Category Archives: Hardware

Powering a Supermicro Micro Cloud node

Leave a reply

You can often find good deals online for blade servers, but the critical question is always: can you power it without an expensive and loud chassis?

Usually blade servers have very little IO onboard, relying on the chassis backplane to provide power and communication. In the case of Supermicro, the Micro Cloud series (such as the Supermicro X10SLE-F) are designed to be used in a chassis providing cooling and power, but they have all other IO onboard. Each node has IPMI, Gigabit Ethernet, VGA/RS-232/USB 2 (SUVI cable), USB 3 (onboard), and SATA connections for up to 5 hard drives.

I previously wrote about resetting the IPMI password on the X10SLE-F, which allows you to reset the IPMI to factory defaults (as most eBay sellers don’t know the credentials and/or don’t reset IPMI before shipping). With functional IPMI, you can install an OS without needing the proprietary KVM/SUVI cable from SuperMicro.

There are a few unique aspects to the X10SLE-F I want to discuss, having had one for several months:

  1. It is quite easy to power
  2. Remarkably low power

The power connector on the motherboard is the same physical connector as the standard ATX 12V EPS (8-pin CPU) connector. Pin 5 must be replaced with 5V stand-by. The easiest method is to buy an EPS extension cable and modify the extension cable, with that you can use a normal ATX power supply with EPS output to power the board.

In addition to 5VSB and 12V, there are two pin headers on the motherboard which must be wired to signal that the blade is connected to the backplane. You will need two “dupont” wires of 2.54mm pitch (Arduino/Raspberry PI GPIO style) to wire the following pins between JP17 & JP18:

  • Purple: JP18 pin 1 (5V) -> JP17 pin 2 (GND)
  • Green: JP18 pin 6 (3.3V) -> JP17 pin 5

Additionally, your board must be running a BMC version older than 3.38 as SuperMicro changed the backplane detection method in newer BMC releases. Updating to BMC 3.38 or newer will result in the board not powering up. BMC firmware versions above 3.00 include an HTML5 iKVM viewer instead of the traditional Java-based iKVM applet. Here is an archive of SuperMicro BMC firmware releases.

In total, with no modification to the motherboard, 2 breadboard cables and an EPS extension cable, you can use the X10SLE-F by itself without the chassis:

Power consumption is very low. I measured 13W idle on a system with the following specs:

  • Intel Xeon E3-1220v3
  • 2x4GB PC3-10600E
  • SATA SSD (OpenWrt x86)

At 13W consumption, the BMC booted and 2 Ethernet connections are active (BMC and one Gigabit Ethernet).

The BMC alone seems to draw ~6W when booted with the dedicated BMC Ethernet connected. With a high efficiency power supply like the Dell DA-2, the idle consumption of the server will be well under 20W. I have never seen an x86 server idle below 20W, with the Dell R210 II only going as low as 23W with all settings to minimum (CPU speed, memory clock, no iDRAC Ethernet).

Using a CPU such as the Intel E3-1220L V3, you could easily power this board with a 12V power source that can supply 40W. For higher TDP CPUs like the E3-1225 V3, I would recommend a 12V power supply of at least 100W.

The X10SLE-F supports Xeon processors and ECC memory, making it a great low-power NAS for up to 5 hard drives. With up to 32GB of DDR3 ECC and a quad-core processor, you can run ZFS or have a low-power hypervisor.

There’s a thread on ServeTheHome about this motherboard if you have any further questions.

Resetting Supermicro X10 series BMC to factory defaults

4 Replies

If you’ve ever bought a used Supermicro motherboard and it came without the IPMI login reset to ADMIN/ADMIN, you may be wondering how you can reset IPMI to factory defaults without booting an OS.

Quick note before we continue: if you have an OS on the board, and have installed the IPMI tools for your OS, it’s easier to reset the IPMI username/password via those utilities than via the following method.

This method requires physical access and an SPI programmer like the ch341a or Raspberry Pi. A SOIC16 chip clip will also make life much easier. The ch341a and SOIC16 chip clip can be purchased online for <$10 USD from various sources (e.g. eBay, AliExpress).

Disclaimer: This information is provided without any warranty. Always take multiple physical backups of firmware before performing any modifications. I have only tested this on the Supermicro X10SLE-F motherboard as it is the only Supermicro board I own. However, looking at the REDFISH BMC update image available on Supermicro’s website, this method should be compatible with all X10 series motherboard BMC firmware.

To start, we need to locate the BMC flash. On my X10 board, this is an SOIC16 chip from MXIC with a capacity of 32MB (256MBit).

U53 (SOIC16, 256MBit) contains the BMC firmware, U5 (SOIC8, 128MBit) contains the BIOS

Dump the contents of the BMC firmware using flashrom (using ch341a_spi):

$ flashrom -p ch341a_spi -r BMC.bin

I always dump the flash twice and compare the dumps using a hashing algorithm like sha1 or sha256, to confirm that both dumps are identical.

If they are not identical, check your physical connection to the chip and whether something on the board is receiving power from your SPI programmer.

Using binwalk, find the JFFS2 region. In Supermicro X10 firmwares, this appears to be from 0x100000 to 0x400000:

$ binwalk BMC.bin
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
109381        0x1AB45         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
109541        0x1ABE5         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
109777        0x1ACD1         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
109913        0x1AD59         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
110057        0x1ADE9         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
112368        0x1B6F0         CRC32 polynomial table, little endian
1048576       0x100000        JFFS2 filesystem, little endian
4194304       0x400000        CramFS filesystem, little endian, size: 11915264 version 2 sorted_dirs CRC 0xD6771DEA, edition 0, 6818 blocks, 1038 files
20971520      0x1400000       uImage header, header size: 64 bytes, header CRC: 0xC5F4666A, created: 2015-10-05 10:52:56, image size: 1537322 bytes, Data Address: 0x40008000, Entry Point: 0x40008000, data CRC: 0x677BDAA8, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: gzip, image name: "21400000"
20971584      0x1400040       gzip compressed data, maximum compression, has original file name: "linux.bin", from Unix, last modified: 2015-10-05 10:49:39
24117248      0x1700000       CramFS filesystem, little endian, size: 5435392 version 2 sorted_dirs CRC 0x43329740, edition 0, 2071 blocks, 309 files

To reset to factory defaults, simply overwrite the JFFS2 region with 0:

$ dd if=/dev/zero of=BMC.bin bs=1 seek=1048576 count=3145728 conv=notrunc

Reflash the modified firmware:

$ flashrom -p ch341a_spi -w BMC.bin

When you power up the board again, the BMC will re-create the JFFS2 region with the default credentials of ADMIN/ADMIN.

Editing the JFFS2 partition instead of overwriting it with zeros seems to invalidate a checksum somewhere, and this causes the BMC to re-initialize the JFFS2 region on the next boot. For that reason, I wouldn’t recommend extracting and editing the JFFS2 region, just zero it out.

Note: you will lose any licensed features in the BMC by resetting it to defaults using this method. However, Peter Kleissner did an amazing job reverse engineering the Supermicro license validation code, and using his work you can generate an IPMI license for your BMC.

With the licensed BIOS upgrade feature of IPMI, you can update the BIOS without ever needing to boot an OS, very handy for when your CPU revision is unsupported by an old BIOS release or if the board happens to have a corrupt BIOS image.

It should also be noted that the Supermicro BIOS updates available from their website appear to be directly flashable to SPI. You’ll lose some SMBIOS information if you use an SPI programmer to write directly to the SOIC8 containing the BIOS, but it can also help resolve some strange issues encountered after the IPMI BIOS upgrade (example below).

I hope this information is useful to anyone trying to get into their Supermicro BMC. Of course, requiring physical access and an SPI programmer is never as easy as resetting the BMC passwords from software and carries some risk that you may corrupt the BMC firmware.

There’s a thread on ServeTheHome about this motherboard if you have any further questions.

Western Digital Weltrend MCU

1 Reply

When we left off in my last post about the Western Digital EX2100, I had corrupted the firmware on the microcontroller responsible for managing mundane system tasks like bringing the main ARM SoC out of reset. This unfortunate bout of curiosity had left me with a fancy brick instead of a NAS.

To recap from the last article, the EX2100 uses a Weltrend WT61P8 series MCU (despite being labelled WT69P803) intended for use in flatpanel televisions. The WT61P8 has a turbo 8052 CPU and is primarily intended for flatpanel television power management as it supports HDMI CEC and Infrared.

Finding the ISP

Consulting the WT61P8 datasheet (PDF) I was able to trace the headers JP3 and JP4 on the EX2100 PCB:

JP3:
Pin 1: Uart Tx (WT61P8 pin 32)
Pin 5: Uart Rx (WT61P8 pin 31)

JP4:
Pin 1: SCL (WT61P8 pin 23)
Pin 5: SDA (WT61P8 pin 22)

If I can be grateful for one thing, it is that this microcontroller is frequently used in Samsung televisions and Russians are very resourceful at repairing and modifying televisions. Through a variety of Russian language forums and with a lot of Google translate, I was able to find that the WT61P8 supports ISP via I2C and that there is software available to flash it. The “Postal2” software is designed for dumping and flashing firmware from televisions, and as the WT61P8 is used in televisions it is supported.

Now my options were either:

  1. Postal2, which is intended to be used with a parallel port to I2C adapter
  2. RT809F ISP programmer, which costs around $70

Since I did not really want to spend money on the RT809F, having no other use for it, I decided to try Postal2.

A trip back to the 1970s

However I faced a problem: Postal2 requires an LPT to I2C adapter, and this is not an adapter you’ll find anyone selling on AliExpress or eBay, so I would have to make it myself.

After searching for schematics, and building one non-functional adapter, I was able to find a correct schematic for an LPT to I2C adapter that people were successfully using with Postal2:

Use 3.3V instead of 5V as the WT61P8 operates at 3.3V

Luckily I already had some CMOS inverters from another project, so I only had to order resistors and an LPT connector. After much soldering, and a trip down nostalgia lane to another era of computing, I finished the adapter:

The micro USB port is for power, with a 5V to 3.3V linear regulator. An equal mix of modern and ancient connectors are present in this adapter.

Then I had to find a PC with a parallel port and install Windows 7 x86. Postal2 seems to be completely incompatible with all x64 versions of Windows.

Finally finished with the prerequisite tasks, I started Postal2 and tried to read the current EEPROM contents from the Weltrend using the SDA/SCL lines on JP4 within the EX2100:

It works!

Examining the downloaded firmware with the firmware file I managed to save from the EX2100 as it was updating, it was clear to me what had happened:

After analyzing the downloaded firmware, it appears mcu_upgrade put the Weltrend into ISP mode and erased a majority of the internal EEPROM, leaving [I assume] only the bootloader. For some reason, mcu_upgrade did not write the firmware back to the Weltrend after erasing. I won’t speculate if my strace on mcu_upgrade was responsible for it… When I reset the EX2100 the Weltrend had no firmware to run, and didn’t bring the main Armada CPU out of reset, resulting in a brick.

Since the bootloader portion of the firmware I dumped and the binary firmware I had from the EX2100 update were the same, I felt confident to upload the firmware binary to the Weltrend using Postal2. My logic: cannot be any more broken than it is now ¯\_(ツ)_/¯

I would not consider the result a complete success. The Weltrend did have a valid firmware again, and it was functional enough to bring the main Armada CPU out of reset. Unfortunately (or predictably) the binary firmware I recovered from the Western Digital update package does not appear to be specifically for the EX2100.

Here is the uart output of the Weltrend during boot before I bricked it:
Yosemite Uart test
nick 2222
nick pwr on
Check_SYSTEM_Command=11
PwrOnCause=10

And here is the uart output after flashing the uP_0.bin I extracted from the EX2100 update archive:
nick 4444
Check_SYSTEM_Command=11
PwrOnCause=00
nick pwr off wol

The codename for the EX2100 is “Yosemite” and unfortunately this string does not exist in the binary firmware uP_0.bin I extracted from the EX2100 firmware update. Looking at the strings in the firmware, it appears that the firmware is intended for the “MyCloudDL2000” (a product that doesn’t exist). There is a DL2100 which is a “Pro” version of the EX2100, using an Intel CPU instead of the Marvell Armada.

Unfortunately the behaviour of the Weltrend with the extracted firmware is different enough that u-boot hangs trying to configure wake-on-LAN for the two Ethernet interfaces. Since u-boot doesn’t get far enough to boot Linux, my next task was stripping out all communication with the Weltrend from the u-boot source to see if I could at least get the NAS to boot Linux.

After building u-boot with WoL configuration disabled, I was able to kwboot u-boot and successfully boot into the Western Digital firmware.

In another stroke of luck, Western Digital has published a new firmware (2.30.181) to resolve a vulnerability discovered in their web management interface. Updating the Western Digital firmware on my EX2100 also updated the firmware on the Weltrend. The firmware update process resolved the issue with u-boot WoL and I had a fully functional unit again.

Obtaining the Weltrend firmware
I was mystified where the firmware from the Weltrend microcontroller came from. There is no file matching the contents of “uP.bin” or “uP_0.bin” in the filesystem of the firmware update, nor is there any firmware blob in the WD GPL archive that I could find.

After much searching in the squashfs filesystem of the update, and through the WD GPL archive, I finally gave up and grepped for a hex string in the firmware in the firmware update binary itself. At this point I was very surprised to find that the Weltrend firmware is simply appended to the end of the WD firmware update.

The firmware is under 50KB, as the size of the flash in the WT61P803 is only 49152 bytes. Chopping the last 50KB of the firmware update file with dd allows you to easily find the offset of the Weltrend firmware with hexdump and grep. The firmware starts with 02 5d a9 which is a long jump to address 0x5da9:


hexdump -C data.bin | grep 02 5d a9

With the offset from hexdump, you can again chop the file with dd to isolate only the Weltrend firmware. There doesn’t appear to be any data in the firmware update file after the Weltrend firmware, so there’s no need to remove any trailing data.

Firmware analysis

I was interested in reverse engineering the Weltrend firmware to determine what tasks it was performing. I know that the Weltrend is responsible for at least:

  1. System power to internal components (Armada CPU, system fan)
  2. Enabling or disabling SATA hard drive power
  3. Reading the ambient temperature sensor
  4. Controlling the system fan speed (PWM controlled)
  5. Power LED (colour, blink rate)

I asked a friend with IDA Pro to disassemble the binary to see if it was possible to isolate specific functions that would give insight into the tasks the Weltrend is performing. Unfortunately, IDA didn’t produce anything immediately obvious. But with some searching, we were able to identify some portions of the firmware, such as the UART command reply method:

While I dislike an opaque microcontroller being used for important system management functions, I’m not prepared to invest a lot of time in learning 8052 assembly so I can reverse engineer the firmware.

Conclusion

If you have a Western Digital NAS using a Weltrend microcontroller and it appears to be bricked, there is route to recovering the firmware and it’s not as impossible as it may seem at first glance. With access to the SDA/SCL header and an LPT adapter can be built from inexpensive components you can flash the Weltrend firmware extracted from the WD update file.

In the future I think it is best if I stick to the task at hand (writing Debian installation instructions) and refrain from messing with proprietary and undocumented microcontrollers responsible for key system functions like power management.

Stay tuned, installation instructions are coming soon!