Category Archives: Embedded

Resetting Supermicro X10 series BMC to factory defaults

4 Replies

If you’ve ever bought a used Supermicro motherboard and it came without the IPMI login reset to ADMIN/ADMIN, you may be wondering how you can reset IPMI to factory defaults without booting an OS.

Quick note before we continue: if you have an OS on the board, and have installed the IPMI tools for your OS, it’s easier to reset the IPMI username/password via those utilities than via the following method.

This method requires physical access and an SPI programmer like the ch341a or Raspberry Pi. A SOIC16 chip clip will also make life much easier. The ch341a and SOIC16 chip clip can be purchased online for <$10 USD from various sources (e.g. eBay, AliExpress).

Disclaimer: This information is provided without any warranty. Always take multiple physical backups of firmware before performing any modifications. I have only tested this on the Supermicro X10SLE-F motherboard as it is the only Supermicro board I own. However, looking at the REDFISH BMC update image available on Supermicro’s website, this method should be compatible with all X10 series motherboard BMC firmware.

To start, we need to locate the BMC flash. On my X10 board, this is an SOIC16 chip from MXIC with a capacity of 32MB (256MBit).

U53 (SOIC16, 256MBit) contains the BMC firmware, U5 (SOIC8, 128MBit) contains the BIOS

Dump the contents of the BMC firmware using flashrom (using ch341a_spi):

$ flashrom -p ch341a_spi -r BMC.bin

I always dump the flash twice and compare the dumps using a hashing algorithm like sha1 or sha256, to confirm that both dumps are identical.

If they are not identical, check your physical connection to the chip and whether something on the board is receiving power from your SPI programmer.

Using binwalk, find the JFFS2 region. In Supermicro X10 firmwares, this appears to be from 0x100000 to 0x400000:

$ binwalk BMC.bin
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
109381        0x1AB45         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
109541        0x1ABE5         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
109777        0x1ACD1         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
109913        0x1AD59         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
110057        0x1ADE9         Certificate in DER format (x509 v3), header length: 4, sequence length: 12291
112368        0x1B6F0         CRC32 polynomial table, little endian
1048576       0x100000        JFFS2 filesystem, little endian
4194304       0x400000        CramFS filesystem, little endian, size: 11915264 version 2 sorted_dirs CRC 0xD6771DEA, edition 0, 6818 blocks, 1038 files
20971520      0x1400000       uImage header, header size: 64 bytes, header CRC: 0xC5F4666A, created: 2015-10-05 10:52:56, image size: 1537322 bytes, Data Address: 0x40008000, Entry Point: 0x40008000, data CRC: 0x677BDAA8, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: gzip, image name: "21400000"
20971584      0x1400040       gzip compressed data, maximum compression, has original file name: "linux.bin", from Unix, last modified: 2015-10-05 10:49:39
24117248      0x1700000       CramFS filesystem, little endian, size: 5435392 version 2 sorted_dirs CRC 0x43329740, edition 0, 2071 blocks, 309 files

To reset to factory defaults, simply overwrite the JFFS2 region with 0:

$ dd if=/dev/zero of=BMC.bin bs=1 seek=1048576 count=3145728 conv=notrunc

Reflash the modified firmware:

$ flashrom -p ch341a_spi -w BMC.bin

When you power up the board again, the BMC will re-create the JFFS2 region with the default credentials of ADMIN/ADMIN.

Editing the JFFS2 partition instead of overwriting it with zeros seems to invalidate a checksum somewhere, and this causes the BMC to re-initialize the JFFS2 region on the next boot. For that reason, I wouldn’t recommend extracting and editing the JFFS2 region, just zero it out.

Note: you will lose any licensed features in the BMC by resetting it to defaults using this method. However, Peter Kleissner did an amazing job reverse engineering the Supermicro license validation code, and using his work you can generate an IPMI license for your BMC.

With the licensed BIOS upgrade feature of IPMI, you can update the BIOS without ever needing to boot an OS, very handy for when your CPU revision is unsupported by an old BIOS release or if the board happens to have a corrupt BIOS image.

It should also be noted that the Supermicro BIOS updates available from their website appear to be directly flashable to SPI. You’ll lose some SMBIOS information if you use an SPI programmer to write directly to the SOIC8 containing the BIOS, but it can also help resolve some strange issues encountered after the IPMI BIOS upgrade (example below).

I hope this information is useful to anyone trying to get into their Supermicro BMC. Of course, requiring physical access and an SPI programmer is never as easy as resetting the BMC passwords from software and carries some risk that you may corrupt the BMC firmware.

There’s a thread on ServeTheHome about this motherboard if you have any further questions.

IoT sensors and time series databases

Leave a reply

In this article we are going to look at a few uses for low-cost sensors and how they can be combined with a time series database (TSDB) and a web front-end to easily visualize the data. For privacy reasons, I will describe a standalone use case where you have the time series database running on low-cost hardware such as a Raspberry Pi (or Chinese equivalent), so the data never leaves your house and the IoT sensors are not directly exposed to the internet.

Energy monitoring

The Sonoff POW is a $12 wireless relay that includes a power measurement IC capable of measuring energy consumption, voltage, current, etc.

The Sonoff POW is based on the ESP8266, and there are a number of third-party firmwares available which add additional functionality like support for MQTT, InfluxDB, Domoticz, Amazon Alexa, etc. The most popular third-party seems to be ESPurna, which is what I’m using.

ESPurna is not infallible, and does occasionally crash. When that happens, the relay cycles, disrupting power to whatever is connected. Since I’m monitoring things I don’t want to be randomly power cycled, I soldered across the relay to prevent it from shutting off the loads. This turns the Sonoff POW from a wireless relay with energy monitoring to simply an energy monitor. However for my purpose that’s fine.

Bypassing the relay in the Sonoff POW requires opening the case and soldering across the relay. Working on the Sonoff POW should only be done when it is not connected to AC (mains power)! After soldering, you should confirm with a multimeter that you have correctly bypassed the relay and not created a short circuit. There have been several iterations of the Sonoff POW PCB, so I cannot provide universal instructions on how to bypass the relay.

On the latest Sonoff POW hardware I own (purchased in mid-2017), you can bypass the relay by soldering a wire (shown in red) between the relay input and output:

The small gold coloured object is the shunt resistor used to measure the current consumed by the load. To keep the energy monitoring functionality intact, it is important that you only solder after the shunt resistor (to the left), not before (to the right), otherwise the shunt resistor will not be in series with the load and the measured current will be 0.

Environmental monitoring

The Wemos D1 mini is a “mini WiFi board” with a large number of “shields” incorporating various sensors or other expansion options.

I was drawn to the Wemos D1 mini because it is supported by MicroPython as well as ESPurna (though not for my intended use case). Since there are many shields available, you can just stack modules to get the desired functionality instead of messing around on a breadboard or soldering onto protoboard.

The Wemos D1 mini is also cheap, you can buy it from China for under $3 with free shipping (at least to the EU). The modules are also quite inexpensive when ordered from China, as long as you don’t mind waiting 4-6 weeks for delivery.

Since ESPurna only supports the Wemos D1 mini with the relay shield, and I wanted to do temperature/humidity/pressure monitoring, I decided to use MicroPython since it has the lowest barrier to entry. Flashing MicroPython on the Wemos D1 mini wasn’t too complicated, there is a forum thread describing how to flash it.

I created a simple python script to report the temperature and humidity to the InfluxDB server every minute. Overall it works well, the only issue I’ve run into is that there is no watchdog on the ESP8266, so if the urequests.post() fails for some reason (DNS resolution issue, packet loss, alignment of the stars) you have to manually reset the sensor using the reset button on the side.

Since these are just around my apartment, I added a “meatware monitoring” feature. When the POST is in progress, the LED on the Wemos is enabled. For a normal POST, the LED will just blink for around a second. If I walk past a sensor and notice the LED is on solid, I just press the reset button. This is not very “production ready” but I’m only monitoring the temperature and humidity for fun, so the motivation to resolve this bug is not very high. I will accept any pull requests to improve the functionality.

Time series database (TSDB)

Time series databases are a relatively new and hyped type of database, as you can probably gather from how incomplete the Wikipedia page is compared to relational databases.

For my application of 5 sensors reporting values every minute or so, there’s no reason a relational database like PostgreSQL couldn’t be used instead. But it’s helpful to learn a new technology, and InfluxDB offers some benefits over a relational database:

  • Engineered for time series data
  • HTTP API
  • collectd API

These are only scratching the surface of InfluxDB’s features, but the HTTP and collectd APIs reduced the amount of effort needed for this project. Otherwise I would have had to write an HTTP API to accept readings and insert them into a relational database. collectd is also useful to collect performance metrics from devices running Linux or BSD, but that’s beyond the scope of what I want to discuss today.

For the Sonoff running ESPurna, there’s no additional programming required as InfluxDB is supported by default. Simply enter the URL of your InfluxDB server’s HTTP API and wait for the sensors to report readings.

Visualising the data

Now, it’s great that we can send data to InfluxDB with very little effort via the HTTP API. We can of course run queries on the data from the influx cli, however this isn’t very useful for getting a quick impression of the data.

time value
---- -----
1512599873138880467 27.23
1512599933609113738 26.57
1512599994180436248 26.5
1512600054652895777 26.31
1512600115149476539 26.31
1512600175695516312 26.26
1512600256017317051 26.21
1512600316488957374 26.07
1512600376985207309 25.99
1512600437407006181 26.02

To visualise the data, I’ve chosen to use Grafana. Grafana is free software that you can use to visualise data from a variety of data sources such as OpenTSDB, InfluxDB, graphite, elasticsearch, and more.

Coupling Grafana with the InfluxDB data source from the Sonoff and the Wemos, we can build clever dashboards to visualise the sensor data:

Sonoff POW monitoring a fridge and microwave, you can see where the microwave was running

D1 Mini with SHT30 shield monitoring temperature and humidity,
can you see when the window was opened?

Security considerations
I would like to add that for security reasons if you are using any IoT devices at home, I would strongly recommend you consider isolating the devices to a separate WiFi access point and subnet to prevent them from communicating with devices on your main network. ISP supplied routers with a “Guest WiFi” mode should be capable of implementing this. Alternatively you can find inexpensive routers such as the Nexx WT3020H which support OpenWrt/LEDE and could be used to implement this.

You could in theory implement this on an SBC with WiFi supporting AP mode (such as the Orange Pi Zero), negating the need for a separate WiFi AP. However you are either faced with a SBC with very limited resources (the Orange Pi Zero has only 512MB of RAM), or an SBC with higher price than a Raspberry Pi with a WiFi router such as the Nexx WT3020H.

Tying it all together
We’ve looked at sensors, InfluxDB, and Grafana in this article. I haven’t mentioned until now that I’m running all of this on an Orange Pi PC, a small single board computer based on an Allwinner processor. For my use case, this hardware is low-energy, low-cost, and meets the performance needs of InfluxDB and Grafana.

There is nothing preventing you from running all of the above software on a different architecture (e.g. Docker on an x86). I chose ARM purely because I had the hardware available, and it is low power. If you’re building a monitoring system from scratch and your processing needs are not significant, then a SBC like the Raspberry Pi or Orange Pi PC is a very inexpensive server you can use with sensors.

I want to close by leaving some installation instructions if you are interested in implementing this yourself. This article is mostly just to inspire you to do your own projects, and is not a novel application of sensors, databases, or data visualization. So in this case, I will leave some links to other people who have written detailed instructions on how to install and configure InfluxDB and Grafana on ARM.

Debian on WD EX2100: Installation instructions

Leave a reply

In the last article on this topic, I unbricked my Western Digital My Cloud EX2100 NAS and said I would provide instructions on how to install Debian. There are partial instructions for how to replace the stock u-boot with one that is capable of booting Debian from a USB stick, but following those instructions requires slightly more knowledge about how the boot process works and omits some important details.

Now that I have the EX2100 working again, I thought it would be good to provide a set of complete, concise installation instructions for anyone else with this hardware who is interested in running Debian (or another Linux) on it. These instructions are also available on the Doozan forum.

Uart
Before we begin installation of Debian, you will need a working uart connection to the EX2100.

There are two possible methods to connect via uart:

  1. solder a header to JP1 the PCB to expose Rx, Tx, and Gnd. This will require opening the enclosure and removing the PCB, which will void your warranty
  2. Using kapton tape (tweezers, and patience), cover the 3.3V and GND pads on the front of the PCB. Use alligator clips to attach to the Tx contact, the Rx pad from JP1, and the chassis for ground. This method is not as reliable as soldering a header to JP1, but does not require disassembly and soldering as the area can be accessed by opening the hard drive bay doors

Test the connection by powering up the EX2100 with a USB to uart adapter or something like a Raspberry Pi. The uart operates at 115200n8.

You should immediately see output from u-boot (see sample below). If you don’t see any output, check your connections and ensure that you have not reversed the uart Tx/Rx.

Uart output

BootROM - 1.73
Booting from NAND flash

General initialization - Version: 1.0.0
Detected Device ID 6820
High speed PHY - Version: 2.0

Load WD Yosemite Serdes Config:
board SerDes lanes topology details:
| Lane # | Speed | Type |
--------------------------------
| 0 | 06 | SATA0 |
| 1 | 05 | PCIe0 |
| 2 | 06 | SATA1 |
| 3 | 05 | USB3 HOST1 |
| 4 | 05 | USB3 HOST0 |
| 5 | 00 | SGMII2 |
--------------------------------
PCIe, Idx 0: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.26.0
mvSysEnvGetTopologyUpdateInfo: TWSI Read failed
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
DDR3 Training Sequence - Ended Successfully
BootROM: Image checksum verification PASSED

__ __ _ _
| \/ | __ _ _ ____ _____| | |
| |\/| |/ _` | '__\ \ / / _ \ | |
| | | | (_| | | \ V / __/ | |
|_| |_|\__,_|_| \_/ \___|_|_|
_ _ ____ _
| | | | | __ ) ___ ___ | |_
| | | |___| _ \ / _ \ / _ \| __|
| |_| |___| |_) | (_) | (_) | |_
\___/ |____/ \___/ \___/ \__|
** LOADER **

Prepare a USB with the Debian rootfs for mvebu
For this step you will need a USB mass storage device of at least 2GB. We will format the device, so ensure you do not have any important data on it. The device must be USB 2.0 as USB 3.0 is not supported in the EX2100 u-boot.

Format the device with a single ext3 partition. Double check the device before proceeding as parted will erase and format the device without confirmation!

sudo parted --script /dev/sdX \ mklabel msdos \ mkpart primary ext3 1MiB 100%
sudo mkfs.ext3 -L rootfs /dev/sdX1

Download the latest Debian rootfs from the Doozan forums.

Now, mount the new partition and extract the Debian rootfs:

ROOTFSDIR=$(mktemp -d)
sudo mount /dev/sdX1 $ROOTFSDIR
sudo tar -C $ROOTFSDIR -jxvf Debian-4.12.4-mvebu-tld-1-rootfs-bodhi.tar.bz2

You can download the EX2100 DTB from the kernel image Doozan user bodhi publishes. Inside the kernel archive is another archive containing the DTBs (linux-dtb-$(kernelversion).tar). Extract the archive containing DTBs, and append the EX2100 DTB to the kernel:

tar -C $ROOTFSDIR/boot/dts/ -xvf linux-dtb-4.14.1-mvebu-tld-1.tar
cd $ROOTFSDIR/boot/
cat zImage dts/armada-385-wd-ex2100.dtb > zImage.fdt
mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-4.12.4-mvebu-tld-1 -d zImage.fdt uImage

Download u-boot for the EX2100 from here. Extract the archive:

tar -zxvf u-boot-a38x-Yosemite_2014T3_PQ.tar.gz

Copy the file u-boot-a38x-Yosemite_2014T3_PQ-nand.bin to the rootfs:

cp u-boot-a38x-Yosemite_2014T3_PQ-nand.bin $ROOTFSDIR/root/

Unmount the USB device from your computer:

sudo umount $ROOTFSDIR

Put the USB device in the rear USB port on the EX2100. The front USB port cannot be used for booting as it is not powered during u-boot execution.

kwboot the EX2100
Now that you have a serial connection to the EX2100 and a USB device prepared with the Debian rootfs, mvebu kernel, and dtb for the EX2100 it’s time to start installing Debian.

You must use a modified version of kwboot to load u-boot to the EX2100 via uart. You can obtain the modified kwboot binary for amd64 from here, and kwboot for armhf from here.

You will be kwbooting a modified version of u-boot which can save environment variables.

With the EX2100 powered off and unplugged, execute kwboot (note this is one line):

./kwboot -f -t -B 115200 /dev/ttyUSB0 -b u-boot-a38x-Yosemite_2014T3_PQ-nand-uart.bin -s 0 -q 1

Plug the DC power into the EX2100. If the handshake is successful, kwboot should display a loading screen:

Sending boot message. Please reboot the target...-�$�"Ufw�$�"U����$
Dfw�$�"U�\�$�"U����$�DUf�$�"Uw��"U����$4"U���$�"Uw�$�"U���$�DUf|fD�&T���$�"U�E�$�"Df3DD�DU�E7$�"U����$4"U���$�"U�E�4"U�/7@� ��$DUw�$�"U����$�DUff�$�"D��fD$U��
Sending boot image...
0 % [......................................................................]

If the handshake was unsuccessful, you will see normal u-boot output as you should have seen in the “Uart output” section above. Repeat the procedure until you see kwboot say “Sending boot image…” followed by loading u-boot via uart. This takes some time (~90 seconds).

After the u-boot image has been loaded via kwboot, it will boot normally (as you saw in “Uart output” section above).

When the uart output gets to the following section:

Enable HD1
Enable HD2

Begin pressing the 1 key to interrupt the automatic boot process. If you were successful you should now have a u-boot prompt:

Enable HD1
Enable HD2
Net:
| port | Interface | PHY address |
|--------|-----------|--------------|
| egiga0 | RGMII | 0x00 |
| egiga1 | RGMII | In-Band |
| egiga2 | SGMII | 0x01 |
egiga0 [PRIME], egiga1, egiga2
Hit any key to stop autoboot: 0
Marvell>> 1111

Set u-boot environment parameters
Press ctrl+c to clear the extra 1 characters and enter the following:

setenv bootdev usb
setenv device '0:1'
setenv load_initrd_addr 0x2900000
setenv load_image_addr 0x02000000
setenv load_initrd 'echo loading uInitrd ...; ext2load $bootdev $device $load_initrd_addr /boot/uInitrd'
setenv load_image 'echo loading Image ...; ext2load $bootdev $device $load_image_addr /boot/uImage'
setenv usb_set_bootargs 'setenv bootargs "console=ttyS0,115200 root=/dev/sda1 rootdelay=10 $mtdparts earlyprintk=serial"'
setenv usb_bootcmd 'echo Booting from USB ...; setenv fdt_skip_update yes; usb start; run load_image; run load_initrd ; run usb_set_bootargs; bootm $load_image_addr $load_initrd_addr'
setenv bootcmd_usb 'usb start; run usb_set_bootargs; run usb_bootcmd; reset'
printenv
run bootcmd_usb

If you performed the previous steps correctly, the EX2100 should now boot Debian from the USB device attached to the rear USB port.

Make a backup of NAND flash
For the changes in u-boot to be persistent, we need to write the modified version of u-boot to NAND.

However, before we do this, we will make a backup of the contents of NAND before modifying it. When booted into Debian, run the following commands:

mkdir nand_backup
cd nand_backup
nanddump --noecc --omitoob -f mtd{0,7}.bin /dev/mtd{0,7}

Make sure you make a copy these backups also in another location!!!

Installing the modified u-boot
Once you have taken a backup of the NAND contents, poweroff the EX2100. Remove the USB and copy the mtd backups you made to your computer for safekeeping.

When you have finished this, follow the instructions again in the “kwboot the EX2100” section but stop at the “Set u-boot environment parameters” section.

This time we will modify the u-boot environment:

setenv bootdev usb
setenv device '0:1'
setenv load_initrd_addr 0x2900000
setenv load_image_addr 0x02000000
setenv load_initrd 'echo loading uInitrd ...; ext2load $bootdev $device $load_initrd_addr /boot/uInitrd'
setenv load_image 'echo loading Image ...; ext2load $bootdev $device $load_image_addr /boot/uImage'
setenv usb_set_bootargs 'setenv bootargs "console=ttyS0,115200 root=/dev/sda1 rootdelay=10 $mtdparts earlyprintk=serial"'
setenv usb_bootcmd 'echo Booting from USB ...; setenv fdt_skip_update yes; usb start; run load_image; run load_initrd ; run usb_set_bootargs; bootm $load_image_addr $load_initrd_addr'
setenv bootcmd_usb 'usb start; run usb_set_bootargs; run usb_bootcmd; reset'
saveenv
run bootcmd_usb

You will then proceed to boot Debian again.

Once in Debian, create /etc/fw_env.config:

echo “/dev/mtd0 0x100000 0x80000 0x20000 4” > /etc/fw_env.config

Check that fw_printenv is able to read the u-boot environment you just saved in the kwboot’d u-boot:

root@debian:~# fw_printenv
CASset=max
MALLOC_len=5
MPmode=SMP
autoload=no
baudrate=115200
boot_order=hd_scr usb_scr mmc_scr hd_img usb_img mmc_img pxe net_img net_scr
bootargs=root=/dev/ram console=ttyS0,115200
…

If you see the u-boot environment variables returned, then the modified u-boot successfully wrote the environment variables to 0x100000

Verify that you are able to write to the u-boot section of NAND (this should be enabled in the EX2100 dtb):

root@debian:~# mtd_debug info /dev/mtd0
mtd.type = MTD_NANDFLASH
mtd.flags = MTD_CAP_NANDFLASH
mtd.size = 5242880 (5M)
mtd.erasesize = 131072 (128K)
mtd.writesize = 2048 (2K)
mtd.oobsize = 64
regions = 0

If instead you see “mtd.flags = MTD_CAP_ROM” then you cannot flash u-boot using the dtb you have booted with. You can download the dts for the EX2100 and build the dtb for your kernel.

If you saw MTD_CAP_NANDFLASH, then proceed to backup the u-boot environment variables to a file:

nanddump --noecc --omitoob -s 0x100000 -l 0x80000 -f ubootenv.bin /dev/mtd0

Erase the u-boot portion of NAND, flash the modified u-boot, and restore the environment variables:

flash_erase /dev/mtd0 0 8
nandwrite -p /dev/mtd0 u-boot-a38x-Yosemite_2014T3_PQ-nand.bin
nandwrite -p /dev/mtd0 -s 0x100000 ubootenv.bin

That’s it, you should be finished. Shutdown the EX2100 and exit kwboot. Using a standard serial console like screen or minicom, connect to the uart if you want to monitor the boot process.

Now when you power the EX2100 it should boot Debian from the USB device plugged into the rear USB port, if it is present. If the USB device is not present, u-boot will fall back to booting the WD firmware from internal flash.

Note that the Western Digital firmware is not fully functional unless you allow it to format your hard drive(s). If you format the drives with the Western Digital firmware and run Debian from USB, then the device should function regardless of the running firmware. If you choose to partition the drives yourself and usually boot Debian, then the WD firmware won’t be very functional as the hard drive(s) are not formatted in the expected layout.