Category Archives: Linux

Debian on WD EX2100

The Western Digital My Cloud EX2100 is a dual-bay NAS based on the Marvell Armada 385 dual core ARMv7 CPU first released in 2015.

In terms of NAS devices available in 2017, it isn’t very special. I would say the only major differences between most other devices in the 2 bay category are:

  • Dual Gigabit Ethernet
  • Screwless and trayless hard drive installation

The dual GigE interfaces are what attracted me to the device over competitors like the Zyxel NAS326 and Western Digital’s own MyCloud EX2 Ultra.

Unlike some other NAS bundles, it’s possible to buy the EX2100 without drives, so you can add your own preferred 3.5″ SATA hard drives. I bought a refurbished unit for 110€, which seems typical for a device with these features. For some reason the resale price of these units has skyrocketed since I bought mine in mid-2017. I personally would not pay more than 150€ for such a device. If you get into the higher price range of these SOHO devices, you’re almost always going to get better value for your money building your own NAS using standard x86 components (such as the HP MicroServer G7/Gen 8/Gen 10) and a distribution like FreeNAS or OpenMediaVault.

Since the vendor supplied firmware is almost always a pile of unsightly hacks, I set to work investigating into how to put a better operating system on the EX2100. If you stick around to the end, you’ll see this particular product also has its share of unfortunate hardware design decisions…

kwboot
Before we get into anything about u-boot or the operating system, we need to talk about kwboot.

kwboot stands for “Kirkwood boot”

Kirkwood is an ARMv5 SoC from Marvell around 2008-2009 that started out in the SheevaPlug (what single board computers were before the creation of the Raspberry Pi) and sooner or later found its way into a lot of NAS devices like the D-Link DNS320 and the Zyxel NSA320.

Coming back to the near-past (2015), and we have the Western Digital EX2100/4100 which use the Marvell Armada 385/388 CPU, which is a dual-core ARMv7 design. However it was known that the Armada SoC could boot from serial because of the SolidRun ClearFog. But the ClearFog uses DIP switches to set the boot source, and most (all?) consumer devices lack these.

ClearFog Pro boot source selection DIP switches

It turns out it is possible to kwboot consumer devices based on the Armada 38x, however you need to apply this patch to kwboot to parse the response from the Armada CPU, which differs from the Kirkwood response. Unfortunately the patch broke Kirkwood compatibility, and was seemingly never merged into u-boot mainline. However, you can still apply it to the kwboot source in u-boot and compile kwboot for use with Armada CPUs.

Once you have patched kwboot, you can use it to test new versions of u-boot via a USB to uart adapter:
$ ./kwboot -f -t -B 115200 /dev/ttyUSB0 -b u-boot-uart.bin -s 0 -q 1

There are some synchronization issues with the magic sequence, so it often takes several attempts before successfully loading via kwboot. A dead giveaway that you need to power cycle the device and try again is when you immediately see u-boot output in the console instead of “Sending boot image…”

A successful attempt should look similar to the following:
$ ./kwboot -f -t -B 115200 /dev/ttyUSB0 -b u-boot-a38x-Yosemite_2014T3_PQ-nand-uart.bin -s 0 -q 1
Sending boot message. Please reboot the target...-�$�"Ufw�$�"U����$
Dfw�$�"U�\�$�"U����$�DUf�$�"Uw��"U����$4"U���$�"Uw�$�"U���$�DUf|fD�&T���$�"U�E�$�"Df3DD�DU�E7$�"U����$4"U���$�"U�E�4"U�/7@� ��$DUw�$�"U����$�DUff�$�"D��fD$U��
Sending boot image...
0 % [......................................................................]

Once kwboot works, you can safely proceed to testing u-boot modifications without the risk that you brick your device, as kwboot runs code in memory without modifying the contents of NAND.

u-boot
Unfortunately mainline u-boot doesn’t support this device, although similar devices are supported, such as the Turris Omnia (Armada 385) and Solidrun Clearfog Pro (Armada 388). It’s no surprise that attempting to kwboot a build of mainline u-boot for these targets on the EX2100 doesn’t work. So currently we have no choice but to use the u-boot source from Western Digital’s GPL archive.

The stock u-boot on the device does not support saveenv. Without modifying NAND, it is possible to boot Linux from USB, however this requires using the uart console and manually entering the boot parameters on each boot.

Naively modifying the WD u-boot source to enable the saveenv command results in corruption of the kernel uImage since someone at WD set the environment offset to 5MB and this is beyond the u-boot partition, corrupting the uImage.

However it is possible to modify the WD u-boot source to save environment variables within the 5MB allocated for u-boot. This requires reflashing u-boot to the device. Before you replace the stock u-boot on your device, you should take a backup of the u-boot region of flash. This can be done from within the Western Digital firmware, but requires a USB to UART adapter and a header soldered to the PCB:
# nanddump --noecc --omitoob -f mtd0.bin /dev/mtd0

Remember to copy this file somewhere off-device, such as a USB key, for safe keeping!

The general steps to replace the stock u-boot are:

  1. kwboot a modified u-boot which saves environment variables within u-boot region
  2. Inside u-boot, run saveenv
  3. Boot Debian from USB or SATA
  4. Dump u-boot env to a file (using nandread)
  5. Erase u-boot portion of mtd0 (using flash_erase)
  6. Flash new u-boot (using nandwrite)
  7. Restore u-boot environment variables (using nandwrite)
  8. Reboot

Dump the u-boot env to a file:
# nanddump -s 0x100000 -l 0x80000 -f ubootenv.bin /dev/mtd0

Erase the u-boot portion of mtd0 and flash the new u-boot to NAND:
# flash_erase /dev/mtd0 0 8
# nandwrite -p /dev/mtd0 u-boot-a38x-Yosemite_2014T3_PQ-nand.bin

Restore u-boot environment variables:
# nandwrite -p /dev/mtd0 -s 0x100000 ubootenv.bin

Integrated MCU
Western Digital decided to use an external microcontroller to handle certain system management functions such as fan control, LED control, and power on/off.

Sadly the microcontroller uses a proprietary and undocumented protocol for communication, and as it turns out this protocol can differ even between Western Digital products!

For the Western Digital EX2100 and EX4100, the integrated microcontroller communicates on ttyS1 at 115200n8, unlike other Western Digital NAS products whose microcontroller communicates at 19200n8.

Thankfully, some of the commands are common, so once communication with the microcontroller has been established, fan control and temperature monitoring are functional. Fan control and temperature monitoring are available through a userspace daemon called “mcm-daemon” (MyCloud Mirror daemon). I have forked the mcm-daemon repository on GitHub and made modifications to support the EX2100/4100.

LED control and power on/off are still a work in progress as the reverse engineered commands used on other WD products do not work on the EX2100/4100.

Debian
The user bodhi at Doozan forums does a great job of providing pre-built Debian images for a variety of Marvell Armada based NAS devices.

Usually I would link to the excellent instructions bodhi normally writes for installing Debian, but since they don’t have the EX2100, writing the instructions fell to me.

But sadly I haven’t got installation instructions written because I’ve bricked my EX2100.

Weltrend WT61P8
Let’s revisit this mystery microcontroller in charge of so many tasks in the EX2100.

Well, after reading that Western Digital My Cloud products contained a backdoor and multiple vulnerabilities I thought I would go and update the WD firmware before continuing to write the Debian installation instructions. The WD firmware resides on the built in 512MB of EMMC, while Debian lives on a USB device, so the installation of Debian does not replace the original WD firmware.

During the WD update process I noticed that it was also updating the firmware of the Weltrend:

16479 root 6624 S /var/www/cgi-bin/system_mgr.cgi
16480 root 2560 S sh -c cd /usr/local/upload/;upload_firmware -a -n 'nas-new-firmware' >/dev/null;cd /
16481 root 4544 S upload_firmware -a -n nas-new-firmware
19436 root 49120 R mcu_upgrade -r -f /tmp/uP_0.bin
19552 root 2720 R ps ax

How curious! Since the Weltrend is very undocumented I was eager to learn more about the firmware it runs.

I found out a good deal more than I’d expected. Firstly, the mcu_upgrade binary contains some interesting strings. Here is a short sample of strings in the binary:

WT61P8
Enable ISP
Set ISP
Erase
Page Erase
Program
Set Address High Byte
Finish

The MCU firmware also has some very interesting strings. Here is a short sample of strings in the firmware:

nick 1111
nick 2222
nick 3333
nick 4444
MyCloudDL2000
Cannot Copy
from Camera
Cannot Move
Storage
Almost Full
Limit Reached
nick pwr on
nick pwr off 1
nick pwr off wol
Welcome to
RTC_ALARM pwr on

Googling the part number “WT61P8” lead to a very interesting datasheet (PDF) describing the microcontroller in detail.

What I found from the datasheet was… not anything I expected to find.

It’s a Turbo 8052 CPU with ~48KB of built-in EEPROM (this is my guess based on the part number and size of firmware mcu_upgrade was sending) and it’s a “Flat Panel Display Control Sub-MCU”

Most information about Weltrend microcontrollers is on Russian language forums dedicated to TV repair. The most common use of this MCU is in Samsung TVs for power management, since it includes an IR receiver and HDMI CEC capabilities.

They do support ISP (In-system Programming) via I2C, if you have the right hardware. There are quite a few Russian articles and YouTube videos on how to program these chips in TVs.

Conclusion

After reading about WD’s numerous firmware vulnerabilities and a back door, which were also present in D-Link NAS products (implying a shared code base or same third party contractor), and then learning that the microcontroller in charge of power management for the EX2100 (and other My Cloud products) was intended for power management in LCD TVs:

My final $0.02: this thing is an utter bodge job in both hardware and software! Don’t buy one of these. It doesn’t matter that can be persuaded to run Debian, it’s terrible value for the price.

You’re far better off getting an older PC and running FreeNAS or OpenMediaVault. Older corporate tower PCs with 2nd or 3rd gen Intel processors like the Dell Optiplex line can easily be purchased for under $150 from places like eBay.

If power consumption is really important to you, then I would recommend something like the Rock64 which has Gigabit Ethernet and excellent USB 3.0 performance with Armbian. It also comes with more RAM than the EX2100 (1/2/4GB while EX2100 has only 1GB) and is a quad core aarch64 instead of dual core armv7!

Best of all, an older PC or Pine64 is going to be cheaper than the EX2100 (or ludicrously more expensive 4 bay EX4100) anyway.

If I ever manage to restore the Weltrend firmware rest assured there will be a follow up article with both the journey of unbricking and instructions to install Debian. Until then, I’m going to take the HDDs I planned to use in the EX2100 and build a FreeNAS in an old PC. ✌️

Disabling Secure Boot on Intel Quark “secure SKU” silicon

Secure Boot is a bit like SELinux: people who use it really like it, and tell all their friends to use it. For everyone else, apart from those who don’t know about or even notice Secure Boot, it’s an annoyance that they almost immediately disable.

We’ve looked at the Intel DK200 from a hardware perspective before. Now it’s time to look at it from a software perspective. “Internet of Things Gateway” is pretty generic, so what can it actually do?

Following the instructions, I tried to register the system on Intel’s website so I could download the Wind River Intelligent Device Platform XT 2.0 SDK. I didn’t get very far:

No WindRiver SDK for you

Stormtrooper #1: This is not the product you’re looking for

Yeah… I guess this is what Mouser meant when they said the DK200 was End of Life.

Since this ships with the Linux Kernel, which is GPLv2 licensed, I believe Intel may be violating the GPL. Specifically:

Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange

But I am not a lawyer, and I am not really that interested in starting a legal battle over the source code for an ancient version of Wind River Linux I am not interested in using anyway.

So let’s go try to build Yocto. The Intel rep did say there was a Yocto BSP coming “soon” but “soon” in Intel time seems kind of variable.

After some hiccups (Yocto needs python2 and GCC <6) I had built a Yocto image and put it on an SD card. Does it boot?

...no

…no

So we can’t boot Yocto because this is a “secure SKU” which means Secure Boot is enabled. Is there some way we can disable Secure Boot? What about updating the BSP to a newer version with Secure Boot disabled?

Back to hardware
If I’ve learned anything from messing around with electronics, you want to make a backup before you start modifying things. This is doubly so if the data in question is related to the booting process. It sucks to end up with a brick, so make a backup!

Taking a backup of flash

Taking a backup of flash

The Intel Quark guide mentions using a Dediprog SF100 to flash EDKII. I don’t have a Dediprog, but I do have an SPI programmer. Unfortunately, none of the Intel documentation I could find mentions the Dediprog header on the DK200, so I had to go hunting.

I traced the pins from the Winbond flash to header J23. J23 is only 8 pins, so trial and error with a multimeter to find the pin mapping wasn’t terrible:

J23 pinout

J23 pinout

Here’s the pinout of J23 in text form:

J23 pin 25Q64 pin Pin description
1 8 VCC
2 4 GND
3 1 /CS
4 6 CLK
5 2 DO
6 5 DI
7 Not connected
8 Not connected

/WP and HOLD pins on the 25Q64FV are not routed to J23, but they aren’t required for flashing.

With the pinout known, I could attach the SPI programmer to the header instead of using the chip clip:

J23 to ch341a SPI programmer

J23 to ch341a SPI programmer

I took a dump of the Winbond 25Q64FV and then for good measure desoldered the chip and read it again to confirm the images were exactly the same. It was strange because the image from the chip clip wasn’t identical. But, the image from the desoldered chip was identical to the image taken from J23, so we’re done here. I wrote the image to a new 25Q64FV and soldered that back onto the board.

Firmware disassembly
Disassembling the firmware which shipped on my DK200, we see that a Secure Boot certificate was created by WindRiver.

I assume that had I been able to download the WindRiver SDK, I would have been able to build and sign Secure Boot with my own certificates. Given that industrial customers spend a lot of time and money worrying about security, I was surprised to see that the Secure Boot certificate in the firmware was created by WindRiver China.

I did try to load up the image in IDA, but not being a power user of IDA, I couldn’t figure out how to get it to analyze the SPI dump, and gave up to try and compile the firmware from source.

Building the BSP

Being Intel, there are hundreds of pages you can read about developing for EDK2 and other really fun things, probably. I didn’t read them.

A document which I did end up reading religiously was the Intel ® QuarkTM SoC X1000 Board Support Package (BSP) Build and Software User Guide [PDF] which describes how to build all the firmware components needed to bring up the X1000 SoC. I found out there is actually a newer version of this document (1.2.1 instead of 1.1) and there are some important differences between the documents I want to get to later.

By building the firmware, we’re hoping for one of two outcomes:

  1. A firmware with our own Secure Boot certificates, or
  2. A firmware which has Secure Boot disabled

Version 1.1 of the BSP Build and Software User Guide includes a section on pages 29 and 30 on how to bundle your own db, kek, and pk certificates:

Page 29 and 30 condensed

Unfortunately if you follow the instructions and try to use a layout.conf which specifies these files, you’ll get an error because there’s no address specified for this data in the image:

I do have a reference file from WindRiver with Secure Boot certificates, so if I was really interested in making Secure Boot work as intended, I could have reverse engineered the address to store the certificates.

The certificates section of layout.conf was removed from the 1.2.1 revision of the BSP Build and Software User Guide. I guess since it no longer works, Intel decided to remove it from the documentation.

So, we can’t install our own Secure Boot certificates in the firmware. What happens if we just leave out the certificates section entirely and build it?

Error 37: Quark signature file not found

Right, so even though there’s now no certificate in the firmware bundle, we still can’t boot.

Interestingly, if you don’t partition the uSD or USB stick correctly, you end up with this pretty screen:

I never saw that in the stock firmware.

Hacking GRUB
So it seems that we can’t include our own Secure Boot certificate in the firmware, due to the sample layout.conf file missing the certificates section, and not knowing the appropriate address to store the certificates.

What if we dig into Error 37: Quark signature file not found a bit more?

If you look in the grub source code included in the BSP, you can see a giant ~1000KB patch that Intel has made to the original upstream code to support the Quark platform.

If you grep for “Quark signature file not found” you’ll find it was added in stage2/common.c:
diff --git a/stage2/common.c b/stage2/common.c
index e96bec2..e122745 100644
--- a/stage2/common.c
+++ b/stage2/common.c
@@ -88,6 +88,8 @@ char *err_list[] =
[ERR_UNRECOGNIZED] = "Unrecognized command",
[ERR_WONT_FIT] = "Selected item cannot fit into memory",
[ERR_WRITE] = "Disk write error",
+ [ERR_QUARK_VERIFICATION] = "Quark signature verification failed",
+ [ERR_SGN_FILE_NOT_FOUND] = "Quark signature file not found",
};

If you grep for ERR_SGN_FILE_NOT_FOUND you’ll find it’s in the following files:
./work/efi/ia32/loader/linux.c:410: errnum = ERR_SGN_FILE_NOT_FOUND;
./work/efi/ia32/loader/linux.c:732: errnum = ERR_SGN_FILE_NOT_FOUND;
./work/efi/quark/boot_settings.c:190: errnum = ERR_SGN_FILE_NOT_FOUND;

Going back to Intel’s modifications to grub, we can see what they added:

It takes a bit of searching, but if you strip out all of the grub_quark_secure logic from linux.c and boot_settings.c, you end up with…

Ta-da! I can boot Yocto Linux

No more Secure Boot!

At the end of the day, the Quark X1000 is an x86: “secure SKU” is nothing but a fuse setting.

The comment should read:

Determine whether or not grub should enforce Secure Boot.

In our case, this is not a mandatory option 😉

Special offer for DK200 owners
As shown above, it is possible to modify the Intel sources to disable Secure Boot. If there are other people have a DK200 from Intel and are interested in running a firmware without Secure Boot, leave a comment with your contact details. Upon request, I can provide a firmware image* with generic Ethernet MAC addresses for you to flash. Note that this firmware is specific to the DK200 (Clanton Hill) hardware.

* No warranty, express or implied, provided for said firmware image. You flash at your own risk!

Allwinner H2+/H3 Ethernet with Linux 4.9-rc8

The Orange Pi PC is not a new single board computer. It’s been released for over a year now, but has mostly been stuck on a heavily patched 3.4 release kernel.

There have been ongoing efforts since the release to have support for the Allwinner H3 in the mainline kernel. In the past weeks there have been new patches released which enable support for the Ethernet MAC on the H3 (and H2+).

Unfortunately this support is not in mainline yet, and won’t make it in the upcoming 4.9 release. However, that doesn’t stop you from taking the patches and applying them against 4.9 yourself.

I wrote a script to compile the kernel from source, applying the necessary patches to the kernel and using a minimal .config file which compiles the sun8i_emac support as a module. You can download the build script from GitHub.

It does try to be somewhat smart: verifying the integrity of the downloaded files, and will bail out if there are errors in patching the source code. But, it doesn’t do toolchain dependency checking because that’s just too complicated. Since the emac support will end up in mainline soon, I doubt it’s worth the time to improve the build script. However if anyone is interested in improving it, the script is released as GPLv2.

The result? If you are patient enough to wait for the kernel to compile, you get a uImage and modules for Linux 4.9-rc8 with Ethernet support:

Allwinner H3 emac performance:

orangepi@OrangePI:~$ iperf -n 1024M -c 192.168.1.150
————————————————————
Client connecting to 192.168.1.150, TCP port 5001
TCP window size: 43.8 KByte (default)
————————————————————
[ 3] local 192.168.1.206 port 42572 connected with 192.168.1.150 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-100.6 sec 1.00 GBytes 85.4 Mbits/sec

Orange Pi Zero (Allwinner H2+):

I’ve also just received my Orange Pi Zero and confirmed that the same kernel works on the Orange Pi Zero, so you can run Linux 4.9 on the Orange Pi PC (Allwinner H3) or Orange Pi Zero (Allwinner H2+) with Ethernet support.

Allwinner H2+ emac performance:

orangepi@OrangePI:~$ iperf -n 1024M -c 192.168.1.150
————————————————————
Client connecting to 192.168.1.150, TCP port 5001
TCP window size: 43.8 KByte (default)
————————————————————
[ 3] local 172.16.4.206 port 54762 connected with 192.168.1.150 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-95.9 sec 1.00 GBytes 89.5 Mbits/sec

Download:
If you want to skip compiling the kernel yourself, I’m providing the kernel uImage and modules here.

I am using u-boot v2016.11 from denx.

Note: HDMI isn’t working on my Orange Pi PC, and since I run it headless I’m not interested in investigating why. If you’re using your Orange Pi PC with HDMI output, you may need to modify the kernel .config file to fix HDMI.