Tag Archives: u-boot

Meraki MX80: buildroot firmware

Ten years ago, before ARM was in everything, many embedded systems that did not justify using an x86 used PowerPC (or MIPS). This leads us to today’s subject: the Meraki MX80, an “enterprise security appliance.”

Meraki MX80 marketing image

Meraki MX80

The MX80 is End-of-Life (EOL) and can be purchased quite inexpensively on eBay.

The MX-series differs from other Meraki networking products of the 2010-era that we have previously covered (the MS220) in that it is PowerPC based (APM86290) with 2GB of RAM and 1GB of NAND flash. The factory bootlog of the device can be found in this GitHub gist.

MX80 PCB with uart header highlighted

MX80 UART header

Removing the cover allows you to connect to the UART header J3 (57600n8), with the pinout:

  1. VCC (don’t connect)
  2. Rx
  3. Tx
  4. GND

Ground (pin 4) is closest to the Ethernet ports.

Obtaining a root shell is very easy as u-boot has a 1 second boot delay and accepts input on UART. The default meraki_boot target sets the bootargs from meraki_bootargs which appends extra_bootargs, so just override rdinit with /bin/sh to prevent the Meraki OS from booting.

setenv extra_bootargs rdinit=/bin/sh
run meraki_boot

Once the MX80 has booted, bring up eth0 (port label Internet on the MX80):

$ mount -t proc proc /proc
$ mount -t sysfs sysfs /sys
$ ifconfig lo up
$ ifconfig eth0 up
$ udhcpc eth0

Once you have functional networking, you can dump the contents of NAND to a remote host for further analysis:

$ cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00100000 00020000 "firmware"
mtd1: 00100000 00020000 "environment"
mtd2: 00040000 00020000 "oops-old"
mtd3: 3fdc0000 00020000 "ubi"
mtd4: 40000000 00020000 "all"
mtd5: 0201d800 0001f800 "part1"
mtd6: 0201d800 0001f800 "part2"
mtd7: 0001f800 0001f800 "board-config"
mtd8: 2001f000 0001f800 "storage"
$ dd if=/dev/mtdblock4 bs=1M | gzip -c | nc -l -p 5000

Running binwalk on “part1” shows us the structure of Meraki’s firmware:

$ binwalk part1.bin

1024          0x400           device tree image (dtb)
16384         0x4000          device tree image (dtb)
131072        0x20000         uImage header, header size: 64 bytes, header CRC: 0xD84034B5, created: 2020-05-29 01:15:36, image size: 2447726 bytes, Data Address: 0x0, Entry Point: 0x0, data CRC: 0xCE3A4A5E, OS: Linux, CPU: PowerPC, image type: OS Kernel Image, compression type: gzip, image name: "Linux-3.4.113"
131136        0x20040         gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
4194304       0x400000        uImage header, header size: 64 bytes, header CRC: 0xDA83B155, created: 2020-05-29 01:16:17, image size: 16719915 bytes, Data Address: 0x0, Entry Point: 0x0, data CRC: 0x2381914F, OS: Linux, CPU: PowerPC, image type: RAMDisk Image, compression type: lzma, image name: "Simple Ramdisk Image"
4194368       0x400040        LZMA compressed data, properties: 0x5D, dictionary size: 67108864 bytes, uncompressed size: -1 bytes

There is a device tree image at offset 0x400 which seems to be for the MX60 (codename bluestone). There is a second device tree image at offset 0x4000 for the MX80 (codename fullerene).

It is not as simple as creating a binary image with the DTB at offset 0x4000, the kernel at 0x200000, and initrd at 0x40000 because Meraki have modified u-boot to have a custom command meraki which reads a header, verifies the contents of the ubi partition part1 or part2 with SHA1, and then sets environment variables from addresses defined in the header.

The layout of the header is as follows:

Header field Data type (value)
SHA1_MAGIC uint32 (0x8e73ed8a)
DATA_LEN int32
SHA1SUM char[20]
MERAKI_EXTRA_MAGIC uint32 (0xa1f0beef)
MERAKI_EXTRA_LEN uint16 (0x0006)
MERAKI_EXTRA_TYPE uint16 (0x0001)
IMAGE_OFFSET uint32 (0x20000)
RAMDISK_OFFSET uint32 (0x400000)
FDT_OFFSETS array uint32 (0x400 or 0x4000)

The FDT used to boot depends on the value of meraki_part_fdt_index in u-boot. For the MX80, the index of the FDT offset is 1, meaning the FDT located at 0x4000 is used to boot. The presence of two FDTs suggests that Meraki are using the same firmware for both the MX60 and MX80. Despite the MX80 being a dual core CPU only one CPU core is usable, there is no SMP support in the kernel provided by Meraki.

To simplify booting, I have written a post-image.sh script which generates the appropriate header and assembles a bootable firmware image as part of the buildroot build process. You can find instructions on how to build the firmware in the meraki-builder GitHub repository.

The 3.4 kernel provided by Meraki doesn’t have any of the features required by OpenWrt (e.g. overlayfs) and buildroot doesn’t have a package manager. If you just want something to boot and run SSH on, then the buildroot image fulfills that need. You will most probably want to customize buildroot to include the packages and configuration that suits your needs. Upstream support in OpenWrt is still a long way away, as the APM86290 does not have support in the mainline kernel.

Meraki MS120 hardware overview

I received an innocent sounding question via GitHub, would the custom firmware I have been developing for the MS220 work on an MS120?

I am an eternal sucker for good mysteries involving hardware, so I found a seller on eBay offering an MS120-8-HW for $95 USD (plus shipping and customs to the EU). A few weeks of buyer’s remorse and waiting, and I had the MS120 in my hands.

One thing that immediately struck me about the MS120 is the material change from aluminum to steel. While I thought Meraki’s use of anodized aluminum in the MS220 series was a silly choice for the larger rack mounted models, it did make me think they were attempting to position themselves as the Apple Computer of networking products (“You pay the premium because we’re different”). Regardless of their intentions with the aluminum MS220 series, it was a precedent and it cheapens the experience to see them swap out aluminum for steel.

Let us continue, because whinging about metal choices is not bringing us closer to answering the original question.


Inside the MS120-8-HW

The MS120 is based on the Marvell Alleycat3 platform, referred to as kelpie-8 in the u-boot source and otherwise known by its marketing name “Prestera.” It is an ARMv7 core running at 400MHz with 512MB of DDR3 and 256MB of NAND flash.

The UART header is J16 at 115200n8 with the pinout:

  1. Vcc (Do not connect)
  2. Rx
  3. Tx
  4. GND

Pin 1 is closest to the SFP cage.

J17 is a mystery jumper. I have not identified its purpose yet.

There is 32Mbit (4MB) of SPI flash present, however as far as I can tell, this is connected directly to the Microsemi SmartFusion 2 and not to the Marvell ASIC. Using a hardware reader and a chip clip, I dumped the contents to examine it. Running binwalk yielded no results.

The entropy graph of the dump suggests that there are multiple copies of the same data stored, which follows Meraki’s design with the MS220 switches where there are primary and backup copies of the bootloader.

Entropy of the 32Mbit flash of the M120

Entropy graph of the 32Mbit flash in the MS120

Further inspection confirms that there are two identical copies of what I think is u-boot stored in the flash, starting at 0x301000. Each copy is approximately 420KB, which would correspond to the size of u-boot for this platform. However, the entropy is much higher than the entropy of u-boot.bin built using the Meraki GPL source, and contains only one readable string: kelpie_top

Perhaps this is the output of u-boot after running doimage to enable Secure Boot and AES-128 encryption?

The PCB traces from the winbond flash appear to go directly to the SmartFusion 2, but the u-boot UART output shows that the BootROM is booting from SPI:

BootROM 1.41
Booting from SPI flash

Is the SmartFusion 2 emulating an SPI device to the Alleycat3 after verifying the integrity of the u-boot binary in ROM?

u-boot then executes an application at memory address 0x0C100000 that prints the value of multiple “SecureBoot Registers” the strings of which do not appear anywhere in the u-boot code provided in the GPL archive:

## Starting application at 0x0C100000 ...

----Security Versions----
SecureBoot: R03.11b39af022017-07-25
SB Core: F01114R18.1680555472017-07-12
Microloader: MG0008R01.0103302017

----SecureBoot Registers----
system_invalid: 0
boot_check_count_error: 0
boot_done: 1
boot_ok: 1
boot_check_count_golden: 0
boot_check_count_upgrade: 2
boot_status_golden: 0
boot_status_upgrade: 1
first_bootloader: 1

boot_error: 0
boot_check_count_error_vc: 0
boot_check_count_error: 0
boot_timeout_vc: 0
boot_timeout: 0
boot_cs_good: 1
boot_config_error: 0
boot_version_error: 0
boot_config_error_code: 0
boot_error_code: 0
boot_cs_good: 1
boot_version_error: 0
boot1_cs_key_type: 1
boot1_cs_return_code: 0
boot1_cs_key_index: 5
boot2_cs_return_code: 0
boot2_cs_key_index: 5
boot2_cs_key_type: 1

----Other Registers----
fpga_version: 001b

Meraki do not include the build toolchain in their GPL archive. Luckily, I remembered that I have encountered this Marvell fork of u-boot before, for the Western Digital EX2100 which also uses a Marvell Armada 385 from the same family of ARMv7 CPU cores. Western Digital does include the Marvell toolchain used to compile u-boot in their GPL archive, good guy Western Digital!

To save anyone else the effort of setting up a development environment with an ancient version of GCC, I have created a Dockerfile that will handle building u-boot using the Marvell toolchain. You can find this work on GitHub.

I reached out to members of the Doozan forum who have been building Linux images for Marvell based NAS devices for many years to see if they had any more information about Secure Boot. Apparently Marvell CPUs will always kwboot before loading from other sources such as SPI or NAND:

Even if a box has secure boot in stock FW (u-boot, kernel,..), you should be able to kwboot it with a non-secure u-boot/spl binary.

I tried to kwboot the MS120 (with and without the Armada patches), but was unable to get it working. For some reason, the BootROM output is printed twice when kwboot is running, which I have not witnessed during any normal boot sequence:

$ ./kwboot -b u-boot.uart -f -t -B 115200 /dev/ttyUSB0
Sending boot message. Please reboot the target.../
BootROM 1.41
Pattern detected on UART-
BootROM 1.41
Pattern detected on UART/

uart.bin was created using tools/marvell/doimage:

$ ./doimage -T uart -D 0x0 -E 0x0 u-boot.bin u-boot.uart

Unfortunately, this investigation leaves us with more questions than answers.

  • What is the contents of the 32Mbit SPI flash?
    • Does the SmartFusion 2 only provide glue logic, or does it also protect/verify the contents of SPI flash?
  • Why won’t the Alleycat3 kwboot?
    • Is the duplicate output from BootROM when kwboot is invoked a clue?
  • What is the purpose of the header J17?
  • Why did Meraki switch from aluminum to steel?

The MS120 series is a completely different platform from the previous MS220 series, which used Vitesse ASICs with a MIPS core, 128MB of DDR2, 16MB of SPI, and 128MB of NAND flash.

The use of Secure Boot will complicate efforts to create a third-party firmware for the MS120 series. However, the more immediate issue is that kwboot does not work and there is no obvious copy of u-boot in SPI flash we can modify to alter the boot process.

Debian on WD EX2100: Installation instructions

In the last article on this topic, I unbricked my Western Digital My Cloud EX2100 NAS and said I would provide instructions on how to install Debian. There are partial instructions for how to replace the stock u-boot with one that is capable of booting Debian from a USB stick, but following those instructions requires slightly more knowledge about how the boot process works and omits some important details.

Now that I have the EX2100 working again, I thought it would be good to provide a set of complete, concise installation instructions for anyone else with this hardware who is interested in running Debian (or another Linux) on it. These instructions are also available on the Doozan forum.

Before we begin installation of Debian, you will need a working uart connection to the EX2100.

There are two possible methods to connect via uart:

  1. solder a header to JP1 the PCB to expose Rx, Tx, and Gnd. This will require opening the enclosure and removing the PCB, which will void your warranty
  2. Using kapton tape (tweezers, and patience), cover the 3.3V and GND pads on the front of the PCB. Use alligator clips to attach to the Tx contact, the Rx pad from JP1, and the chassis for ground. This method is not as reliable as soldering a header to JP1, but does not require disassembly and soldering as the area can be accessed by opening the hard drive bay doors

Test the connection by powering up the EX2100 with a USB to uart adapter or something like a Raspberry Pi. The uart operates at 115200n8.

You should immediately see output from u-boot (see sample below). If you don’t see any output, check your connections and ensure that you have not reversed the uart Tx/Rx.

Uart output

BootROM - 1.73
Booting from NAND flash

General initialization - Version: 1.0.0
Detected Device ID 6820
High speed PHY - Version: 2.0

Load WD Yosemite Serdes Config:
board SerDes lanes topology details:
| Lane # | Speed | Type |
| 0 | 06 | SATA0 |
| 1 | 05 | PCIe0 |
| 2 | 06 | SATA1 |
| 3 | 05 | USB3 HOST1 |
| 4 | 05 | USB3 HOST0 |
| 5 | 00 | SGMII2 |
PCIe, Idx 0: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.26.0
mvSysEnvGetTopologyUpdateInfo: TWSI Read failed
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
DDR3 Training Sequence - Ended Successfully
BootROM: Image checksum verification PASSED

__ __ _ _
| \/ | __ _ _ ____ _____| | |
| |\/| |/ _` | '__\ \ / / _ \ | |
| | | | (_| | | \ V / __/ | |
|_| |_|\__,_|_| \_/ \___|_|_|
_ _ ____ _
| | | | | __ ) ___ ___ | |_
| | | |___| _ \ / _ \ / _ \| __|
| |_| |___| |_) | (_) | (_) | |_
\___/ |____/ \___/ \___/ \__|
** LOADER **

Prepare a USB with the Debian rootfs for mvebu
For this step you will need a USB mass storage device of at least 2GB. We will format the device, so ensure you do not have any important data on it. The device must be USB 2.0 as USB 3.0 is not supported in the EX2100 u-boot.

Format the device with a single ext3 partition. Double check the device before proceeding as parted will erase and format the device without confirmation!

sudo parted --script /dev/sdX \ mklabel msdos \ mkpart primary ext3 1MiB 100%
sudo mkfs.ext3 -L rootfs /dev/sdX1

Download the latest Debian rootfs from the Doozan forums.

Now, mount the new partition and extract the Debian rootfs:

ROOTFSDIR=$(mktemp -d)
sudo mount /dev/sdX1 $ROOTFSDIR
sudo tar -C $ROOTFSDIR -jxvf Debian-4.12.4-mvebu-tld-1-rootfs-bodhi.tar.bz2

You can download the EX2100 DTB from the kernel image Doozan user bodhi publishes. Inside the kernel archive is another archive containing the DTBs (linux-dtb-$(kernelversion).tar). Extract the archive containing DTBs, and append the EX2100 DTB to the kernel:

tar -C $ROOTFSDIR/boot/dts/ -xvf linux-dtb-4.14.1-mvebu-tld-1.tar
cd $ROOTFSDIR/boot/
cat zImage dts/armada-385-wd-ex2100.dtb > zImage.fdt
mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n Linux-4.12.4-mvebu-tld-1 -d zImage.fdt uImage

Download u-boot for the EX2100 from here. Extract the archive:

tar -zxvf u-boot-a38x-Yosemite_2014T3_PQ.tar.gz

Copy the file u-boot-a38x-Yosemite_2014T3_PQ-nand.bin to the rootfs:

cp u-boot-a38x-Yosemite_2014T3_PQ-nand.bin $ROOTFSDIR/root/

Unmount the USB device from your computer:

sudo umount $ROOTFSDIR

Put the USB device in the rear USB port on the EX2100. The front USB port cannot be used for booting as it is not powered during u-boot execution.

kwboot the EX2100
Now that you have a serial connection to the EX2100 and a USB device prepared with the Debian rootfs, mvebu kernel, and dtb for the EX2100 it’s time to start installing Debian.

You must use a modified version of kwboot to load u-boot to the EX2100 via uart. You can obtain the modified kwboot binary for amd64 from here, and kwboot for armhf from here.

You will be kwbooting a modified version of u-boot which can save environment variables.

With the EX2100 powered off and unplugged, execute kwboot (note this is one line):

./kwboot -f -t -B 115200 /dev/ttyUSB0 -b u-boot-a38x-Yosemite_2014T3_PQ-nand-uart.bin -s 0 -q 1

Plug the DC power into the EX2100. If the handshake is successful, kwboot should display a loading screen:

Sending boot message. Please reboot the target...-�$�"Ufw�$�"U����$
Dfw�$�"U�\�$�"U����$�DUf�$�"Uw��"U����$4"U���$�"Uw�$�"U���$�DUf|fD�&T���$�"U�E�$�"Df3DD�DU�E7$�"U����$4"U���$�"U�E�4"U�/[email protected]� ��$DUw�$�"U����$�DUff�$�"D��fD$U��
Sending boot image...
0 % [......................................................................]

If the handshake was unsuccessful, you will see normal u-boot output as you should have seen in the “Uart output” section above. Repeat the procedure until you see kwboot say “Sending boot image…” followed by loading u-boot via uart. This takes some time (~90 seconds).

After the u-boot image has been loaded via kwboot, it will boot normally (as you saw in “Uart output” section above).

When the uart output gets to the following section:

Enable HD1
Enable HD2

Begin pressing the 1 key to interrupt the automatic boot process. If you were successful you should now have a u-boot prompt:

Enable HD1
Enable HD2
| port | Interface | PHY address |
| egiga0 | RGMII | 0x00 |
| egiga1 | RGMII | In-Band |
| egiga2 | SGMII | 0x01 |
egiga0 [PRIME], egiga1, egiga2
Hit any key to stop autoboot: 0
Marvell>> 1111

Set u-boot environment parameters
Press ctrl+c to clear the extra 1 characters and enter the following:

setenv bootdev usb
setenv device '0:1'
setenv load_initrd_addr 0x2900000
setenv load_image_addr 0x02000000
setenv load_initrd 'echo loading uInitrd ...; ext2load $bootdev $device $load_initrd_addr /boot/uInitrd'
setenv load_image 'echo loading Image ...; ext2load $bootdev $device $load_image_addr /boot/uImage'
setenv usb_set_bootargs 'setenv bootargs "console=ttyS0,115200 root=/dev/sda1 rootdelay=10 $mtdparts earlyprintk=serial"'
setenv usb_bootcmd 'echo Booting from USB ...; setenv fdt_skip_update yes; usb start; run load_image; run load_initrd ; run usb_set_bootargs; bootm $load_image_addr $load_initrd_addr'
setenv bootcmd_usb 'usb start; run usb_set_bootargs; run usb_bootcmd; reset'
run bootcmd_usb

If you performed the previous steps correctly, the EX2100 should now boot Debian from the USB device attached to the rear USB port.

Make a backup of NAND flash
For the changes in u-boot to be persistent, we need to write the modified version of u-boot to NAND.

However, before we do this, we will make a backup of the contents of NAND before modifying it. When booted into Debian, run the following commands:

mkdir nand_backup
cd nand_backup
nanddump --noecc --omitoob -f mtd{0,7}.bin /dev/mtd{0,7}

Make sure you make a copy these backups also in another location!!!

Installing the modified u-boot
Once you have taken a backup of the NAND contents, poweroff the EX2100. Remove the USB and copy the mtd backups you made to your computer for safekeeping.

When you have finished this, follow the instructions again in the “kwboot the EX2100” section but stop at the “Set u-boot environment parameters” section.

This time we will modify the u-boot environment:

setenv bootdev usb
setenv device '0:1'
setenv load_initrd_addr 0x2900000
setenv load_image_addr 0x02000000
setenv load_initrd 'echo loading uInitrd ...; ext2load $bootdev $device $load_initrd_addr /boot/uInitrd'
setenv load_image 'echo loading Image ...; ext2load $bootdev $device $load_image_addr /boot/uImage'
setenv usb_set_bootargs 'setenv bootargs "console=ttyS0,115200 root=/dev/sda1 rootdelay=10 $mtdparts earlyprintk=serial"'
setenv usb_bootcmd 'echo Booting from USB ...; setenv fdt_skip_update yes; usb start; run load_image; run load_initrd ; run usb_set_bootargs; bootm $load_image_addr $load_initrd_addr'
setenv bootcmd_usb 'usb start; run usb_set_bootargs; run usb_bootcmd; reset'
run bootcmd_usb

You will then proceed to boot Debian again.

Once in Debian, create /etc/fw_env.config:

echo “/dev/mtd0 0x100000 0x80000 0x20000 4” > /etc/fw_env.config

Check that fw_printenv is able to read the u-boot environment you just saved in the kwboot’d u-boot:

[email protected]:~# fw_printenv
boot_order=hd_scr usb_scr mmc_scr hd_img usb_img mmc_img pxe net_img net_scr
bootargs=root=/dev/ram console=ttyS0,115200

If you see the u-boot environment variables returned, then the modified u-boot successfully wrote the environment variables to 0x100000

Verify that you are able to write to the u-boot section of NAND (this should be enabled in the EX2100 dtb):

[email protected]:~# mtd_debug info /dev/mtd0
mtd.type = MTD_NANDFLASH
mtd.size = 5242880 (5M)
mtd.erasesize = 131072 (128K)
mtd.writesize = 2048 (2K)
mtd.oobsize = 64
regions = 0

If instead you see “mtd.flags = MTD_CAP_ROM” then you cannot flash u-boot using the dtb you have booted with. You can download the dts for the EX2100 and build the dtb for your kernel.

If you saw MTD_CAP_NANDFLASH, then proceed to backup the u-boot environment variables to a file:

nanddump --noecc --omitoob -s 0x100000 -l 0x80000 -f ubootenv.bin /dev/mtd0

Erase the u-boot portion of NAND, flash the modified u-boot, and restore the environment variables:

flash_erase /dev/mtd0 0 8
nandwrite -p /dev/mtd0 u-boot-a38x-Yosemite_2014T3_PQ-nand.bin
nandwrite -p /dev/mtd0 -s 0x100000 ubootenv.bin

That’s it, you should be finished. Shutdown the EX2100 and exit kwboot. Using a standard serial console like screen or minicom, connect to the uart if you want to monitor the boot process.

Now when you power the EX2100 it should boot Debian from the USB device plugged into the rear USB port, if it is present. If the USB device is not present, u-boot will fall back to booting the WD firmware from internal flash.

Note that the Western Digital firmware is not fully functional unless you allow it to format your hard drive(s). If you format the drives with the Western Digital firmware and run Debian from USB, then the device should function regardless of the running firmware. If you choose to partition the drives yourself and usually boot Debian, then the WD firmware won’t be very functional as the hard drive(s) are not formatted in the expected layout.