Category Archives: Firmware

MS220-8P: Custom firmware from scratch

To follow up on my previous posts about modifying the firmware for the Cisco Meraki MS220-8P, I have more progress to report.

Since I could not figure out how to fix serial output from userspace when booting from u-boot, I went back to booting with RedBoot and tried to focus on improving userspace from my first sloppy attempt to coerce the Meraki firmware into booting from NOR.

Here is the current situation:

  • RedBoot without CRC or kernel size checks
  • Kernel 3.18.123 with xz compression
  • Busybox userspace based on buildroot 2020.02.1
  • Meraki kernel modules: vtss_core, proclikefs, merakiclick, elts_meraki, and vc_click are successfully loaded during boot
  • The click router is successfully initialized, creating the interfaces arping, linux_mgmt, sw0_pcap, and wired0

However, networking is non-functional. The switch broadcasts DHCP requests, but no packets are ever passed from the switching ASIC to the management CPU:

arping    Link encap:Ethernet  HWaddr 88:15:44:73:22:31
          inet addr:169.254.55.143  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::8a15:44ff:fe73:2231/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:33 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:5546 (5.4 KiB)

linux_mgmt Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.255.255.255
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

sw0_pcap  Link encap:Ethernet  HWaddr 00:01:02:03:04:05
          inet addr:169.254.83.119  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::201:2ff:fe03:405/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:5476 (5.3 KiB)

wired0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

On the other side:

tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
06:02:32.423798 IP6 (hlim 1, next-header Options (0) payload length: 36) fe80::201:c0ff:fe1f:9fb2 > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff1f:9fb2 to_ex { }]
06:02:33.097129 IP6 (hlim 1, next-header Options (0) payload length: 36) fe80::201:c0ff:fe1f:9fb2 > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff1f:9fb2 to_ex { }]
06:03:20.969517 IP (tos 0x0, ttl 64, id 20247, offset 0, flags [none], proto UDP (17), length 332, bad cksum 2a8b (->1677)!)
    0.0.10.10.68 > 10.10.255.255.67: [bad udp cksum 0x17ca -> 0x03b6!] BOOTP/DHCP, Request from 88:15:44:21:65:10 (oui Unknown), length 304, xid 0xd3650d76, secs 123, Flags [none] (0x0000)
          Client-Ethernet-Address 88:15:44:21:65:10 (oui Unknown)
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Discover
            Client-ID Option 61, length 15: hardware-type 255, 44:21:65:10:00:03:00:01:88:15:44:21:65:10
            SLP-NA Option 80, length 0""
            NOAUTO Option 116, length 1: Y
            MSZ Option 57, length 2: 1472
            Hostname Option 12, length 13: "m881544216510"
            T145 Option 145, length 1: 1
            Parameter-Request Option 55, length 14:
              Subnet-Mask, Classless-Static-Route, Static-Route, Default-Gateway
              Domain-Name-Server, Hostname, Domain-Name, MTU
              BR, Lease-Time, Server-ID, RN
              RB, Option 119
            END Option 255, length 0
06:04:25.620540 IP (tos 0x0, ttl 64, id 13784, offset 0, flags [none], proto UDP (17), length 332, bad cksum 43ca (->2fb6)!)
    0.0.10.10.68 > 10.10.255.255.67: [bad udp cksum 0x178a -> 0x0376!] BOOTP/DHCP, Request from 88:15:44:21:65:10 (oui Unknown), length 304, xid 0xd3650d76, secs 187, Flags [none] (0x0000)
          Client-Ethernet-Address 88:15:44:21:65:10 (oui Unknown)
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Discover
            Client-ID Option 61, length 15: hardware-type 255, 44:21:65:10:00:03:00:01:88:15:44:21:65:10
            SLP-NA Option 80, length 0""
            NOAUTO Option 116, length 1: Y
            MSZ Option 57, length 2: 1472
            Hostname Option 12, length 13: "m881544216510"
            T145 Option 145, length 1: 1
            Parameter-Request Option 55, length 14:
              Subnet-Mask, Classless-Static-Route, Static-Route, Default-Gateway
              Domain-Name-Server, Hostname, Domain-Name, MTU
              BR, Lease-Time, Server-ID, RN
              RB, Option 119
            END Option 255, length 0

The primary reason for this appears to be because Meraki is using the click modular router. Click seems to be an attempt to lobotomize the Linux kernel networking stack for the sake of writing academic research papers. The kernel side of click was never upstreamed to mainline and is no longer maintained. Let us pour one out for the poor soul at Cisco who has to maintain this.

Since Meraki was spun out of MIT and the researchers who wrote Click were also at MIT, the relationship between Meraki and Click is clear. Finally Click found a practical use, in a now-obsolete product line from Cisco.

More reverse engineering is necessary to make Click functional enough to have basic connectivity to the management CPU.


The strace output of switch_brain is available in this gist. By first running switch_brain, and then overwriting the default traffic rules with “allow all” I was able to SSH to the management CPU:

/ # echo "allow all" > /click/nat/common_switch_nat/from_smc_filter/config
/ # echo "allow all" > /click/nat/common_switch_nat/from_mgmt_filter/config
/ # tail /tmp/messages | grep dropbear
Jan  1 00:15:49 buildroot authpriv.info dropbear[680]: Child connection from 10.10.10.1:39248
Jan  1 00:15:50 buildroot authpriv.notice dropbear[680]: Password auth succeeded for 'root' from 10.10.10.1:39248
/ # w
USER            TTY             IDLE    TIME             HOST
root            pts/0           00:00   Jan  1 00:15:50  10.10.10.1

I think it is now very clear that the “only” thing blocking full access to the management CPU is the Click configuration. My hope is to build the configuration from the switch_brain strace output and package that into an initscript.

I have updated the meraki-builder repository with the buildroot config file and overlay directory I use to inject init scripts (among other things).

If you would like to contribute, I have written some installation instructions for the current development firmware. If you find the correct voodoo of Click commands to access the management CPU, please open a PR 🥰

u-boot, boot args, and compressed kernels on the Meraki MS220-8P

The last time we looked at the Cisco Meraki MS220, it was possible to boot kernel 3.18.123 compiled from the Cisco GPL archive. The ability to compile the kernel, with our own command line and modules opened up new possibilities for firmware modification.

That was a very good start, but there is more improvement possible. Support for the Microsemi VCore III SoCs was added to u-boot in late 2018. The MS220-8P contains the Luton26 SoC, but with only 10 ports. The MS220-8P appears to follow the Microsemi PCB090 reference design, and it is possible to build and boot u-boot (v2019.10):

U-Boot 2019.10 (Apr 04 2020 - 09:26:23 +0000)

MSCC VCore-III MIPS 24Kec
Model: Luton26 PCB090 Reference Board
DRAM:  128 MiB
Loading Environment from SPI Flash... SF: Detected mx25l12805d with page size 256 Bytes, erase size 64 KiB, total 16 MiB
OK
In:    [email protected]
Out:   [email protected]
Err:   [email protected]
Net:   Could not get PHY for miim-bus1: addr 0
Could not get PHY for miim-bus1: addr 1
Could not get PHY for miim-bus1: addr 2
Could not get PHY for miim-bus1: addr 3
Could not get PHY for miim-bus1: addr 4
Could not get PHY for miim-bus1: addr 5
Could not get PHY for miim-bus1: addr 6
Could not get PHY for miim-bus1: addr 7
Could not get PHY for miim-bus1: addr 8
Could not get PHY for miim-bus1: addr 9
Could not get PHY for miim-bus1: addr 10
Could not get PHY for miim-bus1: addr 11

Warning: [email protected] (eth0) using random MAC address - 62:bf:42:f4:e4:5f
eth0: [email protected]
Hit any key to stop autoboot:  0 
luton #

u-boot for the Luton26 specifies the environment to be saved at 1MB. This does not match my desired flash layout, so I redefined the ENV_OFFSET and associated variables in board/mscc/luton/Kconfig:

# 64KiB
config ENV_SECT_SIZE
        default 0x10000
# 128KiB at 512KiB
config ENV_SIZE
        default 0x20000
# at 512KiB
config ENV_OFFSET
        default 0x80000 if ENV_IS_IN_SPI_FLASH

Though for some reason this is not working as anticipated, and the environment is still corrupted after saveenv (╯°□°)╯︵ ┻━┻


With u-boot, I was finally able to test booting a compressed kernel. I wonder why Meraki is not compressing the kernel, since compression does not appear to slow down boot at all, and the compressed kernel is much smaller than an uncompressed kernel. Using xz compression, the 3.18.123 kernel is ~2MB, instead of ~5MB. Some fairly trivial modifications were necessary to achieve this.


With u-boot support, testing is much easier. u-boot has network support on the Luton26, making it possible to tftpboot a kernel. This vastly increases the speed of testing as you don’t have to write the kernel to SPI flash before testing changes.

Getting the handoff between u-boot and the kernel for the command line was difficult to figure out. The original bootloader, Redboot, doesn’t have the kernel command line built-in the command line is always compiled into the kernel. In that context, it’s logical why Vitesse didn’t include support for reading the bootloader command line in the kernel (and why Cisco didn’t modify the kernel to include support, since they’re shipping Redboot in these switches).

u-boot was placing the variables somewhere in RAM, but for some reason the kernel wasn’t getting them in its boot command line. u-boot has a humorously titled page on exactly this issue: Linux Kernel Ignores my bootargs. After digging through the u-boot source, it was clear that argv are being passed at the start of DRAM at 0x80000000:

in include/configs/vcoreiii.h: 
#define CONFIG_SYS_SDRAM_BASE           0x80000000
in board/mscc/luton/luton.c:
gd->bd->bi_boot_params = CONFIG_SYS_SDRAM_BASE;

After much searching through the 3.18.123 kernel source code, it became clear that prom is typically where the boot args receiving code is typically placed, so I took inspiration from the lantiq prom code to copy the bootloader argc and argv.

Now, it’s possible to tftpboot compressed kernels with boot args!


Except… there is no serial output from userspace. printk messages are displayed on serial, but no userspace programs (e.g. init) seem able to print out /dev/ttyS0 when using u-boot.

Compiling the kernel with /dev/ttyprintk support and adding custom scripts in /etc/init.d shows that userspace is indeed alive:

[    4.779000] [U] S05printk starting; userspace is alive
[    5.689000] random: dropbear urandom read with 85 bits of entropy available
[    7.356000] [U] Hello from S51printk; userspace is alive but serial is broken

/dev/ttyS0 seems normal, and the memory map is the same as RedBoot (which is logical, since it’s the same kernel version):

[    7.706000] [U] S99 starting; we print stuff
[    7.712000] [U] Now we print /dev
...
[    8.150000] [U] crw-------    1 root     root        4,  64 Jan  1 00:00 ttyS0
[    8.159000] [U] crw-------    1 root     root        5,   3 Jan  1 00:00 ttyprintk
[    8.207000] [U] Now we print /proc/iomem
[    8.222000] [U] 00000000-07feffff : System RAM
[    8.227000] [U]   00000000-00000000 : Crash kernel
[    8.232000] [U]   00100000-0048cc83 : Kernel code
[    8.237000] [U]   0048cc84-0057610f : Kernel data
[    8.241000] [U] 40000000-4fffffff : Serial interface
[    8.247000] [U] 50000000-5fffffff : Parallel interface
[    8.252000] [U]   50000000-50000010 : gen_nand.0
[    8.256000] [U]     50000000-50000010 : gen_nand.0
[    8.261000] [U] 60000000-60ffffff : Switch registers
[    8.266000] [U] 70000000-701fffff : CPU Registers
[    8.271000] [U]   70100000-7010001f : serial
[    8.279000] [U] Now we print /proc/vmallocinfo
[    8.294000] [U] 0xc0000000-0xc1001000 16781312 __ioremap+0x128/0x4ac ioremap
[    8.301000] [U] 0xc1004000-0xc1009000   20480 jffs2_lzo_init+0x18/0x84 pages=4 vmalloc
[    8.309000] [U] 0xc1009000-0xc100c000   12288 jffs2_lzo_init+0x24/0x84 pages=2 vmalloc
[    8.317000] [U] 0xc100c000-0xc100e000    8192 __ioremap+0x128/0x4ac ioremap
[    8.324000] [U] 0xc100e000-0xc1010000    8192 __ioremap+0x128/0x4ac ioremap
[    8.331000] [U] 0xc1010000-0xc1035000  151552 deflate_init+0x44/0xf8 pages=36 vmalloc
[    8.339000] [U] 0xc1035000-0xc1041000   49152 deflate_init+0x98/0xf8 pages=11 vmalloc
[    8.347000] [U] 0xc1041000-0xc1045000   16384 n_tty_open+0x2c/0x14c pages=3 vmalloc
[    8.355000] [U] 0xc1045000-0xc1066000  135168 xz_dec_lzma2_create+0x54/0x9c pages=32 vmalloc
[    8.363000] [U] 0xc1080000-0xc1281000 2101248 __ioremap+0x128/0x4ac ioremap
[    8.370000] [U] 0xc1285000-0xc1289000   16384 n_tty_open+0x2c/0x14c pages=3 vmalloc
[   97.943000] random: nonblocking pool is initialized

I haven’t solved the issue yet of why serial output is broken in userspace 😔
If anyone has experienced broken serial in userspace but not for kernel printk, I’d love to hear how you resolved the problem!

Modifying the Cisco Meraki MS220-8P firmware

Following the analysis done by Leo Leung of the Cisco Meraki MS220-8P boot process, I wanted to share more information I’ve uncovered about the firmware source code, layout, and modification.

Cisco were not very forthcoming with the source code, and initially tried to claim that because the product was past the End of Sale, they were under no obligation to provide the source code. I am not a lawyer, but my understanding is that this claim is in violation of GPLv2 Section 3b, which states the the vendor must make the source code available for at least 3 years after the last distribution:

Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange

https://opensource.org/licenses/gpl-2.0.php

I am extremely disappointed to see that Cisco is still not forthcoming with GPL source code. They have little to gain by being so difficult, and it is very disappointing to see them claiming they aren’t under obligation to provide the source code due to the product being discontinued.

To save anyone else the trouble of dragging the source code for Meraki Vitesse based switches (MS22, MS42, MS220, MS320) out of Cisco, I have mirrored the source code into 2 repositories on GitHub:

  • Stage 1 (NOR bootloader, kernel 3.18.122, includes RedBoot LinuxLoader)
  • Stage 2 (NAND firmware, kernel 3.18.123)

Please note: the kernel modules used to control the switch ASIC (vtss_core, merakiclick) are not included in the above source code. If you ever plan to build your own kernel and want to use the switch, you will need to extract the kernel modules corresponding to the above kernel versions from your switch.


My initial attempts to modify the firmware followed Leo’s instructions to disassemble the mtd region boot1. I was perplexed by the contents of the boot1-patched-post data that Leo’s script extracted from the image. The data in this region has an extremely high entropy:

The data is quite large too, approximately 200KB or 5% of the boot1 region. That’s a lot of space to give up on an embedded system for no apparent reason!

I decided to zero out the contents of this boot1-patched-post section and see what effect it would have on the boot process of the switch. The result was a kernel panic, cannot open initrd:

[    2.537000] devtmpfs: error mounting -2
[    2.541000] Warning: unable to open an initial console.
[    2.548000] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -2
[    2.556000] Please append a correct "root=" boot option; here are the available partitions:
[    2.564000] 1f00          131072 mtdblock0  (driver?)
...
[    2.674000] 1f15            2670 mtdblock21  (driver?)
[    2.679000] mkp_lg: VFS: Unable to mount root fs on unknown-block(0,0)
[    2.679000] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

So the kernel can no longer find the initramfs, even though the XZ archive is still present in the region. This got me thinking that the region must contain information the kernel uses to extract the XZ compressed initrd.

The build instructions I received from Meraki with the source code stated:

To build OpenWRT firmware:

cd meraki-firmware/openwrt
cp config-elemental-3.18 .config
make oldconfig
make -j1 BOARD=elemental-3.18 OPENWRT_EXTRA_BOARD_SUFFIX=_3.18


To build Linux-3.18 kernel:

cd meraki-firmware/linux-3.18
cp ../openwrt/target/linux/elemental-3.18/config .config
make CROSS_COMPILE=../openwrt/staging_dir_mipsel_nofpu_3.18/bin/mipsel-linux-musl- ARCH=mips oldconfig

make CROSS_COMPILE=../openwrt/staging_dir_mipsel_nofpu_3.18/bin/mipsel-linux-musl- ARCH=mips prepare

touch rootlist
make CROSS_COMPILE=../openwrt/staging_dir_mipsel_nofpu_3.18/bin/mipsel-linux-musl- ARCH=mips vmlinux

After following these instructions and ending up with a vmlinux that was far too big to fit into the flash region, I figured out how to build the kernel such that it fits into the boot region. This also clarified what the contents of boot1-patched-post from Leo’s extraction script contains.

To start, if you are building the 3.18.122 kernel for the SPI flash (first stage boot, before kexec) you absolutely need a rootlist file. For the stock Meraki firmware, I have re-created the contents of the rootlist file:

file /bootsh ramdisk/bootsh 755 0 0
slink /init bootsh 777 0 0
file /kexec ramdisk/kexec 755 0 0
file /MERAKI_BUILD ramdisk/MERAKI_BUILD 755 0 0
dir /bootroot 755 0 0
dir /dev 755 0 0
slink /lib bootroot/lib 777 0 0
slink /etc/ bootroot/etc 777 0 0
slink /usr bootroot/usr 777 0 0
slink /sbin bootroot/sbin 777 0 0
slink /bin bootroot/bin 777 0 0

The Meraki provided instructions produce an ELF vmlinux, which is not what at all what the switch is booting. You must use objcopy to strip vmlinux before you can get it to what the switch expects in the firmware:

mipsel-linux-musl-objcopy -O binary -S vmlinux vmlinux.bin

By following the above steps, building the kernel with the above rootlist file contents and stripping vmlinux with objcopy, vmlinux.bin is the data that goes into the boot1 region immediately after the 32 byte header. Pad the header + vmlinux.bin with zeros to ensure it is 3932160 bytes total.

Aside: You’re probably not interested in including the same files in your ramdisk as the stock Meraki firmware. Unless you have modified the NAND image to spawn a console and kill their management daemons, you’ll have the same end result as the stock firmware. However, it is a useful test to ensure that you can build and boot the kernel yourself:

Linux version 3.18.122-meraki-elemental ([email protected]) (gcc version 5.4.0 (GCC) ) #4 Sun Mar 1 14:44:24 UTC 2020

The meraki_stock branch of the 3.18.122 repository includes the defconfig and rootlist file you need to build the 3.18.122 kernel.


Now onto more useful matters, modifying the bootloader. I grew tired of having CRC errors when testing my changes, so I patched the CRC check out of the bootloader. Later on, I decided I wanted to try booting 3.18.123 directly from NOR, except the stripped vmlinux is nearly 5MB, too large. So I patched out the boundary check in the bootloader. You can find both versions of the bootloader (without CRC and without CRC/boundary check) in the 3.18.122 repository on GitHub.

Meraki appears to supply a common firmware for all their Vitesse based switches. The MS220-8P is based on the luton26 ASIC, while the larger switches (MS22, MS42, MS220-24/48, and MS320-24/48) appear to be based on the Jaguar-1 ASIC. Since you don’t need the kernel modules for other models in the firmware, you can delete them (and other unnecessary Meraki daemons) and fit the entire firmware in roughly 8MB using XZ compressed squashfs.

With this combination, you can boot Linux 3.18.123 from NOR, with a squashfs on NOR:

[    0.000000] Kernel command line:  console=ttyS0,115200 mtdparts=m25p80:0x40000(loader1),0x4c0000(boot1),0x800000(bootubi),0x300000(jffs2) root=/dev/mtdblock3 rootfstype=squashfs ubi.mtd=gen_nand.0 mem=0x7FF0000 ramoops.mem_address=0x7FF0000 ramoops.mem_size=0x10000 ramoops.block_size=0x10000

Since the kernel version matches the modules shipped by Meraki in their firmware, you can load the modules required to manage the ASIC:

# lsmod
vc_click 251569 0 - Live 0xc35eb000 (PO)                                                                                                                                                   
elts_meraki 4132849 1 vc_click, Live 0xc30ab000 (PO)                                                                                                                                       
merakiclick 1587774 2 vc_click,elts_meraki, Live 0xc2854000 (O)                                                                                                                            
proclikefs 5189 2 merakiclick, Live 0xc007b000 (O)                                                                                                                                         
vtss_core 663855 1 vc_click, Live 0xc13f0000 (PO)

This approach is not without its own problems. The Meraki firmware expects / to be mounted rw, something that isn’t possible with squashfs.

In summary, with the following components you can boot directly from NOR, without touching NAND at all:

Please understand that I have not solved all the issues with this approach, and above resources are intended as a guide to encourage future development. To illustrate, here is a flashable image for your MS220-8P that illustrates booting entirely from NOR. Although I would like to caution you that this is a proof of concept, not a functional firmware. If you have any improvements please submit your pull request the meraki-builder repository!

The above information is the product of months of reverse engineering, development, and testing on an MS220-8P. It is my hope that by providing the GPL archive, my modifications, build scripts, and documentation that others will find a more elegant way to run the firmware entirely from NOR, which would significantly decrease the complexity to running a custom firmware on the MS220-8P.

That’s all for now. I will continue working on this project in the background and may have more updates in the future.