Tag Archives: cisco

Meraki MS350 hardware overview

Leave a reply

The Meraki MS350 (MS350-24 and MS350-48) series switches offer 24 or 48 ports of Gigabit Ethernet. The MS350-24X offers 16 ports of Gigabit Ethernet, and 8 ports of multi-Gigabit (1/2.5/5/10G) Ethernet. All models have four SFP/SFP+ uplink ports, a dedicated remote management port, and stacking capabilities via QSFP. Today we will be looking at the MS350-48 and MS350-24X models specifically.

MS350-48LP from Meraki’s datasheet

Here is a quick summary of the MS350 specs:

  • Intel Atom C2358 CPU (2C/2T, 1.74GHz)
  • 2GB DDR3 ECC RAM (SODIMM, Unigen U25U7210N8FD-BDD-CCHF1)
  • 16MB SPI flash, 2GB NAND flash (TSOP48 NAND on motherboard, USB via Phison)
  • MS350-24X: 30 Network interfaces (16 Gigabit Ethernet, 8 mGig Ethernet, 4 SFP+, 2 QSFP stacking)
  • MS350-48: 54 Network interfaces (48 Gigabit Ethernet, 4 SFP+, 2 QSFP stacking)
  • MS350-24X: MA-PWR-1025WAC (identical to PWR-C2-1025WAC)
  • MS350-48: MA-PWR-250WAC (identical to PWR-C2-250WAC)

The MS350-48 uses the Broadcom BCM56547 (A0) ASIC, with BCM84740 PHYs. The MS350-24X uses the Broadcom BCM56546 (B0) ASIC, with BCM82578 and Aquantia AQR405 PHYs. PoE versions of the switch use the Broadcom BCM59121 PSE controller.

MS350-48:

00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)
00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02)
00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02)
00:0b.0 Co-processor: Intel Corporation Atom processor C2000 QAT (rev 02)
00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)
00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)
00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 1.0 GbE Backplane (rev 03)
00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)
00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries Device b547 (rev 01)
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries Device b547 (rev 01)

MS350-24X:

00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)
00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02)
00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02)
00:0b.0 Co-processor: Intel Corporation Atom processor C2000 QAT (rev 02)
00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)
00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)
00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 1.0 GbE Backplane (rev 03)
00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 1.0 GbE Backplane (rev 03)
00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 1.0 GbE Backplane (rev 03)
00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)
00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries Device b546 (rev 11)

Both models have the same USB devices present:

Bus 001 Device 002: ID 8087:07db
Bus 001 Device 001: ID 1d6b:0002
Bus 001 Device 003: ID 13fe:5200

MS350-24X and MS350-48 both use coreboot as the bootloader, although the MS350-24X model has a different build. In both cases, the ROM has the following layout:

00000000:00010000 reserved
00010000:0070ffff bk1
00710000:00dfffff bk2
00e00000:00ffffff coreboot

The cbfs contains the following:

FMAP REGION: COREBOOT
ms350-24x_w25q128.bin: 16384 kB, bootblocksize 1024, romsize 16777216, offset 0xe10000
alignment: 64 bytes, architecture: x86

Name                           Offset     Type           Size   Comp
cmos_layout.bin                0xe10000   cmos_layout      1396 none
fallback/romstage              0xe105c0   (unknown)       21624 none
fallback/ramstage              0xe15a80   (unknown)       49421 none
fallback/payload               0xe21c00   simple elf      23042 none
config                         0xe27640   raw              4676 none
revision                       0xe288c0   raw               566 none
(empty)                        0xe28b40   null          1209432 none
mrc.cache                      0xf4ffc0   mrc_cache       65536 none
cpu_microcode_blob.bin         0xf60000   microcode       84992 none
(empty)                        0xf74c40   null            45912 none
fsp.bin                        0xf7ffc0   fsp            389120 none
(empty)                        0xfdf000   null           134040 none

coreboot was built with an ELF payload (miles) which by default loads and jumps into the bootkernel FIT image located at 0x10000. A secondary bootkernel exists on flash at offset 0x710000.

This is very similar to the MX84 as they are both based on the same Rangeley platform.

The entire MS350 series is based on the Intel Atom C2000 series CPU, which Meraki also used in the MX84. Sadly, the MS350 also suffers from the AVR54 errata, as the C2358 in both the MS350-48 and MS350-24X is the B0 revision.

LPC_CLK is exposed on pin 1 of J35, with R3635 carrying 3.3V (MS350-48 and MS350-24X). Therefore, you can add a 100 Ohm resistor between R3635 and pin 1 to pull up the LPC clock. Just be sure to use an “extended-life” resistor for the modification, you wouldn’t want to compromise the MTBF of your Meraki product with anything sub-par 😉

100 Ohm resistor to pull up LPC clock (MS350-24X)

If you wish to flash your MS350, you will need to remove or socket the SOIC8 SPI flash (SK_U1).

This is because there are other devices powered by the +3.3V voltage rail used by SPI flash, which interferes with your ability to read/write the contents of flash. I prefer the Wieson G6179-10 SOIC8 socket (available from Adafruit). People outside the US will probably find it easier to desolder the flash and use a SOIC8 socket with prototype wires, as the G6179-10 is difficult to obtain for a reasonable price.

The UART header is J31 on both the MS350-48 and MS350-24X and follows the standard Meraki UART pinout (1: VCC, 2: Tx, 3: Rx, 4: GND)

Similar to the MS210/225 series, the Broadcom SDK implements the packet engine in userspace, using the linux_kernel_bde and linux_user_bde kernel modules to interface with the ASIC. In the Meraki firmware, the packet engine is a component of the userspace click daemon, which loads the bcm_click shared object during click router initialisation.

Idle power consumption:

  • MS35-48: 54W
  • MS350-24X: 96W

GPL source code for the MS350 was requested from Meraki in July 2023. At the time of writing, they have not provided any. I will update this post with links to the source code when it is provided.

Meraki MS210/MS225 hardware overview

Leave a reply

The Meraki MS210 and MS225 series switches offer 24 or 48 ports of Gigabit Ethernet, four SFP/SFP+ uplink ports, a dedicated remote management port, and stacking capabilities via QSFP.

Meraki MS210-24 and MS210-48

The MS210/MS225 series are based on the Broadcom BCM56160 “Hurricane3” ASIC, and the Broadcom BCM82756 10G PHY. PoE models contain the Broadcom BCM59121 PSE controller. All switch models have 16MB of SPI flash (MX25L12805D), 256MB of NAND (MT29F2G08ABAEAWP), and 1024MB of DDR4 DRAM.

MS225-48LP internal PCB

MS225-48LP switch internals with PoE midplane removed

The Meraki codename for the MS210 and MS225 series is “brumby” and all brumby switches run the same firmware release (switch-arm). The MS250 is essentially the MS225 with hot-swap power supplies (similar to the MS220/MS320).

Keen readers may be wondering why the MS210 series has only SFP ports while the MS225 has SFP+ ports, given they are identical hardware and run the same switch-arm firmware. The answer is market segmentation; Meraki decided to artificially limit the speed of the MS210 SFP ports to 1G, even though the MS210 hardware is capable of 10G via SFP+. Early in the boot process switch_brain checks the switch model, and if it identifies as the MS210 series the SFP port speed is limited to 1000M.

The stock Meraki boot process uses u-boot on SPI to load a “bootkernel” (also from SPI), which then initializes NAND and using kexec boots the main firmware. The firmware layout follows the standard Meraki practice of having A/B firmware images: bootkernel1, bootkernel2, part.safe, part.old.

If you wish to flash your MS210/MS225, you will need to remove or socket the SOIC8 SPI flash (U18). This is because the ASIC is powered by the same +3.3V voltage rail as the SPI flash, and will attempt to boot when you attach your flashing device, which interferes with your ability to read/write the contents of flash. I prefer the Wieson G6179-10 SOIC8 socket (available from Adafruit). People outside the US will probably find it easier to desolder the flash and use a SOIC8 socket with prototype wires, as the G6179-10 is difficult to obtain for a reasonable price.

MS225 with SPI flash socket installed

Unlike the MS120, the MS210/MS225 do not implement secure boot, so all that is needed to develop on the platform is to recompile and flash u-boot from the Meraki GPL release and then interrupt the boot process and provide your own firmware build (e.g. via TFTP).

The UART header is J31 on both the 24 and 48 port models and follows the standard Meraki UART pinout (1: VCC, 2: Tx, 3: Rx, 4: GND) at 115200 baud.

The Broadcom SDK for the BCM56160 series implements the packet engine in userspace, using the linux_kernel_bde and linux_user_bde kernel modules to interface with the ASIC. In the Meraki firmware, the packet engine is a component of the userspace click daemon, which loads the bcm_click shared object during click router initialisation.

There are no public datasheets available for any of the Broadcom chips used in the MS210/225. While you can find information on OpenBCM, as far as I can tell the API provided by OpenBCM (via the kernel modules) which is used to implement the packet engine has no public documentation. If anyone has more information, please get in touch 😀

Meraki MS120 hardware overview

2 Replies

I received an innocent sounding question via GitHub, would the custom firmware I have been developing for the MS220 work on an MS120?

I am an eternal sucker for good mysteries involving hardware, so I found a seller on eBay offering an MS120-8-HW for $95 USD (plus shipping and customs to the EU). A few weeks of buyer’s remorse and waiting, and I had the MS120 in my hands.

One thing that immediately struck me about the MS120 is the material change from aluminum to steel. While I thought Meraki’s use of anodized aluminum in the MS220 series was a silly choice for the larger rack mounted models, it did make me think they were attempting to position themselves as the Apple Computer of networking products (“You pay the premium because we’re different”). Regardless of their intentions with the aluminum MS220 series, it was a precedent and it cheapens the experience to see them swap out aluminum for steel.

Let us continue, because whinging about metal choices is not bringing us closer to answering the original question.

MS120 PCB

Inside the MS120-8-HW

The MS120 is based on the Marvell Alleycat3 platform, referred to as kelpie-8 in the u-boot source and otherwise known by its marketing name “Prestera.” It is an ARMv7 core running at 400MHz with 512MB of DDR3 and 256MB of NAND flash.

The UART header is J16 at 115200n8 with the pinout:

  1. Vcc (Do not connect)
  2. Rx
  3. Tx
  4. GND

Pin 1 is closest to the SFP cage.

J17 is a mystery jumper. I have not identified its purpose yet.

There is 32Mbit (4MB) of SPI flash present, however as far as I can tell, this is connected directly to the Microsemi SmartFusion 2 and not to the Marvell ASIC. Using a hardware reader and a chip clip, I dumped the contents to examine it. Running binwalk yielded no results.

The entropy graph of the dump suggests that there are multiple copies of the same data stored, which follows Meraki’s design with the MS220 switches where there are primary and backup copies of the bootloader.

Entropy of the 32Mbit flash of the M120

Entropy graph of the 32Mbit flash in the MS120

Further inspection confirms that there are two identical copies of what I think is u-boot stored in the flash, starting at 0x301000. Each copy is approximately 420KB, which would correspond to the size of u-boot for this platform. However, the entropy is much higher than the entropy of u-boot.bin built using the Meraki GPL source, and contains only one readable string: kelpie_top

Perhaps this is the output of u-boot after running doimage to enable Secure Boot and AES-128 encryption?

The PCB traces from the winbond flash appear to go directly to the SmartFusion 2, but the u-boot UART output shows that the BootROM is booting from SPI:

BootROM 1.41
Booting from SPI flash

Is the SmartFusion 2 emulating an SPI device to the Alleycat3 after verifying the integrity of the u-boot binary in ROM?

u-boot then executes an application at memory address 0x0C100000 that prints the value of multiple “SecureBoot Registers” the strings of which do not appear anywhere in the u-boot code provided in the GPL archive:

## Starting application at 0x0C100000 ...

----Security Versions----
SecureBoot: R03.11b39af022017-07-25
SB Core: F01114R18.1680555472017-07-12
Microloader: MG0008R01.0103302017

----SecureBoot Registers----
system_invalid: 0
boot_check_count_error: 0
boot_done: 1
boot_ok: 1
boot_check_count_golden: 0
boot_check_count_upgrade: 2
boot_status_golden: 0
boot_status_upgrade: 1
first_bootloader: 1

----Upgrade----
boot_error: 0
boot_check_count_error_vc: 0
boot_check_count_error: 0
boot_timeout_vc: 0
boot_timeout: 0
boot_cs_good: 1
boot_config_error: 0
boot_version_error: 0
boot_config_error_code: 0
boot_error_code: 0
boot_cs_good: 1
boot_version_error: 0
boot1_cs_key_type: 1
boot1_cs_return_code: 0
boot1_cs_key_index: 5
boot2_cs_return_code: 0
boot2_cs_key_index: 5
boot2_cs_key_type: 1

----Other Registers----
fpga_version: 001b

Meraki do not include the build toolchain in their GPL archive. Luckily, I remembered that I have encountered this Marvell fork of u-boot before, for the Western Digital EX2100 which also uses a Marvell Armada 385 from the same family of ARMv7 CPU cores. Western Digital does include the Marvell toolchain used to compile u-boot in their GPL archive, good guy Western Digital!

To save anyone else the effort of setting up a development environment with an ancient version of GCC, I have created a Dockerfile that will handle building u-boot using the Marvell toolchain. You can find this work on GitHub.

I reached out to members of the Doozan forum who have been building Linux images for Marvell based NAS devices for many years to see if they had any more information about Secure Boot. Apparently Marvell CPUs will always kwboot before loading from other sources such as SPI or NAND:

Even if a box has secure boot in stock FW (u-boot, kernel,..), you should be able to kwboot it with a non-secure u-boot/spl binary.

I tried to kwboot the MS120 (with and without the Armada patches), but was unable to get it working. For some reason, the BootROM output is printed twice when kwboot is running, which I have not witnessed during any normal boot sequence:

$ ./kwboot -b u-boot.uart -f -t -B 115200 /dev/ttyUSB0
Sending boot message. Please reboot the target.../
BootROM 1.41
Pattern detected on UART-
BootROM 1.41
Pattern detected on UART/

uart.bin was created using tools/marvell/doimage:

$ ./doimage -T uart -D 0x0 -E 0x0 u-boot.bin u-boot.uart

Unfortunately, this investigation leaves us with more questions than answers.

  • What is the contents of the 32Mbit SPI flash?
    • Does the SmartFusion 2 only provide glue logic, or does it also protect/verify the contents of SPI flash?
  • Why won’t the Alleycat3 kwboot?
    • Is the duplicate output from BootROM when kwboot is invoked a clue?
  • What is the purpose of the header J17?
  • Why did Meraki switch from aluminum to steel?

The MS120 series is a completely different platform from the previous MS220 series, which used Vitesse ASICs with a MIPS core, 128MB of DDR2, 16MB of SPI, and 128MB of NAND flash.

The use of Secure Boot will complicate efforts to create a third-party firmware for the MS120 series. However, the more immediate issue is that kwboot does not work and there is no obvious copy of u-boot in SPI flash we can modify to alter the boot process.