Tag Archives: secure boot

Meraki MG21 hardware overview and secure boot bypass

Leave a reply

The Meraki MG21, introduced in 2019, is a Cat 6 LTE gateway intended for fail-over connectivity. It features a soldered modem module and either two internal (MG21) or two external (MG21E) antennas.

Meraki MG21 LTE gateway

Here is a summary of the MG21 specs:

  • Qualcomm IPQ4029 (ARM A7, 4 cores @ ~700MHz)
  • 512MB DDR3 RAM (soldered)
  • 128MB of NAND flash (Winbond W29N01HV)
  • Cinterion PLAS9-X LTE Cat 6 modem module (LCC, soldered)
  • Gigabit Ethernet (x2, QCA8072 PHY)
  • Nano-SIM slot

There are no screws holding the MG21 together, the case is glued. As Meraki used glue and not adhesive to hold the MG21 together, heat does not help in opening the device. To open the MG21/MG21E: guitar picks and Isopropyl alcohol are recommended, with a lot of patience.

Opening the MG21 with guitar picks and Isopropyl alcohol

The 3.3V UART header in the MG21 is J5, which is unpopulated, and follows the standard Meraki pinout (1: VCC, 2: Tx, 3: Rx, 4: GND) with a 115200 baud rate. It looks like Meraki may have planned to ship the MG21 with an integrated u-blox module (U22), however on my production units the module is absent.

540-00144-01 48RLEQ01.0GA 2019.08.22

With the summary aside, let us focus on the secure boot status of the device. For context, see Breaking secure boot on the Meraki Z3 and Meraki Go GX20.

U-Boot 2017.07-RELEASE-gf49d105aeb-dirty (Jul 13 2020 - 11:22:51 -0700)

DRAM:  242 MiB
machid : 0x8010001
Product: meraki_Tie_Fighter
NAND:  ONFI device found
128 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
machid: 8010001
ubi0: attaching mtd1
ubi0: scanning is finished
ubi0: attached mtd1 (name "mtd=0", size 112 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 896, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 4, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 235/60, WL threshold: 4096, image sequence number: 2046230850
ubi0: available PEBs: 157, total reserved PEBs: 739, PEBs reserved for bad PEB handling: 20


Secure boot enabled.

Read 0 bytes from volume part.safe to 84000000
No size specified -> Using max size (29196288)
Valid image
## Loading kernel from FIT Image at 84000028 ...

Foreshadowing: You will notice that this output is very similar to that of the Z3 and GX20.

Unfortunately changing the EEPROM value to the MR33 (stinkbug) does not work, because Meraki have removed support for the legacy non-secure boot devices from recent U-Boot builds:

U-Boot 2017.07-RELEASE-gf49d105aeb-dirty (Jul 13 2020 - 11:22:51 -0700)

DRAM:  242 MiB
machid : 0x8010001
No product detected! (Major Number 30)
NAND:  ONFI device found
128 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
machid: 8010001
ubi0: attaching mtd1
(...)

Secure boot enabled.

Removing the BGA NAND and replacing the u-boot region with a dump of the Z3 2018 U-Boot build, U-Boot is still performing signature validation:

U-Boot 2017.07-RELEASE-g39cabb9bf3 (May 24 2018 - 14:07:32 -0700)

DRAM:  242 MiB
machid : 0x8010001
NAND:  ONFI device found
128 MiB
Using default environment

(...)

Secure boot enabled.

The reason for this is that the EEPROM is not found. But why? We have a clue from the stock bootlog of the device:

[   15.287320] i2c /dev entries driver
[   15.302889] at24 0-0056: 8192 byte 24c64 EEPROM, writable, 32 bytes/write

The EEPROM in the MG21 has the address 0x56 instead of 0x50 as on the Z3. This causes the downgraded Z3 U-Boot build to not detect the EEPROM.

The Meraki Go GR10 (Maggot) also has the EEPROM at address 0x56:

struct eeprom_i2c_config
{
	uint16_t gpio_scl;
	uint16_t gpio_scl_func;
	uint16_t gpio_sda;
	uint16_t gpio_sda_func;
	uint16_t eeprom_addr;
};
/* valid eeprom configuration for insects */
static const struct eeprom_i2c_config valid_eeprom_i2c_config[] = {
    { 20, 1, 21, 1, 0x50 }, // Stinkbug, Ladybug, Noisy Cricket
    { 10, 4, 11, 4, 0x56 }, // Maggot
};

However, the GR10 uses different GPIO pins to access the EEPROM.

I do not have the U-Boot source code of the MG21 to review (see endnote). Lacking the U-Boot source code, we can hexdump the Z3 and MG21 U-Boot regions from the flash dumps and compare.

Z3:

00044360  0f 00 14 00 01 00 15 00  01 00 50 00 0a 00 04 00  |..........P.....|
00044370  0b 00 04 00 56 00 00 00  00 f0 f4 a1 ea ea fb 01  |....V...........|

MG21:

000452f0  0f 00 14 00 01 00 15 00  01 00 50 00 14 00 01 00  |..........P.....|
00045300  15 00 01 00 56 00 00 00  00 f0 f4 a1 ea ea fb 01  |....V...........|

Decoding the structs from the hexdump we can infer the C source code used in the MG21 U-Boot build:

static const struct eeprom_i2c_config valid_eeprom_i2c_config[] = {
    { 20, 1, 21, 1, 0x50 }, // Fuzzy Cricket, Fairyfly, Heart of Gold
    { 20, 1, 21, 1, 0x56 }, // Tie Fighter
};

The only difference between the MG21 and Z3 is in the EEPROM address, the GPIO configuration remains the same.

Reviewing the datasheet of the at24 EEPROM, we can see that the address is set by the first 3 pins (A0-A2) being pulled to ground or Vcc. Since the EEPROM has the address 0x56, that must correspond to the bitmask 110 or: A0: 0, A1: Vcc, A2: Vcc.

After some verification on the PCB, removing the surface mount resistor R50 (4.7k) above U6 will remove Vcc from A1 and A2, changing the EEPROM address from 0x56 to 0x50.

The signed Z3 2018 U-Boot build now properly detects the EEPROM at address 0x50 and disables signature validation on the payload.

The chain-loaded U-Boot I used as a proof-of-concept is based on the Z3 GPL source code provided by Meraki in 2021, which does not include support for the MG21. Networking is non-functional, which makes further development challenging as images must be (slowly) transferred via UART.

Some readers may be wondering about the MG41. This secure boot bypass does not work on the MG41.

Meraki has signed the MG41 bootloader with a unique device certificate (x-wing), so cross-flashing U-Boot from another device such as the Z3 will not work.

Although the FCC internal photos of the MG41 show both NAND and EMMC, the production MG41 has only EMMC present. The boot_meraki_qca function has been re-written as boot_meraki_mmc_qca. During this re-write, Meraki removed the vulnerable switch statement that aborts enforcing signature validation on legacy products.

tl;dr

  1. MG21 uses the same device signing certificate as the Z3 and GX20
  2. Overwrite u-boot on NAND with dump from Z3 running 2018 release
  3. Change Product ID in EEPROM to device without secure boot (MR33)
  4. Desolder R50 to change EEPROM address

The MG41 is not vulnerable to this technique.

Model Meraki Board Part number
MG21 Tie Fighter 600-89010
MG21E Tie Fighter 600-89010
MG41 X-Wing 600-119020
MG41E X-Wing (unknown, let me know in comments)

There is still a long road ahead to support the MG21 with any custom firmware such as OpenWrt. Downgrading U-Boot on the device is not easy due to the weather proofing of the device, and the use of BGA NAND.

The GPL source code for the MG21 and MG41 was requested from Meraki in April 2024. At the time of writing Meraki has not provided any of the requested source code.

Meraki announced the end of sale of the MG21 and MG41 in March 2024, and stopped selling the MG21 and MG41 on 2024-09-18.

Breaking secure boot on the Meraki Z3 and Meraki Go GX20

Leave a reply

Meraki used to be friendly to open-source. Older devices and Meraki firmwares offered a root shell over UART, and it was relatively easy to flash custom firmware to the device. Later, they began locking down devices so that people could no longer repurpose the hardware. Their latest “innovations” include the use of secure boot, ostensibly to improve device security but with the side effect of making claimed and EoL devices e-waste.

“Z3 is a remote work gateway with integrated Wi-Fi, secure access to corporate and multi-cloud resources, and features that delight SD-branch cloud platform users.”

Today we are looking at the Meraki Z3 and Meraki Go GX20. Both the Z3 and GX20 ship with secure boot enabled, meaning you cannot boot an alternative firmware like OpenWrt on the devices, and they are useless if the previous owner has not “unclaimed” them in Meraki’s dashboard.

Meraki chose to use the same u-boot release for their Qualcomm IPQ40xx based devices (MR33, MR30H, Z3, Go GX20). The MR33 ships without secure boot. However, since secure boot is enabled on the Z3 and Meraki Go GX20, u-boot on those devices must be signed with a certificate matching the one burned into the QFPROM fuses, or the device will not boot. In the case of the Z3, u-boot is signed with the fuzzy-cricket attestation ca1 certificate:

California1
San Francisco
Cisco Meraki1%0#
fuzzy-cricket attestation ca1
Product Security0
170728184745Z
170827184745Z0

The boot flow on an ipq40xx device with secure boot enabled can be summarized in the following diagram (adapted from the LineageOS docs):

Since Meraki use the same u-boot for devices with and without secure boot, u-boot must determine at some point whether it should enforce signature verification on the payload (a Flattened Image Tree Image). That check happens here:

static int do_meraki_qca_boot(cmd_tbl_t * cmdtp, int flag, int argc, char * const argv[])
{
   /* unsupported boards */
   switch(get_meraki_product_id()) {
      case MERAKI_BOARD_STINKBUG:
      case MERAKI_BOARD_LADYBUG:
      case MERAKI_BOARD_NOISY_CRICKET:
      case MERAKI_BOARD_YOWIE:
      case MERAKI_BOARD_BIGFOOT:
      case MERAKI_BOARD_SASQUATCH:
      case MERAKI_BOARD_WOOKIE:
         return 0;
      default:
         break;
   }

   /* Check 0: check/force secure boot on */
   force_secboot();

   /* Check 1: boot diagnostic */
   run_command("run check_boot_diag_img", 0);

   /* Check 2: boot part.new if upgrading */
   do_upgrade_boot();

   /* Check 3: if node specific unlock exists, chainload u-boot */
   if (devel_crt_valid()) {
      do_chainload_uboot();
   }

   setenv("part","part.safe");
   run_command("run boot_signedpart",0);

   setenv("part","part.old");
   run_command("run boot_signedpart",0);

   run_command("reset",0);

   return 0;
}

Note that if the board is STINKBUG (MR33), LADYBUG (MR74), NOISY_CRICKET (MR30H), YOWIE (MR42), BIGFOOT (MR52), SASQUATCH (MR53), or WOOKIE (MR84) then verification of the FIT image is skipped, as these devices don’t have secure boot enabled by default.

The 2017.07 u-boot update for the MR33 is a special case, as that doesn’t actually enable secure boot, it just maliciously bricks the device if the user attempts to interrupt boot.

How this practice is permitted under the EU Unfair Commercial Practices Directive (Directive 2005/29/EC of 2005) is quite frankly a mystery to me, but perhaps someone with more legal expertise can weigh in.

u-boot determines the device it is running on by reading the “Board major number” from an I2C EEPROM (24c64; silkscreen U32) on the PCB. Modifying the contents of the EEPROM from a device model with secure boot to a device without secure boot is enough to disable signature verification of payloads in u-boot. You can find the “board major number” byte at offset 0x49 and a backup copy at 0x1049 in the EEPROM.

Original Z3 boot output:

Format: Log Type - Time(microsec) - Message - Optional Info
Log Type: B - Since Boot(Power On Reset),  D - Delta,  S - Statistic
S - QC_IMAGE_VERSION_STRING=BOOT.BF.3.1.1-00096
S - IMAGE_VARIANT_STRING=DAACANAZA
S - OEM_IMAGE_VERSION_STRING=CRM
S - Boot Config, 0x00000025
S - Core 0 Frequency, 0 MHz
B -       261 - PBL, Start
B -      1340 - bootable_media_detect_entry, Start
B -      2615 - bootable_media_detect_success, Start
B -      2629 - elf_loader_entry, Start
B -      7204 - auth_hash_seg_entry, Start
B -   1382243 - auth_hash_seg_exit, Start
B -   1447600 - elf_segs_hash_verify_entry, Start
B -   1569288 - PBL, End
B -   1569312 - SBL1, Start
B -   1657984 - pm_device_init, Start
D -         6 - pm_device_init, Delta
B -   1659502 - boot_flash_init, Start
D -     87535 - boot_flash_init, Delta
B -   1751079 - boot_config_data_table_init, Start
D -     14006 - boot_config_data_table_init, Delta - (419 Bytes)
B -   1767784 - clock_init, Start
D -      7575 - clock_init, Delta
B -   1778758 - CDT version:2,Platform ID:8,Major ID:1,Minor ID:0,Subtype:1
B -   1782246 - sbl1_ddr_set_params, Start
B -   1787231 - cpr_init, Start
D -         2 - cpr_init, Delta
B -   1791720 - Pre_DDR_clock_init, Start
D -         5 - Pre_DDR_clock_init, Delta
D -     13140 - sbl1_ddr_set_params, Delta
B -   1804998 - pm_driver_init, Start
D -         2 - pm_driver_init, Delta
B -   1875945 - sbl1_wait_for_ddr_training, Start
D -        27 - sbl1_wait_for_ddr_training, Delta
B -   1893463 - Image Load, Start
D -   1311048 - QSEE Image Loaded, Delta - (268504 Bytes)
B -   3205010 - Image Load, Start
D -      2116 - SEC Image Loaded, Delta - (2048 Bytes)
B -   3215195 - Image Load, Start
D -   1307290 - APPSBL Image Loaded, Delta - (292616 Bytes)
B -   4522910 - QSEE Execution, Start
D -        56 - QSEE Execution, Delta
B -   4529089 - SBL1, End
D -   2961858 - SBL1, Delta
S - Flash Throughput, 1970 KB/s  (563587 Bytes,  285975 us)
S - DDR Frequency, 672 MHz


U-Boot 2017.07-RELEASE-g39cabb9bf3 (May 24 2018 - 14:07:32 -0700)

DRAM:  242 MiB
machid : 0x8010001
Product: meraki_Fuzzy_Cricket
NAND:  ONFI device found
ID = 1d80f101
Vendor = 1
Device = f1
128 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
machid: 8010001
ubi0: attaching mtd1
ubi0: scanning is finished
ubi0: attached mtd1 (name "mtd=0", size 112 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 896, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 124/51, WL threshold: 4096, image sequence number: 1218587189
ubi0: available PEBs: 337, total reserved PEBs: 559, PEBs reserved for bad PEB handling: 20


Secure boot enabled.

Read 0 bytes from volume part.safe to 84000000
No size specified -> Using max size (16584704)
Valid image
## Loading kernel from FIT Image at 84000028 ...
   Using 'config@1' configuration
   Trying 'kernel@1' kernel subimage
     Description:  wired-arm-qca Kernel
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0x84000134
     Data Size:    1962384 Bytes = 1.9 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x80208000
     Entry Point:  0x80208000
     Hash algo:    sha1
     Hash value:   1a716f7999511396baa166ef3986165f40c4c1c7
   Verifying Hash Integrity ... sha1+ OK
## Loading ramdisk from FIT Image at 84000028 ...
   Using 'config@1' configuration
   Trying 'ramdisk@1' ramdisk subimage
     Description:  wired-arm-qca Ramdisk
     Type:         RAMDisk Image
     Compression:  uncompressed
     Data Start:   0x841df3b8
     Data Size:    14496712 Bytes = 13.8 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x82200000
     Entry Point:  0x82200000
     Hash algo:    sha1
     Hash value:   6c03ebf4feff11ed19dadcd2b6afe329d74e6671
   Verifying Hash Integrity ... sha1+ OK
   Loading ramdisk from 0x841df3b8 to 0x82200000
## Loading fdt from FIT Image at 84000028 ...
   Using 'config@1' configuration
   Trying 'fdt@1' fdt subimage
     Description:  Fuzzy Cricket Device Tree
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x84fb2874
     Data Size:    38216 Bytes = 37.3 KiB
     Architecture: ARM
     Hash algo:    sha1
     Hash value:   25733212c52b5e5803c8fe02a64fd229f30a2ac4
   Verifying Hash Integrity ... sha1+ OK
   Loading fdt from 0x84fb2874 to 0x89000000
   Booting using the fdt blob at 0x89000000
   Loading Kernel Image ... OK
   Using Device Tree in place at 89000000, end 8900c547
Using machid 0x8010001 from environment

Starting kernel ...

After modifying the Z3 EEPROM to have the board major number of the MR33:

Format: Log Type - Time(microsec) - Message - Optional Info
Log Type: B - Since Boot(Power On Reset),  D - Delta,  S - Statistic
S - QC_IMAGE_VERSION_STRING=BOOT.BF.3.1.1-00096
S - IMAGE_VARIANT_STRING=DAACANAZA
S - OEM_IMAGE_VERSION_STRING=CRM
S - Boot Config, 0x00000025
S - Core 0 Frequency, 0 MHz
B -       261 - PBL, Start
B -      1340 - bootable_media_detect_entry, Start
B -      2615 - bootable_media_detect_success, Start
B -      2629 - elf_loader_entry, Start
B -      7204 - auth_hash_seg_entry, Start
B -   1382292 - auth_hash_seg_exit, Start
B -   1447925 - elf_segs_hash_verify_entry, Start
B -   1569633 - PBL, End
B -   1569657 - SBL1, Start
B -   1658338 - pm_device_init, Start
D -         6 - pm_device_init, Delta
B -   1659859 - boot_flash_init, Start
D -     87566 - boot_flash_init, Delta
B -   1751466 - boot_config_data_table_init, Start
D -     14002 - boot_config_data_table_init, Delta - (419 Bytes)
B -   1768163 - clock_init, Start
D -      7570 - clock_init, Delta
B -   1779133 - CDT version:2,Platform ID:8,Major ID:1,Minor ID:0,Subtype:1
B -   1782622 - sbl1_ddr_set_params, Start
B -   1787606 - cpr_init, Start
D -         2 - cpr_init, Delta
B -   1792095 - Pre_DDR_clock_init, Start
D -         5 - Pre_DDR_clock_init, Delta
D -     13141 - sbl1_ddr_set_params, Delta
B -   1805373 - pm_driver_init, Start
D -         2 - pm_driver_init, Delta
B -   1876249 - sbl1_wait_for_ddr_training, Start
D -        27 - sbl1_wait_for_ddr_training, Delta
B -   1893981 - Image Load, Start
D -   1311108 - QSEE Image Loaded, Delta - (268504 Bytes)
B -   3205588 - Image Load, Start
D -      2122 - SEC Image Loaded, Delta - (2048 Bytes)
B -   3215674 - Image Load, Start
D -   1307310 - APPSBL Image Loaded, Delta - (292616 Bytes)
B -   4523411 - QSEE Execution, Start
D -        56 - QSEE Execution, Delta
B -   4529587 - SBL1, End
D -   2962011 - SBL1, Delta
S - Flash Throughput, 1969 KB/s  (563587 Bytes,  286173 us)
S - DDR Frequency, 672 MHz


U-Boot 2017.07-RELEASE-g39cabb9bf3 (May 24 2018 - 14:07:32 -0700)

DRAM:  242 MiB
machid : 0x8010001
Product: meraki_Stinkbug
NAND:  ONFI device found
ID = 1d80f101
Vendor = 1
Device = f1
128 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
machid: 8010001
ubi0: attaching mtd1
ubi0: scanning is finished
ubi0: attached mtd1 (name "mtd=0", size 112 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 896, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 124/52, WL threshold: 4096, image sequence number: 1218587189
ubi0: available PEBs: 337, total reserved PEBs: 559, PEBs reserved for bad PEB handling: 20
Read 0 bytes from volume part.safe to 84000000
No size specified -> Using max size (16584704)
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Read 0 bytes from volume part.old to 84000000
No size specified -> Using max size (16547840)
Wrong Image Format for bootm command
ERROR: can't get kernel image!
resetting ...

Note the change in output, the device was previously meraki_Fuzzy_Cricket but is now identified as meraki_Stinkbug, so the EEPROM modification worked. Unfortunately, the format of the FIT images differs between secure boot and non-secure boot, so the device no longer boots the stock firmware.

The MR33 board major number was chosen as u-boot selects config@1 from the FIT image for the MR33, which is the same config index used by default for the Z3. It would be possible to select any of the device models mentioned above that don’t implement secure boot, but doing so would mean u-boot attempts to boot from a different config@ entry.

Since the Z3 and Meraki Go GX20 have secure boot enabled, we cannot replace u-boot or the device will fail to boot.

0x000000000000-0x000000100000 : "sbl1"
0x000000100000-0x000000200000 : "mibib"
0x000000200000-0x000000300000 : "bootconfig"
0x000000300000-0x000000400000 : "qsee"
0x000000400000-0x000000500000 : "qsee_alt"
0x000000500000-0x000000580000 : "cdt"
0x000000580000-0x000000600000 : "cdt_alt"
0x000000600000-0x000000680000 : "ddrparams"
0x000000700000-0x000000900000 : "u-boot"
0x000000900000-0x000000b00000 : "u-boot-backup"
0x000000b00000-0x000000b80000 : "ART"
0x000000c00000-0x000007c00000 : "ubi"

However, by modifying the device major number, the (signed) u-boot will now allow booting unsigned payloads. The easiest way to proceed is to then build our own modified u-boot as a FIT image, and put it into the ubivol part.safe. After we do this, the boot process now looks like:

Format: Log Type - Time(microsec) - Message - Optional Info
Log Type: B - Since Boot(Power On Reset),  D - Delta,  S - Statistic
S - QC_IMAGE_VERSION_STRING=BOOT.BF.3.1.1-00096
S - IMAGE_VARIANT_STRING=DAACANAZA
S - OEM_IMAGE_VERSION_STRING=CRM
S - Boot Config, 0x00000025
S - Core 0 Frequency, 0 MHz
B -       261 - PBL, Start
B -      1340 - bootable_media_detect_entry, Start
B -      2615 - bootable_media_detect_success, Start
B -      2629 - elf_loader_entry, Start
B -      7270 - auth_hash_seg_entry, Start
B -   1382352 - auth_hash_seg_exit, Start
B -   1447639 - elf_segs_hash_verify_entry, Start
B -   1569353 - PBL, End
B -   1569377 - SBL1, Start
B -   1658051 - pm_device_init, Start
D -         6 - pm_device_init, Delta
B -   1659570 - boot_flash_init, Start
D -     87594 - boot_flash_init, Delta
B -   1751205 - boot_config_data_table_init, Start
D -     14010 - boot_config_data_table_init, Delta - (419 Bytes)
B -   1767914 - clock_init, Start
D -      7572 - clock_init, Delta
B -   1778887 - CDT version:2,Platform ID:8,Major ID:1,Minor ID:0,Subtype:1
B -   1782376 - sbl1_ddr_set_params, Start
B -   1787361 - cpr_init, Start
D -         2 - cpr_init, Delta
B -   1791851 - Pre_DDR_clock_init, Start
D -         5 - Pre_DDR_clock_init, Delta
D -     13143 - sbl1_ddr_set_params, Delta
B -   1805130 - pm_driver_init, Start
D -         2 - pm_driver_init, Delta
B -   1876340 - sbl1_wait_for_ddr_training, Start
D -        27 - sbl1_wait_for_ddr_training, Delta
B -   1893870 - Image Load, Start
D -   1312117 - QSEE Image Loaded, Delta - (268504 Bytes)
B -   3206486 - Image Load, Start
D -      2118 - SEC Image Loaded, Delta - (2048 Bytes)
B -   3216578 - Image Load, Start
D -   1308369 - APPSBL Image Loaded, Delta - (292616 Bytes)
B -   4525372 - QSEE Execution, Start
D -        56 - QSEE Execution, Delta
B -   4531549 - SBL1, End
D -   2964253 - SBL1, Delta
S - Flash Throughput, 1969 KB/s  (563587 Bytes,  286154 us)
S - DDR Frequency, 672 MHz


U-Boot 2017.07-RELEASE-g39cabb9bf3 (May 24 2018 - 14:07:32 -0700)

DRAM:  242 MiB
machid : 0x8010001
Product: meraki_Stinkbug
NAND:  ONFI device found
ID = 1d80f101
Vendor = 1
Device = f1
128 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
machid: 8010001
ubi0: attaching mtd1
ubi0: scanning is finished
ubi0: attached mtd1 (name "mtd=0", size 112 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 896, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 5, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 214/99, WL threshold: 4096, image sequence number: 2086049366
ubi0: available PEBs: 0, total reserved PEBs: 896, PEBs reserved for bad PEB handling: 20
Read 0 bytes from volume part.safe to 84000000
No size specified -> Using max size (507904)
## Loading kernel from FIT Image at 84000000 ...
   Using 'config@1' configuration
   Trying 'kernel-1' kernel subimage
     Description:  Kernel
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0x840000d0
     Data Size:    381216 Bytes = 372.3 KiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x87300000
     Entry Point:  0x87300000
     Hash algo:    sha1
     Hash value:   89c319b76738e71147631f87311fc8f31e8ac8aa
   Verifying Hash Integrity ... sha1+ OK
## Loading fdt from FIT Image at 84000000 ...
   Using 'config@1' configuration
   Trying 'fdt-1' fdt subimage
     Description:  Insect DTB
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x8405d2d4
     Data Size:    235 Bytes = 235 Bytes
     Architecture: ARM
     Hash algo:    sha1
     Hash value:   86c47255d86f2bd6301e7772ca65b3548493875b
   Verifying Hash Integrity ... sha1+ OK
   Booting using the fdt blob at 0x8405d2d4
   Loading Kernel Image ... OK
   Using Device Tree in place at 8405d2d4, end 840603be
Using machid 0x8010001 from environment

Starting kernel ...



U-Boot 2017.07-DEVEL (Apr 01 2024 - 14:50:01 +0000)

DRAM:  242 MiB
machid : 0x8010001
Product: meraki_Stinkbug
NAND:  ONFI device found
ID = 1d80f101
Vendor = 1
Device = f1
128 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
machid: 8010001
Net:   MAC0 addr:e0:cb:bc:11:22:33
PHY ID1: 0x4d
PHY ID2: 0xd0b1
ipq40xx_ess_sw_init done
eth0
Autoboot in 5 seconds
FUZZY CRICKET #

We could proceed by putting OpenWrt directly in part.safe however if we ever had a bad flash there would be no way to recover except by manually reflashing NAND. Chain loading a u-boot build does delay booting by a few seconds, but offers a way to recover from a bad flash with just UART and tftpboot.

With the modified u-boot on flash, we can now tftpboot the OpenWrt initramfs and install it to flash with sysupgrade.

“Meraki Go offers simple networks for serious business.”

The Meraki Go GX20 uses the same PCB as the Z3, but without the WiFi radios populated (the lack of WiFi means you won’t find the GX20 in the FCC database). So, does the above technique work for the Meraki Go GX20?

U-Boot 2012.07-g03cdfe19e00f [local,local] (Aug 29 2017 - 11:59:45)

DRAM:  498 MiB
machid : 0x8010001
ERROR: Unknown board
NAND:  ONFI device found
ID = 1d80f101
Vendor = 1
Device = f1
128 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
machid: 8010001
Hit any key to stop autoboot:  0
Creating 1 MTD partitions on "nand0":
0x000000c00000-0x000007f00000 : "mtd=0"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=0"
UBI: MTD device size:            115 MiB
UBI: number of good PEBs:        920
UBI: number of bad PEBs:         0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     7
UBI: available PEBs:             364
UBI: total number of reserved PEBs: 556
UBI: number of PEBs reserved for bad PEB handling: 9
UBI: max/mean erase counter: 2/0
Read 0 bytes from volume part.safe to 84000000
No size specified -> Using max size (3284992)
## Booting kernel from FIT Image at 84000000 ...
   Using 'config@1' configuration
   Verifying Hash Integrity ... sha384,secp384r1:wired-arm-qca-RT-SECP384R1_1-rel+ OK
   Trying 'kernel@1' kernel subimage

No, unfortunately for u-boot 2012.07 which shipped on the Meraki Go GX20 I purchased, changing the device major results in ERROR: Unknown board and we can see from the sha384,secp384r1:wired-arm-qca-RT-SECP384R1_1-rel+ OK output that signature checking is still enabled.

However, the Meraki Go GX20 is a very similar device to the Z3. Is it possible that there is another way?

Examining the u-boot region of the Meraki Go GX20 with strings reveals the following:

California1
San Francisco
Cisco Meraki1%0#
fuzzy-cricket attestation ca1
Product Security0
170728184745Z
170827184745Z0

So the u-boot binary on the Meraki Go GX20 is signed with the same signing cert as u-boot on the Z3. What if we replace u-boot on the Meraki Go GX20 with the signed u-boot from the Z3 dump?

U-Boot 2017.07-RELEASE-g39cabb9bf3 (May 24 2018 - 14:07:32 -0700)

DRAM:  242 MiB
machid : 0x8010001
Product: meraki_Stinkbug
NAND:  ONFI device found
ID = 1d80f101
Vendor = 1
Device = f1
128 MiB
Using default environment

Bingo. And since the Z3 2017.07 u-boot release contains the secure boot downgrade bug, we can exploit it on the Meraki Go GX20 as well.

I am curious to see how u-boot 2012.07 on the Meraki Go GX20 implements signature validation, since unlike u-boot 2017.07 changing the device major does not disable signature verification of the payload.

Unfortunately, this is not possible. The GPL source code for the Meraki Go GX20 was requested in December 2022 and Meraki has yet to provide the u-boot source code for the device.

The EU and other jurisdictions claim to take the right to repair and e-waste seriously, but their actions thus far have ignored the elephant in the room. Many of the devices being sold in the past 5 years come with secure boot enabled, and thus are locked to running the OEM’s software.

In the case of the Meraki prodcuts, you have purchased the device and Meraki are selling a license only for the cloud management and hardware replacement service. But, if you stop paying Meraki, buy a “claimed” second-hand device, or Meraki discontinue support for the device you now have a worthless brick.

“Cisco Meraki may find it necessary to discontinue products for a number of reasons, including product line enhancements, market demand, technology innovation, or if the product simply matures over time and needs to be replaced by something functionally richer.”

To be clear, you are not leasing the hardware, and this is not “Hardware as a Service” you have bought the device in question and own it.

There are hundreds of SMD resistors on the device’s PCB. Any one of them could be tied to a GPIO which is polled by the SoC BootROM to enable or disable secure boot. Incorporating this into the hardware design would cost nothing, and would allow consumers the choice to re-use devices that were resold without proper decommissioning, from companies that were liquidated, from devices that were recycled as e-waste, or from devices where the manufacturer has decided to “maximize shareholder value” and end support. Physically opening the device and adding or removing a component to disable secure boot does not compromise user security in any way.

Routers like the Z3 and Meraki Go GX20 may appear to the untrained eye to be highly specialized devices, but in reality they are very similar to a Raspberry Pi with additional network interfaces, e.g. they are general purpose embedded computers.

It is past time that regulators start considering the user-harmful practices of companies in enforcing secure boot on these devices with no way for consumers to exercise their rights. This anti-consumer behaviour increases costs and creates more e-waste, solely for the benefit of the bottom line of companies that are (often) not even based in the same economic area. We should not, and cannot, accept such a status quo. There is no reason for these devices to become e-waste!

Impact: Device owners with physical access to their device and the appropriate hardware flashing tools can install a custom firmware.

Could this be accomplished remotely? No, not unless it is chained with another exploit providing Remote Code Execution (RCE) on the device. Anyone with an RCE on Meraki devices would not waste it on flashing a FOSS firmware, they would use it to build a botnet.

Can Meraki patch this? Yes, see below it is trivial for them to add a check in u-boot 2017.07 whether secure boot is enabled and enforce signature verification. Patching this is left as an exercise to the reader.

Will they patch it? Meraki announced the EoL of the Z3 in March 2024. They have previously been rumoured to update u-boot to brick devices when users attempt to flash a custom firmware. It remains to be seen if they will continue this anti-consumer practice on the Z3 and Meraki Go GX20.

Responsible disclosure: Yes, this has been responsibly disclosed to you, the device owner.

2024-06: The above method has also been confirmed to bypass FIT signature verification on the Meraki Z3C.

2024-07: Meraki inadvertently addressed this vulnerability in U-Boot builds dated U-Boot 2017.07-RELEASE-gf49d105aeb-dirty (Jul 13 2020 - 11:22:51 -0700) and later when they removed support for devices without secure boot (MR33, MR30H, MR74).

Disabling Secure Boot on Intel Quark “secure SKU” silicon

3 Replies

Secure Boot is a bit like SELinux: people who use it really like it, and tell all their friends to use it. For everyone else, apart from those who don’t know about or even notice Secure Boot, it’s an annoyance that they almost immediately disable.

We’ve looked at the Intel DK200 from a hardware perspective before. Now it’s time to look at it from a software perspective. “Internet of Things Gateway” is pretty generic, so what can it actually do?

Following the instructions, I tried to register the system on Intel’s website so I could download the Wind River Intelligent Device Platform XT 2.0 SDK. I didn’t get very far:

No WindRiver SDK for you

Stormtrooper #1: This is not the product you’re looking for

Yeah… I guess this is what Mouser meant when they said the DK200 was End of Life.

Since this ships with the Linux Kernel, which is GPLv2 licensed, I believe Intel may be violating the GPL. Specifically:

Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange

But I am not a lawyer, and I am not really that interested in starting a legal battle over the source code for an ancient version of Wind River Linux I am not interested in using anyway.

So let’s go try to build Yocto. The Intel rep did say there was a Yocto BSP coming “soon” but “soon” in Intel time seems kind of variable.

After some hiccups (Yocto needs python2 and GCC <6) I had built a Yocto image and put it on an SD card. Does it boot?

...no

…no

So we can’t boot Yocto because this is a “secure SKU” which means Secure Boot is enabled. Is there some way we can disable Secure Boot? What about updating the BSP to a newer version with Secure Boot disabled?

Back to hardware
If I’ve learned anything from messing around with electronics, you want to make a backup before you start modifying things. This is doubly so if the data in question is related to the booting process. It sucks to end up with a brick, so make a backup!

Taking a backup of flash

Taking a backup of flash

The Intel Quark guide mentions using a Dediprog SF100 to flash EDKII. I don’t have a Dediprog, but I do have an SPI programmer. Unfortunately, none of the Intel documentation I could find mentions the Dediprog header on the DK200, so I had to go hunting.

I traced the pins from the Winbond flash to header J23. J23 is only 8 pins, so trial and error with a multimeter to find the pin mapping wasn’t terrible:

J23 pinout

J23 pinout

Here’s the pinout of J23 in text form:

J23 pin 25Q64 pin Pin description
1 8 VCC
2 4 GND
3 1 /CS
4 6 CLK
5 2 DO
6 5 DI
7 Not connected
8 Not connected

/WP and HOLD pins on the 25Q64FV are not routed to J23, but they aren’t required for flashing.

With the pinout known, I could attach the SPI programmer to the header instead of using the chip clip:

J23 to ch341a SPI programmer

J23 to ch341a SPI programmer

I took a dump of the Winbond 25Q64FV and then for good measure desoldered the chip and read it again to confirm the images were exactly the same. It was strange because the image from the chip clip wasn’t identical. But, the image from the desoldered chip was identical to the image taken from J23, so we’re done here. I wrote the image to a new 25Q64FV and soldered that back onto the board.

Firmware disassembly
Disassembling the firmware which shipped on my DK200, we see that a Secure Boot certificate was created by WindRiver.

I assume that had I been able to download the WindRiver SDK, I would have been able to build and sign Secure Boot with my own certificates. Given that industrial customers spend a lot of time and money worrying about security, I was surprised to see that the Secure Boot certificate in the firmware was created by WindRiver China.

I did try to load up the image in IDA, but not being a power user of IDA, I couldn’t figure out how to get it to analyze the SPI dump, and gave up to try and compile the firmware from source.

Building the BSP

Being Intel, there are hundreds of pages you can read about developing for EDK2 and other really fun things, probably. I didn’t read them.

A document which I did end up reading religiously was the Intel ® QuarkTM SoC X1000 Board Support Package (BSP) Build and Software User Guide [PDF] which describes how to build all the firmware components needed to bring up the X1000 SoC. I found out there is actually a newer version of this document (1.2.1 instead of 1.1) and there are some important differences between the documents I want to get to later.

By building the firmware, we’re hoping for one of two outcomes:

  1. A firmware with our own Secure Boot certificates, or
  2. A firmware which has Secure Boot disabled

Version 1.1 of the BSP Build and Software User Guide includes a section on pages 29 and 30 on how to bundle your own db, kek, and pk certificates:

Page 29 and 30 condensed

Unfortunately if you follow the instructions and try to use a layout.conf which specifies these files, you’ll get an error because there’s no address specified for this data in the image:

I do have a reference file from WindRiver with Secure Boot certificates, so if I was really interested in making Secure Boot work as intended, I could have reverse engineered the address to store the certificates.

The certificates section of layout.conf was removed from the 1.2.1 revision of the BSP Build and Software User Guide. I guess since it no longer works, Intel decided to remove it from the documentation.

So, we can’t install our own Secure Boot certificates in the firmware. What happens if we just leave out the certificates section entirely and build it?

Error 37: Quark signature file not found

Right, so even though there’s now no certificate in the firmware bundle, we still can’t boot.

Interestingly, if you don’t partition the uSD or USB stick correctly, you end up with this pretty screen:

I never saw that in the stock firmware.

Hacking GRUB
So it seems that we can’t include our own Secure Boot certificate in the firmware, due to the sample layout.conf file missing the certificates section, and not knowing the appropriate address to store the certificates.

What if we dig into Error 37: Quark signature file not found a bit more?

If you look in the grub source code included in the BSP, you can see a giant ~1000KB patch that Intel has made to the original upstream code to support the Quark platform.

If you grep for “Quark signature file not found” you’ll find it was added in stage2/common.c:
diff --git a/stage2/common.c b/stage2/common.c
index e96bec2..e122745 100644
--- a/stage2/common.c
+++ b/stage2/common.c
@@ -88,6 +88,8 @@ char *err_list[] =
[ERR_UNRECOGNIZED] = "Unrecognized command",
[ERR_WONT_FIT] = "Selected item cannot fit into memory",
[ERR_WRITE] = "Disk write error",
+ [ERR_QUARK_VERIFICATION] = "Quark signature verification failed",
+ [ERR_SGN_FILE_NOT_FOUND] = "Quark signature file not found",
};

If you grep for ERR_SGN_FILE_NOT_FOUND you’ll find it’s in the following files:
./work/efi/ia32/loader/linux.c:410: errnum = ERR_SGN_FILE_NOT_FOUND;
./work/efi/ia32/loader/linux.c:732: errnum = ERR_SGN_FILE_NOT_FOUND;
./work/efi/quark/boot_settings.c:190: errnum = ERR_SGN_FILE_NOT_FOUND;

Going back to Intel’s modifications to grub, we can see what they added:

It takes a bit of searching, but if you strip out all of the grub_quark_secure logic from linux.c and boot_settings.c, you end up with…

Ta-da! I can boot Yocto Linux

No more Secure Boot!

At the end of the day, the Quark X1000 is an x86: “secure SKU” is nothing but a fuse setting.

The comment should read:

Determine whether or not grub should enforce Secure Boot.

In our case, this is not a mandatory option 😉

Special offer for DK200 owners
As shown above, it is possible to modify the Intel sources to disable Secure Boot. If there are other people have a DK200 from Intel and are interested in running a firmware without Secure Boot, leave a comment with your contact details. Upon request, I can provide a firmware image* with generic Ethernet MAC addresses for you to flash. Note that this firmware is specific to the DK200 (Clanton Hill) hardware.

* No warranty, express or implied, provided for said firmware image. You flash at your own risk!