coreboot for CompuLab Intense PC

I am very pleased to announce that coreboot now supports the CompuLab Intense PC and MintBox 2! 🎉🎉


Building coreboot
The instructions for building coreboot yourself can be found on the coreboot Wiki. You will need a Linux system with typical development packages installed such as build-essential.

Select CompuLab and Intense-PC in the Mainboard section of the coreboot menuconfig:

You need to decide at this point whether you wish to use the internal full-height PCI-Express slot for mSATA or as PCI-Express:

If you have not installed an additional mSATA SSD in your Intense PC, then you do not need to select this option. Selecting the mSATA option is only required if you have installed an mSATA SSD and want to use it in the Intense PC:

Because coreboot does not have full support for the embedded controller (EC) in the Intense PC right now, the choice of using mSATA or PCIe cannot be made at runtime. If later you wish to change the function of the slot, you need to rebuild coreboot while selecting the appropriate choice of mSATA or PCIe.

Note that the mSATA port is limited to SATA 3Gbps speeds. This is a hardware limitation of the Intense PC design, and cannot be changed by flashing coreboot.


It is important to include the Firmware Descriptor Table (FDT), ME, and GbE regions of flash. Specify these files in the Chipset section:

You can choose yourself if you want to run me_cleaner on the ME or not. Note that if you choose to run me_cleaner, all SATA ports will cease to function. This is not a coreboot specific bug, the same behaviour occurs on the CompuLab firmware when me_cleaner is run. It may or may not be possible to fix this issue, more research is needed to understand the root cause.


If you want to have video during POST, you also need to include the Intel VGA BIOS in the image. Specify this in the Devices section:

In theory coreboot graphics init is supposed to initialize the Intel HD graphics without the need for the VGA BIOS, however without the VGA BIOS I was unable to get any video output until the Linux kernel started booting. This makes using the bootloader menu or troubleshooting pre-boot issues very difficult.


I would recommend you enable logging to cbmem at a minimum. This will allow you to access the coreboot boot log in Linux using the cbmem utility. If you have trouble booting the Intense PC after flashing coreboot, I would recommend you enable logging to UART, and use the included serial dongle to debug coreboot via RS-232 (115200n8). UART support for the Intense PC should be accepted to coreboot master shortly.


The default boot order of SeaBIOS seems to be SATA HDD if present, then PXE boot (if compiled with iPXE). It is possible and easy to change this, by specifying a bootorder file to include in cbfs when building coreboot.

I have created a boot order file which searches for boot devices in the following order:

  1. USB 2.0 devices
  2. USB 3.0 devices
  3. SATA devices (in order: 2.5″ internal, mSATA, eSATA, FACE module)
  4. iPXE

You can download the bootorder file and include it in cbfs. If you don’t include iPXE as a payload, remove the last line of the bootorder.txt file. If you are not building SeaBIOS as a payload, then you do not require this file.


After building coreboot, but before flashing, we need to split the coreboot.rom file into two 8MB files. This is because the Intense PC has two 8MB NOR flash chips totaling 16MB.

Split the coreboot.rom file into two 8MB files “SC1.bin” and “SC2.bin” using dd:
$ dd if=build/coreboot.rom of=SC1.bin bs=1M count=8
$ dd if=build/coreboot.rom of=SC2.bin bs=1M skip=8


Extracting binary firmware components
You may notice above that several portions of the initial Intense PC firmware are required to successfully build coreboot. The Intel Descriptor file (otherwise known as the Flash Descriptor Table or FDT), Management Engine firmware, Gigabit Ethernet region, and VGA BIOS.

If you have not yet installed the CompuLab firmware update to address CVE-2017-8083, you should be able to dump the entire firmware using flashrom in Linux:
# flashrom -p internal:laptop=force_I_want_a_brick -r intense_pc.bin

If you have already patched your system, then flashrom will be unable to dump the firmware:
Enabling flash write... Warning: SPI Configuration Lockdown activated.
FREG0: Warning: Flash Descriptor region (0x00000000-0x00000fff) is read-only.
FREG2: Warning: Management Engine region (0x00003000-0x00cfffff) is locked.

You will have to use a hardware method to dump the firmware from the chips. As an example, using a ch341 based SPI programmer and flashrom:
# flashrom -p ch341a_spi -r sc1.bin
# flashrom -p ch341a_spi -r sc2.bin

Confirm that sc1.bin contains the start of the flash descriptor:
$ hexdump -C sc1.bin | head -2
00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000010 5a a5 f0 0f 03 01 04 03 06 02 10 12 20 01 21 00 |Z........... .!.|

Do not forget to concatenate the firmware together into a 16MB image before running ifdtool:
$ cat sc1.bin sc2.bin > intense_pc.bin

You can then extract the regions from the firmware using ifdtool (included in coreboot/utils):
$ ifdutil -x intense_pc.bin

The output of ifdtool should appear as follows:
File intense_pc.bin is 16777216 bytes
Flash Region 0 (Flash Descriptor): 00000000 - 00000fff
Flash Region 1 (BIOS): 00d00000 - 00ffffff
Flash Region 2 (Intel ME): 00003000 - 00cfffff
Flash Region 3 (GbE): 00001000 - 00002fff
Flash Region 4 (Platform Data): 00fff000 - 00000fff (unused)

ifdtool will output files for each flash region. For building coreboot, we are interested in the following regions:
flashregion_0_flashdescriptor.bin
flashregion_2_intel_me.bin
flashregion_3_gbe.bin


If ifdtool doesn’t work for some reason, verify that you have concatenated the firmware files in the correct order. Or, if you don’t want to use ifdtool, you can split it manually using dd.

Intel Descriptor file/FDT:
$ dd if=intense_pc.bin of=descriptor.bin bs=4096 count=1

Management Engine:
$ dd if=intense_pc.bin of=me.bin bs=4096 count=1028 skip=3

Gigabit Ethernet region:
$ dd if=intense_pc.bin of=gbe.bin bs=4096 count=2 skip=1


You can extract the VGA BIOS from within Linux. First, verify the PCI ID of the Intel integrated graphics controller. On the Intense PC, this should be 00:02.0:
$ lspci | grep Graphics
00:02.0 VGA compatible controller [0300]: Intel Corporation 3rd Gen Core processor Graphics Controller [8086:0166] (rev 09)

Once you have confirmed the PCI ID, you can dump the VGA BIOS to a file:
# echo 1 > /sys/devices/pci0000:00/0000:00:02.0/rom
# cat /sys/devices/pci0000:00/0000:00:02.0/rom > vgabios.bin
# echo 0 > /sys/devices/pci0000:00/0000:00:02.0/rom

vgabios.bin should be exactly 65536 bytes and begin similar to the following:
$ hexdump -C vgabios.bin | head -2
00000000 55 aa 78 e9 b8 e9 30 30 30 30 30 30 30 30 30 30 |U.x...0000000000|
00000010 30 30 90 24 e9 a9 23 a0 40 00 b0 0a 30 30 49 42 |00.$..#[email protected]|

It is recommended to also dump the video bios table (VBT) to a file to include in cbfs, as the VBT table is expected by Windows:
# cat /sys/kernel/debug/dri/0/i915_vbt > vbt.bin

vbt.bin should be exactly 6144 bytes and look similar to the following:
$ hexdump -C vbt.bin | head -2
00000000 24 56 42 54 20 53 4e 42 2f 49 56 42 2d 4d 4f 42 |$VBT SNB/IVB-MOB|
00000010 49 4c 45 20 64 00 30 00 b8 10 e7 00 30 00 00 00 |ILE d.0.....0...|


Flashing coreboot

The following instructions are provided AS-IS and with no warranty, express or implied. Flashing coreboot can turn your computer into a brick and will void your warranty. By following these instructions you acknowledge these risks and assume all liability.

To flash coreboot onto your Intense PC, you will need an SPI programmer supported by flashrom.

An inexpensive option is a CH341 based SPI programmer (<$2 USD from eBay/AliExpress):

Another useful tool which can also be used for flashing is the Bus Pirate (~$25 USD):

The Raspberry Pi should also work. Here is a detailed post on how to use the Raspberry Pi to flash firmware to a NOR flash.

Unfortunately for us, the NOR flash in the Intense PC is in a WSON package (very very thin small outline no lead package) so a SOIC clip or SOIC socket will not work.

Because the Intense PC uses the chassis as a heat sink, you need to remove the motherboard from the Intense PC chassis to access the NOR flash. To do this, first remove the hard drive and hard drive carrier secured by a single screw:

Next, remove the 4 screws securing the bottom plate to the chassis:

Next, remove the retaining screw of the FACE module:

Next, remove the screw and two stand-offs securing the motherboard to the chassis. The screw is by the Ethernet ports, and the two stand-offs: one near the audio ports and one under the FACE module:

Disconnect the WiFi antennas (if installed) and disconnect the front panel connector near the SODIMM sockets. You should now be able to lift the motherboard out of the chassis.

You will find the two NOR flash modules on the reverse side of the motherboard:

You will need to solder connections to the pads beside each chip to back up the original firmware and to flash coreboot.

If you’re using the ch341 based programmer, then the flashrom commands would be the following:
For the NOR flash near SC1: $ sudo flashrom -p ch341a_spi -w SC1.bin
For the NOR flash near SC2: $ sudo flashrom -p ch341a_spi -w SC2.bin


Conclusion
If you value open-source software and want an alternative to the closed-source and infrequently updated CompuLab firmware, then coreboot is a great choice for the Intense PC/MintBox 2.

However building and flashing coreboot on the Intense PC is not without its risks. You will void your warranty and specialized equipment such as a soldering iron and SPI flashing tool are required.

I was disappointed to find multiple vulnerabilities in CompuLab’s Intense PC firmware. These serious vulnerabilities and CompuLab’s rather lackluster response inspired me to port coreboot to the Intense PC.

I am not an expert on the inner workings of the x86 platform and boot process, so I could not have successfully completed this port without the assistance of the excellent autoport tool.


Coreboot advantages

  • 🎉Open-source firmware🎉
  • Better memory (RAM) compatibility than the CompuLab firmware
  • Memtest86+ and iPXE can be included as a payload in flash
  • Verified boot supported via vboot

Limitations

  • VBIOS is required if you want any video output before the kernel framebuffer is initialized
  • VGA hand-off to Windows is still not working
  • ME firmware is still necessary for most users as me_cleaner breaks SATA
  • Currently no easy path to support UEFI
  • No FACE modules except for the included 4 port USB2.0 FACE module (FM-4USB) are supported (due to lack of additional FACE modules to test)

Please note that due to copyright concerns I cannot distribute binary firmware components such as the ME firmware or video BIOS. Additionally, for technical reasons I cannot provide a fully built, flashable coreboot image for your Intense PC. This is the reason for the “Extracting binary firmware components” section of the article.

If you experience issues building or using coreboot, please leave a comment or subscribe to the coreboot mailing list and ask your question there.

The coreboot project and I make no guarantee these instructions and the resulting firmware won’t turn your system into a fancy brick. The instructions produce a bootable firmware on my hardware (MintBox 2) at the time of writing, although this could change at some point in the future.

Please exercise caution and common sense when modifying system firmware and ensure you always have a backup of the original firmware on another device should something go wrong.

CVE-2017-9457: CompuLab Intense PC lacks firmware signature validation

Summary
CompuLab have not enabled signature checking of firmware updates for the Intense PC product line. This allows anyone in possession of the Phoenix UEFI update program to write a modified UEFI firmware to system flash. DOS/Windows versions of the Phoenix utility are easily obtained online, allowing a local or remote attacker to install a persistent firmware level rootkit to the computer, or to corrupt the system firmware, causing a denial of service.

Installation of a modified firmware can occur entirely in the background, without any user interaction, and once performed is virtually impossible to difficult to detect using operating system utilities. Physical access is not required.

Product description
The CompuLab Intense PC is fanless mini-PC. A model pre-installed with Linux Mint is also marketed under the name MintBox 2. The system firmware is the same for the Intense PC and MintBox 2. CompuLab also sell the Intense PC with an extended temperature range for industrial applications.

The product was introduced in mid-2013 and is still being sold through Amazon US, Amazon Canada, Amazon Germany, Amazon Spain, and directly from CompuLab.

Affected products

  • Intense PC (Intense PC Value, Intense PC Business, Intense PC Pro)
  • MintBox 2

Impact
Any software running with local administrator privileges has unrestricted access to read and write the system’s firmware.

An attacker can modify the contents of the system firmware to install a persistent rootkit/bootkit, or to corrupt the firmware causing the computer to cease functioning.

The attack only requires local administrator privileges, and can be executed either by using an existing OS-level exploit to gain local administrator, or via tricking the user into running an executable (e.g. via an attachment in a phishing email).

Proof of Concept
The proof of concept provided for CVE-2017-8083 can be leveraged for this vulnerability as well. The proof of concept uses the Phoenix UEFI Winflash utility to write a modified firmware to flash. Please refer to the article about CVE-2017-8083 for a detailed description of the proof of concept.

The latest CompuLab firmware for the Intense PC (20170521) modified with the upstream EDKII shell can be downloaded here.

Mitigation
At this time there is no means for the end user to enable Capsule Signature verification or to prevent the Phoenix update utility from updating the system firmware.

Therefore Intense PC owners should consider the following options:

  • Ensure your operating system is up to date with the latest security patches. Do not run software from untrusted sources.
  • Do not connect your Intense PC to any networks with internet access (i.e. air-gap the computer).
  • Discontinue your use of the Intense PC and consider replacing the computer with one from a different manufacturer who implements signature validation for firmware updates.

Should CompuLab decide to improve the security of the Intense PC firmware by enabling Capsule Signature validation, then the above recommendations would no longer apply. However, in my communication with CompuLab regarding this issue no indication was given that they have any plans to enable Capsule Signature verification in a future update. Therefore, it seems very unlikely to me CompuLab will issue an update which enables Capsule Signature verification.

Disclosure timeline:
6 June 2017: Issue reported to CompuLab
6 June 2017: CompuLab confirms that “Default settings of this source tree [Phoenix SecureCore Tiano Enhanced Intel Ivy Bridge CPU Panther Point M] has disabled Capsule Signature option.”
6 June 2017: Issue is reported to MITRE
6 June 2017: Vulnerability is assigned CVE-2017-9457
7 June 2017: CompuLab are informed that the vulnerability has been assigned CVE-2017-9457 and details of the vulnerability will be published after 45 days

CVE-2017-8083: Intense PC lacks BIOS Write Protection

Summary
CompuLab Intense PC and MintBox 2 fail to properly write protect flash regions, allowing an attacker with local administrator privileges to write arbitrary code to the platform firmware. This could allow a remote attacker to install a persistent firmware level rootkit to the computer, or to erase the system firmware, causing a denial of service.

Installation of a modified firmware can occur entirely in the background, without any user interaction, and once performed is virtually impossible to difficult to detect using operating system utilities. Physical access is not required.

Product description
The CompuLab Intense PC is fanless mini-PC. A model pre-installed with Linux Mint is also marketed under the name MintBox 2. The system firmware is the same for the Intense PC and MintBox 2. CompuLab also sell the Intense PC with an extended temperature range for industrial applications.

The product was introduced in mid-2013 and is still being sold through Amazon US, Amazon Canada, Amazon Germany, Amazon Spain, and directly from CompuLab.

Affected products

  • Intense PC (Intense PC Value, Intense PC Business, Intense PC Pro)
  • MintBox 2

At the time of discovery in March 2017, the latest firmware for CompuLab was dated 21 June 2016, and did not enable write protection on any flash regions.

Impact
Any software running with local administrator privileges has unrestricted access to read and write the system’s firmware.

An attacker can modify the contents of the system firmware to install a persistent rootkit/bootkit, or to corrupt the firmware causing the computer to cease functioning.

The attack only requires local administrator privileges, and can be executed either by using an existing OS-level exploit to gain local administrator, or via tricking the user into running an executable (e.g. via an attachment in a phishing email).

Proof of Concept
The firmware update from CompuLab was downloaded, decompressed, and loaded into UEFITool.

The default UEFI shell provided in Phoenix SecureCore was replaced with a newer version of the UEFI shell from EDK2:

The Phoenix SecureCore UEFI Shell was replaced with the EDK2 UEFI Shell.

The modified update was then written to the system firmware using the Phoenix UEFI Winflash utility:

Phoenix UEFI Winflash

It was later realized that the Phoenix UEFI Winflash utility includes a flag enabling a silent firmware update from the command line:

Phoenix UEFI Winflash supports silently updating the firmware from the command line

Using the /remote2 option removes all visual notifications that a firmware update is in progress. Additionally, when used with /console or /remote2 options, the Winflash utility does not reboot the platform when finished. The system continues to function normally, and there is no indication to the user that a firmware update has taken place at all.

Additional information
Output of the chipsec utility:

python chipsec_main.py -m common.bios_wp
################################################################
## ##
## CHIPSEC: Platform Hardware Security Assessment Framework ##
## ##
################################################################
[CHIPSEC] Version 1.3.0
[CHIPSEC] Arguments: -m common.bios_wp

WARNING: *******************************************************************
WARNING: Chipsec should only be used on test systems!
WARNING: It should not be installed/deployed on production end-user systems.
WARNING: See WARNING.txt
WARNING: *******************************************************************

[CHIPSEC] API mode: using CHIPSEC kernel module API
[CHIPSEC] OS : Windows 8.1 6.3.9600 AMD64
[CHIPSEC] Platform: Mobile 3rd Generation Core Processor (Ivy Bridge CPU / Panth
er Point PCH)
[CHIPSEC] VID: 8086
[CHIPSEC] DID: 0154

[+] loaded chipsec.modules.common.bios_wp
[*] running loaded modules ..

[*] running module: chipsec.modules.common.bios_wp
[x][ =======================================================================
[x][ Module: BIOS Region Write Protection
[x][ =======================================================================
[*] BC = 0x08 << BIOS Control (b:d.f 00:31.0 + 0xDC)
[00] BIOSWE = 0 << BIOS Write Enable
[01] BLE = 0 << BIOS Lock Enable
[02] SRC = 2 << SPI Read Configuration
[04] TSS = 0 << Top Swap Status
[05] SMM_BWP = 0 << SMM BIOS Write Protection
[-] BIOS region write protection is disabled!

[*] BIOS Region: Base = 0x00D00000, Limit = 0x00FFFFFF
SPI Protected Ranges
————————————————————
PRx (offset) | Value | Base | Limit | WP? | RP?
————————————————————
PR0 (74) | 00000000 | 00000000 | 00000000 | 0 | 0
PR1 (78) | 00000000 | 00000000 | 00000000 | 0 | 0
PR2 (7C) | 00000000 | 00000000 | 00000000 | 0 | 0
PR3 (80) | 00000000 | 00000000 | 00000000 | 0 | 0
PR4 (84) | 00000000 | 00000000 | 00000000 | 0 | 0

[!] None of the SPI protected ranges write-protect BIOS region

[!] BIOS should enable all available SMM based write protection mechanisms or co
nfigure SPI protected ranges to protect the entire BIOS region
[-] FAILED: BIOS is NOT protected completely

Output of the Intel Flash Programming Tool (FPT):

Intel’s fpt utility showing full write access to flash regions on the Intense PC

Through my discussion with CompuLab support, it has emerged that the issue is due to CompuLab not running CloseMnf prior to shipping. CloseMnf stands for “Close of Manufacturing” and hardens the system by setting write-protect flags for the various flash regions in the Master Access Section of the Descriptor Region.

Intel documentation regarding CloseMnf:

Disclosure timeline:
1 March 2017: Vulnerability is reported to CompuLab via their support email address
2 March 2017: CompuLab replies they will create a beta BIOS to address the vulnerability
6 March 2017: I request a timeline to fix the issue
7 March 2017: CompuLab replies they will create a beta BIOS for testing and they “will provide an official public release in the future”
8 March 2017: CompuLab replies with instructions to run closemnf via the Intel FPT tool
8 March 2017: I inform CompuLab I am waiting for the official BIOS update to resolve the issue
8 March 2017: CompuLab replies with copy of Intel FPT tool and requests “not to publish or disclose this information”
8 March 2017: CompuLab is informed that details of the vulnerability will be published on 4 June 2017
23 April 2017: Issue is reported to MITRE
24 April 2017: Vulnerability is assigned CVE-2017-8083
3 May 2017: CompuLab communicates that they will delay fixing this vulnerability until Intel provides an updated ME firmware to address CVE-2017-5689
4 May 2017: I inform CompuLab that details of this vulnerability will be published on 4 June 2017 as previously discussed
11 May 2017: CompuLab sends a proposed fix for testing, the update script fails due to invalid command syntax for flashrom
14 May 2017: I inform CompuLab of the invalid syntax and provide the correct usage, and confirm that the fix enables write-protection on the ME/BIOS/GbE regions of flash
15 May 2017: CompuLab replies with a revised update script
15 May 2017: I inform CompuLab that the syntax of the revised script is correct, however my unit has already been updated so I cannot re-test
4 June 2017: Details of the vulnerability are published.

CompuLab have provided an update to address the issue.

I can confirm that the Phoenix update utility still functions so it is still possible to update the BIOS even after the FDR has been locked.

Disabling Secure Boot on Intel Quark “secure SKU” silicon

Secure Boot is a bit like SELinux: people who use it really like it, and tell all their friends to use it. For everyone else, apart from those who don’t know about or even notice Secure Boot, it’s an annoyance that they almost immediately disable.

We’ve looked at the Intel DK200 from a hardware perspective before. Now it’s time to look at it from a software perspective. “Internet of Things Gateway” is pretty generic, so what can it actually do?

Following the instructions, I tried to register the system on Intel’s website so I could download the Wind River Intelligent Device Platform XT 2.0 SDK. I didn’t get very far:

No WindRiver SDK for you

Stormtrooper #1: This is not the product you’re looking for

Yeah… I guess this is what Mouser meant when they said the DK200 was End of Life.

Since this ships with the Linux Kernel, which is GPLv2 licensed, I believe Intel may be violating the GPL. Specifically:

Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange

But I am not a lawyer, and I am not really that interested in starting a legal battle over the source code for an ancient version of Wind River Linux I am not interested in using anyway.

So let’s go try to build Yocto. The Intel rep did say there was a Yocto BSP coming “soon” but “soon” in Intel time seems kind of variable.

After some hiccups (Yocto needs python2 and GCC <6) I had built a Yocto image and put it on an SD card. Does it boot?

...no

…no

So we can’t boot Yocto because this is a “secure SKU” which means Secure Boot is enabled. Is there some way we can disable Secure Boot? What about updating the BSP to a newer version with Secure Boot disabled?

Back to hardware
If I’ve learned anything from messing around with electronics, you want to make a backup before you start modifying things. This is doubly so if the data in question is related to the booting process. It sucks to end up with a brick, so make a backup!

Taking a backup of flash

Taking a backup of flash

The Intel Quark guide mentions using a Dediprog SF100 to flash EDKII. I don’t have a Dediprog, but I do have an SPI programmer. Unfortunately, none of the Intel documentation I could find mentions the Dediprog header on the DK200, so I had to go hunting.

I traced the pins from the Winbond flash to header J23. J23 is only 8 pins, so trial and error with a multimeter to find the pin mapping wasn’t terrible:

J23 pinout

J23 pinout

Here’s the pinout of J23 in text form:

J23 pin 25Q64 pin Pin description
1 8 VCC
2 4 GND
3 1 /CS
4 6 CLK
5 2 DO
6 5 DI
7 Not connected
8 Not connected

/WP and HOLD pins on the 25Q64FV are not routed to J23, but they aren’t required for flashing.

With the pinout known, I could attach the SPI programmer to the header instead of using the chip clip:

J23 to ch341a SPI programmer

J23 to ch341a SPI programmer

I took a dump of the Winbond 25Q64FV and then for good measure desoldered the chip and read it again to confirm the images were exactly the same. It was strange because the image from the chip clip wasn’t identical. But, the image from the desoldered chip was identical to the image taken from J23, so we’re done here. I wrote the image to a new 25Q64FV and soldered that back onto the board.

Firmware disassembly
Disassembling the firmware which shipped on my DK200, we see that a Secure Boot certificate was created by WindRiver.

I assume that had I been able to download the WindRiver SDK, I would have been able to build and sign Secure Boot with my own certificates. Given that industrial customers spend a lot of time and money worrying about security, I was surprised to see that the Secure Boot certificate in the firmware was created by WindRiver China.

I did try to load up the image in IDA, but not being a power user of IDA, I couldn’t figure out how to get it to analyze the SPI dump, and gave up to try and compile the firmware from source.

Building the BSP

Being Intel, there are hundreds of pages you can read about developing for EDK2 and other really fun things, probably. I didn’t read them.

A document which I did end up reading religiously was the Intel ® QuarkTM SoC X1000 Board Support Package (BSP) Build and Software User Guide [PDF] which describes how to build all the firmware components needed to bring up the X1000 SoC. I found out there is actually a newer version of this document (1.2.1 instead of 1.1) and there are some important differences between the documents I want to get to later.

By building the firmware, we’re hoping for one of two outcomes:

  1. A firmware with our own Secure Boot certificates, or
  2. A firmware which has Secure Boot disabled

Version 1.1 of the BSP Build and Software User Guide includes a section on pages 29 and 30 on how to bundle your own db, kek, and pk certificates:

Page 29 and 30 condensed

Unfortunately if you follow the instructions and try to use a layout.conf which specifies these files, you’ll get an error because there’s no address specified for this data in the image:

I do have a reference file from WindRiver with Secure Boot certificates, so if I was really interested in making Secure Boot work as intended, I could have reverse engineered the address to store the certificates.

The certificates section of layout.conf was removed from the 1.2.1 revision of the BSP Build and Software User Guide. I guess since it no longer works, Intel decided to remove it from the documentation.

So, we can’t install our own Secure Boot certificates in the firmware. What happens if we just leave out the certificates section entirely and build it?

Error 37: Quark signature file not found

Right, so even though there’s now no certificate in the firmware bundle, we still can’t boot.

Interestingly, if you don’t partition the uSD or USB stick correctly, you end up with this pretty screen:

I never saw that in the stock firmware.

Hacking GRUB
So it seems that we can’t include our own Secure Boot certificate in the firmware, due to the sample layout.conf file missing the certificates section, and not knowing the appropriate address to store the certificates.

What if we dig into Error 37: Quark signature file not found a bit more?

If you look in the grub source code included in the BSP, you can see a giant ~1000KB patch that Intel has made to the original upstream code to support the Quark platform.

If you grep for “Quark signature file not found” you’ll find it was added in stage2/common.c:
diff --git a/stage2/common.c b/stage2/common.c
index e96bec2..e122745 100644
--- a/stage2/common.c
+++ b/stage2/common.c
@@ -88,6 +88,8 @@ char *err_list[] =
[ERR_UNRECOGNIZED] = "Unrecognized command",
[ERR_WONT_FIT] = "Selected item cannot fit into memory",
[ERR_WRITE] = "Disk write error",
+ [ERR_QUARK_VERIFICATION] = "Quark signature verification failed",
+ [ERR_SGN_FILE_NOT_FOUND] = "Quark signature file not found",
};

If you grep for ERR_SGN_FILE_NOT_FOUND you’ll find it’s in the following files:
./work/efi/ia32/loader/linux.c:410: errnum = ERR_SGN_FILE_NOT_FOUND;
./work/efi/ia32/loader/linux.c:732: errnum = ERR_SGN_FILE_NOT_FOUND;
./work/efi/quark/boot_settings.c:190: errnum = ERR_SGN_FILE_NOT_FOUND;

Going back to Intel’s modifications to grub, we can see what they added:

It takes a bit of searching, but if you strip out all of the grub_quark_secure logic from linux.c and boot_settings.c, you end up with…

Ta-da! I can boot Yocto Linux

No more Secure Boot!

At the end of the day, the Quark X1000 is an x86: “secure SKU” is nothing but a fuse setting.

The comment should read:

Determine whether or not grub should enforce Secure Boot.

In our case, this is not a mandatory option 😉

Special offer for DK200 owners
As shown above, it is possible to modify the Intel sources to disable Secure Boot. If there are other people have a DK200 from Intel and are interested in running a firmware without Secure Boot, leave a comment with your contact details. Upon request, I can provide a firmware image* with generic Ethernet MAC addresses for you to flash. Note that this firmware is specific to the DK200 (Clanton Hill) hardware.

* No warranty, express or implied, provided for said firmware image. You flash at your own risk!

Reassembling a firmware from pieces

Working on firmware is always interesting. Modern x86 computers are incredibly complicated, due to the evolution of the architecture over the last 40 years, and it’s difficult to debug issues past “Well it doesn’t POST, better try something else.”

Unlike most ARM/MIPS systems, where you have a UART console or something to see output from u-boot, if you mess up the firmware on an x86, you’ll have a non-communicative brick on your hands.

Of course you can also have firmware issues on ARM/MIPS, if you manage to corrupt u-boot on SPI flash, but since u-boot is open and not proprietary it’s easy to rebuild it from source and flash it again to recover.

Not so in the PC world. UEFI is horrendously complex compared to u-boot, and Intel’s reference implementation known as TianoCore is usually “improved” by several middlemen before going into the final product.

Trammel Hudson’s excellent talk from 33c3 (slides) on Bootstrapping a slightly more secure laptop highlights the situation with UEFI quite well:

Farm to table firmware

In this case, we’ve got TianoCore from Intel, insyde adds some magic fairy dust, this goes to Compal (the ODM of this laptop), and finally Dell (who would probably also claim to add magic fairy dust). In the end there’s a lot of proprietary fairy dust floating around, and we probably wouldn’t be able to boot the laptop if we just build TianoCore from source.

TianoCore itself is open source and BSD licensed, which is why all the vendors use it. Intel manages porting TianoCore to their new platforms, and since it’s BSD licensed, it means that someone like insyde can take the working base from Intel and add their proprietary fairy dust without having to release the modified source code.

SPI flash
The start of flash contains a region called the Flash Descriptor (PDF; page 3) which is programmed at system manufacture and tells the system where different firmware components are present on flash. Think of it as a partition table for the system flash. Under normal circumstances, the flash descriptor prevents the user from reading and/or writing portions of ROM. If you try to use tools to read or write the ME regions of flash, you’ll get a error similar to this:

Error 26: The host CPU does not have read access to the target flash area. To en
able read access for this operation you must modify the descriptor settings to g
ive host access to this region.

And this makes sense. If the system allowed unrestricted write access it would be trivial for some malware to write itself into the system firmware, and then you’d have a persistent rootkit. In my opinion, blocking the ability to read portions of the firmware serves no purpose except to discourage reverse engineering attempts.

Thankfully, there is a method known as Flash Descriptor Security Override Strap which can be used to disable the flash descriptor protection.

The first step is to locate the ME_FWP pin in the circuit diagram:
Override Strap

Now that we’ve located the pin on the logical diagram, we need to find the HDA chip itself so we can see which pins we need to bridge to disable the flash descriptor lock on the ME region:
la-7731p_sda

In this case, it’s most convenient for us to short pins 5 and 9:
la-7731p_hda

You could also short pins 1 and 5, but this requires a very steady hand and small instrument. However pins 5 and 9 are connected to surface mount components located away from the chip. These components (a capacitor and a pad) are much easier to access with a wire:
la_7731p_pin5_pin9

Amazingly, this area can be accessed without completely disassembling the laptop. Just taking off the palm rest and keyboard, which is about 8 screws, is enough to access the pins. They’re right under the LVDS cable to the display. Thanks, Compal!

At this point, I have two options:

  1. Find or buy an SPI dump online and flash that
  2. Find a way to dump the firmware from the working laptop without soldering

As you read in the previous post, I only managed to find an E6320 SPI dump online, and ended up with a laptop that worked-ish. I tried for many hours, but I wasn’t able to find any free SPI dumps for the E6230 online. After seeing many forums promising to sell you the firmware for outrageous prices, I finally found one that wanted only 8 złoty ($2) to download the SPI dump for the E6230. Principles be damned, I’ve wasted enough time trying to get this to work. I paid up and downloaded the files for the 4MB and 8MB chips.

Did I really get a valid firmware for $2? Yes!

But it’s BIOS A12, the current version is A16, but none of the Dell update utilities work! I installed Windows to try the update tool from Dell, but the laptop just rebooted without updating the firmware. I tried from FreeDOS and again, the laptop would just reboot when it got to flashing. Hrmm…

So, at this point I’m going to cheat a little: I have a working E6230, but I decided when I started this that I would not touch it with a soldering iron or heat gun. If it ain’t broke, don’t fix it!

Can I get a full firmware dump from the working E6230?

Intel FPT

Intel FPT is a proprietary command-line utility created by Intel for flashing firmware files through the computer’s internal SPI flashing interface.

Alright, let’s go dump the firmware! First you need to remember to apply the Flash Descriptor Security Override Strap, or the CPU will block your read attempt to the ME region.

However, once we’ve done that, FPT will allow us to dump regions of flash to a file.

At first I tried to dump just the ME and BIOS, using the A12 firmware from above, I reflashed these regions using the FPT utility from FreeDOS. The verification of the BIOS region failed, and when I rebooted I had a brick again.

But, the FPT tool lets you dump all the regions at once! I dumped the entire flash, but now we’ve got a 12MB file and I don’t know where the split is between the 8MB and 4MB flash chips.

After more searching, I found the Intel ME System Tools, including a utility called Flash Image Tool, which allows you to import the firmware image file created by FPT.

Well, the flash component density matches what’s in the laptop, so I guess this is correct.

Flash Image Tool also lets you build a new firmware image:

Writing ROM image file “C:\Users\hmartin\Documents\MESYS\Flash Image Tool\v8.1.10.1286\Build\outimage.bin”.
Writing file “C:\Users\hmartin\Documents\MESYS\Flash Image Tool\v8.1.10.1286\Build\outimage(1).bin” (size = 8388608)
Writing file “C:\Users\hmartin\Documents\MESYS\Flash Image Tool\v8.1.10.1286\Build\outimage(2).bin” (size = 4194304)
Writing MAP file “C:\Users\hmartin\Documents\MESYS\Flash Image Tool\v8.1.10.1286\Build\outimage.map”.

Image size = 0xC00000 bytes

Interesting… outimage(1).bin is exactly 8192KB, and outimage(2).bin is exactly 4096KB. I wrote these two images to the respective chips and put them into the laptop. The moment of truth had arrived.

It boots!

When you’re trying to work through firmware issues, it’s really helpful to have the flash chip in a socket. Soldering and desoldering gets really old when you have to do it more than a couple of times.

Thoughts on the sale of firmware images

I’m against the sale of these firmware images. I realize that it takes a non-zero amount of time to get the image, but the whole experience of BIOS forums just leaves you feeling dirty. You have no way to verify before payment that the files they’re providing even work. Combine this with the fact that most websites want $10-$20 for the SPI dump, the experience leaves a bad taste in your mouth.

Is it worth it to pay? It depends entirely on your willingness to pay. Is this your only computer? Do you need it working now? How much of your own time are you willing to invest to learn about UEFI firmware? I’ve probably put 20 hours into this project, and I still don’t understand the internals of UEFI.

The wrong way to clear a BIOS password

I was browsing on eBay and ran into a listing for a Dell E6230. By now you might guess where this is leading:

Dell Latitude E6230 12.5″ i3-2350M 2.3GHz 4GB RAM Admin Pass Set AS IS READ!

s-l1600

The whole laptop, everything working perfectly, except… it has a firmware password set, and the seller doesn’t know the password. Well no bother, it’s cheap (83 EUR), I know how to flash SPI chips. Should be a piece of cake to clear this password! I also had an E6230 motherboard with a Core i5-3320M which I had found on eBay for 40 EUR, so my plan was to swap the boards, and then work on clearing the password.

These boards contain two flash chips: Winbond 64Mbit (8MB, U52) and Winbond 32Mbit (4MB, U53) located to the left of the SIM card slot.

f_0020914

The first step was to dump the SPI flash from U52 and U53 on both boards as a backup before flashing anything. This seemingly went well, but when I flashed the i5-3320M firmware dumps to the board with the i3-2350M, I had a brick.

It turns out that several of the pins to each chip are connected to the same pins in the PCH. Important pins like Data Out, Clock, and Data In are shared:

PCH SPI wiring

U52 and U53 share data out, data in, and clock lines to the PCH


I’m not sure exactly what happened, but my guess is that by leaving the flash soldered to the board, I somehow managed to get data from both chips. Alarms should have been going off in my head when my SPI reader dumped 8MB from a 4MB chip.

Not wanting to risk wrecking the i5-3320M board by desoldering the flash, I turned to the internet and discovered the terrible world of BIOS image dumps. It’s nearly impossible to find any website offering SPI dumps for free. Of course you can download the firmware update tool from the manufacturer, but those are meant to be run only on the intended hardware, and they only contain regions of the SPI flash like the UEFI firmware and Intel Management Engine.

I was able to find an SPI dump of the E6320, which is a 13.3″ laptop one generation previous to the E6230 (12.5″). As I had nothing to lose, I desoldered U52 and U53 and flashed the E6320 images to each chip. To my great surprise, the board passed POST, albeit with a warning about unsupported hardware. I was even able to enter the UEFI setup utility.

But now I am running a firmware intended for a completely different laptop. The E6320 has a QM67 (6 series) chipset, and the E6320 has a QM77 (7 series) chipset. The USB 3 ports on the side don’t work, the internal SATA port doesn’t work, and PXE booting doesn’t work. The only port that seems to work, at all, is a Mini PCIe USB port:

mini_pcie-usb30-1_zps098e038b

My next thought was to try the Dell E6230 BIOS update tool, which can run from DOS. I put FreeDOS on a USB stick in the above adapter and installed that in the WWAN Mini PCIe slot. Unfortunately for me, Dell has put checks into the update utility to check that it’s running on the correct hardware. This makes total sense, if a user downloads the wrong update for their laptop, they shouldn’t end up with a brick.

However, it didn’t suit my purpose. I wanted Dell’s BIOS update utility to ignore the fact that it was running on an “E6320” and instead flash the firmware for the E6230, the actual hardware.

Having been foiled by Dell’s checks, I decided to load up the utility in IDA Pro to see if I could bypass the check. A bit of string searching and I found the target, a jnz:
ida_jnz

Changing this check to a jz and I tried again. This time the utility didn’t complain about the machine being an E6320, but as soon as the flashing process started, the laptop shut off. So what happened? My best guess is that Management Engine shut down the platform.

Management Engine
The Intel ME has existed since the mid-2000’s, and is now deeply integrated into all of Intel’s modern x86 SKUs. The ME can provide additional functionality like a TPM (implemented in firmware), cryptographic acceleration, DRM, as well as other patented and super duper proprietary stuff. There’s a fairly comprehensive feature list available on Wikipedia.

Since Intel doesn’t actually release documents on the ME, it’s hard to come by actual information on the inner workings. It’s also why some libre people are concerned about buying newer laptops: the ME is integrated into the PCH, cannot be disabled, runs an OS with direct memory access to system RAM and has never been audited.

Trammell Hudson is currently experimenting with coreboot on the Lenovo X230, and it seems like there’s a non-zero chance that he’ll succeed in disabling the ME.

Anyway, I was on vacation earlier this year and had lots of time to kill in airports/planes/trains so I read a book about the Intel ME called “Platform Embedded Security Technology Revealed” You can download the ebook for free as a PDF or EPUB from the publisher.

Using knowledge from the above book, we can conclude that the security number of the Series 8 ME firmware must be equal to the security number of Series 7 firmware, or the ME would not allow the platform to power on.

Unfortunately, the updater managed to overwrite something important in flash before the ME cut power, because now I’m stuck with a brick again.

Stay tuned for part 2!

Allwinner H2+/H3 Ethernet with Linux 4.9-rc8

The Orange Pi PC is not a new single board computer. It’s been released for over a year now, but has mostly been stuck on a heavily patched 3.4 release kernel.

There have been ongoing efforts since the release to have support for the Allwinner H3 in the mainline kernel. In the past weeks there have been new patches released which enable support for the Ethernet MAC on the H3 (and H2+).

Unfortunately this support is not in mainline yet, and won’t make it in the upcoming 4.9 release. However, that doesn’t stop you from taking the patches and applying them against 4.9 yourself.

I wrote a script to compile the kernel from source, applying the necessary patches to the kernel and using a minimal .config file which compiles the sun8i_emac support as a module. You can download the build script from GitHub.

It does try to be somewhat smart: verifying the integrity of the downloaded files, and will bail out if there are errors in patching the source code. But, it doesn’t do toolchain dependency checking because that’s just too complicated. Since the emac support will end up in mainline soon, I doubt it’s worth the time to improve the build script. However if anyone is interested in improving it, the script is released as GPLv2.

The result? If you are patient enough to wait for the kernel to compile, you get a uImage and modules for Linux 4.9-rc8 with Ethernet support:

Allwinner H3 emac performance:

[email protected]:~$ iperf -n 1024M -c 192.168.1.150
————————————————————
Client connecting to 192.168.1.150, TCP port 5001
TCP window size: 43.8 KByte (default)
————————————————————
[ 3] local 192.168.1.206 port 42572 connected with 192.168.1.150 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-100.6 sec 1.00 GBytes 85.4 Mbits/sec

Orange Pi Zero (Allwinner H2+):

I’ve also just received my Orange Pi Zero and confirmed that the same kernel works on the Orange Pi Zero, so you can run Linux 4.9 on the Orange Pi PC (Allwinner H3) or Orange Pi Zero (Allwinner H2+) with Ethernet support.

Allwinner H2+ emac performance:

[email protected]:~$ iperf -n 1024M -c 192.168.1.150
————————————————————
Client connecting to 192.168.1.150, TCP port 5001
TCP window size: 43.8 KByte (default)
————————————————————
[ 3] local 172.16.4.206 port 54762 connected with 192.168.1.150 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-95.9 sec 1.00 GBytes 89.5 Mbits/sec

Download:
If you want to skip compiling the kernel yourself, I’m providing the kernel uImage and modules here.

I am using u-boot v2016.11 from denx.

Note: HDMI isn’t working on my Orange Pi PC, and since I run it headless I’m not interested in investigating why. If you’re using your Orange Pi PC with HDMI output, you may need to modify the kernel .config file to fix HDMI.

Intel DK200 IoT Gateway

Earlier this year I was at a conference and heard from other attendees that the Intel booth was giving away IoT gateways. Never one to turn down free conference swag, I hurried over to the Intel booth and was told to pick up a gateway out of a pallet of boxes just delivered (and rapidly disappearing).

The Intel IoT gateway series is codenamed Moon Island, but the design targeting the transportation market is codenamed Clanton Hill. Clanton Hill known to us mortals as the “Intel DK200 Series Gateway Solution for the Internet of Things (IoT)” quite the mouthful.

Let’s get down to it.

Availability
Unless you happen to be at a conference where Intel reps are handing these out like candy, I don’t think it’s practical to try and buy one yourself:

dk200_mouser

Some interesting details to note about this product:

  1. Although released in 2014, the DK200 still costs more than the new MacBook (3,712.50 EUR versus 3,199 EUR)
  2. It’s End Of Life

When a low volume product goes EOL and you still have stock, I guess giving it away at conferences is the next logical step.

Hardware Specifications
The DK200 (datasheet) is targeted toward the transportation industry, and it really shows in the appearance of the device:

Only available in 'Cosmic Black'

Only available in ‘Cosmic Black’

I don’t work in the transportation industry, and have never seen connectors that look like this before. They’re very well made, and I suspect probably do a good job of keeping dust, dirt, and debris out of the ports. Since I don’t wish to make a mess by throwing dirt and debris at it, I’m going to have to trust the engineers who designed it.

The build quality is quite good, as one might expect from a device selling for 3,700 EUR. Nearly every screw is secured with loctite to prevent vibration from loosening them:

DK200 screw with loctite

No pentalobe nonsense here

However, I was surprised to find that despite all the physical hardening applied to the enclosure, I couldn’t find any information on an IP rating. In fact the top and bottom of the case don’t appear to offer any additional dust or water seal. There’s clearly been a lot of thought put into the design of this enclosure to withstand vibration and dirt, so it’s strange that there doesn’t seem to be water protection of any kind.

Processor
The Intel Quark series SoC was introduced in late 2013. The X1020D in the DK200 is a single core SoC based around a 80486 core running at 400MHz, with modern I/O and memory.

dk200_x1020d

In 2014 a leaked product roadmap suggested a successor to the X1000 series named “Dublin Bay” to be released in 2015. Then news emerged that “Dublin Bay” had been cancelled, to be replaced by “Liffy Island” and “Seal Beach” which would be released in 2015. As of late 2016 Intel has not released a direct successor to the X1000 series, and there is no new news of “Liffy Island” or “Seal Beach” being cancelled (or released). So it’s anyone’s guess whether Intel is still even interested in the IoT gateway market.

Storage
The DK200 doesn’t include any of the typical storage buses like SATA, NVMe, or NAND (EMMC). This is not overly surprising given the embedded nature of the hardware (requiring lower power) and the simplicity of the Quark processor.

The only storage option is a micro SDHC card, and the DK200 includes an 8GB class 4 micro SD card:
dk200_sdhc

Given that it’s a class 4 card, the performance is quite poor. Use of an SD card for storage isn’t a bad decision per se, but the DK200 uses ext3 for the root partition. Ext3 is not a flash aware filesystem. SD cards have only basic wear leveling, and ext3 has no wear leveling. So it hardly seems like the appropriate combination of storage and filesystem for a headless embedded device with an expected lifetime of 5-10 years.

Input and output

  • Dual 100Mbit Ethernet controllers
  • 3 x USB 2.0 host and 1x device
  • Audio in/out
  • CAN bus
  • RS-232
  • GPIO
  • 1x half-height mini PCI-e slot (populated with Intel 7260)
  • 1x full-height mini PCI-e slot (unpopulated; for 3G modem/GPS)

The Intel documentation also mentions ZigBee, however this is an external device, presumably attached via the USB bus.

Power consumption
Development platforms aren’t known for being highly optimised devices. They often include extra I/O which would not necesssarily be included in the final product, and as such do not have the same energy efficiency as a finished product.

This being said, I was quite surprised that a device intended for 24/7 operation in an embedded environment, and especially serving the “Internet of Things” market, could be so energy inefficient. Issuing a poweroff command in Linux results in the platform going into an S5 (shutdown) state. I was surprised to discover that the energy consumption in the S5 state is 2W. This seems quite high for a device which includes an ignition input for automatic power-on and shutdown.

When booting, the device peaks at 7.9W consumption, while the idle power consumption is 7.5W. This is almost certainly due to the added peripherals as the TDP of the Quark processor is only 2W.

It’s difficult to see how Intel expects the Quark platform to compete with ARM. My PandaBoard ES, an ARM-based development board from 2011, peaks at 4W, idles at 2W, and draws nothing when off. Now some might argue that comparing an ARM board from 2011 with an Intel IoT gateway from 2014 isn’t valid, but they do have a lot of similar features. Now, I will grant that the PandaBoard is not in a rugged enclosure with fancy connectors, but since it cost 95% less than the DK200 does, there’s some room in the budget for an enclosure and funky connectors. And, since Texas Instruments has stopped making OMAP chips, the PandaBoard gets about the same amount of vendor support as the DK200!

Software

I will be exploring the software of the DK200 in a follow up post. Stay tuned!

The real meaning of 10^9

I recently bought a 128GB USB mass storage device from Amazon here in Germany. The price I paid for this unbelievable amount of storage in my pocket? Just 22 Euros.

As you maybe know from a previous article, I’m never exactly pleased when I buy a storage device and find out that the manufacturer is actually selling less capacity than they’re advertising. Rather than turn this into another rant, I’d like to start a discussion with storage manufacturers about what 10^9 means.

First, some definitions:

Gigabyte: 1,000,000,000 bytes is the actual number of bytes in a “gigabyte” which has historically had the acronym “GB” shown.

And when you think about it, it makes sense. Recall your SI units, where you have kilo (1,000), mega (1,000,000), giga (1,000,000,000), tera (1,000,000,000,000), peta (1,000,000,000,000,000)… you get the point.

Gibibyte: 1,073,741,824 bytes. This is the number of bytes in “GiB” which is what computers typically operate in, because it’s a power of 2 (2^30).

So, why the confusion? Well, when you buy a computer with 8 “GB” of RAM, you’re actually buying a computer with 8 GiB of RAM. But, for historical reasons, it’s much more common to see electronics advertised with “GB” instead of “GiB” (although in recent years things seem to be changing, at least on the software side).

Some smart executive at a storage company long ago figured out that if they were advertising products as having 1 GB of capacity, that was actually 10^9 bytes, not 2^30 bytes, and they could increase profits if they started selling devices which were only 1,000,000,000 bytes instead of 1,073,741,824 bytes. That’s like 7% less storage!

https://commons.wikimedia.org/wiki/File:High_five!!.jpg

The celebration probably looked something like this

And in fact, storage manufacturers have been quite clear about this for a long time. They state quite clearly, in small print, on the back of the box or at the bottom of their product webpage, that the actual size of a Gigabyte is really 1,000,000,000 bytes. So, when you plug that shiny new storage device into your computer, and see 119.2GiB, well that’s just you failing to do the math to account for the difference between 10^9 and 2^30.

In fact, many manufacturers have support pages dedicated to outraged people who buy a device and haven’t read the fine print. Here is the SanDisk website explaining device capacity:
http://kb.sandisk.com/app/answers/detail/a_id/46/kw/capacity

I am not here trying to argue that electronics manufacturers should advertise storage devices advertised in GiB. They’re correctly advertising the capacity of the devices in Gigabytes, the SI unit. It would probably help SanDisk even more if on their website they used the appropriate definition for 2^30, which is Gibibyte. But as I said, consumers have gotten used to reading “GB” so when they see “GiB” they don’t understand what the difference is.

I know, I can hear you thinking right now “Okay, so what? Get to the point already, you’ve been droning on for over 400 words. Hey, are you paid by the word?”

First, I make no money from this website. And second, I said this was a discussion, not a rant. In a discussion, you must provide context and frame the problem. Otherwise it’s just blatent complaining.

Remember that SanDisk page (pictured above) where they helpfully explained the difference between a Gigabyte and Gibibyte for us? Well, I didn’t show you the whole page. Here is what is written directly after the screenshot pictured above explaining a gibibyte:
sandisk_capacity_explanation_revealed

So, basically what SanDisk is telling us here is that the actual size of the device isn’t even the advertised capacity multiplied by 10^9 bytes, it’s actually less. This is basically the legal equivalent of them saying “trust us, it really has 128,000,000,000 bytes inside, but you can’t use them all.”

Somehow, our governments have decided that this kind of advertising is legal.

I went back and looked at the Amazon.de page where I bought the product. It might shock you, dear reader, but there was no fine bullet point in the specifications saying “actual user storage less”:
amazon_de_128GB_cruzer_blade

This is understandable. If people saw “actual user storage less” mentioned in the product advertisement, they would probably be suspicious of the amount of storage they were actually buying, and sales would suffer.

I thought I would go look at other retailers to see if “actual user storage less” was mentioned anywhere on their websites. Here is the same product listed on Amazon.com:
amazon_com_128GB_cruzer_blade

What about NewEgg.com?
newegg_com_128GB_cruzer_blade

Okay, so the companies selling these devices aren’t overly eager to include this fine print, which SanDisk actually includes on their website. In small text, at the bottom of the page:
sandisk_com_128GB_cruzer_blade

I emailed SanDisk about this to ask why the “actual user storage less” wasn’t mentioned on any retailers website, and they responded:
sandisk_reply_1

While I disagree with their reply, I understand that retailers have some freedoms in how they advertise a product. However, with this in mind, I am sure that a small army of SanDisk lawyers would co-sign a cease and desist letter if I started advertising their products in any way which they determined was harming their brand value. But then something curious happened…

sandisk_reply_2

SanDisk claims that they don’t have any control over how retailers advertise their product, but then they state that these companies are “authorised distributors and resellers.”

Given the incredibly high percentage of counterfeit products being sold these days under the label of a well known brand, it’s clear that manufacturers need a trustworthy outlet to sell their goods, or consumers might begin to doubt the quality of their brand. That’s the economic impact of “electronics priacy” [PDF].

Managing your supply chain and maintaining your brand image costs companies millions, if not billions, of dollars every year. It’s serious business. People go to jail for importing and selling counterfeit products.

So, when a company claims that they have no control over how their product is advertised, I find that a bit difficult to believe. Legally they may not have an obligation to require retailers mention “actual user storage less” but morally and ethically they should ensure that their retailers do not advertise their product in a misleading way.

SanDisk is selling a “128GB” USB stick, which has a raw capacity of ~125GB (116GiB):
sandisk_cruzer_128GB_fdisk

They also mention in small text, on the back of the package, that “actual user storage less.” Too bad they didn’t state this anywhere on the actual retail page.

At this point, anyone who is sane will do the math and ask the question “You received 2.2% less capacity than was advertised. Why have you wasted your time writing about this?” and that’s because I’m scared of the precedent this is setting.

It’s true, it’s seemingly pointless to sit here and discuss the missing 2.2%. But 10 years ago, you would spend a lot of money to buy a USB stick which was 2GB. Even now, most people pay phone companies tens of dollars per month to transfer 3GB over 3G or LTE, or $90 per month if you’re unlucky enough to own a smartphone in Canada:
canadian_cell_plans

So even though as a percentage, it’s relatively small, it a not-insignificant amount of capacity that’s missing. If this was a 1TB device, you would be missing 23GB, and this is even before we get into Giga versus Gibi and formatting…

When I buy a device where the primary function is storage, I expect to be buying a device which can contain $CAPACITY * 10^9 bytes of data. This is important for things like data recovery. If I need to make an exact duplicate of the data on a storage device which is 128GB, I don’t want to have to worry about buying a device from a specific manufacturer, model, or serial number range to be able to store the data. This isn’t swapping the PCB on a broken hard drive, it’s just buying a simple storage device!

I’m scared of what message we’re sending to manufacturers when we allow them to sell us products with vague statements like “actual user storage less” and what it means for the future of the industry.

If I buy a smartphone which has 16GB of space, I expect to be able to use less, because the primary function of a smartphone is to be a pocket computer. I understand that capacity is required for the operating system, and that the actual capacity available to me will be less than the advertised amount.

But to buy a device whose sole purpose is to store data, and have that device provide less capacity than advertised. That’s scary, not for the 2.2% that I can’t use today, but because tomorrow it might be 5%, and in 10 years 20%.

Why is the capacity less? Perhaps they’re using NAND which doesn’t have space over provisioned for error correction and wear levelling, allowing them to fit a few more on a wafer. Or maybe the chips have bad regions which they’ve mapped around, meaning you get a slightly smaller capacity. I doubt we’ll ever get an explanation apart from “actual user storage less.”

SanDisk isn’t the only manufacturer doing this, but they’re certainly the worst offender that I’ve found. Recently I purchased some 16GB micro SDHC cards from Transcend, and they’re 0.5% under capacity as well. Luckily I have found that Samsung’s EVO line appears to at least provide the advertised capacity in 10^9 bytes. But, how much longer until everyone clues in to the get out of jail free card that is “actual user storage less” and starts selling devices under the 10^9 capacity?

The moral of the story here is: complaining on the internet is useless. Vote with your wallet. Return anything which is not actually 10^9 bytes of capacity, and rate it accordingly to warn other users. I’m not going to support a brand which, in my opinion, allows retailers to advertise their products in misleading ways. Besides, the performance sucked (4.5MB/s sequential write). You get what you pay for.

Linux 4.5 on a Bay Trail tablet

This post is a short update to my original article on booting Arch Linux on a Bay Trail tablet.

I originally wrote this for 4.4.5, but I wasn’t fast enough, and 4.5 was released before the post was completed, so might as well continue with a 4.5 kernel.

To simplify the build process I took the PKGBUILD for linux-mainline in AUR and modified it to build a mainline kernel with patches for SDIO WiFi on BayTrail.

If you’d like to build the kernel yourself (and you happen to run Arch Linux) you can download the PKGBUILD.

The firmware for the rtl8723bs card is in its own package, in keeping with the Arch Linux best practices for separating firmware from the kernel package. Download the firmware PKGBUILD.

Or, if you’d rather just have a newer kernel on your tablet which is already running Arch Linux, you can download the pre-built kernel package, and the firmware package.

I will be submitting both of these packages to AUR shortly.

Turns out you can actually get GRUB working with a menu if you build a standalone version of grub. However, the issue is that even though the grub menu works, there’s some issue with modesetting and you’ll never see any console after grub hands off to the kernel. You can download the standalone version of grub if you want to try, I wasn’t able to get any usable installer environment out of it. You can download standalone grub for ia32 (i686), you will also need grub.cfg.

$ tar -Jxf bootia32.tar.xz
$ cp bootia32.efi /mnt/archiso/EFI/boot/bootia32.efi
$ cp grub.cfg /mnt/archiso/EFI/boot/grub.cfg

Since grub video handoff isn’t working well, the only way I was able to successfully boot was to drop to command line by pressing c at the menu, and typing the following:

set root=hd0,msdos1
linux /arch/boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201603 video=VGA-1:[email protected]
initrd /arch/boot/x86_64/archiso.img
boot

There is a small issue with kernel oops, which has been present since at least 4.4.5:

[  164.281827] NMI watchdog: Watchdog detected hard LOCKUP on cpu 2
[  164.281913] Modules linked in:
[  164.281962]  intel_rapl intel_soc_dts_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw iTCO_wdt snd_soc_sst_bytcr_rt5640 iTCO_vendor_support hid_multitouch gf128mul glue_helper dcdbas ablk_helper cryptd pcspkr hci_uart snd_intel_sst_acpi mei_txe joydev input_leds snd_intel_sst_core btbcm snd_soc_rt5640 evdev snd_soc_sst_mfld_platform mousedev btintel mei lpc_ich mac_hid snd_soc_rl6231 thermal snd_soc_sst_match dw_dmac dw_dmac_core tpm_crb snd_soc_core bluetooth processor_thermal_device int3400_thermal int3403_thermal acpi_thermal_rel int3402_thermal i2c_hid int340x_thermal_zone snd_compress intel_soc_dts_iosf tpm_tis rfkill_gpio snd_pcm_dmaengine battery ac97_bus ac spi_pxa2xx_platform crc16 tpm snd_pcm i2c_designware_platform
[  164.283185]  acpi_pad i2c_designware_core 8250_dw snd_timer snd processor soundcore sch_fq_codel nfs lockd grace sunrpc fscache ip_tables x_tables overlay squashfs loop nls_iso8859_1 nls_cp437 vfat fat sd_mod uas usb_storage scsi_mod hid_generic usbhid hid i915 mmc_block button i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_intel xhci_pci drm xhci_hcd intel_gtt wmi serio video sdhci_acpi sdhci led_class r8723bs(O) cfg80211 rfkill mmc_core
[  164.283978] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G           O    4.5.0-byt #1
[  164.284073] Hardware name: Dell Inc. Venue 8 Pro 3845/XXXXXX, BIOS A02 12/29/2014
[  164.284169]  0000000000000086 9ad5a4512f59852f ffff880039d05b50 ffffffff812d25d1
[  164.284284]  0000000000000000 0000000000000000 ffff880039d05b68 ffffffff81116550
[  164.284399]  ffff880038ca8000 ffff880039d05ba0 ffffffff81156b4c 0000000000000001
[  164.284513] Call Trace:
[  164.284552]    [] dump_stack+0x63/0x82
[  164.284645]  [] watchdog_overflow_callback+0xe0/0xf0
[  164.284733]  [] __perf_event_overflow+0x8c/0x1d0
[  164.284815]  [] perf_event_overflow+0x14/0x20
[  164.284894]  [] intel_pmu_handle_irq+0x1e1/0x460
[  164.284980]  [] perf_event_nmi_handler+0x28/0x50
[  164.285062]  [] nmi_handle+0x5e/0x130
[  164.285133]  [] default_do_nmi+0x48/0x120
[  164.285207]  [] do_nmi+0xe2/0x130
[  164.285274]  [] end_repeat_nmi+0x1a/0x1e
[  164.285349]  [] ? poll_idle+0x39/0x80
[  164.285420]  [] ? poll_idle+0x39/0x80
[  164.285490]  [] ? poll_idle+0x39/0x80
[  164.285558]  <>  [] cpuidle_enter_state+0xf3/0x2f0
[  164.285655]  [] cpuidle_enter+0x17/0x20
[  164.285728]  [] call_cpuidle+0x2a/0x40
[  164.285800]  [] cpu_startup_entry+0x2c5/0x3a0
[  164.285878]  [] start_secondary+0x165/0x1a0
[  164.285964] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 4.144 msecs
[  164.286069] perf interrupt took too long (32852 > 2495), lowering kernel.perf_event_max_sample_rate to 50100
[  172.707012] perf interrupt took too long (32619 > 4960), lowering kernel.perf_event_max_sample_rate to 25200

You’ll see a lot of these, however WiFi still continues to work, and the tablet didn’t kernel panic for me in the installer environment.

Hopefully someone finds this useful. I’ll have a write up on installing and using Arch Linux on the tablet in the coming weeks.