Tag Archives: gigabyte

Gigabyte MJ11-EC1 PCIe Bifurcation

The Gigabyte MJ11-EC1 motherboard is an ITX motherboard with an AMD EPYC 3151 (4C8T) onboard. These motherboards were being liquidated from the Gigabyte G431-MM0 GPU server in 2023, and could be purchased for around 60 Euros in the EU. The bare-bones G431-MM0 can still be purchased for around 170 Euros.

The MJ11-EC1 is very similar to the Gigabyte MJ11-EC0 with the main difference being the MJ11-EC0 has a PCIe x16 slot while the MJ11-EC1 has a SlimSAS (SFF-8654 8i) connector for use with the GPU riser in the G431-MM0.

You can purchase the SlimSAS cable (~18 Euros) and a PCIe riser (~15 Euros) from several AliExpress sellers. The added cost of the cable and PCIe riser does reduce the value proposition of the motherboard somewhat. Additionally, user testing showed that PCIe bifurcation was non-functional on the SlimSAS port, meaning only a single PCIe device could be recognized on the SlimSAS 8i port unless a PCIe switch was used.

However, I can demonstrate that bifurcation does work on the MJ11-EC1, and in fact it is possible to access all PCIe x16 lanes if you add the unpopulated U2_1 SFF-8654 connector. All the passive components for the SFF-8654 connector are already present on the motherboard, so only the physical connector needs to be added to unlock an additional 8 lanes of PCIe.

Two MJ11-EC1 motherboards, one with U2_1 unpopulated and one with a SlimSAS 8i connector soldered

However, PCIe Bifurcation does not work under every condition. The following scenarios do not work:

Cable Adapter Bifurcation working
SlimSAS 8i to dual 4i SlimSAS 4i to PCIe x4 No
SlimSAS 8i to 8i SlimSAS 8i to Dual NVMe No
Dual SlimSAS 8i to 8i PCIe x16 (JHHP1B) No

The first attempt was with a SlimSAS 8i to dual 4i cable. Unfortunately, bifurcation did not work with this cable, only one device was visible.

Only one device is recognized

The second attempt was with a SlimSAS 8i to dual NVMe adapter. Again, only the first NVMe device was visible, so I do not recommend purchasing this for use with the MJ11-EC1.

Only one device is recognized

I then tried this dual SlimSAS 8i to PCIe x16 adapter, which did not work at all. In my subsequent discussion with the vendor in the AliExpress dispute, it appears this adapter is only compatible with their PCIe x16 to SlimSAS riser. So despite using the SFF-8654 connector, it is not standards compliant with SlimSAS 8i and cannot be used with the MJ11-EC1. Do not purchase this.

This adapter does not work at all. Avoid purchasing the “2 Port SlimSAS 8i x2 to PCIe 4.0 x16 Slot Adapter Card SFF8654 Riser Card GEN4”


The following combinations are fully functional:

Cable Adapter BIOS configuration PCIe devices Bifurcation working
SlimSAS 8i to 8i PCIe x8 x8x8 2 Yes
SlimSAS 8i to 8i PCIe x8 with 4xNVMe riser, 2xNVMe x8x4x4 3 Yes
SlimSAS 8i to 8i PCIe x8 with 4xNVMe riser, 2xNVMe x4x4x4x4 4 Yes

The following SlimSAS 8i to PCIe x8 adapters were used during testing and worked as expected. The adapters were purchased with my own funds and I have no relationship to the brands or sellers.

CEACENT CNS41CX16W

The CEACENT CNS41CX16W places a decoupling capacitor (C21) in the path of the power connector. Cover this in glue/epoxy or it will get knocked off the board.

“SFF-8654 8i to PCIe 4.0 x16 External Graphics Card Adapter SFF-8654 8i Adapter Card” N-P548-A

The designer of this adapter does not seem to have considered the lack of clearance between the SFF-8654 and SATA power connector. I would call it “challenging” to plug in the SFF-8654 connector.

Most SFF-8654 to PCIe x8 adapters from China seem to have fundamentally flawed physical layouts, which is unfortunate given they are otherwise inexpensive and effective.

This NVMe adapter is great, my only wish is that they made a 1U compatible 2 NVMe version as inexpensive as the 4 NVMe model.

There are dual SlimSAS 8i to PCIe x16 adapters available, however they are cost prohibitive. Given the PCIe bifurcation options available in BIOS and the fact that there are 16 accessible PCIe lanes, I suspect a standards-compliant adapter (e.g. Ceacent CNS52CX16R) would work to expose all 16 lanes.


MJ11-EC1 with two PCIe x8 adapters; HP 544FLR-QSFP installed and BIOS configured for x8x8 bifurcation:

05:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
06:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
(...)
# lspci -s 05:00.0 -vvv
05:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
        Subsystem: Hewlett-Packard Company InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter
(...)
                LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
# lspci -s 06:00.0 -vvv
06:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
        Subsystem: Hewlett-Packard Company InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter
(...)
                LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

MJ11-EC1 with two PCIe x8 adapters, each with 4xNVMe adapters, NVMe 1 and 2 sockets populated; BIOS configured for x4x4x4x4 bifurcation:

05:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980 (DRAM-less)
06:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller BG4 (DRAM-less)
07:00.0 Non-Volatile memory controller: SK hynix 960GB TLC PCIe Gen3 x4 NVMe M.2 22110
08:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01)
(...)
# lspci -s 05:00.0 -vvv
05:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980 (DRAM-less) (prog-if 02 [NVM Express])
        Subsystem: Samsung Electronics Co Ltd Device a801
(...)
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x4
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
# lspci -s 06:00.0 -vvv
06:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller BG4 (DRAM-less) (prog-if 02 [NVM Express])
        Subsystem: KIOXIA Corporation NVMe SSD Controller BG4 (DRAM-less)
(...)
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <32us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x4
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
# lspci -s 07:00.0 -vvv
07:00.0 Non-Volatile memory controller: SK hynix 960GB TLC PCIe Gen3 x4 NVMe M.2 22110 (prog-if 02 [NVM Express])
        Subsystem: SK hynix Device 0000
(...)
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x4
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
# lspci -s 08:00.0 -vvv
08:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD
(...)
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (downgraded), Width x4
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-


This was a fun modification but the economic case is dubious at best. With the SFF-8654 8i cables being roughly 18 Euros each, and SlimSAS 8i to PCIe x8 adapters ranging in price from 18-22 Euros (+shipping) the additional cost to fully utilise 16 PCIe lanes easily exceeds the entire cost of the motherboard.

The SFF-8654 connector (Amphenol U10-B074-200T) proved very hard to source from Western Distributors as they are discontinued, and are not available at a reasonable price on AliExpress. I ended up purchasing them on Taobao via a Chinese based forwarding service. The cost for QTY 20 was 523¥ on Taobao, plus forwarding agent fees, shipping to Europe, and VAT.

Soldering the connector is also a nightmare. The ground plane on the MJ11-EC1 is very effective at dissipating heat. I used a pre-heater, hot air station (set to 200C with high flow to avoid melting the plastic), leaded solder, sticky flux, and still required some touch up work with a very fine tip on the soldering iron to fix bad connections.

It should be noted that bifurcation of the populated U2_2 SFF-8654 port works. Anyone owning the MJ11-EC1 wishing to do that just needs to flash the MJ11-EC0 BIOS via the BMC to expose PCIe bifurcation settings in BIOS, and they should be able to install two PCIe x4 devices (subject to the limitations mentioned above regarding cables/risers).

The real meaning of 10^9

I recently bought a 128GB USB mass storage device from Amazon here in Germany. The price I paid for this unbelievable amount of storage in my pocket? Just 22 Euros.

As you maybe know from a previous article, I’m never exactly pleased when I buy a storage device and find out that the manufacturer is actually selling less capacity than they’re advertising. Rather than turn this into another rant, I’d like to start a discussion with storage manufacturers about what 10^9 means.

First, some definitions:

Gigabyte: 1,000,000,000 bytes is the actual number of bytes in a “gigabyte” which has historically had the acronym “GB” shown.

And when you think about it, it makes sense. Recall your SI units, where you have kilo (1,000), mega (1,000,000), giga (1,000,000,000), tera (1,000,000,000,000), peta (1,000,000,000,000,000)… you get the point.

Gibibyte: 1,073,741,824 bytes. This is the number of bytes in “GiB” which is what computers typically operate in, because it’s a power of 2 (2^30).

So, why the confusion? Well, when you buy a computer with 8 “GB” of RAM, you’re actually buying a computer with 8 GiB of RAM. But, for historical reasons, it’s much more common to see electronics advertised with “GB” instead of “GiB” (although in recent years things seem to be changing, at least on the software side).

Some smart executive at a storage company long ago figured out that if they were advertising products as having 1 GB of capacity, that was actually 10^9 bytes, not 2^30 bytes, and they could increase profits if they started selling devices which were only 1,000,000,000 bytes instead of 1,073,741,824 bytes. That’s like 7% less storage!

https://commons.wikimedia.org/wiki/File:High_five!!.jpg

The celebration probably looked something like this

And in fact, storage manufacturers have been quite clear about this for a long time. They state quite clearly, in small print, on the back of the box or at the bottom of their product webpage, that the actual size of a Gigabyte is really 1,000,000,000 bytes. So, when you plug that shiny new storage device into your computer, and see 119.2GiB, well that’s just you failing to do the math to account for the difference between 10^9 and 2^30.

In fact, many manufacturers have support pages dedicated to outraged people who buy a device and haven’t read the fine print. Here is the SanDisk website explaining device capacity:
http://kb.sandisk.com/app/answers/detail/a_id/46/kw/capacity

I am not here trying to argue that electronics manufacturers should advertise storage devices advertised in GiB. They’re correctly advertising the capacity of the devices in Gigabytes, the SI unit. It would probably help SanDisk even more if on their website they used the appropriate definition for 2^30, which is Gibibyte. But as I said, consumers have gotten used to reading “GB” so when they see “GiB” they don’t understand what the difference is.

I know, I can hear you thinking right now “Okay, so what? Get to the point already, you’ve been droning on for over 400 words. Hey, are you paid by the word?”

First, I make no money from this website. And second, I said this was a discussion, not a rant. In a discussion, you must provide context and frame the problem. Otherwise it’s just blatent complaining.

Remember that SanDisk page (pictured above) where they helpfully explained the difference between a Gigabyte and Gibibyte for us? Well, I didn’t show you the whole page. Here is what is written directly after the screenshot pictured above explaining a gibibyte:
sandisk_capacity_explanation_revealed

So, basically what SanDisk is telling us here is that the actual size of the device isn’t even the advertised capacity multiplied by 10^9 bytes, it’s actually less. This is basically the legal equivalent of them saying “trust us, it really has 128,000,000,000 bytes inside, but you can’t use them all.”

Somehow, our governments have decided that this kind of advertising is legal.

I went back and looked at the Amazon.de page where I bought the product. It might shock you, dear reader, but there was no fine bullet point in the specifications saying “actual user storage less”:
amazon_de_128GB_cruzer_blade

This is understandable. If people saw “actual user storage less” mentioned in the product advertisement, they would probably be suspicious of the amount of storage they were actually buying, and sales would suffer.

I thought I would go look at other retailers to see if “actual user storage less” was mentioned anywhere on their websites. Here is the same product listed on Amazon.com:
amazon_com_128GB_cruzer_blade

What about NewEgg.com?
newegg_com_128GB_cruzer_blade

Okay, so the companies selling these devices aren’t overly eager to include this fine print, which SanDisk actually includes on their website. In small text, at the bottom of the page:
sandisk_com_128GB_cruzer_blade

I emailed SanDisk about this to ask why the “actual user storage less” wasn’t mentioned on any retailers website, and they responded:
sandisk_reply_1

While I disagree with their reply, I understand that retailers have some freedoms in how they advertise a product. However, with this in mind, I am sure that a small army of SanDisk lawyers would co-sign a cease and desist letter if I started advertising their products in any way which they determined was harming their brand value. But then something curious happened…

sandisk_reply_2

SanDisk claims that they don’t have any control over how retailers advertise their product, but then they state that these companies are “authorised distributors and resellers.”

Given the incredibly high percentage of counterfeit products being sold these days under the label of a well known brand, it’s clear that manufacturers need a trustworthy outlet to sell their goods, or consumers might begin to doubt the quality of their brand. That’s the economic impact of “electronics priacy” [PDF].

Managing your supply chain and maintaining your brand image costs companies millions, if not billions, of dollars every year. It’s serious business. People go to jail for importing and selling counterfeit products.

So, when a company claims that they have no control over how their product is advertised, I find that a bit difficult to believe. Legally they may not have an obligation to require retailers mention “actual user storage less” but morally and ethically they should ensure that their retailers do not advertise their product in a misleading way.

SanDisk is selling a “128GB” USB stick, which has a raw capacity of ~125GB (116GiB):
sandisk_cruzer_128GB_fdisk

They also mention in small text, on the back of the package, that “actual user storage less.” Too bad they didn’t state this anywhere on the actual retail page.

At this point, anyone who is sane will do the math and ask the question “You received 2.2% less capacity than was advertised. Why have you wasted your time writing about this?” and that’s because I’m scared of the precedent this is setting.

It’s true, it’s seemingly pointless to sit here and discuss the missing 2.2%. But 10 years ago, you would spend a lot of money to buy a USB stick which was 2GB. Even now, most people pay phone companies tens of dollars per month to transfer 3GB over 3G or LTE, or $90 per month if you’re unlucky enough to own a smartphone in Canada:
canadian_cell_plans

So even though as a percentage, it’s relatively small, it a not-insignificant amount of capacity that’s missing. If this was a 1TB device, you would be missing 23GB, and this is even before we get into Giga versus Gibi and formatting…

When I buy a device where the primary function is storage, I expect to be buying a device which can contain $CAPACITY * 10^9 bytes of data. This is important for things like data recovery. If I need to make an exact duplicate of the data on a storage device which is 128GB, I don’t want to have to worry about buying a device from a specific manufacturer, model, or serial number range to be able to store the data. This isn’t swapping the PCB on a broken hard drive, it’s just buying a simple storage device!

I’m scared of what message we’re sending to manufacturers when we allow them to sell us products with vague statements like “actual user storage less” and what it means for the future of the industry.

If I buy a smartphone which has 16GB of space, I expect to be able to use less, because the primary function of a smartphone is to be a pocket computer. I understand that capacity is required for the operating system, and that the actual capacity available to me will be less than the advertised amount.

But to buy a device whose sole purpose is to store data, and have that device provide less capacity than advertised. That’s scary, not for the 2.2% that I can’t use today, but because tomorrow it might be 5%, and in 10 years 20%.

Why is the capacity less? Perhaps they’re using NAND which doesn’t have space over provisioned for error correction and wear levelling, allowing them to fit a few more on a wafer. Or maybe the chips have bad regions which they’ve mapped around, meaning you get a slightly smaller capacity. I doubt we’ll ever get an explanation apart from “actual user storage less.”

SanDisk isn’t the only manufacturer doing this, but they’re certainly the worst offender that I’ve found. Recently I purchased some 16GB micro SDHC cards from Transcend, and they’re 0.5% under capacity as well. Luckily I have found that Samsung’s EVO line appears to at least provide the advertised capacity in 10^9 bytes. But, how much longer until everyone clues in to the get out of jail free card that is “actual user storage less” and starts selling devices under the 10^9 capacity?

The moral of the story here is: complaining on the internet is useless. Vote with your wallet. Return anything which is not actually 10^9 bytes of capacity, and rate it accordingly to warn other users. I’m not going to support a brand which, in my opinion, allows retailers to advertise their products in misleading ways. Besides, the performance sucked (4.5MB/s sequential write). You get what you pay for.