Tag Archives: intel

HP MicroServer Gen8: two PCIe, too furious

1 Reply

The HP ProLiant MicroServer Gen8 was quite the impressive budget server when it was released in 2013.

Behold the power of some CPU cores, a maximum of 16GB of DDR3 RAM, and 4 to ?? disks

Intel was dominating desktop CPUs, and a typical desktop or workstation motherboard of this era had a single PCIe 3.0 x16 slot, and then several PCIe 2.0 slots (e.g. x4 and x1). The MicroServer Gen8, due to its small size (slightly larger than ITX), was limited to only one PCIe 2.0 x16 slot. You also did not have the option to bifurcate the PCIe x16 slot, as is common in recent years.

At least, that is what everyone thought. This adventure started when I found a very interesting blog post about bifurcating the PCIe x16 slot on Intel socket 1155:

As mentioned above, I want to set 16=8+4+4, so the values of CFG[5] and CFG[6] should be both 0. So we can ground CFG[5] and CFG[6] by connecting N35M35 and L37M37. (VSS is GND, and VCC is PWR)

This information is from the Desktop 3rd Generation Intel® Core™ Processor Family, Desktop Intel® Pentium® Processor Family, and Desktop Intel® Celeron® Processor Family datasheet:

Desktop 3rd Generation Intel® Core™ Processor Family, Desktop Intel® Pentium® Processor Family, and Desktop Intel® Celeron® Processor Family

The author found that by physically bridging the pads corresponding to CFG[5] and CFG[6] they could force the PCIe x16 slot to bifurcate into x8x4x4. Since there is no official support for PCIe bifurcation on the Ivy Bridge generation, this was an interesting finding and one which I feel went largely unnoticed.

The author used silver paste, which is more permanent than what I wanted. I decided to go a more reversible route: using kapton tape and very thin strips of aluminum foil (precision cut with Fiskars scissors from a roll of baking foil).

LGA1155 CPU modified for x8x4x4 bifurcation

It is important to note that by default CFG[5] and CFG[6] are pulled up to 1. Masking the pads with kapton tape is ineffective, they must be grounded to an adjacent GND pad.

WARNING: You can cause permanent damage to your CPU and motherboard by bridging incorrect pads. Always verify that the pads you have bridged are correct before installing the CPU into the socket.

So, I obtained HP MicroServer Gen8 and got to work testing this.

Bridging both CFG[5] and CFG[6] for x8x4x4 resulted in the system hanging indefinitely during the “Processor Initialization Complete” step in POST.

HP MicroServer Gen8 BIOS POST screen

Boot stuck at “Processor Initialization Complete”

Bridging only CFG[5] to set bifurcation to x8x8 did work, and the system passed POST.

LGA1155 CPU modified for x8x8 bifurcation

With this modification, you can install two PCIe devices in the PCIe x16 slot of the MicroServer Gen8. Unfortunately due to physical constraints in the chassis, it is not possible to install a half height card on top of a half height riser (e.g. x8 and dual NVMe) but you can install a dual or quad NVMe adapter and use two PCIe x4 lanes (e.g. for an M.2 SATA controller and an M.2 2.5/5G NIC). If you do not mind PCIe cards hanging out the side, you can use a dual x8 riser like the one pictured below.

HP MicroServer Gen8 with x8x8 riser installed

Here is the lspci output of two ConnectX-3 Pro 40G network cards installed in a MicroServer Gen8:

# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C204 Chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
07:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
0a:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

Of course the MicroServer Gen8 is very under powered and can only dream of saturating even a single 40G link.

# lspci -s 07:00.0 -vvv
07:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
        Subsystem: Hewlett-Packard Company InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter
        Physical Slot: 1
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at fbe00000 (64-bit, non-prefetchable) [size=1M]
        Region 2: Memory at f6000000 (64-bit, prefetchable) [size=32M]
        (...)
        Capabilities: [60] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 116W TEE-IO-
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 256 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (downgraded), Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        (...)
        Kernel driver in use: mlx4_core
        Kernel modules: mlx4_core
# lspci -s 0a:00.0 -vvv
0a:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
        Subsystem: Hewlett-Packard Company InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at fbf00000 (64-bit, non-prefetchable) [size=1M]
        Region 2: Memory at f8000000 (64-bit, prefetchable) [size=32M]
        (...)
        Capabilities: [60] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 116W TEE-IO-
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (downgraded), Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        (...)
        Kernel driver in use: mlx4_core
        Kernel modules: mlx4_core

To anyone still running a MicroServer Gen8 and would like more expansion, I hope you find this information useful.

Meraki MS350 hardware overview

2 Replies

The Meraki MS350 (MS350-24 and MS350-48) series switches offer 24 or 48 ports of Gigabit Ethernet. The MS350-24X offers 16 ports of Gigabit Ethernet, and 8 ports of multi-Gigabit (1/2.5/5/10G) Ethernet. All models have four SFP/SFP+ uplink ports, a dedicated remote management port, and stacking capabilities via QSFP. Today we will be looking at the MS350-48 and MS350-24X models specifically.

MS350-48LP from Meraki’s datasheet

Here is a quick summary of the MS350 specs:

  • Intel Atom C2358 CPU (2C/2T, 1.74GHz)
  • 2GB DDR3 ECC RAM (SODIMM, Unigen U25U7210N8FD-BDD-CCHF1)
  • 16MB SPI flash, 2GB NAND flash (TSOP48 NAND on motherboard, USB via Phison)
  • MS350-24X: 30 Network interfaces (16 Gigabit Ethernet, 8 mGig Ethernet, 4 SFP+, 2 QSFP stacking)
  • MS350-48: 54 Network interfaces (48 Gigabit Ethernet, 4 SFP+, 2 QSFP stacking)
  • MS350-24X: MA-PWR-1025WAC
  • MS350-48: MA-PWR-250WAC (identical to PWR-C2-250WAC)

The MS350 RAM can be upgraded: the 4GB (HMT351A7CFR8C) and 8GB (HMT41GA7MFR8C) PC3-12800E SODIMMs have been confirmed to work. Non-ECC SODIMMs are not compatible.

Model Codename Part number
MS350-24 (P) BigPony 600-36010 (P: 600-36020)
MS350-48 (LP/FP) BigPony 600-36030 (LP: 600-36040, FP: 600-36050)
MS350-24X Clydesdale 600-48010

The MS350-48 uses the Broadcom BCM56547 (A0) ASIC, with BCM84740 PHYs. The MS350-24X uses the Broadcom BCM56546 (B0) ASIC, with BCM82578 and Aquantia AQR405 PHYs. PoE versions of the switch use the Broadcom BCM59121 PSE controller.

MS350-48:

00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)
00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02)
00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02)
00:0b.0 Co-processor: Intel Corporation Atom processor C2000 QAT (rev 02)
00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)
00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)
00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 1.0 GbE Backplane (rev 03)
00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)
00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries Device b547 (rev 01)
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries Device b547 (rev 01)

MS350-24X:

00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)
00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02)
00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02)
00:0b.0 Co-processor: Intel Corporation Atom processor C2000 QAT (rev 02)
00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)
00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)
00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 1.0 GbE Backplane (rev 03)
00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 1.0 GbE Backplane (rev 03)
00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 1.0 GbE Backplane (rev 03)
00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)
00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries Device b546 (rev 11)

Both models have the same USB devices present:

Bus 001 Device 002: ID 8087:07db
Bus 001 Device 001: ID 1d6b:0002
Bus 001 Device 003: ID 13fe:5200

MS350-24X and MS350-48 both use coreboot as the bootloader, although the MS350-24X model has a different build. In both cases, the ROM has the following layout:

00000000:00010000 reserved
00010000:0070ffff bk1
00710000:00dfffff bk2
00e00000:00ffffff coreboot

The cbfs contains the following:

FMAP REGION: COREBOOT
ms350-24x_w25q128.bin: 16384 kB, bootblocksize 1024, romsize 16777216, offset 0xe10000
alignment: 64 bytes, architecture: x86

Name                           Offset     Type           Size   Comp
cmos_layout.bin                0xe10000   cmos_layout      1396 none
fallback/romstage              0xe105c0   (unknown)       21624 none
fallback/ramstage              0xe15a80   (unknown)       49421 none
fallback/payload               0xe21c00   simple elf      23042 none
config                         0xe27640   raw              4676 none
revision                       0xe288c0   raw               566 none
(empty)                        0xe28b40   null          1209432 none
mrc.cache                      0xf4ffc0   mrc_cache       65536 none
cpu_microcode_blob.bin         0xf60000   microcode       84992 none
(empty)                        0xf74c40   null            45912 none
fsp.bin                        0xf7ffc0   fsp            389120 none
(empty)                        0xfdf000   null           134040 none

coreboot was built with an ELF payload (miles) which by default loads and jumps into the bootkernel FIT image located at 0x10000. A secondary bootkernel exists on flash at offset 0x710000.

This is very similar to the MX84 as they are both based on the same Rangeley platform.

The entire MS350 series is based on the Intel Atom C2000 series CPU, which Meraki also used in the MX84. Sadly, the MS350 also suffers from the AVR54 errata, as the C2358 in both the MS350-48 and MS350-24X is the B0 revision.

LPC_CLK is exposed on pin 1 of J35, with R3635 carrying 3.3V (MS350-48 and MS350-24X). Therefore, you can add a 100 Ohm resistor between R3635 and pin 1 to pull up the LPC clock. Just be sure to use an “extended-life” resistor for the modification, you wouldn’t want to compromise the MTBF of your Meraki product with anything sub-par 😉

100 Ohm resistor to pull up LPC clock (MS350-24X)

If you wish to flash your MS350, you will need to remove or socket the SOIC8 SPI flash (SK_U1).

This is because there are other devices powered by the +3.3V voltage rail used by SPI flash, which interferes with your ability to read/write the contents of flash. I prefer the Wieson G6179-10 SOIC8 socket (available from Adafruit). People outside the US will probably find it easier to desolder the flash and use a SOIC8 socket with prototype wires, as the G6179-10 is difficult to obtain for a reasonable price.

The UART header is J31 on both the MS350-48 and MS350-24X and follows the standard Meraki UART pinout (1: VCC, 2: Tx, 3: Rx, 4: GND)

Similar to the MS210/225 series, the Broadcom SDK implements the packet engine in userspace, using the linux_kernel_bde and linux_user_bde kernel modules to interface with the ASIC. In the Meraki firmware, the packet engine is a component of the userspace click daemon, which loads the bcm_click shared object during click router initialisation.

Idle power consumption:

  • MS350-24: 45W
  • MS350-24P: 57W
  • MS350-48: 54W
  • MS350-24X: 96W

GPL source code for the MS350 was requested from Meraki in July 2023. At the time of writing, they have not provided any. I will update this post with links to the source code when it is provided.

2.5Gbit Ethernet for TinyMiniMicro labs

1 Reply

2.5Gbit Ethernet is finally at an affordable price, but modern platforms do not offer much in the way of upgrade paths. Desktop PC motherboards typically dedicate all PCIe lanes to graphics or NVMe, leaving you with tough choices to make if you want to upgrade your network card. The situation is even worse for small form factor and embedded devices.

Or so I thought, until I discovered an M.2 A+E key 2.5Gbit Ethernet card based on the Realtek RTL8125B.

This tiny M.2 2230 card can be installed in the M.2 WiFi slot present on many motherboards. If you were not already using WiFi, this means you have a “free” upgrade path to 2.5Gbit Ethernet, without sacrificing any higher bandwidth PCIe slots. Better still, M.2 A+E keyed slots are commonly available in the “TinyMiniMicro” segment of small-form-factor PCs. This allows you to install 2.5Gbit networking in the Asus PN50, or an HP T640 thin client. You can also find mini-PCIe to M.2 A+E adapters, allowing you to install the NIC in a device that predates M.2.

Also attractive is the price, I bought two for 15.70€/piece (including VAT and shipping) from AliExpress. This is only a small premium over what a full-size PCIe card with an RTL8125B costs (typically around 13€).

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 50
        IOMMU group: 8
        Region 0: I/O ports at e000 [size=256]
        Region 2: Memory at fe910000 (64-bit, non-prefetchable) [size=64K]
        Region 4: Memory at fe920000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fe900000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 01
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp+ ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00000800
        Capabilities: [d0] Vital Product Data
                Not readable
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-
        Capabilities: [168 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [178 v1] Transaction Processing Hints
                No steering table available
        Capabilities: [204 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [20c v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=306176ns
                L1SubCtl2: T_PwrOn=150us
        Capabilities: [21c v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 
        Kernel driver in use: r8169

iperf3 testing shows that we can achieve consistent results over 2.4Gbit/s between the RTL8125 (M.2 A+E) installed in an HP T640, and the RTL8156 (Framework USB-C module).

Accepted connection from 192.168.10.2, port 45494
[  5] local 192.168.10.1 port 5000 connected to 192.168.10.2 port 45496
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   294 MBytes  2.47 Gbits/sec
[  5]   1.00-2.00   sec   295 MBytes  2.48 Gbits/sec
[  5]   2.00-3.00   sec   295 MBytes  2.47 Gbits/sec
[  5]   3.00-4.00   sec   295 MBytes  2.48 Gbits/sec
[  5]   4.00-5.00   sec   295 MBytes  2.47 Gbits/sec
[  5]   5.00-6.00   sec   295 MBytes  2.47 Gbits/sec
[  5]   6.00-7.00   sec   295 MBytes  2.48 Gbits/sec
[  5]   7.00-8.00   sec   295 MBytes  2.47 Gbits/sec
[  5]   8.00-9.00   sec   295 MBytes  2.48 Gbits/sec
[  5]   9.00-10.00  sec   295 MBytes  2.47 Gbits/sec
[  5]  10.00-10.00  sec   568 KBytes  2.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.88 GBytes  2.47 Gbits/sec                  receiver

Owners of the lesser powered thin clients should take note: the HP T530 (AMD GX-215JJ) can only manage around 1.6Gbit speeds in iperf3 testing.

You do not typically associate Dupont wires and high bandwidth being a good match, but somehow it works.

Realtek still has a bad reputation in some corners, so for those interested there are also sellers offering the Intel I225-V 2.5Gbit in M.2 B+M 2242 form factor.

The chip revision is SLMNG (B3) which from internet lore seems to be the revision where all the show-stopping bugs at link speeds above 1000M were resolved. I did not notice any instability in my iperf3 testing, the adapter was able to reliably maintain 2.45Gbit/s.

02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 50
	IOMMU group: 8
	Region 0: Memory at fe700000 (32-bit, non-prefetchable) [size=1M]
	Region 3: Memory at fe800000 (32-bit, non-prefetchable) [size=16K]
	Expansion ROM at fe600000 [disabled] [size=1M]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
		LnkCap:	Port #3, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [140 v1] Device Serial Number 88-c9-b3-ff-ff-b5-19-bc
	Capabilities: [1c0 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [1f0 v1] Precision Time Measurement
		PTMCap: Requester:+ Responder:- Root:-
		PTMClockGranularity: 4ns
		PTMControl: Enabled:- RootSelected:-
		PTMEffectiveGranularity: Unknown
	Capabilities: [1e0 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
		L1SubCtl2:
	Kernel driver in use: igc

I have not been able to find anyone selling the I225-V in the M.2 A+E form factor. However, you can adapt the M.2 2242 B/M key to an A+E key with an inexpensive passive adapter.

Performance is unaffected, but you should check that you have physical clearance for such an adapter as it extends the card length from 42mm to 53mm. This prevents one from installing the I225-V in the HP T530, as there is insufficient physical clearance for the card with the M.2 A+E adapter.

Finally, the I225-V M.2 designs I have seen are using larger perpendicular headers as compared to the Realtek, meaning they are less likely to fit in small/thin devices like the HP T640 thin client. Given the choice, I would stick to the Realtek for M.2 A+E applications rather than adapting the Intel I225-V.