Author Archives: Hal Martin

About Hal Martin

In my free time I like experiment with hardware and embedded systems. Here I write about personal projects and random adventures into firmware land.

From SATURN with love: a bad USB power supply

3 Replies

Recently I needed another USB charger for my devices. I went to Saturn and picked up this Innergie adapter, which is rated for 10W (2A at 5V).

innergie_front

innergie_rear

But, it doesn’t work. Sure, it can put out 2.5A, if you short it. I managed to get 0.6A out of it charging a 24Wh lithium-polymer battery, but at that current the voltage drops to 4.5V, which is outside of the USB specification and will not charge a mobile phone or tablet.

innergie_fluke_voltage_current

When I did plug in a mobile phone, it drew about 0.3A (1.5W) but the charging light did not turn on. The Innergie cannot even hold up at 1.5W, it dropped to 4.75V!

I lost the receipt, otherwise I would return it for my money back. So into the junk pile this goes… Lesson learned, next time save the receipt!

Installing CentOS 7 with a chroot

2 Replies

I needed to install CentOS 7 on an embedded PC with UEFI and 2 SSD disks in mdadm RAID1.

While I’m sure the guys at Red Hat work very hard on CentOS, the installer is a piece of cr*p, especially when it comes to disk partitioning. I have never hated any installer more than the CentOS disk partitioner. I don’t know what happened. The disk partitioning tool in CentOS 6 installer was fine, I had no problems using it, but in the 7 installer it’s just a nightmare to do anything. In my opinion Windows installer does a better job of disk partitioning than the CentOS 7 installer.

While many people who like CentOS will proclaim that the error is between the keyboard and chair, I welcome them to provide a write-up and screenshots of how to accomplish my desired partitioning scheme using the graphical installer. If the instructions are shorter than this blog post, next time I won’t use a chroot to install.

I loosely based my method off of this post, but immediately found I had to deviate because I didn’t have USB install media with all the required commands on it.

Required tools:
1) USB stick with some Linux distro on it (I prefer the Gentoo minimal installer, it’s small and it includes lots of useful utilities)
2) USB stick with the CentOS minimal installer on it
3) About 2GB of free space, you can use a ramdisk, or create an extra partition using the free space on the USB sticks (CentOS occupies about 800MB, Gentoo about 300MB)

Steps:
1) Boot the Gentoo installer off the USB stick
2) Partition your disks however you like using gdisk, fdisk, or parted
3) Create your mdadm array(s)
4) Plug in the CentOS 7 USB stick and mount it to a temporary mount point (e.g. /tmp/centos)

Inside the CentOS USB stick you will find LiveOS/squashfs.img, you need to loopback mount this:

livecd ~ # mount -o loop /tmp/cinstall/LiveOS/squashfs.img /tmp/csquashfs/

Now we have yet another image to mount, this one within the squashfs image:

livecd ~ # mount -o loop /tmp/csquashfs/LiveOS/rootfs.img /tmp/croot

Finally we have a Linux filesystem. But unfortunately we cannot use it for anything as it is mounted read-only and there is no resolv.conf present, so no domain names can be resolved. This is why you need ~2GB of free space somewhere (or 4GB of RAM).

livecd ~ # mkfs -q /dev/ram1 1572864
livecd ~ # mkdir -p /tmp/ramdisk
livecd ~ # mount /dev/ram1 /tmp/ramdisk
livecd ~ # rsync -avHp /tmp/croot /tmp/ramdisk

Now that we have the installer rootfs somewhere writable, copy /etc/resolv.conf to the filesystem:

livecd ~ # cp /etc/resolv.conf /tmp/ramdisk/etc/resolv.conf

Mount your destination partition for CentOS somewhere you can access from within the chroot:

livecd ~ # mount /dev/vg0/centos /tmp/ramdisk/mnt

Chroot to the installer environment:

livecd ~ # chroot /tmp/ramdisk

Download the CentOS release RPM and install it to the destination partition:

bash-4.2# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/centos-release-7-0.1406.el7.centos.2.3.x86_64.rpm
bash-4.2# rpm --root=/mnt --nodeps -i centos-release-7-0.1406.el7.centos.2.3.x86_64.rpm

Because yum is missing the yummain module in the installation environment, we need to download and install the yum RPM on the installer partition:

bash-4.2# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/yum-3.4.3-118.el7.centos.noarch.rpm
bash-4.2# rpm -i --nodeps yum-3.4.3-118.el7.centos.noarch.rpm

Now finally we can run yum on the destination partition to install CentOS:

bash-4.2# yum --installroot=/mnt update
bash-4.2# yum --installroot=/mnt install -y yum
bash-4.2# yum --installroot=/mnt install -y @core kernel
bash-4.2# yum --installroot=/mnt install -y grub2-efi efibootmgr lvm2 mdadm \
dosfstools kernel

Now unfortunately I hit a small snag: the Gentoo installer isn’t EFI aware. Exit the chroot, but remember to copy /etc/resolv.conf to the destination partition:

livecd ~ # cp /etc/resolv.conf /tmp/ramdisk/mnt/etc/

Poweroff and unplug the Gentoo installer USB stick. Plug in the CentOS installer stick and boot to the rescue environment. Skip rootfs detection.

Mount the partition/LV slice containing your CentOS installation:

sh-4.2# mkdir /mnt/centos
sh-4.2# mount /dev/vg0/centos /mnt/centos
sh-4.2# mount -t proc proc /mnt/centos/proc
sh-4.2# mount --rbind /dev /mnt/centos/dev
sh-4.2# mount --rbind /sys /mnt/centos/sys
sh-4.2# chroot /mnt/centos /bin/bash

Save the mdadm array information to the mdadm.conf configuration file:

bash-4.2# mdadm --detail --scan > /etc/mdadm.conf

Format your EFI boot partition:

bash-4.2# mkfs.vfat /dev/sda1
bash-4.2# mkdir /boot/efi
bash-4.2# mount /dev/sda1 /boot/efi

Install grub:

bash-4.2# grub2-install
Installing for x86_64-efi platform
...
Installation finished. No error reported.

Now, dear reader, this is the part where you do not see the hours I spent debugging why dracut would not find my root partition (hint: see Fedora wiki for dracut debugging steps). The tl;dr is that the mdadm array wasn’t being assembled, for reasons still unknown. To solve this we need to add our array UUID as an additional kernel parameter for grub:

bash-4.2# MD_UUID=$(mdadm -D /dev/md0 | grep UUID | awk '{print $3}')
bash-4.2# grubby --update-kernel=/boot/vmlinuz-3.10.0-123.20.1.el7.x86_64 \
--args="rd_MD_UUID=$MD_UUID"
bash-4.2# grub2-mkconfig -o /boot/grub2/grub.cfg

Absolutely verify in /boot/grub2/grub.cfg that the correct rd_MD_UUID was appended to linuxefi, or like me, you may be left wondering why your system won’t boot.

Check with efibootmgr that a menu entry was created:

bash-4.2# efibootmgr -v
...
Boot0014* grub  HD(1,800,32000,SUPER-LONG-UUID)File(\EFI\grub\grubx64.efi)

Don’t forget to set a root password:

bash-4.2# passwd
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully

Configuring the hostname (using hostnamectl, or editing /etc/sysconfig/network and /etc/hostname), udev rules for eth* interface names, and static network configuration in /etc/sysconfig/network-scripts/ifcfg-eth* is left as an exercise for the reader.

After rebooting

If when you reboot you find that you cannot login as root using the password you specified, it’s probably SELinux. Normally I hate disabling SELinux, but in this case I was so tired of spending a day and a half debugging booting issues, I just disabled it and went on with the setup.

Despite what the wonderful CentOS installer tells you, you do not require a separate /boot partition. Right now this is what the partition layout looks like:

Disk /dev/sda: 128.0 GB, 128035676160 bytes, 250069680 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1         2048       206847    100M  EFI System      EFI System
 2       206848    250069646  119.1G  Linux RAID      Linux RAID

Because the contents of /boot/efi is static, and cannot be mdraid, remember to copy the contents of /dev/sda1 to /dev/sdb1 so that if your first drive ever dies, you will still have the required EFI components to boot off the second drive.

Overall, I can’t say this was a lot of fun. But I did learn a lot more about dracut and the CentOS booting process, and I can still say with confidence that I hate the CentOS 7 installer.

Unitymedia is blocking VPN connections

28 Replies

unitymedia is a German ISP, and I made the unfortunate choice of using them as my internet service provider.

Unfortunately, they’re not a very good ISP, because they are using deep packet inspection (DPI) to throttle or block VPN connections. I would expect this sort of behaviour from certain countries (a full list of countries engaging in internet censorship can be found here) but never in Germany.

Obviously this claim cannot be made lightly. There must be some sort of proof to back up anyone who is claiming that a German ISP is filtering customer’s internet connectivity.

Could this be a mistake?
Possibly. Unitymedia does not have enough IPv4 addresses to give each customer a global IPv4 address. They are instead using something called DS-Lite in which each customer has a unique IPv6 address, and IPv4 connections are tunneled over IPv6 (4in6) to the carrier where customers then must go through carrier-grade nat (CGN) before reaching the internet.

Other people have written about frequent TCP disconnections and poor connectivity on unitymedia (DE, English version here), but I believe they are going farther.

The evidence
I had a strong suspicion from my regular use that unitymedia was filtering connections, but I needed a good way to prove it.

Since the most common use for a VPN is to get around region restrictions (honourable mention to YouTube DE and GEMA) or to download copyrighted content illegally, I thought torrenting Ubuntu 14.04 would be a good example use case.

I installed Transmission and configured it to require encrypted connections. This will ensure that any network connections to peers will be encrypted UDP. I should also note that IPv6 was disabled on the computer, so only IPv4 connections are possible.

So, for the first test, I downloaded the torrent for Ubuntu 14.04 LTS AMD64, and started downloading.

encrypted_udp_torrent_novpn

Everything is still working, good.

Now I paused the torrent and logged in to a paid OpenVPN service on UDP port 1194. To prove that the connection is working, I will run a ping test to Google’s public DNS. This is what happened next:

encrypted_udp_torrent_udpvpn

Okay, so we are still downloading, but much slower than before. Unitymedia would not be the first ISP in the history of the uncensored world to throttle VPN connections.

However, we cannot browse to websites. Chrome and Firefox just sit forever waiting for data from the remote server. To test my VPN connection, I tethered my phone to my PC and logged in to the VPN. Websites load perfectly.

But, what if we tried a TCP-based OpenVPN connection? Let’s connect to an OpenVPN endpoint using TCP on port 443. In the upper right xterm I have logged into my router and I am filtering for TCP RST packets on my upstream WLAN connection (to the unitymedia router). We can see that the ping is still running, although we have not started to download anything yet.

encrypted_udp_torrent_tcpvpn-working

I resume the torrent and all of a sudden I receive 2 TCP RST packets and the connection is dead.

encrypted_udp_torrent_tcpvpn-blocked

This is absolutely unacceptable. Under no circumstances should an ISP ever be inserting TCP RST packets into a user’s connections. Here is the PCAP file with the TCP RST packets that ended my connection.

ISPs inserting TCP RST packets to disrupt user connections is sadly not new. Comcast in the United States has been known to forge TCP RST packets for users who are using bittorrent, calling it “Network Management.” The EFF even compiled a report on Comcast’s shady behaviour. There is a Wikipedia article on the TCP reset attack if you want to know more about how it works.

An ISP is providing a service to paying customers, and at no point should they ever interfere or attack a customer’s connections. Sadly these days the terms of service (ToS) are usually in favour of the service provider, and not the customer.

I find it ironic that in Germany, which has spied on its citizens in the past, I cannot use such privacy enhancing technology as VPNs.

There are many legitimate uses for a VPN such as: encrypting your internet traffic from your ISP, anonymizing yourself to avoid higher prices in online shopping, or getting cheaper plane tickets for your holiday.

I have confirmed that this issue is not isolated to my Unitymedia connection. I have a friend living in another large German city, and their connection faces the same hostile behaviour from Unitymedia.

Unitymedia’s filtering is also not limited to commercial VPN services. My work requires I use a VPN connection to access internal services, and I frequently experience disconnections, dropped packets, extreme throttling, and corrupted packets while trying to work.

So, why am I writing this? I live in a free country, I can vote with my wallet and move to a different ISP. This is true, but if I did that, people who do not have the technical knowledge to diagnose connection problems would never know that Unitymedia is censoring their internet. Unitymedia would just be blasted for their DS-Lite implementation, and things would never improve.

In conclusion:

linus-torvalds-220612

Unitymedia is sending forged TCP RST packets to disrupt TCP VPN connections. They are throttling, and I suspect also filtering, UDP VPN connections to a severe extent.

thatwouldbegreat