I’ve been living in Switzerland for about 4 years now. Ever since I moved here I was more than happy with the offerings from init7, probably the best ISP in Switzerland and definitely one of the best in the world. They are not the cheapest, but their reliability and service quality far outweigh their asking prices. I don’t think I’ve ever had a case where my speeds measured more than 5% off than the advertised maximum and I’m sure I’ve never had any unplanned downtime at all. To compound on that, the company understands its power-user client segment and are more than happy to accommodate more advanced configurations.
A year or two ago init7 started offering 10 and 25Gbps symmetric fibre connection. Apart from the initial setup fee (CHF222 for the latter option) the monthly cost remained the same. I’ve been with an 1/1Gbps connection for a while and it’s more than enough for most power users, not just regular clients (foreshadowing). However, in the meantime I decided to retire my remote server I had running at Hetzner and attempt to selfhost everything, with the procedure becoming a game of “where can I go next?”
The Server
I upgraded my workstation to an Intel 14900K, and my previous hardware (based on an Intel 9900K) replaced my existing local server (which used to be an Intel 3770K). The most significant reason compelling me to upgrade, is the need to run a lot of virtual machines (on Proxmox) and the 3770K server was limited to 32GB of RAM, versus 64 that I had on the 9900K workstation.
My server at that point ran Unraid, a couple of VMs (Linux and Windows), a Domain Controller and a Kubernetes cluster for experimentation. I used to have a dedicated Raspberry Pi running PiHole which was moved to an LXC in Proxmox.
I obtained a static /48 IPv6 Prefix from init7 (for free!) as well as a static IPv4 address, then moved to move my few websites (including this one!) & database servers from the Windows machine I had on Hetzner to two Debian VMs, hardened and set to their own restricted VLAN.
The connection, preparation
As I mentioned, 1Gbps is plenty for almost every usage, including medium usage websites (that I have none of, currently). However, I started thinking about init7’s offer and started investigating my options. They offer a router, but after reading Michael Stapelberg’s experiences with it (one of the first people to dive into and share his experience with upgrading his connection to 25Gbps!) as well as its price, I started looking into alternatives.
On my previous router (running on an TP-Link Archer C7) I used OpenWRT and I find that router OS brilliant. Could I keep using that? It turns out that yes, I could, since there are x64 builds available. So, one extra VM later and my OpenWRT router was connected to the 1Gbps media converter using a PCIe pass-through of my 4-port NIC and… all was golden. The only external networking hardware I had was three TP-Link RE650 units (also running OpenWRT) in order to provide WiFi to the apartment and be able to cover all of it, as well as offer guest & IoT networking access (on their own VLANs of course).
At this point I realised that I’m pretty much putting all my eggs into one basket, but I have daily 3-2-1 backups and I can get back to a functioning state quickly if I need to. I kept my old server, albeit decommissioned, just in case there’s a hardware problem that I can’t resolve quickly enough. Since I am hosting everything for myself mostly, it doesn’t make much sense to have two Proxmox machines running (+ a raspberry pi for quorum) just to achieve HA; wouldn’t do me much good anyway if the network adapter were to go down.
But I was ready for the next pieces of the puzzle, that is optics and fibre hardware.
The connection, optics
I realised that buying hardware for fibre-optical connections can be very tricky. There are different protocols, wavelengths, cable types and one mistake means no networking. Luckily, init7 has comprehensive information about the optics requirements, but it’s easy to feel lost when considering the hardware to use. Here’s what I ended up with:
- Two Mellanox ConnectX-4 (MCX4121A-ACAT) from Jacob.de with two SFP28 ports
- An AOC (active optical cable) from FS.com, compatible with Mellanox (more on this later)
- An Optical Transceiver compatible with init7’s requirements (again, from FS.com)
The plan was to place one of the Mellanox cards on my server and the other to my main workstation, since I wanted one machine that can take advantage of the (up-to) 25Gbps connection. In theory this would speed up the communication between the server and the workstation, but there wasn’t much point to it after all. My NAS uses spinning disks, so at best it can utilise 1-2Gbps of traffic and my workstation backups (using restic) are limited by those disks.
The connection between my workstation and my server would happen through the AOC cable. This is an optical cable that is soldered to a transceiver. It has the advantage of being significantly cheaper than a an optical cable with separate extra transceivers (like, one third of the price or so).
The transceiver I bought however, I used for connecting my optical wall-wart to the Mellanox card on my server with the patch cable. Order the wrong type or protocol or wavelength of that, and at best you won’t have any connection, at worst, the cables won’t even fit! I triple-checked everything, but even the slightest thing can throw you off (prime example being, the difference between optical connector types (APC/UPC) where mixing the two can cause actual damage to the sockets).
There is one other factor one needs to consider with both AOC & Transceivers, and that’s the firmware they run. In theory, all can run a generic firmware and be compatible with one another, but if you have Cisco hardware for example, it’s a good idea to have a transceiver with Cisco firmware. FS.com allows you to customise this, so I chose Mellanox for the highest compatibility.
My cards and AOC cable arrived way earlier than init7’s activation and I realised that Jacob.de actually sent me used (I don’t mind) Dell (I mind) cards. Firmware upgrading using Mellanox’s firmware isn’t possible unless you crossflash, but I wasn’t courageous enough to do that (yet). Thankfully, despite the different firmware between the cards and the transceivers/AOC, they did work without a hitch as soon as they connected.
The cards only support Ethernet. It is possible to buy versions of those that support InfiniBand, but for my usage that would be a waste of money.
The day has come!
Until init7 enabled my 25Gbps, I had to keep using my 1Gbps media converter for Internet uplink, since the transceiver on the Mellanox card wouldn’t communicate with the 1Gbps fibre directly. As soon as the switch happened, the media converter couldn’t get a signal anymore and it was time to switch the cable.
It worked immediately! In terms of speedtest.net benchmarks, I got about 12Gbps/11Gbps from my workstation, however testing from my router VM (to which I have the Mellanox card directly passed through) or my Proxmox hypervisor using iperf3 and the speedtest.init7.net address, I can get ~22Gbps in both directions with two parallel connections. Not always; at peak hours the network is busier, but the lowest I’ve seen is ~6Gbps.
My workstation is running Windows and using the windows build of iperf3 I seem to be limited to 12Gbps. I am still not sure if this is due to the PCIe 3.0 x8 card running at a PCIe 4.0 4x slot, however even at PCIe 3.0 speeds, 4x should allow for 32Gbps of maximum data transfer rate. I still haven’t spent any time troubleshooting this (and Mellanox cards have very deep tuning capabilities that I’ve yet to look into), but iperf3’ing to the router VM (where the other end of the optical connection lies) gives me about 20/16Gbps. Keep in mind, the cards are passed through directly to the VM and are not paravirtualised. They also have enough CPU cores and hardware offloading is enabled in OpenWRT.
What good is it for?
So… why? Well, my first thought was to improve my remote backup scenarios. I am using Hetzner’s Storage Boxes which offer very cheap pricing per terabyte, however I’ve discovered that they are kinda limited to ~300Mbps. I could rent speedier boxes, but it can get expensive fast and I need about 4TB of space.
I tried testing my speed to an ftp server that I use regularly, which managed to saturate my Gigabit connection, only to find out that it’s really not capable of more than 1.5Gbps. The websites I have are pretty light and OpenVPN connections to my infrastructure were never a problem even with the Gigabit uplink.
So, what good is it for then? Nothing really for my current use cases. But as I realised that the only realistic use cases I’ve seen were speedtests, perhaps I could offer a server for such purposes; the bandwidth would go to waster otherwise.
Speedtest servers
I’ve created another VM, installed the server script from Ookla and registered my server. The installation was pretty straightforward and It should be visible by the time I’m writing this. You should be able to find my server as James Kavakopoulos (vtable) and it should support 12/11Gbps on normal days and up to 20something on better days (depending on traffic). It supports IPv4 as well as IPv6 connections.
I have also set up an iperf3 server at speedtest.vtable.org:9999 if that’s more your thing.
I live near the Zurich Airport and I suspect the infrastructure should be at least decent. Give it a try and let me know! I am absolutely welcoming any improvement advice you may have as well and finally, if I can help with any questions you may have about doing the switch, feel free to ask!
Note: I’ve disabled comments here; comment spam’s too much hassle to deal with. Reach me at blog[at]vtable[dot]org. Thanks!