The ultimate small business network
At my office, there are two recurring challenges in the area of file management and access. One is that I backup all of the data for my law practice daily using a full backup rather than incremental backup. So instead of adding only new files or changed files, I simply create a new archive of the entire file system and I get one massive .tar.gz file. That's fine for now, but it means that to backup my files, I need to copy over a gigantic archive file from my server to my workstation. Until August 2019, due to being unable to re-wire the office under my then-existing sublease, I was using a Netgear Orbi wireless setup that maxed out at about 60 MB/s. Transferring a 380 GB archive file took hours. The second challenge is that I want to work with files directly on my server so that everyone sees the latest version of each file. This can be problematic because the latency and throughput of the wireless network are not guaranteed, so I end up often copying the files to my local machine for editing.
Solution: a 10 gigabit wired network to replace my wireless network. I put this provision into my sublease when I was renewing it, and it was accepted. So on the weekend of August 3, 2019, I ran a Cat 6 cable from my office to the server room. That was the easy part, although it was no cake walk due to the large amount of mineral fibers and dust in the ceiling.
Wiring my office.
Wiring was only part of the challenge; I also had to upgrade my router, switches, and computers. I selected the Intel X550-T2 host adapter for my server and router, and an ASUS host adapter for my workstation. The Intel X550-T2 is a great product that works out of the box on Linux and the pfSense operating system. The ASUS host adapter required me to install the driver from the CD before using it on Windows 10, which was somewhat annoying but not unreasonable. Anyway, The X550-T2 has two 10GBaseT ports and installed nicely into both my router and server.
Intel X550-T2 installed in my pfSense router
Not quite the Ferrari of routers, but close
I had a huge problem with my pfSense appliance, in that it would not boot after five years of service (lol). I replaced the power supply but it didn't fix the issue; I was able to log into IPMI but the console screen was blank with no signal. It turned out the CPU was dead and I had to replace the motherboard, which I did. Side note: the prices on the Netgate web store are pretty high considering that with not too much effort I can recreate any of their products by ordering the parts from a local systems integrator that I work with, and installing pfSense myself. I plan to create my own router for the next project and make it have redundant Titanium-level power supplies for maximum reliability...you can't get that from Netgate.
For switches, I chose the Netgear 5-port 10 gig unmanaged switch (part no. XS505M), $399 as of this writing, because the 8-port switch in the same model series was double the price. This one has four 10GBaseT ports plus one SFP+ port for future upgrades if I ever decide to go the fiber optic route. In my current setup, fiber optic is not desirable because I want to be able to use Cat 6 wiring everywhere to support gigabit devices that may not be possible to upgrade to a 10 gig level. I can also daisy-chain a gigabit switch into one of the 10 gig ports in the event I need to connect multiple one-gigabit devices.
Once I got the 10 gig adapters installed, there was still a lot of work to be done. I ended up troubleshooting the pfSense router for hours because it turns out the Intel X550-T2 does not support traffic shaping with the pfSense built-in driver, at least as of version 2.3.5. The error message was that the driver does not support altq. So I was going crazy trying to figure out why the LAN had no internet access until realizing that the driver issue was causing the LAN firewall rules to not load. Once I deleted the traffic shaper, all was well. I also should note that the X550-T2 seems to conflict with the built-in Intel gigabit ports on the Supermicro C2758 board, such that the ports cannot be assigned to the WAN connection. This was an easy fix by assigning the WAN port to the secondary port on the X550-T2 and letting it run in gigabit mode. But that would be a problem if I had gotten the host adapter that has only one 10 gig port. So if you replicate this setup, be sure to get the X550-T2 instead of the X550.
Success!
There is wide variability in the transfer speeds; not sure what the hills and valleys mean, but it could be a result of the 100' long cable I am using. Note: this could be alien crosstalk (AXT); see my graphs here.
Update: I figured out the problem-- it was my ASUS network card. There must be a driver issue. Anyway, it was a piece of junk-- it died after three months. I got another Intel X550-T2 and tried that out...perfection!
I now max out my server's hard drives without even getting close on the network's capacity. This is ideal for working with files on the server and also transferring large backup files to my workstation. I recently added a RAID to my workstation that does about 950 MB/s, so when I upgrade my server down the road, I will be able to max out the connection. Not in any hurry here, for financial reasons-- 10 gig networking equipment is premium-level product. Speaking of which, here is the price list:
Total: $1,900
It was all worth it to save tons of time copying backup files, and increasing the reliability and responsiveness of my overall setup.
August 8, 2019