Let’s talk switch OS’s

The summer/fall update post about the state of the homelab is coming up soon, but I’m already well into the fall season and I wanted to share an initial post on some thoughts about open-source network operating systems, particularly for network switches.

I recently purchased an Edgecore/Facebook Wedge100–a 32 port, 100GbE switch for my homelab and to play around with some relatively current-gen hardware. It’s the BF-32X variant, meaning that while it’s based on the same open source design from Facebook, it uses the Barefoot Tofino ASIC for 3.2 Tbps of forwarding capacity.

Because it’s a commodity ToR switch, and because I bought it used, this also means it doesn’t come with any support contract. I expected an uphill battle playing around with getting the thing to work, but I severely overestimated the state of open-source network operating systems. It came preinstalled with ONIE (Open Network Install Environment) that’s advertised to help you install any NOS available for your platform. I was expecting to be able to use Facebook’s Open Switching System (FBOSS) on top of ONF’s Open Network Linux, but was met with limited, failing support for Barefoot’s Tofino.

Instead, I opted to install Microsoft’s SONiC. Documentation is *severely** limited, and found the overall install process to be very unintuitive–including requiring troubleshooting on my part and setting up my own HTTP download server for ONIE to discover the .img properly. GitHub issues aren’t always addressed, even when there’s a common issue among users. I guess this makes sense, as SONiC and commodity switch hardware isn’t necessarily meant for a homelab, and instead is meant for a team of network engineers in a large enterprise. The adage, “you get what you pay for,” still rings true here–I paid just over $50 per 100GbE port, whereas competing products go for well over $1000 per 100 GbE port.

Still, for those saying that SONiC and FBOSS will replace EOS in large part, SONiC has a long way to go to replicate the experience and support of EOS and Arista.