« Back to home

Perils of automatic invisible software updates

A couple of days ago I noticed my Google Fiber uplink was only 100Mbps instead of 1Gbps. While debugging with the tech via phone, it seemed as though any time I had something connected to port 4 on the network box, the speed dropped. I moved cables around until the speed was back to normal, and they scheduled a tech to come swap out the box.

Today the tech arrived and was unable to reproduce the problem. Even with port 4 connected to the switch — the highest traffic network segment — everything was normal.

Finally, he checked the network maintenance logs. It turned out that I just happened to run a speed test during the exact time period when Google was pushing out a firmware update to my network box. It probably dropped the link to 100Mbps while it was flashing the firmware and rebooting. The fiber UI is very barebones and doesn’t show that information at all, so I had no way to know that.

Making details invisible to users is great, until the moment when things don’t work as expected.