As I’ve written before, date and time calculations are surprisingly difficult. What’s more, even the bits that aren’t difficult often get messed up. Sony have just demonstrated this.
Apparently, a firmware bug in the real time clock in some models of PS3 lead the RTC to think that 2010 is a leap year. This confused the PS3’s OS, which was then unable to connect to PlayStation Network. This, in turn, caused havok with all kinds of games, even single player games with no network play, because PS3 trophy support requires network connectivity. It also disabled all DRM-protected content. Sony advised people not to use their PS3 until engineers could find a fix, as content might get corrupted. In the event, the ‘fix’ was to wait until the next day, when the RTC moved to a valid (but incorrect) date, and the OS would once again recognize it.
I happened to be using my PS3 when the date hit March 1st 2010 UTC (or February 29th 2010 PS3 time, you might say). I was playing “inFAMOUS”, a game by Sucker Punch. I got an error message saying I’d been logged out of PSN; a little while later I got a trophy sync error. I finished the game that evening, so I qualified for more trophies, but of course I won’t have gotten them. I didn’t particularly care about the trophies, and PSN failures aren’t exactly unheard of, so I thought no more of the issue.
Not all games are as well-written as Sucker Punch’s, however. Many people reported that “Heavy Rain” crashed on startup; apparently it assumes that if the network is enabled, the PSN trophy server must be reachable. “Flower” also died.
There are a few observations I’d make about this whole situation:
Firstly, yes, it’s a stupid error to find in code. However, if you read RISKS Digest you’ll see the same error happening in a big system every few years.
Secondly, the scale of the failure was undoubtedly due to the system brittleness required for DRM. The existence of time-limited rentals requires that the DRM system not trust the system date, and fail to authenticate if the date appears incorrect. This means that even indefinite rentals (which are dishonestly called “purchases”) ended up with copy protection errors when you tried to run them; people reported that none of their downloaded games worked.
Thirdly, this kind of slapdash QA is all too typical of the video game industry nowadays. My impression is that the quality of game software is getting worse, now that consoles are assumed to have an Internet connection. It used to be expensive to recall and replace faulty discs, so games spent a long time in QA; now, the developers will just throw a faulty game out so long as it’s more or less playable, and plan to fix it with a point release. Games even get released to consoles with sound that doesn’t work properly (“The Orange Box” for PS3, for example). Sometimes a patch eventually fixes the problems, sometimes it doesn’t.