Anatomy of a crash, part V (la grande finale)

Early Wednesday morning Daniel tries out an idea for forums. This idea means the forum most likely will be slower than usual, but at least it’s something. We open up the forums around 7.30, to see if the speed is sufficient and not just frustrating to users.

It turns out forum speed is so-so, but perhaps it could do. However, we also know that the load from users will at least be doubled during the day and it may affect forum speed negatively. Daniel comes up with a new idea to try out, and we close the forums again at 08.00 in order to try it out. About half an hour later the forums are back up again, now working even better.

Having the forums online again felt really comforting, not only because we know how much they mean to our users – but it also meant we could start communicating for real again. And for us “typers”, after two long frustrating days without really being able to do that much I have to say it felt really good to be able to roll up the sleeves and start working for real.

One thing had to remain offline though, the youth academies. The youth applications were up and running on our normal disks, trying to catch up. But with the parity check running (again) and the forums being on the same disks, it was at this time impossible to predict how long time that catching up would take, and then we also have to find out if it’s possible to open up the academies at the same time as the parity check was running. We think it will be, but one can’t be sure until we’ve tested it. In two hours, around 11 o’clock we hope to be able to make a prediction about the catching up at least.

Two hours later, after sorting out some issues with shop purchases made during the “missing” 30 minutes on Monday, we get a prediction about youth: It still has a lot of catching up left to do, it’s still some 19 hours behind schedule. With a bit of luck it could mean we could be up in the evening, but a more realistic estimation is that youth would need another day offline. Each and everyone of us, I think, takes a deep breath. That was not what we were hoping for.

We have no other choice to communicate that youth will probably be down until noon Thursday, and prepare for getting tomatoes thrown at us in the forums. It’s never pleasing to be the messenger of bad news. Five hours later, at 16:00 we get some good news at least. The youth engine is now only one hour behind and we can try to open the academies again – luck is perhaps on our side?

Only a few minutes later we can note that it was only a five minutes moment of luck. The disk performance is not enough, so we have to close youth again. All we can do now is to wait for the parity check to complete. On Thursday morning we can only establish that it will not be completed to noon, it needs another 24 hours. Finally, Friday morning at 7 we can open up the academies again, and we can get some rest after a long exhausting week.

This is where the story ends. But as you can figure out, there is still a chapter left in this case. We still got one disk in need of a parity check, and then we also need to move the main game back to the normal disks again. Exactly when this happens will be announced on My Hattrick. However, this “anatomy” ends here. Writing it has been a lot more pleasurable than experiencing it, that’s for sure.