Anatomy of a crash, part III

The story goes on, this part just handles about 30 minutes. But hey – those minutes were intense and a real rollercoaster ride:

16.19 Mattias emerges from one of numerous telephone conferences with devs and suppliers during the day, to tell the whole team some really heart-breaking news on our Skype chat:

[2010-08-24 16:19:07] Mattias: OK, bad news:

We will not be up for another 24 hours. We are copying the data to our old disk system, but at this rate (since the disk check is going on at the same time), we won’t be finished for another 20 hours. And then we need to let ht.exe catch up.

There is no chance that we can delay the matches that long, so they will be played on schedule.

[2010-08-24 16:19:40] Peter: πŸ™

[2010-08-24 16:20:19] Mattias: There is a chance that we can abort the parity check. Our supplier is trying that out on a test system, to see if its possible. If so, we will be online later this evening.

[2010-08-24 16:23:42] Mattias: I suggest we inform the users about the possibility that we will be offline for another 24 hours and explain why.

[2010-08-24 16:24:09] Mattias: Also, we need to tell them that the matches will be played as scheduled.

[2010-08-24 16:24:31] Niclas: this is indeed bad news.To say the least

[2010-08-24 16:25:59] Anne: So something along the lines of We might have to keep the site offline for another 24h. We are working really hard to find a solution to be up and running before then. Unfortunately, matches scheduled during those 24h will be played.

From having been a crash that we could, in a sense, live with, we now have to face the consequences of a real disaster, which threatens to leave the site closed through in effect the entire Cup and friendlies round, affecting every single user in the game. It’s just really heavy to digest. I think at this moment all of the more community-faced members of the team silently pencilled in a rough few weeks in their mental calendar. I know I did.

And then:

[2010-08-24 16:35:27] Mattias: stop

[2010-08-24 16:36:28] Mattias: We have just aborted parity check on the youth disks and we increased performance by 8 times.

[2010-08-24 16:36:41] Mattias: So we will probably up later tonight.

This still meant we would miss the start of the cup games, but it was a big relief that it looked possible that we could be up again tonight. But we still couldn’t be sure what we would get out of the system once we reopened it.