Goodness, that was a doozy wasn't it? Up, down, up, down, then rollback... What on earth happened?
At 8:54pm server time (pacific timezone) the portion of the server that stores record of activities like item loots, item trades, quest completions crashed.
At 9:03 users reported issues, and after some brief troubleshooting the sysadmins were notified at 9:11. It looked like loginserver as well as itemshop/houses/mailbox/AH were nonfunctional.
At 9:42 the servers were rebooted and came online at 9:56. When this did not fix all the problems, more troubleshooting happened.
At 10:01 the servers went down again, for a more lengthy fix. Fortunately, the server keeps regular backups of item and quest interactions, and the information up to 8:52pm (most recent backup) was restored. Of course, the backups stop being recorded when that process crashes.
At 10:44 the servers were available for play again. Item shop functions like Depot and Advent Calendar missing.
At 5:19 Sysadmin was notified of missing shop functions, which he still works on now.
To sum up: All info between 8:52pm server time and 10:01pm server time has been effectively rolled back. Since the servers were offline for much of that, and since a recent backup was available, we expect total loss to be relatively small. If, for example, you were questing please recomplete them as normal. If you lost any items that cannot easily be replaced (like, say, an OD weapon from a high-leveled instance) due to server issues, please submit a ticket. However...
Since our searchable logs are stored in the same portion of the server that crashed, we expect some data loss there as well. We can and will research your tickets as best we can. It's likely that the CoMas will want to review the situation before the GMs close any related tickets. Please remain calm while we clean up.
