[UPDATE 9:20 PM PST] The rolling restart is complete, including cleanup of crashed regions. Thank you for your patience. In addition, we have collected some excellent data from this restart that should make the next one even less intrusive – ensuring that regions are not kept down for extended periods even if something goes awry.
[UPDATE 8:25 PM PST] Rolling Restarts are 90% completed.
[UPDATE 7:25 PM PST] Rolling Restarts are 75% completed.
[UPDATE 6:51 PM PST] Rolling Restarts are 60% completed.
[UPDATE 5:49 PM PST] Rolling Restarts are 40% completed. – Kate
[UPDATE 3:15pm PST] We’re about 20% of the way through. One of the central databases was looking a bit unhappy, so we slowed the restart down somewhat from the last time. The restart team is taking the opportunity to improve the restart process as it goes, with a team of engineers watching various systems and trying to reduce the amount of post-restart cleanup that will be necessary. A shout out to Prospero Linden, CG Linden, and Sabin Linden.
[UPDATE 1:45pm PST]
The restart has been in progress for several minutes now. 5% done already!
[UPDATE 1:15pm PST]
We’ve verified the new fix in place, and will begin the rolling restart shortly in the next few minutes. While QA was verifying the change, the release team worked to combine the “push the updated code to all servers” into the rolling restart process itself to save time so we can start the process without waiting for several hours. (We’ve been meaning to do that for a while anyway.)
[UPDATE 11:48am PST]
Final verification of the code turned up a bug blocking the release. Fortunately, the fix was easy to identify (it had been made in another “branch” and just needed to be ported.) We’ll need to re-verify with the fix in place, which will take about an hour, and then comes the time consuming bit – pushing the updated code out to all of the servers that make up Second Life, a process that takes several hours. The restart can’t begin until this is complete. We’ll see if we can’t hasten this a little bit, though.
Also: the “fix related to age verification” is to correct an issue where verified accounts are incorrectly blocked from parcels that require verification, after a teleport.
An updated ETA on the restart will be posted when available.
[UPDATE 11:04 a.m.]
The rolling restart will start slightly after 11 a.m. Pacific.
[Originally posted December 29, 2007 @ 3:43PM PST]
To push out server-side fixes related to age verification and a few causes of sim crashes, we will be performing a rolling restart of the Second Life servers on Wednesday, January 2nd 2008. Logins will be unaffected, and no new viewers will be required.
The restarts will begin between 10 and 11am Pacific, and take about 4 hours to update all simulators, followed by some cleanup work. Each region is given a 5 minute warning, then restarted (which takes approximately 5 minutes). The restarts will be on a simulator-by-simulator basis rather than region-by-region, so it will not be possible to postpone restarts for a given region and the restarts will occur in “random” order relative to the map.