Update 2008-05-15 09:50 : Today’s rolling restart is complete. There are currently about 100-150 regions down; we are working on getting them back up right now.
Update 2008-05-15 09:05 : Today’s rolling restart is about 81% complete.
Update 2008-05-15 08:27 : Today’s rolling restart is about 62% complete.
Update 2008-05-15 06:59 : Today’s rolling restart is about 22% complete.
Update 2008-05-15 06:30 : Please note that regions restarted during this rolling restart may be down for as long as 30 minutes. This is because of recent changes to how regions are assigned to hosts. We will try to mitigate this downtime in the future, but for the current restart, please do not panic if your region does not come back as fast as it has in previous rolling restarts.
Update 2008-05-15 06:01 : The second half of the rolling restart has begun.
Update 2008-05-14 09:42 : The first-half rolling restart is complete. There are about 150 regions still to start up after the rolling restart; those should be complete within 10 minutes or so. The rolling restart to the rest of the grid will go forward tomorrow morning at 5AM as planned.
Update 2008-05-14 08:35 : The rolling restart is 80% complete.
Update 2008-05-14 07:27 : The rolling restart is 49% complete.
Update 2008-05-14 06:15 : The rolling restart is about 17% complete. We’ve made some changes to the system infrastructure over the last few weeks. These changes affect how hosts are assigned regions to run, and have reduced the load on our central database. Indeed, our central database has been humming along fairly well recently — not completely without trouble, but doing far better than it did in April. Alas, the asset cluster seems to have figured out that things were working better there, and has jumped in to bite us. One side affect of these changes is that regions are taking longer to start back up during the rolling restart than they did in previous rolling restarts. As such, if your region goes down, don’t panic if it doesn’t come back in 10 minutes. However, if it stays down more than 30 minutes, please contact support.
Update 2008-05-14 05:26 : The rolling restart to the first half of the grid will begin momentarily. I will post “% complete” updates hourly.
Update 2008-05-13 21:42 : I’ve temporarily closed off comments on this blog post so that the 150-comment limit isn’t used up with questions about current asset server problems; I want to save room in the comment thread for comments relevant to this topic.
Update 2008-05-13 20:06 : the 1-rack roll is finished. Updated regions are on version 220.127.116.11495, non-updated regions are on version 18.104.22.168182.
We will be rolling out a deploy of an updated version of the Second Life server over the course of the next three days. This patched version of 1.21 includes some high-priority bug fixes; these include a fix for repeated IM-to-email messaging on login due to corrupted data, as well as a few other crucial security fixes. This is not a major upgrade of the server, and in any event no new client downloads will be needed because of it.
The rolling restart will proceed in three stages.
- Tuesday, 05/13, 7:30PM : we will do a pilot rolling restart to about 150 regions
- Wednesday, 05/14, 5:00-9:00AM : we will do a rolling restart of half of the grid
- Thursday, 05/15, 5:00-9:00AM : we will do a rolling restart of the rest of the grid
As with all rolling restarts, regions should
only be down for a few minutes for up to 30 minutes. If your region stays down for more than 20 45 minutes, it almost certainly means something went amiss; in that case, please contact support. There is no easy way to predict exactly when a given region will be restarted, and it will not be possible to delay the restart once it has been announced for your region.