Update 2008-07-30 07:48am : But for a small amount of residual clean-up, the rolling restart is complete.
Update 2008-07-30 05:45am : The second-half rolling restart has begun. Today, all regions still on 1.22.4 will be restarted, upgrading them to 1.23.4.
Update 2008-07-29 09:08am : The first-half rolling restart is complete, but for a small amount of cleanup.
Update 2008-07-29 05:09am : The rolling restart of even numbered hosts has begun.
Update 2008-07-25 10:04pm : Those regions (about 1/10 of Second Life) that had been running server version 1.23.3 are now running server version 1.23.4. (90% of the grid remains on 1.22.4.)
Update 2008-07-25 03:32pm : We have found and corrected some bugs in the code. These were very rare crash modes (the crash rate on 1.23.3, even though a bit higher than 1.22.4, still corresponds to a mean time between server crashes of nearly a month). As such, they’re not easily reproducible, and we’re not 100% sure that the bugs found were responsible for the crashes we saw (although it’s likely). Our plan is to roll out 1.23.4 to the pilot group of hosts and regions currently running 1.23.3; that will happen tonight at 7:30 or 8:00PM. The full roll will happen early next week. The schedule below has been updated.
Update 2008-07-25 05:21am : We have decided to delay further rollout of 1.23.3, but we are not going to roll back the regions currently on 1.23.3. The simulator crash rate is not alarmingly higher, but we have seen a couple of new crash modes that we want to investigate before rolling out 1.23.3 further. We will update more as we know more.
Update 2008-07-24 01:22 : The rolling restart to deploy 1.23.3 to 1/10 of the grid is now complete.
We have found and fixed the bug that was causing an increased crash rate in Server version 1.23.2, the version that was initially rolled out earlier this week.
We will be doing a pilot roll to about 1/10 of the grid very shortly. We will watch the crash rate of servers, and look for other problems, throughout the rest of the day. If all goes well, we will deploy 1.23.3 to the rest of the grid according to the schedule:
- Fri evening 07/25, 7:30-9:30PM : the 1/10 of the grid running 1.23.3 will be updated to 1.23.4.
- Tue morning 07/29, 5-10AM : one half of the grid;
- Wed morning 07/30, 5-10AM : the rest of the grid.
As with all rolling restarts, regions will receive warnings starting five minutes before they restart. Regions will only be down for 5-10 minutes. If your region stays down for more than 20 minutes, please contact support.