[Update 1:31PM PST 08/11/07]
Late yesterday we pushed out a network configuration update to the affected hosts which appeared to resolve the issue. Initial data looks promising, but we’ll be continuing to monitor and investigate.
[Update 4:26PM PST 08/10/07]
Following the additional tests, we have concluded that code initially suspected to be at fault was not the root cause and so we will not be making any visible progress on this issue until Monday. To be clear, this does not mean that work on this issue has ceased (in fact, the opposite is true), it merely means that no patch or other fix is likely until next week. Several options in addition to the already written patch are also being researched in parallel, and it is looking like this issue may be the result of a network configuration problem rather than a “bug”. We will update again on Monday or as more information becomes available. Thank you for your patience.
[Update 1:39PM PST 08/10/07 ]
The fix is now being looked at by both development and our QA staff as we get deeper into testing it. Depending on the result of these tests, we may do a limited push of the patch later this afternoon before a full grid-wide deployment. Fortunately, as noted before, this patch will not require a rolling restart so the actual patch deployment shouldn’t cause further interruptions. More updates to come as the testing results come back from QA.
[Update 11:28AM PST 08/10/07]
Our development staff continues to test the potential fix, but does not yet have an ETA for grid-wide implementation. We will continue to post updates as we have them. Thank you for your continued patience.
[Update 9:50AM PST 08/10/07]
Last night, our development staff rolled a potential fix out to a small number of simulators for testing. They will be evaluating the results this morning in consideration of a grid-wide roll-out. Once this evaluation is completed and a decision has been made, we will post further updates.
[Update 6:48PM PST 08/09/07]
The development team is still hard at work on this. We’ll post significant developments or resolution if they occur tonight; otherwise, we’ll check in tomorrow morning.
[Update 5:50PM PST 08/09/07]
Work continues into the evening. Thank you for your patience.
[Update 4:30PM PST 08/09/07]
Our development team continues to work on this issue, but we don’t have any further updates at this time. Thank you for your patience.
[Update 3:30PM PST 08/09/07]
While our developers are still working on a fix for the issue, we do have one small update. It appears that the fix will not require a rolling restart. We will continue to update as we get more information. Thank you.
Just before 2pm our Support staff began hearing from a number Residents about various Secondlife service issues in particular regions. It appears that these issues may have been going on for a few days, but were masked by the network outages we have been working through.
The issue can cause a number of different problems such as: an inability to search or use the map, failure to rez, failures while attaching, L$ balance not being displayed and odd IM behavior. Because there is the possibility of content loss, we encourage you to verify that Search works in the region you are in before attempting to rez any objects.
Our development team has identified the issue and are currently working on a fix.