[2:30 PM –Chiyo] It is taking a while for the system to recover from the stress of today’s earlier problems. That means for the next few hours you may experience intermittent problems with teleports, rezzing objects, scripts, transactions, appearance etc.
This also affects many regions that are still offline and have not restarted as of yet. The current stressload on the database from everyone’s return is delaying this process. Please be patient as things work themselves out to return to normalcy. This process may seem slow at the moment but it is progressing as fast as we can make it happen. Thanks : )
[12:55 PM –teeple] We’re open. The outage has been successfully addressed by our service provider. Traffic has been diverted around a faulty router within a major internet carrier’s facility. Please be patient logging in for the next few minutes until the initial surge of logins has been processed. If possible, ride the login process out, rather than quitting and restarting, in order to keep your places in the login queue.
Some regions are still being returned to service, and Operations and Concierge will be working with those as quickly as possible.
[12:06 PM –teeple] Our upstream provider is continuing work to resolve the outage.
[11:01 AM –teeple] As Jack remarked earlier, this is an extraordinarily prolonged and unusual disruption. Many high-profile services and corporations are feeling the pain this morning, and full scale efforts are underway to isolate and fix the root cause.
[10:11 AM] We are still trying to put our fingers down on the cause for the problem, please bear with us a while longer! We apologize again for this interruption of your weekend plans!
[09:07 AM] At the present time, in order to resolve the current operational state of Second Life, we will need to force log out all users currently still logged in. We apologize for the inconvenience and hope to have the issues resolved as soon as possible. Updates will be posted here. -Lotte
[08:30 AM] Our Ops team is still working with the technicians from our ISP to fix the networking problems. Stay tuned for updates! -Lotte
[07:30AM 04/05/08 REOPENED] It seems the problem is even harder to nail down than we earlier supposed. We need to close logins again to go back to diagnostics. -Lotte
[3.30 AM] [RESOLVED] – The doors are open and we are back. It turned out there were multiple issues with our ISPs network which have been worked around for now. The network provider is working to resolve the issue permanently – Matthew
[2.30 AM] Folks, it’s been a rough night so far and we apologise for the lack of service right now.
Like most network providers today, the ISPs who provide bandwidth for Second Life actually bundle many network links into large virtual links between their data centers. Traffic for many customers then flows across these bundles. Starting at approximately 19:30 PST, some of the special routing that handles aggregation of these bundles at the ISP level malfunctioned causing us severe packet loss of over 50% on the portion of traffic going to Linden Lab’s data centers. We had no option but to disable logins.
Since this is a highly unusual failure, it’s been complex and time consuming to diagnose. We are working with our network provider’s engineers towards a solution and will be back up just as soon as we can.
— Jack Linden
Resident logins have been disabled until our toplevel routing issues have been resolved.