Region Data Corruption / Loss Problem

We have started to receive reports of regions losing many of their small objects after the region crashed. This was determined to be caused by an Internet Networking hardware problem between our data centers and one of our service providers. We are already in contact with the service provider and we hope to see this resolved quickly.

Should you see massive object loss in one of your regions, please submit a support ticket and provide the name of the region.

This entry was posted in -Miscellaneous, Customer Service, Resident Experience, Service Status. Bookmark the permalink.

15 Responses to Region Data Corruption / Loss Problem

  1. Yes, there is a problem keeping linked flying objects linked after sim crossings (^_^) I noticed this in the New open water protected sims east of Third Space.
    I do thank you so very much for adding the new open protected land and water in SL Thats So Very Awesome! ,,,=^_^=,,,
    Great job!

  2. SeanMcPherson Senior says:

    Can anyone from LL describe the sequence of operations that takes place between a client, a sim/server, and the database where the objects reside, and how objects would just ‘disappear’ after a crash? Are the objects expected to reappear after the network issue is corrected? Also, the blog references massive object loss for a ticket; if a small number of objects is missing, and if it is anticipated they’ll not return, is that to be considered ‘acceptable losses’ that aren’t recoverable?

    Not hammering on LL, but I am looking to understand what might take place to threaten all the objects that it is quite simply impossible to backup legally with the current system, and what changes LL might be making to the architecture going forward to resolve these issues.


  3. Kate Linden says:

    @seanmcpherson Thank you for your topic related comment and it is a good question. Network corruption occurs between the simulator and asset server, causing some simstates to be unusable, and some to come up garbled. When they come up garbled, huge numbers of objects are affected.

    Poppy Linden has a wiki page which answers your question more in depth:

    Kate Linden

  4. Puppet Shepherd says:

    Does this problem cause objects to become unlinked and some of the prims to get deleted, but not the entire object? I lost some dolphins this way last week – the root prim remained, but the rest of the dolphins – the other prims – vanished completely.

  5. SL Player™ says:

    That’s why I only rez my fully furnished house only when I’m there. I have way to many valuable things to lose and it keeps people out to. =)
    I’ve had a friend place a cube under my house as a reference point to know exactly where to place my cursor to rerez it all in one piece. So when I chose ‘Select Only My Objects’ the cube never gets taken.

  6. Amtor Wopat says:

    Nice to peek into the technical background to problems – I don’t envy you tracking down this one :-))

    Try and get into the habit of posting explanations – some of us are interested in how it all works, or doesn’t …

  7. Tegg B says:

    You realise you have more chance losing stuff rezzing it and derezzing daily it than leaving it stable. The more often data moves about the more chance of it corrupting, you may be increasing your chances of losing something by more than 300 times, rezzing and derezzing daily compared to leaving something worth $10 US sit for a year.

  8. Kim Trefusis says:

    As with alll “Bad news, good news” situations, it is best to appreciate the “Good News” and handle the “Bad News” the best way we can.

    My good news was from Tegg B saying that keeping stuff over a long period was a lot better than rezzing and derezzing. I have a lot that I need to keep inworld all the time and really have no choice.

    The other good news is to see this openness of LL to talk to us at a shared level, so that there just may be fewer posts complaining about being left out of the loop. For me, SL is not a game but it does play an important social and commercial role on my life and will do so increasingly in the future. So here’s to a closer and longer lasting relationship between LL and the SL residents. By the way, I was sorry to hear about your dolphins, Puppet. I hope you can replace them, soon.

  9. Very Keynes says:

    Interesting explanation, and I am sure that a lot of time, effort and cost went into identifying the ISP’s problem for them. “This enabled our ISP to track down specifically which node on which router in the path was failing. ”

    But as it is happening again, only a month later, I would suggest that it is time your ISP got a new hardware supplier or you got a new ISP. It is, after all your reputation on the line, and I would imagine with you are, with your traffic, one of their larger customers.

  10. Jessica Hultcrantz says:

    Thanks for the link to Poppy’s great technical explanation, more of that please!
    It help sunderstanding the real problems instead of getting angry over things working without any useful information on the blog.
    Please, share the info for everyones best!

    And besides that… The Dallas location messing again? That one is and has been the worst to connect to of all locations regarding packet loss and loading problems for a long time now.

    They probably have more bad routers for the international trafic :-b

  11. Ryu Darragh says:

    #10 Very: LL has no real choice as to the ISP they use. They can’t use dialup or cable for SL so they connect to the tier 2 or tier 1 providers using fiber or ethernet cable. At that level, you find that ISPs all use the same cabling to connect to the backbone of the internet the same way multiple “competing” power providers use the same set of wires to provide power to homes. They could switch ISPs and still wind up using the same hardware they were before. Even if the went through the astronomical cost of having new fibers and /or ethernet cables put in to yet another ISP, they would probably end up being carried by the same provider as before to get to the backbone of the internet.

  12. SeanMcPherson Senior says:

    Thanks Kate! Networking and databases are a large part of what I’ve done as a living, and it’s always good to share an understanding of the events. You never know, one of the various Avatars might have a great idea that helps the LL staff. And if nothing else, when people see there’s an actual explanation for the issue, it really helps them avoid feeling like “Well, we had another issue, are they just saying it’s something technical, when they’re just spilling sodas into the network rack during a game of lawn darts in the data center?” Obviously, that’s not the case w/ LL, but it’s always helpful to ‘user morale’ to get solid answers.

  13. Duckie Dickins says:

    #5: I have noticed this behavior with sculpted prims and a few normal prims. Were your dolphins sculpted? I had 2 identical flower vases with a sculpted stemmed flower. One of them the flower disappeared but the stem remained. The fun part is the prim count on both copies is the same. I have also had a security orb which the prim was no modify (not just the scripts) change from sphere shape to a flat cylinder. Might want to check the jira for any open issues regarding this.

  14. Andrew says:

    I have faced the problem of data loss. Recently I have lost my data from windows. I have used Stellar Phoenix Windows data recovery software to recover my data. It is very easy to use and provides good data recovery solution. You can check this software from the site

Comments are closed.