More info on this week’s grid problems.

For the curious, I want to shed more light on the causes of the stability problems Second Life has experienced over the past four days. First, some background: the two facilities which currently make up the grid use encrypted IPSec tunnels (a form of VPN) to securely communicate with each other over the Internet. This is what allows, say, a simulator in Dallas to query our databases in San Francisco. Needless to say, if these tunnels don’t work, SL doesn’t work very well either.

On Wednesday, we experienced a failure wherein our primary bandwidth provider started dropping about 95% of our IPSec traffic bound from one site to another, but only for some of the tunnels. Other, non-IPSec, traffic was largely unaffected, and so this was a very difficult problem to diagnose and, crucially, get the provider to fix. While waiting for that, we started moving banks of simulators onto unimpaired tunnels. Once a good percentage of the SL was back up, we enabled logins, but didn’t get all the sims back up until the provider fixed the problem.

At 3AM on Thursday, we had trouble again and two failures actually overlapped. Our provider started dropping traffic again, with a different set of tunnels affected, and, around the same time, our largest tunnel failed entirely. This is apparently due to a bug in the router model we’re using, which the vendor believed was fixed but which, it seems, wasn’t really fixed. I still don’t have all the answers on that yet, but we’re now in the process of taking those routers out of service. It was several hours before the bandwidth provider fixed the traffic problem and we were able to get the failed routers back online.

While all of this has been going on, and still today, a serious bug in the Second Life software is causing 5-minute lock-ups of the communication channel between simulators and our various databases. This is a difficult bug to describe, but essentially, we see a very tiny percentage of database queries stall indefinitely after returning their data. Our software notices after five minutes and re-starts the database client, but by then some number of queries have usually been lost. This causes affected sims to occasionally (or in a few cases regularly) experience inventory problems, search problems, transaction problems, the works. Without any code changes, the bug seems to have gotten more common over the past week; on Friday evening the engineers working on the problem identified a potential fix, which we’ll be testing over the weekend.

Finally, there was an unrelated problem Friday night: a disk drive in one of our databases failed in a peculiar way, which caused it to stay in service but perform far too slowly to keep up with load. (Normally drives which fail are automatically shut down by our RAID systems.) Logins and some residents’ inventory remained unreliable while we diagnosed the problem.

The end result is a pretty rough week for Second Life Residents, and I can only offer apologies for that. The good news is that we have many projects in the works which will help improve the reliability of SL. We’re working on new infrastructure which will help us withstand the failures above, as well as store and transmit data more reliably, and we’re working on reducing both client and server crashes. Several of these projects are new, thanks to our growing technical team. We’ll share more information on these efforts in the near future.

This entry was posted in Operations, Service Status. Bookmark the permalink.

164 Responses to More info on this week’s grid problems.

  1. Tmayla Rotaru says:

    Thank you for trying to keep us in the loop on what’s happenening ‘behind the scenes’. I think you all are doing a remarkable job when I consider the shear volumn of data you’re dealing with in real time environment.

  2. Cargo Tomsen says:

    Thanks for the details. Reassuring to hear that it just seems to have been a bad week rather than some deep rooted problem. Fingers crossed that normal service had now been resumed…

  3. Hank Mcalughlin says:

    Glad you found the problem. I, and others, I’m sure, will be eagerly awaiting the fix.

    One question, though. Will this fix the strange vehicle behavior? Because when I’m riding a vehicle, if it gets up to too high of a speed or goes into just the wrong area(Not an unreasonable speed or an area I’m not allowed in), it spins out of control, crashes into(Read:Through) the ground, or just starts accelerating out control until I relog, and I’m having problems getting anywhere, or even finding places where I can rez another vehicle. So, will the fixes address that?

  4. Miche Rainbow says:

    Does explain a lot….would like to know which version this bug was introduced in, pertaining to the bug with the DB queries.

    All in all, i think this explanation makes a lot of sense, holds together with the facts on hand…..keep up the good work on working around the unlucky problems and everything, and i hope the fix that’s being worked on over the weekend works ^.^

  5. Amanda Ascot says:

    Thank you so much, Ian. I was really hoping that you guys would be able to track down these issues, quickly, but I don’t live on hope. I know full well how complex this system is, and things can go wrong in ways that overlap and interact to make it very difficult to fix.

    For the most part, SL seems to be behaving as well as it has in a long time, but I’m sure you’ll get the usual rash of complaints. I, for one, am happy that the Grid is as least functional, again.

  6. Well atleast someone sheds the damned light on problems AINT THAT GREAT!!!!! W00t W00t!!!!

  7. Kiwoo Gandini says:

    First time bothering posting in a blog.

    What a week its been, loss of inventory, loss of L$, inability to log in. At this time im getting the message “Login failed. Unable to connect to a simulator” So, for me the log in issues arent fixed.

    The problems over the last week are recurring problems we have all seen countless times before, normally they rear there ugly head again after a major patch.

    So will this “fix” thats being tested this weekend, be a fix. Or just another bandaid that will fall off in short time.

    My faith in LL is as stable as the grid.

  8. BW jinxing says:

    time will tell…time will tell.

  9. Skyler Goode says:

    Wow – It’s great to get some info on this! It HAS been a tough week, but it’s great to know someone out there is working on it. Way to go and keep us posted.

  10. Cat Gisel says:

    Hang in there ya’ll, Some of this info is over my head, but I know you are trying 🙂

  11. Zandor Maltese says:

    thank you Ian for letting us know the real scoop and treating us like adults.

    all though i hate the down time i can understand and emphasize with your plight.

    again thank you for the real scoop.

  12. Shidji Woollahra says:

    Explanations like this are needed to keep people (myself included) from jumping to conclusions. It’s good to hear such a thorough explanation… helps to reduce the rather large and jaded opinion pool out there that consists mostly of you guys not knowing what you’re doing.

  13. Bobo Decosta says:

    So what are you doing actually of me getting back to the AV is was before? Since your latest grid problem that so said was fixed made me end up nude not able to edit my appearance while my clothes are loading. The are downloading whole day now. Still no fix! Instead of portraying like all is well! fix your grid and let me please enjoy something i paid for. I didn’t pay to walk like a nude grandma. I can’t talk to my customers like that and can’t open my new club looking like that!

    Clear cache, ethernet connection, reinstalling second life, restarting the region still leaves me naked!

    When will we get compensated for all this? I pay monthly 295 USD to end up in all kind of problems just because you can’t manage to provide what you want the whole world to believe you deliver.

    Why did i till know never received a version of the SL client that would just work fine? Why is that? I can imagine some versions end up bad, but all?

  14. Avril Sodwind says:

    no the truth is that this voice shit creates more lag and is ruining everones fun, make a new viewer 1 with voice 1 without, that both are live… i cant stand this, you took off the votes stuff that i put in about 5k worth of lindens in on the votes and now you FORCE everyone to have this voice crap on their viewer ALOT of people do not like the idea of voice

  15. Wyald Woolley says:

    Ohmygawd, as you were describing what had gome on over the last few day I couldn’t help wondering if you weren’t being given a sign from GOD that you should stop trying to do his job and stop being the morals police for SecondLife.

    Yeah, His thumb may be hoovering over the BIG RED BUTTON right now…ready to Zzzapp the crap out of something else until you get the message. I suggest you start some good clean living and offer up a sacrifice of some kind, like maybe giving up VOICE or paying back the citizens for all their lost inventory. Remember, it ain’t a sacrifice unless it hurts a little.

  16. Solomon says:

    One very, fundamental, basic change would help a lot… make the bloomin’ database transactions frikkin’ *ATOMIC*. It will prevent lost inventory from failure to rez, PERIOD…

  17. Bobo Decosta says:

    This is just bad news! Again new stuff while SL still works like crappy software!

  18. Marni Grut says:

    I like how everyone get new feature but looses all there stuff!

  19. notagambler says:

    “””””””””””which the vendor believed was fixed but which, it seems, wasn’t really fixed.””””””””””””

    from what i do know that ??

  20. Aeper Jie says:

    I belive Sl is to big and to advance for current Tech. or the internet is?

  21. Asmo Aya says:

    Well, as much as I detest LL and the pile of crap SL has become, I must say this is FINALLY a step in the right direction… Some HONESTY and OPEN COMMUNICATION instead of just taking people’s money and pretending it’s “none of their business” why things are in the state they are in. It doesn’t excuse the mess and it doesn’t make things any better for paying customers, but at LEAST it seems maybe some of the arrogance is fading… keep heading in that direction and I might even lift my curse…. 🙂

  22. Locke Tarveler says:

    Agree with Solomon. The Het-Grid is here; why are you still playing with Voice and WindLight? You can change the way the grid talks to itself and the asset server now! Why isn’t this happening? And if it is, PLEASE tell us that it is! You need to slow down the development of WindLight, and get some of the people working on that moved onto the problems. Once the asset server does exactly what it’s supposed to 99.99% of the time, then we can talk about pretty skies.

    Regarding the weeks problems; it sounds like your vendors and providers owe you a refund or two. Good luck keeping tabs on the external problems, and even better luck with fixing the internal ones.

    For all the Lindens and the Grid, Cheers.

  23. Ian, this is a great post with technical details as to the outages. It seems like only one of the issues, and not even one that caused the grid to close, resides in the realm of the “code” itself- everything else is vendors. In that, it’s a stroke of bad luck, but a good thing to know. I’m expecting a rolling restart or client update next week for that bug fix 😉

  24. simon sugita says:

    ty for keeping us residents up 2 date thumbs up and keep up great work !

  25. Lobo Faulkner says:

    This explanation is all well and good but maybe fixing and stabilizing the existing product before adding features that cause a drain on the system. We are loosing inventory that we can not recover loosing sales because items are not being delivered and there is no way for us the paying customers to recoup our loss due to system wide outages or shortfalls. Granted one or to inventory items can be tolerated but when over time you are loosing inventory that starts to amount to a real cost what si Second life going to do say sorry but we cant help. I appreciate the explanation but is steadily loosing confidence in the program as a whole especially considering the cost of owning a island and trying to operate several business ventures in world. Fix what you have already before introducing any new features that open the doors for more stress on the system. Hope this post doesn’t get deleted and your willing to listen to criticism as well and you so ready to take praise.

  26. Tom Bender says:

    Hey Guys… stop whining…

    Shall I come and mess with your tiny lil home use router?
    A lil falg… a lil click and you surf like 10 years ago on your phone line.
    Thank you LL for that great background info – that makes absoltely sense.
    Seriously… stop whining and complaining here – slowly it’s getting old.

    Add serious comments – serious critics which helps LL to find the bug.
    And not about how the casions went down the drain and camping chairs pay less.
    Hell I never used a camping chair in my whole SL.
    Just think:
    Power consuption of a PC – compared to what you get with camping…

    Hehe
    Did I mention that I love my Platic?

    Live long and prosper

  27. richard smith says:

    over the past 4 days?

  28. Brett Finsbury says:

    hey what ever it takes get it fixed. thanks for the heads up on what is going on.

  29. Dirk Felix says:

    This build and client is pure poop. Crashing all day. Im at crash #23 for the past two hours. Your CEO needs a reality check and engineering should be ashamed of this spaghetti code that is terminally flawed.

    Voice was a horrible idea and was implemented poorly. Most of those who wanted voice can get better results in using Skype and still active in-world. Please become a publically traded company so we can demand resignations. 😉

  30. Lex Neva says:

    Dear lord. Someone send these poor folks a beer 😮

  31. I can’t log in or update, im having to do it manually and still will not work. Is this causing the problem on my end, or am I just having a problem for the first time in 10 months?

  32. Clint Dibou says:

    Good work catching it. As a network egineer I can appreciate what you have been through…been there many times. Thanks for letting us know a few details, it helps many of your members understand what is happenig and enables them to see you are working on the issues.

  33. Nefertiti Nefarious says:

    And where is the Linux viewer for Voice? I see that Windows and Mac have avai;lable updates.

  34. Kiwoo Gandini says:

    Yes, I agree thyings do go wrong. Its a very complicated piece of software and hardware thats running the grid. The problem is none of the problems we have experienced this week are new, they have been ongoing for at least since i started SL 13 months ago.

    We can live with delays in additional features being added to SL. The problem is adding more bricks to a already shaky foundation will, at some point, bring the whole structure crashing down on all our heads.

    Get SL stable. Get it running as it should, then and only then add new eye candy.

  35. Anderson Philbin says:

    Thank you for explaining what has been going on. I realise a technical explanation goes over many people’s heads, but you do need to give details i.e. start with the Executive Summary and then give the real reason. Yesterday’s explanation of a “RAID failure” didn’t cut it with me. Today’s is much better.

  36. Maite McMahon says:

    One week… only one single week without any kind of log in problems, inventory loss, rezzing failures, grid downtime…*sighs* Wouldn’t that be nice…

  37. drako nagorski says:

    “The end result is a pretty rough week for Second Life Residents, and I can only offer apologies for that. The good news is that we have many projects in the works which will help improve the reliability of SL. We’re working on new infrastructure which will help us withstand the failures above, as well as store and transmit data more reliably, and we’re working on reducing both client and server crashes. Several of these projects are new, thanks to our growing technical team. We’ll share more information on these efforts in the near future.”

    in other words, LL is starting to listen. really…. we can go without voice or fancy skies, but what we cant go without is a reliable and fast SL.

  38. Sy Beck says:

    QUOTE: This is apparently due to a bug in the router model we’re using, which the vendor believed was fixed but which, it seems, wasn’t really fixed.

    Bummer aint it LL when people tell you they’ve fixed something and it just gets worse.

  39. Nefertiti Nefarious says:

    PS: My sympathies on the VPN tunnel problem. I have to use them occasionally when telecommuting and they can be erratic and hard to keep open.

  40. Machaer Janus says:

    Over 2000 servers, in 2 cities. Im surprised SL works at all. Thanks Lindens, for keeping us inworld!

  41. Rooke Ayres says:

    Ian,

    Thank you for letting us know what was going on and why. It doesn’t undo any of the problems it caused the user community, but at least it’s a step in the right direction by keeping is informed.

    Now we have some idea why things have been the way they were over the last 4 (more like 5) days.

    I hope that LL continues to be this informative in the future. It will go a long way in letting the users know that we’re not being treated like mushrooms – kept in the dark and fed a lot of bullstuff.

  42. Fledermaus Messmer says:

    As a former computer tech (hardware, discrete component level), I fully empathize and sympathize. Simultaneous issues are a bear to troubleshoot. Given what your crew were up against, I am amazed at how quickly and well you kept things running. Well done to all!!!!!

  43. Naz Fride says:

    It’s frustrating that we are having all these problems, but it makes it MUCH MUCH easier to tolerate when LL actually gives us updates and explanations, instead of just acting like it’s not happening. This post is exactly the kind of information we like (need, deserve, whatever) when there are problems like these. Keep it up and thanks.

  44. Cocoanut Koala says:

    So . . . when is all this mess going to be fixed?

    Do we have to wait for the “several projects in the works?”

    (And Voice does nothing for me but give me YUGE packet loss. I had to go back to the previous version in order to move in SL at all. Shouldn’t that tell you something?)

    coco

  45. richard smith says:

    i’m laughing #27

  46. Arthur Fermi says:

    Wouldn’t it make more sense with a mission critical application to not rely on the internet for transport? IPSec adds packet over head, and the internet is prone to slow down when big things happen.

    If I was looking for high speed bandwidth between co-locations i would be looking at multiple MPLS pipes. You can get gigabit speeds easily, and the cost is lower than frame relay on a per K basis. The other option would be to rent some of the dark fiber that is in the gorund and run direct.

    There is, with VPN any way to ensure capacity, reliabiltiy or QoS of your connections.

  47. Nimue Jewell says:

    Thanks so much for the update! Nice to hear why things have been bumpy lately and that fixes are in the works.

  48. Bobo Decosta says:

    @Tom Bender Why should we stop whining here? We pay for second life and demand service all we get is rubbish and be kept in the dark of what is going on!

    Maybe you should stop giving compliments here to people that don’t deserve it! Most subscriptions i have envolve much less money, yet the service is 10 times higher!

  49. Cat Cotton says:

    Good to know, and thanks for being honest and upfront about the problems many of us are currently experiencing. I can only hope that these issues will be a thing of the past ; in the very near future. I am glad you were upfront; as these issues have wavered my support of Second Life in recent months.

    Catherine Cotton aka Martin Magpie.

  50. Thanks for the details. I don’t understand a lick of it, but I’m sure you have someone that does. What I *do* get kind of explains why the animations in my eCollar seemed to suddenly stop working, though I had not moved from one sim to another.

    Keep cracking! I’d send donuts if I could.

  51. Matti Deigan says:

    Thank you for the insight, however i belive next time it might be better to explain it as its happening than after >_> many of us begun to suspect things

  52. I really appreciate the details about the problems and hope this will be a continuing trend. Communication seems to be lacking lately in these problems.

  53. Able Whitman says:

    Ian, thank you very much for a thorough and informative update. It really helps put things in perspective, and I appreciate all the effort you and the rest of the team are putting in this weekend to deal with these myriad problems. Good luck!

  54. Teeny Leviathan says:

    I barely understood the explanation, but it was nice to finally hear whats going on.

  55. Kept in the dark? How does posting a full run-down of this week’s problems and a plan for the near future equate to keeping us in the dark? I think that LL provides plenty of info about SL’s growing pains and problems, more than most companies would be willing to share.

    Thanks for the status update, LL. Keep up the good work! 🙂

  56. Lance Sismondi says:

    I got stuck in a VPN tunnel once during an earthquake. I can tell you that is no place to be during a big 6.6’er! I guess taking BART to SF from the east bay could be worse. Damn VPN tunnels.

  57. It is not 100% clear if it is related or not, but it seems over the course of the crashes my router got fried.
    It would have just been nice to have a note, that something goes wrong with logins. Like with the DNS server problems in Germany, that required to use an US DNS server: no note, nothing on the blog! (just some helpful people on the betagrid who new a solution)
    If all your friends are obviously online, but you can’t get in, you really think the problem is on your side!
    Now, it is good to know at least a bit of what was going on, or still is going on (even if it is difficult to understand). I am still not able to stay longer than 10 seconds in SL without getting frozen. But, then, my router half died, maybe because I was trying to hard to figure out what was going on. (So, SL friends, see you back on Wednesday. When I will hopefully get a replacement)

  58. Astarte says:

    Thank you Lindens for not just ignoring us and starting to give us some of the reasons why the probelms, it is much appreciated, and we can start to understand some of the problems you are coming up against trying to scale this world of ours for more people to use.

  59. The Baron says:

    It is certainly working better, and I find the explanation very useful, as I would otherwise put the previous problems down to my own equipment, network or ISP being intermittently flaky. I hate troubleshooting things that are working properly, you can never fix them.

  60. Arthure Zabblebeebles says:

    thanks for all the hard work! cinnabons for everybody!!!!

  61. Anonymous says:

    @Tom (are you on a ) Bender…

    Like some of us have posted here; we pay a LOT of $ to work in this environment (if that’s what you want to call it (work; failed TP’s, lost inventory, etc)). When I am expected to pay tier on land that NO one can get to to purchase my products, due to LL’s 20% downtime per month now, I and EVERYONE else that pays tier has EVERY right to complain;
    To ALL that say we are “whiners”; if you dont like reading the complaints here, dont load the page. THIS IS THE ONLY place LL seems to listen to the moral majority. I would much rather be called a “whiner” than a “whino” 😉

  62. Bo Rossen says:

    Well… let’s see if all of this is going to lead somewhere…

    And agreeing with a lot of folks around here.. get RID of the crappy voice feature… Skype is so much easier…

    While your at it… just make one thing for sure… get that asset server working like it’s supposed to pls pls pls… I still don’t understand why this has to be a problem all the time… MySQL is SOOO easy to use LOL…

    anyway… hope next weekend will be better *grins*

  63. Xylona Fride says:

    While the explaination is nice, my main concern is the money I’ve lost due to inventory loss. I doubt that SL will offer compensation, but someone might want to warn people faster in the event that it happens again. The fact that for several days I was in the dark about why I wasn’t getting my inventory items rezzed or even returned is a pill I don’t want to swallow. The fact that I am only left to ask for items back (some of which have still not returned) makes me even more bitter. When SL stops having it’s continual problems is when I’ll start paying for an account.

  64. xeta dix says:

    thanks for the details.

  65. Eon Peterson says:

    I’m hoping Ian’s post explains why some of my house is missing… walls, floors, whole rooms. This has never happened to me before in nearly a year of being in this house. Funny thing is though, that although I can’t see the floor in one place (just a drop to the ground below), I can walk on it as if I’m walking on air. Has anybody else experienced similar problems?

    I hope your fixing all the problems sorts this out too, it seems as if it will.

  66. Ann Otoole says:

    why even post a comment. they are ignored. the same symptoms of a cobbled together architecture are ever present. despite a huge objection to a certain feature it was shoved in and then the system goes brain dead for a week+ resulting in income losses for the residents.

    why even bother trying to be transparent if you cannot bring yourself to be transparent enough in real time to just tell everyone at logon that they cannot spend or build without major risk of loss?

    there isn’t much confidence left for linden research now. it will take a very long time to repair this sort of reputation damage.

    try listening to your customers instead of sneering at them.

  67. Mandi says:

    sorry for all the problems, but why is it when you get one thing fixed everythign else goes haywire, gawd if this was RL ppl would have already taken their business elsewhere, might i again suggest a suggestion box( s) in sl so that those of us who do play in world can offer suggestions ( not stupid shit either) that might just make it a pleasant experience for all
    * offers a toast* ” here is hoping for a better SL week next week!”

  68. Proc Shepherd says:

    Hi

    I work Worldwide Customer Care at AT&T for clients like SL. And yes we do have Service level issues from time to time. I don’t know if SL is our customer (I’ll look). But as a provider I acknowledge that providing this service to our customers is our highest priority. And when problems arise we will do everything in our power to resolve them.

    As a provider, I want everyone to know that we are concerned about how these issues affect all of you. Its not the revenue that counts. Its the benefits our customers enjoy. And a guarantee of that enjoyment.

    For me personally, I’ll take this one to heart. And I’ll make sure all of our customers get the service we promise to deliver to them. And thats you. Its a tall order. But an order that we agreed to fill.

  69. Miguelito says:

    Once again we have fundamental problems with this game and the users are divided into two factions: the blind Linden supporters, who seem to believe that LL can do no wrong, and the people yelling at LL to fix the problems before introducing new features and new bugs.

    Except you guys need to get something through your heads: The people who work at Linden Labs aren’t listening to you tell them what to do. They don’t care how much you pay. I knew someone who spent $2000 a month on this game. $24000 a year. When he asked LL for help, they shot him down and blamed his problem (a badly crashing sim that needed to be reset every day) on his own doing without even bothering to explore the idea that it might be a problem on their end.

    It’s a known issue that the money we spend goes to whatever they feel like working on–not what will improve the experience for its paying customers. Unless there is some dire emergency that requires intervention or the game will end up not working at all, they feel your money is much better spent on development rather than quality of the product. It’s time to accept that and either deal with the fact that the Linden staff have no souls or concept of customer service, or stop using the game (or at least stop spending money on it, like I did–Linden Labs will not get another penny of my money until they give me something worth paying for).

    I can only hope that Linden Labs will see the errors inherent in their business practices someday and will look at their game, which will by then not even have five-digit concurrency any more, and realize they made a mistake.

    I hope none of you are holding your breaths for that, though. It could take a long time.

  70. Alyson Tomsen says:

    Thanks for the heads up. It has indeed been a rough week and I want to thank you for taking the time to explain what happened. I don’t understand most of it, but I get the gist.

    And I agree with some of the other comments – fix the current problems, make the grid stable, THEN go for the voice and pretty skies.

  71. Ossified Ogre says:

    I, for one, welcome our tunnel masters to the 100th level of experience.

    Lindens, Thank you. A few forthright details can make a big problem easier to swallow. Thank you for those details.

  72. U M says:

    Again Dallas? Thank you for clearing reason. Its helps understand alittle more then if not explained.
    But are we going to get replied in anyway?

  73. Shimmer Ellis says:

    “Thank you Lindens for not just ignoring us ….”

    LMAO ! I hit the floor on that one..

  74. Andrew Scott says:

    We need some more solid and tangible reassurances here.
    I am relatively new to SL, and it feels like this sort of thing is all I know of it.

    Never the less, I have been spending 10 to 18 hours a day trying to set up an business, and considerable cash, and I have to say it feels a bit like throwing it into the wind and just hoping that it finds its way.
    This is big money now, a huge time and investment for so many – peoples livelyhood – and you can no longer afford the luxury of feeling that you are “allowing’ us to play in your sandbox.

    The fact that we have absolutely no way of backing up our work done online to hard disk is a giant leap of faith for us, and for what you charge us for the privilege of doing business with you, you HAVE to offer us some security in that.

    To loose a shirt, or a jet ski from my inventory that I paid the equivalent of 25 cents real money for – well, I’ll get over it, but to see that happen and then think the same could happen to all of my inventory, my time, money and hard work…. well, how would you feel?

  75. LinXoa Yue says:

    Thanks for the information. Yes indeed a ROUGH week lol. But I must also offer my 2 cents…..
    The Trevally Sim is horrible. My objects fail to rez, or when rezed sink into the virtual earth as though they are sinking. Secondly, objects in BUILD MODE do not behave correctly, ie becoming unlinked spontainiously, will not accept position commands, will not accept delete commands, can not be taken into inventory, etc etc.
    Personaly I believe voice has over taxed and crashed your hardware. And on the subject of VOICE, I have yet to meet a resident, paying or not, that welcomes voice into our second life. Honestly it seems as though Linden Labs proceeds with ideas without any regard for what thier customers actually want in thier second life. Voice is for FIRST life NOT second life.

    Thanks and happy grid-running! 🙂

  76. Renato says:

    WOW finally someone who tells us more then we appoligise and thankyou for your patience. Kudos IAN for actually taking the time to report to us on what is /has/was happening. We all know you don’t/can’t control how hardware and your service suppliers work. My hats off to you Ian for Doing this maybe you can rub off on the rest. But yes i agree fix current then consider other options

  77. Zarah Dawn says:

    Not ignoring us? they’ve been ignoring the problems & us since I started in SL tho I have to admit it worked better then than now. The letter from the residents asking LL to repair what needs fixing & stop adding stuff til it is, wasn’t worth the paper it was written on. Basically LL said ” Ok, we got the lettter now go play your game & give us your money.”
    Ok, so we know what the problem is. Is this fix just fixing the symtoms or is it going to fix the base of the problems in SL? I’m tired of not being able to work & build in SL because of one problem or another.

  78. U M says:

    “The end result is a pretty rough week for Second Life Residents, and I can only offer apologies for that. The good news is that we have many projects in the works which will help improve the reliability of SL. We’re working on new infrastructure which will help us withstand the failures above, as well as store and transmit data more reliably, and we’re working on reducing both client and server crashes. Several of these projects are new, thanks to our growing technical team. We’ll share more information on these efforts in the near future.”

    Well lets hope this is true. But when I see it, then I will believe it! How many times have we heard the ” it will never happen again”? I Just hope you people learned Outdated hardware needs replacing PERIOD! What I don`t see really in this meassge is besides the HDD being replaced, not nothing else is? So inturn i seeing a repeat of failures based of the lack of better judgement of replacing old and warnout hardware.If we were sitting at 2 million accounts, I say ok you use current hardware. But we are not reaching 10,000,000 accounts spells ! MELTDOWn. If hardware is not replaced because of the heavy useage. So Ian, if you really want us the users to be relaxed do a little pervention. REPLACE PARTS THATS NEED IT!

  79. jdehlic Jie says:

    FINALLY a real explaination. I think many SL users get more upset at some of the generic explainations than the actual crashes. NO one wants crashes, but treating paying users as slightly dim kindergarten students only makes matters worse. Thanks for the explainations, good luck with the repairs, and don’t ask for a downtime refund from your service providers unless you offer one to us as well.

  80. Supa says:

    Since my other post was moderated I suppose I better keep this “fan boy friendly” and recommend another slice of last weeks compensation be given for the repurchase of lost inventory or whatever way LL can provide its “paying” customers.

  81. Sindy Tsure says:

    Wow.. Tough week.

    /me buys the grid monkeys a case of banana bear.

  82. Leena Deschanel says:

    THANK YOU. A week or two ago, I asked that LL provide it’s residents with as much info as possible regarding problems as they are encountered as well as keeping us regularly updated as to the status of the problem. Maybe you saw my post, maybe you didn’t, but it’s good to see this flow of information coming out of LL. Knowing what’s going on makes dealing with it that much easier.

    Cheers, and thank you for keeping us up to date. 🙂

  83. Melanie Milland says:

    Hi,

    I really feel left out when everyone around me voices.

    On the table in front of me is a brand new headset, and my PC even has a connector it fits into – but I am on LINUX and you keep delaying the Linux voice release and even refuse to give an ESTIMATED time.

    Please let us know if we can expect voice in a week, 2 week, a month, 2 months or a year…. I mean, really, you must know SOMETHING!

    On another note, @26, I actually went and the place is a casino, just by a different name. It is exactly the kind of place that blights SL, with loads of camping and things that take your money.

    What a devious move, using the very blog here to advertise a casino!

  84. Ryu Darragh says:

    Heh.. I and my partner appreciate the info. Thanks, Ian. Noticed the oddities when the red scripted object beacons in the 0,0,0 corner of the SIM appeared frozen. Usually, like prims and avatars which appear in those corners, script updates only flash in and out in a fraction of a second. Lately, they have hung there like small red novas. Curiously, they (and all the prims and avatars) appear in that corner in relative positions to each other that they will have when they finally do get placed in world.

  85. Jazzman says:

    Wow wee…
    Nice to see everybody back on the blog..it was a rough week..SL crapped out so I actually read a book and checked out some other virtual worlds. Lindens are still ahead of the curve it would seem to this neophyte observer of this kinda thing, but competition exists for those of us suitably po’d.
    I’m not…with the exception of this past week I think I have had good value for my small monthly tier and premium account. endless amusement frankly; in particular when somebody’s hair rezzs in their crotch or a shoe comes through the ear.
    I feel bad for you folks not getting satisfaction but a great many people do, and having been an SL victim of inventory robbery before($20kL) I sympathize…but in the end you can walk if you want.
    It’s the only option, be sorry to see you go but a smaller game would be better generally I think. I don’t like the label whiners because if you pay you have a right to complain and THIS is the place.
    I’ve met piles of folks loving voice and awaiting the sky thingy whatever it is, but yes…keep those tunnels open Lindens and shore them up before showing us the new sky. It’s sometimes very dark in the tunnels and the place is full of trolls and faeries and vampires, and goreans everywhere you look..nevermind the furry things scuttling about and getting in my wife’s jam.

  86. Raudf Fox says:

    It is good to see things explained in depth now and it makes it much easier for us to have patience with Second Life when we know what’s going on and what’s being done about it. I really appreciate this news!

  87. Athena Whizenhunt says:

    WORK ON THE BASICS! I could care less about casinos and the things that “blight” SL ( some of which I enjoy). if you don’t want to go to a casino and camp then don’t go! I personally think the things that “blight” SL is SL working poorly- less time cleaning up our activities and more time making activities possible_ it was my understanding that SL was for us to create- if it works properly we can do that providing we don’t get stuck with a bunch of well meaning moralists-where did they come from anyway?

  88. Janet Rossini says:

    I very much appreciate the update on what happened. Most of the details are more technical than I need, but it gives me a feeling that someone cares, and is on top of things.

    I am in strong agreement with those here who say that LL should focus on getting things working well, and keeping them working well, rather than on “enhancing” our experience with weather that no one can share and other such new features. I am keeping an open mind about Voice, and can see some places where it may be useful.

    HOWEVER, you promised that voice would not affect performance, and everyone I know who has downloaded the voice client is reporting many many crashes, as often as every 10 minutes, even with voice turned off. That’s a performance issue, and a big one.

    I have heard that LL employees get to work on whatever they want to work on. That makes it a great job, I’m sure but it doesn’t make my Second Life a better life.

    If you need help learning how to do testing and root cause analysis and other modern approaches to improve things, I can put you in touch with the consultants and training people we have used in my company to our great delight. I’m sure they’d love to come help you.

    Thanks,

    Janet Rossini

  89. Pingback: The Gird of Late « The Grid Live

  90. brinda Allen says:

    well at least I know why I suddenly went from 6ft 9..to 5ft 2 and no eyes and looking like I had mange and..and..hey…it could be a lot worse..there could be no SL.It seems to me that the wonderful times I have in world are well worth the problems that I sometimes experience..(I’ve tried stomping my feet and throwing a tantrum…doesnt work any bettr here than it did when i was !) Thanks LL, all in all we’ve got it made!

  91. Thanks for the explanation. Doesn’t sound like it’s been a fun week for anyone.

  92. Melanie Milland says:

    @53: I’m not a “moralist” – the blight comes from camping, which ruins entire sims.

    Incidentally, if all the connections used by camping Avies were dropped, SL would have 50% less load and would maybe work right.
    Think about it. I belive camping has no place in SL.

  93. Nevera Stooge says:

    gripe, gripe, gripe….

    complain, complain….

    whiiiiiiiiiiiiiiiine!

    Just my 2 cents

    😉

  94. U M says:

    Don`t sence of the issue here. it was hardware and networking issues . Don`t start taking things off subject matter and bring in campers. We all know the problems they cause but lets keep this issues on topic. Atleast for the sake for the future for it to not letting it reoccure again. Stay focused

  95. Ihatewhiners says:

    thx for the update Ian …. and for those whiners just be happy sl is running 😀

  96. Claudine Chantilly says:

    Excuse me, Mr Linden.. when will it be safe to install things on my clit without losing it?

  97. IloveSL says:

    cheese anyone? 🙂

  98. Noisey Lane says:

    O cool – I just spent $1499L on an item I never received – I guess I’ll have to rely on the good intentions of the vendor to make good – even tho it’s not his fault. Or dance for another month to make it all back so I can wait for another bug to screw up my transaction again.
    Lost inventory should be compensated – it must all be logged somewhere and easy enough to verify.
    This is not the first time its happened this month either – hardly good news for vendors and the SL economy – I’m getting nervous every time I purchase something and that is NOT good for business.
    If I ever get my non-delivered item, it’ll be the last big purchase until I can be guaranteed I will get what I pay for – if this were RL and I didn’t get what I paid for, consumer laws would protect me – what is Linden doing to protect consumers and is the lack of protection in breech of US laws – u stopped gambling in SL because it broke RL laws – do the same for consumer protection sometime now-ish!

  99. Cookie_Juran says:

    If second life is to succedd and bring in more investments, as in ‘land purchases’, then it HAS to be more stable. To many people are getting discouraged and turn away from the game. To me that is a shame becasue I , and others, see the potential here. PLEASE PLEASE put all your talent on that…STABILITY!!!!

  100. Dyslexia Sprawl says:

    Good communication LL… it makes a big difference when we know what’s actually happening! Thanks.

  101. Noisey Lane says:

    Melanie, some of us don’t have credit cards or live in countries that SL will not allow credit use so camping is one of the few legal ways to make money in SL – Your attitude is elitist and patronising. Get a RL and let the campers camp. There’s plenty of other sims with no camping to be by yourself in if that’s what you want.

  102. SL Player™ says:

    #30

    It would be nice to know why their not doing that.

  103. Nerketur Kamachi says:

    Ah, it all makes sense now. Thank you LL. Finally we are told what happened. If only you told us this sooner xP. Anyway.. I downloaded the “voice-enabled” client, but not the newer version, simply because I never use voice. Either way, though, You guys are doing quite well at keeping SL going, even with all the problems. I think I’ll Volunteer, so I can help you guys with the issues, and perhaps let the people be heard ^.^ Anyway, I hope all goes well, and everything behaves from now on.

    PS: Ignore all coments that complain about the same things. the “No gamble” rule was a good move. those machines did cause lag, and, Voice is cool, but unnessecary in my opinion. I’ve never really crashed, and I’m happy with SL. Usually a re-login fixes any problems.

    PPS: To all Complainers: Complaining all the time just doesn’t help. Whether YOU see it as a problem or not doesn’t mean everyone else does. The reason SL is laggy is because of all the complaining, and griefing, and all the negative attitude xP

  104. Thanks for the low down

  105. Melanie Milland says:

    Cost. Since they are driving away their paying customers, yet roll out one expensive shiny after another, there cannot possibly be any money left for the most expensive class of data connection known to man.

  106. SL Player™ says:

    #56

    …you should go to Welfare Island sometime. You’d be surprised there is very little lag at all. I was able to walk and fly around with ease despite the very long line of campers.

  107. Erbo Evans says:

    Thanks for the info, Ian; not only is this a good explanation of what was going on, it provides more interesting technical details about SL’s implementation (meat and drink for geeks like me).

    For those that bemoan LL working on voice and Windlight while these things are going on, I’ve no doubt they’re working on behind-the-scenes stuff that will improve these situations, too; those just aren’t as “visible” as strictly client-side features like voice or Windlight. (OK, voice needs servers, too, but those aren’t the Grid servers.)

    And for those who say that LL should be replacing all its outdated hardware…I’ve seen brand-new hardware fail pretty spectacularly. Replacing a machine that’s old but still working well can be a crapshoot. Of course, SL probably pushes these machines fairly hard, which has to be factored into the replacement decision as well.

  108. Thanks for taking the time to write that blog … if all the whiners are honest .. of course its not easy managing this kinda setup … SL is an amazing place, sure there are flaws.. but being honest like this is a huge comferter …and i thank you for what you have created, what you are trying to fix, and for intoducing me to friends, and one especially, that i would have never met if it wasnt for SL …

  109. U M says:

    “And for those who say that LL should be replacing all its outdated hardware…I’ve seen brand-new hardware fail pretty spectacularly. Replacing a machine that’s old but still working well can be a crapshoot. Of course, SL probably pushes these machines fairly hard, which has to be factored into the replacement decision as well.”

    Oh please save it…………….If your talking about person computers don`t bother even saying anything.

    Facts of the matter is they screwedup! Period! Why in the world are you protecting LL? The already said what the problems were. But If you don`t think a HDD problem is not related to old system hardware. I sure hope you not incharge of anyone elses computers…

  110. Dana Hickman says:

    I appreciate the more in-depth explanation about the latest batch of problems. Thanks Ian for taking the time to write that up.

  111. U M says:

    As I said earlier, I am happy Ian explain what happened but i don`t want to see this type happening again. Again changing hardware for age is nothing new to experienced hardware experts. No matter what people say periodical checking of hardware wear and tear is basic 101 tech understanding.

    It ws wonderful Ian explain in detail what happened, but unless they are more careful next time might be worse then what happen.

  112. Luna Larsen says:

    thanks for the informations ian 😉 nice to see lindens don´t ignore us.

  113. Wolfgang Zhichao says:

    Thank you for the update, it helps explain alot.

    LL, please keep up the detailed information when a problem happens, dont baby us. Most of us know what your talking about when you say a VPN, or if they dont, they know someone who can explain the issue.

    In keeping with the topic, will the invintory items that have been missing this week be returned now that the issues has been fixed? I know of a few of my friends who have lost items due to not being rezed in world but were removed from the invintory.

  114. Alicia Sautereau says:

    Thanks for an explenation finally, but could the communication be abit faster please?

    also i`m kinda suprised u`ve posted this on a saturday, wich shows that we are being heard and some stuff finally being done or rlese it would have been posted on monday 🙂

    but seriously, with the past weeks, we could set our clocks for the moment the grid starts to fail, friday evening, any explenation asof why this is happening?
    we`ve been online with 47k at the same time and u hardly noticed it a month or 2-3 back so that isn`t the issue

    also now u feel what we felt over the past months:
    “which the vendor believed was fixed but which, it seems, wasn’t really fixed”

    even top voted jira issue`s aren`t being fixed and a 3rd party client is better then the official version with the amount of fixes in it wich are bugs or just annoying things residents hate, i`d be blushing…

    but i have 1 lil request as as a paying customer and for my tenants, when serious issue`s come up, BLOG them right away, not every one is a coder and can spend 2 minutes writing something simple to be updated by some 1 else later for more so that i don`t have to tell people with problems “it`s weekend… wait till monday”

    thanks

  115. U M says:

    Let`s get Ian a big SMILE 🙂

  116. Anylyn Hax says:

    great, thanks for the details.
    that was plan and the do, now lets see the check and the act :)))
    can we see the action list and the responseble persons ?

    kidding, 😉
    (send my greetings to your quality department ;))

  117. Alexandra Rucker says:

    THANK YOU for providing details. It may not seem like it, but it IS appreciated by a lot of us residents. 🙂

  118. Jman Udet says:

    Thank you Lindens for posting that info. I know it’s hard to explain it all while the crisis is going on, but please keep posting entries like this every once in a while on the problems that have occured. Also, thank you for all the hard work your doing. I know that your doing a lot in the background and can’t really tell us, which is why you probably see a lot of complaints on here, but there is a bigger picture goin on.

    For those complaining about Voice slowing you down, it shouldnt if you have it turned off. It’s your choice to have it on, and Voice communicates with completely separate servers, so yes it will add on more traffic, but only if you have it turned on.

    As for those complaining about downtime and such and about how much money you have in the game, dont forget it is your choice to put money in the game in the first place. You can do quite a fair amount in the game without having to put in a single penny. So if you know that the grid has these problems, why do you put money into it? Ive put a ton of money into this game, but I know the risks, know the TOS, and am willing to live with the consequences. Over time, as Linden Labs rolls out their ideas and plans of changing the way their servers communicate, or how to manage growth, im sure we’ll see a big improvement. But to implement changes and make sure you have a solid plan to do so takes time, you can’t just say “lets throw in this product, it’s proven reliable for xyz company, it’ll work here”. There’s more variables that go into play here. And while engineers work out to solve the problems that affect us now, others are looking into the future on where the company is going, and what they need to do to make sure that future doesnt run into the same problems that we’re facing now. Hang in there ppls, you’ll see larger improvements in the future, just that LL cant say about them yet.

    Keep up the good work LL 🙂
    *waits for the bashing to come….

  119. Calin McKinney says:

    Thanks so much for the detailed description in language that even many non-tech folks can understand. It seems that the best thing to do is plan pretty far ahead and design the hardware to be able to handle the ever increasing loads of land and avatars that are sure to come. Seems like having banks of servers assigned to various regions or parts of the world would be best..so that the whole grid isn’t effected by specific hardware failures, but rather the region (say 50 sims) assigned to that hardware.

  120. Marriah Dinzeo says:

    Thank you so much for shedding light on these issues. Although I’m trying to remain positive that these problems will be fixed in the near future, I am almost certain that only more problems will come, and with that, you’ll have even more pissed off residents on your hands than what you already do.

    Either way, somebody finally let us know what’s been the causes of all the damned problems this past week, which is all some of us ask for.

  121. Loniki Loudon says:

    Oh yes, nothing to do with voice…
    Its just total coincidence that all the current problems started that friday after the casino ban when you patched in voice and not four days ago…. Just coincidence… I suppose the grid shutdowns in the middle of the night to fix voice is coincidence also.
    And I also assume that voice doesn’t use this same security protocols or that the old viewer without voice capability (that works so much better) is all coincidence too.
    The big words sure convinced me…

  122. lost ares says:

    Good to see that we are being kept informed ..all be it a little long in arriving 🙂

    4 days ? … hmmmm … try the last 7 maybe 8 days … this all started last friday or as its now become known ( BLACK FRIDAY ).

    It looks like your on the road to getting it fixed at least … drive careful 🙂

    Good luck

  123. Man, there’s a whole lot of hardware problems going on (and not just this last week). This failed, that failed. Granted that’s the nature of the beast when dealing with electronics in general, just…damn.

    Thank you Ian.

  124. Storm Chatnoir says:

    Fix The World.

    I’d be a lot more sympathetic to your problems and ‘thank you for your patience’ messages if you hadn’t made this situation yourselves over the years.

    The fact is, SL isn’t going to get better quickly, and it won’t get better *AT ALL* if LL doesn’t sit down and fix the basics. As the community has been begging them to do for years. We could have waited for the bells and whistles, and the ice cream on the cake, We understood that SL was to be a work in progress, constantly evolving.

    Why didn’t you?

    It took a long time for me to move from ‘slightly annoyed’ to ‘Furious’ and it will take time for this paying customer to get back to ‘satisfied with my SL Experience.’ –

    Show me you’ve changed.

    Show me you care.

    Show me I’m doing the right thing by standing by you, SL.

    Show me I’m not waisting my time.

    Fix The World.

  125. Bebop Vox says:

    LoL i love some of these responses. obviously not long term MMO players. As any game always has server problems (even though these are more than normal) and every game is always being updated and theres always things for people to complain about.

    Also just like any MMO the creators are not actively seen with the community, while focus should be on the games content and support. Were lucky enough just to get a direct blog from them about whats going on than the normal log in message “The servers are down and being worked/updated, ETA: 2 days”

    So yes thanks for the update, I’d still say sell to another company like Sony who can put a couple billion into the project. =)

  126. WyldFox says:

    Thanks for the report. It explains alot on what is going on.

    To those who are PO’d, you ppl can either quit or stop complaining. There are better ways of doing that.

  127. I’m glad to see these follow-up posts after the “We’re working on it” posts when we’re all frustrated. I have been very frustrated with my inventory issues, and submitted a support ticket. Just as they do in support (offering an explanation to the problem when it’s resolved), it’s good to see the explanation and resolutions on the blog too.

  128. Sam Brodie says:

    Thanks for the detailed explanation Ian, is much appreciated here, and goes along way to cool some of the anger and frustration going around. As a few have said, just would be nice if you guys could be more vocal about whats going on, when it’s going on, especially when we all know it is beyond a mess. Even if you don’t know why, just acknowledge that things are not right, but you are doing your best to get to the bottom of it. Constant “Resolveds” does not make for happy campers… err residents, when the platform is obviously still flailing around like some huge dying animal. This is the first time inventory loss has hit me… and I very much feel for the “whiners” that have lost large amounts of money and are screaming. And closing comments… I do understand why you do, but… this is the only way most of us have to let you, LL, know our feelings. Sometimes just being able to vent is a good thing, not necessarily useful, but good.

    Thanks again, your post was much appreciated by me 🙂

    Sam

  129. nina says:

    well none of that meant anything to me, but im sure the more technical people will dumb it down for me :o)
    i do appreciate a dry technical blog over a bad attempt at spin.

    cheers ian

  130. Dwayne Lancaster says:

    I understand, we all have our computer problems.

  131. Nairi Gegaku says:

    Wow. A Linden that listened. SOMEONE GET THIS GUY A RAISE! XP Anyway. Good luck fixing the issues, I hope everything gets sorted out, and I do agree with getting the base problems sorted out, then moving on to Windlight, cause I like it better than the voice stuff >_>

  132. Smith Barnard says:

    I don’t care! It’s the game stupid! All I want is what I pay for and all that entails! I pay for 3 memberships and when the game is down, or isn’t allowing me to do what I want to do, I can’t help but feel the game owes me something back! I don’t care about all your technical woes anymore then you care for my game troubles! Although some times they are intertwined. Test updates more thoroughly and longer, and test again! Have secondary plans in place for situations that recently occurred and test them from time to time! Plan! Plan! Plan for the inevitable, foreseeable, and for the uncertain! It’s the game stupid!

  133. Firelight Simca says:

    Thanks a lot for the detailed explanation. I know that not all folks are techs and such, but it still helps to hear what happened.

    It tells me (and others) that you understand the problem which means that you can work on fixing it.

    Firelight

  134. Chaise Torok says:

    Tough break guys, go get some well earned beers. If you’re not using Cisco gear, now’s a good time to start 🙂

  135. Kianna Rossini says:

    How about fixing my characters lost database of inventory… and SL before adding stuff like Voice which I have yet to even meet someone who uses it. Thanks.

  136. Marianne McCann says:

    Keep on keepin on, LL!

  137. Avril Sodwind says:

    Well said Storm

  138. Mahala Peccable says:

    Well, thankyou for finally providing some actual information. You are right, it has been a tough week in SL, lost inventory, wierd ass problems all the time, a nightmare time to buy or sell land to boot. Please just get the main grid working properly. Voice and whatever are really not any use if you cant move, can dress, can’t even talk to your friends. Like many others I pony up big $$ for my hobby and it has been very very disappointing to have had such a bad week – what a waste of money.

  139. TomasG says:

    Just out of idle curiosity, why are you using VPN Tunnels for critical infrastructure and connections to the backbone and not dedicated T1 or better yet G3 lines?

    And aren’t you using automatic fallover redundancy?

  140. Gawyn Philbin says:

    I would suggest a DB system meant for large scale queries like you’re using, such as Oracle. MySQL, while incredible for a free system, is NOT meant for large-scale databases, and it’s scaling is sketchy to say the least. Yes, I can reliably say this, as I’ve been working with MySQL for 8 years.

    As for all of you ragging on poor Ian and LL in general about their hardware problems (most specificly, the RAID drive problem they had), RAID systems are a lot more difficult to troubleshoot then a normal system like you’d have in your home PC, especially when dealing with large RAID banks as I’m sure they must be using.
    I personally have dealt with such systems were one drive in the chain half-fails, but doesn’t automatically hand off it’s functions to the next in it’s chain. It’s a bloody annoying problem, and does take time to diagnose. I’m not saying I’m happy about the downtime or problems myself, because frankly, it upsets me to no end. Regardless, as much as the human factor may or may not have been an issue in this, the computer factor was much more an issue. “To error is human. To really screw things up takes a computer”.

    On the other hand, I tend to agree with a lot of Residents, that more R&D on current issues would be much more beneficial to us as a whole, then working on the features such as voice, windlight, and whatever other “pretty things” you might have currently in the works.
    I sincerly hope all things are taken into consideration over the next little while, and you allocate your resources into the appropriate places.
    Regards,
    Gawyn

  141. Thanks for the detailed info! Have a good weekend.

  142. 3G Indigo says:

    Thanks for the info. I love SL, but it has been difficult to live with this week (bit like my RL partner). LOL

  143. Ravanne Sullivan says:

    VPN = Unreliable!!!

    Using a VPN for a mission critcal data path is very poor design. A VPN is good for a backup route or for a cheap connection between two or more locations where where if communications fail it will not bring the business to its knees. When is LL going to start thinking like a real business and make important design decisions accordingly?

  144. Tegg B says:

    Good work to the ground crews, though some reassurance would have been nice from higher up.
    I suspect something is till not right I’ve been stealth crashing since early in the week, and it’s still happening.

  145. Spry Homewood says:

    @25 Too funny. When we complain about bugs and bad fixes LL never listens or acknowledges us. As soon as it happens to them they want the whole world’s shoulder to cry on so all of a sudden they use the blog to have a whinge at someone else’s tech. Too funny, the irony is killing me.

    Anyways Lindens, this is a lot better than saying nothing to us. Keep it up! And let us know when it’s your screw up too, you might get a hard time for it and it might be deserved but we’ll respect you more for being transparent.

  146. Dahlia Trimble says:

    Thanks for the info, it really helps. Sounds like a bad stroke of luck. Hope it’s better with the new architecture changes you’re bringing in. It’d be nice to get the pretty skies too! 🙂

  147. Megaera Cioc says:

    Someone ought to tell Sen. Ted Stevens that the internet isn’t a series of pipes after all: it’s TUNNELS!!

    But seriously, you guys have been doing a much better job of keeping us informed about what’s going on when there are problems. I appreciate it.

    Now, if you could do as good a job about warning us about rezzing issues a lot sooner. I really *hate* it when I lose inventory and then you warn me about it 3 days later …

  148. honeydripper says:

    thank you for keeping us informed….really does make us feel as if we are not being ignored out here

  149. Avacea Fasching says:

    I could only hope you received a credit from your third party providers.

    I could only hope you will pass that along to your valued customers.

  150. Giulio Prisco says:

    I appreciate the efforts that LL is doing – it is a complex technical system with heterogeneous hardware and software subsystems, and it is perfectly understandable that something stops working every now and them. What is important is that there is a skilled team of specialists ready to analyse and fix problems, as it is the case.
    I think the “final solution” will be a distributed grid run by independent operators on their own hardware. This will, at least, ensure that problems are local and only affect parts of the grid rather than the whole.
    In the meantime, kudos to LL and keep on the good work.

  151. ellisdee welinder says:

    On one hand I appreciate the news, although some of this news seemed to come 3 to 4 days late.

    On the other hand, I am missing inventory items, my business interests have been effected, and as you stated, this was a pretty rough week for residents.

    Perhaps something more needs to be done by Linden Labs to make amends for these matters of late than just another “we apologize for any inconvenience this may have caused and we appreciate your patience.”

    Apologies only go so far, and out patience is the same way too, it only goes so far.

  152. Deira Llanfair says:

    Thanks for the info Ian – all the hard work behind the scenes is very much appreciated by me, as is being kept informed. It seems that SL needs a major redesign and rebuild before all these issues can be properly sorted out and in the meantime it is a case of “make do and mend”. Of course, major redesigns are not without risk too….worrying – I don’t envy you. Best of luck!

  153. magiquemerlin says:

    Excellent information post Ian – thanks for that.
    Hopefully the ops can relax a bit next week and grow their nails back.

  154. Giulio Prisco says:

    Concerning “on the subject of VOICE, I have yet to meet a resident, paying or not, that welcomes voice into our second life”:
    Besides being irrelevant, this comment is completely wrong.
    I have a network of hundreds of contacts in SL, and must of them are either actively using voice, or upgrading hardware, buying mics and headsets etc. to be able to sue voice. Also, all the sims that I use to visit are voice enabled, with no exception.
    We should remember that everyone tends to seek and spend time with like-minded people and complaining about the lifestyle of others does not make sense.
    The secret of life is simple: do your own thing, and let others do their own thing.

  155. U M says:

    third party providers? well some are no help really. infact they have as much problems as the main viewer.

  156. XLR8R says:

    Can we correlate Voice and the latest trouble
    Hummm

  157. Cinthya Vavoom says:

    OMG come on now people!

    your being sheep.

    The problems have been much longer then 4 days. We’ve been having nothing but issues since voice has went live. and thats been longer then 4 days.

    LL will never admit that voice is causing some the problems we’re having. but anyone with a little bit of a brain will know.

    but then LL needs sheep and sheep are exactly why nothing will ever change here.

  158. AnnaLatexa Beaumont says:

    Hello Ian,

    thank you for this information *smiles*. I think, this kind of policy makes it easier to understand what is happen behind the doors of our beloved grid and in result we all have more belief in our future. And btw. thank you for this lovely place to be *kisses* to all of you at Linden Labs.

  159. Jordguitar Flasheart says:

    Lets hope that all of this will work well.

  160. sanou woodget says:

    Problems are not resolved for me i lost a few my shape and don’t find it in my inventory… what’s happened??? and when can i be me ??

  161. Lucas says:

    Thanks for explaining what broke during the five days of frustration for all of us
    My avatar had many sex changes due to this and it drove me nuts, could not build or rez objects at times and I even lost a 5 buck fire kit I had bought, thank God it was only 5 L
    I wonder why do you Lindens have the servers spread over the continent, Why not centralize everything and then have smaller mirror servers
    And why not a direct connection to the ‘net and dump the VPN tunneling thing, And are you running on Windows , Unix, and Linux based servers
    The last time I was on things were pretty normal, Well except certain objects turn red when I load the ” building ” menu
    I am off to log in using the optional update

  162. Wolvie Howton says:

    Great explanation thankie for that and as a ISP techie i can fully understand it, been there done that (remembering his week of hell with network failures) and it sucks bigtime especially when things fail without a clear seeing where the problem is.

    Sadly some peeps are not able to understand it, yes i bitch sometimes at SL more out of frustration when my AV starts doing weird things, or even prims for that matter.
    But then i remember how sometimes Murphy’s law comes in to play, and that mistakes can be made.
    Software development is not easy i think 40% of the time bugs left and right have to be fixed, it’s the nature of it nothing that can be done about it.

    So i don’t blame voice because it a separated service from SL if people ever take the time to look in there Windows Task manager they would see that it’s a stand alone program and if they would read the FAQ’s like they read the blog they would know the voice servers are not in anyway connected to the grid.

    So i can whine and cry murder, but i will not because i know how hard it is to run servers and router and in general big networks they just sometimes fail in the most weird places.. shit happens and sometimes it’s stacked high.

    LL good job on telling the residents, do this more often though 😉

    Wolvie

  163. Tijn Erde says:

    What about the failing TP’s and missing “Buy” pie-buttons that are still plaguing my home region and you keep referring to the blog? Is that going to be solved by network fixes as well?

  164. Sling Trebuchet says:

    I think that this blog posting is the most intelligent thing that LL have done since I started in SL in January.
    The explanation makes sense to me. I think that even non-techs will take some comfort in it.

    It seems to me that user confidence in LL was at an all-time low. (ok, in my 7 months). This was more to do with incredibly bad communications rather than the grid problems.

    There will still be problems and bugs. The LL leopard won’t change its spots overnight either 😉
    We’ll have lots of bitch-fests, and I’ll be bitching too.

    Its a real comfort to know that LL
    1. have some sort of decent handle on what is going on
    2. have realised that they really need to keep the users informed as fully as possible

    Thank the $deity for that!

    Now if only LL would listen to and engage with the people who make the world of SL.

Comments are closed.