Further Information on Second Life Issues, 1st-2nd October

I’m Neuro Linden, one of the new team of Production Operations engineers here at Linden Lab tasked with assisting our support and operations staff with maintaining the health of the grid. I’d like to supply more information about the Second Life service problems of the past few days.

There were two primary problems: a change to the way we perform rolling simulator updates, and a hardware failure with the VPNs which allow distant Second Life servers to communicate. I’ll describe both of these failures in detail, below the fold …

We’ve recently changed the method used to deploy rolling simulator updates, in an effort to speed up and stabilize the process. Unfortunately, when we used the new system on Monday, it left many regions either shutdown, unable to report their status to our core systems, or running the old software version. The release team has identified several bugs in this new system, and are working to correct them before we use it again. Thursday’s rolling restart will use the old system.

So, after the new update method did its damage, and developers and operations staff were recovering the affected regions (at about 8pm PDT), a fault developed with two of our VPN servers. One was recovered successfully by operations, but at 11:30pm PDT, in the process of attempting to recover the second, it suffered a hardware failure, so we set out to replace it. While we were configuring the replacement machine, it suffered a ‘stall’, which required us to intervene physically, and we weren’t able to get it up and running until 6am PDT. This returned our VPN to full service and most of the affected regions came back to life.

Some regions still had issues after this time, and our developers and production operations staff worked through the morning recovering them, with the bulk recovered by around 9am PDT.

We offer our apologies to all residents affected by these problems, and I would like to re-iterate Ian Linden’s previous comments that we are working on several projects to improve the reliability of Second Life, one of which is replacing our VPN systems with more robust technology.

This entry was posted in Operations, Service Status. Bookmark the permalink.

101 Responses to Further Information on Second Life Issues, 1st-2nd October

  1. When will we have a single day where SL runs smoothly????????

  2. U M says:

    Thank you I look forward to this new process. No we don`t have a single day of “SMOOTH Server”.

  3. Jonathan Sprawl says:

    Could one or more Lindens please elaborate on how the new methodology for performing rolling restarts was (or was not) tested in simulation before being used on the main grid? The pattern that repeats here seems to be one of finding bugs in the production environment that millions of residents pay to access.

  4. Kerik Rau says:

    Hahahaha #1

    The answer is obviously no.

  5. Fattimah Collas says:

    I am unable to log on to SL.. I have been unable since 6:15 pm (pst)
    Is this problem gonna be fixed tonight?? Am I the only one that is unable to log on??
    Some answers would be appricated….

  6. knitemarco0731 Tripsa says:

    Thanks for the update Nero, but what is gong to be done about all the packet loss and terrible lag since last update? The previous version had none of this, (well yeah we all get lag, but one hit on movement and I am 800m away????) little too much. I really like SL, so I want to try to be positive about outcomes. Will this new update tomorrow fix anyof this new lag?

  7. Wow thanks! That was a nice gesture coming out and saying it like that. I could’t upload textures or log in for long periods of time.. one of my sims went down for almost a full day. I was a bit more worried then usual.

  8. Ann Otoole says:

    How about giving us a break for the man made breaks? Stop rolling out new anything for a very long time. Your back to the old pattern of destroying the economy weekly. SO STOP IT.

  9. Rubie Vita says:

    Well I am a newbie to second life ..only three days old and already getting frustrated so I can understand how all the older members etc must feel …in the three days I have been playing I can never stay on line for more than about an hour or maybe two at most with out crashing …I do run a dual processor and have tried all the suggestions made for fixing crashing problems with no sucess ..maybe when you get stuff stablised maybe I can stay on line longer ..thanks

  10. Gennifer Meredith says:

    Wow, just out of nowhere, a huge crash, and then trying to log back in, ‘Region is Off Line’ or something.

    How about a little warning, please? I understand that SL is beset with issues, how could something this big not be, but at least let people know that you’re going to kick them off.

    Please?

    Thank you.

  11. richard says:

    we still can have sex right?

  12. U M says:

    There once was a time when Second Life was a small and close group. Then One day some linden decised ! ” HEY LETS TRY TO GET 10,000,000 USERS”.Well a short time after ( 1 year later ) There was no Second Life because it was hacked and left for dead” The End…….

    Well lets Hope it wil be around longer then that. Too many people have Too much time and money invest on sl.

  13. knitemarco0731 Tripsa says:

    to #11
    Not according to my bed, no.

  14. Poolzy says:

    just fix it get it done so sl can run smooth im so sick of it not bein done yet

  15. #5 Possible resent firewall settings change?

    Everyone else, your utilizing a cutting-edge technology that is shaping the future of internet communications and utilization. If you prefer not to have sporadic issues I would suggest signing up somewhere else that uses old technologies, is barren of users and has no vision of the future.

    SL is blowing away competition because they don’t just think outside the box.. there building it.

  16. Poolzy says:

    oh and get rid of hippie pay and all that CAUSE IT DONT WORK IT SUCKS!!!!!!!!

  17. Votslav Hax says:

    thanks for the ‘heads-up’ neuro… and thanks for the explanation of the problems that have plaqued us all these last few days… keep, up the good work!!!

  18. U M says:

    @15 ummmmmm I think.

  19. Jopsy Pendragon says:

    Thanks for the informative and detailed update!

    Good luck with the changes!

  20. Janine Whetmore says:

    If it aint broke, don’t fix it. Hopefully you might grasp that idea some day.

  21. Karyn says:

    Also good luck with the changes. As we all know, computers have a mind of their own… 🙂

  22. Ive been having serious problems trying to get my 8 bots on to go camping. Please get this software fixed soon cause I’m losing tons of L$ not being able to log in 8 bots at one time….

  23. concerned says:

    Well we appreciate the hard work and late night to correct this issue grid wide performance issues after a bad update you decide to try yet another new system,PLEASE listen residents want a stable grid,they want to buy in confidence,they want to be able to walk across sim borders and not die,If you tell us a rolling restart is due then ok we can plan around it we cannot plan around you testing a new system,Sort the grid then look at improving what is basically a 5 minute island hop.I do know you guys work really hard on advancing this platform but we need a platform everyone who meets specs can enjoy and i have sadly lost 3 long term friends in the past 30 days because they no longer want to be here 2 joined 2005,1 2006.

  24. Skyye Perfferle says:

    I wonders why bother “mature” rating or no. Many time I be told is missing clothes me by friend. Screen self all look ok. Some other I no have clothes, some other has clothes me.

    I teleports into new sim. Many time I need wait 3-4-5 minute before world rezz. If tries move before that I lags bad maybe to next sim or even more.

    Understands you working hard infrastructure but this be getting worse. Needed laugh. Yesterday had question “Is you SL experience be get better or worse?” No could click any response.

  25. johnny says:

    hmm traffic hits 30k+ we all suffer your hardware cant cope

  26. Macro Voss says:

    Your honesty is much appreciated Neuro. I’m a software developer. Nothing in this world is bug-free. Its just great that people are able to use this cutting-edge technology for minimal cost (or free of charge).

  27. Pingback: Second Life News for October 4, 2007 « The Grid Live

  28. Todd Vedelander says:

    I noticed they took our ad money just the same, so they didnt feel all that bad the little bastards.

  29. Michael Timeless says:

    Neuro
    Thank you. It is refreshing to get some details. I’ve noticed in the last week that several of the lindens have been making the extra effort to let residents know what’s going on. Most of the residents that are complaining seem to be the unregistered ones who complain about everything.
    Examples of this are the ones who can’t bother to look at the login screen. If it says there are other residents logged in then obviously the system is at least partially working so why bother posting an obvious question. As for the lag, It’s amazing that I can run this on my laptop from work on a wireless link with a 5k inventory and several scripted attachements in a sandbox with 10 or more users and don’t see the lag. Perhaps the unregistered residents who are camping should try going to a different sim.

    I am not an apologist for the Lindens, but when they do communicate with us there are too many whiners. Instead of whining, write down your complaint and submit a bug fix. As for the one who said he was bounced 800 meters that is a communications server bug, (client to server so look to your internet connection). I’ve been working with Microsoft word (a simple word processor) since it was dos based and it isn’t bug free why would you expect cutting edge software to be bug free? Start reading before responding.

  30. You know, my experience the past month or so (after the power outage and near constant run of crashes and patches and stuff) has actually been pretty good. There were entire days when I could log in and not have problems. It’s gotten rough again the past few days, but at least I haven’t lost any items or money like before. Thanks for the heads up, Neuro, and I’ll be sure to buckle in on the 4th.

  31. Welcome Neuro, I usually like to complain about things, but it is a refreshing change to get an explanation of problems SL is having, and an awareness that LL is working to correct them, I have felt in the dark a lot of times about the workings behind the curtain and it often leaves me feeling like Linden Labs doesn’t pay attention, I like this blog, it makes me feel like I am a part of the process of SL, rather than a victim of it. Good work, keep it up.

  32. Jakyl says:

    at ThaBiGGDoGG Richez, how about getting a job in sl instead of camping, it’s just a waste of sl…

  33. Kathy Hallard says:

    Is this “robust technology” thing will involve finally fixing that ‘object is missing from the database’ error that makes peoples’ items randomly unusable? You know, the one reported in http://jira.secondlife.com/browse/SVC-553

  34. Selkit Diller says:

    Neuro, I am impressed and pleased with your honesty. Thank you, from a long-standing resident and outspoken critic. This is a very refreshing change, and I genuinely appreciate your candour.

  35. Prismatica Palisades says:

    Since the rolling updates I have became a premium member. However I’m wondering how long that will last because since these updates my system periodically freezes and I crash once every hour minimum. Before them I could run Second Life for hours without a crash.

  36. knitemarco0731 Tripsa says:

    #29
    Hey Mike,
    I appreciate the thought, but no, internet connection is fine, no other problems reported on my internet or wireless conenections.
    I sent in bug report with snapshot of this incident. I had used my multipose bed which was 750m high, and I ended at 0,0,0 on the ground. Even took a picture! Most of the problems I am personally experiencing have been since the new update was installed on Monday. Up until then, my lag was very low, and now I am experiencing packet loss consistently in the 20’s. It is always redlined which wasn’t the case before. I am also not a *techie* just a resident who is trying to learn and understand the ins and outs. So when you talk mainframe,bits,bytes and values, and paramaters, too me it is BLAH BLAH BLAH 🙂 I just want a smooth experience without all the problems. Thanks for keeping us Informed about this issue!~

  37. Rachel Darling says:

    Overall, I pay about $500 per month for this “free or almost free” service. The reason I do this is because SL is not just a game for me; I run a successful in-world business. Not surprisingly, when SL is experiencing down-time, instability, lag, broken tps, broken search, or even overall general “weirdness,” sales go down signficantly. This is a trackable trend that can be shown in a graph. So not only am I NOT receiving the service I have paid for, I am also losing income.

    It’s all very well and good for LL to admit that they have a problem, though it is a bit redundant — we can see when there are issues. Having this confirmed is far less important than seeing some progress in solving the root of the problem. SL will not survive if LL continues to simply fix issues as they occur and then apologize for the “inconvenience” to their customers. I know of no other company who would find this acceptable, including other Online game platforms. Banks, newspapers, web hosting companies, game companies, airlines and more all have mission-critical hardware and software systems which they manage to keep running with relatively little down-time or serious glitches. Why is this such a challenge for Linden Labs? And more importantly, what is the company’s commitment to answering this question?
    Let’s face it, guys — this is not just the developers’ sandbox any longer. It’s time to grow up and act like a real software company. Your customers expect nothing less, and will go elsewhere if you do not.

  38. EBCY.Clift says:

    Slifers,
    Problems were, are, will occur again, obviously . This is IT.
    I just appreciate the way Linden Team give us more explanations from now on. Thanks Neuro.

  39. EliteData says:

    im sure the team @ SL are sweating bullets when something goes awry.
    its not easy and painless to do upgrades and configurations while thousands upon thousands of members are online.
    i think the client software should have a tick enabler in the preferences to choose open GL, Direct X or software rendering though.
    since the client seems to prefer openGL, i had to change my video card from an ATI to Nvidia (Nvidia works better with open GL, ATI works better with Direct X)
    this would solve a good amount of “lag” problems, as some of the lag users are experiencing are from image rendering.

  40. Danny says:

    I left SL about 9 mths ago after being a resident since late 2003. I left because of all the lag, the crashs every few hours or more, the griefers, the lack of suport for premium members (I was one)….etc, etc.
    I came back in to SL after copius amounts of publicity on our Australian TV just to see if the new hype was true.
    Guess what….same ol’ same ol’. I have been crashing every 30 mins or so in some sims, lag seems to be worse than before and still the dickheads are in abundance… Whats that saying….Once bitten twice shy…? or maybe Buyer beware….phud! Nero thanks for the info…but really after all this time , still Lindens have to put pen to paper to justify the unjustifiable.

  41. johnny says:

    censorship dont you just love it

  42. maggie mcardle says:

    Thanks Nuero, was wonderign what was happening,then realised new viewer!…well as one of your former critics, i would like to thank you for at least “speeding” up the info on issues. keep it up!

  43. Nahona Nakamori says:

    People complaints of lot of thing about SL, me too of course, that’s good to pass nerves on something and let’s stress go off. let me say that: At the current Opensim development stade, In Linden Labs official grid, we can use attachements, hud, sit on prims and are not limited to a bunch of end-user useless mathematics functions and llsay as scripting commands. So all is not so dark 🙂

    Come on Lindens, this post is a good examples of what to do, people need more communication from you.

  44. bedeboop says:

    It would be nice if the game could be left alone for a while. All these upgrades and fixes just seem to make it worse and accomplish very little except creating new problems. Was wondering…at one time worked for a development firm, when we installed new programs and updates we ran a parallel system for a while….any possibility of that happening before just sticking it to us, come what may?

  45. Pervee DeVinna says:

    Oh great, another “We offer our apologies to all residents affected by these problems…”. You guys cash in real money and now also deduct taxes – maybe you then should behave like a real company and not only apologize but stand in financially for your mistakes.

    For now you not even manage do deliver a basic stability for weeks now. Even the odd web-based friends list is not working for days. Items keep vanishing due to your nifty little problems, inworld businesses loose customers and sales.

    Come on, start to behave like a business….

  46. johnny says:

    hmm 346$ i pay now for each sim thanks to vat i pay for maintenece of my sim it hasnt been maintained for 3 months how can i compeate with a no eu resident with a mainland sim he payes 195$ a month to you i have to pay 346$ how the hell is that fair i cant compeate lower eu members sim rates

  47. johnny says:

    at least sex is good in the lag cum pending………. lmao

  48. Masuyo Aabye says:

    Do people not grasp that SL is an ever changing, ever evolving thing? sure theres a few screw ups, but they fix them. if all the whiners knew how complicated SL actually was, they’d shut up and realise that LL is doing a good job keeping it running.

  49. Winter Ventura says:

    I’ve had a number of vendor failures over the last few days, really odd ones, like scripts unable to talk to the web, or the web unable to talk back.. it’s nice to know I wasn’t hallucinating. Thanks for the informative post.

    Seriously, those of us who complain could easily become LL supporters, if you just avoided the corporate doublespeak and told us that problems were occurring and what you were doing.

  50. Tegg B says:

    Ann Otoole Says: “How about giving us a break for the man made breaks? Stop rolling out new anything for a very long time. Your back to the old pattern of destroying the economy weekly. SO STOP IT.”

    Can LL please make a few isolated islands on their own grid to work the original Version 1.0,0 viewer and promise never to ever update again for those who hate bug fixes and new features. Then banish them there for their own pleasure.
    I want bug fixes and new shiney, keep it rolling out before we get bored.

    Thanks for the explaination Neuro, keep us well fed and we won’t eat you, it works for Torley 🙂

  51. Anderson Philbin says:

    I had to laugh when I read this in the blog post announcing the rolling restart: “we want to wait until the service is in a stable state before we make any sweeping changes”. Obviously, there’s no point making it unstable if it’s already unstable. And they did deliver! I’m constantly amazed at how fragile the infrastructure is.

    Second Life is a game and half the fun is wondering whether it will be working today. The blog is always my first stop before logging in. C’est la vie.

  52. Sweet says:

    # 22 ThaBiGGDoGG Richez Says:
    October 3rd, 2007 at 8:49 PM PDT

    Ive been having serious problems trying to get my 8 bots on to go camping. Please get this software fixed soon cause I’m losing tons of L$ not being able to log in 8 bots at one time….

    ** THIS is the reason i deleted all my campers i had 4 to help newbies out and give back some to the community i deleted them all cus of these bot campers i hope more will follow suit as campers and traffic dont really do much anymore all are bots sad

  53. Misfit March says:

    please i just want one day were i log in not in noob clothing with boobs and able to TP to work i havent been able to tp anywhere out of the region for 3days and had stalling or crashing infact i had so much trouble i stayed away coming back on off chance it been fixed and finding out otherwise, Listen guys i paid my account up front in advance for a year and all SL done is cost me money than they add V.A.T and still give crap service FIX this mess and stop updating till you have a working FIX cos i feel you dont test it first in fact all of SL is your TEST BED… 😦

  54. Fattimah Collas says:

    I still can’t log into SL.. and nobody is saying when it is going to be fixed… I miss out on alot today… MONEY!!!! Because of SL..
    Nobody is giving out any answers as to when many of us are going to be able to sign on again… Geez this is Ridiculous~!!!!

  55. Sweet says:

    knitemarco0731 Tripsa Says:
    October 3rd, 2007 at 10:25 PM PDT

    #29
    Hey Mike,
    I appreciate the thought, but no, internet connection is fine, no other problems reported on my internet or wireless conenections.
    I sent in bug report with snapshot of this incident. I had used my multipose bed which was 750m high, and I ended at 0,0,0 on the ground.

    Reason is ur bed scripts are very unstable at that hight take it to around 300M *shakes Head*

  56. AWM Mars says:

    It’s good to see that issues are BEING reported, rather than waiting for bug reports from the lab rats.

    If this is part of the new changes, then confidence maybe restored for the long suffering and most welcome. We know life isn’t perfect, but being kept in the dark for so long, makes us feel like Mushrooms.

    I sense a new wave of energy from the team at LL, lets hope this is a sign of things to come.

    Great Start.

  57. Martin Willis says:

    Ich werde das Gefühl nicht los, dass wir alle BETA-Tester sind und dafür auch noch bezahlen müssen. Ich wünsche mir einen Tag, an dem SL mal nicht abstürzt und an dem mal alles so funktioniert, wie es funktionieren sollte!

  58. U M says:

    Lets just hope they keep focus!

  59. Frans says:

    Hi Neuro I appriciate it you come forward and explain what happend. But would it be really be to much to ask, that during these things someone posts that LL is aware of it

    It was really bizare, the blog was celebrating that LL was doing a new agressive roll out, but at the same time the grid was having issues, with uploads, rezzing things from inventory and we could see concurrency drop considerable.

  60. Fellatione Aabye says:

    A VERY MEANT THANK YOU NEURO

    For once and maybe even for the first time a LINDEN with B.LLS (the first one i ever read) … who dare to face the SL Citizens …. who has taken time to read all the comments (complaints) on hte blogs … and most important doesn t take us SL Citizens as little children. (What a refreshment)

    Clear explanations …

    Let s just hope that all will be corrected in due time ….

    Again Thx very much Neuro “The Guy With B.lls” Linden

    ps; Hope to read you a lot on the blog ….

    *****

    Und für unserem freund MARTIN WILLIS (comment 57) .. Steck dir die sorgen an den Hut freund …. SL ist wie SL ist … mit gute und slechte laune ….

  61. nina says:

    laughing at #15.
    what a dumdum.

  62. Burgess Miles says:

    Most webmail suppliers are deleting, or at least shutting down, accounts that haven’t been used in 1-3 months. Wouldn’t it be a good idea if the same thing would apply to SL? There must be hundreds of thousands accounts that are never used…

  63. Psistorm Ikura says:

    lol @ #15, really.

    now to stay on topic:
    I sometimes dont understand why people complain that much. I havent had severe packetloss in months now. I rarely crash (maybe once or twice every X days, and thats most likely because I work on heavy 3dsmax scenes in the background and SL runs out of video memory), and generally, the SL experience is pretty much stable for me.
    while I agree that there may be people who have a really different experience, they should wait patiently and understand one thing:
    LL is working on a MAJOR infrastructure rework, which will get rid of the central asset server and the rather unreliable VPN servers and replace all this with more recent and stable technology. do you REALLY think they are going to waste development resources on patching up holes now instead of rolling out the big project as soon as possible, especially when its the hardware infrastructure that is the major problem, and NOT some software bugs?

    seriously, guys, read the blogs and THINK. theyve explained it long and extensively whats going to happen to improve the grid, yet you folks act like it never was said and we will use VPN until 2020 or something…

    on other news, thanks for informing us what went wrong, I like extensive information like this rather than being left in the dark 🙂

  64. Ron Crimson says:

    Hey, I happen to agree with #15. And going around calling someone a dumdum just because you don’t share his/her opinion is oooh sooo mature, y’know. *rolls eyes* o_O

  65. Lana Madonna says:

    “”22 ThaBiGGDoGG Richez Says:

    October 3rd, 2007 at 8:49 PM PDT
    Ive been having serious problems trying to get my 8 bots on to go camping. Please get this software fixed soon cause I’m losing tons of L$ not being able to log in 8 bots at one time….
    “”

    Camping bots are a pain in the ass and shouldnt be allowed, theyre the reason me and a lot of other store owners won’t have campers anymore.

    This upgrade is the worst for lag in a while, I cant move anywhere, freeze and keep crashing, can’t load or erit vendors or chairs. However, I know it will be fixed so I’ll just be patient. I have to say SL was running the best in a while on the last upgrade so why tamper with it??

  66. Tegg B says:

    Fattimah Collas Says: I still can’t log into SL.. and nobody is saying when it is going to be fixed… I miss out on alot today… MONEY!!!! Because of SL..
    Nobody is giving out any answers as to when many of us are going to be able to sign on again… Geez this is Ridiculous~!!!!
    So what, think of all the many more people who can’t PLAY!
    Then again, there’s nothing wrong with SL, 30,000 other people are in, and the same people have the same problems for months, it’s got to click sooner or later the tinistst possibility that’s it’s not SL?Perhaps some people should get a RL job, get a real internet provider and buy a better computer instead of whinning they can’t run enough campers on their 20th century dinosaur.

  67. Hi Neuro,

    welcome to the team I’d say. Hope you will keep on posting what’s going on behind the scenes. It doesn’t really matter whether it is positive or negative news, as long as we know what is going on, it helps a ton of people otherwise left in the dark.

    I’m looking forward to see the VPN’s move to a more stable solution, and while it will no doubt come with more downtime and regions being offline, the ultimate goal is a more stable environment.

  68. Demelza Beaumont says:

    Neuro, I see you’re new here. Welcome to the bear pit.

    As an ambassador of the new culture of openness, can I ask you to go to the Lindens with a suggested change of working practice? When you have a planned change to the main grid, such as the new rolling restart method or the new sim code being rolled out, would the Linden’s be prepared to publish on this (or another) blog the testing procedures and results *before* putting the change on the grid?

    Doing this would mean that (a) we can have confidence that the change has been properly tested and (b) that “many eyes” can see where additional tests could be added which might turn the unforeseen into the foreseen and avoided.

  69. tracey says:

    how about fixing the group issue that has been going on for months, your support dont reply to emails about it…

    when an object is deeded to certain groups the owner shows up as (???) this stops our secure scripts working..

    we have a lot of land and our partner has 2 islands, yet you dont reply to our support emails…

  70. Nano Ashby says:

    Nice clear factual explanation thanks 🙂 lots of growing pains, but it’s obvious we are growing and evolving – I feel a lot happier knowing *why* our user experience was changed – and what LL did to correct it, ie, replacing the failed hardware and resurrecting the failed vpns. Let’s remember what a complicated piece of stuff this is 🙂

  71. As a Web software developer with experience on some very large projects, I would say that Second Life shows poor quality control. The number of serious bugs that are uncovered AFTER going live with changes, is not acceptable by any well-run organization.

    Some people here want to blame those who have not paid as being “whiners”. I do pay, and I believe we can all detect experiences that should not be happening, with major problems constantly appearing and disappearing, then reappearing with the next update of the viewer or network. It also appears they need high-end network consulting advice in order to avoid dependence on points of failure (such as the VPN mentioned) – too often a machine fails and it takes hours to recover. If set up correctly, it should be either invisible or require only minutes to recover.

    It seems the company accepts too low a reliability as being the norm. A technology director with high standards should be placed in charge of all this, so reliability can improve. The fact that SL is still probably the leading virtual world in terms of maturity and number of members is not an excuse for poor quality control and technology implementations.

  72. Raul Crimson says:

    Thanks, Neuro.
    Some days ago i was talking with some friends that the worse thing in LL was the lack of communication, all we understand (most of us) that SL as a system is quite complex, and is normal to have problems sometimes, but lately Lindens seemed hidding from the residents (no Town Halls, no Life Help, etc…).
    I’m really happy with this post, Neuro, because it makes me think that things may be changing. I hope the rest of Lindens will follow this example.

  73. Pepper Haas says:

    @Martin

    I have to agree — I know SL has a Beta Grid but then why is it, that upgrades, restarts, etc., introduce so many unexpected problems, not just occasionally but all the time?
    And then, I often report bugs I encounter after such a “fix” (being a good little Beta tester!) but then when I read the blog on the next upgrade it doesn’t fix any of them. For instance for weeks now, when you click “edit appearance” it shows the back of your avatar. Two upgrades and three rolling restarts later it’s still not fixed.
    So yes, why do I feel like we’re all Beta testers here? Beta testers, moreover, whose reporting isn’t certain to reach the people who can do something about it?
    LL should thoroughly test everything on the Beta grid, where it has to run smoothly for at least three months before it’s inflicted upon the main grid, with the accompanying down time and related inconveniences for all users.

    Also, yes, I agree: GET RID OF BOT CAMPERS. Keep the rule that says no alts for non-preems and limit alts for Premium to two. People should also be required to register their alts so that a ban to one will affect them all.

  74. Ziggy March says:

    #1 … never

    #9 … dont forget you have a first life too !!!

    #16… yes please ban camping & sl will be a much better place…btw why the hell do people setup a sim and pay 2.5k to set a bunch of campers there anyway doh !!! oh yes i remember .. to get a false impression of there traffic count … so others would setup a shop there only to find out there only traffic is noobie and bot campers !!! yay for smart move, dance from happyness folks !!!

    #22… read comment to 16 !!! doh or talk to All Gore, he will explain it too you better… raise your electric bills for 3 L$ per 10 min yes its worthed !!! doh you people are so silly !!!

    #33… exactly… why the hell do we contsantly get the message (read tons) “Gesture is missing from database” & ” cant load Gesture please try again” doh while just getting online and doing nothing !!! LL fix that shit please !!!

    #62… get rid of the non used accounts … put a non actif limit on them and send them an email after 6 weeks or 3 months to ask if there still want to login… so in the end the REAL WORLD will get an honest resident number… not like now … in every newspaper or story or blah blah they say Second life has over 9 mill users !!! LMAO

    #71… thank you !!!

    #72… to myself i will copy this post to be sure it gets posted this time… (bonus … my land is on sale VAT included)

    To Neuro… goodluck mate

  75. Burgess Miles says:

    agree, get rid of the campers and restrict number of alts! And shut down accounts not used for 1-3 months, as the webmail providers do!

  76. Ziggy March says:

    sorry LL double posts are the resul from the first post not showing up !!! dont blame the cookie, but feel free to delete this one and the first !!! Mwah /hug Lindens

  77. Getting rid of campers and restricting alts seem like good ideas. However, is there any evidence that they cause the overall instability, bugs, lag, and so on? Possibly lag… but I doubt there are THAT many campers to explain the lag.

    But the lag is probably not everyone’s #1 problem – it is the unreliability, loss of inventory at times, loss of business income due to downtime, etc. I do not personally have those problems, but I can see how bad it must be for those who do. It still seems to me that from the very highest level of Linden there must be a shift in attitude and methodology. Currently it seems that bugs are regularly re-introduced into the system and that point failures (single machine) bring down the environment for everyone.

    It needs a steady hand, and support from the ownership of Linden, to start fixing these problems.

  78. Aya Pelous says:

    I will no longer be putting money itno second life until I know there is a stable protocol involved with secondlife.

    Ive cashed out 2 weeks ago

  79. Jim Perhaps says:

    I want to share my opinion here about the overal performance of both the Grid as they call it and Linden Labs. First a little history. I have been here since 12/2005. I pay my membership annually. I pay tier. More then some less then others.

    Little more history. A number of years ago I ran a small but profitable hosting service. My clients ran 2D graphical chat services. The software that ran these services was at one time owned by Time Warner. The software came on most new retail computers along with AOL, Prodigy, Compuserve and other such commercial services. Despite the financial might of Time Warner there venture into the world of graphical chat went bankrupt. Enough history. Now to my opinion.

    In order for me to provide these services I had to rent/lease servers. The company that provided my servers offered packages for as little as 99 dollars to up to several hundred. Though I was only paying $99.00 a month for my servers I could call this company 24/7 and after a couple minutes if that on hold be talking to a real live human. What was even better is this human with the help of support staff could solve problems very quickly. This company though now under a different name due to a merge hosts over 40,000 servers.

    Now I see LL basically as a hosting service. Their performance compared to my previous experience with a hosting company is often pathetic at best. For 195 dollars a month in tier they should be able to off what most web hosts offer for 4.95 a month. 99% uptime. I understand that the software behind SL is a little more complicated then Apache or some other web service. Regardless Linden Labs fails to provide quality service.

    I am sure there are many dedicated grunt level Lindens but I expect that the upper management has little regard for it’s customers. To me it seems they fail to listen and fail to care and fail to provide quality service. Perhaps they should reconsider name themselves all with the last name of Linden and take on various names. That is get out of the special group and think and act like those that are paying the bills.

    In short it is my opinion that Linden Labs does not know how to to provide quality hosting services as well as many teenagers running hosting services that provide 99% up time from their bedrooms after school and on the weekends. I think frankly the Lindens have the take the money and run mentality.

  80. Grid Borked says:

    The october started so nicely economy wise…Thanx LL for destroying this, however small trend. Who decided rolling out a new release to the month change was a good idea? Revenues are now pretty much dead again. Wonderful. I was just starting to forget about the illegal way of imposing VAT on EU customers…

    Please don´t roll out more than once a month, please please

  81. Ziggy March says:

    #1 … never

    #9 … dont forget you have a first life too !!!

    #16… yes please ban camping & sl will be a much better place…btw why the hell do people setup a sim and pay 2.5k to set a bunch of campers there anyway doh !!! oh yes i remember .. to get a false impression of there traffic count … so others would setup a shop there only to find out there only traffic is noobie and bot campers !!! yay for smart move, dance from happyness folks !!!

    #22… read comment to 16 !!! doh or talk to All Gore, he will explain it too you better… raise your electric bills for 3 L$ per 10 min yes its worthed !!! doh you people are so ***** !!!

    #33… exactly… why the hell do we contsantly get the message (read tons) “Gesture is missing from database” & ” cant load Gesture please try again” doh while just getting online and doing nothing !!! LL fix that **** please !!!

    #62… get rid of the non used accounts … put a non actif limit on them and send them an email after 6 weeks or 3 months to ask if there still want to login… so in the end the REAL WORLD will get an honest resident number… not like now … in every newspaper or story or blah blah they say Second life has over 9 mill users !!! LMAO

    #71… thank you !!!

    #72… deleted
    #76… deleted

    #80…to myself i will copy this post to be sure it gets posted this time… (bonus … my land is on sale VAT included)

    To Neuro… goodluck mate

  82. Fellatione Aabye says:

    Geeeee Just checked the agenda …. again a rolling restart at 1800 gmt …on this thurday 😦 and this after a black monday , a darkgrey tuesday, a somewhat lightgrey wednesday

    So Dear SL Brothers and Sisters, please join me in a prayer

    Dear God Linden,

    We all pray to you
    so Second LIfe will be ressurecting from her ashes
    like once did the Phoenix

    We pray to you
    so our beloved Second Live will no longer suffer
    from buggs, downtimes and rolling restarts

    We pray to you
    So our AV’s would not loose any of their precious inventory items

    We pray to you
    So we can login again without any problem

    AMEN

    Dear Second Life Brothers and Sisters please give a very nice applaus for Neuro the only Linden who dare to come down from the Linden sky to talk to us ….

  83. Arthur Fermi says:

    Once again, VPN is a problem child for this type of application. VPN packets are larger, there is a processing cost to encrypt and decrypt. Not to mention when traveling over the internet your connection can be perfect on both ends, but something in the middle that you do not control can be slow. To ensure QoS, which I would think be a permium you would want to have dedicated circuits between your server farms.

  84. Lazarus Longstaff says:

    Excellent report Neuro. Professional and packed with rich detail.

    Thanks a heap and a stack 🙂

    Laz

  85. Villanova Beaumont says:

    Thank you for the update Neuro. Fact is that SL is still so unstable that even basic functions like logging in, moving, teleporting, making transactions or building are constantly hampered by freezes, crashes and all sorts of other glitches. Fantastic concept but really difficult to enjoy, neither for pleasure nor for business.
    Pity.

  86. mimi says:

    Hi neuro
    thanks for the update!! Its great we now know what is going on! We appreciate your post a lot! lol i like your name 🙂

    I wonder if this trend of beeing opener will happen with some of the issues that could be more easily resolved than fixing bugs?
    Like information about age verifications companies safety, mentioning the VAT to europeans before buying islands… things like that? (I know its not your concern.. just wonder… maybe a linden could be hired for that too..?)

    @29: why do you assume the complainers are unregistered ones?
    Its not just affecting the unregistered ones, these problems are affecting registered ones just as hard. Just take a look at JIRA and see many of the big names (shop owners of well known big shops in SL) have voted on issues which are mentioned here too many many times. They are not exactly unregistered ones!

  87. martin magpie says:

    I read recently that SL peaked a year ago this month and has since stagnated. I also read that the average person only logs in for 12 min. I honestly believe that the grid issues have everything to do with that. I would love to see great vast improvements before SL becomes a footnote in the virtual online experience. Don’t get me wrong I adore SL unfortunatly it’s become one very long commercial lately. Good luck with the newest havoc version. I am in a wait and see mode.

    Mar/Cat

  88. Ishara longstaff says:

    well al i can say is thank for keeping us up dated on the blog, but be nice to be updated in world before something that effect grid is used. so we have choice and can leave, if we feel we have a need to.

    also LL had better what there backs goodle are working on there own metashere space (my world) and i know japnise are working on one too for the playstation 3, it gonna be called “home” and another japanise firm is working on another simmlar sl world opeing soon called “meet me”

    so if LL dont fix bugs faster they might become underdog instead front leader.

    Prepare yourselfs for competion LL the future migh be rougher for You!

  89. Glory Takashi says:

    because Ron Crimson #64 you and mr #15 don’t seem to get it. Second life is far from cutting edge it is actually old very old tech the software itself is nearly 6 years old the game has been open for over 4 that is not cutting edge. Not even close. This far into it they are having far to many issues. Sure programming is difficult but for the 2 years I have been here quality control is their biggest problem… there isn’t any!

  90. The XO says:

    …3…2…1…. cue the ranting… oh you already did! :-p

    First off… pop quiz:

    1. Do you know anyone who runs 16,000+ servers on a 24/7 basis?
    2. Out of those 16,000+ servers (simulators) most are online most of the time – don’t you think thats impressive?
    3. Now add in scripting, gravity, physics, building, prims, communication, web, voice, etc, etc – isn’t that even more impressive?

    Anyone in IT *knows* you will have hardware problems, failures and software bugs, glitches and crashes. Nothing is ever perfect. Not Microsoft, not Linux, not Apple. When Word or Outlook crashes, you bang the desk, swear at Bill Gates and the restart it – you live with it day in, day out. Fact!

    In SL it’s no different. Apart from the fact there are tens of thousands of servers, hundreds of databases and gigabits of bandwidth being used every second! You didn’t think your prims and house just “magically” stayed there did you?

    One such glitch or hardware failure can have a *massive* cascade effect and effect many other things. Forgive the example, but look at the ripples when you throw a stone into a pond. Sometimes it’s not a stone that gets tossed in, but a massive building block.

    LL do annoy me at times with their poor decisions – but these are not technology decisions, more “public relations” things. Technology wise, they really can’t be faulted. They are introducing backups and redundancy into more of their systems, but it takes time.

    The moral of the story: It’s going to rain sometimes so carry an umbrella, enjoy the sun while you can!

    Yours,

    The XO

  91. The XO says:

    @ Arthur Fermi #82:

    Bang on the money!! Well said 🙂

  92. Ishara longstaff says:

    to some other posters:
    Hey some of us do rely on camping and contests for Lindens, not all us are premium you know?!! an with service so poor from LL why should we bother upgrading? im still waiting for stability before i do upgrade;

    Also it’s not like we dont then pass those lindens on in form shopping so u premium guys and girls do benifit in that way. think of us as money pump!

    and basic accounts dont get alts! as far as I know. and gratful LL didnt end account after 1-3 months
    if ever moved house, it takes BT an eteernaty with ISP to swich u back on! i was without a net conection for 5 months.

    so e-amiling me wouldnt have been that effective now would it! ;-p
    besides as some posts show some people go away and after long spell come back to LL second life

    it up to acount creator to kill off there own Avitar! if dont want it.

    and LL sory this took up two posts.

  93. yuriko nishi says:

    the problem is that it gets worse over time. if you stay on this road you are on atm, sl will behave like an early alpha version soon. it´s ok for a game, but not for a business platform…

  94. DR Dahlgren says:

    We all know that problems can occur. Hardware fails, that missing EOL finally catches up to you. I am still amazed that LL has been able to scale up to the level it has.

    There is a lot of good work here by LL. That said, the major problems, or perceived problems, truly distract from the achevments.

    1) Paid Members perceive LL as being unresponsive to their concerns and needs.
    2) Those of us who understand things such as version control, wonder if LL does. It certainly appears at times that they do not.
    3) I realize that in some ways, SL is still and always will be a “Beta” environment. However, it often seems as though no testing is perfomred on updates of various types at all.
    4) LL is getting cheaper by the day. Quit trying to make every penny now, and plan on creating a stable virtual world where you can continue to reap the bounty.

    The way things are going: Gee, these virtural worlds that make up the internet are so cool. What ever happened to Linden Labs, the ones that really started it……

  95. Musicteacher Rampal says:

    When will the “friends online” on the website be fixed?

  96. Klaatu Congrejo says:

    Thanks for your detailed and informative post Neuro.
    This is a refreshing change and I applaud you for facing the community and treating us like responsible adults – something that has been seriously lacking at LL of late!
    @89 The XO. I couldn’t agree with you more! LL really CAN be annoying sometimes and its not because of their constantly innovative improvements to the grid. You’re right, its because of their poor ability in communicating with the SL community. Something that I, and many others in these blogs, have been banging on about for months!

    Well done Neuro – and we welcome your refreshing frankness.

  97. Alexander Regent says:

    Oh man….what a painful week for LL IT department!

    Thank you for the hard work…. I do complain every once in a while about stuff… but it can’t be easy doing your job… and thank you for doing it!

  98. starhunter Gall says:

    #62 ) Agreed, there are alot of unused accounts floating in SL taking up server space, Email the owners of inactive acounts , give them a set time to responde , if nothing is heard back then poof, . And I know there are military personal who play, but they even get to check emails often. Let the ones who play, spend money and enjoy SL actually get to enjoy it.

  99. Alyx Sands says:

    Could the people who whinge on about “no introducing of new stuff, fix old bugs” please note that this rolling restart was just that? AND the next one today will be the same?

  100. Rose Grooms says:

    Well Michael Timeless maybe alot of us are fussing…umm, maybe just because there are a lot of issues that need to be resolved… And if you’d think about it, maybe there’s a reason why we’re still unregistered users. You think maybe it’s because we’re not willing to pay for an unstable service? You think we might be waiting for the sim to stabilize before we register and start throwing our money away?

  101. Lynn Silverstar says:

    starhunter, I’d have to wonder if doing so to (idle) paid accounts would be a good idea. Especially for a friend who can’t log in (she uses her alt to be with us), yet still uses that account to hold her group land.

    Now, a 6 month window for free inactive accounts might be a good idea. It might actually bring the resident numbers in line with reality.

Comments are closed.