[Resolved] Database, asset server issues.

[RESOLVED 10:00 PM PDT] These issues have been resolved.  Thanks for your patience.

[UPDATE 8:38 PM PDT] We are still trying to nail down the problem. Thank you for your continued patience.

[REOPENED 6:25 PM PDT] We are currently experiencing some Asset Server Issues and investigating the cause. Updates will be posted here when we have further information. Thank you for your patience

[RESOLVED 5:00PM PDT] Everything should have returned to normal by now as the repairs have finished. We apologize for any inconveniences you may have encountered during this time.

[UPDATE 4:00PM PDT] We still have not gotten word on the status of the repair. We apologize that this is taking so long. We hope to receive word soon.

[UPDATE 3:25PM PDT] Our vendor has dispatched an engineer to the facility. We are waiting for them to complete the repairs so we can bring the system back up to full speed. You may still notice intermittent problems until those repairs are completed. Further updates to follow.

[UPDATE 2:40PM PDT] The problem has been traced to a faulty piece of equipment and we are attempting to have repaired or replaced. Once that has been done things should return to normal. We will keep you updated on our progress.

[2:14PM PDT] We are investigating intermittent database and asset server issues. These may affect logins, teleporting, viewing scripts/notecards, and rezzing items in world. We are aware of the problem and working to isolate the source. We will have updates as we have more information. -Chiyo

This entry was posted in -Miscellaneous. Bookmark the permalink.

150 Responses to [Resolved] Database, asset server issues.

  1. JayR Cela says:

    Hello :_)
    is nice you tell us on the Blog / however I believe that an in-world announcement would be greatly welcomed as well
    other than that / thanks :_)

    JaayR Cela

  2. Mimi Coral says:

    Thank you for letting us know in a timely fashion though I didn’t have to read this blog to realize we are having issues inworld. Good luck ❤

  3. la le lu says:

    not you lindens found the problems. us users found it.
    just check here:


    you should recognize it really before the userbase do. get your monitors
    ready. get some smart snmp guys for your nagios.


  4. Hiro says:

    It is good as long as the comment is open. I don’t feel the rage.

  5. Kandy Tomorrow says:

    Get a clue!

    If you were actually researching you would realize this is a recurring problem, and stop patching the problem for a few hours, and actually fix it.

    This is ridiculous.

  6. Kieran says:

    This is always happening is second Life. What happens to the poor old people who are actually paying as a member! Good luck solving this one!!

    – Kieran.

  7. SecondLife Is Just A Sad Place Anymore says:

    JUST another SADDDDDDDDDdd old day in SL for the weekend LL, last night was bad enough and no my sister has no clothes on and can’t put none on either and all you got to say is we are working on the issue , well you all have been working on the issues since when Jan , 08 and still nothing is FIXED . GET OFF YOUR LAZY BUTTS AND START TO WORK ON THE MAJOR PROBLEMS IN GAME , INSTEAD DON’T BE DOING FACELIFTS ON THE WEBSITE AND BLOG


  8. topmate says:

    Kudos to the Linux Users group who warned each other in world that problems were occurring.

    Funny that, in world warnings are actually quite useful.

  9. Namarie Merlin says:

    GREAT !! It’s the first time that we have these problems !! … The only thing that work always well there is our payments for tiers !

  10. Walentine Andersson says:

    Wow, is the asset server down?

    This must be the first time this ever happend to SL?

    Maybe you should change from Commodore 64 to Commodore 128?

  11. Funny says:

    Ok, you are aware that this is getting ridiculous do you?
    Instead changing CEO, bringing in GLOW in LSL, and all minor stuff like that try do something with freekeen asset server as it crashes more than windows itself…

  12. Kewl Planer says:

    maybe you guys should look into spending more than $20 for the hardware you are running the asset server on, ya know, just a thought.

  13. Raven Primeau says:

    Ho hum, another weekend of SL blues

  14. Fleur Rasoir says:

    I am very worried about what is going to happen to my house right now, I put it down prior to reading this post and then I picked it back up, but it didn’t go into my inventory and it seems as though it is still down on the land (but now invisible, because I can’t leave the area my house WOULD be in) so have I lost my entire house for good because of these problems? I hope not.

  15. Rhonda Nessen says:

    **For those who pay**

    Hmmmm maybe you shouldnt pay if you stay so unhappy? Problems are going to happen….. so why are you paying for something that you obviously dont think is worth it? Im sure i will get some hard remarks for this but think about it, seriously! You hollar, complain about loosing money, being a paying customer and getting “crap” in return. D O N T P A Y!!!!!

    **For those who dont pay**

    Is this sl necessary to live?? I think not, yes its fun and addicting, hell Im addicted too, im on here every moment when I am not at work during the week and all weekend when sl is working. But seriously venting hateful smart**s remarks just because you cant get on????? WOW! It’s a scary thought that a big percentage of people on sl are in the age range of their 30’s (at least that is what i have been told) and act that way 😉 At any age really……

  16. Alyx Sands says:


    So moaning and whining will help? Since when? And a Commodore 64 didn’t even have a hard disk. I still *have* one in working condition.

  17. zoha boa says:

    @7 “JUST another SADDDDDDDDDdd old day in SL for the weekend LL, last night was bad enough and no my sister has no clothes on ….”

    Dam, And i can’t TP to see her!!

  18. Jenny Carlos says:

    Pffft have had problems reading , Saveing scripts and haveing items revert to older versions ALL day long now.
    Your just now starting to see this lol.

  19. Dianne Davies says:

    Thanks..for letting us know but I really believe “inworld” notifications are very important..for everyone..and I agree..stop “tweaking” the pretty bits here in SL and get the foundation steady..

  20. Rhonda Nessen says:

    Teleporting is already working again and it hasnt even been 10 minutes, you people are wayyyyy to impatient!!!

  21. Meganutter says:

    however, this usually is a recurring problem on the weekends, it start friday evening, lays low for saturday, and all of a sudden pops up again saturday night..
    really od…

    oh, i know, maybe it has to do with user/server load relations?!
    yeah, take a look at that

  22. Funny says:

    @15 – Rhonda Nessen
    Thats because you never actually tried to build something in sl, or write more than 2 lines of LSL…
    If you are stucked with project with 218 prims, and 3000 lines of LSL code, you wouldnt be so calm, and trying to sell us that stuff you do..
    “oh if you dont think life is fair – kill yourself” thats your analogy for this right? oh how mature isnt it


    pathetic incompetent techs…. please do us all a favor a FIRE THEM! 6 weeks of servers that beak down daily should wake you all the HELL up FIX THIS ISSUE ALREADY!

  24. Anthony Hocken says:

    Think it’s fair to say that SL is fundamentally broken by design because this has been ongoing for so long now. I can’t see there being a stable robust virtual platform with the richness of SL until we see a competitor come out with a better design from the ground up. Maybe Sony’s Home might evolve into something worthwhile.

  25. SecondLife Is Just A Sad Place Anymore says:

    @15 , shut up , I pay to have good service which in the past 3 yrs being in game we did till they lefted dmc , zyngo , and started to update more , bring in game like a new sun dumb things , because you NON PAYING MEMBERS WANTED THIS OR THAT , I pay because if I don’t I lose my 2 sims I own . So before you know all the details I would advice you to shut the heck up , put a sock in your mouth and chew it up . If you don’t pay for SL , I feel you don’t need to complain about the poor service in game . PLAIN AND SIMPLE . SHE IS STILL NUDE AND STILL CAN’T PUT CLOTHES ON LL PLZ COME ON NOW FIX THE PROBLEM … Quit eating donuts and drinking that nasty vendor coffee lmao

  26. Bill Schmo says:

    when are we gonna see the light at the end of this tunnel.. 3rd time in less then 24hrs that i know of, and the first time cost me 2hrs worth of scripting and ghost prims stuck in the sim… which are still there o.0 come on guys if the hsptl systems I work on all day did this i would be out of work, and the company would be out of biz X(

  27. Rhonda Nessen says:

    @22 lmao I have built things and scripts

    from 22…..”oh if you dont think life is fair – kill yourself” thats your analogy for this right? oh how mature isnt it”………..

    WHAT??? are you serious?? lol

  28. Lady says:

    Ohhh, i thought i was the only one having trouble (since 4 hours ago..)

  29. Anna Shinohara says:

    Trouble is people in RL not doing SL better. Just making R$.

  30. Liny Odell says:

    I should also add that abuse reports are going to have a tendency to fail because the database is down. That isnt a good thing to have happen.


  31. Samuel Geiger says:

    Multiple studies have been done in the MMO feild that suggest that it takes roughly 50,000 PAYINGsubscribers at that usual 14.95/mo mark just to break even. Now, we have LLthat might be lucky to have 1,000 paying members at 9.99/mo. So, when are you guys finally pulling the plug? lol

  32. hugsalot says:

    Even the competitors have the SAME EXACT PROBLEMS. Hell even IRC networks have the SAME EXACT PROBLEMS. So please people shut up already. Wake me when the Lindens stop running bit Torrent seeds on the asset servers. Thankies.

  33. hugsalot says:

    Monthly fees isn’t the only source of income that LL gets. Even from basic members they get a nice wad of cash from the Lindens I earn and later sell on LindeX. So please shut up again, tired of the ignorant whining.

  34. i really wish you would play this game instead of just takin our monies.Maybe then you would know just how we feel when your system goes down and we spend countless hours attempting to get in and you dont say a thing. But not to worry you’ll most likely send us a questionier asking us if the service got better or not.this is one unhappy camper.

  35. SecondLife Is Just A Sad Place Anymore says:


  36. Linsey Nolan says:

    Just downgraded my account to the free, i will NOT pay for this any longer. i will sell my land and NOT pay any more TIER, i’ve had it with SL !!!

  37. la le lu says:

    i don’t pay but as a pro scripter i pay my living from the jobs i do in LS. i fear the bad performance head me into another direction sooner or later. in fact: the most LL techies who don’t do their jobs should fear for their positions and also the rest of the LL should fear for the future of LS.

    hipihi is upcoming, twinity is upcoming othe MMO’s will follow. the time for LL gets shorter. they now should find the right pace on the market or it will close at the end.

    as a unix/linux engineer since some decades i don’t like to switch to something else with no linux support. my only escape direction might be cobalt, a http://opencroquet.org/ split and/or http://opensimulator.org .

    LL, really. we all cook with water. ask for help is nothing you need to feel ashamed. but actually people starts laughing at you. that means,
    they don’t take you and your efforts not serious anymore.

  38. Funny says:

    @ 27 oh yeah LMAO
    like what? llOwnerSay (“You are beautifull and smart girl”); right?
    point is that facelift of website and blog is more important than asset server that HOLDS EVERY TINY LITTLE ITEM IN WORLD.
    So thats normal?
    and whats wrong with my example?
    well if you dont like it, kill yourself, same as yours “if you pay, dont pay anymore, if you didnt you dont have any rights to complain” you think yours is better…

  39. AWM Mars says:

    Shopping List for the New CEO:

    1) Stabalize the platform.
    2) Stabalize the platform.
    3) Stabalize the platform.

    Miss anything?

  40. Anna Shinohara says:

    By the way… I am naked too… sob

  41. Tiara Calvert says:

    This is just getting overly lame. This place has declined, steadily for over a year. And every damn day “something” simply doesn’t work. Every weekend there are huge issues. We complain because unfortunately there is no other place like it currently and most if not all of us have invested time and money here. This is a cascading failure in front of our eyes. And no amount of contunued failings and or customer complaints on the issues that matter, seem to be heeded. Sickening.

  42. Asset Server Fails every weekend says:

    6 weeks of servers that beak down daily? Please check back the log until end 2006, when i lost 150 USD when the asset server broke, so unfortunately these ‘Issues’ are here since forever.

    Linden Labs never refunded that money btw, although admitting it was due to an error in the software a year later that assets went missing from the database -.-
    Basically simplified it’s like this: You give me stuff, I throw it in the ocean and say to you: Yeah, sorry it’s gone, but do not blame me, because i have a paper here that says that you cannot blame me for it. Would you punch my face, or? Well, i would and make you pay for every penny i lost.

    So this papping and nursering, without actually being decisive and totally ignoring the losses of the residents, is building no goodwill.

    I think now would be a good time for M Linden to step in and do something about the prolonging asset server issues.

  43. Midnite Rambler says:

    Ho Hum……..used to be that my morning routine was

    Get out of bed
    Make morning coffee
    Log in to SL and relax with coffee

    For the last month or so it has been –
    Get out of bed
    Make morning coffee
    Check blog to see which issue it is this morning for why I can’t log in
    Find something else to do

    /me cries

  44. um, is an incapacitated IM facility inworld an asset server issue? i can’t seem to find my IM window, at least not since the new viewer update. it just doesn’t show if i click 😦 or is there a gagging order on me and my inworld gobbinesses?

  45. Jessica Hultcrantz says:

    OK so that is why it is impossible to get dressed then *grins*
    Yep, SL keeps this girl naked at the moment *laughs*

    Why wasn’t there an in-world broadcast about the asset server failure ?!?!?!?!?!

  46. Jessica Hultcrantz says:

    oh yes… forgot one thing… ™ !

  47. iliveisl says:

    yeah i like it too when you allow comments. thank you.

    now i have a very diff experience than most it seems. sl has been getting better. i have not had a crash in several months. i dunno if it’s cause i have a decent machine (like $600, but put together myself) or if it’s the “pipe” i have (fibre optic – way fast). but sl works very well for me. graphics and bandwidth always set to absolute max

    in case i do experience tech difficulty, i switch to rl and have a martini! LOL, usually that tech issue is my scripting!

    thanks LL, i am cruising along and you guys have given me an opportunity to make additional income. i find it hard to fault you on that! 🙂 *looks for olives*

  48. How Many Excuses Can You Invent? says:

    Now we are experiencing crashed every five minutes! Is someone actually running SL these days?

  49. Funky Monnett says:

    the asset server broke??

    O RLY ?

  50. Anna Shinohara says:

    It’s fun watching payers ask non payers to shut up. They don’t want us pushing to make things better. Sweet things…

  51. hugsalot says:

    My point is that even non-paying monthly members also contribute to the income that LL gets. Like people who buy and sell Lindens on LindeX. So just because some of you folks own land and pay tier every month doesn’t make you superior to everyone else.

  52. Raven Primeau says:

    Bitch slappin! Yayay! Reading the hissy bitchy comments can be more fun Than SL these days. So please boys and girls, give us more *Nurrnurrdeenurrnurr* posts

  53. Annabel Vuckovic says:

    …and I am bald…..

  54. Anna Shinohara says:

    hugsalot, yes. that’s me buying clothes and renting a bit of sand and beach. And I have a voice.

  55. Non Paying User says:

    All those so-called “issues” are annoying enough, but why oh why do they have to stirr up some kind of paying vs. not paying customers fight?

    Of course I see that it is especially hard for people who pay fees every months instead of buying L$s just when they need them. But it is not the fault of the basic accounts that LL(tm) hardly values the premiums. I think such thoughts only poison the atmosphere… and speaking for myself, if I see a business owner demanding restrictings on basic accounts, I’m really not inclinded to support their business as a customer, but go find another shop.

  56. carmichael says:

    omg all my groups disappeared……come on lindens,u had 3 days already of failures,fix it,no excuses anymore

  57. Burgess Miles says:

    …and please realize that Havok4 has only brought havoc. Some say that sims crash less…well, I don’t know about that, but I do know that if that is so, then it has been at the expense of just about everything else experience-wise have become worse, much worse. More lag, much jerkier movements, harder to control camera, a lot of scripted things not working…I could go on and on….

  58. Tobias Novi says:

    “The problem has been traced to a faulty piece of equipment and we are attempting to have repaired or replaced.”

    Oh please , don’t try to “repair” it … just change the blown up hard-drive , instead of try to find the right turnscrew and fiddle around for hours..

    And no , these is not sarcasm , if I look at what happens the last months I really start to think you guys really sit there and try to do such things..

  59. carmichael says:

    yes totally agree

  60. Alan Radius says:

    more and more and more database asset server issue ……
    SL is now running at 50% of the time but i pay for 100% and is expelsive for this bad quality of service !
    I propose that all SL men and women post in games blogs how SL run every very well 😦 …………………………………………………………….

  61. JYM says:

    To those who sit here and complain about the issues. Why are you still using the service? Those people who are Premium Members have a choice to stay or leave if your unsatisfied with the service than leave. Let LL / SL do their job as they always have been. I’m surprised LL hasn’t came out with the ToS to where people who keep complaining they get suspended or banned as I’m sure they are annoyed by complaints they do receive when they would rather hear how to make the product better or at least try too. People there is a First Life and last time I checked it’s not that reliable as Second Life 🙂

  62. Waldorf Bing says:

    “We are investigating intermittent database and asset server issues. These may affect logins, teleporting, viewing scripts/notecards, and rezzing items in world.”

    We have this message all week now ! The problem is resolved, and the problems come back…..

  63. Sinuna Falta says:

    Good that comments are open…
    Better to let the anger out than try to gag everyone and hide under a rock.

    one question which I don’t expect anyone to answer.. but what the heck…

    1.) Is someone with a last name of Linden EVER going to stand up and say. “The problem with our asset servers is ….. and ….. is what we plan to do about it”?

    One thought… Even if no one is going to come clean on this 6+ month old issue… won’t you please stop pretending like it’s a surprise evey single time? It’s truely insulting.

    Thank you
    Sinuna Falta

  64. Twinks says:

    I love SL, it gives me opportunities that I would never otherwise have to meet people around the world and get creative in so many ways. It is hard though, when you put money into items just to have them dissappear because of server problems. I understand that it happens, but I know people who have lost everything. I think they need to give us a stippend when ever the server goes wonky. That way we can make up a bit of what we have lost, or at least be compensated for our inconvienience.

  65. Asset Server Fails every weekend says:

    This is 2008. 😮
    Where redundant hardware and software is common.
    In other words, quite impossible to have failing hardware to mess up online services IF you have your hardware/software setup done the right way.

    So, to me this says something… and it ain’t good.

  66. carmichael says:

    this is not cuba or iran where ur killed for your opinion # 61….perhaps you should move to another game that would support such behaviour..we can say what we want to say and personally i actually complain very little and have critized other who have but i see why they are so upset now,lindens are great innovators,i agree but this is getting ridiculous

  67. Ashwyn says:

    This just sounds like a mysogynistic attempt on LL’s part to keep girls naked! Freakin’ chauvenists!

  68. Linda Brynner says:

    Maybe you should need a CCO.
    Chief Crashing Officer, LOL hahah.

  69. carmichael says:

    hahah i meant move to a game that supports banning you for ur opinion not murder lol,but u get what i meant

  70. la le lu says:

    i also like to know what piece of equipment they mean?
    soldering a cpu or what? should all be fault tolerant and every little router comes with snmp these days. they should just monitor and replace the litter.

    also they could run a monitoring bot to recognize in-world problems and bring them to the nagios pages.

    but realize. it’s weekend and the staff is short. just 4 or 5 lindens in-world right now. don’t know whats behind the curtains. thats to the fact of the propagated open communications..

  71. +1 For Weekends says:

    Its gonna happen again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND again AND AGAIN!!!!!!!!!!!!!!!!!

  72. SoulsurvivorTOSP says:

    It affected the Lindens transaction too.. Never receive payout when suppose to..

  73. la le lu says:


    i like the lindens to call a ‘slim your inventory’ week. get rid of unused textures, objects and landmarks. maybe that helps… up to them..

  74. Bambee Pelous says:

    REPLACE IT. Don’t bother wasting time to repair it REPLACE it THEN repair the broken one.

  75. Cat Cotton says:

    *raises and eyebrow.

  76. Claudine Chantilly says:

    Twelve months ago Cory Linden said:

    “There are other errors that could manifest themselves as inventory loss, such as dataserver stalls, failure to complete transactions correctly, failure to properly cache the inventory on the client, or failure of the asset system to deliver an asset pointed to by inventory. Two separate teams are examining these problems, and there are numerous fixes either recently rolled out or in QA. The first team is focused on fixing existing bugs, while the second team is developing a more robust, distributed transaction system. We are also reviewing the design with outside developers familiar with building transaction systems. As we get farther into the design process on the next generation transaction system, we will publish the spec as well.”

    So, perhaps someone could tell us what that second team has been doing over the last year, please?

  77. Ronald Dunheved says:

    Maybe we all should just be happy.
    Very happy !!
    Very happy that the LL techies are LL techies and not Boeiing techies……

  78. Bunnie Pelous says:

    SL is arfed? I am shocked and appalled. Ok, not really :p

  79. Anna Shinohara says:

    Let’s all go march naked to Linden Island… lol
    I am against smile and swallow philosophy…
    Hugs to all, included all Lindens, of course.

  80. alf lednev says:

    #10 LOL, though i i think they use a Vic 20 and need to upgrade to a Commodore 64. A 128 would be their fantasty or perhaps an Amiga 500.

    If its hardware, they have had months to buy a new one, unless the bean counters have vetoed any expenditure of course till next financial year. IT Geeks talk tough but roll over to accountants. Neither should be allowed to run a business under any circumstances. New CEO, a job for you.

    If its software, again they had more than enough time to fix or hire a consultant to fix. I love all the wannabe IT “experts” in here, defending Lindens from a techo point of view, they have NO clue of business nor indeed of project management.

    It’s easy, SL fixes the asset server, replacing hardware and replacing software. In all the months they have been told (vocally) of this issue, they have just applied bandaids and ignored the real concerns of Users. This minor change this week of finally allowing blogs to be open is itself meaningless as they don’t bother to reply or explain.

    Hint! Reserve slot 150 for a summary by the Linden posting the blog, to tell the mushrooms what is going to happen, what has been done,. Don’t pitch at the klingons in the basements but to real people and don’t talk of “patience”, there is none now.

    Start thinking marketing and upping the stripend, put some money back into the SL economy, its deader than a Linden’s customer service class.

  81. Prospero Linden says:

    The problem with the asset servers is a bit complicated. The root of the problem is that it’s a huge data store that needs to be rapidly accessed by eighteen thousand different running processes on four thousand different hosts with a minimum of latency…. That’s not an easy problem to solve.

    Our asset servers are based on an external vendor’s product. It’s a multi-machine system that has enough redundancy built in so that one or two machines can go down without any loss of data. (It’s not RAID, but the result is similar to RAID.) Every so often, we need to add more nodes to the asset server because we need more storage space. Also, every so often, we have to call in the vendor to replace memory chips or similar things in the asset servers, because all hardware will sometimes have problems.

    The problems the other day started when, after maintenance of one node, when the node came back online the entire cluster froze up. The vendor hasn’t told us exactly what happened, but the asset server came back online after we hard-rebooted all of the nodes. The problems we’ve been having since there are something that the vendor is looking into. They got more severe an hour or two ago, and the problem has been escalated at the vendor that provides these machines for us.

    What we’re doing about it: we’re working on optimizing the way we use the asset servers in two ways. First, we want to make it so that lesser-used assets get stored in a different way so that we won’t need to add nodes to the asset server as often. Second, while we do have a backup mirror asset server in another data center, at the moment it’s not possible to do a live fail over to that backup server. We are working on making that possible. (It’s not trivial to do, and will take some time to complete.) When that happens, it will both allow us to do some more “down time” maintenance without having to take SL offline for days, and will also mean that we’ll have an option to get things running again rapidly when the entire cluster freezes up.

  82. Maybe the Lindens should call the Ghostbusters:

    If there’s somethin’ strange in your neighborhood
    Who ya gonna call – ghostbusters
    If it’s somethin’ weird an it won’t look good
    Who ya gonna call – ghostbusters

    But maybe salvation is near for Second Life. Read the breaking news in my blog at http://villainbaroque.wordpress.com – lol

  83. Prospero Linden says:

    Oh, and, by the way, you *could* get a hard drive for your Commodore 64…. Creative Micro Designs had 10MB and 20MB models. 🙂

  84. Asset Server Fails every weekend says:


    I agree about the inventory, BUT, as a scripter I make backups every 30 seconds or so of my objects, because i lost so many hours due to ‘issue’s already. Soooo, I know it’s not helping but fook it, I make a quadrazillion backups to be sure i don’t lose work where i put many hours in. So yeah, 20k items and growing, but until i see some stability in SL, I will not cleanup my inventory. Everytime i ‘Take’ an item from world it’s always a surprise if it returns to your inventory or not, and this has been since I have been in SL, more then 2 years now.

    Personally I have never seen a service with so many users where the backbone of everything is items/assets and a possibility those assets can get lost into the void. That in a sense is a new experience and totally unexpected one nowadays.

    Lag? sure np.
    Crashing? ok, badly developed, but can happen.
    Inability to access the database? hrmmm, are we going back 20 years in terms of hardware/software, where redundancy was uncommon?
    Losing assets (that cost REAL money/time)? No, that is totally out of the question, should not happen.

  85. Tegg B says:

    “She can no hold at 64000 for any longer kaptun”
    “It’s physics Jim! Those TP’s are bad news…….”
    “Eject the bots Mr Spock and set the blog comments to 0%”
    “Which ones are the bots Jim?”
    “Send a RedShirt team of newbs in to find out!”

  86. Kerik Rau says:

    @ JYM

    You can’t just “pack up and leave” on a whim. Often times people have several hundred dollars worth of assets invested and to just leave would mean abandoning that investment.

    @ la le lu

    opencroquet looks very interesting. Especially when you consider that LL tries to market Second Life as a “business” platform to many companies (I am sure they enjoy crashing every 20 minutes randomly during meetings).

  87. I’m having issues of my own objects not allowing me to change permissions. A linked object of 90 prims that i created cannot change permissions…even when unlinking and inspecting every prim (no contents, all), permissions cannot be changed.

  88. Rooke Ayres says:

    @ #82 – Prospero Linden

    I don’t envy you guys at all. At best, SL is a monstrously complicated system.

    Although I have been critical at times ( Wha’? Who me? Naaa! 😛 ), you guys are doing an amazing job at something few people could even comprehend.

    Keep up the good work, and don’t let the problems get you down.

    Hang in there. 😀

  89. Ciaran Laval says:

    Guys I work in support, I’ve been through periods where everything you touch turns to …well the opposite of gold, but this has been going on for way too long.

    If it’s an issue with the external company then it’s time to flex your muscles, this really can’t carry on like this, someone has to get a grip.

  90. Zarah Aichi says:

    ya know… as often your hardware breaks down it almoust sounds like you buy the second hand from the LOWEST bidder… or maybe old russian stuff… i wish you could take back the wednesday downtimes again becouse then it worked a day or two afterwards. now it keeps break downb 2 HOURS after a “update”… so please.. stop buy lowbudget items! PLEASE!!!

  91. Car says:

    Well at least this explains why my “body parts” arent loading and SL is telling me I need to contact someone because of it…

  92. Tobias Novi says:

    🙂 finally some information that’s worth to called information …
    and a bit of hope that someone rethinks about the concept of outsourcing the elementary things … oh wait , who works at the land store ?

  93. Maruko Sakigake says:

    I trust in LL! A day all will be nice adn working^^ The guys need only time. Im curious to know how many Tera of database they managing.

  94. Crap Mariner says:

    This is when I thank my lucky stars that I’m too busy with other things to do.

    Ahhh… springtime chores and honeydo lists! Never have you been so pleasant to read!

  95. Bambee Pelous says:

    I fail to understand why the fail over to the redundant asset cluster isn’t possible. Shouldn’t this have been addressed previously. during say, I don’t know… like any one of the 100 times this has happened in the last 6 months?

  96. Ivana Pawlowski says:

    Gawd, have you not heard of redundancy? Holding spares of critical pieces of equipment? Please Linden, this is every friday and saturday night, some of us lose a LOT of money becuase people don’t trust transactions any more.

    The equivalent of “we sent someone to the hardware store for fusewire” doesn’t really cut it.

    Consider building some redundancy into your systems before we all lose the last bit of faith we are hanging on to and the other sort of redundancy happens.

  97. …after all, it’s just an online game – right?

    Folks here tend to forget that occasionally, I think.
    Tell me that you never experienced anything to go wrong in RL. SL just imitates that behavior 😉 And yes – I am a merchant too.

  98. la le lu says:


    thanks to don’t tread us as children ! thanks to let us know what really happened. this give back a bit of confidence. i hope you get this hot switch between the data centers ready with highest priority.

    and yes, maybe this could be one of katt’s first jobs, to initiate a ‘slim your inventory week’.

  99. ActionJackson says:

    OK, Lindens, it’s time to admit you can’t fix these problems on your own.

    PLEASE go out and find yourself some experienced, smart, capable technical consultants to clean up this mess.

  100. Harmony Deschanel says:

    @33 How do you figure that Linden Lab charging you 3.5% of the L$’s you cash in is making them money? You pay that charge in L$’s not US$’s …. So they’re NOT making anything off you SELLING your lindens, except for maybe the $0.30 charge the charge the buyers.

  101. R. H. says:

    Great Timing!
    I was just about to idle away a few hours inworld too, after spending the afternoon in usenet… not that this forum is any different than the ever present flame wars there :p
    So…let’s see if there’s anything worth watching till the premature
    [RESOLVED] posting

    Cya all Tue when this SL™ networked virtual world gets kickin’ again
    all around
    /me mutters where are those olives?….

  102. just me says:

    if you need to get rid of ghost prims, set em to temporary

  103. Bambee Pelous says:

    @102 that would be great if they could be edited.

  104. Selkit Diller says:

    Kids, before you get too screamingly upset, anyone remember SL 1.7? There’s a reason why we were distributing “SL 1.7: I See Gray People” T-shirts during that. Kinda ironic, given that the shirts almost never loaded. What’s gotten better, (in comparison), *IS* the asset cluster, as unbelievable as it is to say that.

    However, what hasn’t gotten better is an incredible number of bots eating resources, and a colossal influx of residents that came faster than the Lindens could support them, by a poor decision to make all account signups 100% free. Now if they’d staggered signups, or even reduced the fee to a token buck, just to keep the bots to a reasonable and accountable level (Used to be six accounts per payment method, I believe), I doubt we’d be in this present mess, but hey. Hindsight’s 20-20.

    Good luck on hot-swapping a system this large and complex; I don’t envy the engineer responsible for that.

  105. Renee Faulds says:

    So Prospero Linden, in the mean time all the businesses in SL will go under. This in turn will help the asset load because hundreds of creators will have just plain left.

    M Linden – how about some compensation on all our tiers because nobody can buy anything – REMEMBER this is WEEK 5 – 5 STRAIGHT WEEKS OF THIS !!!

  106. Maxwell Faulkland says:

    To all of you at Linden Labs, I’d like to bring this thought forward. If you have yet to use IBM Bladecenters for server hardware, why don’t you invest in picking 1 or 2 of those up. I’ve constantly seen ads run for those things on TV, and who knows….they just may help SL run a tad smoother than conventional server hardware. Leaving you to teal with less of the problems like what has been recently plaguing you.

  107. Maxwell Faulkland says:


  108. Darien Caldwell says:

    Nice comment Prospero. Detailed enough to be informative, and yet not so detailed it makes your frontal lobes throb. 🙂

  109. Bayou Miles says:

    ok i’m sure you all have noticed that over the past year, the PLANNED interuptions of service are now scheduled for non-corporate peak hours? the big money boys and girls are in SL now, so we had better get used to things breaking down on weekends and evenings when the important boys and girls have already packed up and gone home for the day.

  110. So tell me if this Paradigm changed has been considered.

    All assets never leave the client unless that client’s Av gives that asset to another Av as no MOD?

    New assets which invariably may never leave possesion of the AV that created them reside on that client’s hard drive.

    All searching for an asset starts with the client’s own harddrive.

    All new assets default to public ownership unless the creator specifically declares otherwise.

    All AVs who come into contact with this public asset gets that asset via a transfer to the AVs client via a peer to peer (not from any asset server).

    A large (maybe even majoriy of assets) will never reside in an asset server.

  111. Sascha says:

    Well a hierachical file system makes sense, I mean if all 13 mio residents have 1k objects, mainly wooden genitals, and logged in for say 2 hours and never come back, store it away from the main asset servers. Or simply move it to the bin :P. It looks like a metaverse of trash for me now.

  112. River Ely says:

    you would need the hard drive to configure for every texture you might come across in SL, wow, there would be so many copies, and what if they are changed while you are off line, updating would drag your viewer to a halt ,,, It would be worse than running Norton av while watching online football while trying to use SL, no way Jose.

  113. Pingback: [UPDATE 4/26] Problems History with Second Life « Damona’s Blog

  114. Harmony Deschanel says:

    @ 81 … Is it possible for some of these replies I see posted in these blog comments to actually be posted as a blog post, making it a little more official than having to come here to scroll down for the yellow boxes, and seeing partial replies here and partial replies there? Write up a nice blog post telling us you’re aware of this problem, this is what we’re trying to do to fix it? And maybe even give us a rough idea of when you anticipate it being over? Also, please link that blog post in all the “In World Issues” blog posts you do from now on? That might help us out understanding more of this.

  115. What would reside in the asset server would be the locations of that asset in world and the client of the creator of that asset which contains the descriptions of that asset.

    Clients desiring the assets they created to be seen 24 hours a day would need to keep their computers running 24 hours a day.

    One of the only way I see SL ever scaling beyong 50,000 users.

  116. Loniki Loudon says:

    Oh, DB down, no biggy as I have had no desire to log into SL and indulge in the crash fest that it has become. The desire to create for no purpose as transactions or TPs fail and business is at an all time low. No, not much incentive to create lately. I am actually surprised to see DB problems with less and less of my friends logging in.

    I am waiting for you people to do the right thing and you know damn well what it is. The bots have to go. You have already passed the threshold where bots are now preventing live people from doing anything on SL. You know damn well the load they place on the data base for your lousy traffic system.

    And knowing that, I don’t blame the hardware or the software and certainly not the database, I squarely place the blame on you for your lousy decisions. You put bots above the people, but remember this, they don’t pay your server fees when that is all you are left with, bots.. Drop your lousy traffic system NOW.

  117. DiSSENT says:

    We aren’t behind this one looks like LL’s bad luck did it for us. We were just prepping for another grid bombing too. Ah well, guess we’ll just leave it alone for today.

  118. Tiny Mind says:

    Yes, the Bots DO have to go. But no one at Linden Lags is man or woman enough to admit that. They are all afraid of their job-security or ‘stepping on the wrong toes.’ So cut them some slack on this one.

    Tiny Rant.

  119. Midnite Rambler says:

    There are many of us I think that would rather LL take SL down for a few days and REALLY fix things. Instead of the ad hoc when something breaks way it is lately.
    A few days pain is better than a month of daily issues, and the lack of confidence this has brought about.

  120. The Todd says:

    Thanks for the updates guys, and the hard work on the issues

  121. Sharon Breyer says:

    maintenance , maintenance , maintenance!!!! i agree Midnight .. shut it all down till its fixed right instead of being used as the damn guinnea pigs in game .

  122. The Todd says:

    and by the way…. the Commodore 64 rules 🙂

    (Sorry for the double post)

  123. Loniki Loudon says:

    No, I am not going to cut them some slack on this, SL is almost unusable. Do you have any idea how well SL would be working with 20k real users minus the 40K bots? If they have to keep the bots, then create a special bot class that disconnects them from the database with no inventory, but sheesh, something has to change here.

  124. Thomas Weiger says:


    Thank you for the kind explination.

    People, yes the crashes suck and yes I realize that running a business under these circumstances suck, but we’re dealing with a monster here.

    Linden labs owns and operates the worlds largest infastructure of its type (built upon Debian linux, YAY). It’s a massive, complex system that WILL have operational glitches.

    Perhaps the Lindens should do some R&D on some propriatary technology to help their situation. Though, doing something like this would cost millions of dollars and countless man hours.

  125. Asset servers may then be distributed on a sub-region location specific basis. They would contain asset locations local to that sub-region.
    As AVS invariably re-visit the same regions, those region’s public assets will be stored on the client of that AV. Never needing to access the asset server for them.

    Users will be encouraged to make assets they create public, in the knowledge that otherwise they will not be seen by other AVs unless they keep their client running.

  126. Hewitt Huet says:

    @85 : ROFLMFAO!!!!!!!!!!!!!!!!!!!!!!!

  127. Shockwave Yareach says:

    Once again, SL is pointless to build in, work in, try to visit friends in…

    I love this game when it works. But that is so infrequent nowdays. Please, whoever is out there who can… distribute the database to every person’s home server. When Yifu Linden walks up to me, my viewer should ask the User server which server Yifu’s stuff is in, then Yifu’s server feeds the viewer with data. That way there isn’t a single database so overloaded that your customers are close to revolt.

  128. Get a new provider? I’dunno just a thought…

  129. Sgeo says:

    @118: Any plans on how to get rid of bots? They connect to SL the same exact way that regular users do. Actually, what would be cool, although it would be a bit late to do something like this, is what 123 suggested, which is what is done in Active Worlds. A special type of user designated as a bot. (Actually, AW doesn’t even give bots special logons, they just log on with the priv. pass of the user and any name). However, I don’t see how such a system can work given the infrastructure already in place.

  130. Traverse Janus says:

    Maybe ths will get through to Linden Labs:


    Revert te whole thing to pre-Havoc 4, pre-Windlight, pre-Voice. Notice we are now unstable with fewer users online than before? Its all the crap that has been added that is breaking your data and asset servers.

  131. Cat Cotton says:

    1. ”
    Our asset servers are based on an external vendor’s product.”

    2. ”
    Second, while we do have a backup mirror asset server in another data center, at the moment it’s not possible to do a live fail over to that backup server.”

    3. Here’s your sign.

    It’s your product don’t blame your bad business decisions on the vendors you choose.

    Failure to rez/asset problems have been going on for a lot longer than you have had “external vendors”.

    You know I love SL but don’t feed me this line, I don’t buy it.

    I now agree with others; we should be compensated for down town as per your own TOS. Which I HAD to agree to before I could access MY ASSETS on Your product.

    Get threw a month and I might start believing your product has staying power.

    But I won’t be sitting around holding my breath;

    Microsoft bought out Truespace/caligari; Get a clue.

    Grrr another down weekend; more money lost.


  132. Cat Cotton says:

    *Time not town.

  133. Zandor Maltese says:

    it might help to ban few few alt accounts i know some people who make 7 or 8 accounts just so they can put them all in world and camp.

    beats me why they do this. but i do know all these alt accounts are taking up resources that could be used by people who are actually in world using the service.

    i don’t buy things in world any more because if the server loses them LL will not replace them or refund your loss.

    i all so down graded my account to a free one till LL gets the stability issues worked out in the 3 years i’ve been a member they seem to have gotten worse over time.

    you crash then ghost on the server what that means is the server does not log you out. so you have to wait to log back in. i could over look that if it weren’t for all the other issues.
    @ 47 no it’s not my computer i’m a hard core gamer so i do a new build ever year or so. the system i have now should have no problem running the SL client.

    i am running a liquid Q6600 over clocked to 3.9ghz with 2 gforce 8800GTX in SLI mode. 4 gigs’ of corsair dominator ddr3 ram.
    and a WD raptor 1000rpm hard drive
    with a 20mb/ps connection so i can say with confidantes it’s not that my computer is to under powered to run the SL client.

  134. Tom says:

    I remember 2 years ago when 20,000 users where on this asset problem arrised basicly Platform Maxed Out at 20.000 users online. But has now seemed to have reached that same point again now there are nearly 70,000 users online and its maxing out with the load can you guys try to limit the number of Bots in any way at all maybe by code or something because as I see it 1 ip connection can use 40+ Bots compered to normal users that just use the 1 veiwer connection. I know its open Source But maybe its time to start Closing the Source otherwise its going to get out of controll with users running Bot after Bot and Hammering the grid Just a Idea but you need to start to limit the amount of connections 1 user can connect to the grid 3 seems a fair number per ip adresse

  135. Galt Hax says:

    Seems to be not resolved at all. Come on guys.

  136. Katerina Kirax says:

    Even after 6 months I’m still a newb – please educate me as to what the purpose of these “bots” is?

  137. mcp Moriarty says:

    Someone needs to vigorously clean out the dead accounts. Also users with more than 2 accounts, I’ve seen a lot people doing this. Pretty sure the TOS says 2 per user max. It’s time to clean up the database. Put a team on it and get rid of the dead weight.

    They eliminated the ratings because of database issues and that was at the most 6 32 bit longs, or 24 bytes per account. Imagine what cleaning out the accounts could yield. It certainly could not have a negative effect.

  138. Tom says:

    @ 136 Katerina.. A bot Is A term for the veiwer code that can be written to do things automated so basicly people can buy land without even being infront of their computer or even camp basicly you can automate your avater to do things inworld even though your not at your computer and its a light weight code so therfore many bots can be run from the same pc

  139. Kewl Planer says:

    only 3 hours to fix? wow, this must be some type of record!

  140. Anthony Hocken says:


    “and a WD raptor 1000rpm hard drive”

    There is your problem right there. You have a slow harddisk 😉


  141. Ryanna Enfield says:

    It takes nearly a minute to send a crash report every time I crash. Previously it would take at the most about 20 seconds. At first I thought it must be my ISP but I’m using a cable connection and my bandwidth is fine. Is it possible that there are so many crash reports being sent and stored on the servers that this is actually affecting performance of the servers used to run SL? Are the servers collecting the crash data separate? I really hope so. Maybe someone who knows more about this can answer my question?

  142. Felix Duesenburg says:

    Greatly appreciate your efforts and the detail of communication. There seems to be a Gremlin at Linden Labs though that makes these things always ever happen on weekends 😉

  143. Can’t rez, asset server not responding, 18:01 SLT, NOT [resolved] at all it seems


  144. Walentine Andersson says:



  145. Hikaru Dreadlow says:

    …The issues are back again in the Turbo area. *sigh*

  146. [18:03] Asset server didn’t respond in a timely fashion. Object returned to sim.

  147. Asset Server Fails every weekend says:

    [17:59] Asset server didn’t respond in a timely fashion. Object returned to sim.
    [18:00] Unable to create requested object. Please try again.
    [18:00] Asset server didn’t respond in a timely fashion. Object returned to sim.
    [18:00] Cannot complete attachment. An attachment is pending for that spot.
    [18:00] Asset server didn’t respond in a timely fashion. Object returned to sim.
    [18:00] Unable to create requested object. Please try again.
    [18:02] Asset server didn’t respond in a timely fashion. Object returned to sim.

  148. Mah…

    [17:59] Unable to create requested object. Please try again.
    [18:03] Unable to create requested object. Please try again.

    Asset server is still down. Stoopid database. xD

  149. BoomSpeed Kumsung says:

    You missed something, it’s acting strangely again now. Sad.. I figured it’d still be fine after an hour…

  150. Miro Collas says:

    “There was a problem saving a notecard due to the following reason:
    Asset request: Failed. Please try re-saving the notecard later.”

    And it is supposed to be resolved?

Comments are closed.