Rolling restart Wed/Thu April 16/17 for 1.21 Server Deploy

[Update 2008-04-16 21:10] Several of the regions that received version 1.21 are showing problems, so we are going to revert them to 1.20.  Many of the regions remain down; they will be back up within 1/2 hour.

[Update 2008-04-16 20:30] The deploy to 490 regions will begin momentarily

[Update 2008-04-16 17:00] We are in the middle of updating the central servers. Note that if you watch the concurrency plots, you will see dips in it as we restart servers that report concurrency numbers. This doesn’t actually mean that people are getting kicked offline, it’s just a reset of the data collection. The deploy to 500 regions will begin later tonight.

We will be doing a rolling restart this Wednesday and Thursday to roll out the patches to the server that were to be rolled out with last week’s cancelled rolling restart. Changes include security patches, performance improvements for Havok4 (including the issue that “openspace” or “void” sims have with Havok4), and code designed to mitigate the load on the central database systems.

We will do this with a usual 3-stage deploy:

  • Wednesday, April 16, 8:00PM : ~500 regions will receive the 1.21 server deploy.
  • Thursday, April 17, 5:30AM : ~1600 regions will receive the 1.21 server deploy.
  • Thursday, April 17, 6:00PM : All of Second Life will receive the 1.21 server deploy; this will take 5-6 hours to complete.

There will be no viewer updates required as a result of this deploy. All regions will receive warnings beginning five minutes before they are shut down. During the rolling restart, regions should be back 5-10 minutes after they are stopped. If your region stays down more than 20 minutes, please contact support.

This entry was posted in Announcements & News. Bookmark the permalink.

150 Responses to Rolling restart Wed/Thu April 16/17 for 1.21 Server Deploy

  1. Awesome I look forward to this and the new land store 🙂

  2. Anonymous says:

    Let’s hope this fixes ||Target Omega and other scripts that no longer function 🙂

  3. Marianne McCann says:

    Is there any list of the specific fixes, or more hints on those “performance improvements?”

  4. Shnurui says:

    Lets hope it also fixes the Adult verification barriers to stop blocking previous verified people.

  5. milady Guillaume says:

    Will this fix the search engine for specific types of events?

  6. Anny Helsinki says:

    ohhhh a open blog.. have the sudents a free day…. lol

    and digital.. ure a walking joke.. love ure posts

    greets anny

  7. Uccello Poultry says:

    The post mentions that a new viewer is not required after the restarts, but will any “fixes” affect any known issues with the Release Candidate viewer or the current standard viewer?

  8. Anny Helsinki says:

    #4, that never will work for europeans…, that isnt a feature,.. only a fault to go agressive against european users.. the biggest group in SL

    why never work? we have free counties here in europe and not a totally one like germany was at worldwar2

  9. Krasker says:

    Blog title dates are off.

  10. Cincia Singh says:

    Adult verification is a good thing #8. Even a “less than perfect” assurance is better than no assurance that I’m dealing with an adult and not someone’s children. And I don’t see it as aggressive, just due diligence. I always look forward to updates because things can only get better over time … they certainly won’t get better by keeping the current code. Ok, time to go play in SL.

  11. Prospero Linden says:

    @7 : this is a server-side deploy, so will include no client fixes. Watch the blog for announcements of updated client versions.

    Re: which performance improvements are included: the primary one is a fix for the fact that H4 regions use more memory than they really need to use. This has its primary impact on openspace regions, which are run 4 per CPU. (Lower resources regions, thus we run more per computer.) Right now, they use up enough memory that the machines they run on (especially for class 4 regions) run out of memory pretty quickly, forcing the machine into heavy use of disk swap space.

    To my knowledge, this does not have the llTargetOmega fix in it, as that one is turning out to be difficult to track down. For an update on that, please see Andrew Linden’s comment today on SVC-1910.

    I will fix the blog title. You think I’d not make that mistake during tax week! Yipers.

  12. Roy Blanchard says:

    Will this fix the recurring breakage of my terra skydiving launch pods and all my rotating objects that keep stopping?

  13. Pingback: Second Life Italia Blog » Blog Archive » Orari previsti dei prossimi interventi di manutenzione

  14. @ Anny Helsinki Says:
    April 14th, 2008 at 3:11 PM
    ohhhh a open blog.. have the sudents a free day…. lol

    and digital.. ure a walking joke.. love ure posts

    greets anny

    ____

    Honey before you try to take me down by saying stupid comments in the blog why don’t you learn how to spell 🙂

    _____

  15. Drake Bacon says:

    And the blog stands still as the grid goes down (secondlife.com/status says it’s down, but no reason why).

  16. Eric says:

    Yet another week of slow sales because on going issues. I know your trying LL but your putting peoples real life income at stake. This is not just a game for everyone. This is my real life income and your slowly destroying it. How about crediting some of the $700.00 a month tier I pay you so I can pay my real life bills since SL does not work correctly? Hell, think about it. I could buy a damn nice house in real life for $700.00 a month and it would function.
    Get-R-Done LL soon please!

  17. Elvis Orbit says:

    @12 Please look at the post before yours number 11. The llTargetOmega problem is the spinning object problem.

  18. Martin Magpie says:

    I am getting the very old and very tiresome error about the asset server not responding in a timely fashion. I hope to GOD that bug is fixed!

    Sorry what I mean to say is; “The dribbles have invaded my sim’s brain again, someone please do let the new Captain know.

    *rolls eyes you want to treat your business like a joke expect others to respond in kind.

  19. Venus says:

    is the grid down…unable to connect to a simulator

  20. Drake Bacon says:

    Okay, back up now…

  21. Naima aya says:

    😀 awesome,

    Did u noticed that SL is down now? Say Unable to connect to (wahtever) lmao.

    Cy in-world when it’s working again 😀

  22. Tarik Flatley says:

    It would be nice if they just fixed the current problems, login’s failing due to unable to connect to a simulator or just timing out, rezzing either takes forever or finally times, TP’s failing or timming out [be stuck in a region for over 30 minutes and maps don’t work either], unable to access inventory, and the list goes on. Contacted support on chat and all they do is scratch their asses and say Duh Dude, nothin’s wrong! Morons!

  23. bigmoe whitfield says:

    patience is a virtue. I have it and will always have it. I meet such a beautiful person here and I am not going to be leaving sl over a few glitches me and her get along really well and these problems dont really bother us much.

  24. Thamala says:

    Hopefully this will fix the IM borkage between sims..

  25. Kelly Linden says:

    The issues with rotating objects (and other moving objects) that stop rotating (or moving) is being worked on right now and will not be in the 1.21 release. We believe we know what is happening in general (objects that are no longer selected still think they are selected) and are testing fixes now.

    The “performance improvement” for void regions is detailed in SVC-2005 (https://jira.secondlife.com/browse/SVC-2005), specifically my comment from April 6th.

  26. Datapanic says:

    Will this resolve the Selected/Sat Upon counts being way high?

  27. Shin says:

    Why waiting so long? This effectively means another week of the Open Space lag (ie, for most of us it will be Friday morning before the rollout is complete). I understand stopping it last week if there was a major problem, and even understand not wanting to do it before the weekend, but why not start this morning? Or tomorrow? If the first part reveals a problem again, it will be next week before you try again. I know the major problem only affects the Open Space sims. Maybe you can roll out that fix to them even if something else will hold up the full update.

  28. Prospero Linden says:

    The reason we’re not doing it today and tomorrow is twofold. First, we needed to make sure we understood the issues from last week and had them fixed before actually rolling code out. That is done, and indeed I believe the updated version is already on aditi. Second, there is the maintenance event going on until Wednesday morning. Breaker work going on at one of the datacenters that hosts our machines will cause machines to lose power for a little while, and we’re going nuts trying to minimize the impact of that. Doing a rolling restart at the same time as that work would be “crossing the streams”….

  29. Eric says:

    #23 that is great and very happy for you. If I was just in SL to play and have fun I would have your attitude but, I’m in business in SL and my main purpose is making a real life living. These issues are having a major effect on my business. A glitch here and there is always expected, but these transaction failures constantly, do not rezz no copy items and the teleport issues. Who the hell wants to buy anything and when they do want to, the damn transaction fails.

  30. Beta X says:

    (Temporary interruption of services) post, i know it’s off topic, i was seeing these myself issues with in-world then login issues

    you can wipe this post if needed, but i wanted to say ty and good luck to you all (LL) lol

  31. Bocan Undercroft says:

    I’m personally very pleased with these latest fixes coming out. Just lately I’ve been able to teleport up to 9 or 10 times in a row without getting kicked out! Certainly better than the 0 or 1 I was living with. Now if my house will quit becoming “phantom-like” at random intervals I’ll be happy… Mind you, I took the opportunity to put a tomb stone down there -very nice. 😉

  32. uh-oh says:

    “Lets hope it also fixes the Adult verification barriers to stop blocking previous verified people.”

    Is it required now? I have been in many adult content places and have yet to see any thing flagged yet. How about a notice of when it is supposed to be used or are all these sims just breaking the TOS?
    I am sure if it is required LL is to busy to notice who is following the rules or not and may require some in world help on this matter.

  33. bigmoe whitfield says:

    I am in business inworld too. But I am not letting any of these effects or problems cause me concern, why should I? there is nothing I can do about system failures or other related problems. so I lose a little bit of money, trust me I recover pretty quick and i do live off of SL too. Linden labs® is one of the best things in the Real world right now and it keeps me happy and I make a little money to live off of.

  34. bigmoe whitfield says:

    to add to my post for 33, I am pro linden and guess that could be considered a “fanboi” yep thats me. Darn proud of it too.

  35. I might be the only one but i thought since the asset servers doing less problems SL was performing quite okay. However not to fix the rotation bug is kind of a show stopper for me because it requires a sim restart.
    Isn’t there a chance that you wait with the new deploy until you got all basic features restored??? We have also been promised for the beginning of April to get control over the group chat.

    Please Please Please, reconsider your schedule, fix the objects and give us group chat control + other basic features, on these 2 days less people will log in, we loose money again and with all due respect all patches before required maintenance the following week before.

    Fixing the basic features would give your coders the chance to improve the current patch, test it to death and the final result would make us all happy!

  36. bigmoe whitfield says:

    I think group chat is borked on the server side if I remember right. something about over 1000+ members in a group causes the problem. Dont rmember the specifics

  37. Shin says:

    Damn..a sensible answer. What’s a chronic complainer to do?

  38. Prospero Linden says:

    Isn’t there a chance that you wait with the new deploy until you got all basic features restored???

    This is actually a pretty deep question, it turns out, and something that we’re balancing every week.

    There’s always tension between rolling out needed fixes (e.g. the openspace memory issue, which we really wanted to get out last week, and database load mitigation issues, which I’m sure you all can appreciate) and holding the release until another feature or fix is ready. The later we allow new patches in, the less testing the release gets in beta, of course; but, also, the later we allow patches in, the more other crucial patches are delayed. The answer isn’t always obvious, and requires judgment calls. Occasionally things don’t work out as we expect, and we end up delaying a rolling restart because we are trying to get something into the deploy that we thought would be possible to do, but turned out harder than we predicted. Naturally, we try to avoid this!

    So, it’s all quite complicated.

  39. bigmoe whitfield says:

    Prospero Linden keep up the great work you and the team do, you guys dont get enough praises but trust me there are those of us that love what your doing even if you are having issues at this time. We love the product and the platform you have brought us. Keep up the good work you guys deserve a pat on the back

  40. Dilbert Dilweg says:

    Progress as promised. 🙂 Look forward to it

  41. Chrysala Desideri says:

    *Doing a rolling restart at the same time as that work would be “crossing the streams”….*

    and what if ZUUL was eating asset servers, would you cross the streams then?

    loved that comment, Prospero 😀 good work and good luck!!

  42. Anny Helsinki says:

    #14,, sorry english isnt my spoken language. woudnt speak against your posts… only find them funny….

    btw, did u speak german free from failures?

    sorry, Digital. love always read ur posts… is that a gesture?

  43. Anny Helsinki says:

    oh Prospero, any idea why the software always try load inactive gestures?

    a answer to my mail would be interessant… because i deactivate most now, and clears cache, installed the last stabe browser… hope u have a idea.

  44. Cat Cotton says:

    Thanks for the honesty Prospero.

    Cat Cotton/Martin Magpie (take your pick 🙂

  45. Verdana Klaar says:

    Anny Helsinki said : “#4, that never will work for europeans…, that isnt a feature,.. only a fault to go agressive against european users.. the biggest group in SL – why never work? we have free counties here in europe and not a totally one like germany was at worldwar2”

    I’m european myself, and do not understand one single piece of what you wrote… wich all the more seems a bit out of topic.

    Some residents are funny 🙂

  46. Chaz Longstaff says:

    llMoveToTarget was broken under Havok 4 beta, but fixed, and worked when Havok 4 released on 1 April 2008. Sometime around 8th or 9th of April, something changed in the OS platform, and it was broken again.

    Does the 1.21 server patch address this?

  47. coventina dalgleish says:

    To remain ignorant of things that happened before you were born is to remain a child.
    o Cicero

    History is the sum total of things that could have been avoided.
    * Konrad Adenauer

    History is a continuing dialogue between the present and the past.
    * James M. McPherson

    He who controls the past, controls the future; and he who controls the present, controls the past.
    * George Orwell

    The first time as a tragedy, the second time as a comedy, the third time an absurdity.

    Those that fail to learn from history, are doomed to repeat it.
    * Winston Churchill

    Just a few ideas for Linden Lab seems all we see are rapidly repeating events from our recent history.

    Attention to detail would also aid in the correction of the many repetitions of the past irritations.

    Again the most Poignant quote in reference Second Life

    The first time as a tragedy, the second time as a comedy, the third time an absurdity.

  48. Angelic Greenwood says:

    OT, but needs to be said.

    TP and searches are DOWN.

  49. Angelic Greenwood says:

    Take that back, saw the times for the maintenance events and got all confuzzled being in EST, now I know it’s not a random “we’re gonna shut everythin’ down and tick everyone off” thing.

  50. Tristin Mikazuki says:

    The server breaker work..is that gona be tonite ? Because is says Monday.. but in brackets it says tuesday.?.?

  51. Angelic Greenwood says:

    It’s the 13-16th, check a couple blogs down and it’ll tell you the specific times.

    TP started working again, but back down now. =\

  52. Dyan Westland says:

    wow once again inworld stuff is not working..is this gonna be a daily thing? should we just be crossing our fingers and hoping to god our transactions go through?? is there some prayer i should say before i try and tp anywhere?? WTFFFF

  53. Urantia Jewell says:

    @51 Tristin,

    As stated: 10:00 PM Monday to 1:00 AM Tuesday 🙂

  54. Angelic Greenwood says:

    Dyan, they actually planned it this time. Check the blog…I think it’s two blogs under this one.

  55. Michel says:

    Can’t login… I haven’t cancelled the login process as suggested, but it’s been on for 10 minutes now… not moving… stuck at only a 10th of the bar

  56. Angelic Greenwood says:

    Try now Michel, was able to TP just now, you may be able to log in.

  57. LoLo says:

    one more day where business will be bad but we still have to pay for advertising… i can understand that things like openspaces wil be fixed.. but please keep unnessecary things like dazzle or more shinies AWAY till our business have been able to at least get even with the payments again. Business in secondlife is like gambling, with all the bugs, non delivered items and IP theft. It is like trying to do business in an underdeveloped country with no government and broken roads.

  58. Sandor Balczo says:

    I would suggest Linden Labs create a complaints section in the website to allow those who constantly repeat the same complaints again and again and again and again to vent their anger while the rest of us try to read the on-topic comments.

    Having said this, I am happy LL realized that the 1.20 server code deployment of a couple of weeks ago is the key factor in the long stretch of malfunctions (including an incredible slowness for mid-range computers like mine) and the current browser with atmospheric rendering is the victim and not the culprit, something the usual and the same whiners suggested somewhere else.

    I still believe we should introduce Intelligence Quotient verification in Second Life rather than adult verification. Some people waste time and effort (of the rest of us) BORING all those who try to come to SL to work and play in peace and some of these people cannot even spell in ANY language. They show an immaturity that irritates me to the core.

    If you are so good at complaining and you think you can fix the problems you so eagerly identify, why don’t apply for a post as a dev in Linden Labs?

    Keep up the good work, Lindens!

    Sandor

  59. I really appreciate the work of the h4 team, especially the quick response after the open space performance problem. Kinda sad having to wait another week for the fix deploy, though. Especially as someone who got his OS delivered shortly before the price cut with no offered compensation. (Well, could give it back at no cost and order a new one once the land store is running – but what with all the work that already was put into the place and that would be lost?) And that for a SIM that since the h4 deploy is not usable for lots of stuff because of the mentioned performance problem. LL should really re-think that “no copmensation” approach – how about a month or two without region fee?

  60. Tintin Blinker says:

    I would like to thank everyone at Linden Lab for all the hard work:)

  61. Al says:

    Ok…lets wait and see… Good luck Linden people!

    I can’t wait to live on my lag free open space Sim.

  62. Pingback: Tonight on ISTE Island: Havok4 Fireside Chat with Sidewinder Linden | The Story of My “Second Life”

  63. wesley says:

    alguém sabe como abrir e jogar esse jogo

  64. Xen Akula says:

    When will we be able to build up to 4096 meters?

  65. @65: We can already – using the RC viewer.

  66. AWM Mars says:

    To be honest, if LL wwere plumbers and they spent 4 years ‘fixing’ my washing machine, I would have sacked them after a couple of attempts, given the complexity of this system, even 4 years is stretching good customer tollerance.

    Given the money ppls are paying every month without any form of compensation for the downtime, loss of earnings (this is not a game for everyone), loss of customers/clients due to inability to provide a service, references to this being ‘Beta’, is getting long in the tooth. 90% of LL’s resources should be working on getting stability…. the one thing I have not personally seen in this platform in the 4 years I’ve been here.

    I fail to understand how old issues keep reappearing, time after time. It suggests that you are not using a central source for the code, but have several versions being worked on at any one time, fixes being not proporgated into other versions, therefore issues once fixed, being reintroduced JMHO.

  67. Xen Akula says:

    @66 Fantastic, thanks!

  68. Actingill Igaly says:

    SL is complex and ground breaking. Anyone who dives into such a world as an alternative to RL work should first realise that they are at the mercy of LL, system failures and constant change. Most people do know this, so for those of you that moan about downtime affecting income: 1st, if someone wants to buy your product or service, they will still want to when the system comes back up. 2nd. If progress and change are such an issue to you, go get a job as a civil servant 🙂
    As i have said in the blog before, change enables LL, and therefore us, to stay ahead of other up and coming virtual worlds; and will inevitably come with occasional problems and downtime. I for one, would sooner be at the cutting edge of this techology with its occasional problems, than a civil servant…..

  69. Royce Boa says:

    I am in Baku, and my skybox arena’s floor disappeared (not just phantom…it’s gone) and a whole bunch of my objects are highlighted red for some reason. Is this connected at all?

  70. Hope this update makes Second Life better and more stable for everyone. The last days were absolutely AWFUL! There was nearly no day where you have not had heavy trouble during European peak times.

    I fully understand that Second Life as a whole is a very complex thing – but: Please work it more seriously… There is no competition (yet), but… there will be one time. And you sure do not want to see your users run away…?

    So, good luck to all of us and them 🙂 Let’s hope for a better future now…

  71. Karine Koba says:

    Will this fix the camera issues? I can’t wait to put back my normal boxing rings with the ropes at my gym.

    Royce highlighted red generally happens when pressing ctrl alt t to see transparency. press it now and see if it disapears. Contact me inworld if you need help.

  72. Royce Boa says:

    Hey Karine..fancy seeing you here, 🙂

    No that was the first thing I checked. It isn’t highlight trans. Someone in the forums gave me another bit of advice on what the red might be, and I’ll check at home tonight. The floor thing is truly disturbing though. Hopefully the guy who made my arena will be on tonight to come fix it.

  73. Anonymous says:

    Is this an exemplary example of lack of IT knowledge? Me scratches his head…

    “This was the result of bringing some login servers online before they were ready. Services will resume shortly.”

    Par for the course, as when unexperienced people mess with stuff they are not experienced with unexpected results are produced…

  74. Anonymous says:

    @Al

    Unless you have Frame Relay or some other god awfully fast AND wide connection, you will be enjoying your open space sim all by your lonesome 😉

  75. Ener Hax says:

    hmm, target omega never pooped out on me? 🙂 and very nice to see the attention for OpenSpace performance. OpenSpace is a real “value add” to sl imo =)

  76. hmr1000 says:

    whatever you guys are doing now seems to be causing total chaos with constant crashes and new bugs appearing everywhere, along with a return to log in crashes that are as bad as I have seen in nearly two years. At the rate you are going I shudder to think where this is going to lead, especially since these updates were billed as a means to gain more stability and to stop these crashes. If so, I can only say you have picked a very strange way to go about stopping them.

  77. Cat walk says:

    I love many of the new features. Instead of always putting the hard working people down at LL try helping them by using the jira to let the right people know what isn’t working right.
    I have seen a big improvement since the new roll out.

  78. Rolling Restart... says:

    @78

    how can you see a big improvement since the new rollout? it hasnt even happened yet?:)

  79. Pingback: The Grid Live » Havok4 Fireside Chat with Sidewinder Linden

  80. Gideon McMillan says:

    What happened to keeping the Google calendars updated for planned downtimes? I spend more time in my calendar than on the blog.

  81. Kathy Morellet says:

    Can anyone tell us if this update is still on schedule for this evening?

  82. Prospero Linden says:

    We are still on scheduled for this evening; see updated posted above.

    I’m going to be offline for the next couple of hours (off to a string quartet rehearsal!), but will respond to questions again later tonight as the actual roll to three racks begins again. Please keep questions on-topic so that we don’t use up the post limit for this post on things not relevant to this post.

  83. Linda Brynner says:

    Sorry, a bit off topic… But I know you Lindens absorp.
    Ever heard of the marketing stats 50 rule ?
    Listen to at least 50 cases… then you know what to do.
    It always works.
    XXX,
    Linda

  84. Stephe Ehrler says:

    Thanks for dealing with this “open space” problem, I’ve been patiently waiting for my sim to return to the pre havoc 4 performance!

    @ 76
    The target omega issue is when linked sets use it, not single prims. It killed the “plamsa generator” in my space station so I know this bug happened on havoc 4 deployment, just like it killed my siggy swimmer. But then if you yourself don’t own one of those swimmers, I suppose you would also think people are complaining about nothing because you personally haven’t experienced it?

    I do get a bit miffed at residents who discount other’s experiences just because they personally haven’t experienced it. I have never seen the issues with nvidia video cards and people crashing myself, but then again I’m using an ATI card so I wouldn’t see it. I’m happy for the people who seem to have zero issues with SL, I don’t think you’re in the majority.

    Stephe Ehrler

  85. Lysana McMillan says:

    @67: If you’re such an expert at programming and implementing a project of the complexity of SL in a bug-free fashion, I strongly suggest you apply for a job with Linden Labs. Clearly, your expertise is going to waste.

  86. Carrah Rossini says:

    will this include a fix to

    http://jira.secondlife.com/browse/SVC-1531

    ? the last official comment by a Linden on that blog said the fix would be out in next server patch deploy (which was done during the previous server patch)

  87. Sidewinder Linden says:

    The specific physics-related fixes are included in this post, http://tinyurl.com/4v54c8, which provided details on the updates that were deployed to the Beta Preview on April 6th.

    The specific physics-related fixes in this build include:

    DEV-13002: Improved physical representation of some thin shapes (As originally reported, this appeared to be a problem with a prim floor becoming phantom after conversion. It turned out to an inaccurate physical representation that left “holes” in the floor in some areas)

    SVC-1531: Physical representation of dimpled, hollowed spheres improved

    DEV-12725: Camera position is no longer affected by phantom objects or avatars (resolution of this bug removed a cause of rapid shifts in camera position for reasons not obvious from the scene in view)

    DEV-12841: Removed a simulator crash mode

    DEV-13080: Reduced tendency of avatar’s feet to fold up to waist climbing stairs

    SVC-1989: Avatar no longer becomes phantom when sitting

    Improved performance of openspace regions

    We have another build with many more fixes in it that is starting QA testing now, and I will blog about the details of that build once we have passed testing and are ready to deploy it to the Beta Preview.

    Regards,

    Sidewinder

    P.S. My apologies for the lack of coordination that resulted in limited information in the initial blog post above.

  88. Sidewinder Linden says:

    @85 Stephe: The Havok4 team is quite aware of the swimmer issue. It is coupled to some other issues that we are working on shortly – it has not been forgotten.

    Regards,

    Sidewinder

  89. Sidewinder Linden says:

    @85 Stephe:
    “The target omega issue is when linked sets use it, not single prims.”

    Actually I have seen this in both cases, with and without linked sets, in items as simple as a single prim with a single line llTargetOmega rotation script in it. We believe (hope) that we have a workable solution for the problem, and it is in testing as I write this. Due to the sporadic nature of the problem, we may not know for sure whether this resolution fixes the real problem until we roll it out, which would be in the deploy *after* the one described in this blog post.

    Sidewinder

  90. Tiny Mind says:

    Wait,,, does this LATEST roll-out mean we’ll have a NEW memory-leak or a fix for an OLD memory-leak. All these little ‘fixes’ get confusing.

    Tiny-est of Minds.

  91. Sedary Raymaker says:

    Well, asset server died, now logins are fail, so I guess I may as well go to bed. Won’t be getting any work done tonight.

  92. Sedary Raymaker says:

    Yay, magic blog comment fixed it. I amaze me. 🙂

  93. Sidewinder Linden says:

    @91: The root cause of sporadic deep time dilation (slow downs) on open space regions was traced to some code that allowed the simulator to allocate too much physics engine memory on this server configuration. This update resolves that problem by limiting the physics engine memory allocation on Openspace servers that run four regions per CPU. /Sidewinder

  94. H. Blanco says:

    Nice work Lindens. I do hope that this “fix” will also take care of the 20 crashes that I was getting all last night right after the download of the new client. Also lets try not to muck this weekend up as the past 2 have already been a FUBAR event.

    Get all your server down, power cuts, updating this and that done today and tomorrow and give it a rest this weekend.

    Thanks.
    H.

  95. Tiny Mind says:

    Omg… get that Time-Dialation fix in there like… YESTERDAY! No more Server Mem Leaks!! Somebody take a picture of this moment! Call Slash-Dot! SL is finally STABLE again! 😀

    Tiny is Miny,

  96. Yanik says:

    GOOD! “Unable to connect to a simulator” Great!
    Groups notices don’t send, group members don’t load. Logged out and not trapped there. Hoorah! Love daily updates! Maybe this is to get it better, but since 4 weeks it getting awfully worse.

  97. Rooke Ayres says:

    @ 96 – Tiny Mind

    Don’t hold your breath. Wait until after they have the fix in place, and everyone up and running with it. The theoretical doesn’t always work out the same when its actually used.

  98. Zenaro Schism says:

    Is anyone having trouble logging in? Getting a error that says “cannot login to simulator.”?

  99. Starlett Nightfire says:

    yes…says the same here

    “cannot login to simularor”

    I’m guessing it is related to the rolling restart?

  100. Zenaro Schism says:

    I’ve tryed logging in to different sims, and it’s not letting me there either..

    i mean good job LL on everything, but i know some people who are online right now, and some who aren’t…

  101. Aaron J. Freeman says:

    I just had a touch time, logging in. Three tries to HIP2 failed, Home failed, and my usual work-place got me rerouted to Governor Linden’s Mansion. TPs to friends seems to work, even though my map doesn’t.

  102. Brynne DeCuir says:

    Yeah, I can’t log in either. Get the same error about cannot login to simulator.

  103. U M says:

    As for the H4 servers……Is there going to be a fix for slowing down of sim FPS? I understand that H4 servers don`t crash but the slowing down of the simulator doesnt improve the over all playable conditions. It seems like a -+ issue?

    Usagi

  104. Kimmie Cleanslate says:

    Is this why I am having connection problems? My BFF was able to get on but I am not! Help!!

  105. CC says:

    whatever they are doing is making my whole internet and modem lose connection not just the game!!!!!!!!!!! this is NOT SATISFACTORY what your doing should NOT effect my internet PISSES ME OFF! your so incomptent INFORM YOUR BLOODY CUSTOMERS IN GAME!!!!!!!!!!!!!!!!WHEN U DO THIS kinda stuff god only knows what damage it can do geeeeeezus grab a freakin brain lindens i swear………. please get competent people and techs working there already!

  106. Sephy McCaw says:

    Ummm Guy and Girl Lindens, are you gonna postpone the remeberance day or holding off restarting those specific regions?

  107. Danball Tureaud says:

    #106 CC, that has nothing to do with Linden Labs, thats your ISP having network issues. IT HAPPENS TO EVERYONE, GET OVER IT.

  108. Melody Regent says:

    Let me just say, the thing that is impressing me most about this current blog is the response from the LL staff. It does make us feel better to see you all responding to things said here. Lets us know you are listening. That makes a world of difference. Thank you

  109. Tiny Mind says:

    :: @ 96 – Tiny Mind
    Don’t hold your breath. Wait until after they have the fix in place, and everyone up and running with it. The theoretical doesn’t always work out the same when its actually used.””

    Wha… what? *sniffles* “Are… are you saying the Memorial-leaks won’t be fixed??? Am… Am I going to crash on SL some more?? Like I have been for MONTHS now?? *Whuuuaahahahhhhh…!*” Runs off crying like Chris Griffin.

    Tiny Crying.

  110. Sidewinder Linden says:

    @110 Tiny: The memory issue addressed by this build is on the server-side – the simulators that run the regions. it does not address viewer crashes. Rather it resolves a region crash mode that throws everyone off a whole island or region at once. /Sidewinder

  111. Ann Otoole says:

    best resolution for viewer crashes i deployed was a new quad core cpu with 4GB ram and a nvidia 8500 gt vid card. full on ultra all maxed and no problems at all. the poor old battered dual sli amd 64 bit box that was almost 4 years old finally blew up with a pop and an ozone cloud. there went the $3500 box replaced with an $800 box that is way superior. all i can say is the new systems are cheap. if your having problems with the viewer consider an upgrade.

    i think i got my money’s worth out of the old one and will seriously enjoy getting my money’s worth out of the new one.

  112. Sean Heying says:

    I hope you can get this 1.21 update rolled out, so many of my friends with openspace sims are having real performance problems.

    Mine seems to have cleared up again, once more simneighbors shows I am the only sim on my server…. which gives credence to the memory issue and also explans why this didnt become too apparent during early adopter testing.

    Anyway, good luck with the bug hunting.

    (Think the swimmer will ever be fixed Sidewinder?)

  113. U M says:

    sidewinder ( should have directed to you )……….

    As for the H4 servers……Is there going to be a fix for slowing down of sim FPS? I understand that H4 servers don`t crash but the slowing down of the simulator doesnt improve the over all playable conditions. It seems like a -+ issue?

    Usagi

  114. As someone being heavily affected by the memory allocation problem killing open space performance I’m really unhappy about the prospect of having to wait even longer for the fix. Can’t you rollout an interim version with *only* that problem fixed – this shouldn’t have negative side affects on other stuff, correct me if I’m wrong.

    Fact is: since the general h4 rollout open space regions are barely usable – and somehow I doubt you will give us a cut in the money we have to pay LL for such a defective product.

  115. Chance says:

    Sure is funny watching these people complain about there business in sl .just like any business theres going to be growing pains . and to think there critizing LL for there not making money in a game due to there efforts in improving this for everyone not just the few that think that making money is more inportant then all in this game involved well personaly i would not buy from any vender that would make them comments. it shows me that there probily trying to take advantage of new technoligy for there own greedy gains.. well tough luck people sometimes the horse wins .sometimes he loses.. thats the gamble . could i suggest maybe getting a real job and after all without LL you wouldnt be here to complain lets keep it in prospective. its a GAME if ya dont like it the power button is right there you choose to come here thats your choice. thank you and PLAY on !

  116. Kayla says:

    Yeah Sl Is Being Mean To Me Today Says Despite Our Best Efforts Bla Bla Bla .. Happens Alot Actually 😦

  117. Prospero Linden says:

    Re: the pain associated with H4 openspace sims, we are aware of it. I’m really sorry that we keep having trouble with the 1.21 deploy. There are other pain points too, associated with the central database, that we’re also working on alleviating.

    I will post more updates above as we know more today about what will happen with the deploy.

  118. U M says:

    I wish people would stick to the subject instead of trying to start problems……….

  119. nimi says:

    Thursday, April 17, 5:30AM~~~~~until when??

  120. sexina Rayna says:

    [Update 2008-04-16 20:30] The deploy will begin momentarily
    [Update 2008-04-16 21:10] showing problems, so we are going to revert them to 1.20.

    loooooool
    I give a hint to the developer team of linden:
    Use the Betagrid to test the new version BEFOR deploy.
    Linden act unprofessionel.
    Ask google how to make a working database for a huge of transactions in one time.

  121. Mook says:

    what about the root cause of sporadic deep time dilation (slow downs) in occupied sims? Since the first 2.0 its dropping tons of acks when theres activity in a less than healthy sim.

    There was a seed change in how connections are handled in 1.20 and since then uprates (and to a small degree downrates, after all they are the cause of the ack flooding) have been atrociously volatile since 1.20 arrived however I doubt the run of the mill user would notice. Please look into why the acks are far more likely to be held up since the new server?

    Im waiting with baited breath to see if tonights fix does anything, but besides the openspace memory issue, theres little in the changelog to suggest that anythings been done for occupied sims.

  122. Shin says:

    #122 Mook, I don’t know what you are thinking of, but most “openspace” sims are occupied.

    Given yet another problem, is it possible that just the patch that fixes the problem with the openspace sims be rolled out just to the openspace sims? It seems to me that we are now looking at almost another full week at best…given the standard LL decision not to roll out leading into a weekend. Unless the patch is likely to take out the sims, I don’t see how it can be worse for openspace sims than being on them now is. Pull out the other “fixes” and apply the openspace patch to openspaces.

  123. Mook says:

    ok, for the record, make a correction to read “fully occupied” if you wish.

    Either way, fixing the memory/swapping issue on open sim servers doesnt sound like anythings been done to ease connection stability problems on “fully occupied” servers.

  124. Gil Druart says:

    @106 – Try not to rant, dear, but you’re not alone. For me it is *specific regions* that are showing the problem – try moving elsewhere as soon as your ping times start to spike. Also please submit a support ticket as LL support is trying to pretend that this problem doesn’t exist at the moment. We must be imagining it or be feeble-minded is the general feedback … which brings me to …

    @108 – Please try not to be as ignorant as the people ignoring support tickets: you are quite, quite wrong. This is a real problem and it is an SL server side problem. My take on what is happening is that SL is flooding the connection and ignoring the bandwidth throttle. For people on low bandwidth ‘last mile’ this has the same impact as a denial of service attack and typically kills the PPP session. For people with higher bandwidth connections this is not a problem presumably because the typical sim is incapable of flooding their link. Why and how this is happening is another matter of course.

  125. Joshe Darkstone says:

    Wow, a blog I can be totally positive about 🙂

    Glad to see these issues resolved, especially the openspace issue, many tenants struggling with this one. Great work on the part of the havok4 team!

    (Now would one of you go smack jack on the forehead for me please? 🙂

  126. U M says:

    125 please repect otehrs on the blog………everyone has the right to voice their thoughts……….besides your wasting blog space.

  127. Darling Brody says:

    *** BUG REPORT ***

    llLookAt is not working on non-physical objects in the region Meyers, but it is still working in other regions. I’ll put it in the jira too, but this might be faster!

    *** BUG REPORT ***

  128. nimi says:

    can’t login

  129. U M says:

    129 look at the blog newest post its all there……..

    Usagi

  130. Mike1 Shepherd says:

    Am I just imagining the fact that you cannot log in to any region right now and nothing on any blog or status shows that? I’ve also observed the users online dropping steadily for the last hour when it should be climbing.

    This could really be a great day of remembrance….if I could get to it.

  131. Angel says:

    ugh I cant log in for all most two hours now. Oh well I got to know rl again 😛

  132. U M says:

    I am in world byt can`t teleport any where profiles are not showing. etc etc etc………i stuck in SL hell at the moment

  133. Sandor Balczo says:

    Well, perhaps Linden Labs have realized that too many people spend time being productive in Second Life and have forgotten this is a VIRTUAL WORLD???? 🙂

    In a response to my blog comment regarding a previous service disruption, someone under the name Broke (my avatar name is real, I would appreciate your using the same identification) said with irony that I am the perfect customer for Linden Lab because of my complacency and said that if I spent a thousand US Dollars a month in tier fees and had lost over a hundred thousand Linden Dollars in inventory last year I would be angry too.

    Well, I have been in SL since the end of May 2007 and spent the equivalent of US$1,247 (or L$305,000 for those who think in L$) out of my own pocket.

    I knew I could not spend a lot of money, so I didn’t waste it.

    I bought land where it costed less, moved when I could find better land and better neighbors at a decent price, tried to open up a store, which I still have, and when I realized I would never be rich I did the best I can do, help others.

    And now I have a SL job as a resident helper that pays my tiers and, having been a premium subscriber since June 2007, I have received a regular stipend from Linden Lab amounting to L$18,000 so far.

    If a disruption occurs, no problem, I am in a team and if someone else can be online when I cannot, customers are satisfied anyway.

    Helping others should be the most thriving business in Second Life, since a lot of people are HELPLESS in many ways.

    I am somewhat disappointed when Second Life cannot be used, but the moment I realized it was not stable (and it has never been for as long as I can remember, if we also consider a time last year when griefing on a large scale was rampant on a lot of sims, which is a rare event now, maybe thanks to more security measures introduced in the server code) I reinvented myself and, since I am paying my own money on this virtual world, tried to reduce risks.

    Life presents risks: some people like to run them and some do not.

    Life also has accidents and acts of God. If those who complain about losing money have taken risks, they should not complain if acts of God happen. It is part of life. One has to roll his sleeves and try to counteract. And if one knows he cannot overspend, they should not.

    As for LL not fixing problems and all other virtual worlds having fewer crashes, well….if I am not mistaken you can build anything you wish in Second Life and write any kind of code you wish. Maybe if Linden Labs started to fetter those who cannot write decent code and cause sim instabilities (and there are quite a few) we could have fewer downtimes. But it would not be a free world anymore, would it?

    Sandor

  134. Bently Upshaw says:

    “Several of the regions that received version 1.21 are showing problems, so we are going to revert them to 1.20.”

    Does this mean that the 6pm update to the rest of the servers is off?

  135. Trin Bailey says:

    Ok…
    Thursday noon and 24 minutes.
    This was supposed to be my birthday party.
    I have been planning this for weeks,
    and was ready to celebrate my first year on SL.

    No friends online,
    no world map…
    You know what is ironic ?
    I know I am not alone on SL,
    but YOU linden tend to make us think we are all alone.

    Is is really worth ?
    whay am I keeping paying tier for such a nightmare ???

  136. Trin Bailey says:

    oh…
    and here is to you, Sandor…
    you’re an @shole, swollen with self-confidence. Be careful, you’ll soon become magalomaniac.

    Keep on pretending you are helping “so-called” weaker people,
    keep on trying make yourself an interestinf person…

    I don’t qive a f#uck,
    I just want to enjoy SL.

  137. Trin Bailey says:

    oh…
    and here is to you, Sandor… !
    you’re an @shole, swollen with self-confidence. Be careful, you’ll soon become magalomaniac.

    Keep on pretending you are helping “so-called” weaker people,
    keep on trying make yourself an interestinf person…

    I don’t qive a f#uck,
    I just want to enjoy SL.

  138. Trin Bailey says:

    Sorry for typos,
    but I am rrrrrrrrrrrreally angry now.

  139. Trin Bailey says:

    and stop telling fairy tales about “God”.
    The only god here (i.e omnipotent people) are the linden. They rule SL.
    Theiy have the power to make it heaven or hell.

    But please, Sandor,
    Don’t talk about “God”…
    It is so “passé” !!!!!

    hahahahahah
    ridiculous

  140. TongKuonuo Kamachi says:

    Back Office to each Region Would be nice–accessible to Region Owners <>, considering these factors:

    >Factor 1: Multiple PCs with different Platforms are Jumping from region to region with Teleporter via AV.

    >Factor 2: Region Owners cannot try a patch and if it cause problems remove it and go back to the old version.

    >Factor 3: Could Make additional profits on selling simulator plug ins and patches. Like adding more Prims.
    Every Region has different needs.

    Consider giving region owners a choice over when they can upgrade or uninstall features on their simulator.

    Consider giving Avatar Users who may be logging on at office with a PII and then logging on at home with a MAC for example. The capability to uninstall and upgrade there SL experience VIA Patches, Plugins, and Add-ons.

    Set up a Market Place for selling these upgrades or giving away the ones that are a must <>.
    Also you can give us a choice between viewers. Which would be cool if also at this market place.

    >>>>>>>>>>This is easier said than done but had to say it cause me and many people i’ve come in contact with and i dont go around bad mouthing SL.. SL is a great place (Definantly a World Wonder) ..but people are complaing and getting upset in-world. Cause This Not Just a game to them its a new way of experiencing life.

    If there is any way of changing upgrade, plugin, add-on logistics. To a more efficient platform that can allow diversified users having different Hardware and Software Possibilities to access it make a choice, try the feature out and remove or keep it on there pc or on their region via Region Back Office. Please consider giving us access to it.

    TongKuonuo Kamachi >DyNasty Island<

  141. Trin Bailey says:

    >Tong…..
    can you repeat, please ?
    Make it simpler, please.

  142. So 1.21 is being hard and not working. Well, 1.20 is not working either is it, nor was 1,19. Why don’t you just roll back to 1.18, when everything at least worked a whole lot better, forget Havoc, Windlimp, Frazzle and all the others shinies?

    Basically nothing has worked since you changed the server code for these additions.

    On top of that, why oh why did you let a whole raft more concurrency on the already overloaded asset servers, let alone plug in a whole new mainland continent. Do you guys have any clue at all what you’re doing?

    Most companies make sure they have sufficient capacity and performance, with some in reserve, before they move on. LL’s Tao seems to be when it’s already broken, and you’ve had four pathetic attempts to fix it, pile on a load more…

    This was the prime complaint in the Open Letter of almost exactly a year ago, and you’ve fixed precisely zero of those concerns. And I guess that’s why from your own Key Metrics, Active Users have been dropping, not just the US for two months in a row, but globally. Must be why you didn’t blog about the release of the February ones, 5 weeks late – hoping people would forget and the bad news would go away again.

    You know, when people have been complaining for a year on performance, it’s generally not a good idea to rewrite both server and client and four major areas: graphics, lighting, physics and scripts and a new viewer architecture on top, and release them all at once. Total madness. How do you expect to find any bugs?

    The quicker a new CEO comes in with a rather large broom and sweeps away all the old crud, the better…

  143. U M says:

    Going back to 1.20? that shows some things are just being done right fromthe start. Is it possible that one needs to “EXPECT” the worse before releasing this? I understand like many do that its trail and error. But these days it seems to be getting daily life with these types of boo boos LL is doing these days.

  144. stripey says:

    It’s good to hear that some issues are being worked on. And way better than that is that Prospero and Sidewinder Linden are both responding to people here. *much greatfull applause to both*

    I may gripe about SL when things mess up, but the ever present lag and crash problems are an ever growing, ever battled beastie. The issues could be much worse.

  145. Sandor Balczo says:

    Trin, I ams so sorry your birthday has been wasted, but this is not a good reason to call ME an asshole.

    That is what I dislike about SL these days. Too many angry people who try to shut up other people who try to help them when they are angry. Fortunately, I am in customer care in real life and this is a common practice.

    Learn to listen, will you?

    Sandor

  146. U M says:

    agreed……………too many upset people and then those calling people whinners………Years ago we didnt have neither of these types. But now thre are those that know-all and those that installed all. then again look at the number of under 18 inthe game these days…..

  147. U M says:

    Another round of WHO IS RUNNING LL ?omg teleport money transation, etc etc etc etc etc!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!gesh 😦

  148. KMeist Hax says:

    1.21 failed again, for the third time… lulz

  149. Pingback: Rolling Restart for 1.21 Server Deploy Wed/Thu/Fri « Official Second Life Blog

Comments are closed.