Rolling Restart Mon-Wed 07/21-23 to deploy server version 1.23.2

Update 2007-07-22 12:30 PM : The reversion is done.  The investigation begins.

Update 2007-07-22 11:56 AM : The reversion is 80% complete and should finish soon. Also, thanks for the reports (via JIRA and comments) about performance issues with sims running 1.23. We have a hypothesis that the server build erroneously included additional debug information, which could lead to memory bloat (and thus crashes) and reduced performance. We’ll be doing additional experiments later today to try and confirm that hypothesis and see if that is the cause of the problems. At this point, though, it’s not much more than a hunch.

Update 2007-07-22 10:04 AM : Okay, you probably saw this coming… Now that the 1.23.2 update is out to half of the sims, we’re seeing a greatly increased simulator crash rate relative to 1.22 – nearly 10 times as high. This was not seen during previous 1.23 deploy attempts, nor on the Preview Grid, nor was this behavior seen on the pilot roll hosts – we’re baffled. Call stack analysis doesn’t point to a single smoking gun, either.

Between sobs and gnashing of teeth, we’re going to revert the 1.23 regions to 1.22 with another rolling restart already underway (expected to complete within about 4 hours), while the dev team attempts to understand what happened.

Please direct any positive karma towards the engineers working on this. When I was a fresh new Linden two years ago shepherding releases, a planned Wednesday downtime deploy – which stretched to 6 hours – would often be followed by additional hours of unplanned downtime and subsequent patches pushed out over multiple days to address show-stopper issues with new features and regressions. And then we’d often live with critical issues until the next iteration. We now have the means to push changes out much more slowly and with limited downtime per region, and in a non-monolithic fashion. For example, the central systems have been happily running 1.23 code for several weeks now. As the code complexity and size of Second Life has increased, the system as a whole is more sensitive to changes – and we also have better monitoring. Since we value the stability of Second Life, this has made us approach the roll-out of recent releases with more caution. It certainly causes more visible action, and angst within Linden Lab, but we believe the stability and quality of Second Life is improved.

Update 2007-07-22 08:56 AM : The first-half rolling restart is complete.

Update 2008-07-22 06:00 AM : The rolling restart for the first half of the grid has begun. Today, we are deploying to odd-numbered hosts.

Update 2008-07-21 09:06 PM : the pilot roll to 304 regions has been completed. Regions which have been updated are running version 1.23.2.92647.

The “fun” (here, here) continues – the issue found that prevented the rollout of 1.23.1 has been fixed and verified, and sat on the Preview Grid over the weekend, along with a couple of security fixes for previously outstanding issues. We plan to roll out 1.23.2 this week, starting with a “pilot roll” on Monday followed by updating the reset of Second Life over Tuesday and Wednesday. See the original 1.23 blog post for a full list of issues.

Here’s the schedule in detail:

  • Monday evening (PST): a pilot roll to 150 regions
  • Tuesday morning, 5AM-10AM : we will deploy server version 1.23.2 to half of Second Life.
  • Wednesday morning, 5AM-10AM : we will deploy server version 1.23.2 to the rest of Second Life.

As usual with rolling restarts, this is a change on the server side; there will be no required client udpates associated with this rolling restart. Regions will receive warnings starting five minutes before they are restarted. There is no way to delay the restart of a given region. Regions should restart within 10 minutes of going down. If your region stays down for more than 20 or 30 minuets, please contact support.

This entry was posted in -Miscellaneous. Bookmark the permalink.

150 Responses to Rolling Restart Mon-Wed 07/21-23 to deploy server version 1.23.2

  1. cosa nostra says:

    remark : The “fun” (here, here) continues

    1/ maybe it is ‘fun’ for free accounts who invest 0 l$/us$/€/£ ….
    2/ but stay professional & respectfull for your customers who invest Real money in here (better leave the ‘fun’ out moving forward)

    thx
    cosa

  2. Ann Otoole says:

    Best of luck with the rollout and hope it all goes nice and smooth this time.

  3. Cappy Frantisek says:

    /me grabs my helmet and settles in

  4. Now Cosa, don’t get all negative.

    I’m sure our friends at Linden Lab are doing their level headed best to FIX our world.

    Just last week, there was another BUG unearthed, and so far, those in charge have refused to even acknowledge it, the poor people, working so hard all day and never achieving a single goal; must be hard.

    I just hope that the next Linden Lab employee that tells a Resident to ‘BUGGER OFF” gets caught. The last time it happened, the logs mysteriously got corrupted. Maybe Linden Lab should look at its own employees for these bugs?

  5. Slartibartfast Magicthise says:

    Oh, come on #1 have some fun.
    When your sim restarts, don’t walk – run!
    Hopscotch the “up” sims, from one to another,
    and rolling restarts won’t be such a bother.

  6. DR Dahlgren says:

    Let me see – A rolling restart vs. the whole world offline for 6+ hours -decisions, decisions… what would I prefer… LOL

    D’d if you do and D’d if you don’t, no way to ever please them all.

    Thanks Joshua for the update.

    DRD

  7. Garn Conover says:

    Woot! Hopefully this goes smoothly so we can work on looking into getting Mono ready for release.. #1 lighten up >.< and #4 do u release how many bug reports there are? also if it is not added to a Triage sometimes they get missed until the right person finds it and adds it to the list.

  8. Villain Baroque says:

    Meanwhile, I crash every second time I upload a texture today and got my first asset server error in a few weeks.
    Seems as if even the servers are in a mild state of panic concerning this 3rd (or is it the fourth already) try to deploy this server update.

    How about contacting the Ghostbusters for some help?

  9. Marianne McCann says:

    Third times a charm?

  10. JayR Cela says:

    Go for it LL
    at least these days (as of late) you do not seem to be doing things anywhere near as haphazardly as you used to. :_)
    As an example / 14 release candidates with the newest viewer. / this is a good thing / rather than rush something out the door / you are doing what it takes to make sure the product is solid.
    I think your doing a great job – all oh ya
    Best of Luck

    JayR Cela :_)

  11. Urantia Jewell says:

    Keep up the good work LL.

    BTW, are we going to see this on the SL Grid Status Page and Calendar?

  12. Firebird Nightfire says:

    Cosa –

    The word “fun” was sarcasm. I have no doubt tracking down this bug has been absolutely no fun at all for all the programmers. I’ll donate my box of freebie prim hair to them, cos I’m sure they’re all bald now from pulling their hair out in frustration! 😛

  13. Brian Engel says:

    Whee, another Rolling restart, followed quickly by Rock throwing from the residents. So its a new week and we be Rolling & Rocking our way through it. Its nice when the world is spinning on greased grooves in a predictable way. I hate wobbles.

  14. Andromeda Quonset says:

    What is the UUID for the animation of pulling out one’s hair? I’m sure that there are a LOT of us that could make good use of it 🙂

  15. Cudaboy says:

    Starting to think rolling restart is synamous to SL. Its like a weekly ritual for the last 3 months

  16. Now Now everyone. Linden Lab has a policy of NO-Rolling Restarts every week. It’s been in your head the last 3 months.

    I trust our OverLords at Linden Lab, about as much as I trust that the logs will not corrupt when a Linden Employee gets an AR filed against them.

  17. Joshe Darkstone says:

    @15 only 1.23 has been a “weekly” ritual, but i do hope they slow down the releases a bit after this. but hey, i added hourly revenue graphs to my spreadsheet so i can watch it happen live 😛

    When are we gunna hear more about this showcase thing? Not sure I’m too keen on LL directing traffic away from tier paying residents stores and into the arms of some chosen few…

  18. Firebird Nightfire says:

    LOL Brian! 😀

  19. Jen Shikami says:

    So will this affect the timing of the Mono rollout later? (I know I keep asking about it… we’re very excited!)

  20. Lloyd Newman says:

    Well, I’ve been using RC 1.20.14 for several days now with nary a twitch (THANK YOU for allowing us to opt out of ‘Dazzle’… or as I called it, ‘BlindMe!’ ) … then the rolling update comes along, and I can’t teleport, and even crash on login…. coincidence?

  21. Tom says:

    Runs for the hills…. Good luck guys 3rd time lucky 🙂

  22. Yukinoroh Kamachi says:

    Any plans to fix hovering buoyancy and llSetBuoyancy ? I still wish there will be a time when I can sell my product again, following the havok4 release…

  23. Fiona says:

    Good luck, guys! 😀

  24. Cathy says:

    OMG,,,,,,the asset system is f’ed up again,,i just bought something for 1200L,,,,the folder would not rez,,,something about item not in database…i relogged and its not there now. Any ideas what to do?

    Crying 😦

  25. Yshmael Zapatero says:

    Wow its like living in NYC everyday something different.

  26. Cappy Frantisek says:

    @24

    Yeah try going outside! RL is better anyway!

  27. Cathy says:

    @26

    Thats how you help someone? You are a real great guy. Thanks for the help. Spare me your attempt at comedy, you suck.

  28. Cappy Frantisek says:

    if it’s gone it’s gone sorry to be the bearer of bad news

  29. Wondering about Bob says:

    @16 Bob….

    You have mentioned this same incident now on several blogs and frankly, its boring. So what if a LL employee told you to “Bugger Off” and you submitted an AR.
    So what if they corrupted the logs afterwards. Who cares?? Its boring. Move on.

  30. cosa nostra says:

    @24 cathy

    1/check your transactions history and take a copy of the transaction
    2/copy this into a notecard
    3/sent it to the seller of the goods ya bought

    PS. if the seller is a professional, he will give ya a new one ;o)

    take care
    cosa

  31. Cathy says:

    @30

    Thank you for the advice. I really appreciate it.

    See Cappy thats how its done. I hope everyone here looks at your profile and doesnt buy anything from you. Thats what i have done.

  32. Rhaorth Antonelli says:

    might want to proof read before hitting the submit button (unless I am reading this incorrectly)

    shouldn’t the line (note the change from reset to rest)

    “We plan to roll out 1.23.2 this week, starting with a “pilot roll” on Monday followed by updating the reset of Second Life over Tuesday and Wednesday.”

    be…

    “We plan to roll out 1.23.2 this week, starting with a “pilot roll” on Monday followed by updating the rest of Second Life over Tuesday and Wednesday.”

  33. Jahar Aabye says:

    For everyone complaining about the rolling restarts:

    You may not realize this, but it makes you look like a noob. Not just a noob, but an extremely ignorant and uninformed noob. Actually, let’s also throw in the term “spoiled brat” as well.

    Why do I say this? Well, let me tell you a story about what SL server updates used to be like before the advent of the het grid. The *entire* grid had to be shut down completely. Not a rolling restart, an entire shutdown. No logins except for Lindens. This would often last for between 3-6 hours. If there were problems with the new code, of course, it could wind up taking even longer.

    This was a fairly regular occurrence, too. There was nothing we could do about it except just take note of the advance notices and plan on having something else to do….well, except for downloading the new viewer that was pretty much mandatory with each update.

    But yeah, go ahead and whine and moan because your favorite region was down for 5-10 minutes during a restart. Please keep it up. It makes us older residents smile and remember how much better things are now.

  34. Alf Lednev says:

    all these rolling restarts. Hmmm perhaps we could all chip in and buy ” Quality Assurance Testing for Dummies” for the lil lindens. Somehow it looks like there is no QATF staffed within Lindens. It really appears they use the old Mircosoft model of QA testing and use paying customers as beta testers.

    And yes over the last 2 days everything seems borked, I miss “ruth” now, Im sick of that white cloud floating that takes forever before someone rezzes. A new phenomenon but one that has fast worn out its welcome

  35. @29

    It never amazes me to see the blind following the blind. You think it a good idea to let the Linden Employee’s run amuck without ANY punishment?

    I’m sorry, when I put my dollars on the line and someone promises me Customer Service, I expect Customer Service, not an Employee who not only couldn’t control his tongue, but also went into the LOGS and Corrupted them on purpose. Promise you this, if that was ANY other company in the US, that person would be on the streets looking for a new job.

    Just goes to show me that Linden Lab is the LAZIEST and INCOMPETENT Corporate Culture I’ve ever had the dis-pleasure to fun into.

    For the facts though, Bub Linden never told me to “Bugger Off”, it was towards someone else, I just got to write-up the story. If Bub ever said that to me, I’d book my trip to California so we could talk it out Man to Coward.

  36. Hi, I wish the Linden team good luck with this new attempt of getting the 1.23 version working. With this one it seems each time they updated a different thing went wrong. And with it we all will be a little step closer to the mono included server version. I’m looking for that :)).

    And actually is a normal thing that they need to test them, we all know that the worst and hardest to find bugs never show up until you try it in public… murphy’s law isn’t it? 🙂 but the good part is they show up and our friendly Linden team works hard to provide us with the more stable and feature rich server versions. Keep doing like that 🙂


    Greetings the SecondSolutions team.
    http://secondsolutions.wordpress.com/

  37. Dilbert Dilweg says:

    Good Luck. :/

  38. Don’t whine about rolling restarts; I remember once when Second Life was offline for almost two days because of a failed update. Updates then the entire grid had to go offline.

  39. Hank Ramos says:

    Dear Linden Labs,

    Ignore the people bitching. Most of us appreciate the hard work and planning you put into rolling out the updates to the grid (much less disruption than in the past)! Keep up the good work!

  40. Joshe Darkstone says:

    I think its fantastic that things have improved so much over time, and they have, greatly. That doesnt mean that we should not comment on the negatives that remain. During a rolling restart my store revenue is impacted by as much as 30% or more. It isn’t just me so I’m guessing its the entire economy thats affected. LL seems to think otherwise and so I’m happy to point it out to them, constructively 🙂

    During the last 3 attempts to rollout 1.23 the first one consisted of a full rollout and a full rollback which was absolutely unecessary. The 2nd and 3rd attempts were limited to the first intial rollout. Good job that. More time with the initial rollouot and a less aggressive rollout schedule just makes a good thing better.

  41. @38

    Yes, I remember that weekend, Linden Lab in their infinite wisdom (*COUGH*) decided to add VOICE that Friday and then couldn’t get it fixed until Sunday evening. Gee, I wonder why folks still think Linden Lab needs more improvement?

    @39

    Please point out to me, within the LAST YEAR, where Linden Lab hasn’t had to reverse something in a MAJOR ROLLOUT. I remember a Linden Lab that didn’t screw up during RELEASES, goes to show you, Cory was the WRONG GUY to fire, should have been the head. Now we all got a puppet with his strings being pulled by the COB.

  42. shuky says:

    Yo lo que estoy es hasta los cojones de Linden labs,no se puede scriptar tranquilo porque te tira en la mitad, te pones a construir editas mas de tres piezas y el avi se te queda bloqueado, has de reiniciar y todo tu trabajo hasta que te das cuenta de que tu muñeco esta bloqueado no ha servido de nada, con la consiguente perdida de tiempo, estas tranquilamente en tu land el cual pagas, no a linden lab en mi caso pero yo pago para que paguen a Linden y resulta que cada dos por tres hay un restart por las brabas, si estas haciendo algo te jodes, te teletransportan a uno de sus sitios y te aguantas, ole, viva Linden Lab y la madre que pario a todos los informaticos que rabajan en este SU “juego”, pongo juego entrecomillado porque muchos de nosotros somos usuarios de pago y estas constinuas ISSUES (lo llaman ellos yo las llamo putadas) te hacen perder un monton de pasta tiempo y nervios, ale señores a seguir asi a vr si nos hacen un mundo paralelo que funcione mejor y les mandamos a tomar por el culo de una puta vez.

  43. @42

    You said a mouthful there Shuky 🙂

  44. Resolved (TM) says:

    @33 Jahar Aabye, you’re absolutely correct.

    Seems most people forget how ‘delightful’ were some Wednesdays not so long ago — up to something like Sep/Oct 2007 if I remind well. All those ritual mid-week announcements stating 2 to 4 hours downtime for the grid that frequently went up to 6 hours or more actually. Why are these good times gone ? hm you all complainers would have enjoyed so much the ‘old’ days i’m sure…

    And for rolling restarts as they occur now, if some here cannot live without SL for a couple of minutes or one hour, then they certainly have a serious addiction problem.

  45. Resolved (TM) says:

    Nostalgia and souvenirs…. That was a mid-march wednesday:

    “the grid will be down for maintenance from 7am to 11am for maintenance”… in fact it was a typo ; was not 11am but 11pm.

    So please stop crying and whining. SL is working (almost) fine now.

  46. Kenny Devoix says:

    @41 nope they didnt reverse anything back then. They just left the screw ups in place till the next release instead of rolling things back. Myself I kinda like the new way of doing rollbacks better.

  47. Sindy Tsure says:

    @39, Hank

    Seconded!!

  48. Pingback: Massively.com SL news

  49. Tristin Mikazuki says:

    Can we get the % of completed again? Thanks

  50. SFFan2 LosAngeles says:

    It would have been nice to have this rollout listed on the SL Grid Status page. I thought that was the policy.

  51. Shadoe Landman says:

    I don’t know if it has anything to do with restarts, but my av is always looking at the ground now. I can’t see where I’m going.

  52. defrag Babii says:

    I must say , Things are looking up …
    Wish you best of luck with this !

  53. defrag Babii says:

    I do however have one wish for Linden Labs !
    fix the asset servers setups ! seems to be the worst problem in sl as a whole ..

  54. Ron Crimson says:

    Bob @ 35: Just zip it already. You’re such a conspiracy theorist, and yes, it IS getting boring.
    I’m all the way with @36 and @39. 🙂

  55. awnee dawner says:

    hey!
    as a simowner i have to say i like the way of rolling restarts i get the message log off and 5 – 10 minutes later boom the sim is up again.

    and with less “rubbish” in the inventory assestservers would have less problems to fetch all the requested datas.

    thank you LL for all the work in the past and in the future

  56. Nano Ashby says:

    ok, deep breath, hold….. and relax, (repeat as many times as needed) SL is a technology and product in devolopment, and we have volunteered to be the mice in the wheels (yes you have) we *all* know SL as a product is far from finished, the tech heads are developing it while we use it, they are literally figuring it out as they go… to those who want a finished product that is not being developed, performs perfectly, etc, then I would suggest you subscribe to a virtual world that is already finished.
    (of which there is none, this is pioneering technology)
    and
    CHILL (nough said)

  57. Tegg B says:

    cosa nostra Says:
    remark : The “fun” (here, here) continues
    1/ maybe it is ‘fun’ for free accounts who invest 0 l$/us$/€/£ ….
    2/ but stay professional & respectfull for your customers who invest Real money in here (better leave the ‘fun’ out moving forward)
    thx cosa”
    Well, if you going to be so deadpan RL serious & critical about small amounts of REAL money, maybe you should go into REAL business in REAL LIFE instead.

  58. Angsty Rossini says:

    Looking forward to the server upgrade 🙂

    Fingers crossed it doens’t hit my region between 5:00am and 6:00am Wednesday when I’m teaching a class!

  59. Frankly i think LL has managed the whole update procedure quite well, try to remember how it was before when the grid was updated on 1 day and everyting failed and afterwards we had to deal with more issues than before.

    Nowadays the grid performance is a lot better and the rolling restarts dont bother me at all. I used to have 30 failed transactions /day few months ago, now it’s 15 per month.

    I can see that others feel the same way because otherwise the 150 comments would already be eaten up witnin 1 hour 🙂

    Good Job LL

  60. I have 9 sims. and personally I have had no trouble with any of the role outs. actually I agree with 44. Life in SL had diffenately improved. Thanks SL your doing a good job, sorry some people just see the glass 1/2 empty. Looking forward to some new stuff. One thing I would love to see is a way to mute groups with out leaving them. and another would be to have more groups then 25. Then if SL was really in a programing mood. Add more roles in those groups and more ability to minipulate them. And yes Im a terrible speller, so all you half full glasses you can tell me how stupid my comment is. hahahhaha love ya anyway

  61. Cappy Frantisek says:

    @ Cathy

    I don’t sell anything, I hardly use SL anymore. Why would I want to be responsible for selling items on a platform that changes so often that scripts are broken? Then it’s MY duty as c content creator to pass out a fix? Don’t thin so, I’m more grounded in RL and the troubles we have there.

    You do remember RL don’t you?

  62. @ 35
    Quote:
    “It never amazes me to see the blind following the blind.”
    End Quote.

    From my point of view, I think it is a matter of
    a Blind following a Lame.

    ROFL

  63. Any news on fixing invisible mesh skirts that you borked not that long ago?

    At least with Wednesdays things generally worked. With these rolling restarts we just get more and more problems. Never had a failed transaction before 6 months ago. Now failure to rez is almost a daily occurrence (never used to happen at all).

    Certainly not through growth, since active users are shrinking, no doubt the reason why Prospero cut them out of the last Key Metrics. So it must be all these shinies then, like the pointless orange “Loading…” you get in inv now.

    Oh, and it would be nice for those of us on Nicholaz viewers, 1.19.1.4 versions, ie the current official rather than the joke RCxx, not to have a Mandatory Update notice displayed right where the Status Reports are…

    Always two steps forward and three backwards with LL.

  64. linnie says:

    #1 said “1/ maybe it is ‘fun’ for free accounts who invest 0 l$/us$/€/£ ….

    It pisses me off everytime i read some prmium users like you blaming us on free accounts not investing anything at all in SL. I buy things, make some stuff that i sell for less than nothing but my main source for getting linden $ is my credit card.

    Please think about what you say about free accounts before saying it.

    Saying that i wish the lindens good luck with this roll-out, i have close frineds waiting for the bugfixes it will correct 🙂

  65. Alfons says:

    I think its getting better… and must admit that RC 1.20.14 works…stable for me. Yes I do crash from time to time…yes I have lost items during crashes…but life goes on…

    So…good luck Lindens 🙂

  66. Beer4breakfast says:

    /me Puts on Crash helmet and shoulder pads in anticipation of rolling restarts

    /me Pops the top off of a beer and pops some popcorn and relaxes in his recliner watching the snivel fest

    Good Luck LL hope this one goes without a hitch

  67. Tristin Mikazuki says:

    Hmm and hour late… I hope this dosent mean this is gona be a rough one…

  68. Prospero Linden says:

    Naah, it’s just one hour late because I was a bit slow getting started…. It will still be done before 10AM unless Bad Things happen, and so far it’s looking good.

  69. Sedary Raymaker says:

    @26: RL is better anyway!

    A few aspects sure, but overall? Not by a long shot.

  70. Dark_Spyro_003 says:

    @1 cosa nostra:
    I am a ‘free’ account, I buy my L$s through a third party exchange.
    Why do you say they need to stay professional with paying customers? Do you mean that they should be impolite to ‘free accounts’ ?

    @Prospero Linden Thanks for keeping us up to date so well, happy you guys found the bug =)
    Keep on the good work.

    — Christophe003 Carter

  71. Tristin Mikazuki says:

    Cool! I hope its smooth lol
    @69 I agree in RL you cant get paied to sit on a bench or watch people rez heh Though I do think some peoples login to RL has bad bandwidth lol

  72. Says pray and crosses fingers all gos well.

  73. ChatNoir Moonsoo says:

    @60: I don’t see a need for that, you can already define your own roles for your group(s) and assign the individual rights/powers to each role. In effect you can create “sub-groups” within a group.

    Splitting up some of the combined rights (more granularity in rights) though would be a good idea…

  74. Jayden B says:

    My region has been updated to “Second Life Server 1.23.2.92647”

    My Teleporters have become very slow, sometimes so slow that they now die (they are temponrez beams) before they arrive at their destination (ground to 4000M)

    I would say that the speed factor of this move is 1/3 slower than it was.

  75. Soap Clawtooth says:

    /me grabs his raid and waits for the next bug to appear 😛

  76. Jayden B says:

    Hmm, my moving prim cap is also amazingly slow now, where the 30 or so prims that used to move when I touched it used to be instant they now take a long time.

    Was something done to llPrimitiveParams? that could affect moving a prim inside a linkset?

    Maybe best to revert?

  77. Prospero Linden says:

    Jayden — if you find a big that you can post a reproduction for (so that one of us at Linden can reproduce it ourselves — a necessary step for figuring out what’s going on), please post the bug and all the information needed to reproduce it on our issue tracker: https://jira.secondlife.com

  78. shuky says:

    The second life of SONY is coming LL hahaha good luck

  79. Cincia Singh says:

    @16 Bob … you AR’d someone for the phrase “bugger off?” OMG you are such the little drama queen! Someone should AR you for trying to waste LL’s time with such trivial things AND for posting off topic in the blog. And, as Dr. Johnny Fever would say to you, “booger!” (hehe)

  80. Jayden B says:

    Jira is too hard Prospero, there should be a simpler method to report it.

    And yes, I have watched Torley’s video and I started that long running “friends online not showing” Jira, I just hate the system.

    It’s cool if you decide to roll it out, at least people who read the blog will knoe what caused them to fall 4000M to the earth when their TP beam dies in transit.

  81. Cincia Singh says:

    @64 … you said “I … make some stuff that i sell for less than nothing …” so, you pay people to take your stuff? Just wondering hehe

  82. Cincia Singh says:

    @ 80 … I’ve filed Jira’s … it’s not so hard?

  83. Sedary Raymaker says:

    @64: It pisses me off everytime i read some prmium users like you blaming us on free accounts not investing anything at all in SL.

    To be fair, he said “free accounts who invest 0 l$/us$/€/£”. If you’re buying linden dollars with your credit card, that obviously doesn’t include you.

    Personally, I don’t have a problem with people who don’t have a premium account. Heck, not needing a premium account is one of the selling points of my rental properties! I think a lot of people forget that those with free accounts can still put money into SL, they just can’t buy mainland. (No big loss, really.)

  84. Sedary Raymaker says:

    @71: in RL you cant get paied to sit on a bench

    You can if you look pathetic enough. Granted, you won’t get paid very *much*, but then, neither do SL campers.

    Now, if I were to try and sit on that bench in an au naturel state like I do on SL, then the results would be quite different. Hehe.

  85. Edward Kyomoon says:

    @61
    If you dont spend much time inSL then why do you spend so much time being an a$$ on this blog? O.o

    @anyone crying about udpates-this is how it goes and yes restarts are MUCH betta then 6hr shutdowns every wednesday

    good luck today Prospero. 🙂

  86. Beezle Warburton says:

    Object communications are slooooooow since the update.

  87. Bagracer Bergson says:

    *Throws rock* only because it has put our business sim JB Islands 1 offline the entire morning and still remains offline

  88. Beer4breakfast says:

    I was just on a region who just had their region restarted and updated only to find 90% of their store and prims on land gone. and when i was there investigating it. the region crashed with a total of 6 users on it.. Doesnt look good lol

  89. Tan260 Talon says:

    Hi my region is runing on the new update, I notice a performance drop of 60% with the new update. Where I could send you the data that I have?

    Thank you

  90. Cincia Singh says:

    I was just out to the JIRA and NO ONE has posted an issue for the 1.23 server update.

  91. Jayden B says:

    My Mean Kitty is slow to move and also my Huddle is also very slow to flip the buttons when you change pages. Two other items with animated linked prims.

  92. Kimo Junot says:

    OK now how did I guess this “restart” would be all borked like normal??? Our sim went down this morning and came back up over and over and over and now it wont come back up. They tell us that they have to do a roll back and it comes back up for 30 seconds and poof it is gone again then they say we have to submitt a ticket and then they they us it will stay off line til they figure out what they did… THIS IS WHAT WE PAY FOR??????

  93. Jayden B says:

    Sine Wave, with just 19 people in it is running very low dilations, normally this place can run 40 agents with very little lag.

    Also Second Life Server 1.23.2.92647

  94. Jayden B says:

    Frame time in my own well adjusted region, normally rock steady on 1.4ms is now jumping from 2.0ms to over 10.0ms, definitely not stable and very poor. Script performance has also dropped remarkably.

  95. Paulo Dielli says:

    There is definately something wrong with the new server. MS times AND SCRIPTS have been DOUBLED !!! I heard it from another sim owner, so I just checked my own sim. Yesterday and before I had about 160 scripts running and 1.3 ms. Today (Second Life Server 1.23.2.92647) I have more than 300 scripts and 2.8 ms !!!

    What the heck is going on now???

  96. Zorins says:

    @33

    Here here. Be happy that they bother tell us ahead of time now; the only warning was often posted late the day before, or worse: at the same time as the grid’s shutdown. A very nasty surprise on Wednesday mornings.

    I’m not all that wizened when it comes to my account’s age, but I do have the sense and knowledge to know better than to complain about routine system maintenance. The fact that the grid was half as stable as it was is a testament alone; implementing the kind of server network that SL has to run is a logistical nightmare, and the LL crew have done a good job, especially recently.

    @56 You hit the nail on the head. SL had never really been attempted before, and any amount of planning can’t prepare a programming crew for the bizarre anomalies of a completely new system.

    I mean, how many of the original Linden staff had taken a hardcore economics course, or majored in PR? I do not mention this in a condescending manner, it is merely a real aspect of products and systems that are flukes. A bunch of like-minded people come together and almost accidentally build a system that is ground-breaking. Perhaps they weren’t the best qualified for the task, and most likely they didn’t believe it could get so huge, but they did it and it did. That is the reality of product development, whatever the product.

    The problem SL has is that it was built around a pragmatic approach on a smaller scale, and almost overnight became a phenomenon. I am sure the Lindens had aspirations that the system would one day become the kind of “other-verse” in Snow Crash etc., but I doubt that they knew it would become SO bloated, cumbersome, and large by this time. I mean, the Grid is running a simulation the size of 1/3 of Massachusetts, with an account population of over 14 million. I don’t care about how many are on at any given time, that is a HUGE, absolutely MASSIVE amount of account information. WOW only has like 10 million, and all that needs to be stored there is a paltry inventory of hard-coded items and some character stats. We’re talking about complete virtual lives here.

    Even if SL loses out in the likely on-coming VR Platform “wars,” LL gets the medal of honor for giving it a shot and getting as far as they do. Though, I have a feeling it won’t end that way.

    And if they would just fix the geometry of the avatar mesh…

  97. Balpien Hammerer says:

    Prospero, this script performance loss is very real. I just spot checked again since last reporting it in the last rollout attempt, cheking out scripted objects in the 1.23 regions versus the 1.22 regions. There is a 1.5x to 2x increase in script execution time.
    http://jira.secondlife.com/browse/SVC-2649

  98. Cappy Frantisek says:

    /me grabs the popcorn and laughs.

    It’s raining in the RW here too, guess I’m just as likely to get wet inworld or not!

  99. Stress B says:

    Hi, just wish to confirm what #89 says, I’ve experienced a performance drop of around 75% on my sim (open sim). Using my usual bench marker, yesterday I had 900+ no scripts, today 234 no scripts

  100. Wolfie Waves says:

    Yep, 1.23.2 defiantly killed script performance. I had to disable scripts in Eternal Calm once it got the 1.23.2 update because it was causing a crap load of time dilation. EC was perfectly fine before the 1.23.2 update. Looks like your gonna have to do yet another rollback :p

  101. Andromeda Quonset says:

    I agree that there is a huge performance hit with this rolling update. My AV has a “Advanced Border & Performance-Scanner v4.6a (WEAR ME!)” by Thomas Conover. Queue was showing a performance level of about 2300 prior to the restart. After the restart, the level dropped to 700. occasionally, it goes down to 149. This is not good.

  102. shockwave yareach says:

    I cannot verify script performance at this time. But it would go a LONG way to restoring my faith in LL if they took a look at it in the updated servers (with a nonLinden account) and assessed the problem. Is this a temporary hesitation as the database for the region is updated or is it a permanent problem? If permanent, please roll things back.

  103. Balpien Hammerer says:

    Oh crud, it happened the last time and it happened again (loss of rezzed objects), except this time the upgrade restart lost several expensive no-copy objects. Where can I get a rollback started?

  104. pat chemistry says:

    @95, our script time doubled as well, yesterday 7-9 ms, today 14-15 ms, while the debug tool reports: 756 scripts, taking a total time of 4.0 ms. I tried for a few minutes to report it on Jira but it is too complicated for me I guess, I get lost on the start page lol and do not have much time for this. Luckily we are still way below lag-levels here so it is not a big problem for us.

  105. I confirm the problems with script performance. Script times doubled up with the new server software.

  106. Jessica Hultcrantz says:

    Hehe, the history repeats, and the only thing that isn’t learned from the history is that the history repeats…

    Fire the quality controllers 😛

  107. Don Duke says:

    My second Rolling Restart for today O_o first one took my sim out for 12 minutes. I hope this one will pass faster.

  108. Spank Lovell says:

    Hell! I feel for you guys.

    I can imagine how frustrating this is. If you need a volunteer sim to test this on then please IM me, I’d gladly help out if possible.

    is now a good time for a group hug? 🙂

  109. Beezle Warburton says:

    Areumdeuli is running 0.5-0.7 Time Dilation

    You are at 264912.2, 238096.9, 79.6 in Areumdeuli located at sim5175.agni.lindenlab.com (8.2.32.176:13002)
    Second Life Server 1.23.2.92647

  110. Alison says:

    I sympathise, I really do. With Linden with the shop owners, with all of you.

    But surely, we all know how the system creaks when we get 50k+ online, and server restarts must punish the infrastructure and the asset servers to hell.

    I suspect you cant find the problem cos there aint one, – you’re just trying to ask too much of the who darn setup. No wonder it’s fine when you try just a few sims.

    maybe do it over a week? spread the load?

    you canna push her any faster capn!

  111. Jack says:

    (rollback)

    LOL

    sorry, i couldn’t keep this useless comment for me…

    today, when the region i was in restarted, i tought: And tomorrow the rollback… and so is…

  112. Deeso Saeed says:

    Dang! Good look with this one guys =(

  113. Deeso Saeed says:

    I meant luck

  114. Ann Otoole says:

    Have we thanked you enough for developing the rollback capability?
    Thank you!

  115. Balpien Hammerer says:

    Folks, I opened this bug:
    http://jira.secondlife.com/browse/SVC-2649
    …to track the damaged script performance. Vote on it. Presently I’m told it’s just my problem, hence no fix and not even assigned to anyone (most discouraging). Take the time and vote on this bug.

  116. Alyx Sands says:

    Oh dang. Have some good karma come your way….let’s hope you find out what actually causes the crashes and slow downs.

  117. Tan260 Talon says:

    The rollback work at sim3327.agni.lindenlab.com (216.82.21.62:13001)
    Second Life Server 1.22.4.90499

    Region performance is good now.

    Thank you

  118. Beer4breakfast says:

    Well thank goodness i had my Crash Helmet and shoulder pads on. I suffered no bruises or concussions

    Think its time for more popcorn . It is like watching terminator 3 . With each episode it gets worse and worse..

    Good Luck Dev team we hope you all can figure it out. Or maybe just skip this one entirely and go straight for Mono release……
    heh one can only wish

  119. Mook says:

    Why dont you leave a day or two between the pilot roll and the first half?

    Night time till the next morning is so brief its like, why bother? It doesnt seem like its giving a useful forewarning.

    (not a dig, just cant see the sense in rushing it)

  120. fword utorid says:

    “between sobs and gnashing of teeth”… wtf. look, software development and server maintenance is hell… you’re doing the best you can. don’t cry or need dental work because of it. if anyone gives you garbage about unexpected issues in the world’s most complicated server farm, tell them my initials.

  121. Cincia Singh says:

    Roll, roll, roll on back! It doesn’t look like we’ll get the bug fixes in 1.23 anytime soon if there’s no single obvious cause. Life goes on and we live on as we have for weeks happily SLing our time away! Keep at it Prospero, you’ll figure it out! I have faith in you and your team.

  122. Cappy Frantisek says:

    /me passes the popcorn!

    wouldn’t a half a day down make more sense? Or is it the load that is causing the stress?

  123. Bud Parnall says:

    Just logged onto my sim. New version. VERY Slow. Like Molasses. Went to a neighboring sim with the old code… back to norm.

    If at first u dont succeed!!!

  124. Good Luck All! Keep hammering on it!

  125. just me says:

    roll roll roll your boat gently drown the stream
    ok that was silly, good luck guys

  126. Cairo Serevi says:

    I’m with you guys!!!
    Keep heads up….and take a break, sometimes a walk around the block works like a charme 😉

    C.

  127. Day Oh says:

    Best of luck guys ❤

  128. Raban Laborde says:

    Again a Failure – it’s the third try on 1.23 and again you failed.

    Only one word: Botch.

  129. Vivienne Graves says:

    Whoever is posting these announcements to the blog might want to look at a calendar; 2007 was last year.

  130. sirhc DeSantis says:

    Oh Joshua – positive karma directed your way. That sounds like a rough one. Good luck with this

  131. Titzalina Nieder says:

    Well, the rollback certainly helped my husband and I…NOT!!!!! We went back to our store in Cryphia to find it had completely GONE. All that was left was a few pieces of furniture floating in the air. Soooo…now we are having to roll back to the 24 hours previous to hopefully find our store and stock. So fingers and toes crossed our business will survive. That’s of course, we don’t suffer a nervous breakdown before that…SIGH!!.

  132. Morpheus Nieder says:

    Pass me the Valium Titzalina, I’m starting to twitch ( Its OK Guys, I’m her Hubby 🙂 )

  133. Rice says:

    @4 – Bob,

    The last Linden that told us to bugger off was a supposedly a “supervisor”, and his “IE crashed”…

    Aside from that, its the same old rolling restart phenom. How could anyone – I mean HOW DARE US RESIDENTS – demand a smooth operation? The smoother things are the more unemployed Lindens. Think of the Children!

  134. Titzalina Nieder says:

    Ok sweety, we will keep each other company in the local nuthouse. Oh and Morphy, I just took the last of the Valium myself.

  135. Zena Juran says:

    Joshua, I did make plans for this update to fail. I wish it didn’t fail and the code worked as it was expected to. But in review, the server code has been flawed since April. And every update since then (I don’t care how many bugs have been “fixed”) has resulted in an accumulation of flaws that any attempt to continue with the same approach and structure will only meet with failure. I did have fun once upon a time here in SL. I’d like to have fun again. 🙂

  136. Joshe Darkstone says:

    /me positively oozes with positive karma, you guys must be pretty knackered with this one. 2 things I have noticed during this attempt that are good news; The land store was taken down for the duration, and my store sales do not appear to be suffering the way they normally do during a rollout. Im hoping those 2 things are related, and that you take some notice of the fact 🙂

    Perhaps a similar adjustment during future rollouts would mitigate the effect on store sales for those of us that are concerned with them 🙂

  137. Question for JOSHUA LINDEN (not anybody else, please): Why aren’t upgrades tested for longer periods of time on a few special sims before they are applied to the entire grid? I think that at this point it is more than obvious that a few days are not nearly enough.

    I, as well as several other users, have asked this same question many, many times, but for some reason it keeps getting ignored by Prospero.

  138. pat chemistry says:

    Thanks for the rollback dear engineers! 🙂 all back to normal now! thanks so much. script time now 8.7 instead of 14-15.

  139. Positive Karma I think is called Dharma. =) I wish all of it to the engineers who have been working on improving things.

    Something might have broke though, I can’t copy textures from a notecard suddenly after doing it for days. Relogged several times, tried different sims.. no luck. I’ll try later.

  140. Tristin Mikazuki says:

    Dang well maybe next time ;-(

    I should start selling pitchforks and tourches for rolling restartes heh.

    Maybe if you guys got more people in the beta grid to test it for longer then a week… like let them buy stuff on beta and transfer it to the main? Ya might get more people on beta grid for testing that way or make an inworld plea for people to goto beta and test the stuff.

  141. Renee Faulds says:

    What you rolling the thing back AGAIN???????

    This the second restart today !!!!!!!!!!!!!!!!!!!!!!!!!!!

  142. Renee Faulds says:

    Please please please !!!

    Start getting it together !

    jeeeez

  143. Sarwat Miles says:

    ouch 3rd time.

    oh well better luck next time guess you need lots of it 😉

  144. @137 toybodacho Ireland:

    Don’t beat up on Prospero Linden too much – he juggling a lot of things, and trying to keep the grid running smoothly before, during, and after these updates. He manages to comment more than I do, which is amazing!

    As to why the upgrades aren’t tested for longer – well, at this point, the 1.23 Server changes have been on the Preview Grid since July 2nd (3 weeks). But most of the issues that have been found did not show up on the Preview Grid. It’s much smaller and gets much less activity and traffic, so it tends not to manifest the same problems – such as crashes under load. With the change rate we want to maintain (i.e. making lots of bug fixes), we also can’t mandate that the code sits on the preview grid for extended periods of time after each tweak, or we’d push to the main grid, notice an issue, roll back, push to the preview grid, then e.g. wait a week before iteration. At that rate, we’d never ship!

    That’s not to say we can’t improve our processes. Prospero Linden and others are working on a strategy to get more focused testing on the Preview Grid, and we’re rapidly iterating on our analysis tools so we can detect anomalies on smaller grids – or subsets of the main grid – before they are widely deployed.

  145. @140 Tristin Mikazuki:

    Joshua Linden sneakily blends into the crowd, waving a torch and pitchfork, storming towards the castle gates. And hopes no-one will notice.

  146. McMillan says:

    Your doing your best and that counts for a hell of a lot! Thanks for the efforts and keep it up!

  147. Cappy Frantisek says:

    om manny padrem ohm

    I hope it goes well!

  148. Indigo Mertel says:

    As a developer, I know how these things are all too well. Lindens, you have all my sympathies. I know you are doing the best you can.

  149. JayR Cela says:

    Hang in there Folks / you will figure it out :_)
    Best of Luck

    JayR Cela :_)

  150. Cincia Singh says:

    @ 144 … why not get the residents that everyone says have all those alts and bots to hit the preview grid and put the new builds under serious load? (sarcasm, not serious!) Good luck with the investigation and I hope to see you pushing out a new 1.23 soon!

Comments are closed.