Simulator Stability Improvement And Crash Rate Reduction Results

Six months ago we announced the beta test process of a new Second Life simulator, with the explicit goal of improving simulator stability. The new simulator went through six months of public beta testing, was in service on over 635 Second Life regions throughout the beta process, and during the public beta Early Adopter Program was found to dramatically reduce simulator crashes.

In addition to improving the Second Life experience by reducing the number of times a region is restarted (which forces all avatars off the region at the same time), lowered simulator crash rates also reduce content loss by reducing the number of times a region must be “rolled back”. Whenever a region is rolled back, any content changes since the last snapshot are lost, and the increased stability of the new simulator has likewise reduced the number of region rollback, and thus reduced the rollback content loss rate substantially.

Now that we have a week of active experience with the new simulator throughout Second Life, I thought it would be good to share the crash rate details so that you can see the impact of the Havokβ„’4 project team’s work on increased simulator stability.

Below are two charts to show the stability improvement provided by the new simulator. The first chart shows the number of regions that crashed across all of Second Life, per day, due to physics engine crashes. Physics engine crashes were the most common cause of region crashes, and have been reduced by 97.2%

Physics Crash Rate Reduction

The second chart shows the number of regions that crashed from all crash causes. We also resolved some non-physics crashes during this project, resulting in a 68% reduction in region crash rates, across all of Second Life due to any cause.

Crash Rate Reducion from All Causes

We are continuing to resolve open issues with the new simulator, to make it even better both in terms of crash rate and functionality, and will be posting updates as this work is done.

Thanks again for all of your support and involvement,

Sidewinder Linden
Havokβ„’4 Program Manager

This entry was posted in -Miscellaneous, Announcements & News, Bugs & Fixes, New Releases. Bookmark the permalink.

150 Responses to Simulator Stability Improvement And Crash Rate Reduction Results

  1. Anthea Shilova says:

    CONGRATULATIONS !!!!!!

    Well done Havokk 4 team, and thanks for the great communication Sidewinder.

    Thanks agains !

    Woooooooot

    –Anthea

  2. U M says:

    well thats what H4 was puspose to do πŸ™‚ Great work!!!!!!!!!!!!!!!!!

    Usagi

  3. Totally Fooled says:

    The crash rate is lower because for over one month now the Crash Logger is not working and we are being Logged out so that doesn’t count as a crash.
    I don’t think data is valid.

  4. Katherine de Leon says:

    Congratulations, Linden Lab (and Sidewinder and Andrew).

  5. Gattz Gilman says:

    What about Class 4 vs Class 5 sim crashes?
    For the people who have sims running on older hardware running the new physics engine, are there slow downs or more crashes versus having it run on Class 5 hardware?

  6. Gattz Gilman says:

    @3 Totally Fooled – They are talking about sim crashes, not client viewer crashes.

  7. KMeist Hax says:

    THIS WAS A TRIUMPH
    I’M MAKING A NOTE HERE
    HUGE SUCCESS

    p.s. >>3 you’re thinking of viewer crashes, not server crashes. the viewer crashes more than Windows ME currently.

  8. Very dramatic results! What’s next?

    πŸ™‚

  9. Sidewinder Linden says:

    @3 Totally: The crash logger component that was not working was the viewer (client) side. The server side crash logging is fine, and this data is for the Havok4 project which is a server-side enhancement. /Sidewinder

  10. ChatNoir Moonsoo says:

    @3: if that was about *Client* crashes, I’d agree, but this is a bout *Server* crashes which don’t get reported (or not) by your client…the servers keep logs of their up- and down-time (simply put).

    If we do criticize LL, and there is plenty of stuff to criticize them for, let’s keep the facts straight and refrain from nonsensical statements that only lower the critic’s (and this the criticism’s) credibility.

  11. Sidewinder Linden says:

    @5 Gattz: Class 4 vs Class 5 is an interesting question. I do not know the answer off the cuff but will look at the data. My suspicion is that this is independent of server class, but I will check to be sure. /Sidewinder

  12. U M says:

    well it was longer for me! the crash logger wasnt working for 6 weeks.

  13. Dilbert Dilweg says:

    Yes Well Done Havoc4 Team . I am very impressed with this server release. On a regular daily cycle our Sim would crash at least once a day . With havoc4 we were able to run 5 days without a restart and without a crash. Aside from the minor glitches that will soon be fixed i am very happy with the results. Thanks Sidewinder for your dedication and hard work on this project!

  14. Sidewinder Linden says:

    @12 U M: The crashes reduced by this project are server-side crashes. The physics engine for Second Life runs on the servers at our data centers and is part of the server simulator code. That is the area that this project was focused on for crash removal. /Sidewinder

  15. Impressive graphs! And not only because they show the improvement, I’m als a bit amazed by the fact that previously 1500 sim crashes a day was quite a normal figure! Then again, I’d need to see it in perspective, 1500 is a lot, but how much percent of all running sims is that.

    Anyway, congrats on a job well done πŸ™‚

  16. Vivienne says:

    The sim crash results are fine. Great!! Sidwinder, you are the best.

    Now please take over viewer development.

    πŸ˜‰

  17. william Fish says:

    in the meantime clients are crashing people left and right if they are using it for more then a few hours at a time without relogging. Memory draining is the key problem.

    ALSO rotation scripts seem to stop working all together unless you re-rez that object.

    Other then that yes sim performance is 100% better…

  18. Sidewinder Linden says:

    @15 Daedalus: I don’t have the exact number at hand, but the total region count these days is somewhere around 17,000 last I heard… If you’d like an exact number I can find it, but this is probably close enough for the rough percent you were looking for (I hope). /Sidewinder

  19. Sidewinder Linden says:

    @17 william; The viewer teams are working to reduce viewer crash rates.

    We do know about the llTargetOmega issue (causing random stops of rotating scripted objects) and have invested days of development time so far trying to find good ways to create and reproduce the problem, as well as to find proposed fix locations. This one is elusive because it is so infrequent, happing every hour or day, which makes it very hard to hunt down. We will find it, and are actively pursuing the cause.

    /Sidewinder

  20. Damona Rau says:

    on 04/05/08 SL has crashes?
    Your kidding, right? For your remembership, 04/05 was saturday with the “big offline”.

    I believe, that the sim stability with the new server is better, but i believe also, that many other problems comes together with the new server version.

    Maybe you’ve read the blog the past week, we have had alot of problems with logins, TPs, transactions etc. All this failures are based on the database / asset server and not the simulators. But if many things borked and not usable, this will decrease the crashes also. So I’m not sure if this statistic is a true one. Let’s talk again, after the main problems with the database is fixed.

    I hope things will be better soon, because i have round about 60% lost in my business with all the problems (transactions, failed TPs, downtimes – and yes, i know it was the provider, not LL)

  21. Captain Noarlunga says:

    @16

    Please take over the whole company …….they sure need a lesson in customer service and Sidewinder and Team are the only true professionals there. Well done Sidewinder and the Team. Pity I wont be staying in SL long enough to appreciate it, after Jacks attrocious handling of SIM pricing.

  22. Solomon Devoix says:

    There are still some quirks that need to be ironed out with H4, but by and large… well done, Sidewinder and team! πŸ˜€

    Your group’s focus, progress, and above all communication are models that the rest of the LL programmers/groups can only dream about. Kudos, well done, and all that!

  23. Solomon Devoix says:

    /me nominates Sidewinder for project manager on fixing the asset servers!!!

  24. Ultra Nefarious says:

    Man, then I would love to know what I’m doing wrong, because I crash more than ever now.

  25. Pingback: JIRA to Clone Sidewinder « Dusan Writer’s Metaverse

  26. Sidewinder Linden says:

    @24 Ultra: Viewer crashes happen to one person at a time.. and I suspect that is the type you are talking about. The crash reduction provided by this new simulator version is on the server side. These crashes throw everyone who’s on a region off at the same time, and in many cases require a restore of the region configuration before the server will start again – causing lost content. There are other teams working on viewer crash rate reduction. /Sidewinder

  27. Wake up in SF please says:

    SL data has no credibility because SL controls the data. Pretty simple.

  28. MadamG Zagato says:

    I rarely crash, but suddenly tonight I have been crashing a lot which is scary. Just ironic that I came over here and found this blog post titled “Simulator Stability Improvement And Crash Rate Reduction Results”. πŸ˜€

    Thanks Sidewinder for the confirmation on the llTargetOmega. I think I’ll go make some bubbles and hopefully things will be better in a few.

    Maddy

  29. Miles Beck says:

    Nice work and excellent communication! Congrats. And Thanks.

  30. Sidewinder Linden says:

    @27 Wake up: I offer openly to discuss how this data is collected with you. I offer as additional evidence a discussion with region owners, and ask them how often their regions crash now, as opposed to before the update. Most in the Early Adopter program reported to me that they never or hardly ever see region crashes any more, which is consistent with the data. Owners of regions converted in the full rollout have been sending me messages with the same commentary, so I suspect that this data is correct. /Sidewinder

  31. Captain Noarlunga says:

    @26 ….. Sidewinder….with all due respect…the major effort seems to be on pushing through new features as per the latest Release Candidate with new Dazzle and other silly stuff. No one seems capable of fixing the basic issues …. and if they are working on it….why cant this be communicated to the thousands of frustrated customers out here? You are about the only person who responds to people…the rest of the blog entries are very rarely open to comment and never have responses from the teams involved. I am sure we would all like to hear what is being done re. asset servers and database issues in some detail, together with what is beingg done about all the other problems that have been around for so long. I could put up with being screwed over SIM pricing if I thought there was a chance of this system working properly in the near future. The impression right now is that all the activity is to get more people hooked and to make it all look financially interesting for an IPO.

  32. Ishtara Rothschild says:

    That’s nice. I’d appreciate if my sim would crash less often and I will, for now, assume that there’s real data behind these graphs, that they were not only produced to have something positive to announce for a change.

    My business would be better off though if my customers would receive their wares after paying for them. And if they receive their purchases, which is rarely the case these days, could use them without “item missing from database” errors or missing scripts. I’d rather live with 2 sim crashes per day as it used to be, but have satisfied customers who don’t give me negative word-of-mouth advertising because your asset server is falling to pieces.

  33. Anthony Beatty says:

    its weird but since this new veiwer im crashing 10-20 times per day even if i lower gfx settings all the way down , on the old RC veiwer i could have totaly maxed settings and not crash for a week, now im getting all kinds of bugs memory errors ,constant crashing ,severe loading times i mean its taking me 30 mins to load a sim now… i hope these things will be resolved my sl is almost unusable now, is this just me or is these problems for alot of people?

  34. william Fish says:

    Sidewinder~

    ty for the response. I figured it was reported already. I wonder if it’s the type or script used for rotation because i have one rotation ball that never stops spinning… meaning it’s good. But only that one.

    if i can help let me know.

  35. That looks like a very good improvement. Of course, since that took a large portion of sim crashes out of the picture it should be relatively simple now to isolate and fix the other sources of crashes : )

  36. Jopsy Pendragon says:

    *THANK YOU*THANK YOU*THANK YOU*

    I can’t tell you how many times I’ve accidentally crashed my home region because of what I was *certain* was physics collision problems. (objects being rezzed on top of objects in motion… objects in motion crashing into each other, etc).

    I’d completely given up on one of my favorite projects because I couldn’t, in good conscience, sell things that I knew would increase the risk of region crashes.

    This is a big beautiful *GREEN LIGHT* I’ve been waiting to see for a long time.

    *THANK YOU*THANK YOU*THANK YOU*

  37. Sidewinder Linden says:

    @32 Ishtara: Which region are you talking about? I’d be happy to check the stats for that region… /Sidewinder

  38. Vittorio Beerbaum says:

    @15: hmmmm… about 15.000 resident regions at the end of Feb, they sell 100 per week (Jack words) so they sold about other 500 until today, then you have to add LL simulators (do they own 1500 regions? …i don’t know, it’s possibile, considering how extended is the governor property) …so Sidewider numbas are almost correct (what news). πŸ™‚

  39. Ciaran Laval says:

    Good stats Sidewinder, especially on the physics based crashes. If it’s like this in 6 weeks or so then the proof really will be in the pudding.

    Nice to see some useful stats on here.

    I don’t suppose you can do anything to deal with the land price crashes can you πŸ˜‰

  40. Way to go H4 team! Great job! And thanks for being so responsive about the bugs.

  41. Impressive! Now if only client crashes could be reduced by an equal amount *poke* πŸ™‚

  42. Back in August at the SLCC I talked to Philip about crashing, telling him I am in game 18 hours a day, and the other 6 hours Im on anti Idle, not loggin out for 5-7 days total. Which means hardly every crashing out of game. Since April 1st. I crash about 10 times a day. Mostly when I open a profile, group or TP. I guess when there are 50% of bots in game running with out a viewer and not interacting with the client, ofcourse there not going to crash. But myself logging in and using SL, I crash way to much now. Most of the time I soft crash, which means I get logged out of SL but I still see the world, I only know Im crashed because the mini map turns red or I get an offline going to my blackberry, which tells me, hmm if Im getting offlines, I must not be logged in to SL anymore.

  43. Argos Hawks says:

    @30
    Sidewinder, you rock! My sim hasn’t crashed since the H4 upgrade went in, and we had frequent crashes before. We keep a crash logger active in the sim, and I monitor the sim stats closely. Not only is the sim more stable, but total frame time is down slightly too since the sim is spending less time calculating physics. Can’t wait for the Mono code to kick in!

  44. Creed Bimbogami says:

    This crash reduction is a bunch pf hooey. My sim has been down for the better part of a day. It crashes and then you say problem fixed? I have had prims dupilcating themselves, prims floating in the air and i’m paying a high enough tier and unable to go where I built a house. If this is stable I long for the older setup. This is the second week of outages and all I do is lose money spent on tier.

  45. T N says:

    If I’m reading the charts correctly, non-physics crashes actually increased significantly after this update.

    It seems like the average # of daily physics crashes for the seven days leading up to the update was about 1308, and the average # of “all causes” crashes was about 1788, leaving about 480 non-physics crashes per day.

    After the update, the average # of daily physics crashes dropped to about 35, and the average # of “all causes” crashes dropped to about 574, leaving about 539 non-physics crashes per day.

    That’s a daily average non-physics crash rate increase of about 11% since the Havok 4 update. (All numbers estimated based on the graphs.)

    Possible explanations:

    1) Previously, it would have crashed for another reason anyway, but it happened to crash on physics first.

    2) Some crashes previously attributed to physics are now attributed differently.

    3) The physics code is causing problems for some other part of the server code due to bugs or strain caused by increased activity.

    4) The rest of the simulator code has become less stable for a wholly unrelated reason.

    5) Sunspots.

    There may be other possibilities as well. It would be nice to know which it is.

    (None of this should detract from the sterling work of the Havok 4 team to increase stability.)

  46. Desmond Shang says:

    Hi Sidewinder, casual anecdotal evidence in Caledon pretty much confirms your graph, it’s been pretty stable region-wise. Other factors – teleporting and whatnot – well, that’s a different thing and a different story.

    We have had a lot of objects de-linked by the changeover; more than I had suspected originally but nothing unmanageable – maybe a few per region of linked objects that were ‘just barely hanging together’ according to the old link rules anyway. The new, seemingly looser link rules has allowed for easy restoration in all cases so far.

    Also, the newly prim-doubled openspaces are doing great!

    All in all a very good show and a very good transition to better physics.

    Desmond Shang
    Independent State of Caledon

  47. Sidewinder Linden says:

    @45 T N: Yes I noticed that in the data as well, but haven’t yet had the time to research it. I do suspect however, that #1 is the probable cause. There were enough cases that the physics engine would spontaneously crash first that it is reasonable that it was masking other problems that would have showed up had the region survived the physics problem. /Sidewinder

  48. Captain Noarlunga says:

    @42….that is weird…… I can run SL happily (with the normal bugs that are present) on an ASUS Eee 4G mini PC that has a 4gb ram drive, 1GB ram and with a 4GB SDHC card as the D Drive. The processor and graphics are below SL recommended standards but it all works fine (running WinXP). I wonder if top spec. PC’s are just to complex πŸ™‚

  49. Sidewinder Linden says:

    @48 Captain: Just to toss one more thing in the mix, dual displays can cause odd video driver bugs to appear that don’t show in single display rigs, and the bugs found sometimes depend on the particulars on all sides (which port, which display resolution(s) etc… Too many variables πŸ™‚ /Sidewinder

  50. Ener Hax says:

    all i know is that sl looks better, never crashes on me, does not bog down my pc, and is tight.

    i have all my graphics set to the max, even distance and keep high frame rates and it just works really well. =)

    thanks Havoc team for doing what must have, at times, been a really daunting task! *ener can’t even hook up a printer* o_O

    congrats all of LL πŸ™‚

  51. It is good that you are working on lowering the amount of crashes in Second Life. Please keep your software people working on all usability issues including inventory loss and failed transactions. Is there anything that can be done to reduce crashing in crowded locations? Reveal Magazine was promoting an event for Relay for Life and I experienced many crashes during the auction and DJ contest.

  52. Miles Sullivan says:

    No it doesnt crash, it slows down to a crawl with upwards of 40-50 pending downloads. Everyone gives up in disgust because clothing, profiles, etc do not load and then the pending downloads have time to recover.

  53. Rift Rehnquist says:

    Can we now see a Cool Graph of the New RC Viewer, to see how many times it has crashed. Assuming that people log their crash report…which they don’t. Cause they figure well it’s pretty much the same cause “I did Something, viewer Crashed it will do it again in a few minutes why log every time”?

  54. @48 Captain maybe LL can change my last name to crash. And Ill be known as Spontaneous Crash instead of Spontaneous Radio.

    Just a Spontaneous Thought

  55. Lindal Kidd says:

    Excellent job, Sidewinder and Havok 4 team!

    Now, please fix the creeping bug in rotations (objects stop rotating after a day or so) and the unstable hover (avatars sometimes sink, especially after turning).

    And my .45 throws a little high and to the left. Could you do something about that, please? πŸ˜€

  56. Hey Sidewinder says:

    Myself and many other scripters would like to know when you will be addressing the issues of many scripted products Havok 4 has destroyed or crippled . Theres alot of talk in-world you don’t hear about the products that for the most part , just don’t work any more and alot of merchants that made the products have a total nightmare on their hands . What are you going to do to help these folks who worked so hard to make sl a better place only to have their efforts underminded by Havok 4 ???

  57. Damen Hax says:

    regarding TargetOmega,..

    I have found that only things I have made and compiled before the last update fail to turn/spin.

    EVERYTHING I recompiled worked and moved like it should, and continues to spin as I speak. The majority had to do with events activating the TargetOmega function (like touch or linked message for eg) as opposed to having them as a state.

    In short, nothing using TargetOmega on the random things I’ve created fail to work, all spin as planned. Only certain scripts where it required an event to trigger the function i found to have issues with.

    Great job with h4 btw, it allows me/us to try things that would have been simply too laggy in the old environment (for the most part), shattering linked prims and having them float away at given speeds for eg =)

  58. webmasteramadeus says:

    that’s wonderful dears, considering i just crashed.

  59. tango bingyi says:

    I believe Havok 4 will be better somehow but i must be the only one to expreience worse performance than ever. How come i was forced to upgrade from 18.5 to 19.1 and crash more than i have ever before? my old 18.5 never gave me these issues.. are some versions better some peeps and not others ?

  60. Talarus Luan says:

    Well, I for one can attest to the reduction in crashes (see http://www.iconia.org/sl/crashquery.php for the Isle of Wyrms stats), though the latest sim build seems to have a nasty habit of crashing and staying down for the count. Both Limbo and Cathedral went down today, and the latter required a region down ticket to get running again.

  61. tango bingyi says:

    only to find he has crashed typing the above comment πŸ˜› standing still lol..

  62. Sidewinder Linden says:

    @56 Hey: We ran a 6 month public beta with hundreds of regions, talked (and are still working with) every product maker who is interested. Had 50 office hours, and are STILL openly inviting anyone with product issues to get in touch so that we can resolve issues… Please make sure that open issues are raised and brought to our attention and we will try to resolve them. /Sidewinder

  63. LoLo says:

    Wow great job sidewinder & team

    BIG THANKS!!! KUDOS!!!

    Keep up the great work I hope you will be doing way more projects (and blogposts!)

  64. Stephe Ehrler says:

    WTG Sidewinder! And thanks for your update on the siggy swimmer issue, didn’t catch you guys are already working on it. Also thanks for dealing with the open space sim issue so quickly! Your team is def the model that other LL teams should follow, responding to resident questions as they arise and keeping us informed on what’s going on. Now I can just sit back patiently and wait, knowing your team is working on fixing SL swimming πŸ™‚ It’s this sort of comunication that people expect and want. We aren’t unreasonable citizens, just need a heads up on what’s going on and why !

    Thanks again!

    Stephe Ehrler

  65. bret barbasz says:

    i crashed so often with the new viewer that i uninstalled it and reinstalled the earlier one. lower crash rates? hardly. my crashes with the new viewer were ten times the number of crashes i had with the earlier one.

  66. Elvis Orbit says:

    As a region owner I have noticed the effects of Havok 4 in a positive way. In both the early adapter and now. My Sim has run stable and as far as I know, has not crashed except for a network issue on the grid. I have restarted it only about 2 times myself since Jan. Great work and it shows.!

  67. Talarus Luan says:

    Actually, I just checked the times on those. It looks like Cathedral was caught in the “700 down region” issue. Not sure about Limbo, as it was several hours earlier.

    Anyway, like I said, definitely a huge thumbs up in the crash-mitigation department. There are a few niggling issues left here and there, but they will likely get mitigated soon.

  68. tabitha Oxide says:

    Crash stats down cuz many of us still cant even login.

  69. Cliff Dieffenbach says:

    I’ve read some pretty ridiculous comments before, but suggesting that H4 undermined scripter’s efforts because of their operable status in an older, less stable platform takes the cake. Things may need to be updated on the object development side instead of some conspiracy theory being suggested.

    Anyway…

    Sidewinder, oddly enough I’m seeing some previous bugs that were tackled show their ugly heads…

    -More than minor sliding on slight grades
    -Steady decent while in hover
    -Falling while climbing stairs

    All the above have jiras for them, but it seems at least these three seem to have got worse recently.

  70. Renato says:

    Funny thing is i sit reading this while awaiting the Grid to stop crashing

    Way to go Havok team!!!!1

  71. Geister Pfeffer says:

    Ok, with region crash, wonderfull,
    but if i only see me, i got crash at moment, and it was the 6th in last 8 hours, so, much more then some weeks before.
    I thnk, my crash rate is going higher, but its only me ?

    ty.

  72. Seven Shikami says:

    Hooray for less physics crashes!

    Boo for my sim being unable to send outgoing IMs and objects, including SLX deliveries, for six hours until a concierge moved us to a new physical server!

    Sorry, guys, but I’m not cheering until one day goes by where I don’t have some transaction/asset server failure, sim crash, communications problem, or whatnot.

  73. Solomon Devoix says:

    Again: “Lower crashes” means lower number of *SIM* crashes, people. As in, “The simulator you are in is currently going down, you’ve been disconnected, press continue to still view chat history…”

    NOT “I rezzed an object / tp’d / looked at a prim sideways and crashed.” That’s a *VIEWER* crash, not a *SIM* crash.

  74. Lance Corrimal says:

    @ almost 50% of the previous commenters:
    sidewinder is talking about SIM crashes… that is the part of SL that runs at LINDEN LABS. or in other words, not on YOUR computer.

  75. eku Zhong says:

    i am getting the descent when hovering again and the falling on stairs sometimes.

    but .. we were in on the early adaptor.. we never crashed once (well not the SIM.. we were offline when the whole world was too) and haven’t since.

    there are some things that dont work on Havok4 but its a small price to pay for SIM stability…

    Sidewinder and team deserve the cudos…
    also for great communication.

    the rest of the borked stuff… still tres borked as is the communication thereof.

  76. Renee Faulds says:

    Very well done Sidewinder and your team !!

    Renee Faulds

  77. Lance Corrimal says:

    @sidewinder: what about almost anything that has physics enabled and interacts with linden water (as in, boats, surfboards, scuba gear, or swim attachments) not working properly? that problem kills about 75% of my favorite SL pastimes…

  78. I’m happy with the reduction in simulator crashes. Way to go!

    Now… since I updated to the new RC last night, I CLIENT crash once every 20 minutes. This is new and frustrating. Could someone e-mail me and tell me if there’s a temporary workaround for this? It seems intermittent and unpredictable. I really don’t want to have to roll back to the old client just to get my work done.

    Please please e-mail me if there’s a workaround. This is a showstopper for me.

  79. Darien Caldwell says:

    To those that think these stats are fake, I can assure you they’re not fake at all.
    My sim of Hypnos was crashing 2-3 time *a day* before I enrolled it in the Havok 4 Early Adopter program. From the moment Havok 4 was put on it, until now (a 6 month period), we have had a total of 2 crashes. And both of those were attributed to bugs we found and reported to the Havok 4 team.
    The sim is a high traffic sim, with a constant population of around 20 avatars there. I suspected once it was rolled out grid wide, similarly huge increases in stability would be reported, I’m happy to see it was so. πŸ™‚

  80. Hodgey Hogfather says:

    Nice job, Sidewinder ! It’s great to fly over a sim boundary without going nuts!

  81. Doug Kavanagh says:

    Well after being overjoyed at how the Windlight Release candidates have run over the last months I’m quite disappointed in the crash rate of 1.20. I’ve logged countless hours in game in pre-1.20 and MAYBE crashed 3 times I can remember. Since I’ve upgraded yesterday to 1.20…I’m crashing about 3-4 times per hour and its starting to get a bit frustrating to say the least. I’ve done the usual clear cache, reduce the viewer to minimums, etc (shame cause I’ve been running at highest quality these last few months) to no avail.

    Also, it seems the preferences for whether I want to run in a window or full screen are not being saved after modifying that option – both pre logon and post logon.

  82. Attica Bekkers says:

    Thankyou! I have noticed less sim crashes. It really does help:) Way to go.
    And Im doing well with adjusting to the changed camera thing:) *does a backflip to prove it * thankyou for your speedy help.

  83. MimIi says:

    Naoki, you can still download the old viewer and go back to that one till its fixed

    A big thanks for Sidewinder and the team you did an amazing job!

  84. Bud Noel says:

    Sidewinder, I really appreciate the communication you’ve brought to the blog and the dilligent effort you and your team are making to improve SL. I have a studio full of Mac’s. I am uncertain which part of all the improvements has specifically ruined SL for me. I can’t tell if the server is better or worse because the forced viewer upgrade has been hell across our studio. I’ve had support tickets in since March 13 about the viewer and only had one response about it saying it should have been fixed by all the new software, I’d agree, it should have been but it’s not. If I tp out of a region, I’m logged out. If I walk across a region border, I’m logged out. My “About Land” is slow as mud. 25% of the time a notecard does not save and says there’s a problem with the server, try again later. I can’t add people to my group most of the time and this has made managing my properties nearly impossible. Many of my other developer friends are as disappointed as I am with performance. After we’ve worked for a year to build our businesses, we’re watching them be torn apart by performance issues, customers running away pissed off because vendors are so slow or don’t work at all. I only wish that another Linden would step forward and let us know wtf is going on and when there might be some stability.

  85. HonkyTonkAngel Weatherwax says:

    I can make graphs on my paint program to to say whatever I feel like. I am one of those that you are crashing more and more every day. Please stop adding more stuff and fix what you have.

  86. Sidewinder Linden says:

    @84 Bud: Please IM me inworld so I can understand these issues better… Many or most of them seem to not be simulator nor Mac-specific, but I am curious to get a better handle on the specifics of your situation. I am not promising that I can fix or even improve it, but have a couple of theories about one of the behaviors that I’d like to check out… /Sidewinder

  87. Sidewinder Linden says:

    @85 HonkyTonk: These graphs were not made up. If you have crashing in situations where not everyone on the region is crashing at the same time, then you are experiencing a viewer crash – the type where the software on your computer crashes. The crashes that we have reduced with this project are on the server-side, and affect all people with avatars on a region at the same time. /Sidewinder

  88. Shortdog says:

    Funny how people read things differently isn’t it?

    Sidewinder, congratulations to you and your team for a great job.

    Hell my release candidate viewer, is as crash prone as any other “Winslows” application. I’ve learned to live with a relog from time to time, sorry people, read the blogs properly and get off sidewinders back.

    There is a viewer development team as well you know.
    And a whole mess of folks who are working on the SL Open source grids as well.
    Thank you Lindens, for opening your server and viewer source to us who like to tinker.
    I for one love this system and will defend it and the teams of very hard working people who have put it together for our enjoyment.

    So it is a little buggy, what isn’t? MS Windows? Hah!
    That is what research and development is all about.
    Learn to live with it, and love it.

    Again, Sidewinder, thanks for a damn tough job, well done.

  89. Pointside Sunbelter says:

    @55 Lindal Kidd……. LOL (.45)

    @58 webmasteramadeus
    *********This is about the SIMULATOR not your client crashes.

    65 bret barbasz
    *********This is about the SIMULATOR not your client crashes.

    85 HonkyTonkAngel Weatherwax
    *********This is about the SIMULATOR not your client crashes.

    What is the deal with ppl not differentiating Sim/Server from client

  90. Vittorio Beerbaum says:

    88:”Thank you Lindens, for opening your server and viewer source to us who like to tinker.”

    ….when did they opened the server? πŸ˜€

  91. Cold Spitteler says:

    excellent work.. havok4 is great! if only the windlight team and the user interface team could provide a more stable viewer to enjoy the new havok

  92. @MimIi: I said, “I really don’t want to have to roll back to the old client just to get my work done.” Still, fewer sim crashes means going back to the old client isn’t such a bad idea after all. I guess I will.

  93. cosa nostra says:

    Sidewinder, my sim is running perfect since I bought it last year in august, with or without HAVOC4. Guess that the biggest problem for simcrashes are caused due to complex scripts, megaprims, etc … on those sims !

    despite all the good work from your dept. which we should encourage ofcourse, it is a fact that SL is experiencing more and more problems, I follow this up on a daily base, I meet and speak to loads of people around the grid, trendwise I dont need to make any graph to the crowd here, it is a fact, the sl fun experience is going backwards !

    SECOND LIFE is promoted as the world that ‘we users’ can create ! right ? So that means you launch a product that is working at all time and the users create there worlds at all times !

    facts :
    It is impossible in rl in any business to launch a product that is still in testphasis and being tuned or upgraded every 2-3 hours (and I know what I talk about) ….
    *car manufacturers will not launch a new car when the engine is still under test developement ! Would they recall the cars worldwide back into the service dept every week ? I guess not !
    *a barebone system running in a company where 10.000 users are connected to it worldwide (24/7). Can they permit to see there barebone being upgraded every 2-3 hours causing the whole company being down for hours or constant interruptions ? NO …. forget it !

    sidewinder, as I already mentioned in other blogs, there is no direction anymore … people just need a SIMPLE STABLE VIEWER that let them CREATE there things in SECOND LIFE or INTERACT with other people and a viewer who is accessible for most people allround the world, cause with 50.000-60.000 avatars connected average in which 60 % are camping for free money and the other 35 % loosing business (people who invested and who were working very hard in-world to compensate parts of their tiers !) …. I am not much impressed … most sims are empty, abandonned or are looking like a dump !

    LINDEN is now focussing on improving the VISUAL QUALITY, I will tell ya straight away that ya will never have the money and the payback to reach a profitable goal for this, cause guess LINDEN LABS is not a charity organisation ? People need to be paid, operational costs, etc …. Do you really think that second life users are going to upgrade there pc’s moving forward to next example like my personal pc (INTEL 6600/4GB MEM/1TB HDD/8800 GTS 512MB VGA …. ) ? If ya want VISUAL QUALITY then people need to stop using SL and move to the REAL GAMING WORLD where the gaming industry sells like MILLION cd’s in different platforms XBOX, PS2/3, PC @ 60 US$ per game ! Compairing this with the approach from linden makes me think about the consequences of your actual strategy !

    anyhow, I see/feel more and more that the fun is more than over, seeing that I am part of A WORLD where I pay 295 US$ per month and paid 1695 US$ for a sim, knowing that the product is still not finished and being tuned every hour of the day ! like ; Will my car start today or not , same comparaison?

    again thx for your hard work sidewinder and your team, but youre only a small fraction of the whole issue, a team is a whole company with a clear vison and strategy, it also doesnt help then to give the customers the impression or false hope in BLOGS like here that it is going in the right direction, please dont do that !

    I hope that the new CEO will have a strategy and a strategy he can execute without being influented or interfered by the board every second.

  94. Sidewinder Linden says:

    @93 Cosa:

    I am not trying to “give the customers the impression or false hope in BLOGS like here that it is going in the right direction”.

    I am reporting on the results of the Havok4 team, in objective factual terms to demonstrate that this project has delivered tangible results that improve stability of the Second Life platform – our stated goal when we announced the beta process for this project. If demonstrating that we delivered on the stated goal of this project is problematic, then I surely don’t know what I should have said instead once the statistics were in hand!

    Regards,

    Sidewinder

  95. Morgana Hilra says:

    Apparently no one has been paying much attention then… last night ALONE, I surpassed this “estimate” with the new viewer.
    The “normal” viewer works now.. with its “normal” crashing.. but, the newest viewer.. OMG… please do not make it apart of the main..
    It’s insane.. we had three other ppl online, we thought it could be the sim, so we moved.. NOPE.
    And its still not working right today.
    For a program that is CLAIMING lower crashes… I have to say.. the data is WAY wrong..
    Where are you testing?
    And please, do not try the “its your machine” crap… because its not.
    I surpass the req machine.. so.. that is NOT the issue.

  96. Sidewinder Linden says:

    @95 Morgana: This is server side crashes, not viewer crashes. Please see initial post and comments above. /Sidewinder

  97. Predikador Preez says:

    Well, Congrats, but , please .. fix the problem about the physical objects with the Havok, the elvators, and other stuffs don’t works correctly …

    Thanks, Pred

  98. Willy Rimbaud says:

    Hats off to the team.. Every step you make to improve the SL experience and stability is appreciated!!

    Pat yourselves on the back and look for the next dragon to slay.

  99. Mitzy Shino says:

    Perhaps a lot of the people complaining about viewer crashes should take their head out of their … and actually check out their PC!

    Yes I do crash sometimes, but its not unusual for me to be in world for 18+ hours at a time with no crash. My PC isn’t great (2.4Ghz, 1.25 Gb Ram, old video card and I run two monitors)

    You know maybe your setup is borked, bad/old drivers, windows crapped out etc.

    Not everything is LLabs fault you know!

    p.s. Congrats Sidewinder & Team, and I have a bug for you… I can’t drive my car on the road outside my house….errr hold that thought couldn’t do it in H1 either….there is no paving! (can we put you in charge of the road building crew?)

  100. Pointside Sunbelter says:

    @95 Morgana Hilra

    They chart is about Server Software Crash Rate!!! The Sim’s
    Not your client crashing.

    ***We had a big meeting today in Pooley about the client crashing, They know its crashing thats why they held the town meeting.

    Server crash rates, everybody say that with me…..
    Hello, can anybody hear me, is this thing on…

  101. ChatNoir Moonsoo says:

    @ attempts to spin this positively notwithstanding: the development of the client took a seriously wrong turn about two versions ago. Featurities. The current viewers are so overloaded with instable features that even running them on minimal settings takes a heavier toll on your system (because of all the internal filtering necessary now, persistent memory leaks, and a bigger “footprint” of the UI itself) than running previous viewers on maximum settings does.

    As nice as it is for us sim-owners to see server-stability improve (and thus reduction in content-loss), it doesn’t mean much for the average user or potential newcomer.

    I’m glad this issue is addressed, but the way things are looking the degradation of user-experience due to an incompetent client-design and its featurities will more than make up for the gain in reliability of the servers.

  102. Hu says:

    Poor Sidewinder, having to answer the same questions over and over from people who can’t tell their butt from their elbow.

  103. J B says:

    Excellent news. The results are very visible in-world and a great improvement. Thanks to the Havok 4 team for all their hard work and thanks for the good updates, I wish all teams at LL could keep as interactive with the populous.

    I’m really looking forward to seeing the Mono team’s product going live as well and would like to hear more from them as that nears production status.

  104. Sofia Westwick says:

    If half of you all have even been part of the Havok4 testing and have attend the meetings you would have an idea of whats going on.

    Also viewer crashes have nothing to do with Islands/sims and is not part of Havok4. The crashes reffered to her are sim/island crashes. Not viewer crashes.

    You need to complain to the Viewer team about viewier crashes.

  105. Isablan Neva says:

    Sidewinder – you and your team rock. Hopefully every project team at LL is paying attention to how you operate.

  106. Vixynne Ariantho says:

    Great, lower sim crashes is always good news…can anyone explain to me why my screen keeps flashing intermittently, or why the v.1.20 seems to have removed map rights? Nitpicky perhaps, but little oddities that were definitely NOT issues before…seems we’re playing whack-a-mole with bugs and issues…smack one down and three new ones pop up!

  107. P Belt says:

    @101 ChatNoir Moonsoo

    LOL so now you are saying that fixing the sim crash rates somehow in some wierd way will give the new user a bad experience or somehow because the client is having problems that they shouldn’t make the sim better.

    Wow that’s a piece of logic there.

  108. Sofia Westwick says:

    Try researching stuff before posting, and not even having a clue what you all are posting about.

    Actually try attending one of the havok4 meetings. Get a clue before posting so you all don’t look like fools

  109. Sindy Tsure says:

    Holy crap, Sidewinder!!

    I hope you and your team are getting tasty bonuses for this.

    Congrats!

  110. The data is correct. Prior to Havok4, my sim was crashing a few times per week. Since joining the beta, and particularly since it went live, I’ve had exactly 1 sim crash and I caused it. The only other restarts were those triggered by me on purpose, or rolling restarts by Linden Lab.

    @Wakeup, you are uncredible because you are a coward who hides behind a fake name. Stop trolling and go back to your mother.

  111. I’d rather have the crashes back thank you. You’d get booted, wait 2 minutes and log back on and carry on. Now we have weeks where you can’t do anything. Something is always down. I wouldn’t call that progress. Stop clutching at straws and coming up with BS excuses and FIX the ruddy core stuff we all want and stop wasting time on these shinies that basically mess everything up.

  112. Craig Solzhenitsyn says:

    I wonder how many people will complain about client crashes when theres a few different clients you can use… (its open source ya know, dont like SL’s? try OnRez, or if ya need just to chat one of the basics like SLeek)

    Stated fact – SERVER crashes are down – woot, CLIENT crashes are unknown – no data to compare with (and before ya say its OMGWTFWAAARGG WAAAAY WORSE! compared to what? itself last week/month/year? without sufficient data you can say that the apple in your hand is pink – no one else cant see it to compare, and Linden labs cant see your client crash data to compare)

    Im still going to ask for a systems poll like valves, i bet theres a fair few peeps on 4 year old lappys complaining about framerate, you cant expect to do 100mph in a relient robin (google it).

    Congrats the HVK4 team, now heres a Q for ya, memory leakage team? whar they be? after a couple of years its about time they gave a update.

  113. Sofia Westwick says:

    @111 that is viewer crashes not sim crashes get a clue! THAT IS NOT A SIM CRASH THAT IS A VIEWER CRASH. A SIM CRASH IS WHEN THE WHOLE SIM GOES OFFLINE

    Try getting a clue before posing and looking foolish

  114. @108 Well smartypants, perhaps you’d like to explain why it is that ever since the server code was implemented to allow Havok and Windlight, we’ve had nothing but problems. Or is it all on the asset servers? Either way, they should not have been implemented until the system can cope, and certainly not adding stuff that requires more db capacity on an already ailing system. It’s called cause and effect.

  115. Sofia Westwick says:

    The grid is made up of more then just sim servers and your viewer has nothing to do with the sim server.

    Also there are data servers and alot of other stuff you couldn’t understand.

    So stop saying your viewer crashes are sim crashes when its infact not and have nothing to do with it and all of the recent problems are not form Havok4 its alot of reasons so try getting a clue!

  116. cosa nostra says:

    @93 sidewinder, most people in here are not IT specialists, academics, etc…. people need a simple stable environment, is it now offered free or not, thats what it is all about !

    I wouldnt in my rl business environment never open a blog like this, because i would shoot in my own foot ! customers are not interested in what you do as a company to improve this or that, trust me ……I launched an upgrade of a new part on my customer base, U know how they reacted, sales dropped, cause they were scared this upgrade would cause problems moving forward for there end products ! customers want a product that works, how less complex or high complex that product is ! and if ya want to introduce something -> be very prudent and very convincing within ya customer base, explain it again and again with proven facts, yes proven facts !

    at linden labs you keep tuning and upgrading 24/7, but you are tuning and upgrading while the engine is running, and people are doing transactions or are building ???

    Please be reasonable …. maybe LINDEN should announce that a new ‘whatever’ will be made to improve customer experience and due to that the grid will be closed for 2 MONTHS or something, that approach I would even accept !

    thx

  117. Can we have some graphs on the increased number of borks and bugs? How is it Sidewinder, that with all that alleged testing, meetings, and feedback, you didn’t pick up things like TargetOmega, vehicles, or the flying/falling at the top of stairs and low objects, or did you just let all that irritating stuff slide right through deliberately? Plus now when in my pool, and I collide with the walls, my head and shoulders up to my elbows disappear into the wall?

    That’s about the only physics I use and notice, and it is ALL WORSE.

  118. Blinders Off says:

    Sidewinder, if you are speaking of the HAVOK grids (the post didn’t say for sure… but gave some indication that MIGHT be the case), then I won’t dispute the data for a few months yet because I haven’t had a chance to examine the Havok system. But if you’re not speaking of the Havok grid… I don’t buy the stats for one moment. Cos my experience and the experience of most people I’ve spoken with is that the last 3-6 months the grid performance has gone down the toilet. I’m hoping your stats are about Havok, cos we sure need a more stable system than we’ve seen in operation for the last several months.

  119. Sofia Westwick says:

    @117 Try attending a meeting and see whats going on. and then maybe you will have a clue about this stuff. if you actually owned an Island/sim you would see the imporvments.

    We talk about all of this stuff in the Havok4 meetings. if you actually went to them you would know and wouldn’t be so clueless.

    so stop complaining and go to one of the Havok4 meetings. they are 2 times a week tuesday at 11 am sl time and thursday at 5 pm sl time.

    So go and stop whining here.

  120. Blinders Off says:

    Well that came across sounding more negative than I meant Sidewinder. Lemme reword that:

    If the stats apply to Havok 4 regions… contratulations. ; )

  121. Shai Khalifa says:

    @84 Bud – I have exactly the same issues as you – been told it’s OpenGL related – can we talk inworld about these issues as I’ve had reports in on them since December 07

    I’d love to know what’s causing them – but 1.18.3.5 worked fine, then from 1.18.4 problems started – 1.19 got worse

  122. Sidewinder Linden says:

    @118 Blinders:
    From the original post above: “Now that we have a week of active experience with the new simulator throughout Second Life, I thought it would be good to share the crash rate details so that you can see the impact of the Havokβ„’4 project team’s work on increased simulator stability.”

    The Havok4 simulator was deployed across all of Second Life on April 1st, per previous posts about release timing. i don’t know what you mean by “the Havok grids”. The chart shows the crash rate on Second Life before that conversion, and after, with a span of 6 days on either side, to show the crash rate for the entire Second Life system with the earlier Havok1-based simulator as compared to the deployed Havok4-based simulator.

    Regards,

    Sidewinder

  123. Pointside Sunbelter says:

    @117 Montana Corleone Says:

    There are lots of entries in Jira at the moment on fixing the bugs dealing with Hav4 go have a look any member can log in. Add your own if you don’t see a particular bug.

    But… this post is simply saying that the crash rates are a lot lower not that its all of a sudden bug free.

  124. Shai Khalifa says:

    … further to my posting at 121 – I vote Sidewinder as overall SL project leader – he talks to us, and he tries to do things for the general population…

    And his and his team’s approach should be held up in front of all the other Lindens as a model for SL project work.

    Well done, and I’m sorry you get criticised for errors that occur as a result of other teams’ blunders.
    πŸ™‚

  125. richard says:

    i think we all should buy new computers would that help?

  126. Craig Solzhenitsyn says:

    oh @95 and other client issue peeps..

    using Valves survey (linden doesnt have one yet – please get one, your client base IS different, and im a geek for this sorta thing) there is (not including unspecified)…

    41 ‘identified’ diferent card sets in use –
    29 different ATI drivers &
    19 different Nvidea Drivers in use across
    6 different bus types with anything between
    32meg and 1gig of onboard gfx memory

    16 known types of sound system (onboard/expansion & USB)

    Anywhere between 1gig and over 1Terrabyte in TOTAL hdd space

    Anywhere between 64meg and 4gig on RAMM

    Network speeds from 32kb/s (yes theres people STILL on that speed using steam, i dunno how) to over 10,000kb/s

    Dont even start on memory bus type/speed – Processor arcetecure/speed/amount of physical processors – If they are using Hyperthreading/ CMOV/ RDTSC0/ CMOV/ FCMOV/ SSE/ SSE2/ 3DNOW/ NTFS/ FAT32/ Windows 95/ 98/ 2000/ XP/ Vista/ Linux in all its forms/ Mac etc.

    (Even thou the client base for SL is different to Valves, i wouldnt be surprised if there is MORE variables in the SL systems in use line up, this is after all an anything goes program, not a ‘1337 gamers’ client base.)

    The variables involve in what your PC is and what could be causing the crashs/conflicts are humungous, you could have the latest and greatest and be crashing because your soundblaster card and GFX card drivers are tripping over each other, the cost or how up-to-date your system is is only part of what it could be.

    BUT there is a way to help – that little box that comes up sometimes saying ‘detected a crash – pls send details’ – fill that in and then they can compare it to other crashs like it and those peoples systems to see if there is a common flaw to fix.

    In other news, i cant get Red Alert 2 to run on my XP Home 3.2 P4 2gig ramm system but can on my 98se 800Mhz 256 ramm system, keh?

  127. Elvis Orbit says:

    @ 125
    No no need to because the sim software run’s at the Linden Labs facilities on their computers not on your home computer.

    Graph reading skills:
    Look at the date on the graph: April 1st and before is Havok1 on the main grid.
    April 2nd and after is Havok4 on the Grid.

  128. John Loo says:

    This is really great news Sidewinder!!

    We should have a team like yours handling the DMCA’s! lol

  129. Harke Hartnell says:

    @89 because:

    1. they’re not reading

    2. they don’t understand how sl works

    3. oh they don’t read

  130. Sidewinder Linden says:

    @117 Montana: (Warning – really really long comment follows – I had thought about writing this up as a blog post or wiki page… maybe I still should…)

    “How is it Sidewinder, that with all that alleged testing, meetings, and feedback, you didn’t pick up things like TargetOmega, vehicles, or the flying/falling at the top of stairs and low objects, or did you just let all that irritating stuff slide right through deliberately?”

    This is a *VERY* important question. People build an amazing, actually astonishing, amount of content in Second Life, using a wide array of tools, and they get used in incredible ways. I thought I had a handle on “what’s in Second Life” at the start of this project, having been inworld for well over a year actively. I was sorely mistaken. There is so much here that I believe we have gone well past the capability of one person to know or understand all of it, and this is not some piece of marketing spin… Live a project like this one, and your view of Second Life’s variety and reach will change forever.

    Ok – enough philosophy! I’ll answer these particular questions specifically, and then comment on the general “how did we decide it was time to deploy this simulator to all of Second Life”.

    ============
    llTargetOmega “Things stop spinning once in a while problem”
    ============
    llTargetOmega – why did we not find this spontaneous stoppage mode?
    We have done a *lot* of work on llTargetOmega, and to be quite clear Kelly Linden has done a *lot* of work straightening out all sorts of things that didn’t work right with this feature, to the point that our QA (test) lead said at one point “this is the first time in the entire history of Second Life that all of Target Omega seems to work”.

    We tried lots of things, ran lots of llTargetOmega scripts, and for whatever reason (we do NOT know why yet!) we didn’t see this problem. Once to my knowledge during the beta we got a report of this happening, and the user said that after a region restart it worked fine. I was a bit concerned, but figured that with 545 regions up at that point, if it was more than some fluke we would have heard about more of them.

    This turned out to be a mistake – my bad. In the software business, it’s always the one you didn’t expect that bites you.

    Why isn’t it fixed yet? There are two core reasons:
    1) llTargetOmega is internally a very complex feature that has parts of it’s functionality in many parts of the system.
    2) We have not (YET) found a way to reproduce the problem so that we can figure out what is causing it. We have plenty of reports that “after an hour” “after a day” “once in a while” things stop rotating.

    We have tried to replicate the conditions in which the stoppage happens. I have copies of several items that residents have found stopped, and rezzed them on our simulators, and they’ve been merrily circling around for weeks with no failures!

    Once we either figure out where this issue is coming from by “poking in the dark” or once we find (or someone hands us) a way to spontaneously make the problem occur, we will have a solution.

    I will be very clear. This problem did not appear other than in a random chance occurrence that seemed to be random chance during beta, and we are not ignoring it now. It is however a very hard problem to solve due to the issues noted above. We have invested, and will continue to invest, substantial resources in figuring out a solution.

    ============
    Vehicles
    ============
    I’m not sure what you mean by “vehicles”, but if you’re trying to imply that we should have known and be able to test every vehicle ever built in Second Life.. I think that is not an expectation that anyone could meet.

    We did as much as we could to reach out to vehicle makers, up to and including holding a meeting with 16 builders, many of whom I *personally* solicited to get involved – people who had ignored the beta process up to that point. I went out of my way to find out about vehicle (and sports) in Second Life to try to get to as many people as possible who might be affected.

    We posted 27 blogs about the process, hoping to (and asking specifically) for people to get involved.

    The goal here was to try to get as much content tested as possible.

    Some changes were purposeful, and have been discussed before, and published in blog posts and office hours transcripts before the rollout.

    I believe that these are hardly the mark of a team that is not paying attention to vehicles, nor a team that is “not trying to get it right”.

    There are many people and products that have shown up *after* the rollout saying “this is broken”… Those we can only resolve once we know about them.

    We have found, through this process, that there were some fundamental issues with the implementation of vehicles in the original simulator. We fixed some of those bugs, only to discover that products relied on those bugs in surprising ways – so in some cases we “undid the fix”. In other cases, working with the beta testers (including many vehicle builders) we consciously chose to keep the “new and more right” behavior, so that better vehicles could be built in the future.

    =========
    Flying/Falling at the top of stairs
    =========
    I do not know what this problem is about… Perhaps it was that we had set a lower stair “trip threshold” with this build than with the previous version, as a slightly more realistic model (that stairs above knee height would not be climbable). From feedback in the last week, we found that there were places in the rest of Second Life where this was problematic, so we changed that in the build that I hope will be deployed later this week.

    =========
    The “GO” Decision, and how it was made
    =========

    It is, frankly, impossible for our team to test any feature against all of the things people have made in Second Life. There is too much of it. It is too varied. Some of it uses “surprising” approaches to get around what we think are designed in limitations.

    I knew, on the way into this project, that there would be no way to authoritatively know “everything will work” when we decide to ship.

    Given that, the decision was very hard to pick a “GO” point. This is the first project I have ever shepherded, in a 23 year career, where I had to think to myself “If we ship now, will we break the world?”, knowing full well that it had to be a guess – because we could not know before depolying the simulator.

    Here is how I made the call, and I am the one who made the final call on this. We did talk about it as a team, and there was team consensus, but at various points I held back, saying “we’re not close enough yet” or “there are these critical issues that really have to be fixed before we can launch”. Ultimately we all reached the conclusion pretty much at the same time that we would not know what else was broken until we shipped.

    A key thing to note about this is that because we KNEW we could not know about, and therefore could not test all of Second Life before shipping, we would DEFINITELY without a shadow of a doubt find new bugs and problems once we did deploy.

    At some levels, this then says that no matter what we do, we will find significant bugs once we roll out to all of Second Life.

    There is another KEY element, and it is in the chart above. Until we released the Havok4 simulator, we also knew that EVERY DAY we were seeing residents lose content due to simulator rollbacks. Holding back was thus not painless. Waiting causes harm just like shipping would in this case.

    We decided that to ship, we would have to meet some key criteria:
    1) We would not knowingly break any content in a way that is not recoverable. In other words, if there were any “item rezzes in a way that if it’s taken into inventory it’s now broken forever” bugs, that would hold the release

    2) If it took script recompiles for things to work, we would hold the release

    3) If the vehicles we know about perform “reasonably well” but could be tuned by either the builders or us, we would ship

    4) We knew absolutely that with an upgrade of this scope, complete 100% compatibility would not only be impossible, but would be a mistake – since it would cripple the future by forcing the new physics engine to be “broken” in ways to match bad behavior in the older one. This leads to the uncomfortable conclusion that, like with operating system upgrades, we will try to maintain as much compatibility as possible, but some older things will not end up being compatible without changes.

    5) If the beta test group, a dedicated group of dozens of residents who worked throughout the process said “it’s good enough” we would ship

    A core assumption of this decision was that we should ship as soon as we believed we would not permanently break content by shipping, to get the reduced crash rates and reduced content loss caused by rollbacks.

    We were told by the beta test group in no uncertain terms (and many early adopters) over the last couple of weeks before release that it’s plenty good enough, and in many ways better than the old simulator – it’s time to ship.

    So I held my breath, and hoped we would not break the world, and we decided to deploy.

    The overall feedback has been good. There are products that will either need to have scripting adjustments, or for which we need to compatibility bug fixes.

    The general response I have gotten is “wise choice, yes there are still things to clean up, but it’s better now than it was”. The open problems are things, as far as I can tell, that can be resolved either by bug fixes or product scripting changes, and I am not seeing bugs related to core product breakage in the rez sense, so I think we did hit a reasonable ship point – especially in light of the reduced crash rates shown above – and the related content loss reduction that goes with higher simulator stability.

    I hope that this provides some more insight into both the thinking behind these decisions, and the frankly terrifying prospect of having to make the “GO” choice in an environment where it is impossible to conclusively know that you have tested enough of the right things to “make everything work” and to know conclusively that no one knows even what “everything” is. We shipped when we thought we were “good enough” and could fix whatever other problems appeared in a non-destructive way.

    I believe that we are in position to be able to fix whatever problems are left in a non-destructive way, we are continuing to provide fixes, and hope to have a new build with several important ones in it deployed this week, and that we did deploy at the right time to drop the simulator crash rate as soon as possible to remove a source of content loss that we were in the position to remove.

    I’ll be happy to discuss whether this was the right choice from any angle with anyone who would like to review the big picture and how the specific choices fit into that decision process… It was not an easy choice, and I hope that this decision was “as close to right as possible”, as I am sure there was not “correct” decision – only some “more right” or “less right” possibilities.

    Best regards,

    Sidewinder

  131. Mimi says:

    Thanks for the long explanation and feedback Sidewinder!

  132. Jenta says:

    Sounds like you made a good and well considered desicion. It helps us a lot to understand what is going on – you did a great job.
    The effort you put into this project, and the way you actively communicated with the residents is very much appreciated.

  133. Keiko Rau says:

    Sidewinder and team…,

    Congratulations on an amazing success. and thank you for the detailed responses, especially @130. Its been said many times in the blogs that other Lindens should follow your example.

    The statistics (I believe them) are amazing. 97% improvement? Outstanding!, though being in IT myself, I understand that a large part of this would have been within H4 itself, but you and your team have done a fantastic job integrating it into SL, and have improved many aspects of the experience in the process.

    Sure, there are still bugs to iron out, but over time, they will be addressed. Its one thing to bring them to your attention, but some of the posters here need to realise that you are one of a small handful of Lindens that actually still care and are doing your best.

    Many of my commnets in the blogs have been to take LL to task when they have made a bad call or done something stupid, but praise where praise is due, and this is such an amazing result. You and the team should be proud of this achievement.

    I hereby nominate Sidewinder for Linden of the year!!

    Thanks for all the effort. Now get some sleep dude, lol.

  134. Verena Snoke says:

    Great work Sidewinder and team.. Sidewinder for Linden of the year, he gets my vote ^^

  135. Donal Randt says:

    I would be happy to see some of the vehicle issues resolved – namely SIM border crossings. But ifI understand correctly, the is nothing to do with H4, so no change expected, right?

    As for overall stability of both the client side and server side…. I saw this blog just now because I had to relog after a crash. (Border crossing in a vehicle got me). It’s almost midnight my time — I logged in this morning just after 9am. No relogs, no crashes. Is it stable? Um, yeah – I would say so! LOL

    It seems to me that the Viewer I cam to know and love as RC4, and H4 are playing quite well together!

    (Yes, I bailed from the RC program because that new one was simply too hideous LOL)

    Keep up the great work, and thanks for the awesome communication Sidewinder!

  136. Wheelerwod Oppewall says:

    Great. Now if I could only film with no lag or if the camera controls worked. And I must have crashed 10 times in the last three hours.

  137. Wheelerwod Oppewall says:

    …..or maybe the crash rate dropped because no one can log in.

  138. Cummere Mayo says:

    Look guys, Sidewinderisnt the person to talk to about VIEWER crashes.
    IF you want things done to fixthe viewr bug Pastrami Linden and Bridie Linden.
    THOSE are the ones that can fix the viewer.

    Sidewider: Well Done!

  139. River Ely says:

    to see more stable servers is a definite plus Sidewinder as they are a part of the fundamental foundation of our game experience. I fear a lot of high impact questions are not being addressed by LL, or so it seems from players experience. Some may appear niggly, some may be poor viewer set up, some may be folks expecting too much. That is why we have the Jyres(Jires?). Unfortunately Sidewinder, because LL offer few real interactive opportunities for players to complain where it is perceived to have impact, your, and every other linden thread will be used as a capping point to remonstrate.
    Good to see the server stability improving and the MTBI reducing. My issues now resolve, for example, arround getting only 7 FPS where before the latest viewer, in the same space, I got 30-40. All I did was upload the latest, so for me, my game play is reduced.
    Thanks for the update Sidewinder πŸ™‚ River.

  140. coventina dalgleish says:

    I have not had any crash problems with the New physics engine and it’s benefits far out weigh the problems that do crop up in scripts. Yes we build many scripted items but normally a quick analysis and the problem is corrected. I do realize there are more complicated problems with some vehicles but while they are not my cup of tea many people seem to like to use them and they are a part of the environment.

    Now if we could take the havok 4 team and assign them to solving the pile of old bugs I am sure we would see a vast improvement shortly.

    As far as the crash problem I still run an antiquated 3.6 p4 agp system and it seems to tolerate most changes well except for the decreasing frame rates as second life programs itself away from the common user computer. I might make a suggestion offer at least 2 viewers one with all the bells and whistles and one for the vast number of people who have not plunked down 2 to 4 grand for a current gamer machine.

    And one last thing that I just have to say I always try to keep the latest RC active as my viewer until the last release with DAZZLE, aptly named because you have no eyes left after an hour. I is very retro windows in the color scheme and quite disconcerting as the tool bars now are the focus rather than the game screen. On a 32 inch monitor it is a head ache training ground

  141. Jessicka Graves says:

    That’s great, you can keep the sim from crashing. Now keep the sim from dropping to 0.01% FPS and we’ll talk. And, get rid of Dazzle, it’s like staring into the high-beam headlights of a brand new car, blinding the heck out of me.

  142. Thank you for the long and detailed and thoughtful response Sidewinder. Maybe there’s one Linden after all that doesn’t deserve a pink slip πŸ˜‰

    OK, when I rez a spinning cube I used to have, then the rotation works, but is much jerkier than it used to be (although my frame rate is the same and I have gone back to 1.18.5.3 since one crash and one log off by the system in 10 minutes and slower rezzing with 1.19.1.4 was enough.

    From SLX forums, the consensus is that TargetOmega doesn’t work when the sim comes up or people first come in to the sim. Doesn’t always restart if the reset scripts, but DOES if they take it back and rerez it. That sounds like a start state of some kind not setting the script or function going, if that’s a help.

    As for the others, kindly explain how to tell if it is a server crash or a viewer crash when the message you get is the system is logging you off, or the Region is restarting, when in fact it isn’t, but it’s only you. That’s the same message you get when it *is* the region. But for me for months 90% of the time it appears to have just been me. But if you are TOLD the wrong thing, how is Joe Non-Geek supposed to know?

    Not of course it means it’s always the viewer, any mismatch in data could be responsible, and that could be the server or the client.

    For stairs etc, yes, you set it wrong. Not only the flying, but many steps that are below the knee now no longer you can walk up, meaning on many sims their steps no longer work. That was a bad thing to change now there are 17K sims out there. That’s as bad as not setting default lighting in Windlimp the same as the old viewer, which has made colour matching nigh on impossible and caused tremendous headaches for clothing designers and builders.

    And for those needing the number, look at Key Metrics. Apart from falling Active User and Premium numbers, and numbers by country, it gives total land, in square kilometres. Multiply by 1,000,000 and divide by 65,536 and you have the number of regions, and add on those sold in the Economic Stats page for this and maybe last month if the Key Metrics are 5 weeks late, and that is regions. Plus, presumably, private regions that aren’t publicly listed or accessible.

    And finally Sidewinder, thanks for acknowledging that SL is so big and complex now you can’t know it all. That’s why it is imperative on any future major decisions to thoroughly run it by a large number of residents first. We don’t want to be stuffed with something like cartoon colours in WL again, because only 100 people were surveyed, and the slots were taken up before some time zones came online.

  143. The problem is that the new viewer, which is not stable for many people, was deployed right after the server, and so we keep hearing about the viewer crashes because, to many people, they are one and the same. And they don’t like to read.

  144. Flaran Riggles says:

    I don’t know if this a viewer or server side problem, but every time I turn round quickly my av seems to move slightly. I’m sure I’ve heard this mentioned somewhere. It’s very irritating.

  145. Kyder Ling says:

    Great job Sidewinder. Really nice work. By chance did this upgrade also affect Simulator FPS?

  146. Whispering Hush says:

    sidewinder those stats rock πŸ™‚ now fix the memory leaks in the viewer plskthnxbai.

  147. I might add that the JIRA really puts a lot of people off, it is difficult to navigate, the search doesn’t work properly, even when things get lots of votes they don’t get assigned for months, non-lindens keep closing and moving issues that then need to get reopened, and you use that word we all hate: Resolved, when it’s no such thing. Now, perhaps if you broke out a couple of specialist blogs, saved this one for general, but had ones for the major groupings, including performance, which always have comments off (sometimes they say all is fixed pat on back but they aren’t and it would be nice to tell them), then people wouldn’t fill up each blog out of pure frustration. But LL has made it harder and harder to say anything, so what do you expect? A good list of what Lindens are responsible for which part would go down well too.

    Then maybe your individual blogs wouldn’t get filled with irrelevant stuff, you wouldn’t get cheesed off, and we wouldn’t get cheesed off. And everybody could get constructive stuff done, and all feel better. Now, we whine a lot, but that’s because we love SL, and want it better, otherwise we would have left long ago, like many have. But we do get upset when the issues that are important to US, are ignored for over a YEAR. See last year’s Open Letter for that.

    You said it yourself Sidewinder, you didn’t know the depth and breadth of stuff in SL. Well we do, we make it all, remember, try involving us a little more first, and giving us what we want and not what you think we want. Basically I think LL in general have no idea what they have created. It’s like they almost don’t “get” what it is they’ve created…

    Once again, kudos for your patient explanations. I know it must be disconcerting not to get all pats on the back with all your hard work, but hey, I have 18 months invested in this many much longer, and we do just see performance getting worse and worse, things getting broken and not fixed. Number one must be the asset servers, so LL should take a break on the next concurrency leap until the upgrade after next. That way, you might have some spare capacity which WL and Havok new features will take up anyway, not max it out all the time, breaking it worse every time it’s almost working properly.

    You can see the dropping active user numbers, and the fact they have been static for 10 months. This must now be the last chance to get it all right, or suffer when the first serious competitor is here. And a couple are entering public beta now. Don’t drop the ball at the last hurdle.

  148. None says:

    The glow feature will be abused and misused! It’s already happening. It’s like walking in to a freaking lamp store and all you see is bubble headed blondes with enough glowing prims on them that you can’t see their heads or anything else. Looks like that thing out of the movie Cocoon. LMAO!

    And the simulators may crash less, but damned if this isn’t the first time this browser has. It’s the first time a SL browser crashes again and again.

    Blah… And this month has been the least profitable yet.

  149. Shadow D. Wolf, Esq. says:

    @142: “But if you are TOLD the wrong thing, how is Joe Non-Geek supposed to know?”

    I’ll assume for the moment that you’re running Windows, so bear with me. When the viewer crashes, it will crash like any other windows program, with a message box that says “This program has encountered an error and needs to close,” or words to that effect. At which point, the crash logger and/or windows error reporting will kick in and report the error. If it’s the result of a faulty graphics driver, you might even see a BSOD, or your computer will appear to spontaneously restart (the BSOD is shown for such a short period of time you don’t even get to see it).

    In the event of a simulator crash, you’ll get a message saying you are being logged-off because the simulator is going down, and you are given the opportunity to review your chat history before closing SL.

    If I understand you correctly, you’re saying that you’re getting the sim-crash behavior, but without the sim crashing. In that case, it’s possible that your connection to SL is being interrupted. Wireless connections are notorious for this, or it could be that your internet connection is having technical issues, or perhaps your router or dsl/cable modem is faulty.

  150. cosa nostra says:

    Close the grid for 1-2 months, sort a stable second life environment out and then come back !

    regards,
    Cosa

Comments are closed.