[ALL CLEAR] Rollback on Multiple Regions at 7pm PST Tonight

[ALL CLEAR 20 Feb 08 – 9:23 PM PST  –teeple]

A handful of regions (fewer than 20) did manage to trigger a poorly-timed rollback this evening.  However, we see no out-of-synch mainland parcel transfers, and Concierge has already dealt with any island parcel troubles.

Thanks for your patience this evening.

[UPDATE 20 Feb 08 – 8:40 p.m. PST –teeple]

It’s looking like we got through this without a mass rollback.

This is apparently the extent of the damage: For these regions, simstates between 5:30 a.m. Pacific and 8:30 p.m. are permanently damaged, and should never be reloaded via rollback.

We’re going to be watching for the next hour, through one complete grid-wide save cycle. (Please see my Comment #36 on why it takes an hour.)

If you did go to the trouble of picking up your assets in world, it’s probably a good idea to wait this next hour out before you go back home and rez your items again, so that we can get a complete set of fresh simstates as a basis.

[UPDATE 20 Feb 08 – 7:32 p.m. PST –teeple]

Operations has their collective finger on the rollback button, but they haven’t pushed it yet.

After investigation, we believe we may be able to avoid a rollback for most of the affected regions (those that haven’t already crashed or been restarted since this morning’s update).

More news shortly…

*****

Approximately 500 regions have been failing to save their state since roughly 3 a.m. PST this morning. This means any changes made to land and content have not been stored.

Below the cut is a list of the affected regions.

If you have built or placed objects into these regions today, please retrieve your work to inventory as soon as possible.

We’re about to begin shouts into these regions every few minutes, warning all residents and guests to desist from building, rezzing and further parcel transactions until we’re ready to trigger a 7PM PST rollback.

Abandoned
Acedia
Akh
Aldebaran
Alderney Island
Aleksandr
Alicante
Almaden
ALOHA FREE PARADISE
Alpha
Amaro
Amatsu Mitsukai
Ambiance Falls East
Anarchy
Antiqua Paradise
Appalachia
Arah
area 51
Argent
Argentium Hills
Argon
Arp
Art
Ascella
Aslan
Astolat
Astur Area
Atiu
Ave Maria
Azov
Azzurra Commerce
Bandicoot
Bangu
Barefoot Beach
Bartlett Narrows
Bas Rhin
Bava
BCI Navigation
Beckett dAlliez
Big Sky Bay
Billydale
Blackslough
Blue Drifting
Bluewater
Boca Grande
Body Cult
Bora Bora
Boulevard City
Bounty Bay
Bounty Bay III
BPF Bouwinvest 2
Bramant
Brimstone Island
Bristol Ferry
Buccaneers Passage
Buckler
Bulgogi
Bundaberg
Buttermere
CA Regions
Caddo
Caledon Greystoke
Calla
Canada
Canopus
Cape Point
Captains Isles 3
Captains Sea Area2
Carbonel
Caribbean Dreams
Caribbean Moon
Carolina
Cartesian Straits
Cartier
Castle Turing
Catalpa
Catax
Catur
Ceneselli
Centaur
Center of Attention
Chessport
Chiffre
Chrysolite Island
Circinus
ClevelandPlus
CMP 3
Cody
Coelacanth
Colcord
Collingwood
Comcast
Coral Beach
Cornerstone Cove
Corona del Mar
corsica_reg11
corsica_reg11
corsica_reg2
corsica_reg4
Coruscant
Croak
Crucial
Curie
Cut Water
Danpoon
Darrin
Dedato
Developer Dock
Devils Moon
Digital Rift
Digital Underground
Discovery
Dolce Vita
Dolphin Riff
Dominari
Dorado
DragonFly Odonata
Drakes Void
Dreamworld
Dryke Island
Dumfoot
Dunai
Durga
Earthsea
Eden Garden
Educators Coop 2
EliteWorldGold
Ellida
Elven Imp Ocean
Enchanted Isle
EnelPark2
Enewetak Atoll
Eostre
ERE Pantalica
Eris
Euiryeong
Europe One
EverLite
Ewalani
EXCEPTIONAL ELEGANCE
Exclusive Island
FairChang Calypso NE
FairChang Mar
FairChang Neptuno
FairChang Vista SE
Fantasies II
Fate Island
Feltia
Femina Isle
Feraco
Fields of Hesius
Filby
First Island NE
First Island North
FirstFlight
Fluid
Forever Never
Fornax
Forum Europe
Friend City
Friends
Friesland
FUJITSU
fukuoka
Furness
Fussingen
Gabetti
Gaia
Garnet
GavenBizDistrict
Glen Lyon dAlliez
Gloria
Goa Beach
Golfo de Nicoya
Gonzales
Good Life
Gor
Grasmere
Great Barrier Reef3
Grebe
Guam Walker 1
Gulf Stream
Hachiman
Haengbok
Haidai Gwaii
Hamlet
HANA JAPAN
Hanalei Bay
Haploa
HAPPY DISPATCH
Hardrada
Harryhausen
Havana
Haven
Hedge Fund
Hell
Help Island 5
Hematite Island
Henderson
Hephaestus
Heracles
Hercules
Hesperus
Heylo
hibiya
Hidden Marvel
Hill of Hope
Hillcrest
Hippotropolis
Home
Hotgates
Howelsen
Hyper
IBM 2
IBM 4
IBM PDC Vancouver
Image Reflections
Immersion Zone
Immidae
Imoa
Inishnee
Ink Cap
Ipala
Ishikawa
Isla Montevideo
Island Type 1
Isle of Faith
Isle of Love
Isle of Tranquility
japon
Jast
JB Islands1
Jenner
Jingbo
Jitsu
Jockeys Ridge
Jukai Urbaine
Kailiauk
Karpov
Kazzkark
Kelly Work Zone
Key Rua Hatu
Koeulla
Kolamaafushi
Kuai hele
Kubrick
Kula 1
Lacrymosa
LAFAYETTE BEAUTE
Lafee island
Lagoa
Lake Serenity SW
Land of Lincoln
Laurin
Laurius
Le Public Systeme
Lepudro
Linden Lab HQ 3
Lipari
LM Sun
Lobo Island
Lost Gryphon Island
Lost Spirits
Lu Dongbin
Ludwigsburg
Lust
Luxury Island
MADE IN ITALY
MagSL Nishiki
Majuro d’Alliez
Makedonija
Makira
Malaga
MalaiKa
manas
Manitoba
Marajo Island
Maxwell
McDonnagh Land 3
McGammon
Melungtse
Messenger
Milo
Minotussen
MIT Media Lab002
Mohah
Monkeh Barrel
Monte Napoleone
Mount Celestia
Muhlif
Mun Saari
Muralis
Muta
Mystery
Nabla
Nadia
NanoGunk NW
Nautilus_10v4
Nautilus_6v4
Neifaru
Nerice
Nerriga
Neverov
New Zealand
NEWare
Nina
Nomanisan South
North Ar Plains
Nossa Senhora Gracas
Novae Artes Domum
Nubia
Nymphs Beach
Nyree II
Oberon
Ocean View dAlliez
Oeland
Old Wives
Omey
Ona
One Manchester
OneClimate
Ontario
Ophiuchus
Orientation Island 15
Orientation Island 39
Orientation Island 61
Ortega Channel
Oshima
Overdrive Island
OWK
Oz Island
PAConsulting 4
Palau
Panther Crossing
Panther Ridge
Panthera
paradise dream
Paragon Land
Passionflower Island
Paw Paw
Pennycat
Perdita
Phoenix Productive
Pietra
Pimushe
Pini
Pleased
Pleasure Bay
Pleon Islands
Plush Chi
Plush Epsilon
Plush Meter
Plush Nano
Plush River XI
Plush River XV
Ponderosa
Porto Seguro
Preis
Prize Banks
Prize Homes
Provincetown Marina
public townscape
Purple
Qlar Fell
Questionable Virtues
Quoddy Head
Racoon Cay
Ragnars Valley
Rakapila
Rannug
RastaOpen1
Reforma
ReiLank
Rekall
Rezzable 12
Rivers
RosaMaria
Rosario B2
Rosario B3
Rosario C3
Rose Garden
Rothenburg
Rubberpups Realm
Ryder Lake
Sammy
San Ramon Sports 3
San Sebastion Isles
Sanibel Island
Santos Anjos
Sapas
Saturnalia
Scaroth
Schertz
Scion City
Scorpius
Sea Monk
Sea of Tranquility
Second Ice
Sedge
sendai
Seoul 4
Serafina Island
Serene
Serenity Island
Seychelles Resort
SGLand2
Shaea ViBe
Shannan
shark beach
Shea Nectaris
shige24
shinano
Shinichi
SHUSHOKU
Silver Shores
Sinuiju
Sipiwesk
Skouros
Sky Island South
SLASH2
SLeek
Snowball
Sonata
Southern Ar
Spaniards Bay
Sparkle
SPIRIT ENTERTAINMENT
Spiritus Sancti
Splash
Spontaneous Heights
SpoonDrift
Star
Starfish
Starrymoor
Straylight
Strianic Ocean
Suhail
Summers Dae
Sumptuous
sunbelt software
Sunnyland Pearls
Sunnyland Sands
Sunset Arts II
Super
Surrendered Scorpion
Sylvia
Tabuks Ford
Talana
Tearaght Island
Telchines
Telstra Experience
Tenshi
Terra Alpha
Terra Neo
Thatcher
The Heart
The L Word East
The Sandbox
The Tempura Region
Theatron
Tikva
Tir na Nog
TNW East
Tompson
Tong
Town and Country
TOYOSU
Tranquillity
Triton
Tuscan Dream
Tweak Isle East
Uijong
Ultimo
Undiscovered
University of Illinois
Upper Aboyo Passage
Urcho Archo
Uru Tetefa
UT Pan American
UTD ArtTech Island
Utopians
uvvy
Vaitahu
Vargas
Varsity
Venetian Shores
Venezia
Venture
Verney
Vesperia
Vicentino
Vinyards of Genesian
Virgin Isle Marina
Visayas Isle
Vleet
Volcan Tenorio
Volksland Downtown
Vosk EasternPlains
Vosk WesternPlains
Wailea Beach
Warhol
Wasteland
Waved
Wellbeing Island
Wild Point
Wintertraum
Winyah
Wisteria Lane
Wolpertinger
Wonderful Land
Wyrd
Xay Tomsen Resort
xFire
Yellow
Yoretow
Yuva
Zebranky
Zeffenhal
Zeno
Zermatt
Zorn

This entry was posted in -Miscellaneous. Bookmark the permalink.

97 Responses to [ALL CLEAR] Rollback on Multiple Regions at 7pm PST Tonight

  1. Susan Homewood says:

    WOW, 500 regions.
    Hope all goes well, and nobody loose any stuff

  2. mac says:

    Great can`t wait 😛

  3. Jayden B says:

    Are there 521 regions by any chance? Quickly counts… Yep!

    /From another blog entry: Server version 1.19.1.80264 has been rolled to 521 regions./

  4. Sean Heying says:

    I would maybe suggest that as a large amount of the time between 3am and now was EU and Asia primetime that you delay a little more so those regions of the world can come online first.

    Imagine all the stuff that will be lost with a full day’s building.

    Good luck Lindens, and I hope you solve the bug before pushing it out onto more people.

  5. Aria Alexandria says:

    uh? what’s the point of this anyway? you made a ops on your new deployed code 😮 silly…

  6. Sean Heying says:

    ps: To everyone in those areas who has bought or sold land today don’t forget that land sales revert in roll backs too.

  7. Ree Indigo says:

    Nice how they treat a commercial environment, isn’t it?

  8. Epocx Timeless says:

    Are the Items that where placed well before save?

  9. Angelina Sinclair says:

    Wait land sales revert in rollbacks?? What about the lindens you wasted in buying that land, does it revert too?

  10. sirhc deSantis says:

    Well at least they are going to warn people. Must admit though its going to be tough on those in and out before the shout out. I was about to call it a night (whinesday hangover?) but make it a habit to check this blog these days. Glad I consolidated to just one reason – and dodged the bullet this time. Good luck LL

  11. Pingback: New LL server code triggers rollback at 7PM « virtualSecond.com

  12. Sandar Gausman says:

    SO, that explains mysterious appearing and disappearing items from inventory

    Caused a bit of confusion when some things were not delivered, this explains it.

    Perhaps you could test new code on a few regions first before you roll itr out in the whole world?

  13. Uccello Poultry says:

    Bless you, bless you, bless you for the in-world notification and for the list here. Huggs and Marshmallows for everyone!

  14. I’d almost say : good that it were “only” 521 regions and not the 1500 or so that were next on the list, but it makes me wonder about the LL QA process.

    If “Linden Lab HQ 3” is indeed a LL sim, they’ll be sharing the joy with the rest of the unfortunate residents. I feel bad for all of them…

  15. coventina dalgleish says:

    Well one might think that almost 18 hours without a word is a bit long to wait for notification of this type of a problem. Again another example of the ho hum attitude by the management. Does no one monitor this type of FAILURE or did you just happen to notice that a major part of your grid was non functional. I suspect we will be seeing this proliferate into other areas now since they have no idea of what caused the failure to save. The game has become top heavy and there is only one final outcome now it seems. Greater and greater cascade failures seem imminent in the future. Again wasting peoples time and efforts through an inability to properly manage the spaghetti
    code that has been building since version 6.0. As much as I enjoy this game I see no light at the end of the tunnel.

    Very disappointed in this apparent lack of concern by the management.

  16. Jayden B says:

    @16 Coventina “Does no one monitor this type of FAILURE or did you just happen to notice that a major part of your grid was non functional.”

    This was failing to save state, that means they would not have known it was faulty until they rebooted a server with the new code and found that newly rezzed items did not come back.

    The monitoring was not at fault, the QA on the server code was. Checking state should definately be in their test scripts, assuming they even have them. That is where they fall down, they don’t test properly before deploying still treating the world as a Beta playground.

  17. Teeple Linden says:

    @Sean: #4 is a valid concern…we had some internal push for more lead time between the announcement and the rollback…but #6 is one of the strong counter-arguments against postponing any longer than a couple of hours.

    Even with 15-minute warnings broadcast into the regions throughout the night, there would still be significant potential for unaware residents to TP in and purchase property, contributing to a growing number of land transactions which would need manual attention before this is all sorted.

    As for what’s already happened with land purchases, we’re pulling lists of parcel transactions for the day, with an eye toward making good on all of them.

  18. Marina Mincemeat says:

    Many TY’s for the warning LL 😉 Oh and @#13 Hello Uccello Poultry i miss my ex neighbor and her store!!! Oh no not off topic so sorry but is there more than 500 regions in all of SL?

  19. Angelina Sinclair says:

    ahh ok that’s good to hear, thanks Teeple. ^.^

  20. Cincia Singh says:

    I notice that I can’t seem to log in to the grid … and when I check “friends on line” I find none of them are inworld. Maybe a larger problem Teeple?

  21. handmadeandroid says:

    I’ve just spent 3 hrs uploading textures, working in photoshop and applying these, sims gone….oh well, I’m trying not to moan 😛

  22. Proch Mcmahon says:

    I guess with all this happening explains why im laggy, my music breaks up and i freeze up.

  23. Bambee Pelous says:

    What a suprise, more issues. Would be nice if you could actually get some of this working for a decent period of time. The QoS on this is frustratingly lacking and the entire system is in a constant state of chaos. now I get to go back and redo an entire days worth or work Yay for me.

    It’s a good thing my bank has better quality control than this. Sigh

  24. Caemlyn says:

    Could we possibly get a rollback to when group chat actually worked?
    Please stop working on the the new and better viewers and fix SL!
    I appreciate the work everyone in LL does and in 2 years this is my 2nd post. I am not a whiner but enough is enough!
    TY!

  25. neurosys sin says:

    o man… its only 6 and I cant login. Any idea of the timeframe for outage?

  26. Teeple Linden says:

    @Coventina & Jayden: Also a couple of valid concerns.

    We have a number of tests we run every time a region generates an hourly archive (called a “simstate”) of itself. If any of those tests fail we get an alert, and diagnosing the failure becomes a high priority task.

    But most of those alerts are based on pattern-matching, and as anyone who’s ever tried to develop a comprehensive regular expression or other pattern-matching algorithm can attest, it can be fiendishly difficult to code defensively for all possible failures.

    Today, the saved simstates were passing all of our acceptance tests. Externally, the simstates were the correct size, shape, and weight, if you will. Internally, the particular corruption which was introduced by the server code deployment failed to trigger any of our pattern-matching alerts.

    Needless to say, we’ll be adding to the list of tests we run on freshly-archived simstates.

    As to the issue Jayden raises…we don’t have anything to tell you yet. Yes, we’re a bit mystified tonight. The simulator code was doing the The Right Thing[tm] on our test grids. In fact, it’s *still* doing the right thing.
    First things first: We’ll get this cleaned up, and then after we’ve investigated, we’ll do some public talking about what went wrong and why.

  27. Veronica Quackenbush says:

    From the perspective of this relatively non technical user, it might be useful if someone could explain how tping in and collecting your objects saves them safely, whereas the objects remaining in world will be lost in the rollback? The reasoning behind my question is simply this: if you can save the objects by taking them back into inventory, could you not likewise save them by autoreturning them to the owner? This could help out a sh*tload of folks who are currently not online. It may be an inconvenience wading through your bloated Lost & Found, but it beats losing everything

  28. U M says:

    Was there a failure within the asset server? a failure hardware failure? Software Failure? Or just a meltdown? These are very dangerous if the cores don`t do their dumps.

  29. Bambee Pelous says:

    Maybe less talking and more fixing? I’ve only been on here since September but there’s been more downtime than any system, game, service or product I have ever used, ANYWHERE, EVER! I just think it’s completely ridiculous. and now I am losing valuable time I spent on something that I will have to recreate thanks to ongoing issues.

    As stated above by Caemlyn, maybe we could stop adding stuff to the system before we actually get the system stable? It’s like sticking a brand new 350 Chevy in a burnt out pinto, you just don’t do it.

    Additionally, there’s no way to archive your work. it would be really nice if I could make a back up of my work to my hard drive and then import it when stuff like this happens. then I would only have to spend a partial day recreating the work instead of the whole day.

  30. Darien Caldwell says:

    @27, I frankly wouldn’t want months of work returned to me, on the off chance that one item would be lost. They are handling in the best way I think, and yes, I own a region affected. Frankly I’m glad they are being more cautious, Had this been deployed grid wide, the potential damage could have been much greater. Unlike some, I’m pleased LL is being dilligent in their efforts to protect residents from unpredictable issues such as this, which are bound to crop up when making changes to such a complex system. As is often said in such cases, “nice catch.” 🙂

  31. KMeist Hax says:

    Veronica: Your inventory is saved on a different set of servers than the simulator contents. When the sim rolls back, your inventory will still be there.

    It would be nice to have a script or something autoreturn everything on the soon-to-be-rolled-back regions, though.

  32. Bato Brendel says:

    Can we get some clarification as to if its the new server code as the issue or not? or some other isssue as in like the ‘central’ system. Granted its interesting to see that the blame has been put on the code since they are doing a rollback but I dont see nothing in black and white saying that its the new server code as to being the cause either. Oh, I know its off topic but can someone please tell me how to get my icon to appear in the blogs Ive looked high and low and got thumbnail icons in the forum allready just send me a IM inworld if you know how. And you guys at LL dont sweat it, $%^& happens just keep up the good work cuz we appreciate it.

  33. Louisa Sartre says:

    pardon me for seeming to be stupid but how long does the roll back take, and will everything just be set back to 3 am?

  34. U M says:

    But if there is no core dump nothing is saved

  35. neurosys sin says:

    is the main grid down now? its 6:42 PST.
    I havent been able to login for an hour now. sez 44K online tho…

  36. Teeple Linden says:

    @32: Bato, the issue definitely started with the deployment of the new server code. But we’re still assessing; we can’t tell you yet what the root issue is because we don’t know.

    @33: Not stupid questions at all! We’re projecting about an hour start-to-finish for the rollback.

    Simstate save times are roughly evenly distributed around a one-hour cycle. So, at any given minute past the hour, roughly 1/60th of the regions on the grid archive themselves. So, we’ll be setting each region’s simstate back to the closest good state we can find around 3 a.m.

    @34: U M, there’s nothing to worry about saving or core dumping in a simulator process itself. Within the same version number, those processes are generic and blank; all the uniqueness is in their simstate files. Current terraforming values, parcel ownership, a snapshot of the state of every rezzed script, resident content…the simstate holds it all.

  37. Jack says:

    Hello everyone, I am new to this game and I am glad I found out why I couldn’t sign in lol, anyway, does anybody know what time SL will be back up at? Thanks alot and I am looking forward to playing this game 😀

  38. Cacho Rossini says:

    I can’t log in… But i have a few friends connected… what’s happening? the land where i start isn’t in the list…

  39. SL Player™ says:

    God help the people whom do not know about this. $$$$$ out the window.

  40. Apologies for the OT entry here but Joppa has left the comments closed so I am unable to post this where it belongs.

    Great and dandy that you’ve posted alternative ways to log on when your home region is down from the “network issues” but how about an ETA on a resolution for said issues? My region has been down for over an hour now.

    Is the problem being worked on, or is it going to wait till after the problem in this blog entry is taken care of?

  41. Cacho Rossini says:

    Second Life Service Status
    Second Life is Open

    The Second Life grid and the Second Life web site are open.

    But i still can’t log in!!!!!!!!!

  42. nina says:

    yep.

    ’06:59 PM, Wed 20 Feb
    Network issues effecting ~450 regions

    There are network issues that are affecting approximately 450 regions…’

    ’04:56 PM, Wed 20 Feb
    [ALERT] Rollback on Multiple Regions at 7pm PST Tonight

    Approximately 500 regions have been failing to save their state since roughly 3 a.m. PST this morning…’

    ’03:53 PM, Tue 19 Feb
    Second Life Server Software Updates (1.19.1)

    Over the next several days we’ll be updating the servers comprising Second Life to our “1.19.1″ version of the server code.
    These changes include a small number of bug fixes…’

  43. U M says:

    Thank you Teepee for that explaination.

  44. Chaz Longstaff says:

    Deep breath everyone, reach for a Plymouth Gin martini, and repeat to yourself, “bleeding edge, isn’t it exciting, darling…”

  45. Veronica Quackenbush says:

    Teeple, thank you so much for setting aside time to keep us posted during what I am sure must be a very trying time handling the rollback *cheers and hugs* However (and KMeist @31 seems to agree with me) it’s still not entirely clear to me why an emergency autoreturn just before the rollback strikes some people as a bad idea, so it would be nice if folks could comment on that (if not for the present rollback, at least for possible future reference). I understand Darien’s objection @30, but any duplicates saved to lost and found could just be dumped later, and the assets otherwise lost would at least be saved. If you don’t feel like wading through your enormous lost and found, clearing it wholesale is a few seconds work, but the people who *needed* to get their things back would not be left out in the cold

  46. Yuo Rang says:

    will the restarts tomorrow still go ahead now?

  47. Please make your mind up !!

    I have already warned our FairChang residents in the listed sims.

    Rollback or not?

  48. Teeple Linden says:

    @45: Veronica, both your and KMeist’s comments and Darien’s concerns make sense to me. Rather than speculate on why that wasn’t part of the game plan tonight, let me wait until this is over one way or the other, and put the post-mortem blogger on the hook for addressing your thoughts.

    @47: Garth, I appreciate that this is an utterly aggravating incident. But in light of the substantial concerns already expressed by residents here over the short notice we were able to give, we’d obviously like to avoid a mass rollback if there’s any possible way to do it. Please bear with us; we should know shortly.

  49. Bucky Barkley says:

    How goes the search for the first *real* CTO? Not so much a hardcore technical contributor, but someone who will assure that stability problems are addressed? Someone who can prioritize, lead internally, and proactively communicate with the residents without having to be goaded into it?

  50. TaraLI Jie says:

    @45 “If you don’t feel like wading through your enormous lost and found, clearing it wholesale is a few seconds work, but the people who *needed* to get their things back would not be left out in the cold”

    You’ve got some 7500 prims, in over 2000 objects. These are scattered over perhaps 800 meters of airspace, across 32K sq m of simland. Anything rezzed from the coalesced object that hits the parcel border may or may not held back – depending on the next parcel’s object entry permissions. Anything that ends up rezzing below ground level or above 762 m will be forced together and out of place. This may cause interpenetration of physical objects – causing the physics engine to make those objects depart their location at a very high speed.

    If you’re absolutely lucky – you rez the root object of the coalesced object at *exactly* the point it was before – assuming you know somehow *WHAT* that root object was, where it was, and it wasn’t 350m up in the air… and as one of the items was a 40m x 40m x 40m sphere, that somehow gets shoved out from your island into a non-existent region…

    I’m sure the Lindens are making sure good SIMstates are saved if at all possible, and will do their best to help you recover something from there if it’s truly crucial – but even so. Huge pain for everyone if they just return *EVERYTHING* on that island.

  51. Veronica Quackenbush says:

    Garth @47, it seems sensible enough to me to act as if the rollback will take place, simply because the consequences would be worse if you didn’t, which is generally a good rule of thumb. As my grandmother used to say, it’s no fun to have egg on your face, but it beats having blood on your hands

    Teeple @48 wow you really are on top of things, that was the quickest I’ve had a response from any Linden, ever (even counting the good old days of live help). When this is over, buzz me an IM sometime when you’re in world, I want to personally give you a BIG luvin hug 🙂

  52. Pingback: WTF were they thinking? (Caution: Rant) - SLUniverse Forums

  53. U M says:

    There indeed was a failure some where alone the way. But where and the what damage was caused might never know.

  54. Zena Juran says:

    HIyas Teeple,

    I don’t have a lot of resources, but I will gladly donate what I do have and volunteer to help out SL in it’s time of crisis. 🙂

  55. Phantom Ninetails says:

    About that network issues blog post right below this one that we cannot comment on. Does that include Crash Me and the nearby Havok4 test sims? I haven’t been able to get to them for several days now.

  56. OK…
    I want to say that this is an improvement.
    No, they still have rollout problems.
    BUT
    In 55 posts, Teeple Linden has responded 4 times …
    That is a 400% improvement over the las 150 entry blog which had no linden answer anything.
    I don’t know if you are just going “above and beyond” Teeple, but thank you for taking the time and effort to read and respond here in a reasonable amount of time….
    I am still not sure SL will survive but today, in this blog, I saw some good stuff happening.
    If the rest of the Lindens take note of Teeple Linden and start responding and actually communicating with us WHILE things are going bad, it may not make the damage any less, but it makes Me feel like it hurts less… Thank you Teeple Linden for taking the time to respond…. (ps I will be glad to bring My 35 years of software development experience to LL to help 🙂 )

  57. teddirez Escape says:

    totally agree with Harleywood, its good to get some feedback in the posts. Cheers Teeps.

  58. Jaca C says:

    So as paying residents we are suppose to check the blog to make sure we “save changes immediately”? Don’t you think you should take your database of paying residents and email them when such a problem happens? I, for one, just read this and sooooooooooo many hours behind – so if it is my land I am SOL

    Jaca

  59. Wolfgar says:

    I Love Ths Game Not Sure Whats Going On Because I dont Know A Whole Alot About Computers And There Guts
    But When I try to get on I Can’t But When I Do Get On It’S Crash Crash Crash But I know The LindinsAre Working Hard To Get Every Thing Back To Working Order It’s Going To Take Some Time I Know That Much

  60. Wolfgar says:

    Yeah Whats Rollback Mean I Dont Have Clue And Is There Any Set Time That Every Thing In 2Nd Life Will Be Back To Normal Thaks Wolfgar

  61. U M says:

    35 years that makes you 55 to 60? um………..ok sure :/

    Anyways Teepee for once instead of having comments close has them up. That alone is some thng.

  62. Pingback: The Grid Live » Second Life News for February 20, 2008

  63. Teeple Linden says:

    @56 & 57: Thanks for the positive feedback! And Harleywood, we’re hiring at http://www.lindenlab.com 🙂

    @58: Jaca C, good question. I’ll offer a hypothesis, which I may be back out here to eat as soon as I have opportunity to check on it: Timing issues aside, a properly-focused mass email wouldn’t have been practical for this particular incident. The parcel ownership maps pointing to potentially affected residents were locked away within the bad simstates, and since we couldn’t get to those properly, we couldn’t have quickly and reliably scoped a mass email.

    But let me escalate your input with an eye toward future contingency plans.

    @59/60: Wolfgar, a “rollback” basically takes an entire region back in time. It completely restores the region to an earlier state, but it also wipes away anything that happened between the target time and the current time.

    A rollback is an all-or-nothing deal for a region, so we only offer them by request to island owners and their estate managers. Mainland rollbacks are very rare, and we do them only as absolute last resort for system errors, never on request.

    We solved our problem without doing that tonight for almost all the regions on that list, and it’s safe again now to build, rez, and buy and sell parcels on them.

    On your other note: We really appreciate your vote of confidence 🙂

    As to your crashes, please log in to the Website, click the Support link at the top of the page, and then log in to support. You’ll find a new ticket there with a little information filled out. It already has internal links to your crash logs.

    Whenever a technician makes a comment or request for information from you, you’ll get an email with a link to that ticket. Please feel free to make any comments or provide any more information on the ticket while the support team works with you.

  64. Wyald Woolley says:

    Thanks guys, it sure looks like you did the best you could given the sneaky way this bug crept up and bit you. From what I understand, many of the things you are doing (and especially this upgrade) are designed to make SL more stable. I can only imagine how difficult it must be to accomplish given how far behind the power curve you started out.

    I suppose many things you’ve already accomplished are making SecondLife more stable on some digital level, however the in-world experience (which is all we have to measure your success by) seems to be less stable at the worst, or not improved at best.

    Here’s looking forward to the day, not too far off, when I log in and stability and lag will be a thing of the past.

  65. Gabbi Rotaru says:

    @45: An auto return (aka copy) of items to a special rollback folder would be a better idea than an auto return to lost and found.

    I am not in an effected region and if I was I would have been seriously P..sed about it. Last night I saw no mention of this then this morning I find this log. The time difference (i.e. my overnight) meant I would have potentially lost a lot of items and probably about 200K l$ worth of stuff I purchased int he last few hours before log off.

    Top of my wish list is an option to perform a local backup to my computer of my complete inventory and preferably my land config as well. How difficult would that be to implement? Is there an open wishlist request for such and item. I can never figure out the jara thing 🙂

  66. Wanja Lubitsch says:

    Since the simstate holds it all, it also holds the “build allowed” flag and the “parcel for sale”.
    So you could have set these regions to “no-build” and each parcel to “not for sale” and disallow every other activity that cause changes that would be overwritten by the rollback.
    Add a warning message everybody gets when arriving (by logon or tp) in that sim.
    Than you can give the time to everybody to collect their stuff before the rollback.

    You have serious quality problems.

    If you only be as fast with problem detection and communicating as you are with claiming them to be RESOLVED when you think they are. You need to improve the grid monitoring. I propose to have a more detailed automated grid-status page on your sites, with a possibility for residents to report misbehaviors of each sim (like “SIM down”, “can’t move in”, “can’t move out”, “don’t rez things”) as an additional input to your monitoring. Of course that would not have helped you with that issue you had here.

    Although the residents have asked you for a feature freeze long time ago, still your adding new features including new bugs that need to be fixed by people who should fix old bugs instead.

    Having three branches of viewers (Release Candidate, Windlight, Dazzle), with different new features is not manageable. Not even for a company with good QA and a fine tuned development, release and rollout management. Just stay away from it.

  67. Gino says:

    Since the Non-rollback a few hours ago, I have returned to my rental property on one of the Sims on the list. And now, everytime I cross the sim boudary into a neighboring sim, I get a popup box in the middle of my screen informing me that I have entered a region with a different version of the sim software (or something like that) Only option is to Close box. But the next time I cross that boundary in either direction it pops up again. Is this supposed to happen? For how long, and for what purpose, need I have to Know this juicy bit of info. How can I disable it? Thanks for anybody’s suggestions 🙂

  68. Gino says:

    I forgot to add, that the Esate owner re-booted the region to see if it would kill the pop-up notice box. But it didn’t help 😦

  69. Nik Woodget says:

    @67 as you step one way, your changing to one version, and when you step back your changing to another

    thus you get the warning each time.

    This is due to the het-grid capability, where the grid can support multiple versions of servers at once. Make a note of of the two version numbers, see if they are the old version. the version we’re talking about here, or havok versions. The versions for havok and the one we’re talking about is okay, if its the old version, then it would seem the sim with that version number wasn’t updated.

  70. Xenox Snook says:

    Dear Lindon, I am afraid I am going to have to agree to what Wanja Lubitsch says. My wife and I have everything we own invested in this, and we have great faith in this media. Our only real concern and fear of this not succeeding, and eventually becoming the replacement of the internet as we know it, is that you as a company focus more on gaining new customers, adding new features and opening up more and more new regions (and thereby servers to service!) instead of first of all fixing what you already have.

    The amount of instability of this system, caused primarily by the addition of new flashy features and expanding the already massive potential of this media, is simply unacceptable to us who depend on being able to work online on a daily basis.

    We very much hope you will come to realize the potential hazard you place on yourself, and thereby on all of us who have invested in SL, by running such a support policy.

    I know how difficult and time consuming it is to get well-qualified personnel enough while expanding as fast and immensely as are you, especially good technicians to service the servers and resolve support issues. We hope to see you reversing this, in our opinion, very short-sighted policy very soon and up-prioritize getting what you have to work to an acceptable standard that better reflects the massive land tiers we pay you for doing so. And please forget about all those flashy new bugged features… SL is great as it is, if it would only work as it’s supposed to!

    Apart from the instability I don’t think SL NEEDS so many new features. You have done a brilliant job with this universe, and it is incredible what can be achieved with the tools you have already given us to work with 🙂 I am in here day and night, and I doubt that even if I reserved half my time exclusively to research, I would still be unable to keep up with all your new features and learning how to use them all – and I am not a slow learner.

    Please invest more in context and functionality! THEN, when you have it all working, you can think about decorating the “Christmas tree” with all the bells and whistles. Just don’t pile it on until the tree keels over, if you catch my drift : )

    Also, I must agree with what Gabbi Rotaru has to say on the subject. Your reasons for not implementing a local back-up feature from day 1, I might be able to understand. But now that the technology is clearly available (you just have to get it from a private entrepreneur for 11,000 L$) you choose to implement graphic enhancements instead of addressing this. That is beyond me!!!!

    I hope you take this advice seriously and give it due consideration, before you get some real competition breathing down your neck, and the rats flee the ship.

    In my opinion you have the edge on this market and the opportunity to become the future of this entire media if you play your cards right. Without doubt there will soon be others trying to take a bite out of your cake!

  71. Fellatione Aabye says:

    Dear Linden Labs,

    First Respect to Teeple who dares to comment …. (One of the few Lindens who dare)

    But overall the experience of SL … is getting worse by every new failing update or whatever you will call it ….

    In Europe we would call that purely incompetence and discontempt towards the residents ….

    Wouldnt it be better FIRST to resolve definitively all the KNOWN issues before adding BY NO-ONE ever wanted gimmicks like windlight … or VOICE .. Because since the introduction of both SL became more and more instable.

    Would not repeat the loss of inventory and rezzing problems (SL Residents start to get used to that phenomenon) … But what if, by Linden Labs, screw ups someone looses a item of great monetary value? You wont refund it … or am i wrong

    But still enjoying SL

  72. Zi Ree says:

    @67: This notice has been moved to the lower right of the scrreen and made passive so it doesn’t interfere with your normal activities anymore. You can see this change in newer versions of the viewer.

  73. Gino says:

    I live on an Estate where only 2 of the 25 regions had their servers updated. Why only those 2, makes no sense to me.. So when will the other regions in this Estate be updated?

  74. Prospero Linden says:

    @Nik Woodget: try out one of the Release Candidate or First Look viewers. The notification you get when moving into a region with a different version is much less obtrusive there. Bear in mind that those viewers are unrelease software, of course. However, I think you will find that crossing regions into a different version is less jarring.

  75. Gino says:

    @72 Thanks for letting me know about newer alt. viewers , by which you mean, I guess , Windlight or Dazzle and a First Look client if they have one. But why does SL force me to make a switch now to one of the Beta clients. They should have held off on these server upgrades untill the official client had been upgraded.

    And since now with the new Het Grid, and adjoining regions possibly running different server versions, WHY does the Viewer(Client) need to inform me at all anywhere on my screen? Why not bury that info on some log file to be consulted IF you need to. Last nite I didn’t notice any change in performance etc. when I crossed back and forth between updated and non- updated regions except for the pointless Notice pop-up.

  76. Gino says:

    @74 “However, I think you will find that crossing regions into a different version is less jarring.”

    Jarring ??? How about Show Stopping , in your tracks, ’till you close the HUGE pop-up window?

  77. S H1 says:

    Hi, i have Problem with my sim. the prims are not right it show me Primitiv on SIM 594 but i only use 586. when i take all. 8 prims are there. but there are no Pirms what happend?Musst i restart?

  78. @77: are there any Linden Plants on your land? They count as one prim each.

  79. Sean Heying says:

    Thanks Teeple for communicating so well, and thats to all the engineers who pulled it off. Not needing to rollback after all was a nice thing for those in affected regions.

    Time to stop the “Dazzle” and live up to Philip’s promise now to focus on core stability.

  80. Pingback: MMODump.com » Blog Archive » Second Life Server Software Updates (1.19.1)

  81. @61 U M that’s right dude… I turned 60 in January. You got a problem with that (muahahaha) I built My first micro computer on a SouthWest Technical Products set using a Motorola 6800 chip in 1975. Which sounds like from your comment that you weren’t even a gleam in your daddy’s eye yet. I read your trolls with amusement but keep trolling U M. I am not the only one I know on here who has been involved with programming computers longer than most of those who come to My island and club have lived.

    @63 Teeple Linden yes, I did apply 🙂 but you aren’t hiring old men that speak mainframe assembler :):):) njo matter how long we have been making the same mistakes you at Linden Labs are going through. But I will send in one more resume just for giggles.

  82. Pingback: MMODump.com » Blog Archive » Linden Lab asks residents in 521 regions to down-tools

  83. Sedary Raymaker says:

    @75: “WHY does the Viewer(Client) need to inform me at all anywhere on my screen?”

    So you’ll know why your vehicle suddenly started misbehaving after you enter a Havok 4 sim, for instance. It helps me know that I need to detach my wing controller or else I’ll be going airborne every time I encounter a small step or a rock.

  84. Pingback: Second Life Server Software Updates (1.19.1) « Official Linden Blog

  85. U M says:

    Hey you are a oldie! 🙂 thats wonderful! see too manytimes people say waht you said and really its not true…….because they can`t be doing it that long if the say 12 in rl………if that was true

    BTW its not Dude its Ms………

  86. @85 U M mea culpa ms 🙂 I would never call a woman a dude (knowingly). and yes it is all true… (I even met My RL wife on SL) 🙂 different world today. Now if we can just get SL to stabilize ONE VIEWER and not make Me open bug reports on 3 different ones….

  87. Fixyour Product says:

    Does anyone at your office know what they´re doing? Back and forth, broke, stabilized, resolved and all the way back…

    What was all that meaningless talking about getting the grid more stabile and improving service?

    The messages on the blog get more shocking by the week and so does the inusability of your product.

    You don´t even manage to keep the hp going…That´s a bloody webpage!

    Get your stuff together folks!

  88. MOLL DEAN says:

    for the first time in more than one year, ALL SCRIPTED OBJECTS in my land FREEMIND is not working…

  89. Stephe Ehrler says:

    Way to go guys! I know it would have been “easier” to just roll these back but you spent the effort to come up with a solution that didn’t create huge headaches for the residents!

    I also agree an “auto return” of everthing in a case like this would be a nightmare. I have a private island sim with ~1500 prims scattered across it. To find all of that in my lost and found (especially if I hadn’t done anything since the last saved state that would even get lost) would surely ruin my day.. I’d rather lose a few objects than have to rebuild the entire sim from scratch..

    Stephe Ehrler

  90. Flux says:

    WHY WASN’T THIS ANNOUNCED INWORLD?
    THATS SO RUDE

  91. Holly says:

    @90, Flux —

    It WAS announced inworld, as the blog says.

    “We’re about to begin shouts into these regions every few minutes, warning all residents and guests to desist from building, rezzing and further parcel transactions until we’re ready to trigger a 7PM PST rollback.”

    There was a blue box every 20 minutes or so. Or did you mean the final resolution, without all the rollbacks?

  92. Damona Rau says:

    @71 Fellatione Aabye

    you haven’t clue what your talking about, right?
    Windlight is more a Viewer specific “thing” and Voice runs on different servers.

    But other things are big problems, as they are:

    – Group-IMs fails very often
    – Group-Notices fails sometimes
    – Friendslist sometimes disappears and with look you get it back
    – bad SIM-Performance on some islands
    (nerver tell me that SIM and Physics FPS between 35-40 is fine on a empty SIM)
    – failed to rezz objects
    – inventory loss and no refunds / replacements
    etc…

    The List is long and LL has my sympathy but sometimes it looks more like patchwork as improvements. As further example the new FirstLook Viewer “Dazzle”. Who needs it realy? Work on the regular, RC and Windlight Viewer (the last one has a much better performance as the RC and regular) before you start to write a new viewer like dazzle.

  93. This is how Linden Lab “Dazzles” us.

  94. U M says:

    Harleywood Guru , that is so wonderful 🙂 awwwwwwwwwwwwww
    I am sorry if i appeared rude ok it was meant that way but I am now sorry. They rejected you because of your age? Thats not right. As I recall there is one linden that is over the age of 55. Call them direct and ask to speak to someone incharge of Software Operations…..They silly not to hear you quilifacations……Which we neeed sorly these days…………..

    Good Luck Love,

    Usagi 🙂

  95. Pingback: Second Life Server Software Updates (1.19.1) « Official Linden Blog

  96. Harris Emmons says:

    Well, I lost a bunch of stuff. I was moving some rarely used objects to a notecard, and when I saved the second time, poof, error message and no objects.

  97. Fairtax Freenote says:

    Has Linden Labs ever considered selling Second Life to another company, one that is better equipped to handle it? Right now, Linden Labs does not appear to have the resources to maintain this game, and the worse it gets, the more people with paid accounts, sims, islands, etc. that are going to leave, resulting in a massive financial loss for LL. If you sold the game, you would make a decent profit, and could put that money towards other projects that LL can handle more efficiently.

Comments are closed.