Second Life Grid availability in April

Concurrency graph during outage

We’ve just updated the Quality Metrics page, and the numbers show what you already know: April was not a good month for Second Life Grid availability. Our internal outage tracking tool estimates that about 630,000 usage hours were lost to global system failures over the course of the month, which is about 1.9% of the total (up from 0.06% in February and 0.22% in March), and resident surveys clearly indicate great unhappiness coinciding with these failures. (We define lost usage as how much time Residents would have spent logged in but did not, due to Grid failures; it is meant as a global availability metric and does not cover local failures like sim crashes, inventory problems, and the like. See actual[black] vs predicted[blue] concurrency graph excerpt, right.) I’d like to address the causes for this, and what we are doing about it in general terms.

What happened, and Why?
So, on to why April was so tough: virtually every piece of mission-critical Second Life Grid infrastructure failed catastrophically at least once during the month. Here are the biggest sources of downtime; I list these not to shirk responsibility, but to illustrate the near-perfect storm:

  • Intra-Grid Network By far, the largest event was the near-total loss of intra-grid network connectivity on April 4/5, caused by a line failure within the backbone of our primary network provider – who delayed, botched, and again delayed the fix, extending what should have been a relatively brief outage for many hours… and then the failure was repeated on a smaller scale on April 25. We’re working with the provider at the executive level to address the risks that allowed this to happen.
  • Central Database We encountered a new crash bug in our central database, resulting in a series of four database outages. We have subsequently identified a work-around, and should be able to avoid this particular crash in the future. In general, eliminating this cluster as a scalability bottleneck and failure point is a very high priority for Linden Lab.
  • Asset Storage Cluster The asset storage cluster crashed during routine maintenance; this was a repeat of the crashes that afflicted us in December and January. The vendor had previously assured us that this problem was fixed, and so we continue to work with them to investigate the cause and potential solutions.
  • Data Center/Transient Data Services Our data center in San Francisco cut power to many of our most critical servers over the course of two days. We had advance notice of this one, but working around the power cuts did cause some disruptions. This was a learning process as well, and we’ve refined our processes for dealing with events like this.

For more information on the systems that make up the Second Life Grid and how, when they fail, they impact the platform, please check the Service Disruptions page. Because component failures are never wholly avoidable, our goal (and the goal of online services everywhere), is to reduce the extent to which the Second Life Grid is affected by the loss of key systems. With a platform this technologically complex, adding the necessary redundancy and failure management is a long process, but we have not been standing still: on average a database crash in April cost about 14,000 lost usage hours, vs 53,000 hours for a similar crash last August (at a time when substantially fewer people were using Second Life). We will continue to reduce the impact (on both logins and in-world functionality) of these crashes, until they cease to be significant, at which point they can come off the Service Disruptions page.

Clearly, though, there is still a great deal of work left to do. Our long term strategy includes specific plans to eliminate the risks associated with all of the failures I listed above. We make progress every day, and continue to hire the best and brightest technical staff at our offices in San Francisco, Mountain View, Seattle, Boston, and Brighton. (See our hiring page.) In the meantime, our service record is not perfect but we are confident that we have identified some key areas to improve and will continue to move forward.

Finally, I’ll mention our new coalesced status reports page, which replaces system status updates on our primary blog. Look there for all Grid-status information, including information about upcoming scheduled outages: http://status.secondlifegrid.net/

This entry was posted in -Miscellaneous, Operations, Service Status. Bookmark the permalink.

149 Responses to Second Life Grid availability in April

  1. U M says:

    Woohooooooooo Someone spoke truth!!!!!!!!!!!

  2. KMeist Hax says:

    best blog post ever. it explains everything.

  3. Tired Sim Owner says:

    And I get compensation how? _ don’t offer open sims which just add to long term costs . I want cold hard cash adjustments for lost simaccess and sales revenue.

  4. U M says:

    Shocking yet pleasing to see truth and these matters. Then agian Ian Linden is that type of Linden. Lets make this a Habit LL ok 🙂

  5. Smokey Newman says:

    Erm we allready knew the stats would be down.

    So what are you going to do to compensate customers. Yes we are customers and we have customers we cant just show them a graph and explain technical issues. Please listen to us all and give is some compensation.

  6. urewshmycmd says:

    you know, no matter how good or bad the news is for something there is allways those that have to complain about what could be. If those of you really wanted to make SL better you would let the lindens do their work and stop complainning everytime something does not go your way. it amazes me the people who post here in the blogs and not 1 of them can do better than the ones working on the issue to begin with. i figure if anyone can do better i’m sure that LL would love to have you and if you feel you can do better feel free to take your attempt at an application.. who knows.. maybe just maybe you can do better.. if so *Clap Clap Clap*..

    ~Jamie~

  7. Smokey Newman says:

    Me again but after the What happened and Why should there be and how can we help you. Those who lost out on sales,those who lost tenants, those who still pay a full tier for a service that did not work.

    Thae fact is people have lost trust and LL are not compensating.

  8. Ann Otoole says:

    And when, exactly, will you implement a true transaction manager ala Tuxedo or equivalent mature proven scalable technology that will provide rollback and logging of failed transactions that, in turn, will automatically close the grid to any transactions when the system exceeds a threshold of failures and notifying/paging staff on the side while it is at it?

  9. Malachi Petunia says:

    “We define lost usage as how much time Residents would have spent logged in but did not, due to Grid failures; it is meant as a global availability metric and does not cover local failures like sim crashes, inventory problems, and the [teleport, search, and asset failures]”

    So in other words this is much worse than LL is actually admitting to, whilst blaming their data center, their database, the power supply. Given that Second Life Grid is being sold as an internet service, by their most charitable possible appraisal they are up only 23.5 hours of each day. And people are dazzled by this “honesty”.

    It is one thing when others use statistics to lie to you; it is another thing when you use statistics to lie to yourself.

  10. U M says:

    But how many times does LL Omit they had issues resulting in this bad of a downturn……….LL is going to have one ***** of a trick coming out of their shirt if to make the following month look real and upbeat….. Population stats are always fixed and lied to show resents from around the world who is logging in……That set of facts and stats will always be lied about

  11. JB Kraft says:

    Nice spelunking! 🙂 I can live with not being able to log in. My ISP goes into the aether from time to time too. The real trouble however is when the grid *is* available. Perhaps superimposing a lag, inventory loss, failed transaction, no tp, client crash graph, just to round out the full experience? It would make that chart so much more informative; like a cross-section of the san andreas fault laid over the marianas trench with the himalayas for a background. Now that’d be purdy! 🙂

  12. Aminom Marvin says:

    This is a step in the right direction, but there needs to be more. Don’t just give us generals. Give us specifics: describe actual points of weakness in the SL system, their causes, and plans of action to fix them.

    The grid has gotten so bad that flying vehicle use is almost completely broken. Using a flying vehicle with nobody sitting upon it except the pilot, there is a region cross failure 1 out of 20 sim crossings, resulting in a need to relog or teleport.

  13. Thank you for the honest and informative post, Ian.

    It’s great to learn of the causes for the major problems, and the actions being taken to prevent similar events in the future. Without this sort of information, we tend to cynically (and rather ignorantly) assume that some Linden spilled their coffee on a server rack, or unplugged the asset cluster so they could plug in their Xbox and play GTA IV. 😉 I kid, but there’s often an air of “Ugh, the Lindens screwed up again!” when something goes wrong and isn’t explained clearly.

    It’s also very nice to hear responsibility being taken where it should. I would have been rather disappointed if this post had been honey-coated, head-in-the-clouds, “Everything’s groovy, you guys need to mellow out” PR fluff.

    So in summary, I thank you and salute you for your honest and informative post! As #4 said, let’s hope this sort of good communication becomes a habit. 🙂

  14. Katt Linden says:

    Folks, please be mindful about what you say here. Wild accusations are not appropriate.

    I hope that you can all appreciate this post, transparent and open about what was causing some big issues.

    Thanks.

  15. Smokey Newman says:

    Katt sorry

    But nobody is helping here we have not even heard the word sorry.

    We Pay Lindens and we are getting no customer satisfaction at the moment.

  16. Damona Rau says:

    As first, thanks for this informations Ian.

    At least there are some questions open:

    Why you have moved the Status Reports to a new website with disabled comments? It smells that you won’t any comments about the outages / failures.

    What thinks LL about a compensation for all the money lost we (the business owners) have had?

    Last, but not least, i’m missing the february in the quality metrics chart (the first chart here: http://secondlifegrid.net/resources/service_metrics).

    I would love to see a chart for the transaction / TPs / Groupchat and Notices outages, the primary things we need fully functional in-world for the economy. If i count all the hours where transactions fails, you will have more then 2,2% lost in April.

    Your on the right way, but there is very long way to go for LL with some changes in the strategy. You can’t hope that we have more patience or that we can accept more excuses, too often the same failures occurs. The symptoms affects the residents but LL have to work on the causes. All the “Resolved” posts are nothing more then “we have fixed the sympton, but we don’t know the cause”, isn’t it?

    I like SL and there is still a little bit hope that things goes better. Since February there is more anger then fun with SL.

  17. Damona Rau says:

    @14
    I’m sorry Katt, but why i should appreciate a normal behaviour like the post from Ian? As “Service Provider” your in charge to keep your customers up-to-date about the failures and outages.

    The residents pays your salaries, please keep this in mind. So it should be one of the important points to keep your customers happy.

    As example, my Internet Service Provider offers compensation when there is an outage from more then 4 hours. Ok, it’s a german company and in germany it’s a normal behaviour from the companys that the customers gets an compensation when failures occurs (and, btw, they are faster with explanations, what kind of problems occurs). And the next thing, when this companys writes “Resolved” then it means resolved and not, that the same failure appears some days later again.

    Maybe it’s better if you write “Fixed”, because resolved is nothing really, isn’t it?

  18. Darek Deluca says:

    People;
    Read your TOS. You are not have any right to compensation!

    Katt;
    Give it a break.
    Frustrated customers who have no recourse except to quit need some outlet.

  19. Almadi Masala says:

    I can live with lag, login problems, and transaction, TP, and IM failures that come and go — but permanently losing a big chunk of my inventory is much harder to accept. Since being hit with such a loss, my use of SL has fallen off to almost nothing. Many of my favorite items are no longer usable, and I don’t trust the system enough to go to all the trouble of replacing them — knowing that they all could disappear again at any time. A month ago I was a very enthusiastic user of SL. Now the fun is gone, and I am seriously wondering whether it is worth staying. Will the inventory database ever be reliable? That is the main question that will determine whether I continue to use SL.

  20. Katt Linden says:

    I’m sorry any of you have experienced downtime and problems.

    Seriously.

  21. DISGUSTED IN LINDEN LABS™® *!@$!! says:

    lindens could at least recover our classified payments for the month of april ………………many people pay 1000s of real life dollars a month for these classifies at least extend them out for us for a few weeks its the least you can do and i’m sure would make people a little more happy Lindens did this last year they extended our classifieds when grid and search was down it is the least you can do to take a step in the right direction. It may not make everyone happy but it would apease the crowds that at least you care something about your customers because at this point in time we are feeling a tad more than TICKED OFF

  22. toybodacho Ireland says:

    I have always wondered why Linden Labs won’t test their so-called “upgrades” (I personally call them “downgrades”) on a few special sims for 3 to 4 months before they apply them to the entire grid. That way they could detect most of the bugs before making us suffer through them.

    @16 Damona: My guess is they moved the Status Reports page to keep the “dirty laundry” out of plain view.

  23. Netsuko Yoshikawa says:

    Thanks for the heads up LL. I can understand that some residents are getting angry, but I also understand that it is not always possible to maintain a 100% uptime in a system that has grown so large and diverse like the SL Grid.

  24. Kahni says:

    Thanks for the breakdown on the information.

    There’s a lot of work to do yet, but being honest about numbers like this, and laying it out like this is refreshing. Much more so than “we’re banging on things”.

    I still think there should be tier refunds based on down-time though.

  25. Paulo Dielli says:

    Yes, we are complaining a lot. But to be honest, there are enough reasons to complain. I have the utmost appreciation for the concept of Second Life and the efforts that it takes to keep it afloat. But it really needs to get more reliable.

    It’s very good to be open about metrcis. In fact, it’s great. But from now on: please just focus on grid’s stability. At the moment we have all the basic things we desired, just keep it with that. And communicate with your customers what you are doing to improve stability. That’s all, nothing more.

  26. Ash Meersand says:

    Three weeks ago lots of library items from the sign-up outfits like Nightclub Female went missing. I spent 12 hours researching the “missing items” phenomenon and became very worried about the safety of my items. Then I decided to make a “missing from database folder” and went on with my Second Life.

    Two weeks ago I was unable to log in. It said it couldn’t find the domain a week ago. I used an alternate, four times slower, Internet service after an initial six hours of fruitless research. Ten more hours of JIRA scanning led me to do the Vista host file changing trick, which worked.

    Less than one week ago, I installed Release Candidate 6. I visited Torley’s Watermelinden Land, played around with Windlight for the first time for a few days, then teleported away; all the terrain on the mainland was the same as Torley’s. I tried researching if I could bake terrain textures like avatar clothing textures. while being harassed by griefers pelting me with couch sized-bullets and pushing me into the next sim.

    I put up with the faulty terrain textures and crossed a simulator boundary in a heavily populated area of the grid for a trivia contest event. Then my character kept walking out of the building on air and getting disconnected from the sim. This repeated three times before Release Candidate 6 said my Internet connection was faulty and offered to teleport me home, which didn’t work, and thereafter I was told “despite its best efforts” I could not connect to the Grid.

    I uninstalled RC6 and re-opened 1.19(4) on my computer, through which I could connect to the Grid and see textures just fine. I played around in the Plum sandbox happily building intricate connected tubes to send spheres through. Then today I looked at the Grid status page and it said they were having severe networking issues.

    I tried connecting today and was told despite “their best efforts” I couldn’t. I used “my best effort” my researching for about eight hours today and yesterday how to get around this. Going directly to login.agni.lindenlab.com with a browser suggested something related to not being able to be redirected with the fault being in the webmaster hands.

    I went to JIRA and filed a detailed bug report, but my connection kept timing out for about two minutes when visiting the website every time after I pushed “submit” and my report was lost. At least this time I’ll have saved my “log” in Notepad.

    I have given up and uninstalled SecondLife. I will reinstall it and try again after I have seen it go three days without asset server, grid, or log-in issues. That sounds more than reasonable to me.

  27. Verdana Klaar says:

    A good number of comments here can be summed up as follows: “I want my money back”.

    This means that the main facet of Second-Life has now shifted from an environment that was a mix of fun and creativity (together with some small business) to an economical system at the detriment of nearly all the rest, where residents log into to buy land, sell parcels or open shops filled with items of any kind (which they have not created themselves for a majority).

    The point is that – just like anything in SL – this new facet of the metaverse has been created by… the resident (your world, your economy).

    Also, if i am not wrong, no one in LL has ever forced nobody to get a premium account, invest some hundreds or some thousands dollars for a land, a shop, or alike.

    In the end it looks somehow as if a category of residents had willingly placed their savings on a stock exchange place, then had lost all because they choose wrong options in investing, and… afterwards call for a refund.

    Some residents are funny.

  28. Jayme Llewellyn says:

    Nice to finally see some honesty from the ‘labs. Maybe now they will realise that what needs to be done is a sever upgrade of the grid for stability and the instillation of police program to detect any bugs and viruses. I’d suggest AVG and Adaware …. but what are the chances of Linden Labs listening to a customer?

  29. Cocoanut Koala says:

    I do appreciate the straight-forward and informative post! Let’s hope these days are behind us!

    Nonetheless, I agree – reimburse us.

    coco

  30. Partington Gould says:

    A nice Honest post. Thank You.

    Grid Status reports on another Blog, not an issue for me. I see 3 Status reports now at the top left of this page. I see the status reports when I log in (RC6). I see no reason to comment on status reports.

    This is just me, I don’t claim to speak for everyone.

    Thanks again for the honesty, it really is a step in the right direction towards what others posting here are asking for.

  31. Darien Caldwell says:

    Ouch. that does look like a nasty graph. There were also a number of issues which revolved around several failed attempts to deploy new versions of the server software. I didn’t see that mentioned. Perhaps the contribution was small, but it did have an impact on overall grid conditions. Otherwise, a good summary of April.

  32. T N says:

    “The asset storage cluster crashed during routine maintenance; this was a repeat of the crashes that afflicted us in December and January. The vendor had previously assured us that this problem was fixed, and so we continue to work with them to investigate the cause and potential solutions.”

    Is that your Isilon cluster? As a potential customer, I’d like to know that.

  33. Shai Khalifa says:

    Kat, if you can’t say anything constructive, it’s probably better not to say anything at all.

    There are some very very legitimate concerns expressed here – and a trite “I’m sorry any of you have experienced downtime and problems.” with the word “seriously” added as an afterthought – is more insulting than anything else.

    The metrics displayed here should be cause for deep concern at LL, and if a professional attitude is to be displayed, some compensation for lost time should be offered. In fact it should become practice.

    When the grid is either down or is in some way disabled (particularly stale transactions) a pro rata compensation should be offered in terms of delays in the resident payment dates of Classifieds, and premium memberships – and tier payments should be provided by LL.

    That way, LL is forced to ensure that there are emergency processes in place, and scaling projections resourced to minimise the financial cost to LL.

    We pay for service, you should repay for lack of service provision.

    Down time should not be classified only as time the grid is not available for logins – but should include time when essential aspects of the software are unavailable – such as transactions, tp, group IM, Classifieds, Search etc.

    And these stats should be readily available on a daily basis to those who pay the exceedingly high rates LL charges for the ability to create the exceptional content everyone enjoys.

    If, according to the TOS, LL own everything we create – then for us to create it, there has to be some form of recognition, and if that is simply by compensating people when they can’t create or manage their creations it would go some way to ameliorating some of the issues we have to constantly deal with – it should be a simple solution.

  34. Jayme Llewellyn says:

    Just thought you should know – though most everyone here probably knows this anyway (everyone bar the Linden’s, that is) – May is starting as a yet another great month for stability. I was on last week with my other avi and I experienced five crashes in my three hours of SL-time. Today, I have been on with my main avi and have had three crashes in an hour – I cannot seem to get past 18 minutes in Real time without crashing. There are some serious problems with the grid stability and Lindens really need to fix it. The above graphs are telling us that.

  35. Pingback: Massively.com SL news

  36. Katt Linden says:

    @22 toybodacho: As @30 Partington notes, the top three Status Reports show up in a feed at the top left side of the page. You can also mouse over the feeds there (just checked it with both Firefox and Safari) to read the initial text in the update.

    You can also subscribe to the RSS feed http://status.secondlifegrid.net/feed/ or to the Twitter https://twitter.com/SLGridStatus.

    They are also always in the Login screen in the Second Life viewer, so each time you log in you see the latest.

    @12, Aminom, and @26 Ash, and the rest of you, I do hear how frustrating it is.

    @23 Netsuko, and Paulo, Kahini, Verdana, Jayme, Cocoanut, Partington, and the rest of you — thank you. I’m glad you’re here.

  37. Umbra Lunardi says:

    While this kind of transparency is a welcome thing, the level of service we experienced over the past month was abysmal. No amount of explanation and “We’re sorry about the problems” compensates the citizens who pay your salary. I pay a mere $95 USD a month to participate in SL and I can safely say that I did NOT get good value for the money in April. I can’t imagine how some of the people who pay many times more than that feel. Sure, the TOS says we’re not entitled to anything, but I agree with many others that a refund of some kind would go a LONG way to creating some customer satisfaction.

    I’m at the point of abandoning SL because Linden Lab is no longer delivering the service I believe I should be getting for what I am paying. I just don’t care about SL anymore. What a shame.

  38. Sindy Tsure says:

    Nice write-up, Ian. TY! As annoying as problems are, it’s always good to see some kind of post-mortem afterwards (and thumbs up to the other Lindens who do it, too!). 🙂

    Also, I’m getting a File Not Found from WordPress when I click that graph excerpt… 😦

  39. Dahlia Trimble says:

    Thanks for the detailed blog post, it helps me feel less frustrated when things arent working and I wonder if the issue is being addressed or not. I understand that things fail and that it’s hard to maintain 100% uptime 24/7 especially when you are dependant on so many things that are outside of your realm of control. Glad to see that you are working to improve the reliability of the grid.

  40. Taft Worsley says:

    Who are you hosting with and using as a backbone service to have such mission critical outages with no warning and no resolve – sound like its time to get off the value host and get a real data hosting facility with a backbone rpovider with some redundancy.

    Funny the last time i recall your data facility had total power loss and had a longer failure due to an issue with the genorators – real backbone providers and data facilities – first test there fail over systems before fail over happenes and two — have more resources to re route and move traffic at a whim- The only time you run into the issues your company has is when your cheap! That means your raking in the dough for the investors but your spending very little or not enough on mission critical services – Answer me this question, how come other companies both large and small don’t run into host and bandwidth issues like second life has blamed on these vendors.

    Funny how its always the next guy – seems that the IT business is full of people who love to blame others – just seems funny to me that companies large and small can have 99% uptime with millions of transactions, yet your host has a outage and it take out your core business — DO YOU REALLY THINK PEOPLE ARE THAT STUPID!!! Give us a little credit – If your host has had issues its not the first time and since its your company life life to make cash – maybe its time to tell the investors to wait for a return on their money and to spend some time getting this system to consistantly working – If that means changing data providers or leaving facility not able to handle your business – so be it — but i have never seen a company as troubled as yours and then turn around and blame the very vendors you keep paying for the shabby service – BET YOU GOT A REFUND FOR YOUR OUTAGE – SEEMS WHATS GOOD FOR THE GOOSE IS NOT GOOD FOR THE GANDER — as always with this piss poorly run company!

  41. Happi Homewood says:

    Fact: SL is still the most stable platform of it’s kind.

    SL is pioneering technology. We should expect these bumps in the road. I, for one, will not leave, I look at the long term possibilities, instead of focusing on the current problems, which are really not that bad, compared to problems in the past.

    @the ppl always telling the Lindens how to run their business:

    Why not apply for a job at LL, if you are as good as you project yourselves, I’m sure LL would hire you right away.

    @LL: Excellent, full transparancy will serve you well in the long run 🙂

  42. Sekonda Huet says:

    Stop faffing around sponsoring dynamic control input devices and focus on fixing the thing.

    Ever since the 1.2 release client was deployed things have been horrendous, not only is the client horrific in every way and there is no easy options to revert to the previous function which many people for many reasons have expressed a great distaste for.

    I get that you can’t prevent “Acts of God” if you will. If the power cuts you can’t stop it. It’s not like Linden is randomly unplugging the servers for the giggles. It hurts them too and costs them money not just us.

    More money desperately needs to be poured into hardware sustainability and less into new mainland areas that servers won’t be able to cope with anyway.

    I do think tho that Linden should always consider reimbursing customers who pay your company. When Blizzard Europe messes up it compensates players with game time.

    Why not toss a few hundred linden towards customers here? What’s a dollar to a company making millions daily…

  43. Seth Ock says:

    Thanks for the info, Katt. I honestly don’t know how you folks can resist telling some of these people who post here to kiss your shiny Linden asset server, but then I suppose that might be perceived as bad public relations or something. My experience with SL is a bit more minimal than many others, however. I make things largely for my own amusement and curiosity and sell very little. To some degree, this is the result of the volume of complaints that decry the dependability of things like the inventory database, or the transaction exchange system. Both of which have cut out on me even with my part-time usage. Add to that a viewer that feels clumsy and leaks memory like a fat man sweating in the desert, and my opinion is that SL is a cool idea that almost succeeds.

    But like Virtual Reality back in the late 1980s, it’s very much in danger of alienating its supporters as being an idea that’s ahead of its time. The technology just doesn’t seem to be there for this sort of idea just yet. With home computers that just barely have the power to support a virtual world, bandwidth that can’t handle the data loads for more than cartoonish representations, networking that grows increasingly unreliable as it scales, and databases that have to cope with creativity and experimentation on an unprecedented volume, the number of failures that can occur between my eyeballs and your vision add up to being almost insurmountable today.

    Still, I hold out hope. I hope the viewer eventually stops leaking memory and gets cleaned up. I hope the database issues get resolved. I hope asset transactions become more reliable. I wish you luck, because I think virtual worlds are a useful idea that will one day be exactly the kind of cultural backbone Philip and Mitch envision them to be. I hope, even with the minimal investments I have made, that SL will be the recognized leader of virtual worlds.

  44. Solomon Canning says:

    Ian & Kat — Thanks for the informative post.

    May I please suggest that network delay and loss stats also be tracked and included in the future? The region viewer network link is an important part of the user experience, too.

    Also, I see that region crashes are way down (presumably because of Havok 4), but the total of region crash + viewer crash seems to be unchanged. Did SL viewers really get that much worse, or is there something else happening there?

  45. Sling Trebuchet says:

    @14 Katt!!

    Ian is making wild allegations
    “our primary network provider – who delayed, botched, and again delayed the fix,”

    Tell him to stop.
    Are we there yet?

    😉

    Seriously though, the “brown-outs” in functionality when the grid is at near-failure are probably more corrosive of satisfaction than is down-time of the graph.

    “We define lost usage as how much time Residents would have spent logged in but did not, due to Grid failures”
    That does not encompass stuff not rezzing, failed TPs and transactions, inventory going poof. This must be causing ‘time Residents would have spent logged in, but did not, due to leaving SL.’
    I’m not leaving of course. I love SL – although sometimes it’s like trying to love a particularly difficult hormonal teenager.

  46. Insky Jedburgh says:

    I want to commend a change in procedure I witnessed today, and feel is reflected with the information laid out in this metric today. When asset servers started to choke, rather than wait until it was crisis level and dump everyone out, or hide behind silence, you made an announcement detailing the problem.

    This allowed us to make informed decisions regarding our actions during the mini crisis today. Please, do that more often. It’s far less insulting than being left to struggle on our own.

    The report in this metric is a good step in the communication direction. There was no way you could have candy coated April though, so it was your only choice. Still I woudl rather hear bad news than no news so I can assess the situation.

    This platform is not what it needs to be, to gain trust among individual and corporate participants. SL is still far too unstable, and prone to unpredictable changes that often lead to failures, to be faithfully used as the resource it could be. Please keep the majority of your focus on functionality for now, and perhaps get the latest browser fully functional and leave it that way for a while. People are getting tired of the constant adjustment to what seem like arbitrary additions of new innovations, that don’t address real issues.

    It would be nice to have a system that we could get familiar with for a while.

  47. While I can agree with some of you that Linden Lab (and please, not the Lindens themself) did its mistakes of course, I find hard to consider some customers accuses appropriate.
    A fast growing system (with massive usage) requires fast growing company politics, but also requires more partners. The more partners (network providers, support teams, outsourced data centers etc.), the more risks something goes for the wrong way.
    So of course LL has to take full responsability for the service they offer (even if the service provider is found guilty), but this doesn’t allow anyone to humilate these guys (and it happens here more and more often) who work hard just like anyone else and certainly are under a big pressure.

  48. Jerry says:

    Sidewinder can fix anything , you folks otta give the man a raise and let him take charge . I don’t see anyone else that can step up to the plate like that man does , he should be running SL .

  49. Very Frustrated Owner says:

    Katt – PLEASE!

    Stop saying how sorry you are and take action!

    People paid Linden Lab thousands of US dollars in April for a service they didn’t recieve. The stats above only mention downtime. I am sorry but if you are going to promote and service an economic platform, having no transactions for vast amounts of every day, doesn’t lend itself to a service that is being provided.

    If my ISP network was down for these amounts of time, I would get compensated, why will Linden Lab not compensate people for the lack of service in April?

    We don’t want apologies anymore (nice at the start), we want compensation and a grid that works. You have lost the confidence of thousands of people and its time to change that. Its been 6 – 8 weeks of problems now, not an acceptable timeframe by any account.

    Also in that time you have slashed the cost of islands, meaning hundreds of people have lost money on their land and now you are planning to change the core principle that the search system is working on, i.e. traffic. Did you intend to press the self destruct button? Either way you have scores of customers that are not just unhappy, they are hopping mad with Linden Lab.

    I would like you to come up with a compensation package for April for everyone that varies in % rebate of tier. After all what other business is allowed by law to charge for a service that isn’t recieved.

    I think the “sorry” card has been over played. Its easy to say sorry, its not easy to go to your CEO and say we have thousands of unhappy customers that feel we are breaking the law with charging for a service that wasn’t provided, we need to look at putting together a compensation package and communicate an action plan to restore confidence in our abilities to provide this service.

    A tough call Katt – but the future success and confidence in Linden Lab rests with your abilities to fix this.

  50. Dinohunden Paine says:

    About time LL found out, that we’re pretty pissed over the downtime and unstabillity. Now we only wait for more serverpower, preferable NOT in the US but also in Europe, to we don’t get those ridiculous ping times, that we have. Drop the new features for 3 month, and work only on the stabillity problem, it’s pretty urgent.

  51. Chavi Skomoroklov says:

    Click on the graphic and the answer is “404 — File not found.”
    And yes, at my company we had a department that was shooting all new kind of ideas into the air, but when it comes to customer satisfaction, it’s all about GET THE BASICS RIGHT.

    🙂

  52. Flew says:

    44 wrote: “Also, I see that region crashes are way down (presumably because of Havok 4), but the total of region crash + viewer crash seems to be unchanged. Did SL viewers really get that much worse, or is there something else happening there?”

    with H4 sims may not crash that much anymore, but it does cause mega lag now. I prefer the good old sim crash to be honest…and just move to an other region. The giant lag in some regions makes me more frustrated then being in a sim that crashes.

    Also since wind/light (or its the viewer) the memory usage on my pc has gone up a lot and and frame rates have dropped a lot. And if you complain about it with support they just ignore you.no response what so ever. even though i gave detailed info about my pc and what problems exactly i am experiencing.

  53. Actingill Igaly says:

    Ian/Katt

    Thankyou for finally posting this long overdue blog. At last, all those that highjack every other blog with crys of “Dont tell us that, fix the system” have a legimate place to voice their feeling.
    I dont believe anyone who understands how cutting edge SL is will expect 100% reliability, stability and uptime from the system; however over the past two months I have watched people get more and more bitter and twisted with each incident. This I suspect, has more to do with Linden Labs silence on the subject than the actual failures themselves.
    Hopefully this will calm the vast majority down. Acknowledging there is/have been problems and providing a place for people to make suggestions or let of steam should have been done a long time before now however.
    As to refunds, I dont want or expect one. I would much sooner the money be used to improve the infrastructure than to temporarily placate the hardcore moaners…..
    Please keep up this level of communication on stability/grid issues going forward.

  54. U M says:

    Anyone thats been around for over 3 years will tell you Ian is indeed on of the most outstanding Lindens when it comes with dealing with the users. So give him a little break, atleast he didnt lie to use about the numbers and try to make things looks rosey when infact they were not…. Thank you again Ian

  55. Craig Altman says:

    The biggest prob for me during April was the daily transaction failures causing people to pay for things and not receive them, often there was no inworld warning this was happening so I took to closing my shops when IMs started to roll in with people angry they didnt get items they paid for, this was complicated by the fact you cannot actually close shops in SL, people can zoom past ban lines and still buy items, which they then dont receive.

    What I would like to know is when you know you have these problems, firstly can you inform people inworld(few read this blog), secondly if L$ transactions are failing can you disable the L$ balance, you have done this before(maybe not deliberately), the balance would read “loading” and any attempt to buy items would be met by “insufficient funds”

    If you disable transactions this way you can globally prevent people losing money during these problems, I think its the lesser of the evils here.

  56. Elwin Jacobus says:

    Thanks Ian for an informative post. It’s good to see public recognition of what have been some pretty catastrophic failures. What is better still is that (and this is just my interpretation of your post, clearly as can be seen in the feedback others disagree strongly) there are signs that all of these issues are being seriously worked on.

    I (and pretty much everyone here) look forward to a more stable grid. Let’s hope that the team at LL can deliver this sooner rather than later.

    The honesty and openness that has been a feature of the blog posts lately is very refreshing and encouraging.

    NOW! Back to the salt mines with you all and go make it all work as it is supposed to.

    /me taps toes impatiently

  57. Taft Worsley says:

    47 Maya Bogdanovich – Obviouisly you must not have any money sunk into this game that has been wasted on downtime – transaction failure, useless classfieds and lost inventory – Funny when you don’t waste your hard earn money how easy it is to talk smack!

    53 “Actingill Igaly Says: I dont believe anyone who understands how cutting edge SL is will expect 100% reliability, stability and uptime from the system” Your right but even 90% is better then what we have been getting here.

    Funny i manage a 4 node cluster of 4x16way SMP systems which does about 273,000 transactions a minute grabbing data from 480 stores nationwide – while it hiccups once in a blue moon — never less then 98% a month including patches and update downtime – and never do i see outagous due to a service provider issue as i said thats why they make redunant links and have a BPG routing protocol. But i guess the toys they have spent their money on like i said before are cheap and must not provide a BGP option for routing.

    Bottom line TOS or not — the people who have invested thousands of dollars in this platform have everyright to complain about this shabby service – it plain out sucks and yes you should be intitled to a refund because as its been stated — anyway you want to candy coat it in a TOS – this is a service and people have not been getting what they pay for out of this company – some of us longer then others.

    Seems like there are more and more people leaving and yet LL doesn’t seem to care — I personally have been reading these blogs for over two years and you know what in that time LL has listen to the complaints all of about zero times – so don’t preach to me about how good this company has been — do your homework and look at the years postings and see how many times they have said sorry only to tap the teir till without fail – funny how they get there money and you get squat – like i said in my last post — I’ll bet dimes to donuts — they got a serious refund from both the colocation facility and the backbone provider if indeed it was there failures to cause outages – but hey WTF do i know i have only been in the IT business 15 years and no nothing about SLA’s!

    All it takes is one person to open the legal gates and this smells of class action lawsuit no matter what the TOS says — BTW the courts have ruled once already the TOS was invaild —- So its not bullet proof.

  58. Vivienne says:

    Katt, the stats only prove what all of us already knew. At least the resons for the outages are transparent, which is progress and an important step towards more transparency. And hiring more and first class technical staff for the Lab is certainly nothing anyone can complain on.

    Nevertheless there are a lot of technical problems additional to the network caused ones. The client software is overfreighted by hardware killing features and very, very buggy. Some server software code needs reconsideration.

    On the communications front: Offered Open Source additions do not really find their way into the LL schedule. Serious bug reporting by residents fails because LL obviously neglects communication on the recommended level, which is the public JIRA (it seems that only a few Lindens find their way to there adressing reports).

    What SL needs is a open channel for dicussing what residents really need, and this is usability. Usability can only be achieved by some kind of effective quality control. Quality control, from the oow. of residents, means that no code should be thrown on the community without being cross checked by dedicated residents who really WORK with the software. Most of the wild and often justified complaining on the official and non.official channels could be prevented by giving residents a word on what is a senseful and working improvement for their daily second life BEFORE an official or RC release. And taking them serious. I am convinced that LL employees want to make this better, so do residents. But in the end the “felt” quality is the one which matters for the user, not the “tech” quality. An open channel and a restricted, selective betatest program which includes direct communication with the Lab and some authority on further release politics would help LL much more than f.e. any RC candidate which, buggy as it is, only contributes to even more nasty complaints and frustration (even if it adds some noteworthy improvements, too).

  59. Carl Temin says:

    I would like to ask if owners of businesses who had difficulty paying their bills last month could be compensated? Maybe a weeks free classified add listings?
    It must be realised that business is business, Lindens get our money no matter what !
    Come on chaps…play fair !

    Carl

  60. Someone spoke the truth.
    Someone showed the facts.

    Now…

    Let’s see someone start paying REFUNDS for april!

  61. RCE says:

    No one told anyone to invest their money in commercial enterprises in a game. If you chose to put your money into an enterprise, you took the risk, and risks carry the possibility of failure. From what I read, LL is attempting to fix problems. Thats what they should do. As far as refunds for lost whatever, hey, live and learn. I appreciate the honesty of the post and look forward to the improved performance of SL as it matures.

  62. River Ely says:

    Thank you for an open and frank report on the problems encountered. I have been asking for a year or more why we have not yet invested in global resources rather than simply USA Based resources. Mirroring systems on a second continent would allow all manner of process’s to gain invulnerability against a severed connection, server system, or administrative deployment error.

    Second Life is a Global Product and I feel confident that as billing can be migrated to establish a financial center in the UK to adapt for VAT payments, so could a remote mirror for asset delivers, or a communications bridge to allow EU residents the ability to log in while a USA based system was up for sched maint.

    I am probably wishing for pie in the sky and many readers will propound methodologies supporting the inverse of my proposals, but, such solutions are currently used by global marketing companies to maintain an adaptive global presence riding over outages and systems loss from time to time.

    The product, SecondLife is rather good and seems to demand people return with high interest, even when malfunctions that would normally deter any sane customer from ever returning. knowing that, push the supportive boat out with some customer based incentives we are all asking for and watch the amazing strength of support you get for improved growth. I mean it, I cant stay away, and I assure you, when I am treated badly by Business Clients, I vote with my pocket book. Again, thank you for the intelligent and well worded simple report. You score well on communications sometimes.

    River 🙂

  63. Deira Llanfair says:

    Thank you for giving us the details about this. It does help to mitigate the frustration to know that you guys are working to get things sorted out long term. A stable system is the number one priority for the continued success of Second Life.

  64. Jessica Hultcrantz says:

    @ kat (20):
    Yes, i have indeed experienced way too much outage at prime time lately… *grumbles showing some fists*

    BUT:
    @ LL ™ ?
    This is the kind of blog postings you need to do more often. The Truth!
    If the lab cares anything about customers it’s time to honestly open up and share this kind of information to explain what the h**k is going on behind the screens.
    Seriously, we, the customers, are fed up with “sorry” and “thank you for your patience”, that is getting ridiculous by now.
    But some decent honest truth blogged with details about why, where, when and why (ever worked with journalism perhaps?) is vital to get customers to understand that there indeed are problems that can’t be solved in a snap.
    Yet again but… if there is a lack of communications (which indeed is true for over 90% of the blog posts about things messing up, customers get angry, you get bad reputation for not caring and everyone get in a kind of hostile mood starting to look for lawyers and digging ditches to fire from, and who benefits from that (besides the lawyers?).
    So keep the straight information flow up and tell the details, even the really bad ones, that is needed.

    Now remember, for every blog post try answering: What, where, why, when, who and how. PLEASE!

  65. U M says:

    There was a **** of alot more issues then transation problems…….But some how everyone made it through it, grant you many lost alot of money. But then as someone said nobody forced anyone to invest money here. But then again how many times have you made even more then typical times? Doesnt it balance out? Or do you expect perfect stats all the time. Ok I said my share of BS as they say inthe past about the lack of support. But thats when LL lied to use about events and other problems. But when they come straigth with us I can`t hold anything on them. Thats why i feel this is a truthy new era with LL. If they keep this up atleast supporting their effort is the least we can do.

    Usagi M

  66. Damona Rau says:

    @27 Verdana
    I don’t want my money back, thats an impossible thing and the maximum of US$50 won’t cover the lost i have (oh yes folks, read the TOS). But Lindenlabs has some more options for a compesation. Let me point some options.

    – For region owners a half or a full month no tiers (could be handout to the Residents)
    – some weeks no payments for classified ads
    – some weeks no group liability
    – some weeks no payments for texture uploads

    The last 3 points are more “hidden payments” we pay each single day. With all the builders and creators it will be a huge bunch of L$ for Lindenlabs.

    I’ve read in some posts about the “growing environment”. Ok, we have more then 60.000 Residents online, but how many from them are nothing more then camping bots? Let me guess: 20.000 as minimum. So we don’t have that much more “living” residents as in 2007 before gambling was banned (July and August 2007: round about 45.000 residents online).

    I won’t count again all problems (too many, look at JIRA Lindenlabs), I won’t read “Resolved” when only symtoms are resolved and not the cause (how often we read “Problems with transactions” and some hours later “[Resolved] Problems with transactions” and some days later again the same?).

    I would love to work together with Lindenlabs to get the Grid and the services more stable, but Lindenlabs should offer an incentive. The only thing we get are excuses “thanks for appreciation” and begging for more patience, but please accept, that these cards are overused right now.

  67. Deira Llanfair says:

    Apologies for taking two slots in a restricted number response list – but I cant edit my first post.

    I just wanted to draw attention to Craig Alman’s post @55 – please, please Linden Lab pay attention to this.

    Transaction failures have a huge impact on commerce in SL – it is bad if everything stops, but it is worse having to deal with a stream of customers whose purchases failed to arrive. For SL to succeed we need a stable system where transaction failures are rare, but with a realistic process for dealing with any transaction problems – a contingency process that does not make the situation worse.

  68. cosa nostra says:

    @36 Katt

    this is the type of information I wanted to see ! it shows visual the performance, also you added the low lights + possible solutions moving forward !

    but ofcourse, a performance like that is affecting your fix customerbase, people who own sims or businessowners small or large, I talk here about a small % because 95 % of the grid residents are travellers or visitors (and have no payment info on file).

    At least a professional company would come up with a commercial proposal to show the FIX base that they fully take responsibility for the weak performance in april ! Saying sorry Katt is not enough, cause this is too easy …

    I really want to see a commercial proposal towards your sim owners, this could help LL moving forward, cause remind that LL is nothing without paying customers, the customers always determine the success of a company, not the company !

    cosa

  69. U M says:

    @61 – Some would term that as “taking the gamble”. But LL has banneded gambling. So shut up, and join the masses saying we are to be needing many refunds for terrible CRAPPY service that we have been to endure for many many many months now.

    I like using up blog posts.

  70. Kugel says:

    Wooo … a lot of anger and frustration here.. not suprised really. SL has been terrible for a moth or more.

    KAT …. from your comment in @14 about wild accusations…fair point 🙂 But I point out the comment in the original posting…

    “….caused by a line failure within the backbone of our primary network provider – who delayed, botched, and again delayed the fix…”

    The word “BOTCHED” is fairly wild in itself. If I was the providers legal representative at a corporate level… then I would be examining the public use and definition of the word “botched” too.

    Anyhoo … yeah… off my soapbox. YES.. a lot of things seem to be being botched, workedaround, band-aided, patched, load restricted, etc, etc… I dont actually see the word FIXED these days. At least not with any belief or conviction.

    Thank goodness I dont run a business/sim/whatever here in SL. Its a minefield. I wonder how many “new” businesses are actually starting up these days.

    Anyhoo… back to SL for me now 🙂 Test the RC6 .. until it crashes..still…again.

  71. U M says:

    “69 U M Says:

    May 10th, 2008 at 3:01 AM
    @61 – Some would term that as “taking the gamble”. But LL has banneded gambling. So shut up, and join the masses saying we are to be needing many refunds for terrible CRAPPY service that we have been to endure for many many many months now.

    I like using up blog posts.”

    This is not me…….I don`t know who is faking posting my name here bt its not me

  72. U M says:

    @67 Use many blog posts as you wanting. I do it all the time and mostly havent much to say.

    We want refunds. lots of refunds. No tiers for may because april sucked so badly.

    Save another blog post for me for later. I havent used enough yet

  73. U M says:

    Nice little trick in faking names here kid………But if you really need to do it you are indeed a low life…….

  74. U M says:

    I are sorry. Do you have a patent on the letters U M ? If you do I change my names. If you dont, then you are the low life and need to stop calling names. stick to the subject. LL is failing to service use properly and should making refunds immediately

  75. Kugel says:

    U M and U M …. stop it. Let me put the kettle on and make you two a nice cup of tea. 🙂 Besides… its obvious you two have different I.P.’s cos of the pretty icon in your posts.

    LOL.. RC6 just crashed AGAIN … and the crash logger works wonderfully..NOT. 😦

  76. U M says:

    @75 Exactly. I am meaning who he think he gets off being saying i fake my name. If not like, change yours.

  77. U M says:

    anyone can tell but the little graphic to the right ho is who look they different.so If your start you better stop before they catch you

  78. Bobo Decosta says:

    @ 61 so SL is a gambling game? Investing on second life is like placing a bet on the uptime level of second life?!

    Glad I stopped gambling on SL’s service level. It’s much like a casino. The house always wins!

  79. Lorna Volitant says:

    I read these forum posts for a giggle at the moaners..it astounds me how much passionate anger is directed at the Lindens, my workthrough for any inworld dissatisfaction would begin and end with the quit button..and if the problems persists, cancel your account..if you are losing money in here due to outages then consider the wisdom of your investment and perhaps redirect your monies towoards RL causes who are crying out for a bit of cash from the wealthy minority who are quite happy to waste cash on megalomaniac follies..I would suggest reevaluating the time you have on your hands and consider a more constructive use of it.
    I see it simply, i choose to pay for a premium account in SL and enjoy the experience whenever I have the time and inclination to login..if for any reason I can’t login, well, there is a big world out there and it is definitely more fantastic and awe inspiring than anything I have found in SL…
    Enjoy what you have and if it’s not enjoyable, don’t be a victim of your own irrational anger, just do something else..
    Thanks Lindens…it’s all good

  80. U M says:

    As I said before Ian Linden atleast told us the truth and should be repected for it…Yes we had a nightmarish 6 or more week. But ateast the said it with truth. Can`t say that about everyone inthe lInden group. But atleast omitted there were problems without telling us it was not as bad or try to make pardons for the stats report.. Lets hope its a trend and they keep it up…..

  81. Porsupah Ree says:

    Thanks for the posting – it’s very much appreciated. As you can understand, one of the worst things a company can do is not communicate, leaving people feeling stranded, leading to the kinds of “FIX IT!” pleas we continue to see. Hopefully, as time progresses, and the message from Linden Lab becomes clear – that server stability is a high priority – we’ll in turn see more understanding from such residents that indeed, LL is working actively to make the platform more stable, even if the task ahead is one of mammoth proportions, given the sheer scale involved, and the legacy code involved.

    As for refunds – it would, I feel, be reasonable to aim for 99% uptime on all sims not running beta code, with an SLA backing this up with suitable financial penalties. $200-300 is, after all, a fair chunk of change, on the scale of a small business net connection or reputable colo service, both of which would offer firm SLAs; LL’s surely been on the receiving end of such, judging by this entry, and others. (Lowering the monthly cost, whilst unrelated, would be highly welcome)

    Please continue this level of openness, and even expand upon it – technical level detail would surely be appreciated by the geekier contingent of residents, and probably make for some good discussions in the comments. ^_^

  82. Martin Magpie says:

    “What happened, and Why?
    So, on to why April was so tough: virtually every piece of mission-critical Second Life Grid infrastructure failed catastrophically at least once during the month.”

    Yes I realize this; and I also realize that my cable company refunds me when I cannot connect to the internet or place a call, or watch tv.

    Ian I appreciate your honestly, Thank you.

    Cat

  83. Martin Magpie says:

    @80 please keep in mind ppl have invested everywhere from a few dollars upwards of entire continents of simulators. It isn’t easy for everyone to just close box when their investment of time and real money is huge.

    Because of Bay City and stability issues I have no future plans to invest any further funds into Second Life or Linden Lab.

    Especially since there is no compensation of my losses as a result of the actions lack of actions on the part of Linden Lab.

    Cat

  84. Alvi Halderman says:

    April Was Really Awfull

  85. Verdana Klaar says:

    @35 (massively.com…) said: “Linden Lab loses 630,000 user-hours in April…”

    Dreadful… A new Katrina may be ? Or something like what’s happening right now in Burma ? Based on an average of – say – 100,000 to 300,000 unique logins a month, this is in the range of 2 to 6 hours. And i guess my random numbers are below reality. I suggest you buy a “massively” anti-nuclear shelter to avoid further catastrophical events of the same kind. (smile please).

    @37 (Umbra indirectly and some others) said : “I pay a mere $95 USD a month to participate in SL and I can safely say that I did NOT get good value for the money in April.”

    Ok, let’s raise the figure to 2 full days of downtime in April (just as a mere example, and for easier calculation sake). Linden Labs should then refund you some 6% of $US95, which makes 5$US out of which you should deduce something like 6% of the stipend as a premium account. Please LL, refund $US4.72 to Umbra. And everyone will be happy.

    Be sure i’ve nothing against you (all the more i do not know you), i just took your blog comment for replying, as it contains figures.

    @37 (Umbra) said: “I can’t imagine how some of the people who pay many times more than that feel.”

    This is my case, i pay more than $95 USD a month, and i mean really far much more. Well, still i do not complain for the very simple reason that i can bear it, that is… live for 1 or 2 days without SL. Do some here know that there is also a life at time outside “in-world” ?

    @37 (Umbra) said: “Sure, the TOS says we’re not entitled to anything, but I agree with many others that a refund of some kind would go a LONG way to creating some customer satisfaction”

    The problem is not the Terms of Service but the total impossibility for Linden Labs to calculate who can be refunded or not, on which basis, and for which amount. Even basic accounts that rent a small piece of land on an estate outside mainland areas should be refunded… but… by the estate owner. More than that, the guy that had earned 50L$ by just camping (on an estate owner or land owner parcel, just to remind – not on a Linden pose-ball – smile), should be refunded in case he used thoses 50L$ to buy an item somewhere and had a transaction failure for it. See how all that is so complex … and bureaucratic as well btw ? hmm ?

    _________________

    Now, on a different and quite more general level, i really wonder why some residents want to know the details on how Linden-Labs will/can solve the scalabitity and reliability issues of their systems’ architecure as a whole. As an SL USER, i frankly do not care “HOW”, i only care about a roadmap together with some committment.

    May be some of these demanding residents only want to work at Linden-Labs, who knows ? (smile)

    And in the end SL is not so badly performing. Could be better… but honestly we all can use it 29 days out of 30 a month. Which is not so bad for a GAME (/me gently winks when reading pompeous terms such as “mission-critical” or others…)

    Ciao ciao 🙂

  86. Aquarius Paravane says:

    Well as far as I can see today (May 10) is another awful day. Failed logins, failed tps, cant move when I do get somewhere. Even the crash reporter crashed. Anything being said about this on the status page? No.

    I’d like to be able to report these issues in a way that would get immediate attention, but I don’t think JIRA or a bug report are right for this kind of issue. Please provide a web link where people can report grid and sim issues, so you can see a real time dashboard of system health from the user perspective.

    I appreciate the honesty in stating the problems. It would be good to get some recognition in terms of tier refunds for lost time. Katt, I appreciate having someone whose job it is to interface with customers, but I suspect few people appreciate empty apologies.

    Although client issues are not the topic of this post, it’s worth mentioning that a lot of hours are also lost to CLIENT ISSUES THAT ARE NOT BEING FIXED. If the client is buggy and crashes all the time, that’s bad enough but it’s simply not acceptable for it to take down the computer it’s running on requiring you to restart via the power button. This started happening to me in 1.18, and it’s still happening on 1.20, where I get about 5 minutes uptime between hard restarts.

  87. interesting…..

    We just lost 20.000 USD (!) thanks to your decision of reducing the Sim Value, but no biggie….

    However, i think the additional losses because of the extreme spike of recent outages (asset server this, logins restricted that) are more than enough a reason for compensation, we lose customers and income just because of the fact the company we are supporting can`t provide us(or meet) the technical requirements. Nothing is done, no compensation is given either. Something is very wrong there and in the end we lose time, money and customers… and LL, so do you.. our customers are your residents.

  88. Verdana Klaar says:

    “We just lost 20.000 USD (!) ”

    1) No one forced you to invest that here

    “We lose customers and income just because of the fact the company we are supporting can`t provide us(or meet) the technical requirements”

    2) your are not supporting anything, you are using this platform to make what you have yourself decided to do (at least i hope so for your mental sake)

    NOW IT IS BECOMING BORING !

  89. Verdana Klaar says:

    My final word here at the attention of Sir Chester Gilks:

    As you inveted that amount of money (20K $US) this means :

    1) you have enough revenues to do so.

    2) you have taken a risk, and the risk happenned (i suggest you read about risk theory and methodologies, but it might be a bit beyond your money-minded self)

    3) you spoil SL as do so many middle-class selfish residents since 6 to 8 months, who do not consider SL a place for creating things for pleasure and dream (something like just a bit of freedom but ok.. are you even able to understand the concept?) but for small money making only

  90. Elle Hextall says:

    @89, i work for the same company and i jus dont get how you could possibly be THAT narrow minded, do you own 32 sims? i doubt it, do you put all your time and energy into a buisness that wouldnt even be there if content creators and designs thought oh its jus a game and went out to work for pixar or another random company where they could earn more? if it wasnt for people doin this there wouldnt be a SL for people to enjoy in so far as things to buy games to play ect as somone way back when decided to put effort into scripting, building and texturing items! if LL could make those things of the same standard why on earth would they be HIRING residents? im sure you are getting bored but no soo bored you didnt post aye? excuse me if im coming across as quite cross but statments reducing days and days of endless work jus so yes we can make a living but if people had said to phillip Linden oh dont waste your time making THAT get a REAL job, there would BE nothing, i dont mean anything personally but when LL groupies come on here to belittle GENUINE complaints, then please tell us where else to have our says cuz i sure as hell dont see them, after all the problems we are entitled to have a moan, damm we are human, you dont like it? well im sorry for you but if this were any other company we WOULD get some kind of better customer service than this, THANKYOU LL for admitting the problems so far but please dont stop, im already downgrading my account and am no longer putting money in as i have inventory loss, lag issues,tp issues,rez issues,sim crossing issues, you name it, this is a place for EVERYONE to have their say but put the pom poms away for a sec while people voice their GENUINE concerns and LET them.

  91. Martin Magpie says:

    Veranda that’s right it was a choice. However when ppl do invest great amounts in a product it says something about that product. Your right no one should invest any further. Good luck with your attitude but more so this product a year from now; when others like me simply do not continue to invest.

    No one is forcing you to read comments either.

    Cat

  92. Almadi Masala says:

    I don’t want a refund or compensation. I want my inventory back. If I can’t have that, then I want to know why the loss occurred and what is being done to ensure that the risk of such losses in the future will be reduced to a more reasonable level. I don’t want to go to all the trouble of rebuilding my inventory only to have it be destroyed again.

    I would also like to see a customer support system that actually responds in a meaningful way in a reasonable length of time — I mean in days rather than weeks or months. I would rather see LL put its resources into fixing the database issues and support system than into compensating me for past losses.

    In response to those who say, “If you don’t like it, just shut up and leave”:

    The reason I speak out is because I don’t want to have to leave. I love SL and still have hope that it can be fixed. But if that is going to happen, then LL needs to hear about what its users are experiencing and what they care about. If we all just shut up and “let LL do their work”, then their work will default to adding more bells and whistles to an already complex and unreliable system.

    I may in fact choose to leave, and maybe soon, but that will be a sad day for me. I don’t want to leave; I want SL to work. That is why I “complain”.

  93. Verdana Klaar says:

    Martin-Cat;

    I never said no one should invest. I said they have chosen to do so. That is quite different.

    And my name is not Veranda (lol). In french, a “veranda” is a room half-way between a glasshouse and a living-room. 🙂

  94. @90 yeah, i must have invented that.. who are you again? so you are supporting SL so well with your waste of commentspace for other people to actually comment on the topic.. if you even cared to read the blog entry (top of the site for fyi 😉 ) SL has major outages this month (and besides, what part of no biggie don`t you get?) i wasn`t asking for compensation for the 20k, i was asking compensation for the outages due to technical reasons (for all premium account residents for the duration of severe technical issues, as a sign of respect) keep offtopic as long as you like, i`m sure it is much appreciated.

    A service not working is a service not being used

    sincerely,
    Sir Chester Gilks 😉

  95. Martin Magpie says:

    🙂 Verdana sorry it’s early yet (sips coffee out on the veranda).
    Historically SL has been divided into two camps those that view it as a game and those that view it as a product/metaverse/marketing tool. I am sure it’s difficult for Linden Lab to keep the balance in Second Life. However I do think anyone who owns more than a simulator should be compensated for lack of stable service last month.

    🙂

  96. U M says:

    Once once in LL history they did a refund on a mess scale. The game was down for days. Now its inthe TOS that they are not liable for downtime losses ( there is some stepulation that they will refund due to some issues )………Think about it if the sellers ( which should be refunded ) can you think about all those money chair bots that will claim they have losses as well?

  97. Aquarius Paravane says:

    One more thing re land prices – the people in charge of national bank base rates have learned through their mistakes not to change them by more than 0.25% at a time. Dropping 10 sims a day into the system is a huge change in land availability which, has sent mainland prices plummeting, to just L$5 per square meter for bottom end mainland, the lowest I have seen in my time in SL. Huge price swings are destabilizing and destroy confidence. It would help to see the land supply managed conservatively.

  98. cosa nostra says:

    @ VERDANA

    listen, if my RL company doesnt supply on time the service or products … we must compensate to our customers ! right, this is how it works in life when ya are a professional business !

    I invested here myself in a sim for my 2 daughters, in this way I can communicate with them when I am abroad (90 % of my time), also I am helping them with SL business projects (small or big) so they learn something from there dad !

    now, if I am pumping 10 K into this service, a service which promotes to start ya own business in here, ref ;

    ‘From the moment you enter the World you’ll discover a vast digital continent, teeming with people, entertainment, experiences and opportunity. Once you’ve explored a bit, perhaps you’ll find a perfect parcel of land to build your house or business’

    I want this service to work, which is obvious, so please leave next time ya comments out of it, enjoy SL, and be thankfull that 90 % of the grid is thanks to the many investors who are working there asses of every day ! not LL, cause there job is to make sure the system is operational so we can access it 24/7.

    cosa

  99. AWM Mars says:

    With the hardware failures, you are insured right?

    When a supplier fails to carryout what they have contracted to, you do sue them and move on to another more proficcient supplier right?

    You do this for your customers yes?

  100. U M says:

    Well Indeed it was hardware failure on a very very large scale but whats worse is Maybe stats are going to be as bad as aprils…..June is coming soon and can you imagen what is going to happen when the under 18 years old kids are inthe game? Bots wil be 80% or more wil be online etc……AWM if you think and I know you know fromthe years you been inthe game. What the outcome will be……Lets just hope it never gets to a very bad meltdown then

    U sagi M usashi

  101. Jamie David says:

    I have read the posts and I was saddened that the first response from a linden was “No wild accusations”!! As this is the only way to get ones issues out in the open there are going to be many who are very passionate. Their frustration is high as is their enthusiasm. In that they take the effort to post at all is a blessing.

    Lets talk about the basics.

    As one who has run a large ISP and a country’s Telephone routing and billing system I understand the complexities of large systems. I also understand the customers needs. They don’t care what happens behind the scenes they just want it to work. Reasonable request. The four instances that are covered were all caused by poor planning. First one is blamed on an unnamed main provider who “delayed, botched, and again delayed the fix, extending what should have been a relatively brief outage for many hours”. Serious accusation!! Slinging the main blame on the provider shows weakness and vulnerabilities. LindenLab is at the mercy of outside providers, as are the residents.

    It would seem to me that LindenLab does not have a good relationship with this provider. It needs to be replaced with more than one. For the ISP I ran, when one of our providers failed us. (We had 4 for back up. 2 Fiber Cable 2 Satellite) there were heavy penalties. They kicked in after 10 minutes of outage or disruption to service. Our post paid, fixed line and corporate clients all are compensated out of this at same rate even though the loss of one would not cause network slow down. With SecondLife we are talking over 24 hours in a month, one would think that compensation to the residents who pay would be in order. A better tactic than posting in a blog that reaches less than 1% of the population.

    The others were internal issues. Hardware and code failure. Understandable if it was a BBS or some system that was for free. SecondLife costs some of us a fair bit of cash. Many are trying to make SecondLife a source of income as pushed by LindenLab. For that we expect forplanning, redundancy, back ups and contingency plans. What happens if? What do we do when ………? What happens if this provider who is so unreliable and uncaring gets knocked out. Are we all dead in the water? What is going to happen this summer when California is hit by their energy consumption and the rolling blackouts are started up again? Is there a back up plan to provide power for the hosting and the network?

    For over a year transaction failures have had a awful effect on the system, stability and the general perception of the resident. People are afraid to spend money. I who hardly spend anything have had in general 1 out of 5 purchasing transactions in SecondLife mess up requiring communication with the creator/seller to get the merchandise I have purchased. This is as fundamental as the stability of the Grid. It never seems to be addressed and is constantly swept under the carpet.

    With the data provided from LindenLab it shows that there is a lost 30 minutes per day on average to the Grid being down. This does not cover the time that the grid was all but unusable. This does not cover the time taken by residents to recover/replace what was lost. The realistic ammount is running at about 2 hours a day where SL is troublesome.

    Concierge, JIRA, SupportTicketing and all other forums of support are a waste in my humble oppinion and quite a few others. The system works fine there is just no one home who cares. File a report and weeks go by before a response, if one gets one. After one of the many grid crashes on one of our sims we noticed that 200 or so prims had gone missing. Mostly furniture, poseball systems and the animations inside the prims. I politely asked for a roll back to pre crash and was gruffly told that it was less than 10% of the prims on the sim so, no. Many hours of work and lindens to replace. How many others have had similar situations where treatment has caused them to give up?

    Another classic example. I was banned from posting here in the blog comments in December. Not one message was sent to me about what I had supposedly done wrong. Three months I posted polite messages in the support section and would call concierge once a month. Then one day it worked again. No message, no explanation. Same happened when LindenLab double charged me twice. Five months of fustration and passing of the buck by staff from one department to another. The wall around LindenLab is better built than of that which divided Germany. Residents want to help make SecondLife better but are frustrated by the gruff responces or silence they get back.

    SecondLife has a 4% retention of customers. 96% or more disappear after a week or so (14 million users only 443 thousand logged in over the last seven days). Between network issues and lack or support it is little wonder. The grid instability hampers comercial acceptance and now general growth. Enough passing blame, lets hear about solutions to making SecondLife more stable!

  102. Ann Otoole says:

    See the thing is, people on LL salaries are not losing any money. Unlike the residents/customers of SL/LL. So it is easy to say “I feel really bad for you given all these outages, especially the constant failing L$ transactions.”

    It would be another thing entirely is people did not get paid rl money for the complete time of any outage not the clear truthful fault of a 3rd party service provider.

    Tie salaries to performance and see how fast this thing gets fixed up.

    But thanks for the numbers. Nice to see some numbers. We need the outages for disruptions involving transactions too. To get this number you will need a real transaction manager system logging, etc. But then there wouldn’t be such a problem as often so over time nobody would care that much any more 🙂

  103. Sexy Partridge says:

    I agree with many of the post here…LL I think it’s time to step up to the plate and offer some sort of compensation. My over all sales were down around 45% for the month of April. I still have to pay my SL bills.

  104. Jamie David says:

    PS. We don’t need yet another web page to find out what is happening. Please put the stoppages and grid issues back on the main blog.

  105. ZenWarrior Fuosing says:

    Yes, it really *SUCKED* for us the past month, but has LL offered *anything* as recompense? No, I don’t think it has. And you wonder why so few pay for your so-called services? I, for one, certainly do not recommend “Sometimes [There’s} Life” to anyone anymore. I prefer my friends remain my friends. They will not if I suggest they even think about giving SL a test run.

    Might I suggest that the Lindens and LL read a few books on customer satisfaction and service quality? Oh, and while you’re at it, how about some books on programming?

    Second Life really sucks right now–really sucks. And again, how about the same percentage of recompense for residents that SL was down–for any given month? Or, does LL insists on just taking our money, telling us how bad things are for us, and then still running like crazy to the bank with our looted cash?

    Is P.T. Barnum running LL and seeing nothing but suckers instead of “paying customers?”

  106. Tegg B says:

    Great now we have Blue UM and Green UM :/

    Thanks Ian & Katt please tolerate the uncouth posters here, who obviously have some upsetnes.
    Many of us realise you are doing your best and replying to residents at times is like putting your head in a lions cage.

  107. ZenWarrior Fuosing says:

    Katt, be quiet. Your comments only make it clearer just how clueless LL is about all of this. We are not at all happy. BTW, what business school classes in customer satisfaction and service quality have you had? Apparently, not a one at any of the business schools at which I’ve taught.

    Words from LL mean nothing to us now. We want a grid which works. If you can’t tell us precisely how and when that will obtain, then please refrain from telling us how to react to all this mess and what to say here.

    Jeesh, are all LL employees so clueless about customer relations? Apparently so. 😦

  108. U M says:

    “107 ZenWarrior Fuosing Says:

    May 10th, 2008 at 6:50 AM
    Yes, it really *SUCKED* for us the past month, but has LL offered *anything* as recompense?”

    Frankly speaking its going to get a little worse then better anytime soon. Last night was just a small possiblity of what could happen if there is anymore hardware failures……..

  109. Zena Juran says:

    I would prefer more emphasis on the Quality of the Grid; I can accept a downtime of roughly 2%. The abiltity to create, communicate, and socialize in an environment that provides a VISUAL as well as AUDITORY experience has been my interest in SL. Now, with the degradation of the Quality of the Grid, it is more frustrating to stay in SL once I am logged in than it is just trying to log in at all.

  110. How it looked from this side:

    You: Update! New features ahead! Tastes great, less crashing! Woot!

    Us: Crash, disaster, business down, *@#&^%(@#.

  111. Heh says:

    Thanks for the honesty LL. I appreciate you letting us know what’s going on. This is funny reading. FIRST, some folks (or maybe just one?) made mention “You don’t even say you’re sorry…” SO Katt comes back and apologizes, then everyone (or maybe just a few?) jumps on her and says , “Don’t say you’re sorry…”

    Sheesh.

    For the “we pay your salaries” statement – please give it a rest. You’re all beginning to sound like yet another bad MTV video playing over and over again. Tell that statement to a police officer the next time you get pulled over for a ticket and see what happens. Make sure you write your experience in the next blog cuz I really wanna know about since “you pay his/her salary.”

    For the Compensation Crowd – you’re never gonna get your lost back so give it a rest too. IF LL were to give ys some kind of compensation, you all would STILL be here demanding something else – why? Because people like you are never satisfied with WHATEVER compensation is given. WHY? Because it would not be enough. WHY? Because you should’ve, could’ve would’ve had more.

    Sheesh.

    My experience here as a sim owner/merchant has been as bad as all of you, but unless any of you “tech wizards” – like Ann O’Toole up there that seems to know EXACTLY how LL could make this world better EVERY SINGLE TIME SHE POSTS – get off your butts and go work for LL to make this OUR WORLD – then for goodness sake GIVE IT A REST.

  112. Promise Bracken says:

    Well. I love the detail.

    I have to disagree with the poster who stated that SL had become all about commerce. The sample of posters here seems to represent a population of merchants. I don’t see many posts by users who are not in business in SL.

    The number of people making money on SL–which is a number something LL promotes, btw, should indicate that the time may be here–or will soon be here when the TOS should be changed in order to better support commerce.

    ATM we have no right to ask for our money back–this is an “as is” service, and we have to live with that. But perhaps LL might be convinced to create a parallel system where transactions and uptime are guaranteed to be stable. I’m not talking about a “SLII,” but a viewer/database which has more backup/redundancy, at perhaps a small additional cost.

  113. Pennysavedpennyearned says:

    OMG, ‘compensate me’? Are you kidding? Let’s see, even if you pay 300 dollars a month, and your downtime was 1.9%, you’re crying over less than 6 dollars? As for ‘loss of sales’, there’s no way to determine hard numbers on that. You’d have to have detailed records of your sales, and comparison charts to other vendors, you’d have to be able to discount any other reason for a change in sales, (other vendors big sale, a lack of new products, the world ecomonic status, etc), and even then, unless everyone one of you is one of the very few that make more than a $1000 a month, , you’re still talking about a few dollars.

    The key word here is ‘entitlement’, which is a sickness in our society.

    Like any other environment, there are going to be times when ‘acts of god’, are going to effect your sales. If you had a physical store, there would be times when weather might affect your sales, or, let’s say the city is working on the streets because the sewage pipes broke, or, well you get my point. Sure, it’s inconvenient, and sometimes, yes, you could even point fingers to say, but sometimes, things just happen and it will effect your business and you suck it up, that’s just part of the risks of running your own business.

  114. U M says:

    What I wonder is if those commerical islands that pay much money per month, do they get refunds if the services randered are beloww useable levels??????

  115. I expressed my unhappiness both to “concierge” service and regular support over the past few months. To say LL didn’t “get it” is an understatement. In the end, I’ve voted with my USD $ and sold the two remaining SIMs I held, ending about $490/month in PI fees. I think the only thing unhappy residents can do if they’re unhappy with LL’s service is to sell their SIMs and neither buy nor rent land. Perhaps if the customers stopped pumping real dollars into the LL economy, maybe LL would understand that they have to run SL like a business, provide real SLA’s for Tier fees, ensure less buggy viewers (instead of leaving viewer fixes to the open source community), and compensate paying users for loss of service. Right now, despite that there are some good Lindens working hard in the customer service group, LL’s management doesn’t seem to understand the basic business ethic of providing customers what they pay for.

  116. Carrie Grant says:

    Katt, as our new Linden to help customers with problems and become the new ‘bridge’ between the higher Lindens hidden in the Ivory tower and us…joe public, I have to say you don’t seem to be doing a very good job so far. The idea is to answer points not to ignore them and add worthless comments here and there. The issue of compensation has been asked over and over but has never been answered. Can you explain why we have to pay for user fees, tier fees, classified advert fees, land advertising fees, plus suffer the loss of sales and have to spend a miserable month dealing with customers complaints and you offer nothing back to your customers in return? I look forward to your reply as I’m sure you’ll do your best to provide us with valuable insight on this…

  117. Merchant says:

    Ok, I’m not a sim owner and I have been losing a lot of sales from all the vendors I have around SL. I would like to know if the sim owner will be compensating ME for the lost of sales I’ve had in the recent months.

    My sales are down 40% all over SL. So, you all DO get a compensation from Linden Labs, will all you sim owners THEN compensate US for paying YOU tiers every month? Will you all tell US that you had gotten some kind of compensation?

    I am very interested to hear from all of you sim owners.

  118. U M says:

    I love you, Ian. I know I’ve alienated all the other Lindens with all my nasty comments in broken english, but you, Ian, are special. You make all my dreams come true, maybe even make me Japanese Liason some day.

    I hate those other mean Lindens that banned me from this blog, so now I have to post using only my initials. Maybe you can fix that for me too, Ian? I give you big bunny kisses!

    U sagi M usashi

  119. Verdana Klaar says:

    @115 Penny said

    ” OMG, ‘compensate me’? Are you kidding? Let’s see, even if you pay 300 dollars a month, and your downtime was 1.9%, you’re crying over less than 6 dollars? As for ‘loss of sales’, there’s no way to determine hard numbers on that. You’d have to have detailed records of your sales, and comparison charts to other vendors, you’d have to be able to discount any other reason for a change in sales, (other vendors big sale, a lack of new products, the world ecomonic status, etc), and even then, unless everyone one of you is one of the very few that make more than a $1000 a month, , you’re still talking about a few dollars”

    Fully agree, and i tried to express the same. Am afraid you can tell and argue common sense either way you want, they (those so-called buninessmen – or women, -> Monty Python ‘Life of Brian’ – smile) will not take it into consideration.

    err… Monty Python , no. Monthly Pythons 🙂

  120. Jahar Aabye says:

    On a more constructive note:

    To what extent have the connections (or difficulties therein) between the colocations in SF and Dallas played a role in these problems? Is having to communicate between the two facilities creating any bottlenecks, and/or are you seeing more connection problems/packet loss there?

    Also, it appears that the Central Database and Asset Storage Clusters seem to be persistent issues. Is this a case where usage is outpacing capacity? Is it simply a scalability issue, a need for more hardware, or is it a situation in which a new system or new protocols will need to be implemented for future growth?

  121. JayR Cela says:

    I wish you all the best of luck

    JayR Cela :_)

  122. Azaiya Aeon says:

    Thank you for the information and openness, allowing comments, and for looking to solutions to improve a system that is far more complex that I could possibly understand.

    To the people with lost sales: I’m not sure that’s something you can blame LL for completely. At least in the US, the economy is in a downward spiral. People have much less disposable income to spend on things like virtual items.

  123. Kaliach Loudon says:

    Lets start with saying I’m getting a rather tingly happy feeling in my tummy…
    You are finally giving me the impression you really ARE addressing my mantra! “Address stability and capacity!” Good job there, this post was what I really wanted to hear for a long time.

    However I will make some critical side notes here:
    Even thought the “real” technical outages may have been only 1.9% the secondary effects and most of all the near loss of service have been much worse imho.
    Lets face it a lot of people just avoid or at least dread the weekends now as you will quite often be barely capable to do anything as all crawls to a near halt.
    As such I get figures more in the category of nearing 25% loss of service especially on the weekends. So it would be nice if Linden Labs figured out a way to measure out grid usability.

    But all in all I’m happy seeing things get addressed and seem to progress well, keep up the good work and most of all keep us posted.

  124. U M says:

    122 said………Also, it appears that the Central Database and Asset Storage Clusters seem to be persistent issues. Is this a case where usage is outpacing capacity? Is it simply a scalability issue, a need for more hardware, or is it a situation in which a new system or new protocols will need to be implemented for future growth?

    More and better hardware is what many have ben calling for for a very long time. They do add better systems but at times i wonder how good those parts are………Failure on the scale we had yesterday was a nightmare……..Await its not even June yet lets hope it never reaches any worse then what we had these past few weeks

  125. Taft Worsley says:

    It not the point you get 6$ back for compensation – its the point it was offered in good faith for acknowledge failures. There is no good faith here – you pay and can’t play — that to me seems like a con game.

    This compensation gesture is not about recovering total loses – its about feeling like a valued customer of a service, you enjoy using and spending you hard earn rl money on – its not about the money in most cases – more the principal!

    And not to mention when you add 6$ up to lets just say 1/3 of the online user base – thats nice chunk of change out of the pocket books of all the management and investors riding high on this bill of goods many of use have been sold with lines like ” Your World – Your Imagination”

    Hmm maybe truth in advertising would have been nice from day one and so many so called suckers by you brown nose tarts – wouldn’t have invested money. But i guess IBM dropping millions on LL as an investment was their bad too.

    Yet Philp is still out there in his shiny clown car, pushing this technology even to this day as a business platform – See last weeks BBC interview. Business platform for who, I know value hosts with better up time then LL has seen in the past – so spare us your lecture about spending money and why compensation isn’t going to matter – its matters when its 20-30K users at $6 bucks a head!

  126. Verdana Klaar says:

    @100 (good number) said “@ VERDANA

    listen, if my RL company doesnt supply on time the service or products … we must compensate to our customers ! right, this is how it works in life when ya are a professional business !”

    just two things.

    1) Never (never) begin a blog comment with “listen”, that is far too italian – milanese espacially – macho amusement when answering a french person (smile please)

    2) i am quite sure your company at corporate level has this kind of image, It’s always one different thing when coming to negotiations at global sourcing “win/win” reality ($$$). Well… not “your” company as you state, but in any case all large companies (smile please)

    @100 (good number) said “@ VERDANA I invested here myself in a sim for my 2 daughters, in this way I can communicate with them when I am abroad (90 % of my time), also I am helping them with SL business projects (small or big) so they learn something from there dad !

    awwww:( Please stop here, i am only tears and sobs now… con la mandolina della comedia dell’arte (more or less). My advice is you should rent or sell that land to them so that your educational duty is refunded. (smile please)

  127. U M says:

    $6 refund? how about it be based on how much one pays per month ( it will never happen) but if it did it should be a % of what one pays they more you pay the more you get back.

  128. Phen Geiger says:

    For the less technically sophisticated, what is the difference between the “cloud computing systems” used by Google which virtually never crash, and the Linden Systems? Please respond to my email address as well as on this site. garrison@rockisland.com

  129. Neferon Nevadan says:

    All i know is i keep paying classified ads .. but SL issues have destroyed my sales. Sigh. I really don’t want to shut down my business and sell off … but it’s a real possibility unless things change.

  130. Jamie David says:

    Why shouldn’t there be compensation to those who pay? I am fairly sure that LindenLab was compensated for the outages. Why not the paying customers?

    A discount to all paying residents in their bill would have been a nice gesture from LindenLab to their committed customers. It is not the amount but the principle. “We feel your pain and we care”. A small token would be well received even though it doesn’t nearly cover the real financial and time loss.

    Currently SecondLife has a monopoly and treats it’s residents so. Those who have issues are berated, brushed off and banned to Cyberia. A few who enjoy associations with insiders or are new and enthused by it all, support the cause blindly and feel that they will be looked after if trouble comes around the corner for all their wonderful supporting comments.

    The system should be fair and equal to all evenly. Not just to the friends of a Linden who show up in so many of these blog postings filling up the limited postings with happy we love you fluff.

    I hear that in the near future the current system of traffic is to be removed. Popular places to be replaced by a “Linden Suggested Places”. Which to me equals insiders and those who leave the apple on the lindens desk.

  131. pantaiputih korobase says:

    well, 1.9% outage time sounds little. Unfortunately, the ‘felt’ outage time is pretty much higher and, LL, did you analyse how many percent of the outage time falls in to US and/or Europe prime time? Can we have this figures, too?

  132. Sling Trebuchet says:

    A second comment to emphasise something from my original comment that was separately mentioned by others (e.g. @55 Craig Altman , @67 Deira Llanfair)

    Transaction failures are perhaps the most damaging outcome of Grid “brown-outs”.
    I’m not a merchant myself. I just fork over 1/2 sim of tier every month because I’m a masochist.
    However I know from Forum threads and the experiences of friends (both merchants and buyers) that products not being delivered after payment are pure poison to community and to confidence.

    Handing over money and not getting the goods is a very personal and immediate experience. Most people react badly, with a sense of outrage.
    I’ve seen more than a few Forum threads in which people assume that they have been cheated and ignored. They might try to send strongly-worded IMs to the vendor without realising that the vendors IMs got capped within minutes of the brown-out starting.
    As for the vendor, if they get even clear polite notecards or IMs, it’s a major hassle to suddenly have to deal with transaction lookups, inventory giving – and the inevitable opportunists.
    If a vendor gets a stream of flames, do they really want that grief on a regular basis? It’s bad enough for everyone when things go borky – but abuse on top of it , with no clear end in sight, must make many want to really give up.

    I’ve been lucky. All I’ve ever lost in a failed transaction was a L$600 pair of shoes. However I’ve noted the very keenly felt sense of hurt caused to both buyers and sellers. I would say that the emotional cost to both buyers and seller massively outweighs any financial cost.

    It looks like we’re into brown-outs at peak times for some time to come. The blog posting http://blog.secondlife.com/2007/02/16/contingency-measures-to-ensure-service-as-second-life-grows/ is just a black museum exhibit I think.

    That being so, could we not have something along the lines of @8 to stop transactions when the grid becomes unstable?

  133. Rayy Giles says:

    Those 4 things you mentioned have been going on for yrs and you have never stopped with all the stupid fluf updates that cause these issues. For yrs now your customer base has been screaming for stability as you give us non important updates. Avies are still Avies, Prims are still Prims and so on. Whats so hard about sticking with the main issuses and fix them. I cant speak for most but i dont give a damn what the sun looks like when it sets if we are crashing because of these fluf updates that cause it.

  134. U M says:

    It use to be worse with the updates remeber every wednesday I believe it was……..But now a days not weekly downtime anymore but more problems and issues……..

  135. toybodacho Ireland says:

    @36 Katt: Thanks, I hadn’t noticed those feeds. What about testing upgrades for a few months before applying them to the whole grid? I’m sure there are a lot of sim owners who would volunteer their regions, especially if Linden Labs were to offer them some type of incentive.

  136. Christos Atlantis says:

    Thanks for the update and dialogue Lindens, let’s all keep our fingers crossed that we will have less mishapes, I know I have had a few in my life, and they are the worse thing in the world.

  137. Alisha says:

    Very nice post Ian. Downtime will happen, but can you please elaborate on solutions to these ongoing problems. It is all well and good knowing what the problems are, but we need solutions.

    Verdana, please check Your attitude as it sounds like You are telling everyone(with a problem) to quit. Thanks

  138. Celina Kamachi says:

    Thank you for all information about that, we all felt it was slow in the last weeks so I hope with the others that the core problem and the asset server problems could be fixed to have a more stable SecondLife. I still enjoy my SL every day.

  139. Darek Deluca says:

    Island “sales” (read setup fees + recurring charges) are way up.

    LL cash flow is doing fine.

    P.T. Barnum was right!

  140. Rayy Giles says:

    @137 Yes that would be nice if they tested on the beta grid for a long time but Lindens dont police beta grid as they should. You have jerks in there just dropping prims everywhere and most duplicate. I work alot in beta because of my SL business and most of the time its impossible because Lindens dont apply the same conduct rules in beta as in main grid. I feel they should but for as long as i have used beta they haven’t. The problem is Lindens think of $$$ first and not their client base. Like non paying accounts should have a limit on inventory which ive said for a long time. Paying accounts pay for their inventory and lose it just as freebie accounts do, that picture isnt right. Theres so many things Lindens could do if they would listen to paying clients and not freebie accounts. Paying accounts feeling like when you go visit a store to get something and someone calls on the phone to ask a question gets a higher priority then you do and your the 1 that drove there. I dont have a problem with free accounts just dont give them the same rights as paying customer do. Think Lindens, that would get thoses freebie accounts to cross over to paying customers. Then with more $$ in your pocket we might as a community get better servers and etc. Thats just 1 idea that makes good business sense.

  141. Martin Magpie says:

    Obviously ppl are upset, I recognize that; I can only hope that Linden Lab does also. Sounds to me like the system is buckling under the strain. However I have read that IBM is doing really well with their open sim even breaking records as far as prims go. Last count they were able to have a stable product with aprox. 45k prims per simulator. Maybe IBM isn’t all bad….

    Ya dig what I am saying LL. You have a small percentage of the population very vocal. You are kidding yourselves if you think the rest of the grid does not share at least in part the comments above.

    All you have to do is long in and ask ppl what they think.
    Customer loyalty is at an all time low, customer satisfaction and grid stability go hand in hand. Failed.

    If you are trying to push your customers out the door, then you are doing a great job.

    Cat

  142. U M says:

    “All you have to do is long in and ask ppl what they think.
    Customer loyalty is at an all time low, customer satisfaction and grid stability go hand in hand. Failed.”

    Didnt they do that a long time ago????? What happened ??????

  143. Cocoanut Koala says:

    If not tier compensation, at least compensation for Classifieds.

    LL refunded Classifieds (actually, gave a free week) once before, as well they should have, and it didn’t cost them a penny.

    And they can do it again, this time for the entire month of April. To make it simpler, refund those Lindens paid for classified for the month of April, rather than giving free weeks (since people may change the amount they want to pay for classified for those upcoming weeks).

    Won’t cost LL a red cent. Not one red cent.

    And would provide a wonderful boost in customer satisfaction, and even provide a nice little boost for the economy.

    coco

  144. Angelas Enoch says:

    Thank you for the update and Katt, keep working hard as you do to make SL Better. At least something is being done about it. I know a lot of Old Timers on SL that just look at it as they say, “Birthing Pains.”

    Keep moving forward everyone..that is all you can do..

  145. Taft Worsley says:

    I guess the cat has Katt’s Tongue since nothing has been posted since late yesterday – which means 1. either she doesn’t work on weekend like the rest of the lindens and hence why weekends are so bad or 2. like i said when she was hired for her position – another waste of lindens money, either way you look at it the silence from your so called press relation person is a tell tale truth about how worthless that postion was to begin with and how much you care about the userbase at large. Long live the opensim project — i guess LL doens’t realize for ever peoson who leave and goes to open sim — is a person surely never to return — oh wait i know you some how thing your going to control the virtual meta universe – now thats funny qwhen you can bearly keep 60k in users online.

    Katt no offense in my ?? but really what is your job for LL, because it seems you really don’t have anything to say or the authority to correct any issues — so explain to us the meaning of your pointless job — oh i know you got the AV icons working for people who fail to load an image — whhot — good work there

  146. Rayy Giles says:

    ROFL @ 146, birthing Pains, are you kidding me, I’m an old timer to SL and that baby should of been born yrs ago. Stop sucking up or acting like a friend, if you were a true friend you would tell them what you think. SL has an aweful customer satisfaction problem. Return of classifieds paid would be nice to compensate.

  147. Cliff Dieffenbach says:

    Growing pains. Everything is a lesson learned.

  148. Martin Magpie says:

    @146
    aka Catherine Cotton 😉 Join Date: 3/18/03

    More like tired of watching SL die a slow and painful death.

Comments are closed.