November 2007 Service Quality Metrics posted

Hello everyone, please note that our Service Quality Metrics through November have been posted at our Secondlifegrid.net site, along with the detailed data in Excel, OpenDocument, and Google Docs formats.

Overview: The Service Outage chart for November reflects the problems that we suffered in the latter two weeks of the month, with over 1.2% unplanned outage, though the Planned outage metric is still excellent, at around 0.3%.

Otherwise good: Viewer crashrates are slightly down (good), viewer framerates are slightly up (also good), and for the region FPS data, while the percentage of regions impacted is up (not so good) slightly for the moderately busy regions, it does remains steadily less than 0.5% for the heavily overloaded servers (good).

Fine Detail:Β  there is a sourcing issue with the details of viewer crash data past Nov 20th specifically with the counts of sessions “disconnected from a region”, but the high-level average on the chart is valid across the month and we do have the history of the actual viewer crashes (an area of strong focus for us in Engineering) during this time.

Related: Yes, I will be posting the updated economic stats under a separate blog post today.

About Meta Linden

Data Warehousing and Metrics Analysis for the world of Second Life. A Linden Lab employee in California.
This entry was posted in Service Metrics. Bookmark the permalink.

60 Responses to November 2007 Service Quality Metrics posted

  1. Kenny Stringer says:

    Be nice to See Auctions as promished ! land still to high !!

  2. Laura18 Streeter says:

    be even nicer to no longer see
    Login failed.
    Cannot connect to version manager
    upon login….

  3. Meade Paravane says:

    TY, Meta!!

    /me wouldn’t mind seeing a Windows/Linux/Mac breakdown, though those kinds of stats aren’t what this blog post is about.. πŸ™‚

  4. Montana Corleone says:

    Good, you’ve had quite a while to massage the econ stats…

  5. Brett Finsbury says:

    My viewer crash rates are way up since I switched to a new computer with Vista installed.
    Windows Vista Home Basic
    AMD Athion 64 Processor 3800+
    4GB of ram
    Nvidia GeForce 6600 GT with 512 MB ram on board
    Running concast high speed internet.
    Clear my cache numerous times a day defrag weekly all my drivers are all up to date.
    I get the black screen and second life not responding freeze ups hourly.
    My system is far more than enough needed to run SL.
    Other people who I talk to about this with Vista have the exact same problems that I am having. Is the second life code not compatable with Vista?
    May Have to dust off the old dinosuar computer and use it again. It was slow but at least it worked with SL.

  6. Gawd…

    Okay whiners… the Metrics are the OVERALL stats. It’s not about YOU.

    Just because system outages might be down doesn’t mean it’s a lie just because YOU have login failures or unusually frequent crash rates.

    The majority of responses to this blog are an absolute hoot. But, I suppose the squeakiest wheels are the ones heard the loudest. The problem is that annoying squeak.

    META: Thank you for your hard, thankless work. I appreciate it anyway. πŸ™‚

  7. Brett Finsbury says:

    and if you dont say some thing it doesnt get fixed. Do you have a point ari or do you just like to hear yourself speak?

  8. LOL – use JIRA or it WON’T get fixed.
    yes, Lindens read the blog. But they (and if I were a Linden) just brease over the whining and look for the USEFUL feedback.

    So, keep on whining.
    “Be heard.”

  9. BluesSweet Tomorrow says:

    I gave up sending crash reports.. figured i would get tossed out from them all lol.. Brand new system yet i crash upon entry most times. Most days i give up trying as i lockup so often.
    I run my cleanup/clean cache/defrag.
    Also on cable here.

    Intel Core 2 Series Processor (2200 MHz)
    GeForce 8400 GS/PCI/SSE2
    OpenGL Version: 2.1.1
    Memory: 2047 MB

    Seems no one has answers to why so many of us are crashing out here.

    oh and OS is winxp home.

  10. Okay – my last post here – don’t want to take up limited talkback slots…

    @ BlueSweet – you mention this: “GeForce 8400 GS/PCI/SSE2”

    Is that an ATI card?

    I have an older nVidia – and my system was stable as rock-solid as it could be. NEVER crashed or anything. Just last week – bought a new video card. Used that ‘framerate benchmarking’ chart they released a short while ago and purchased based off that.

    I installed the card (an ATI) – and wow – performance was awesome – but then BOOM.
    System ran GREAT – unless I was running SL viewer.
    I mean HARD CRASHES – Blue Screen of death and all of it (Windows XP Pro – current patches.) I could be in SL for as little as 30-seconds up to 30-minutes (as long as I didn’t move) before a sudden soft crash (application) or hard restart or ‘black-screen’ or blue screen of death. it seemed random.

    What I am getting at here is this: there are real hardware (driver) issues that are the causes of 95% of these problems people have with SL crashing all the time.

    NO – I AM NOT saying it is your fault or your machine’s fault.
    It *IS* Linden Lab’s fault because they are not ‘properly’ supporting the appropriate drivers and protocols – and I’m referring to CURRENT drivers for the most-used chipsets (ATI and INTEL).

    So, yes – the problems are on your end – your hardware and drivers, but it’s Linden Lab that are not supporting those hardware and drivers – and they should.

    So, I went back to my 4-year old nVidia video card.
    Suddenly – the crashing has stopped.
    This past Saturday – logged in at 10AM – spent all day without a single crash (a relog or two because of the memory leaks). 😦

    So – nVidia card = my system is stable – never crashes SL viewer.
    ATI card (brand new – year-old model) = constant crashing – even blue screens of death – and ONLY in Second Life viewer.
    Remove ATI card – put old nVidia back in there – suddenly rock-solid stable again.

    Okay – YOU do the ‘math’.

    1+1 will equal 2 no matter how you slice it.

    Look – for everyone having problems – I feel for you. That was a miserable week for me. All I’m saying it – use the JIRA and tell Linden Lab to properly support current graphic drivers (as I suspect that’s where 95% of most people’s problems are occurring).

    Proclaiming your difficulties on the blog is just a waste of space. At least get a little mileage for your effort. In the JIRA, it’s logged and tracked – even if it might be 10 or 15 years before an engineer takes it seriously.

    Oh – drop a complaint notecard to Philip Linden, too. heheheh.

  11. Not a Linden Fan says:

    why use Jira, its another flawed part of SL. Only things that even get looked at there are one from the clique. If your not one of them they just shut it down as resolved. Like there ticket system. Takes 6 weeks to get a response and its always some BS auto response that is no help at all.
    SL could be a good game, but the priorities of what to do are so mixed up. All they care about is drawing more people for quick cash but don’t seem to care about the long term residents. Prove me wrong LL, stop with the little bs programs and windlight and weekly tips and put all resources in to fixing this thing.

  12. MaxDesoto says:

    Meta,
    You wrote: “while the percentage of regions impacted is up (not so good) slightly for the moderately busy regions, it does remains steadily less than 0.5% for the heavily overloaded servers (good)”

    What do you mean when you say “heavily overloaded servers”? Asset Servers? The server operating the “moderately busy reegions”? Can you enlighten us on where the problem actually lies?

    @Ari Ummm, a Geforce video card, is by definition an NVidia/AMD card, not an ATI. Your advice would be more helpful if you were not starting from a faulty premise.

  13. Doris Haller says:

    I see no change. It stays as it is. It is not getting worse, and not getting better.

    That’s bad.

    BTW, creating JIRA entries does not mean it gets fixed…. it may just get ignored on a much higher level…

  14. Chris says:

    Thank you Meta and concur with Montana: would love to see the new econ stats when there’s time.

    @Ari – I didn’t read a whine til yours at 6, I even saw a thank you @3 which you said didn’t exist. You did have something useful re: the card and drivers, but you can use a little tact in delivering it and maybe use less of the posts to say it. Peace

  15. Renee Faulds says:

    but……

    1. Does this include “Crash Logger” crashes?
    2. Crash Logger only functions at the BEST 50% of the time.

    and the way December is going your going to really have to bend and manipulate these figures to show that it is holding steady as it is terrible now.

  16. Pingback: Latest Second Life service metrics available : The Metaverse Journal - Australia’s Virtual World News Service

  17. DadOZ says:

    Hmm… ty for the stats. I wonder if LL can (or do) get more technical info from crash reports. Im talking the kind of info the “Steam” application (Counterstrike, HL, etc) obtains when they carry out user surveys.. e.g. full hardware and software info, various related driver info, DX and OpenGL files version info. Hope the crash reports contain that level of detail. Some folk seem to point SL crashs to ATI cards and not to NVidia, etc.
    I complain like hell (TO MY SELF)… running a P4, 9800Pro and 1GB of memory its kinda masochistic…. oh well….

    /me runs back to the nearest SL dungeon

    TY LL, Happy (problem free) holidays all. πŸ™‚

  18. Elissa Bristol says:

    LOL Ari, just another of those priveleges to get her griefing AR’s responded too and probably because she is big business. Ari, I suggest you get some mouthwash πŸ˜›

  19. Dominique Dench says:

    Oh man!!! you talk about sl for november is very stable, but how many peoples have the same problem, every days, i have duo core 2 , dont no if sl support duo core 2 but every of my friend have duo core 2 to, and have the same problem, sorry my english not very good speak more french, beh!!! lindens please fix this , dont no how many time have this problem, but so long about 2-3month.

    so dont talk about my config, am engineer of sound so, i no what is a good computer our not, i run everygame on hight np ,so you no some peoples talk about MEMORY LEAK! probably sl have. specialy builder, modder and texturer have this stupid problem, each time moving camera ( alt-mouse) have many program freeze ( program not responding) thats very frustrating for every builder, how to fix this? thats the problem number 1 for me and for some of my friend build everydays in sl, program freeze every 5 min if i see world into camera mode (alt vs mouse move) need to fix this please, am sure this is a problem about memory leak, cause sl doest support duo core 2 fully probably.

    If you not understand corectly what i say here, i will try to talk one of my friend talking good english to seen to your sl team , a mail about thats big problem, appears every days. sl doest responding all the time, sl freeze and sometime crashed.

  20. Kimo Junot says:

    Viewer Crash rates are down????? how so???? lol just about everyone I encounter in SL seems to be crashing all the time

  21. Fluf Fredriksson says:

    Am hoping the 1.18.6.1RC crash logger will help bring a better picture of true crash stats. I’m usually good for 3 crashes an hour at the moment, sometimes more, sometimes less.
    The only person I know not crashing regularly runs an old pre-new search version of the client. Guess I’m just unlucky though. Some people seem to manage to stay online for hours. As did I back around May-July.
    Shrugs.

  22. Luciftias Neurocam says:

    Viewer Crash rates are down????? how so???? lol just about everyone I encounter in SL seems to be crashing all the time

    I’m currently teaching a course on statistics. When we get to the section about mean, median, and mode, you are welcome to sit in.

  23. Dallas Seaton says:

    “The Service Outage chart for November reflects the problems that we suffered in the latter two weeks of the month, with over 1.2% unplanned outage…”

    Yes, the unplanned outages went up by OVER 600% in November from the month before, and based on what we’re seeing so far, December won’t be any better. So much for improving stability, it appears.

    “Viewer crashrates are slightly down (good)…”

    Yes, they’re down VERY slightly, but its still interesting to note that what you’re calling “good” is that over 20%, or in excess of 1 in every 5 logins ends in a crash! Amazingly damning, in my opinion, in terms of anything even remotely approaching “stability” in SL.

    As for those of you talking about submitting crash reports and wondering if they’re used – think about it for a moment. There are upwards of concurrent 50,000 users per day who are logged into SL. I don’t know what the total number of logins is, but my guess is through a 24 hour day, probably at LEAST 4 times the concurrent user peak. So lets call it 200,000 logins per day most likely. Now note that over 20% – or over 40 THOUSAND per day end up in crashes. So lets say that only 1 in 4 sends a crash report, that says LL is inundated with 10 THOUSAND detailed crash reports per day or over a quarter of a MILLION per month! Now tell me, how many of those tens and hundreds of thousands of crash reports do you think ever get even glanced at? πŸ™‚

  24. Eclaire Looming says:

    well this is all very interesting but is sl having database problems right now? just a question and i think it is a fair one

  25. Fenleab says:

    I’d rather see the, “You cannot log in till 00:00:00 PM” or w/e notice, fricken most annoying thing evar

  26. Xi Taurog says:

    @the whiners: DO submit a JIRA issue for your issues – LL can’t fix your poor attitudes, but I submitted a JIRA recently for a big problem and it was fixed within days. I’m nobody to them, but I reported a valid, reproducible problem, and it was quickly acted on.

    BE SPECIFIC – whining about stuff not working and being vague about it doesn’t help anyone. Think about it – if they don’t know what causes the problem, and see it happen for themselves, how can they learn enough about it to fix it? Yes, I have my share of odd random happenings as well, but I don’t expect them to find a fix for it immediately if I can’t provide enough details so they can reproduce it.

    Be constructive with your feedback. πŸ˜‰

  27. Rascal Ratelle says:

    New security updates fir Quicktime has been released, this update is recommended. There is also an update for windows.

    http://www.apple.com/search/downloads/?q=Quicktime+security+updates

  28. Dragger Lok says:

    New required windlight client jams up solid- not a chance of logging in- Dell XPS720 quad core nvidia 8800 ultra, should’nt be any issues with my system-

  29. U M says:

    Ok beta is beta when its screamed at us at the start. But you tell me you people spent time on this client? Shakeshead :/ I understand it takes time and effect for you to get things correct but OMG what in the world is this release?!

  30. Minni Trotter says:

    Great job!!!! WOW as you proclaim partial success in improving SL for us all I find it virtually unplayable. Everything is gray, avatars take 10 minutes to even be visible, attachments wont load, inventory shows 0 items.

    Clear cache/relog, I am ruthed for about 20 minutes as of this post now… Inventory still gone, world still gray, etc.

    Maybe is just people in my part of the country (usa) huh?

  31. mari says:

    Please, Crashes down- and it is fine. NO NO NO. This is not a stable system! Can you imagine this being a bank – your money is gone, we are looking for it. Our homepage is gone – we are looking for it. With the growth, well deserved because of the wonderful world SL is– you need to make a choice- keep patching this puppy or design something that CAN do it. Or skunk work a team to design a larger program and then land the old one into it.

    And if you are massaging these numbers for venture cap reasons- shame on you. The data is sitting somewhere. Cute lawyers with BMWs and houses in Marin can hire cute number grinders with pocket protectors and condos in Sunnyvale to dig through them. So don’t even bother.

    Why not get the bucks for a new system? Don’t spin it defensively. Did Henry Ford have a problem when he had to built another bigger plant?

    I don’t think I have ever* not* crashed a session. I am tired of cache clearing, DSL restarting all the little tricks to nurse this baby along. It really destroys the “you are in another world” feeling when lag is talked about more than the weather.

    Or is lag the weather in SL?

    And most of all. I am tired of being bald, when my inventory is lounging on another planet in a galaxy far far away.

  32. U M says:

    Its not the matter of Whining about this. Its the matter of continues problems. Nobody likes to complain or likes a service with problems. Its just no fun.

  33. U M says:

    Pastrami Linden please stop closing the windlight thread! what is the sence of that?

  34. Argent Stonecutter says:

    Pastrami’s closing the Windlight thread when there’s a known problem, so lots of extra posts about that problem don’t fill it up. That’s telling you he *is* reading it.

    I like the idea of “lag as weather”.

    Anyway, the thing I really don’t like in the new Windlight, myself, is the new login screen… which may have some kind of effect on the service quality metrics as people deal with firewall and proxy issues who had previously been able to ignore it.

    But there’s people on the sldev list working on getting the old login screen working with the new API, which should let the open source viewers bypass all that extra gunk.

  35. Domchi Underwood says:

    I just wanted to say that I’ve seen *HUGE* improvements with the Windlight viewer for Vista / Core 2 Duo / NVidia 8800 over the past couple of months. Official viewer used to crash and take Vista down as well… about once or twice a day. Windlight viewer crashes about once a week, and doesn’t crash Vista anymore. Of course, when it does, I still can’t send crash report because of http://jira.secondlife.com/browse/VWR-3016 πŸ™‚

    …and there’s still that 0.5/1 FPS problem whenever I look at many objects / avatars, even if sim is running at 45 FPS, which is not the problem with slow hardware since I get up to 70 FPS in less crowded conditions. (http://jira.secondlife.com/browse/VWR-3155)

  36. WarKirby says:

    The viewer crash rates are down a little, but still at almost 20%. That’s clearly a problem.

  37. Laura18 Streeter says:

    just my 2 cents worth,
    the JIRA is great but too complicated for the average non-techie user IMHO. And even finding it half the time is perverse – support its not, all that bit about misfiling issues, can be a bit intimidating.
    Linden’s reply in MISC-636 about round-robin DNS issues is usefult to me but way beyond the average layman. I like the fact i can add a comment to the Crash Logger now tho

  38. Tegg B says:

    U M Says:
    Pastrami Linden please stop closing the windlight thread! what is the sence of that?

    Maybe people should stop posting off topic in his thread like we are here doing to Meta’s πŸ˜›
    Instead of closing perhaps just deleting off topic posts, why should his job be harder due to other departments problems?

  39. Jazzman Jiballa says:

    If the Service Quality Metrics isn’t the place for folks to complain about almost anything I can’t think where they might.
    You would not believe the lag that’s been coming down in Aswang the past few weeks. I think there’s puddles of lag throughout the greater Aswang area, some residents are piling sand bags in defense, but the lag just just coming down.
    /me shouts to the sky…”When is it gonna stop”

  40. U M says:

    34 well that is all well and said. But still if there are problems with this version of wond light then they better off pulling if back. It does appear there is problems with the client.

  41. mimi says:

    Pastrami seems to be doing his best, he writes long explanatiory blogs, tries to communicate. Who knows, maybe his next blog will contain a few fixed to trouble that is mentioned there by many people? I like that he’s trying to explain the idea behing everything, no vague/avoiding answers, I very like this more communicative kind of lindens ^^

    I seem to have little extra problems with windlight crashes, compared to the other clients.

    Windows XP
    AMD Athlon ™ 64 dual
    2.20 GHz
    2.00 GB
    NVIDEA GeForce 7600 GS

    I keep the draw distance low always, because otherwise I have much more lag and trouble. I crash a few times a day, some days all the time, some are just a few crashes.

    I hope they will change something in the looks of WL, the problem with overshadowing/overlighting is still very bad, but who knows?

    I wonder if the statictics say something about the amount of premium members compared to the new people entering and the amount of growth in the overall population?

  42. Melody Marlin says:

    Thanks again for this Meta and congrats on the improving signals. I hope it continues at a pace faster than the lowering of the US trade deficit but I appreciate it’s ‘something’

    @Ari and Xi and the whiners at the whiners crew: If I buy a brand new car (and I could’ve with what I’ve paid into this over the years), and (verifiably according to these metrics) 1 in 5 days I go to start my car, it won’t run – I think I have a good reason to complain. This is not yours so don’t take it personally. I’d take that car back and say fix it. I would not take the car in and have to explain precisely, with a demonstration what’s wrong with it, that’s not my job as the end user who pays good money on the product. If, when I signed up to this years ago, and this was a ‘beta’, yes I’d be expected to gather every detail I could on what made my crash happen. But this is not, supposedly, ‘beta’. Let people have a gripe if it’s there, it can’t be a personal affront, might make them relax a bit even. When I began, there were Linden’s attending events in clubs, teaching areas, at shop openings – interaction. Now, if we’re lucky, we can catch them in ‘office hours’ and it’s rare to have a Linden sighting walking about town so to speak. This is the limit to many people’s interaction with Linden’s unless they’re concierge and we can meet a few at the invitational parties. Let em vent. So long as they aren’t taking 5+ posts per blog, it harms noone. Go to a share-holders investor meeting sometime and tell me nobody rants – it’s not much different here.

  43. Storyof Oh says:

    Off topic but the last post open…in UK papers today breaking news….how all personal details of learner drivers/those taken theory tests since September 2004-April 2007 (3 million gestimate) have gone missing from from a computer estabilshment in …………. IOWA……..
    …why were these details stored?
    …why in IOWA !!!! ???
    …how happy are you to let anyone loose with your ID info?
    …how many of these missing ID’s are now nicely verified on SL???
    …this has been known about by government since June 07…
    …nice one

  44. U M says:

    Can`t login…………to windlight. anyone else have this problem now?

  45. Pepper Haas says:

    I’m still on the pre- new search viewer and plan to stay there until I stop reading on the blogs what a bad idea it was to upgrade

  46. Jane Doe says:

    Tks for the quality metrics but…
    WHERE ARE THE OCTOBER/NOVEMBER ECONOMIC KEY METRICS??????
    ARE YOU AFRAID TO SHOW US HOW BAD YOU ARE????

  47. U M says:

    still can`t login………………support ticket filed. and no reply et whats going on.

  48. Pingback: OMG!!! The Service Quality Metrics!!! « A Crimson world

  49. sloopy cooder says:

    I still have about a 25% failure for teleporting. If it doesn’t go quickly, it’s all over and I’m logged out, and it takes several mins to login again. Can LL please fix teleporting? This is a pretty basic function. I’m not alone – all my SL friends have the same issue. The grid is not stable. Do a feature freeze and JUST FIX BUGS please!!! All the new features have made stability MUCH MUCH worse. Time to stop, take a breath, and look at what you are doing. Get things stable before you do ANYTHING more. The latest regular viewer is fairly stable, but it too could use some help.

  50. Raven Primeau says:

    I often wonder if “Death by TP” and crashes & freeze ups help boost the 1,XXX,XXX logged on in the last XXdays figures, not to mention relogs to remove shoes and hair from bums.

    As a tip when it’s real bad, try TPing with minimal prim attachments, it often helps me.

    P.S. NEVER EVER TP in a total prim outfit if you value your decency.

  51. Argent Stonecutter says:

    JIRA appears to be hung.

    Too many people reporting login and random exits from the new Windlight client?

  52. So, where are those economic statistics??

  53. @#10

    “@ BlueSweet – you mention this: β€œGeForce 8400 GS/PCI/SSE2β€³

    Is that an ATI card?”

    No Ari this is an nvidia card just like the brand you say has OK drivers. I use an ATI card and have very few crashes so lets stop trying to blame someone else for these problems. Yes some video glitches like rendering problems can be blamed on video drivers but not application crashes.

    As far as the JIRA, agreed, unless you’re very techy, you will get lost just trying to figure out how to submit something and likely will get it deleted if one of the techy guys thinks it’s done wrong.

    I submitted a bug I’ve had to deal with for almost a year in “viewer/building” because it happens when I’m building (I apply a texture to a prim and one face doesn’t end up textured while the other sides do)

    Someone decided it was sorta like a bug someone else submitted over six months ago in “graphics”, which this bug is now lost among 350 very minor rendering bugs. It’s been in the system for 6 months, 20 people have voted yet no one is even assigned into looking at it?

    Why should I bother adding more to this list of bugs if this MAJOR issue with building is being ignored? I even offered to show any linden in world how I can reproduce this consistantly to help them, heard nothing in reply 😦 Maybe they can’t reproduce it but me and other unfortunatly can.

    Stephe Ehrler

  54. Bobo Decosta says:

    where are the economic stats? We haven’t seen them for months. Sounds like LL is to embarrassed to show them.

  55. Montana Corleone says:

    And we’re still waiting more than a day later…

    Well, not a gushing outage one, not that December’s will be much better with the asset server fiasco.

    So here are the Econ figures? We’re more concerned about those than a bit of outtage. You’ve had way much longer to massage them or think of a spin for these this month, or are you hoping Christmas will get in the way and we’ll forget? Must be bad, and continuing the downward trend in almost everything, otherwise we would have had a gushing blog by now…

    I mean last month you picked the only upward number to gush about lol.

  56. Confused says:

    Yeah umm so later today, turns into tomorrow, which is gone now, and still no economics ? Dubya Tee Eff Lindens…. Get with the program. It doesnt take that long to fake your own data ….

  57. Keiko Rau says:

    @Meta

    …though the Planned outage metric is still excellent, at around 0.3%.

    Once a planned outage has exceeded its scheduled window, it becomes an unplanned outage, so as long as you schedule planned outages sparingly, this figure is always going to look good.

    I would hope that you are counting the hours beyond the planned window as an unplanned outage, rather than missing it – the recent planned grid shutdown for asset server maintenance ran at least three hours overtime!

    Come to that, how much notice does a planned outage need to still be a planned outage? If we get a blog message saying something like “we need to shutdown the grid tomorrow for essential mainenance” that (in my opinion) should not be called a planned outage. Planned outages require planning, which implies plenty of time for notice. If there isnt at least a weeks notice, then its an unplanned outage.

    I hope these issues are taken into account in these metrics, otherwise they’re just more fluff.

  58. River says:

    TY Meta πŸ™‚

  59. Montana Corleone says:

    Well I hope you include yesterday’s asset problems in next months. Although the grid was only down briefly, another resolved issue (when it’s far from) meant you couldn’t rez anything for four hours is quite a chunk of the day.

    I really wish you’d be careful with the use of “fixed”, “resolved” and “All Clear”. Yesterday after the grid was back up it worked for two minutes before I got Unable to create requested object errors.

    Instead of “resolved”, how about “We have our fingers crossed and are praying that what we’ve done might fix it.”?

  60. Rapidshare says:

    I completely agree with all that here is told
    So you can find the information on it on my search resource
    http://fileshunt.com

Comments are closed.