Frank Ambrose: Updates on the Second Life Grid

Hello, I’m Frank Ambrose, the Senior VP of Global Technology, and I’d like to take this opportunity to let you know about some of the work we’re doing on the Second Life Grid.

By way of introduction, I’m a recent hire here at the Lab, having joined to lead our global technology team. Specifically I’ll be focused on grid infrastructure and our stability initiatives. As noted in the press release, I come to the Lab from many years at AOL (and prior to that MCI), where I experienced the kind of explosive growth, global scale and inherent stability challenges we face here at Linden Lab.

More than anything else, my tenures at those companies taught me the direct relationship between platform stability and user experience. I’m looking forward to applying that lesson, and a host of others, as we work to maintain, build and improve this complex virtual world. I am keenly aware of the pain that any service outage can cause and am both excited and confident that Linden Lab has focused the right resources to achieve this critical objective.

Given the complexities in our architecture, our stability efforts span many individual areas, most of which were detailed by Ian Linden’s May posting. Some areas will be addressed through short-term initiatives, while others will require significant re-architecture, software changes and new physical hardware. Throughout it all, we’re committed to making the transition to a more stable world as seamless and transparent to you as possible. To that end, members of my team will be using the blog regularly to provide updates on plans and progress towards meeting our stability goals.

As part of our wider stability plan, we’re targeting 4 major infrastructure points both with long-and short-term goals: Intra-Grid Network, Asset Storage Cluster, Central Databases, and Host/Transit Data Services. The strategy is to develop and deploy near-term solutions to improve stability, while looking more broadly at our architecture (hardware, software, networks, etc). In the near term we’ve got a number of projects in flight to address some of these problem points. A couple of examples are:

– Asset collection. We’re collecting many assets that are on our storage clusters, but are rarely (if ever) accessed. These assets take up critical space on the clusters and potentially degrade performance and stability as we hit volume thresholds. We’ll be moving these files to different storage mechanisms and, while they will still be easily accessible, it will help us to avoid pushing the limits of our existing storage clusters, while still preserving all existing assets in a reliable storage environment.

– Reducing the need for VPN connections.  Since we don’t encrypt communication between simulators and our databases, there needs to be a safe means to communicate across data centers and so we use VPN connections. The connections don’t scale well and can be unreliable, so establishing a new communications mechanism, that is both safe, scalable and reliable, is another short-term project.

These projects are just a sampling of the work that is currently being done to improve stability, and I’ll be reporting on their progress, as well as other short-term projects, in the coming months.

We have a lot of work to do but be assured that we have the right resources and internal focus to achieve our stability goals. From personal experience, I’ve encountered many equally complex challenges, especially in my time at AOL, and these problems are all solvable with the right level of attention and technical talent. We certainly have both, now we will start delivering.

This entry was posted in Announcements & News, Bugs & Fixes, Development, Operations, Resident Experience. Bookmark the permalink.

76 Responses to Frank Ambrose: Updates on the Second Life Grid

  1. summerseale says:

    “By way of introduction, I’m a recent hire here at the Lab, having joined to lead our global technology team. Specifically I’ll be focused on grid infrastructure and our stability initiatives.”

    /me hands Frank a waterproof parka and football pads. “Welcome, and here is your standard equipment which you’ll be needing starting today.”

  2. Toy LaFollette says:

    does this mean LL will now give free CD’s in each bag of groceries? BTW thanx for deleting my prior post, I was quite serious.

  3. Kyder Ling says:

    Guys, I think he meant AOL as in when AOL was the only real contender when it came to Internet like when their population and usage increased radically.

    I sure would rather have a guy from AOL than say, net Zero… >_>;

  4. Pingback: RSS SL Blog: Frank Ambrose: Updates on the Second Life Grid - SLUniverse Forums

  5. Thats coming alot from a person that came from a failing company…. Can you save this company to not go in the way of AOL went? When we log on SL next time will it say: YOU GOT MAIL!

  6. mrcvp207 says:

    Will the wider stability plan of Second Life require us to also upgrade our videocards?

  7. Porsupah Ree says:

    Welcome! Thanks for the introduction. I do hope, within the limits of normal commercial confidentiality, you’ll bring us updates that, at times, are of a technical nature. After all, there’s no shortage of geeks amongst the SL populace, and I know I find it fascinating to learn more about what underlies the grid, and just what can and does go wrong, from a technical perspective.

    The seldom-accessed assets certainly sounds like a promising avenue. Can you share any stats on what sort of numbers/percentages of assets have lain unused for, say, a year or more?

  8. Aminom Marvin says:

    This has been the blog post I have been waiting for: a talk about what components of SL need improvement and why they fail, what needs to be done and will be done, and what is happening short and long term. While this post talks in broad generals, it tells enough to get a feel for the focus of LL in these areas. Frank, you sound like a man of action, and, if the results follow your words, will be invaluable to LL. Good luck!

  9. Ashelyn Dryke says:

    As long as the asset servers stop frakking up and load the entire inventory, and as long as textures load faster, then I’ll be quite happy.. >.>

  10. Welcome and good luck. One observation: while transparency is good, invisibility is better. I came to the blog today after a crash, to check the status and see if it was worth trying again. It has been a bad few days in SL. I don’t feel much better being told all the major restarts, upgrades, and meltdowns that are going on at LL. I would feel much better if all that were going on without completely borking the grid and demolishing my business for several days. I hope your experience with building railroads while the train is bearing down on you will help!

  11. While I do, really, honestly, love these announcements by people embracing the Laboratory, and love to see all of their ideas of how they are going to Fix The Grid – really, it is very refreshing – I seem to be feeling a little… ennui, now.

    Perhaps older residents could advise me, if there are any left. Is this something that rises on one’s third birthday, three years of seeing the exact same promises?

    In any case: please wake me when things work. I have a little sign around my neck, it is quite obvious.

  12. JZ says:

    Well, i am a 4 year believer in Secondlife and I tip my hat to you Frank. Let’s hope something positve DOES come from this undertaking. Lord knows many believers are wishing that this virtual world can be as stable as the RL. Well dream and belive anyways.

    Good Luck!! You’ll need it :))

  13. Jessica Hultcrantz says:

    Whatever about old compaines….

    As long as things actually gets done and the promises of a more stable grid gets reality instead of just promises, then it’s the right way finally…

    And please, don’t stay silent and wish us all a happy day with the standard phrase “thank you for your patience” ™

    Keep us updated with good technical postings thank you!

  14. nika talaj says:

    Thank you for the initial posting, Frank, hopefully the first of many. And glad to see this hire finally made; LL certainly has been searching for a while!

    Whatever one may think of AOL and MCI’s service offerings, certainly they have massively scaled networks, and that’s what’s relevant to this hire. I think your hire also represents a significant investment in stability by LL; I cannot imagine that your compensation package is modest.

    Also, applause for the committment to move away from VPNs … long overdue, for all the reasons you cite. And more.

    Welcome!

  15. BlckCobra Shikami says:

    After the horrible time from february till july, august was quite promising and things seemed more stable again. But last week has proven that it was just a short break and we are facing a lot of unhappy customers who lost things and lost all faith in LL support and technics and come to us merchants for help (most even don’t bother anymore to submit a ticket because they are tired of the standard responses). It’s crucial for the future of SL that those problems get addressed. It’s frustrating for everyone to deal with those problems.
    Many merchants spend their online time with replacements and sorting out fraudulent replacement requests because we don’t have any chance to verify those things.
    LL support tells us merchants not to replace things and at the same time tells customers to ask the merchant for a replacement. This is a horrible situation for many of us.
    It’s really time for LL to grab the keyboard and redesign certain aspects of the assert, database and communication system to make it a reliable environment. We are talking about a system where people spend a lot of real US$ and EUR not a simple fun game for a few bucks.
    I wish you and all of us lots of luck and success for the next steps towards more stability and reliability.

    /me hands you a bullet proof vest

    Cobra

  16. Ciaran Laval says:

    I’m a tad concerned as to why you’re Frank Ambrose rather than Frank Linden, although we cry out for the Lindens to be Frank, this appears to be a shift in culture.

    Scalability is a big issue here, a very important part of that is group limits and yet I see no mention of that. We’re in an age of social networking frenzy, the group limits are a serious setback.

  17. Aquarius Paravane says:

    Welcome Frank and thanks for the post and your commitment to server stability. Server stability was greatly improved until a week ago. It would be good to get an explanation of what actually caused the recent outages.

    It would be better still to see somebody step up and give an equal degree of commitment to fixing the client stability issues that have continued for over a year. Client stability has wrecked user experience on certain platforms which experience the client randomly freezing every few minutes, or crashing randomly every few minutes or crashing the computer it’s running on several times a day when the computer would otherwise run for weeks without being shut down.

    Regarding assets, the last attempt to clean up “unused” assets resulted in textures and scripts that were in use in products being deleted permanently and irrevocably. Please provide some assurance that this won’t happen again, and explain how assets that have been offloaded will still be accessible when called upon. Texture loading already takes a stupidly long time on some platforms and there is no way to demand that some particular texture be loaded ahead of all the others, for example by clicking on it.

    It would also be good to get a commitment to preventing content theft while allowing content creators to move their own creations to and from other 3D platforms.

    If you get these major issues out of the way then you can actually start to work on improving user experience.

  18. Sindy Tsure says:

    Welcome to the fun, Frank!

    I’m also curious why you’re not Frank Linden…

    You’ve got a “{insert wiki link}” still in there, which I think is supposed to go to http://wiki.secondlife.com/wiki/Service_Disruptions#Linden_Network . Maybe.

    Like others, I’m supportive and glad to hear what you’ve said but it’s sorta stuff we’ve heard before. Hopefully, improved (and expanded) communications will give us more insight into what’s happening. People who can’t make office hours don’t get to hear a lot of it…

    Can I have a bear?

  19. Vivienne Graves says:

    @16: What makes you think increasing group limits is workable when with the current limits, five out of ten large group IM sessions fail and in order to reduce database load the time group notices are archived has been reduced from one month to two weeks, OR the most recent 200 notices (where it was unlimited)? Those two things in themselves are pretty clear indicators of why increasing group limits just isn’t technically feasible at the moment; hopefully, when the underlying network and database issues have been addressed, it will be, but now is not the time to expect it.

  20. Ciaran Laval says:

    @19 They’re not workable, yet they’re crucial to the operation, hence why I asked why there’s no mention of them,

  21. Relic Starbrook says:

    Welcome Frank! Thank you for your leadership and actions toward a more stable grid!

  22. Relic Starbrook says:

    @ 20 I would be willing to pay for more groups (pass me a bucket of 25 more…for a reasonable price)

  23. Paddy Wright says:

    @15 Cobra

    I am indeed one of those customers who lost a relatively valuable item during the recent 6 days of turmoil. I thank Cobra for being patient and allowing me a ‘goodwill’ replacement. (the products he/she sells are brilliant) I agree that technical instability more than anything else could wreck SL and its economy faster than any other factor. Consumers will loose faith, in the face of growing and more competent competition. I therefore welcome Frank and his teams initiatives and sincerely wish him well.

  24. Ann Otoole says:

    So will you be implmenting a true transaction based architecture with a enterprise scale transaction manager along the lines of Tuxedo? So whenever the database or network fails to respond in a time window the transaction is safely rolled back? I.e.; no further failed L$ transactions resulting in loss of money, no further loss of assets on rez or teleport, etc. And how do you intend to stop assets from mysteriously vanishing from the account’s inventory? What measures are in place to log all access to the database that does not happen as a course of normal middleware transactions? Will you provide us with a third tier of inventory for long term asset storage so we can move items off to the slower access compressed archive area? This could help a lot if we could intentionally migrate assets to the “long term parking lot”.

    Will you add an indicator to the UI that shows when the grid is not feeling well so we don’t have to rely on grid wide blue messages?

    There is a lot of work to be done but the most important part is a transaction manager that can set an indicator in the UI when the failure rate goes out of tolerance. Nothing like a little +/- 3 standard deviation programming to help stability.

  25. Christos Atlantis says:

    Hello frank, welcome to grid.

    I had writen a long harse letter about grid stability and SL, but decided not to post it here, please test your codes before you need 10 days to deploy them and please stop making versions that half the world does not have the computers to run, all of us at SL have lost alot of friends to SL improvement and alot of us have lost business to those improvements.

    I do hope your job is really about SL stability, true stability not buy the new computer to get stability. Again welcome and I hope you do make a diffrence because so far the SL experience has been mainly unstable.

  26. Renee Faulds says:

    Glad to hear more promises and good luck.

    If you would have waited 2 more months to announce this it would have been exactly “one year” ago Phillip said the ‘exact’ same thing.

  27. Eric says:

    mrcvp207 Siad:
    Will the wider stability plan of Second Life require us to also upgrade our videocards?

    I feel SL is dumbed down for those with weak hardware already. That’s like back in the day when a lot of websites were still designed for 800 x 600 resolution because there were still a few people with weak computer systems, long past when they should have been designed for higher resolutions.

    Crank it up!

  28. Welcome, Frank. I wish you the best of luck in your new position. It is nice to see someone with your experience hard at work for us residents.

  29. Soap Clawtooth says:

    This would be my to-do list.

    1) Go and buy some sun fire X4200’s
    2) Speed up the transfer of the DB from SQL to Oracle.
    3) Convert Server software to linux (if not already) 😀

  30. Soap Clawtooth says:

    oh also

    4) Develop grid prioritizing, so burden from empty sims is shunted to support sims that are more active.

    5) Develop automatic backup, so if one server goes down another server can take over instantly.

  31. Duke Mcdonnagh says:

    Welcome,

    Hope you know your stuff your going to need it. Here are my questions:

    1. Why should SL not be responsible for Lost content? They have the keys to the database and aren’t the responsible for making it safe and readily accessible?
    2. Why do we not have fully redundant systems?
    3. Why do we continue to have Teleport Issues daily?
    4. Why do we have login and network failure issues daily?
    5. Why is the first solution reboot the servers and reload cache as opposed to finding root cause?
    6. why do we have lost transactions and delays in transactions becoming available for review?

    The definition of insanity is to continue to do the same thing and expect different results.

    Duke

  32. Darien Caldwell says:

    I agree with most of the commenters that what parts are proelmatic seem to be rather obvious. But the first step in any project is defining what the goal of the project should be, and identifying what has to be accomplished to reach that goal.

    So good first step, I wish you and your team much success.

  33. Darien Caldwell says:

    /me dies of embarrassment at that glaring typo. problematic….

  34. Digital Digital says:

    Aol…

  35. Soap Clawtooth says:

    Well, the server stability and lag issues are the big 2.

    Prioritizing burden system will automatically load-balance. This means resources from under-utilized areas (all those open islands with no players on) will be used in busy regions.

    and then built in fail-over capabilities, meaning should a server fail, another just takes over with minimum, if any, disruption to residents.

    But generally just better servers than they have for a start.

  36. Sindy Tsure says:

    /me takes all Dariens stuff after she dies of embarrassment.

  37. Alicia Sautereau says:

    Welcome aboard (again)

    No offense intented with the following:

    Linden Lab has been promising stability for the past 1.5year that i`m here and 4 years for the vets, the only stability that we`ve seen is by 3rd party patchers, like Nicholaz, to clean up the mess that should have been fixed along time ago.

    I can`t even imagine a change unless you got carte blance in man power, funds and access to change the entire backend of second life.

    btw, i hope your team is not working with the TAO of Linden or your team will endup playing tetris instead of fixing things, just look at the top voted jira entries that are still out standing…

    Good Luck, you`ll need it =)

  38. Escort DeFarge says:

    Excellent news. Welcome Frank!
    (Nice hire, Mark!)

  39. Shug Maitland says:

    Welcome Frank (not a Linden?)
    a couple of suggestions re. system loading.
    There has been discussion here in previous blogs re traffic, something must change! at any given time the system is supporting thousands of campers and bots. needless load to those of us who are actively trying to live our second lives!

  40. Joshe Darkstone says:

    Ive long wondered about the use of VPNs in the network architecture. While I use them for personally connecting to resources in disparate offices to check on network infrastructure, using them as part of the network service itself is a bad idea. they are notoriously unreliable, subject to dropped connections, and interminably slow and transferring data – especially in the form of files. ftp is 10 times faster to transfer a file then a filecopy operation, even when they both run on the VPN. Will be glad to see that bit go.

    And welcome aboard, Im glad to see someone approaching this witht he proper background.

  41. Mitzy Shino says:

    @37 no I have a copy of Darien’s will right here, and she willed it all to me, with FULL PERMS, give it back!

  42. Sandor Balczo says:

    Perhaps Frank Ambrose is not Frank Linden because he doesn’t need to be in world. Or perhaps he has a different kind of employment agreement.

    I don’t care as long as I can start travelling on trains again and be able to trigger my animations and see them work, which has been impossible since mono!

    Sandor 🙂

  43. Coventina Dalgleish says:

    Welcome Frank, well you are now in the fire )) the frying pan was hot but this is the cooker. You will notice many savvy people here who do understand the operations of a large complex network. In this case , I have not seen the flow chat, what seems to be a convoluted setup.

    I can give a bit of advice though ignore the Tao and make your coders responsible for what they create. Most problems are created by these every two to three week bug fix updates that cause the system to crash and burn for several weeks after then when they become stable a new batch of sloppy code dumps us right back to hating the game.

    I run several machines on several networks from old tech to well )) newer old tech and one of the problems you will encounter is the bells and whistles take the pleasure of the game from many who would be ardent supporters but just can not get the performance required to play with any satisfaction. I can tell you for a fact on my older tech machines, 3.0g p4 nvid 6800 ultra the game must be totally stripped once a month and re installed or it simply will not work.
    This is far worse than MS and is the cause of many of your customer complaints. This is a total removal through the registry level including linden research entries. The only reason I bring this up is tonight was kill LL on the slower machine and reload.

    Arriving from AOL I am sure you have a vast experience with frustrated customers )) well you have a different cross section here from very knowledgeable to neophyte. The goal should be to keep the neophytes happy and continue as customers. My store seems to be a drop zone for noobs , no idea why but we spend a good amount of time educating them in the basics of how to survive and have a good time in game.

    Assets have been a problem since at least 2004 most likely caused by the network but the Lab needs to take some effort to back up all the uuid keys of every item created. Yes, I know a massive undertaking but you could recover a massive amount of storage if you would kill all the unused accounts. If it has not been used for 90 days put in limbo for another 90 then kill it. Look at your real users and support them the play one time and quit accounts still exist in your data base.

    Grid stability and asset reliability would go a long way to creating a happy user base and an increasingly successful game.

    Oh and I am a noob going on 4 years ))

    Well Frank again good luck,but talent does not require luck does it?

  44. Pingback: Dusan Writer’s Metaverse » Head of Second Life Stability: He Isn’t a Linden

  45. Katt Linden says:

    You folks are on top of every detail, impressive as always.

    Frank is of course a Linden, although he isn’t Frank Linden. He’s FJ Linden.

    Please be kind when messaging him inworld! As he said in his post, he’ll keep blogging to let everyone know about what his team is working on, and when he has time, we’ll arrange for him to participate in a Forum Discussion so you can ask him questions, over the course of a few days, online so those in all timezones can share the discussion and easily read the thread to catch up and see FJ’s answers.

  46. Frank Ambrose (FJ Linden) says:

    So when people told me that we had a passionate resident base, they weren’t kidding!! OK, I wanted to respond to a couple of the common themes in your comments to my post, so here goes…

    – I AM a Linden, FJ Linden to be exact. I should have posted my Linden name and will do so for future posts. Conspiracy theorists can rest, just getting used to the new name 🙂

    – I do appreciate all the comments and suggestions, and believe me I read them. So much to do to get the infrastructure to an acceptable level, but there are certainly some immediate issues we are addressing, the VPN’s are at the top of the list. I’d love to get that one out of way soon, and some attention to the network layer will get us there.

    – One of my initial observations is that many components of the infrastructure are “over engineered.” That’s not a criticism, its just a fact. There are a lot of reasons for this and we’ve already taken steps to remove some complicated code that was deployed and contributed to some of our recent instability. It was designed with the right intentions, but just didn’t deliver.

    Finally, I want to restate that we are committed to fix the grid. It’s my singular focus. But I also know that words are cheap and we need to start delivering on these promises. If my days at AOL taught me anything, its to have a thick skin, and sense of humor. Whatever the comments may be, I’d much rather have an active group of passionate residents (frustrated as they may be), than a silent, disengaged group. By the way, you can say what you want about AOL’s business, but we did build and manage the largest network and host infrastructure of any ISP in the world and faced many of the same issues as Linden Lab. We overcame them all. Lots of bad press about AOL recently, but none of it was ever about stability or scale!!

  47. Dear Linden Labs,

    ACTIONS Speak Louder Then WORDS.

    PS. Good job removing people’s comments on how things REALLY are…

  48. Bucky Barkley says:

    Welcome Frank!

    I too am a former AOL’er, having worked on AOL.Com in the 90’s and then in 2005. Frank is coming from a complex network that serves millions, and although I dont know him personally, I actually know that the AOL experience is going to come in handy!

    The one thing I ask of you, Frank, is to keep blogging on a proactive basis. Dont forget to communicate. Cory was disappointing in this regard, and not much of a leader. Joe Miller is virtually non-existent on this blog. Residents will be looking to you to talk about problems and progress. We truly need a top tech person that doesnt have be goaded into talking.

    Hopefully this is a great signal that M Linden is serious about putting a staff of communicative adults in charge of key functions of LL. Ditch the Tao, dictate priorities, and make the stability of SL rock solid! That is what is needed to win back credibility on the part of third parties who might invest in this platform.

    Good luck Frank, and welcome!

  49. Bryon Ruxton says:

    Ditto! Bucky. Well said!

  50. dusanwriter says:

    Hmmm. Well….first FJ, welcome aboard, and apologies for conspiracy theories. Your background is a great asset to the team. I DO wonder a few things:

    – You have some background in identity and trust systems – will this be tapped into at all do you think, especially with regards AWG etc?
    – One of the values of SL is its global reach. I’ve been trying to get a project up and running linking to Lesotho in Africa for an AIDS project for example – but I wonder if you have any comment on whether the current infrastructure needs to be improved for access in different regions?
    – And much more importantly, has anyone talked to you yet about getting decent prim hair or have you taken care of that?

  51. Kraelen Redgrave says:

    Welcome Frank Linden! Just ignore the grumpy people! 😀

    I for one appreciate someone stepping up to the challenge. I wish you the best of luck!

    I do have a question though… Why on earth do you keep all of the assets that are no longer in use? I myself have uploaded numerous textures, sounds, and animations that I have since deleted from my inventory. Noone else can use them without the UUID. Surely removing obsolete assets completely rather than moving them would be a better option?

  52. Eko Fei says:

    So I’m wondering if all these wonderful new stability changes have anything to do with the fact that I logged in today and 70% of my inventory is missing? Anyone?

  53. The infrastructure is over-engineered you say?

    How is that infrastructure these days? HTTP web services in a sane design or ye olde messaging system?

    “By the way, you can say what you want about AOL’s business, but we did build and manage the largest network and host infrastructure of any ISP in the world and faced many of the same issues as Linden Lab. We overcame them all. Lots of bad press about AOL recently, but none of it was ever about stability or scale!!”

    That’s to say nothing of stability of the average AOL user’s mind of course, no offence 😉

  54. Jebediah Spatula says:

    It’s good to see someone come in who has real world experience facing the same type of challenges SL is currently facing. Hopefully we’ll see more than just words this time. Good luck and don’t forget your helmet. 🙂

  55. Welcome to the Lab FJ 😀

    I really look forward to the results of your work. I’m not an angry moaner… passionate, but not angry!! I’m riding the wave and waiting for *the* creme de la creme release!! At the moment I have very limited access to SL because my laptop (with ATI graphics & running Vista) requires me to reinstall SL after ever logout, crash and failed login….. 4 reinstalls in an hour is becoming a tad irritating 😦 In the meantime my 6144sq M store is running itself and my customers think I’ve jumped ship!!! I look forward to being able to lift the anchor and steer my ship again…. your work should hopefully let me do this 😉

    FULL STEAM AHEAD!!

  56. Katt Linden says:

    BTW, for better clarity and to answer questions from a few of you, I edited the author tag line to make it clear that Frank Ambrose is indeed also FJ Linden. Thanks Dusan and Sandor!

  57. Shug Maitland says:

    okay, FJ Linden, much better!
    and yes do somethigh about the hair!!!
    but the big question:
    when will you have a doll or teddybear?
    .
    .
    Welcome to SL priorities!

  58. Phoenixa Sol says:

    Thank you, I do wish you much luck! I’d especially like to see issues such as Jira 6110 be distant memories rather than daily fear cards.

  59. Pepper Haas says:

    I am glad they finally hired someone whose computer knowledge isn’t limited to making superficial changes to the GUI, giving us many things we don’t want such as a floating Release Keys button and giving us a double Search window etc.
    So, here’s hoping that this is not just talk but we will indeed have more stability. It sounds like a good plan anyway, thanks for communicating.

  60. Welcome Frank!

    Intereting to hear you talk about OVER ENGINEERING not doing the job.

    As in #29
    If , for instance, the Asset Server was an Oracle Database and TABLES were PARTITIONED on DATE OF LAST ACCESS then removing rarely used assets from 99% of common SQL searches might be automatically handled by the database.

    Now obviously all of this suggestion might be moot in the absence of detailed knowledge of the problems at hand but I doubt if this Asset problem is unique to LL and would caution you to be looking for similarly OFF-THE-SHELF solutions – avoiding more OVER ENGINEERING.

  61. Uke Takashi says:

    Dear SL users and Frank.
    I was and still am an AOL user, I have been there for 10+years and to thos that say AOL is failing you may be correct but it wasn’t a program problem rather one of poor managment and lack of for sight.
    The problems LL has may be something that can never be totally corrected in till there is a change in how data is transmitted and stored.
    I take heart in the Idea that people are working on it and welcome any improvement.
    A small perhaps useless idea is have inventory items not used in say 2 months be stored on the owners client, then sent back to assest servers when that person uses the item again.. I am sure you can incript it so it can only be accessed via in game.
    Good luck Frank
    Peace and Gentle Winds to you’
    Uke

  62. Pingback: Massively.com SL news

  63. Pingback: New Linden Lab VP with an old message | MMORPG-BAY

  64. Hello and welcome, FJ! (Frank!) I hope you are well-blessed in your endeavors! Please don’t mind the “nay-sayers”. Most are well-intentioned, and perhaps a bit jaded over promises made, that haven’t always carried through, or didn’t go quite the direction they had hoped. Sadly, a few of the nay-sayers just enjoy griping, no matter what LL states or proposes. You will soon learn to pick those people out, and take what they say with more than a grain or two of salt. Anyways, what you say sounds promising, and I look forward to seeing the improvements you mentioned! May God and Goddess bless you and Linden Labs, for making the quality of my life better than what it was before I discovered SL! 🙂

  65. Wayfinder Wishbringer says:

    A ray of hope hits Linden Lab.

    “More than anything else, my tenures at those companies taught me the direct relationship between platform stability and user experience.”

    I wish to welcome you Frank. Your kind of experience, understanding and recognition of the user experience is what I believe will be a great asset to Linden Lab and Second Life. Not only am I pleased to see such talent and vision join Second Life… I applaud the vision of those at Linden Lab who decided to bring you on board. Because that not only means that LL is recruiting the skillsets needed to fix the problem… they’re recognizing there is a problem

    And that’s something we customers have been waiting for quite some time. 🙂

  66. nika talaj says:

    Frank, if you are still reading comments, you may wish to cast your eyes over this short forum thread. It highlights a problem (one which you are uniquely positioned to solve!) that causes some businesses to leave SL. The business owner in this case is a well-respected European machinima maker who I know has been making promotion SL videos for RL and SL enterprises, and appears to be expanding his company’s service offerings … probably outside of SL.

    http://forums.secondlife.com/showthread.php?t=282114

    p.s. I have no ties to the company in question, and do not wish to imply that videos are their only product/offering. That’s just the only one I personally know about, from seeing his postings 🙂

  67. Wayfinder Wishbringer says:

    BTW (and scuse the double post but lots of room still left), Frank, please note the generally overall positive response above. Even the posts that are negative– are respectful and constructively negative (which I have no doubt you realize = user asset!). That in itself is a big accomplishment. The fact that you folks got a positive comment from me is a big accomplishment. So far everything I’ve read from you FJ, has rung true and good… which is a fine change.

    And yes, while ‘platform stability affects user experience’ would seem a very obvious statement– it’s one that we very seldom hear coming out of Linden Lab. It’s not the simplistic statement that’s good to hear– it’s the fact it’s coming from the senior VP of Linden Lab… a recognition of a simple, basic truth that seems to have too long not been recognized (at least, publicly).

    Simplifying hardware? Go for it! Altering VPN. You bet! And if I may disagree with a prior poster who said basically “ramp it up”… I’m one of those who believe functional non-eye-candy is better than non-functional eye candy. So if the system has to be reduced a bit graphics-wise to get it to work with more people… I’d do it. I don’t think that needs to be done because the versatility of the current graphics settings allows almost anyone to play. The problem was in letting users KNOW about how to do that (which should have been on the splash page, frankly). But your comments on major changes being made to both hardware and software is something us “pro users” have been aware of and waiting for a long time. It’s REALLY good to hear you talking about such things.

    May I put my 2 cents in here regarding something that needs to be done asap… one of the most damaging situations currently on SL? Texture loading. A number of us (including Dan Linden) have been running stats lately… and 95% of all texture loads are taking between 10 and 40 seconds to load. That is probably one of the most experience-damaging aspects of Second Life. Reality check: if 1000 textures take even ONE SECOND EACH to load… the average sim will require almost 17 minutes to fully rez. Since many sims have far more than 1000 textures, that number increases proportionally. So the reality: LL needs to get individual texture loading to UNDER ONE SECOND on an average. 10-40 seconds is just unbelievably bad performance. It damages the SL experience. It damages the shopping experience. It damages merchant sales. It’s just abysmal and imo on my list of “the most damaging things right now to Second Life”… texture loading problems is #1.

    And yup, we’re an enthusiastic (and often wordy LOL) group. As you realize… that means passion. Every negative message sent to Linden Lab means someone CARES (if they didn’t care, they’d just join the millions who no longer use Second Life). So whether you receive good comments, bad comments, angry comments, friendly comments, hateful comments, I’m glad you have “thick skin” and realize that at least you’re getting COMMENTS. The alternative is an apathetic user base, which is a bad thing.

    So welcome again! I have to honestly say this is the most encouraging blog I have read in a very long time.

  68. Edward Griffith says:

    Dear Frank,

    Welcome. Dumping VPN is a good start. But BEFORE you mess with anything else please make TEXTURE LOADING a priority. Current texture load times of 30 seconds are a total PITA. They are killing SL faster than crashes and failed transactions.

    You STILL use UDP for texture xfer!!! OMG – that is just wrong. One hiccup and you are back to reloading from scratch with incremental times of 10 seconds per increment!

    HTTP for textures is a duh fix, and it’s YEARS overdue. HTTP is not a radical technology for graphics. It isn’t difficult to code. It works. UDP was dumped by everyone else so long ago, LL’s continued use of it is frustrating in the extreme.

    I support SL. I truly want it to grow and thrive, but it isn’t in a vacuum. There is competition waiting to eat SL’s lunch. When you get comments regarding SL from this month’s Tech Crunch 50 conference (Virtual worlds – really?) “the platform itself from a technology standpoint, isn’t that interesting anymore… there’s serious limitations with it” that should be a wake-up call to everyone in LL.

    Why is SL loosing 1% user base per month?
    You can’t shop – FIX THE TEXTURES
    You can’t travel FIX THE TEXTURES
    You can’t see yourself or other people FIX THE TEXTURES
    and so on.

    So let’s review : FIX THE TEXTURES. Then fix transactions and groups and bother to get group IM’s right.

  69. Welcome to the Zoo Frank. I hope you deal with the Infrastructure better then you did with my locs.

    I have just one question for my friends at Linden Lab.

    Why couldn’t you have hired someone from say CISCO where they really do spend time in the lab? Was it just too much money to spend? I don’t mind having Frank around, but I really don’t see where Frank is an IP Expert as much as a Telecom Expert. I mean, let’s face it, when AOL needed more lines, they called the PHONE COMPANY, and MCI Locs were laughable (and yes Frank we’ve had the pleasure) at best.

    I am glad to see Linden Lab making some sort of headway, I just don’t see that I agree with the hiring decisions.

  70. Taff Nouvelle says:

    Welcome Frank, If all goes to plan this will be the best thing to have happened in SL for years.
    Just a thought about textures. We have a texture cache on the viewer, is there a texture cache on each region server so that regularly accessed textures are held locally to that region.
    Maybe that is already being done, it really is too logical not to 🙂

    Good luck, hope to see you in world now and then 🙂

  71. Zi Ree says:

    @72 Jayme Llewellyn: Uh … seriously … You’re expecting Second Life to work with an GeForce Mx440? It doesn’t even support the needed OpenGL and shader version. I’d suggest getting a used 6600 for USD 10 (they aren’t worth USD 10 anymore but shipping cost will probably be higher than the actual value) and see Second Life working (not great but at least working). Blaming Linden Lab for a serioulsy outdated graphics card is really the wrong way to go. Your CPU power has nothing to do with the needed OpenGL support and also nothing to do with texture memory on the graphics card.

  72. Pingback: WTF Objects older than 2007 can't rez? - SLUniverse Forums

  73. Pingback: Ongoing Updates from the Grid from FJ Linden « Official Second Life Blog

  74. Pingback: Ongoing Updates from the Grid from FJ Linden | SECOND LIFE

  75. Pingback: Official response to openspace decision

Comments are closed.