Thanks to all those who took part in the event yesterday, for those that missed it – here is the transcript for the October 26th Concierge Townhall event.
Click below to read on..
[14:56] Jack Linden: We’ll get started in a short while. Fritz Linden will be fielding the questions – by direct IM to him only please
[14:59] Robin Linden: Hi everybody, we’re waiting for Joe to join us
[14:59] Robin Linden: He’s going to talk to you for a few minutes about some of the work he and the dev team are doing. If you have questions, please IM them to Fritz
[15:02] Robin Linden: Please say hey to Joe Linden. Joe is the VP Platform Technology and Development. Joe will say a few words and then answer questions.
[15:03] Joe Linden: Hello, all. It’s my pleasure to join you today to answer some of your more technical questions, listen to you concerns, and hear your thoughts on how we can continue to make Second Life a more stable, productive, and enjoyable platform for you and your customers. My role at LL is to focus on the scalability and reliability of the platform as we grow our user base, so I can imagine that you’ll have some great questions for me. 😉
[15:04] Joe Linden: First, I thought I take just a few minutes to give you an overview of the development process at LL today. We have a large (and growing) body of software professionals, grouped into 4 development “studios”. Each studio selects projects based on our long-term architectural roadmap, our short-term priority list, and from a pool of “hot issues” that arise from time to time. At any given time, each studio has between 5 and 8 projects rolling. Each project undergoes a thorough design review, most are coded using modern agile development methodologies (i.e. pair programming), have specific unit tests built to verify the code functions as designed, then each project moves into a QA cycle that may last from several weeks to several months depending on the complexity of the design.
[15:05] Joe Linden: Many of you know we have a main production grid (called “agni”) and a beta test grid, but you probably aren’t aware that we maintain a set of seven other Second Life grids solely for internal QA purposes. We are constantly moving projects in and out of these test grids during the development life cycle, providing design and operational feedback to the developers. When a project nears its completion, it enters a final verification phase where a detailed test plan is completed by the QA department, and if it fully passes that round of testing, the code is then “merged” down into a branch that is targeted for release. We then use an independent group of testers to reverify that the task is complete in a separate round of QA on the “release candidate” branch. And, finally, that branch is usually made available for a period of time for resident testing on the preview grid. So, in essence, each project, feature, or bug fix undergoes four separate rounds of testing, and must pass each of them.
[15:07] Joe Linden: So, that must have many of you asking “Why are there so many rough spots in our deploys recently?” The answer is fairly simple, really: Second Life has now become a nearly organic thing composed mostly of the varied content created by its residents over a three year period. Our test grids, including the public Beta Test preview, are only a general and limited approximation of the production grid with its 22 TB of objects and assets, and 12-15K concurrent user base. So, even a carefully crafted, thoroughly tested feature may behave in ways we simply can’t simulate or anticipate when deployed to the main grid. As I have said in a blog post or two, our deploy days are much like monitoring a Shuttle launch at NASA in that no matter how much “simulation” or training those teams undertake before a mission, there are always surprises and events that require immediate troubleshooting, quick decisions, and action. We do much the same as we launch new code on every other Wednesday morning.
[15:08] Joe Linden: Without bogging down in details, I can tell you that we are working on a way to deploy and test new code using the richness of the production grid to limit some of the kinds of “surprises” we have had to deal with recently (missing textures, grey avatars, etc.) This effort will go a long way to improving the stability of each release, limit our overall downtime for the deploy, and allow us to correct problems without having to bring the entire grid down. More on that work in a later blog post.
[15:09] Joe Linden: So, to your questions and issues…. Let me anticipate two of your first questions and provide you with some new information.
[15:10] Joe Linden: First: Overall simulator performance degradation and time dilation spikes. Believe it or not, this issue has been a top priority for the development team for over three months. We have had (and continue to have) some of our most senior development talent solely dedicated to finding and fully correcting the cause of this serious problem. My news today isn’t that we have fixed the problem, but I can say the team has advised me today that they believe they have found the cause and have something ready to test later this evening. I can assure you that when we determine that this longstanding issue has been fixed, I will announce it on the blog, and we’ll plan to push it out as a rolling update as soon as practical. We will not wait for the next planned downtime to push this out. We know how serious this is and it’s our top priority.
[15:11] Joe Linden: Second: Our next priority is to identify all of the causes of the, so-called, “silent crashes” such that regions will sometimes crash with no apparent warning or cause and do not properly save their “simstate.” This is serious because it also risks the short-term content loss of objects created but not saved since the last hourly simstate save. We also have an active team working on this problem, and while their solution isn’t as imminent as the performance problem, we believe we now know where the problem lies and may have a fix soon for this equally serious problem.
[15:12] Joe Linden: We’ll keep you posted on both of these critical issues. Now, on to your questions…
[15:12] Robin Linden: OK – open for questions
[15:12] Araldia Primbee:What are the chances of uptime indicators on our islands. This would be handy to be able to know if the sim has crashed while owner offline
[15:13] Joe Linden: I saw this as a suggestion earlier and I think it’s a great idea. We have the information available and it shouldn’t be difficult to surface it for you. I’ll take it as a future feature suggestion. Thanks.
[15:14] Kingsbury Yeats: will we be able to purchase bettter servers or higher performance sims
[15:15] Joe Linden: We’ve already announced that we’re upgrading to a new (Class 5) server design and as we run out of the existing inventory we’ll just start integrating them in. We won’t be purchasing any more class 4 systems.
[15:16] Joe Linden: So, there’s really no upgrade path per se, but I believe Ian Linden posted some additional details in the blog.
[15:17] Kiwini Oe: How have required/recommended system requirements changed over time, and how do you decide when to require more out of client systems?
[15:18] Joe Linden: We haven’t changed our client system requirements in quite some time (over two years) and we continue to strive to support widely available systems. They do need to meet those requirements, which means residents who attempt to experience Second Life with a system that doesn’t meet the published spec will probably not have a great time. But, we don’t intend to climb the “spec ladder” to require even more modern equipement than we already do.
[15:20] Jesse Murdock: How do you intend to deal with the bandwidth deficiency caused by the rapidly growing number of residents, and why are all sims not getting 1 Gb ethernet gear throughout, only the new class 5’s
[15:20] Joe Linden: We plan to replace equipment every three years (we do it more frequently than that now) but there really isn’t a bandwidth bottleneck on the simulators today (and none are Class 5s) — the system is architected to provide sufficient bandwidth with the 100MBps ethernet — we just need to solve the problems I alluded to earlier.
[15:23] Wagahai Oddfellow: Is there a plan for us to be able to see what scripted attachments are worn on our visitors?
[15:23] Joe Linden: Beyond the current top scripts display and beacons?
[15:24] Joe Linden: There is nothing in the works for attachments in that regard.
[15:24] Wagahai Oddfellow: worn scripts are not shown in top scripts
[15:24] Joe Linden: Yes, I understand. Not at this time.
[15:24] Rraven Moonlight: when will we as Sim owners be able to host our own sims locally on our own gear via remote access to the main grid?
[15:25] Joe Linden: we have no timeline for that capability.
[15:25] Umphrey Sachs: why and how often do sims change servers? I notice on some of my land that the sim has changed over the past few weeks.
[15:26] Joe Linden: The system is designed such that any region may come up on any available CPU (of the proper class) out of a pool of “spares”. There is really no way to know which machine a given region is going to come up on and no reason to “peg” a region to a given CPU. Anytime a region goes down for any reason, it may come up on another machine.
[15:28] Dai Vega: are there plans to release a separate application for LSL development?
[15:29] Joe Linden: Hmmm… No plans for that, no. There are many plans to improve the scripting language and capability, but not as an external app. Sounds like a great project for an enterprising resident. 😉
[15:30] Effsey Nelson: Q.Island owners are currently unrepresented on the Feature Vote. Could there be a Feature Vote section exclusive to Island Owners only?
[15:30] Robin Linden: That’s a good idea Effsey.
[15:30] Effsey Nelson: thank you
[15:31] Robin Linden: What we’re doing right now is evaluating an open source project managment system to replace the current tool. It would allow a similar voting and proposal structure, and it has categories to make searching considerably easier.
[15:31] Robin Linden: We’re working on authentication issues, and if it passes, we should be able to port the whole system over. It’s a much friendlier system than the current one so I’d like to hold off making any changes to the current tool until we deicde if we’re ready to make that change.
[15:32] Felicity Marat: is there any plan in the works for those of us with customers who run up large bills and then just leave our islands without paying?
[15:32] Robin Linden: I think the real challenge is for you to work out a system for paying
[15:33] Robin Linden: it has to operate outside of the Linden system for now.
[15:33] Robin Linden: of course then the risk becomes the tenants. It’s something you should think about as you work out your covenant.
[15:33] Bill Stirling: question/suggestion: can your billling department consolidate all the individual purchase billing in a month into one monthly billing? it is hard to track in our rl books when you bill each purchase on a different date in the month.
[15:34] Cyn Linden: Hi Y’all! Actually yes Bill, there are plans in the works there. More details after I get an update from the finance team
[15:35] Cyn Linden: We realize it is a pain, I am eager to change they way that reporting works
[15:35] Umphrey Sachs: can you make statement about custom names and/or name changes.
[15:36] Robin Linden: We believe that names play a very important role in Second Life. That’s why we haven’t offered custom names so far.
[15:36] Robin Linden: However as more and more people are starting businesses and need a name to identify with the business, and also as we continue to muddy the line between virtual and real, it’s making more sense
[15:37] Robin Linden: We have a plan to allow both the purchase of a unique name and also the purchase of a last name in bulk.
[15:37] Robin Linden: By that I mean to purchase multiple accounts with the same last name. What is not yet done would be the ability to change names. So if you create a unique name you will need to create a new account.
[15:38] Jesse Murdock: clarification: if a sim comes up on a random machine and isnt pegged to a certain one, how do you say that a new estate bought after the upgrade get s a new machine?
[15:38] Joe Linden: Good question… As I indicated, the region will always come up on a machine of the correct Class, so if you bought a Class 5 it’ll always come up on one.
[15:39] Angel Fluffy: Is there any plan to allow Island Owners to set up direct debits (with the donor resident’s permission), so that we can automatically take our “tier” bill from island residents?
[15:40] Jack Linden: No.. not at this time.. we have no plans for that i’m afraid
[15:40] Wagahai Oddfellow: Is there a plan for a SLA (Service Level Agreement) for sim owners?
[15:40] Robin Linden: There is a SLA built into the TOS. I don’t know that it’s possible to build a different SLA in just yet, but I’ll take a look.
[15:41] Nexus Nash: I’ve always been ‘pro’ open spaces in SL. Question: Instead of putting up more ‘voids’ per sim, would it be possible to have ‘bigger’ sims (in area) however maintain them to current limits (15k prims, 40ish avatars)
[15:42] Robin Linden: I’m afraid not Nexus
[15:42] bartelby Pegler: question for Joe: Can you explain how maintenance fixes get back into the code build which is in test for the next release cycle
[15:43] Joe Linden: We maintain a separate code branch for maintenance fixes and (regardless of the planned set of features for a given release) we test the maintenance items on an ongoing basis and roll those into the release branch separately so, they are triaged and fixed based on our assessment of priority, severity and the affect they’re having in-world. I hope that answered your question.
[15:45] Effsey Nelson: Q. When a region crashes, I have no way to see what caused the crash. Could we get a “last” crash report added to the Estate/Region tools?
[15:45] Joe Linden: Effsey: you want to read a crash report and core dump? I dont know that would do you much good actually. Please send me your thoughts on what you’d like to see in an email to email@example.com
[15:46] Chance Unknown: Q: when will appropriate training be given to liasons to help them appropriately address estate owners as paying customers in addition to those that use mainland when concierge is offline?
[15:47] Robin Linden: Right now the liaisons along with Live help are mighty busy.
[15:48] Chance Unknown: yes they make that abundenly pparent when you talk to them.
[15:48] Robin Linden: We know that the current system isn’t scalable, and we also know that it isn’t possible any longer to offer the same support services to everyone. how we address that is still being discussed.
[15:48] Robin Linden: We have a communitiy roundtable meeting coming up next Thursday afternoon. I blogged about it recently.
[15:49] Robin Linden: There’s a discussion going on in the blog comments, and we’ll continue it at the meeting. I think, given the interest level, that I’ll schedule additional meetings to talk with people as we go.
[15:49] Elliott Eldrich: Every so often I have strange things happen on my islands. At those times, I request help from a linden, and the solution invaraibly becomes “restart the sim”. Any reason I should not just go ahead and do a restart on my own when I feel it is necessary?
[15:51] Joe Linden: That is usually a course of last resort — and it isn’t always necessary to do that to isolate the problem and find a more permanent fix.
[15:51] Kiwini Oe: Will there be a predictable lead-time for new island purchases, so we can set our clients’ expectations better when we are ordering new islands?
[15:52] Cyn Linden: Actually, yes. We have a better supply of hardware now, so we can keep a more consistent stock level.
[15:52] Dai Vega: are there plans for Estate Tool like access from the website, so that banning, adding estate managers, etc. can be done wihtout logging into the SL client?
[15:53] Joe Linden: Dai: what you describe is exactly where we’re headed with many of the “under the cover” changes we’ve been making to the platform …
[15:53] Joe Linden: I can’t announce timing for anything specific yet, but you will see many features like that available when we’re done with that work.
[15:54] Chance Unknown: Q: since estate customers are expected to fill the function of ombudsman of the regions under their control, when will we have the remainder of the linden menus open to us to assist with those functions?
[15:54] Robin Linden: Whether we do it with additional Linden menus or some other way, we know you need additional controls.
[15:55] Robin Linden: For example, being able to better manage disputes is something you’ll see
[15:55] Chance Unknown: this includes diagnostics too probably
[15:55] Robin Linden: Yes, Chance, it could also include diagnostics, so both technical and social controls and of course the billing controls we already mentioned
[15:56] Jana Fleming: Any plans to hire more concierge staff and possibly extend the service into a 24 hr one?
[15:56] Robin Linden: We definitely plan to offer 24/7 service. When that will happen I can’t say yet.
[15:58] Angel Fluffy: Q3: any plans to fix the ‘run other scripts’ setting so it works all the way up to the parcel ceiling and thus is effective at stopping attachments in skyboxes, therefore reducing lag at popular places in the sky?
[15:59] Joe Linden: Angel: I’m not aware of that particular issue, but let me take it and look into it. I’ll get back to you on it.
[15:59] Jack Linden: Okay folks, i think that’s it for this meeting, thanks for taking part!