[TRANSCRIPT] Cory Linden Town Hall

Thanks to all who attended our Technical Town Hall with Cory Linden, whether you were onsite inworld or listening to the Linden Town Hall Closed Captioning group IM (apologies for the group IM borkage, we’re working on fixing this too, see Issue Tracker bugs like VWR-513).

Also, Cory previously shared details of our progress, and you can view fixed bugs reported on the Issue Tracker, sorted by votes.

In case you missed it or would like to catch up, read on for the transcript, thanks to Socrates Linden for this cleaned-up version:

Jeska Linden: Today’s Town Hall is Text Only -
Jeska Linden: All questions should be submitted via the Linden Town Hall Questions group – the group has open enrollment, simply join and paste your question into the Group IM. Questions will be answered in the order they appear, duplicate questions and any technical support or individual support questions will be skipped.
Jeska Linden: If you’d like to follow along with the Town Hall, please join the Linden Town Hall Closed Captioning Group, where we will be repeating the discussion.
Jeska Linden: We will be posting a full transcript to the blog after the event, you may also want to read Cory’s Pre Town Hall blog post here: http://blog.secondlife.com/2007/05/02/cory-linden-town-hall/
Jeska Linden: Also, please hold offers of friendship and IMs to Cory until after the event, as it can be disctracting!
Jeska Linden: Ok, Cory, take it away.
Cory Linden: Good afternoon, everybody… amazing to see so many avatars in one place
Cory Linden: And I think since I put up the blog post we should go straight into questions
Cory Linden: I know you will have a lot of them, so we’ll go through as many as we can
Faith Matova: somebody do me a favor and ask them at least why they’ve waited almost 2 years to upgrade to MySQL5?
Cory Linden: Because until very recently the performance limitations of MySQL 4.*’s multithreading
Cory Linden: hasn’t been the high nail on performance
Cory Linden: plus we wanted to make sure it was very stable before we switched
Cory Linden: the process of upgrading is now ongoing, but will take time with 50+ MySQL instances running
Jeska Linden: Daaneth Kivioq: I wanted to konw – since it is clear that permromance issues are closely related to concurrent log ins, why throttling is not beeing used more agressively – i.e. – starting booting non-verifieds when the load gets high?
Cory Linden: We’re trying to be careful about not using it too aggressively and building metrics to better know when to use it. The building of the metrics is ongoing.
Wrestling Hulka: One of the problems I have with the client is the integration of the Mozilla framework for the browsers. From my estimate the inclusion of Mozilla has increased the size of the client download by at least 10MB and increased the memory leak. As of the current client there aren’t many instances where the internal browser is used, instead we are prompted to use an external browser. When can we expect all the external links be sent to the internal browser to justify the great size of the client, and memory leakage? I’m speaking specifically about the My Account History, Manage My Account, Scripting Guide, Scripting Portal, Official Linden Blog found on the Client menu. And if there are no plans to use the internal browser more efficiently, then what is the justification for having it?
Cory Linden: woah
Cory Linden: As we have talked about in thepast, the integration of firefox has taken *way* longer than any of us expected
Cory Linden: we aren’t the only folks who have found this difficult and the good thing about firefox 2.0 is that they are working on making that process easier
Cory Linden: we have an external team working on it as well, so our goal is to be doing a more significant release soon. However…
Cory Linden: as I pointed out in the blog post, most of our development resources are focused on existing bugs and scaling, so firefox has been a bit starved for resources
Kage Seraph: Cory, if you were a super hero, which would you want more, the mutant ability to scale things at will or the ability to make them more stable by the power of your mind?
Cory Linden: scaling versus stability isn’t really a choice… I would want to do both
Rob Adelaide: What impact will the impending voice system have on the main grid as we know it now?
Cory Linden: Since voice is using external servers for mixing and sends a very small stream to each client, the impact of on teh grid
Cory Linden: as a whole will be minimal from a scaling standpoint. From a use/business/community standpoint, the impact will be incredible…
Cory Linden: we’ve been doing all our linden meetings in the voice client for a month+ and it is amazing how well it works…
Cory Linden: in fact it has had a measurable impact on development, too, since multiple groups are doing their morning standup mtgs in the voice client as well.
Sweet Valentine: Can we please have a way to back up our own inventorys things we buildto help with inventory load and losses? im not asking for backing up copy items just teh items we make ourselves and also what is being done about inventory losses there was a patch posted that they said was fixing this but people are still losing items daily ?
Cory Linden: Agree that backing up items you have full perms on is a good idea. We really feel it is important to build that system in a way…
Cory Linden: that scales and is robust enough to use more broadly. SL has suffered from rapidly deployed systems that we then have
Cory Linden: to rely on for years and years –XML-RPC for example — so this isn’t a quick fix. Would like to work on it in Q3, however.
Katie Welles: As you’re working to fix “Search” — can you do something about “abuse of key words?”
Cory Linden: We are starting on a major reworking of search in SL. We’ve talked in the past about how search in sl is a different problem than on the web…
Cory Linden: but we think we have a good approach that will be more resilient to simple spam-based attacks. More details as the design is refined.
Digital Digital: I would like to know whyat are we supposed to do when we lose such an expensive item in second life for example I bought a slingo machine for 16KL and it’s gone, I have tried everything also reported it many times and never get a responce back in e-mail or anything.
Cory Linden: As is spelled out in the terms of service, you do create content at your own risk in SL. We are making every effort to reduce that loss and I am sorry that it has occurred…
Cory Linden: and numerous recent releases are directly addressing underlying stresses and scaling issues in inventory and assets.
Kylix Petit: Where does stability and performance fall in the priorites for LL development…. There is a problem literally every other day and it seems like there is VERY much room for improvement in this regard.
Cory Linden: As I spoke about in the blog post, 69% of the development staff at LL are currently on scaling and stability…
Cory Linden: and that percentage is rising over the next few weeks. It is the highest priority and the focus of the majority of our design, coding, and QA work.
Cory Linden: We are also working on the next gen architecture to allow far more scaling thean the current design.
Vienna James: I am concerned about releases that have blatant issues, why are they being missed in Beta?
Cory Linden: Many problems only appear when we hit high load. Part of the solution to this is better synthetic testing, but the beta grid exists for people to expose problems…
Cory Linden: You can all help by exploring the beta grid and posting bugs and repro cases to the public jira.
Lewis Nerd: When can we expect to see “Traffic” removed? It serves no purpose, is entirely inaccurate and gamed, and most likely adds to many other problems we have
Cory Linden: Removing the current traffic metric is part of the new search project, so the goal is to pull it as soon as possible.
Spank Lovell: I’d like to ask what Quality control measures LL has in place to police the testing and release of both client updates and patches the the SL client and backend systems. It seems to me that each release introduces issues or breaks previously stable features?
Cory Linden: As I mentioned, the challenge with a system as complex as SL is testing w/o full load. As we continue to builkd out the dev team…
Cory Linden: We will be working on better synthetic tests and benchmarks. In addition, nearly all new/refactored code is being …
Cory Linden: written with unit tests, whcih help. But the best way for you all to help make SL better is to use the beta grid and report repro cases when you find them.
Cedric Boyd: Since i’m not getting any answer from the support concerning this question: when will the map be updated again? The last map from the sim i live on was up2date about four weeks ago. Live Help didn’t answer to my question and the support still not too .
Cory Linden: We know about the bug and it is actively being worked on. We will try to provide more details after the meeting.
Sascha Vandyke: Can’t the patches be tested on a beta grid, because they always render the whole sim unstable?
Cory Linden: Patches are tested on the beta grid, but often times bugs do not appear until full load testing.
Duckling Kwak: Is the explosive land expansion a factor impacting grid performance? If yes, has LL evaluated other alternatives to managing the rising cost of land? The increased land has had a modest impact on land prices. If it’s costing SL more in performance, it’s not a worthwhile tradeoff.
Cory Linden: Not really. There are a couple of load points related to sim count, but one of them was removed last week and the other is being designed away.
Nite Zelmanov: Does the new search functionality being worked on include the discontinuation of traffic based rankings to illiminate the camping/bot impact of people gaming that system?
Cory Linden: Traffic was our first attempt to create relevance ranking and it clearly is imperfect at the best of times….
Cory Linden: we’re pretty confident that we can generate better relevance ranking using other approaches.
Cory Linden: we’re pretty confident that we can generate better relevance ranking using other approaches.
Rita Cummings: Is there any project about improving Sound in Sl? Your sound component seems to capable of much more then what the current functions allow as to do. Will you expose more functionality in the near future?
Cory Linden: Voice is going to generate a lot of new music and audio options.
Cory Linden: and our effort is there more than other audio features.
Cory Linden: but sound bugs are going into public jira nad the studio focusedon bugs will look at them.
Chaos Mohr: What is being done to address land database issues which result in things such as parcels being paid for and not transferring, parcels being sold to multiple people etc? This seems to be a major technical issue which keeps cropping up and hasn’t been fixed yet.
Cory Linden: These are likely symtpoms of the broader stresses on the central db and transactions..
Cory Linden: as I mentioned int he blog post, we are attacking these problems in two ways, to first identity weak spots in the current system and to build a better transaction system as well.
Robaato Yoshikawa: do MySQL instances are load balanced, or the architecture is based on only assigning different server to residents, if the answer is no, have LL looked at tools like Sequoia: http://sequoia.continuent.org/HomePage
Cory Linden: Right now the different residents go to different servers. Ops will get the sequoia pointer, but more broadly we want to go a more scalable architecture long term. More details on that very soon.
Magi Merlin: What focus are you putting into improving pre-release testing in order to reduce our pain?
Cory Linden: We’ve hit this one a few times. It is the majority of our design work, we post all potential code to Beta grid, and we hope that more of you hit the beta grid to test it.
Sharrah Brendel: What about the loss of live help..it has been inpossible to contact a live person in SL for over two weeks. I have left 5 tickets and never heard a word…Where have all the Lindens gone?
Cory Linden: There’s an extensive blog post about that, will post into chat in a moment.
Torley Linden: http://blog.secondlife.com/2007/04/26/introducing-liaison-specialist-teams/
Torley Linden: http://blog.secondlife.com/2007/04/25/support-update/
Torley Linden: ^ See those blog posts for more details.
Keiki Lemieux: I often have upset customers who have lost animations or other no copy items when trying to transfer them from their inventory to a HUD. Other customers have lost their HUD altoghether. Since yesterday’s update, I’ve had 3 customers contact me with inventory loss issues. Two questions, 1) Is there a particular resource that I can point someone to who is experiencing inventory loss issues, like a page in the wiki? and 2) If you are noticing inventory loss after users attempt to move items in a particular way, can you provide a list of suggestions that I can pass on to my customers to avoid inventory loss?
Cory Linden: There are inventory related bugs on the public jira, also we have been putting a lot of code into QA to address some of thos eproblems. For more details, see blog post on transactions.
tx Oh: q1) i still can’t modify scripts in group deeded object but i alowed to edit them. q2) if i change objects inventory, it takes a lot of time to get it updated. q3) the read from notecards is very slow. WHY?
Torley Linden: Public JIRA Issue Tracker – http://jira.secondlife.com
Cory Linden: On specific bugs, please make they are in the public jira issue tracker, please add specific repro cases if you have them, and the studio focused on current bugs will be able to hit them.
Nel Shan: How do the Lindens feel about the Open Letter Project, and what it implies about the state of Linden/resident communications?
Cory Linden: I can’t speak for all Lindens, but as I mentioned in the blog post, I thought that it did a good job of coalescing issues into one thread.
Synack Fitzgerald: has LL considered migrating to a database that scales better that mysql and pays more attention to data integrity (postgresql is free, but oracle or db2 would be viable options as well)?
Cory Linden: We don’t want to work on a model where we attempt to scale a central db. Instead, the effort is to build out a more scalable architecture.
Cory Linden: And MySQL master->slave->slave setusp we use are quite reliable.
Kymber Schnook: The blog indicates that LL will provide an exporter for Maya for the creation of sculpted prims. The range of pricing for different versions of Maya runs from $2,000 to $7,000 USD. Will you make available a more affordable means of export at the time of introduction of the sculpted prim?
Cory Linden: I agree that only being able to export from Maya is suboptimal. We expect other exporters to be written and someone could take the open source client and build an editor there as well.
Cay Trudeau: Do you think arranging physichal server on other continents, like Europe for instance, would ease up on the server load, ease the connections and allow better performance for European residents? Would that also lift the load from sending texture data across the wire?
Cory Linden: We absolutely want to deploy servers overseas. We have an architectural quirk in how we talk to the dbs — a layer of single threaded dataservers –
Cory Linden: that would suffer in situations with greater ping latency, so we need to fix that problem first, whcih is being worked on.
Cory Linden: once that is fixed, we will begin international deployments. This is critical since over 65% of our use is from outside the US.
Daisy Beauchamp: regarding 1.15 and graphic cards in laptops not working: 1. Why is it inconsistent? Why can I login and stay logged in for hours after 5-10 attempts? 2. Will SL ever support other graphic cards, or should I consider selling my Island and counting my losses? 3. Is it possible to use an older version of SL if you have issues?
Cory Linden: Laptop drivers are still an issue for all highperformance 3D applications. Over time, they are improving, but not as quickly as any of us would like.
Cory Linden: We are discussing how to drive the client onto both higher and lower end cards, but right now more development is on system wide scaling.
Doc Nielsen: What we are asking for is that these problems are addressed immediately, ahead of new features, and that we are able to see tangible improvements. Are you going to do this – yes or no?
Cory Linden: As I previously mentioned — and was in the blog post — 69% of development is currently on bugs and scaling. That number will increase this month as we continue to hire.
Cory Linden: Also, this is a good time to remind folks, we are hiring developers aggressively, including offices in Seattle, Boston, and all over the Bay Area.
Cory Linden: Plus, we will always look at smart folks anywhere in the world since SL is working quite well for distributed software development. Oh, Brighton, too!
Sam Stork: Will there ever be a good IDE? Two seperate clients for example, which would make things a lot easier for everyone?
Cory Linden: We are starting the design process for making the UI even more end-user customizable. Plus it is open source, so I think the answer is yes, but scaling/stability is more important right now.
Thorian Pennell: I would like to know if it’ll be possible for group owners to limit chat ability of members?
Cory Linden: Voice chat within groups is on the roadmap, so yes.
Sascha Vandyke: If there’s a database issue and deployment problem, why don’t you put a time limit on free accounts?
Cory Linden: We actually had time limits long, long, long, LONG ago. The problem is that people take a while to learn to use SL, plus great events or builds often bring folks back in weeks/months/years down the road.
Cory Linden: It would be bad for all of our businesses to lose that ability for casual users to come back in. Plus, as we have discussed, free accounts are less of the load than you might expect.
Alexandra Rucker: When will groups truly be fixed? I had to relog 3 times to see the town hall QUESTIONS group, and people still can’t see what’s being pasted to the captions group?
Cory Linden: That is a known bug in the public jira that is being worked on.
Torley Linden: Groups IMs Not Working Properly on the Issue Tracker – https://jira.secondlife.com/browse/VWR-513
Arguss Sleeper: Is it possible to have a “real time” sim status map so that when there is a rolling restart or issuses concerning the ability t0 use the full grid a person would know in advance where not to tp to?
Cory Linden: We currently have an internal tool that does that. We eventually want to be expose portions of that functionality via webservices.
PetGirl Bergman: when wil large groups be able to send outnotices again – mine are over 1000 and have stopped working?
Torley Linden: For many of these bugs, you can look them up on the Issue Tracker (Public JIRA) – Group Notices failing – https://jira.secondlife.com/browse/MISC-37
Katrina Bekkers: since the Open Letter indeed HAD a follow-up from LL, is this *the* way to contact them from now on? since the undercutting of other venues to contact them (and no, a comments-disabled blog and an unresponsive support/tracker don’t count), you’re endorsing resorting to 2000+ residents signed petitions to make issues heard now?
Cory Linden: What Torley said…
Cory Linden: In this case, since scaling and stability are the primary internal focus right now this was an excelletn opportunity for a conversation.
Cory Linden: As the commuynity team has been bloggin about, there are various methods for contacting us and talking to Lindens, including office hours, etc.
tx Oh: over 90% open bug reports are unassigned. why?
Cory Linden: Torley is on this, so we are trying to provide better feedback on when bugs are being worked on. Not all will show the specific Linden who is on it, but we will indicate thos we are on.
Nel Shan: One of the major issues that comes up with inventory loss is the fact that there is no way to verify the loss. This leads to the impression that the Lindens believe that anyone complaining about inventory loss is trying to scam Linden Labs. As well as trying to reduce inventory loss, will there be a way of verifying it introduced?
Torley Linden: Re: bug status, you can read the Issue Tracker instructions – https://wiki.secondlife.com/wiki/Issue_tracker
Torley Linden: See the procedure and the bottom section about “WorkingOnIt Linden”
Cory Linden: That’s a very good idea and I will pass it along to the people focused on the issues.
BlckCobra Shikami: Are our bug reports and crash dumps taken serious and analysed or do you just wait 2 month and send an eMail (like today) asking wether the bug still matter? Does it make sense for us to send bug reports and crash dumps or do we betetr save our time on that?
Cory Linden: Please start focusing that energy on the public jira. That is the best vector for demonstrating repeatable bugs and getting the data to the people working on the problems.
tha Poindexter: Have you considered holding events in the beta grid for better testing? For instance, would it help if these Town Hall meetings were held on that grid?
Jeska Linden: (that’s from Samantha Poindexter – sorry!)
Cory Linden: We have in the past and will continue to do so in the future. Town halls there is a good suggestion.
ascha Vandyke: After the update my island is unstable and support can’t find the issue. Will there be better debug tools coming to fix it?
Cory Linden: In the long run, we want to expose as many of the management and support tools we have for land to the residents so that you can provide better data back to us…
Cory Linden: but that is going to take time.
GreeterDan Godel: people are here expecting answers to their big questions. But it is clear that a lot of people’s big concerns don’t have immediate answers. Should people accept SL as it is and hope for slow change? Or is there reason to expect fundamental changes to the biggest problems any time soon?
Cory Linden: The answer, of course, lies somewhere in between. We are working to fix bugs and enable incremental improvement. At the same time, we are…
Cory Linden: building the foundations for the next gen architecture that will radically improve our ability to scale.
Cory Linden: We also expect that some changes that might seem incremental, such as voice, will actually be transformative in terms of the avaialble design space for residents.
Melchoir Tokhes: What’s the story with the Friends list and the online status indication? Can we expect to see (please) a way where we can check the online status and update the client side on a per-person basis?
Torley Linden: Friends list not working on the Issue Tracker – https://jira.lindenlab.com/browse/SL-40433
Cory Linden: My blog post went into some detail on this. In addition there are details on the public jira. Plus, we are continuing to roll out fixes to thsi system.
Salazar Jack: What will have to happen to eliminate region crossing rubberbanding, for everything from simply walking across a region boundary to traveling between region in vehicles. Is there a way to make that seamless and unnoticeable?
Cory Linden: As many of you know, the region crossing process involves packaging up the objects, scripts, script state and movin ghtem between physical machines…
Cory Linden: there are still some approaches available to us to mask that latency and I feel that we will be able to improve that a lot.
Cory Linden: However, the resources to address that are currently on broader scaling/stability issues.
Michael Dale: Regarding teleporting, it is our only real method of travel within SL. It’s constant breakdowns hinder everyone’s ability to perform basic and routine functions within SL. When will this issue be addressed and corrected?
Cory Linden: We are currently working on better metrics for detecting teleport issues and stomping out the bugs in the system.
Phoenix Psaltery: How does LL plan to deal with urgent scaling issues through 3Q2007 as the concurrent logins continue to grow?
Cory Linden: As the blog post talks about, most of the development effort is on those issues, plus the next gen architecture will enable us to achieve far larger scale than we can today.
Cory Linden: OK, last question…
Radslns Hutchence: As residents, invested in making SL work, what can we do, in addition to being patient, to help with any of these initiatives?
Cory Linden: Patience obviously helps, but more than that, please help the community build the pieces it needs to make SL a better place…
Cory Linden: If you are a programmer, contribute to the open source project or come work for us…
Cory Linden: If you have bugs and can repro them, add data into thepublic jira…
Cory Linden: Help with the wiki and documentation…
Cory Linden: Spend time helping out in the welcome areas and help island…
Cory Linden: As bill and ted would say, be excellent to each other.
Cory Linden: Thank you all very much for all the great questions.
Cory Linden: I will be making blog posts about known issues more regularly as we move forward as well.
Jeska Linden: Thanks for coming to the Town Hall everyone. Please note the transcript from the Town Hall will be posted in the Blog.
Jeska Linden: That’s at blog.secondlife.com
Cory Linden: And as we will be talking in more detail about the next gen architecture, search, etc, soon.
Cory Linden: Thanks, everyone.