Hello everyone, I am finally publishing our Public Metrics in Excel and OpenDocument and Google Doc formats through July 2007. I apologize about the delays in getting these posted, but hopefully the addition of some new data should make this worthwhile.
The new data, as highlighted by Phillip during his keynote at SLCC last weekend, now includes client crash rates, and both server and client performance statistics in terms of Frames Per Second. I have taken the step to publish these figures as Daily statistics for every day since January 1, 2007. As Phillip indicated, we will also be including metrics on Inventory Loss shortly- I expect to include it in August’s data. We will be focusing on discussing these new metrics during my office hours in Beaumont at 10 AM this Friday morning during my weekly office hours- please come ready to discuss them, these meetings have been very fruitful lately and I appreciate everyone’s input. Talk to you tomorrow!
More explanations of the new key metrics below the fold.
(* Edit: Links to XLS and Open Document versions amended- thanks! )
There has been lots of coverage of Phillip’s SLCC keynote and related topics in many blogs and interviews and as he pointed out, there is a huge focus for us on system stability and performance, and these new reports are the beginning of us being more transparent about how we’re doing on those fronts.
The First new tab is “Client Crashes” and it includes 3 new metrics per day – each a percentage of these events occurring per session on the grid. The first metric is “% of sessions that ended abnormally” – which is the percentage of sessions that did not end with an explicit logout event. The second is “% of users that were kicked off a simulator”, i.e. the outcome of the “you have been disconnected from the simulator” error message, which can occur because the sim went down (rolling restarts, planned or unplanned grid outage, or sometimes simulator crashes) as well as communication problems between your client and the simulator you are using, and the third is “% of Logged-in users reporting a crash” which gets populated if you say “yes” to the crash reporter tool and send us details about how/when the crash occurred. The crash reporting is very useful for us to find trends of particular problem code, client hardware, software drivers, and other details that are very useful in debugging. Please continue to send us your crash reports, thanks!
The second new tab is “Server FPS” and it has two metrics regarding simulator performance – “% of daily usage with sim > 35 FPS” and “% of daily usage with sim > 20 FPS”. Server performance can vary anywhere below 40 Frames Per Second, depending on number of avatars using the sim, number of active scripts, use of physics on the sim, and several other factors. Extremely busy sims that are exceeding the number of avatars that can comfortably use it at a single time will see their server FPS decreasing. Any number between 35 and 40 is pretty much maximal server performance, and any number below 20 is a simulator experiencing problems and guaranteed most of the users are experiencing strong “lag”. Typically simulators experiencing lower than 20 FPS will also be experiencing suboptimal “time dilation” figures.
The third new tab is “Client FPS” which will be affected by the Server FPS for the simulator you are using, but much more so it is controlled by your local machine hardware (and network speed) and your client configuration settings. Client FPS is the most direct measurement of your specific experience in second life, whether it be laggy or speedy. It is definitely possible for 2 users on the same server (that has server FPS of 40) to each have drastically different Client FPS due to hardware and settings differences. For residents with slower or older hardware, *please* take the time to turn down your draw distance and change your other graphics settings for second life so that you can experience less lag- you have more control over this than you might expect! We record the average FPS for all client sessions and bucket them into statistical “quartiles”, and report on 3 metrics. “Highest Viewer FPS for lowest quartile” is essentially the best performance that our lowest-performing 25% of users are experiencing. “Median Viewer FPS” is the overall average client FPS. “Lowest Viewer FPS for highest quartile” is essentially the worst performance seen for our highest 25% of users – these are definitely our users with the newer video cards and well-tuned client configuration settings. I know we have users that are experiencing *much* higher than 25 FPS with very fancy new hardware. If you are experiencing great FPS, talk to your friends about your settings and your choice of hardware- you might be able to help them out! In this Client FPS data you can see a great trend of better performance that occurred around the end of March when we release our new texture pipeline code.
I look forward to talking to you more about this new data tomorrow morning, thanks for reading!