Starting about 10 days ago, we began randomly asking around 1000 people per day who log into Second Life whether they feel their experience is getting better or worse, with an additional option to give some keywords or a sentence about why they feel that way. The results so far are interesting and I hope will be quite helpful in helping us to prioritize our work.
As a particular example, the problems we had over the last weekend with a database failure that temporarily broke searching in Second Life and slowed grid performance were immediately measurable as an increase of about 15% in the number of people who answered that their experience was getting worse. As we have gradually repaired and restored service, the satisfaction level has improved.
While you might argue that we should already know (and of course we do) that there are lots of frustrations and bugs to fix in Second Life, this statistic is potentially very useful in quickly reporting the magnitude of impact across the overall population of Second Life to either improvements or problems. Knowing the specific impact of a change can greatly help us to prioritize our work. For example, several months ago we released an update that greatly improved viewer framerates – typically by about 25%. But how much did that change improve satisfaction with Second Life? I would love to have known, so that we could decide how much of our development bandwidth to focus on incremental improvements to framerate versus working on crash rates, for example.
Another appeal of a system like this is that is can give us almost instant feedback when something is going wrong. We have a number of statistics on Second Life (like overall concurrency, for example) that we watch in real time on large monitors in the office and also on internal web pages. When concurrency drops abruptly for any reason, we instantly start looking for problems without needing to wait for any sort of automated alerts to go off. In a similar way, we are going to watch the real-time performance of this satisfaction level with the hope that changes might show us problems that would not be visible in any other way.