Before we hit the weekend, I wanted to update you on our progress on Grid stability. We have taken a number of major steps this past week to improve database performance, which included offloading high impact queries to slave databases, and completing a master database conversion on Thursday. These actions, along with many other performance tweaks, have significantly improved query response times and reduced load on the central database.
While I am optimistic that these measures are helping to stabilize the Grid, I also want to be ready for any event contingency. We have mapped out a detailed response, escalation, and triage plan across the entire company. One of the major changes for this weekend will be the corrective steps we take if the central database becomes overloaded and unstable. Past procedures were to almost immediately block logins, to allow for load to subside and protect the database from a complete crash (which is what happened last Sunday). This is a rather drastic approach to protect the database, and we’ve now put some “throttling” mechanisms in place that would be our first step to reduce load vs. blocking logins. Throttling literally means turning features and functions off, and potentially degrading in world experience. One of these throttles would disable group queries to the central database, and would cause group features to be unavailable during that period of time. While degrading the resident experience is not a preferred step, we believe that it is far less punitive than the drastic measure of blocking logins. So if some in world features are temporarily disabled, you would still be able to log in, and enjoy other in world features that do not tax the central database.
We have an entire support team that is standing by to react to any potential problem, and will aggressively communicate with the residents, through multiple means (in world, web, blogs, etc) to keep you informed of any problems and progress to correct. While I am optimistic that none of these recovery steps will have to be taken, I think it is very important to continue proactively reaching out to the resident community on our plans.
I’ll be in the forums later to answer questions, and continue to thank you for the very constructive and helpful feedback.