Database / Page Load Issue
Thursday, February 18th.
Earlier this morning, for about 5 minutes, some pages were not fully loading. The problem took care of itself, and we investigated the root cause. After noticing the master lookup database was trying to read from disk occasionally, we decided the best course would be a reboot so that we could watch the memory usage upon restart. Our planned reboot was for midnight.
During the afternoon, around 3:45 EST, the problem became too severe. We had no choice but to reboot ahead of schedule. Unfortunately, this took the entire site down as that database is critical to core functionality. To make matters worse, the slave copy was affected by the same problems. Entire site functionality came back online at 4:30 EST.
There are two issues we must address here:
1. What originally caused the database to read from disk? While we investigate, we’re also placing orders for more hardware to have on hand.
2. Our downtime page should show proper styles (http://www.flickr.com/photos/wufoo/3665372897/). Something has changed in our new environment that is causing this to break. Our initial tests worked, but this live scenario did not. Perhaps it was the forced reboot. We’ll be finding out the root cause.
We’re sorry for any inconvenience caused. Please let us know if you have any follow up questions at http://wufoo.com/support, and thanks for your understanding.
