Status Update
As many of our users have noticed, the site has been having trouble staying healthy lately. Before getting to the details, please note that while we have moved away from critical into a much more stable zone, we are not fully in the clear just yet. We may have to bring the site down for patching 1 or 2 more times this week. If maintenance is required, we will try to give as much advance notice as possible. You have our sincerest apologies for the disruption to your forms.
Recently, an unknown bug was introduced into the application. What it did was copy records when certain actions were executed. First, it would add 2 records, then 4, then 8, 16, 32 and so on. It happened so gradually that none of our alerts picked up on it. Before long, 10 million+ records were trying to be copied. The dataset grew so large that any action we tried to perform would eventually overload the server. This cascaded down to backups, integrations, slow queries, and many other processes we run daily.
For the past few days, we have worked on becoming stable. We have started the paired slave database from a clean wipe to remove the corruption the dataset caused, so replication and backups are healthy. We have also stopped all code from interacting with the dataset, so the site is stable for the time being. The last step is to delete and clean up the dataset. Given all of that, here are the issues that may affect Wufoo during this week:
1. When the database gets overloaded, form submissions can sometimes result in a blank page, or a maintenance error page. This situation usually lasts a maximum of 15 minutes, and only affects a subset of all connections. We don’t anticipate this happening again, but it is a possibility. If you notice it, please report it to support@wufoo.com.
2. Backlog with integrations. There is a chance that some of your third party integrations will take longer to process.
3. Controlled downtime. When we actually try to deal with the large dataset, we may have to take the site offline. We will try to limit these attempts to 9pm PST / midnight EST, and for small windows. We’ll also provide as much advance notice as we can.
Thanks for being patient with us, and for the support. Remember to follow frequent updates on twitter at http://twitter.com/wufoo and to email us any questions at http://wufoo.com/support/
