Wufoo Status
Server Move
As planned, our server move has begun. We’ll be working on it over the next 4 hours, and we’ll update this post if there are any advances/delays in the time frame.
UPDATE
Three and a half hours in, all services have been restored. Everything went as planned, and all accounts should be functioning as expected. Thanks for your patience as we went through this move.
Power Problems
We just received word that there are power problems at the data center. All customers at the center are affected. Will know more soon.
*Edit on 5/19 3:30 EDT*
It appears the unthinkable is happening again. As we were evaluating the downtime yesterday, power went out again. The root of the problem bypasses our redundant circuits, and we are completely dependent on a third party.
*Edit on 5/19 5:00 EDT*
Service has been restored. We will be following up on the Wufoo blog with more details about what has happened over the past two days. A brief overview is below:
On Monday night, and again on Tuesday afternoon, we had a power outage at our data center. Please note that that we have redundant circuits that are supposed to be on two independent power systems, but all circuits went down. Without a power source, all service became unavailable.
The downside to a power loss on this scale is that all core level services are affected. This significantly increases the time to get all servers online. Networking on critical level servers must be brought up first, and then all application level servers go through a crash recovery process.
We are currently looking for answers to why redundancy with the power did not work, and where there is room for improvement on the recovery process. We will follow up on our blog with a much more detailed post that will address all of the questions we have been receiving.
New Database Server - 5/15/2009
At midnight we made the switch to a new primary database server (our master lookup database). We tried to do this with no downtime, but it didn’t work as expected. Wufoo was unavailable for 37 minutes during this cutover.Outage Earlier Today
Monday, May 11th, 2009, we were down between 5:00pm and 6:15pm EST. The trouble began when one of our DB servers had a process list build up that began affecting the web servers. For about 10 minutes, starting at 5:10pm, the DB server recovered and the site was responsive. We thought the problem was fixed. Shortly after, we found out the server could not fully recover, so we began the reboot and crash recovery process. This, combined with the rebuilding of cache, took the remaining time.
We have a few leads on potential symptoms that could have caused this downtime. We’ll be investigating and hopefully coming up with a concrete answer. We apologize to everyone who has been affected by this outage.
Unexpected Reboots
Tonight, April 29th 2009 at 12:30, we had to reboot one of our primary servers. The service was unavailable for a few minutes while this took place. We apologize for any inconvenience caused.Server Reboots
We will be doing a round of reboots and general maintenance to the servers tonight, 4/16/09, at 7pm PDT. There will be a brief window when Wufoo will be unavailable. Plan for roughly 15 to 30 minutes of unavailability for your forms. Normally, this sort of maintenance can be done with no downtime as we have redundant servers and can bring down one at a time. But with the settings we have to change tonight, we have to ensure no submissions are received as there could be problems with data integrity if that were to happen.Free Plan Outages - 4/2/09
We will be doing maintenance tonight that will affect new free plans created within the past 3 weeks. There are also some very old accounts that will be affected by this as well. We are streamlining everything into our new server configuration. These recent accounts are affected because they were temporarily created in a holding server. No paid accounts, and a majority of free accounts will not be affected by this downtime at all. To those of you who are without forms, we apologize and will get your account working as soon as possible.Downtime Tonight
This post is to inform you that we will be performing maintenance on Wufoo tonight. The scheduled outage is:
Saturday, March 27th, 2009 12:30am EST - 1:30am EST
While the scheduled time slot is one hour, we do want to alert you to potential uncertainties. The maintenance could take as little as 10 minutes, or it could span longer than the hour scheduled. If it does take longer, we will update our status blog. The downtime is for planned changes that should improve the I/O performance of our main database server. We have been having on and off problems for the past 3 weeks, and we intend to permanently fix the problem so that you can rely on Wufoo without doubt.
Unexpected Maintenance
Edit - 1:18pm EST - All servers are back up. Any accounts on the affected servers will be slow as the cache rebuilds, but all functionality is there. The speed will get progressively faster over the next 10 minutes.
It is 12:55pm EST, and we are currently performing maintenance on a couple of our servers. Not all accounts are affected, but about 60% of forms will have problems loading. The maintenance was triggered by extreme slow downs in page load times. The root cause of this issue is from a migration we are doing to more powerful hardware. The older hardware couldn’t handle the migration during peak hours. We have since cancelled the migration (and will resume at night time) and reboot our servers. Everything should be back to normal within 30 minutes, if not sooner.
