Wufoo Status

Downtime to Expand Storage

Thursday, December 15th at 9:30pm PST, we will be expanding our storage capabilities. This will require the entire site to be taken offline for an unknown amount of time. In our best case scenario, we expect the site to be down for 30 minutes. The worst case scenario would put us at a 90 minute downtime. During the downtime, we will bring up existing forms as soon as possible, but possibly limit the ability to create new forms. All public forms and reports will see error messages as shown in this screenshot.

Thursday, December 15th at 9:30pm PST

[ View All Timezones ]

If you’re up or interested, you can follow our progress during the downtime on our Twitter feed.

Quarterly Patching

Wednesday, November 9th at 10:00pm PST, we will be performing routine updates on all of our servers. For two specific servers, we require the site to be offline in order to reliably patch them. Patching for those two specific servers will begin at 10:00pm PST. During that time, we are expecting all services to be unavailable for 10-20 minutes. All public forms and reports will see error messages as shown in this screenshot.

Wednesday, November 9th at 10:00pm PST

[ View All Timezones ]

If you’re up or interested, you can follow our progress during the downtime on our Twitter feed.

Quarterly Patching

Thursday, August 4th at 10:00pm PST, we will be performing routine updates on all of our servers. For two specific servers, we require the site to be offline in order to reliably patch them. Patching for those two specific servers will begin at 10:00pm PST. During that time, we are expecting all services to be unavailable for 10-20 minutes. All public forms and reports will see error messages as shown in this screenshot.

Thursday, August 4th at 10:00pm PST

[ View All Timezones ]

If you’re up or interested, you can follow our progress during the downtime on our Twitter feed.

New Domains and Languages

We’re beginning to roll out new domains and languages. From an end user perspective, you may see yourself redirected from http://wufoo.com to http://wufoo.eu, or some other Wufoo domain depending on your language, and where you are located in the world.

The way redirection currently works is that when you log in, you are directed to the proper domain. This only affects private pages in the Wufoo admin panel. Any public page, such as a public form, will work across all domains. For example:

http://username.wufoo.com/forms/my-form/

and

http://username.wufoo.eu/forms/my-form/

will both work, so there is no need to change anything on your end. The only noticeable thing from an end user perspective is the potential URL change upon login.

Going forward, we will begin to offer language selection, which will also change your domain accordingly. Please direct any questions or concerns to Wufoo support. Thanks!.

Revisiting June 20th Maintenance

Our previous maintenance scheduled for June 20th never occurred because more preparation was required. We have rescheduled the downtime for this Thursday, June 30th. We are expecting the service to be unavailable for 15 to 30 minutes. During the downtime, public forms and reports will see error messages as shown in this screenshot.

Please note that this event is more complicated than our standard patching cycles, so there are two unique circumstances.

• Prior to the downtime, we will have to prepare certain items. The exact time it will take to get things ready is not concrete, so we can’t give an exact time for when the site will go down. We’re shooting for a range between 7pm and 9pm PST, and we will update this post as we get closer to give a better window.
• Roughly 3 hours before the downtime begins, we will have to turn off new form and account creation. Existing forms and existing accounts will function normally during that 3 hour window.

——

Thursday, June 30th beginning between 7pm and 9pm PST (TBD)

[ View All Timezones ]

If you’re up or interested, you can follow our progress during the downtime on our Twitter feed.

Potential Downtime

UPDATE - This downtime never occurred. Will be rescheduled at a later date.

Today, June 20th at 9:40pm PST, we will be upgrading our file servers to allow for more storage space. This change will allow the site to handle more users, forms and file uploads. We will try to do this without any downtime, but there is a chance we have to put the site into maintenance mode for stability. If that is the case, we are expecting all services to be unavailable for 15 minutes. All public forms and reports will see error messages as shown in this screenshot.

Monday, June 20th at 9:40pm PST

[ View All Timezones ]

If you’re up or interested, you can follow our progress during the downtime on our Twitter feed.

Downtime for New Servers

Today, May 24th at 9:15pm PST, we will fully cutting over to our new servers that were added to alleviate some of our recent problems. This change will allow the site to handle more users, and it will also minimize the risk of one server affecting the performance of others. Downtime will begin between 9:15pm PST and 9:30pm PST. During that time, we are expecting all services to be unavailable for 5-15 minutes. All public forms and reports will see error messages as shown in this screenshot.

Tuesday, May 24th at 9:15pm PST

[ View All Timezones ]

If you’re up or interested, you can follow our progress during the downtime on our Twitter feed.

Quarterly Patching

Today, May 19th at 9:15pm PST, we will be performing routine updates on all of our servers. For two specific servers, we require the site to be offline in order to reliably patch them. Patching for those two specific servers will begin between 9:15pm PST and 9:30pm PST. During that time, we are expecting all services to be unavailable for 5-15 minutes. All public forms and reports will see error messages as shown in this screenshot.

Thursday, May 19th at 9:15pm PST

[ View All Timezones ]

If you’re up or interested, you can follow our progress during the downtime on our Twitter feed.

Status Update

As many of our users have noticed, the site has been having trouble staying healthy lately. Before getting to the details, please note that while we have moved away from critical into a much more stable zone, we are not fully in the clear just yet. We may have to bring the site down for patching 1 or 2 more times this week. If maintenance is required, we will try to give as much advance notice as possible. You have our sincerest apologies for the disruption to your forms.

Recently, an unknown bug was introduced into the application. What it did was copy records when certain actions were executed. First, it would add 2 records, then 4, then 8, 16, 32 and so on. It happened so gradually that none of our alerts picked up on it. Before long, 10 million+ records were trying to be copied. The dataset grew so large that any action we tried to perform would eventually overload the server. This cascaded down to backups, integrations, slow queries, and many other processes we run daily.

For the past few days, we have worked on becoming stable. We have started the paired slave database from a clean wipe to remove the corruption the dataset caused, so replication and backups are healthy. We have also stopped all code from interacting with the dataset, so the site is stable for the time being. The last step is to delete and clean up the dataset. Given all of that, here are the issues that may affect Wufoo during this week:

1. When the database gets overloaded, form submissions can sometimes result in a blank page, or a maintenance error page. This situation usually lasts a maximum of 15 minutes, and only affects a subset of all connections. We don’t anticipate this happening again, but it is a possibility. If you notice it, please report it to support@wufoo.com.

2. Backlog with integrations. There is a chance that some of your third party integrations will take longer to process.

3. Controlled downtime. When we actually try to deal with the large dataset, we may have to take the site offline. We will try to limit these attempts to 9pm PST / midnight EST, and for small windows. We’ll also provide as much advance notice as we can.

Thanks for being patient with us, and for the support. Remember to follow frequent updates on twitter at http://twitter.com/wufoo and to email us any questions at http://wufoo.com/support/

Server Tuning

We’ll be performing some checks this evening, May 6th, to try to improve performance, and isolate some issues that we have noticed on a recurring basis. This work will begin at approximately 9pm PST / midnight EST, and we will be performing tests for a couple of hours. We aren’t certain, but this may result in periodic downtime. We don’t anticipate a worst case scenario of more than 1 hour of downtime. If that turns out to be the case, all public forms and reports will see error messages as shown in this screenshot.

We’re sorry for the late notice — we just recently found out about these issues. If you’re up or interested, you can follow our progress during the downtime on our Twitter feed.