February 2017

No incidents reported for this month.

January 2017

[Scheduled] Infrastructure Maintenance
The scheduled database maintenance was completed successfully.
Jan 1, 14:11-16:25 UTC

December 2016

No incidents reported for this month.

November 2016

No incidents reported for this month.

October 2016

DNS Service Provider Degraded Service
Dyn has issued an update that the DDoS incident is resolved. More information, including a preliminary post-mortem, can be found on their status page: https://www.dynstatus.com. We'll continue to monitor the ControlShift platform and will post any updates.
Oct 21, 16:30-22:28 UTC
DNS Service Provider Degraded Service
This incident has been resolved.
Oct 21, 13:32-13:38 UTC

September 2016

No incidents reported for this month.

August 2016

Email delays
SendGrid has indicated that email delivery is now happening normally.
Aug 30, 21:00-21:54 UTC
Old petitions showing up in moderation queue
Petition moderation queues should be back to normal.
Aug 22, 17:58-21:31 UTC

July 2016

Request Queing
Platform performance is back to normal and we have resolved all known issues.
Jul 22, 20:03-20:15 UTC
Elevated Error Rates
The error rate is back to normal and platform is operating normally.
Jul 22, 19:41-19:52 UTC

June 2016

Heavy Load
We've provisioned additional servers to handle the increased traffic. We'll continue to monitor the situation, but sites should be operating normally.
Jun 24, 18:37-18:50 UTC
[Scheduled] Infrastructure Maintenance
Redis infrastructure has been updated successfully.
Jun 1, 12:00-12:19 UTC

May 2016

[Scheduled] Infrastructure Maintenance
Redis infrastructure has been updated successfully.
May 18, 12:23 UTC

April 2016

No Incremental Webhooks
This incident has been resolved.
Apr 6, 05:49-20:29 UTC

March 2016

Investigating
Service has been restored, still investigating root cause.
Mar 15, 06:59-07:11 UTC

February 2016

Possible outage
We're back online and everything should be working as normal.
Feb 29, 20:02-20:15 UTC
DNS resolution issues
Everything seems to be working again.
Feb 23, 15:37-16:07 UTC
Error pages across platform
We fixed the issues causing sporadic error pages throughout the platform. We'll continue to monitor performance, but the site should be fully operational again.
Feb 19, 18:16-19:21 UTC
Database Migration Causing Performance Issues
We've corrected the issue and the platform should be operational again.
Feb 17, 19:03-19:11 UTC
[Scheduled] Database Upgrade
We've completed the database update with less than one minute of customer facing disruption and the platform is now operating normally.
Feb 9, 13:00-13:26 UTC

January 2016

Intermittent outages
This issue has been resolved. The intermittent outages were caused by malicious users of a customer site attempting to use the platform to post commercial content. We're continuing to monitor the situation, but the platform should be fully operational again.
Jan 11, 17:48-18:15 UTC
Issue with Custom Signature Fields
We've rolled back the issue-causing commit and signature fields should be working normally.
Jan 8, 15:52-16:19 UTC
Continuing Disruption
DNS propagation should now be complete and the platform should be operating normally. This outage was caused by a routine maintenance operation that had cascading consequences that we did not anticipate while applying the change. In response we've updated our procedures to require explicit sign off on production changes before they are applied in addition to tests in our staging environment. We've also discovered that a class of maintenance operations which we had previously believed to be low risk can actually cause persistent outages through side effects that we had not fully anticipated. We're updating our procedures to review these operations much more carefully and develop strategies for making such changes without downtime.
Jan 5, 16:09-16:43 UTC
Brief Outage
We experienced a brief outage caused by routine maintenance this morning. We changed the configuration of our load balancers to add a new larger block of IP addresses to support new customers and other growth of the platform. This change propagated through the infrastructure in a way that we had not anticipated and caused some old ip address ranges to be blocked before the new ones were completely in service. To prevent this from recurring we've updated our maintenance procedures to perform this operation in a multi-step fashion that will allow for zero downtime.
Jan 5, 15:45 UTC

December 2015

Logging in with Facebook not working
This incident has been resolved. Facebook login should be working correctly.
Dec 17, 15:00 - Dec 18, 18:21 UTC
[Scheduled] Infrastructure Migration
The scheduled maintenance has been completed.
Dec 16, 13:38-13:38 UTC
Reports of Network Connectivity Issues
Our upstream provider has indicated that the network issue should now be resolved.
Dec 3, 22:15 - Dec 4, 03:52 UTC

November 2015

Local Groups Forums
This incident has been resolved and forums should be displaying correctly.
Nov 19, 19:38-22:48 UTC
Upstream Network Issues
We're still up and operating normally. Declaring the incident closed.
Nov 1, 16:25-20:10 UTC

October 2015

No incidents reported for this month.

September 2015

No incidents reported for this month.

August 2015

[Scheduled] Database Update
We're now back online with significantly more storage space on our database servers. Maintenance complete.
Aug 9, 12:31-12:39 UTC

July 2015

No incidents reported for this month.

June 2015

Continued Scheduled Maintenance
Our maintenance took longer than expected, but we've completed transitioning our database server to new equipment. All sites should now be fully operational. If you have any outstanding issues, please send us a support email.
Jun 21, 14:04-15:34 UTC
[Scheduled] Database Server Update
The scheduled maintenance is taking longer than expected. We're migrating to new database server equipment, and the time to make that transition is taking significantly longer than predicted.
Jun 21, 12:00-14:00 UTC

May 2015

Problems with Petition Moderation
This incident has been resolved.
May 16, 12:42-13:53 UTC

April 2015

No incidents reported for this month.

March 2015

[Scheduled] Server Update
The scheduled maintenance has been completed.
Mar 4, 21:49 UTC
Intermittent Outages
We're operating normally again.
Mar 2, 21:07-21:18 UTC

February 2015

Platform down
We've resolved this incident, and are now operating normally.
Feb 2, 18:13-19:20 UTC

January 2015

CVE-2015-0235 GHOST: glibc gethostbyname buffer overflow
All systems have now been patched to resolve the GHOST glibc vulnerability. This resulted in less than 1 minute of downtime as the last of our machines were rebooted. Technical information here: http://www.securityfocus.com/archive/1/534555
Jan 29, 14:11 UTC
[Scheduled] Scheduled maintenance on web site infrastructure
The scheduled maintenance has been completed.
Jan 27, 21:00-21:30 UTC
Continuing Load Balancer Issues
This incident has been resolved.
Jan 27, 02:53-02:53 UTC
503 Errors on Customer Sites
While performing routine maintenance our infrastructure provider encountered a problem that caused the platform to become unresponsive. This issue has been corrected, though we are scheduling a maintenance window for tomorrow Tuesday, January 27th at 4 pm EST to perform further improvements on the underlying infrastructure to prevent this issue from happening again. The underlying cause was related to our HA Proxy instances experiencing a "split brain" where both machines thought they were the primary machine that should be serving requests. This is the same issue that caused an outage on December 15th, and we expect once the infrastructure changes are done tomorrow it will be definitely resolved.
Jan 26, 18:37-21:45 UTC
Unplanned Connectivity Outage
Our infrastructure provider has notified us that they experienced network instability after the planned network maintenance window this morning. This caused a second outage after the planned maintenance window: http://status.railsmachine.com/incidents/qn2983sr4m5q
Jan 11, 19:16 UTC
[Scheduled] Data Center Networking
The scheduled maintenance has been completed.
Jan 11, 04:00-04:15 UTC

December 2014

503 Errors on Customer Sites
While performing routine maintenance our infrastructure provider encountered a problem that caused the platform to become unresponsive. This issue has been corrected. The underlying cause was related to our HA Proxy instances experiencing a "split brain" where both machines thought they were the primary machine that should be serving requests.
Dec 15, 19:08-19:21 UTC
Problems with image and file uploads
Connectivity issues corrected, and uploads are working properly.
Dec 10, 19:16-21:20 UTC