We’d like to provide an initial update on our ongoing investigation and analysis related to database instability issues that impacted the Velocity application earlier this week.

On Monday, May 20th, we began to see sporadic occurrences of an unexpected error resulting from a small portion of queries to our AWS Aurora Postgres cluster. These errors resulted in 500 errors when accessing certain Velocity pages, but not at a high enough rate to trigger our alerts. We proceeded to investigate the issue, however. Since the errors appeared to be related to internal Aurora functions, and they subsided independently, we incorrectly considered the incident to be resolved.

On Tuesday, May 21st at 9:28am (EDT), we were alerted to delay in our data processing subsystems responsible for updating the metrics powering our reporting interfaces. Upon examination, we observed elevated load in our secondary instance of AWS Aurora Postgres.

After manual intervention to reduce DB load failed, our site reliability engineers took the action to bring a new secondary instance in an attempt to resolve the issue. This operation was unsuccessful, and after further diagnosis we were able to identify an elevated failure rate for a query accessing a subset rows in at least one table, suggesting that issue we observed on Monday was still ongoing.

During this period of time, we disabled our background processes responsible for data updates and notified customers via in-app messaging and our status page that their data may be temporarily out of date. Additionally, during this period, certain requests to our web interface resulted in 500 errors leading to some portion of site functionality being unavailable.

We worked with AWS technical support to understand what was causing those rows to be inaccessible. Through diagnostics, AWS was eventually able to narrow the problem to inconsistent Aurora internal metadata for a small portion of rows across a subset of our tables. In conjunction with us, AWS proceeded to run a repair operation correcting the internal metadata. Following this operation, and after a period of catching up on data updates, we were able to resume full service of the Velocity application.

We are working with the Aurora team at AWS in order to conduct a root cause analysis to understand what triggered the errors accessing the affected rows. We are also conducting a root cause analysis internally to identify opportunities for improvements in our incident response. We will post the results of this analysis to this status page.