We've taken a number of steps over the past few weeks to prevent a recurrence of this issue:
- Installed additional monitoring tools to keep a closer eye on database performance specifically
- Put caching in front of our most expensive queries (which also helped page-load performance)
- Decreased logging (to relieve I/O pressure as much as possible)
- Failed over to a new, re-indexed instance
After doing all this, and not seeing a repeat of decreased insert performance, we're ready to call this incident resolved.
Posted Jul 08, 2016 - 16:04 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 23, 2016 - 12:49 EDT
Update
We're continuing to investigate the source of Wednesday's slow database inserts, but haven't yet identified a root cause or causes. We have identified some expensive queries that we're going to optimize or remove and are working with our database vendor to identify any other problems. While we did see a brief period of slower-than-normal performance yesterday afternoon (2:45-50PM EDT), customer-impact was low with analysis results delayed by no more than 30 seconds.
Posted Jun 17, 2016 - 11:26 EDT
Update
Insert times have returned to normal. We're still investigating the root cause.
Posted Jun 15, 2016 - 12:09 EDT
Investigating
Our service responsible for inserting analysis results into the database is reporting a delay of up to 3 minutes in handling its messages. Customer's may see analysis events and results appear to be delayed because of this. We're investigating.