We've manually moved the service to the log position before the error occurred. This means we are no longer re-processing old work and the service's performance is back to normal. Unfortunately, we do now have to catch up on the backlog of work from that point.
Analysis results are currently delayed by 30 minutes and we expect to recover within the next hour.
Posted Apr 27, 2017 - 10:11 EDT
Identified
One of our services responsible for writing analysis results to the database, has encountered an error managing its position in its log of work to process. This has resulted in it re-processing from an older point in time. This re-processing is safe, but means we are unable to get to current work at this time.
Users may be experiencing builds that don't appear to start or finish. We're currently investigating ways to manually move the service to the correct position in the log.