[Velocity] Data gaps
Incident Report for Code Climate
Resolved
We have narrow down this issue to a bug in our data processing pipeline. We ran an audit process to validate all of our data. And had identified that only a small subset of customers were impacted. We have taken the appropriate remediation steps and we will be reaching out to the affected customers. We're working on improving the visibility of our data processing pipeline so we become aware of any issue as early as possible. We apologize for the inconvenience.
Posted Dec 12, 2019 - 14:59 EST
Monitoring
We have identified that on December 4 around 10:00AM ET a release that introduced a bug on our data ingestion pipeline was deployed and an engineer rolled back that deployment at 10:08AM ET bringing the system back to a healthy state. At the time, we believed that the data pipeline would self recover and the data would be successfully ingested. Today, after becoming aware of some data gaps and further investigation, we have determined that unfortunately, the jobs that failed during the previously mentioned incident didn't retry, causing some gaps. We are currently in the process of backfilling the data. We will be posting an update once this process completes.
Posted Dec 09, 2019 - 23:04 EST
Investigating
We have received reports of missing data. We're investigating and we will share an update soon.
Posted Dec 09, 2019 - 17:10 EST
This incident affected: Code Climate Velocity.