Back to overview
Degraded

Authentication API error rates higher than normal

Dec 23 at 05:40am EST
Affected services
Cerebrum vID API
Document Verification API
Authentication API

Resolved
Dec 23 at 06:00am EST

System status is now stable. We're sorry for the inconvenience.

Created
Dec 23 at 05:40am EST

During a scheduled maintenance window intended to optimize our database infrastructure's security and performance, Cerebrum experienced a brief period of service degradation between 05:37 AM EST and 06:00 AM EST. This affected the vID, Authentication and Document Verification APIs. Full system health was restored promptly, and no data was compromised.

What happened?
A subset of users attempting to authenticate or verify documents during this window may have experienced intermittent server errors or request timeouts.

05:37 AM EST: Infrastructure optimization began, resulting in intermittent connectivity.
05:39 AM EST: Connectivity was partially restored (service degraded).
06:00 AM EST: All services returned to full operational capacity.

Why did this happen?
The degradation resulted from a complex interaction between network topology changes and our automated recovery systems:

Topology Propagation: During the migration of database nodes to optimized subnets, network propagation delays prevented the application layer from immediately synchronizing with the new primary database node.

Provisioning Latency: When our automated systems attempted to refresh the application instances to resolve the synchronization issue, a dependency in the deployment pipeline caused a delay in provisioning the necessary application artifacts. This extended the recovery window by approximately 20 minutes.

What were the solutions applied?
Engineering teams intervened manually to bypass the provisioning latency and expedite the service refresh.

We have updated our deployment pipeline configurations to ensure high-availability access to application artifacts during future maintenance windows.

All systems were verified stable by 6:00 AM EST.