Resolved (8:40 AM PT) The issue is resolved. Error rates returned to normal (about 0.014%, the same as before the incident) by 8:30 AM and stayed there while traffic kept growing. The component that routes requests to the right district used less than half of its new capacity at peak.
Root cause: The morning traffic surge exceeded the default concurrency limit of our request-routing component. Requests over that limit were turned away before they reached the Aeries application, which is why users saw a 503 error.
Impact: Some requests failed with a 503 error between about 8:05 AM and 8:30 AM PT. At the peak, around 8:20 AM, roughly 9–10% of requests were affected. Most requests succeeded throughout, and a retry or page refresh would often have worked.
Data: We found no data impact. The errors occurred before requests reached the application, and no data was altered.
Next steps: We already have monitoring and an automated process that requests additional capacity as usage approaches the limit. In this case, traffic rose faster than that process could respond. We are adjusting our capacity thresholds so that headroom is increased earlier, and we are reviewing capacity across the rest of our infrastructure and looking at ways to make the routing step more efficient.
Monitoring (8:28 AM PT) We increased the routing component’s capacity from 1,000 to 5,000 concurrent requests. Rejected requests dropped to zero at 8:28 AM, and we are monitoring error rates as traffic continues to grow.
Identified (8:25 AM PT) We confirmed the Aeries application servers were healthy and passing health checks. We traced the errors to a capacity limit on the component that routes each request to the right district, and we are increasing that capacity.
Investigating (8:05 AM PT) We are seeing intermittent 503 errors for some requests to Aeries SIS and are investigating. Refreshing the page or retrying may resolve the error.