AM1- BTC API Printing Degradation
Date: May 20, 2025
Time: 18:50 PDT to 20:30 PDT
Status: RESOLVED

Summary
On May 20, 2025, from 6:50 PM to 8:30 PM PDT, the BTC Print API in the AM1 region experienced a temporary outage that impacted printing operations for some users. The issue was caused by a failure in one of the core services responsible for processing API requests. The system has since recovered, and normal functionality has been restored.

What Happened
A failure occurred in the inbound gateway service, which handles incoming API calls related to printing. This failure affected the service’s ability to process requests, leading to a partial outage of the API printing functionality. The issue was initially silent, there were no immediate failure alerts, resulting in a delay before detection and resolution.

Resolution
Once the issue was identified, our team quickly redeployed all instances of the affected service to restore stability and resume operations.

We implemented fixes in the service code to enhance the reliability and responsiveness of the health and readiness endpoints, ensuring early detection and reporting of failures. Additionally, a new metric has been implemented to detect silent failures automatically, enabling proactive intervention and minimizing service downtime.

Next Steps
To prevent recurrence and improve detection, we have taken the following actions:

• Fixed the gateway service’s health check mechanisms to better detect failures early.
• Automated the recovery process for the inbound gateway service to reduce recovery time in future incidents.
• Continued investment in monitoring enhancements to improve visibility across all critical services.

We sincerely apologize for the inconvenience and appreciate your patience while we addressed the issue.