Postmortem: AM1 - Printing Unavailable
Date: May 9, 2025
Time: 07:20 AM – 09:09 AM PDT (10:20 AM – 12:09 PM EDT)
Status: Resolved
Summary:
On the morning of May 9, some customers experienced issues printing through the API in our AM1 environment. While portal-based printing remained available at first, a second, more widespread issue later disrupted both API and portal printing.
What Happened:
The incident began when an internal configuration issue led to limited capacity on one of our printing services, affecting a small portion of API print jobs. Shortly after, during recovery efforts, a surge of reconnection attempts from print gateways overwhelmed the system, causing a temporary disruption in both API and portal printing.
Resolution:
Our team acted quickly to isolate the issue and restore full service. Printing capabilities were fully recovered by 08:00 AM PDT, and Status page was closed at 09:09 AM PDT (12:09 PM EDT).
Next Steps:
We have restructured how incoming traffic from print gateways is managed, separating it into distinct resource pools. This change improves system stability during periods of high traffic and ensures smoother operation moving forward.
We appreciate your patience and understanding while we resolved this issue.