Postmortem: EU1 Printing and API Errors
Date: April 21, 2025
Time: 12:00 AM – 3:20 AM PDT (3:30 AM – 6:20 AM EDT)
Status: Resolved
Summary:
On April 21 at approximately 12:00 AM PDT (3:30 AM EDT), we began receiving reports of issues affecting printing functionality and API responses in the EU1 region. Our engineering team immediately began investigating and worked to stabilize the service.
What Happened:
The issue was caused by an unexpected spike in system load, which impacted the performance of both the printing and scheduling services. This led to connection failures and printing disruptions for some users.
This issue occurred independently of any recent infrastructure maintenance activities.
Resolution:
Service was fully restored by 3:20 AM PDT (6:20 AM EDT) after restarting key components and setting up an additional service to manage traffic and restore proper communication.
Next Steps:
We are currently in hypercare and closely monitoring the platform. As part of this effort, we will be implementing infrastructure improvements to prevent similar issues in the future, along with enhancing monitoring and alerting capabilities.
We appreciate your patience and understanding, and we apologize for any inconvenience caused.