EU1 - Printing Slowness via API

Postmortem: EU1 – Printing Slowness via API
Date: May 7, 2025
Time: 2:54 PM – 6:03 PM PDT
Status: Resolved

Summary:
On May 7, some customers using the API in the EU1 environment experienced delays and failures when attempting to print. The Web Portal was unaffected and remained fully operational during the incident. Our team investigated and restored service by 6:03 PM PDT.

What Happened:
An unexpected spike in traffic to our printing API created a large backlog of print jobs that couldn’t be processed in a timely manner. This backlog affected performance for customers using the API, resulting in slower responses and timeouts. To recover, we increased processing capacity.
 
Resolution:
Service was fully restored by 6:03 PM PDT after increasing processing capacity to handle the backlog.

Next Steps:
We are making several improvements to prevent similar issues in the future, including:

• Strengthening API request validation and implementing traffic smoothing to prevent system overload.
• Enhancing monitoring and alerts to detect unusual activity earlier.
• Introduction of a job timeout mechanism to prevent stuck jobs from blocking the entire queue.

 
We appreciate your patience during this incident and remain committed to delivering reliable service.