AM1 - Printing Error

Root Cause Analysis (RCA)

Incident: AM1 – Printing Error

Date & Time

• Monday, Jan 12: 13:50 – 14:38 PST

Incident Summary

On Monday, January 12, AM1 experienced a printing error that impacted printing functionality for a short period. The issue was isolated to a specific service and did not result in data loss.

Root Cause

The issue was caused by a disk space limitation within Kubernetes for Windows containers, where a soft limit of 20 GB was reached during normal operation. 

As disk usage increased, temporary files and unused assets accumulated within the affected containers, which led to degraded performance and printing failures.

There is an underlying infrastructure configuration that limited the disk space available to the service. As normal operational data accumulated, the service was unable to write additional data to disk, resulting in degraded performance and temporary service failures.

Detection & Diagnosis

The team observed printing failures in AM1 and identified elevated disk usage within the affected btxmlexecutor containers. Further investigation confirmed that the container-level disk limit had been reached.

Corrective Actions

• Removed unused fonts and temporary files from the affected containers
• Disk usage returned to safe levels and printing functionality was restored
• The issue was fully contained following cleanup 

Resolution Status

The issue has been resolved, and printing in AM1 is operating normally.

The system remains stable under monitoring.

Preventative Actions

• Implement production monitoring of container file system usage in the short to mid term
• Proceed with modernization of print services, separating stateful data from stateless microservice containers, to prevent similar issues in the future 

Final Notes

• No data loss occurred
• The issue was isolated and short-lived
• Longer-term architectural improvements are planned to reduce recurrence risk