AM1 - Print Portal Access Issues

AM1 - Print Portal Access Issues
Dates: May 16, 19, and 20, 2025
Times:

• May 16: 11:27 AM – 12:16 PM PDT
• May 19: 07:36 AM – 08:18 AM PDT
• May 19: 11:10 AM – 11:27 AM PDT
• May 20: 06:30 AM – 09:28 AM PDT

Status: Resolved

Summary
At specific times on May 16, 19, and 20, some users experienced difficulty accessing the BTC Print Portal in the AM1 region. These intermittent issues were due to temporary service degradation during peak traffic periods. All services have since been restored and are operating normally.

What Happened
The access issues were caused by a limitation in how the Print Portal server handled increased activity. During each of the incidents, one server instance became resource-starved and was unable to handle incoming requests, resulting in degraded performance and intermittent unavailability for users accessing the portal.

Resolution
To mitigate the impact, our team restarted the affected services during each event to restore access. After identifying the root cause, we applied a configuration fix to address the scaling issue. Additional supporting services are also being introduced to help offload internal calls and improve performance under load.

Next Steps
To prevent recurrence and improve reliability, we have:

• Implemented enhanced monitoring and alerting on the load balancer that distributes traffic across portal servers.
• Set proactive capacity thresholds to detect early signs of performance issues.
• Continued work to introduce supporting services that reduce pressure on the main portal components.

‌

We appreciate your patience and understanding while we worked to resolve this issue.