On the evening of March 11th at 20:00 PST , our monitoring systems detected a critical issue affecting one of our cluster nodes. This node encountered difficulties accessing our back-end servers used to store container images, which directly led to a failure in redeploying one of our essential services. As a result, users may have experienced temporary disruptions in service availability. Upon identifying the issue, it was revealed that incorrect firewall rules were preventing the cluster node from establishing a connection with the back-end servers, thus hindering the service redeployment process. The issue was resolved by updating the firewall rules to allow proper communication between the affected cluster node and the back-end servers. We were able to successfully redeploy the service and restore full functionality within a short period after identifying the misconfiguration.
Posted Mar 19, 2024 - 16:35 PDT
Resolved
On March 11th at 2000 PST, we experienced an issue with one of our cluster nodes not being able to access our backend servers, this caused one of our services to not redeploy and caused an issue in re-deploying the service. This issue has since been fixed and the firewall rules have been updated with the change. We are investigating what happened to cause this and will have a root cause analysis of what happened.