A recent incident involving a payments service crash in a Kubernetes cluster has underscored the need for automated investigation tools. The crash, which occurred at 1:49 AM, was triggered by a spot instance that caused a cascade of events leading to the service's demise.
Introduction to the Incident
The on-call engineer was paged, and a team was assembled to investigate the incident. The initial 10 minutes were spent reviewing deployment history and configuration, but this line of inquiry ultimately proved fruitless. The team then delved into log analysis, which showed the service starting up normally before being terminated without error.
Log Analysis and Initial Findings
The logs indicated a potential out-of-memory (OOM) kill, prompting the team to check resource limits. However, memory usage was within acceptable limits, and Prometheus metrics did not reveal any unusual patterns. The team then moved on to investigate the healthz endpoint, which they assumed was broken. After 10 minutes of testing, they determined that the endpoint was functioning correctly.
The Breakthrough: Node Events
It wasn't until one of the engineers suggested checking node events that the team stumbled upon the root cause of the issue. A node had cycled at 1:47 AM, causing the Redis cache to restart. The payments service liveness probe, which relied on Redis connectivity, was timing out due to Redis's slow startup time. This realization led to a simple two-line fix in the YAML configuration, adjusting the liveness probe timeout and initial delay.
Reflection and Future Directions
The incident highlights the importance of automated investigation tools in streamlining the debugging process. By correlating signals from different sources, such as Kubernetes events and node logs, these tools can help teams identify root causes more efficiently. The development of tools like Causa, which can receive PagerDuty signals and pull relevant Kubernetes events, is a step towards reducing the time and effort spent on investigation.
AI summary
Ödeme servisi arızasının 47 dakika içinde çözülmesini sağlayan factors ve soruşturma süreci
