Kubernetes' hidden spam filter drops events after 25: ideal for that all-nighter debugging session no PM asked for

A recent debugging effort in a Kubernetes cluster revealed an issue with missing events for pods with multiple init containers. The problem occurred when pods had more than 8 init containers, resulting in only the first 8-9 containers' events being recorded. The issue was traced to a hardcoded rate limit of 25 events in the client-go library, which is used by the Kubernetes API server. This limit is designed to prevent event storms, but it can be easily exceeded by pods with many containers generating similar events rapidly. The problem was solved by increasing the limit to 30, allowing all events to be recorded. This issue highlights the importance of understanding the event spam filter in Kubernetes and its potential impact on observability in production environments. The Kubernetes community is working on a solution to make the limit configurable, which would help prevent similar issues in the future.