I am attempting to read a CSV file using the standard pandas read_csv function in Python, but I keep encountering a "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 10: invalid start byte" error. It seems like the file isn't encoded in standard UTF-8. Does anyone know how to identify the correct encoding or a way to bypass this error so I can process my data without losing the special characters?
3 answers
The byte 0x87 usually indicates that your file was saved using an encoding other than UTF-8, most likely Windows-1252 or MacRoman. To fix this in Python, you should specify the encoding parameter in your open() or read_csv() function. Try using encoding='latin-1' or encoding='cp1252'. If you are unsure of the exact format, the 'chardet' library is excellent for detecting the encoding automatically. Simply read a small portion of the file in binary mode and let the library guess. This is a very common issue when handling legacy datasets or files exported from older versions of Excel on different operating systems.
Have you tried opening the file in a text editor like Notepad++ or VS Code to see what encoding they detect at the bottom status bar? Sometimes a quick manual check is faster than writing a detection script.
You can also try the errors='ignore' or errors='replace' flags in your read function. It will let the script finish running even if some characters are malformed.
I agree that this works to get the script running, but be careful! Ignoring errors can corrupt your data if the "invalid" bytes are actually important symbols or foreign characters. It is always better to find the right codec if the data integrity is crucial for your analysis.
That’s a practical tip, Brian! If the text editor shows it as 'ANSI', then using encoding='cp1252' in your Python script will almost always solve the issue. I’ve found that many corporate CSV exports still default to Windows-specific encodings rather than the modern web standard of UTF-8, which causes these headaches for data scientists.