python - How to fix "latin-1 codec can't encode characters in position" in requests

Question

I am having trouble with encoding in python 3. When I was testing on my PC I get no errors:

Python 3.7.3 (default, Jun 24 2019, 04:54:02) 
[GCC 9.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> print(requests.get('https://www.kinopoisk.ru').text)

everything good.

But when I ran this code on my VPS a have following error:

Python 3.7.3 (default, Apr  3 2019, 19:16:38) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import requests
>>> print(requests.get('https://www.kinopoisk.ru').text) 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 393-401: ordinal not in range(256)

The python versions are the same. I don't know what is going on.

How do I fix it?

Answer 1

If your environment uses the C os POSIX locale, Python 3.7 automatically coerces that to a UTF-8 aware locale, according to pep-538.

So it seems that your PC has an UTF-8 or C locale set, while your VPS uses latin-1.

Try running the following in an interactive Python session on both machines:

import sys
import locale

print(sys.getfilesystemencoding())
print(locale.getpreferredencoding())

It you do not want to change the locale on your VPS, you could set PYTHONUTF8=1 in its environment, or you could use the -X utf-8 option with Python.