python read file encoding and convert to utf-8

install package

1. install libmagic

        $ brew install libmagic
   

2. install python-magic

        $ python setup.py install
   

code read file encoding

• reference determine the encoding of text file

        import magic
  
        blob = open('file.csv').read()
        m = magic.Magic(mime_encoding=True)
        encoding = m.from_buffer(blob)
        print(encoding)
  

code convert file encoding

• reference how to convert a file to utf-8 in python

        source = open('file.csv')
        target = open('file_utf8.csv', "w")
  
        target.write(unicode(source.read(), encoding).encode("utf-8"))
  

  • but after convert this file cannot cat