Change File Encoding to utf-8 via vim in a script

Question

Hi,

i just got knocked down after our server has been updated from Debian 4 to 5. We switched to UTF-8 environment and now we have problems getting the text printed correctly on the browser, because all files are in non-utf8 encodings like iso-8859-1, ascii, etc.

I tried many different scripts.

The first one i tried is "iconv". That one doesnt work, it changes the content, but the files enconding is still non-utf8.

Same problem with enca, encamv, convmv and some other tools i installed via apt-get.

Then i found a python code, which uses chardet Universal Detector module, to detect encoding of a file (which works fine), but using the unicode class or the codec class to save it as utf-8 doesnt work, without any errors.

The only way i found to get the file and its content converted to UTF-8, is vi.

These are the steps i do for one file:

vi filename.php
:set bomb
:set fileencoding=utf-8
:wq

Thats it. That one works perfect. But how can get this running via a script. I would like to write a script (linux shell) which traverses a directory taking all php files, then converting them using vi with the commands above. As i need to start the vi app, i do not know how to do something like this:

"vi --run-command=':set bomb, :set fileencoding=utf-8' filename.php"

Hope someone can help me.

Answer 1

This is the simplest way I know of to do this easily from the command line:

vim +"argdo se bomb | se fileencoding=utf-8 | w" $(find . -type f -name *.php)

Or better yet if the number of files is expected to be pretty large:

find . -type f -name *.php | xargs vim +"argdo se bomb | se fileencoding=utf-8 | w"

Answer 2

You could put your commands in a file, let's call it script.vim:

set bomb
set fileencoding=utf-8
wq

Then you invoke Vim with the -S (source) option to execute the script on the file you wish to fix. To do this on a bunch of files you could do

find . -type f -name "*.php" -exec vim -S script.vim {} \;

You could also put the Vim commands on the command line using the + option, but I think it may be more readable like this.

Note: I have not tested this.

Answer 3

My first thought was to put those two commands into ~/.vimrc, but apparently it does not work:

prompt>echo æøå | file -
/dev/stdin: UTF-8 Unicode text
prompt>echo æøå | iconv -f UTF-8 -t iso8859-1 > iso
prompt>file iso
iso: ISO-8859 text
prompt>cat ~/.vimrc
:set bomb
:set fileencoding=utf-8
prompt>echo :wq | vim iso
Vim: Warning: Input is not from a terminal
prompt>file iso
iso: ISO-8859 text
prompt>vim iso
... manually entering those commands ...
prompt>file iso
iso: UTF-8 Unicode (with BOM) text
prompt>