Lesson 7.5: Converting text files: line endings and encodings
Why text files from Windows misbehave on Linux: CRLF versus LF line endings, seeing invisible characters with cat -A and file, the 'bad interpreter' error, converting with dos2unix, unix2dos and tr; then character encodings: UTF-8 versus the old Windows-1254 Turkish code page, and converting with iconv.
17 min
You will learn to
- Explain LF and CRLF line endings
- Detect CRLF with file and cat -A
- Explain the 'bad interpreter: ^M' error
- Convert line endings with dos2unix, unix2dos or tr -d
- Explain character encodings and UTF-8
- Convert a Windows-1254 file to UTF-8 with iconv
Key terms
- Line ending (newline)
- The invisible character(s) that end a line in a text file.
- LF
- Line Feed, byte 0x0A, written \n. Linux, macOS and other Unix systems end lines with LF alone.
- CR and CRLF
- Carriage Return, byte 0x0D, written \r and shown as ^M. Windows ends lines with CR followed by LF (CRLF).
- dos2unix and unix2dos
- Convert CRLF to LF and back, in place. -i reports the counts, -n IN OUT writes a new file.
- cat -A
- Shows non-printing characters: $ at line ends, ^I for tabs, ^M for CR, and M- for bytes above 127.
- Character encoding
- The rule that maps characters to bytes. The same bytes read with a different encoding give different letters.
- ASCII
- The 1963 American standard: 128 characters (English letters, digits, punctuation, control codes), one byte each. No ş, ğ or ı.
- Code page (Windows-1254, ISO-8859-9)
- Old one-byte encodings that use the 128 free values above ASCII for one language group. 1254 and ISO-8859-9 (Latin-5) are the Turkish ones.
- Unicode and UTF-8
- Unicode gives every character of every script a number; UTF-8 stores those numbers in 1 to 4 bytes, keeping ASCII unchanged. It's the default on Linux and the web.
- iconv
- Converts text between encodings: iconv -f FROM -t TO [FILE] [-o OUT]. iconv -l lists the names it knows.
← Transferring files: wget, curl, scp and rsync · All lessons · Editing files with nano →