Removing ^M in imported Windows files

  • HOME
  • >
  • LINUX
  • >
  • Removing ^M in imported Windows files
Last Updated: 

When you import files from Windows (or even from old Mac) to Linux, you will most find the ^M characters at the end of each line. Systems based on ASCII or a compatible character set use either LF (Line Feed, 0x0A, n) or CR (Carriage Return, 0x0D, r) individually, or CR followed by LF (CR+LF, 0x0D 0x0A, rn). Below is a list of each OS convention:

  • LF: UNIX and UNIX-Like systems, Linux, AIX, Xenix, Mac OS X, BeOS, Amiga, RISC OS...
  • CR+LF: CP/M, MP/M, DOS, OS/2, Microsoft Windows (all versions)
  • CR: Commodore machines, Apple II family and Mac OS through version 9

The different newline conventions often cause text files that have been transferred between systems of different types to be displayed incorrectly. For example, files originating on Unix or Apple Macintosh systems may appear as a single long line on a Windows system. Similarly, when viewing a file from a Windows computer on a Unix system, the extra CR may be displayed as ^M at the end of each line or as a second line break.

You can convert with a text editors relatively small files easily. For larger files on Windows NT/2000/XP you can use the following command:

TYPE unix_file | FIND "" /V > dos_file

On Unix, a DOS/Windows text file can be converted to Unix format by simply using the tool dos2unix or by removing all ASCII CR characters with the command tr.

tr -d '\r' < inputfile > outputfile

You can add a bash alias to your shell startup script and create an easy to remember variations of the tr command for each purpose. Using Bash as an example, edit your .bashrc and add these lines.

alias cvtCR="tr '\r' '\n'"
alias cvtCRLF="tr -d '\r'"

You now have two new commands that you can type from the command line.

To try your new aliases out right away, without opening a new terminal, you can either reload your shell or use the dot command (aka source).

[me@linux ~]$ source .bashrc

To convert an old MAC file you would now type the command cvtCR < MACFILE > UNIXFILE. This will read a file named MACFILE and create a file named UNIXFILE that has all of the r's converted to n's For DOS files, you just want to remove the darn r's so in cvtCRLF the -d tells tr to delete them.

If you want to update inplace a file, you can do so with vim using the search and replace command. Example: :%s/^M//g.

👉 You get the cariage return symbols in Vim with the keystroke CTRL+V+ENTER.

Related linux posts that you may like...
The Complete How To Guide of Bash Functions
Learn how to write shell scripts with bash functions. This guide includes examples and best practices on how to define, call, and debug functions in bash.
How To Create Simple Menu with the Shell Select Loop?
The select loop is not a regular shell loop. It can be used in Bash to generate a simple menu from which a user can select numbered options.
What is the Right Way to do Bash Loops?
Looping over a list of numbers or words is a building block in shell scripts. Learn how to write Bash loops, including for loop, while loop, and until loop.
What is the Best Way to Count Files in a Directory?
Learn how to count the number of files in a directory using the Linux command line ls, find, and a native bash shell solution with globs and arrays.