Removing ^M in imported Windows files

Bash - GNU Shell, Tips 1 Comment »

When you import file from Windows OS (or even from old Macintosh OS) you most likely have the ^M at the end of each line. Systems based on ASCII or a compatible character set use either LF (Line Feed, 0×0A, n) or CR (Carriage Return, 0×0D, r) individually, or CR followed by LF (CR+LF, 0×0D 0×0A, rn). Below is a quick list of OS using which convention :

  • LF : UNIX and UNIX-Like systems, Linux, AIX, Xenix, Mac OS X, BeOS, Amiga, RISC OS…
  • CR+LF : CP/M, MP/M, DOS, OS/2, Microsoft Windows (all versions)
  • CR : Commodore machines, Apple II family and Mac OS through version 9

The different newline conventions often cause text files that have been transferred between systems of different types to be displayed incorrectly. For example, files originating on Unix or Apple Macintosh systems may appear as a single long line on a Windows system. Conversely, when viewing a file from a Windows computer on a Unix system, the extra CR may be displayed as ^M at the end of each line or as a second line break.
You can convert with editors relatively small files. For larger files on Windows NT/2000/XP you can use the following command:

TYPE unix_file | FIND “” /V > dos_file

On Unix, a DOS/Windows text file can be converted to Unix format by simply using the tool dos2unix or by removing all ASCII CR characters with the command “tr“.

tr -d ‘\r’ < inputfile > outputfile

You can add an alias to your shell startup script to create easy to remember variations of the tr command for each purpose.

Using bash as an example, edit .bashrc and add these lines.

alias cvtCR=”tr ‘\r’ ‘\n’ ”
alias cvtCRLF=”tr -d ‘\r’ “

You now have two new commands that you can type from the command line.
To try the commands out right away, without opening a new terminal, you
need to tell bash to re-read it’s startup file by typing

source .bashrc

Now you are ready to try out the new commands.

To convert an old MAC file you would type
cvtCR < MACFILE > UNIXFILE This will read a file named MACFILE and create a file named UNIXFILE that has all of the r’s converted to n’s For DOS files, you just want to remove the darn r’s so in cvtCRLF the -d tells tr to delete them. If you wan to update a sample file you can use you favorite editor : VI !

:%s/^M//g

Becarefull, you get the cariage return symbols by the keystroke CTRL+V+ENTER.

How to manage an idle ssh connection ?

Bash - GNU Shell, Tips No Comments »

You probably already have an idle sssh connection that you can’t leave because of  a tcp timeout somewhere on the network or a mistaking command that you have execute. What to do with this idle session ? Easy, just use the exit sequence ~.(tilde and a period)  then you could open a new session properly.

Raid 5 Monitoring on Dell Power Edge 2650 with afacli

Bash - GNU Shell, Case Study 4 Comments »

I had to monitor the Raid 5 status of two PE 2650 with a PERC3/Di and running under a GNU Debian. You can get the CLI tool afacli from dell website. Originally it was only available for windows or red-hat and it was necessary to tweak this stuff using Alien software for converting rpm to deb packages.

<meta content="OpenOffice.org 2.0 (Linux)" name="GENERATOR" /><meta content="Nicolas Brousse" name="AUTHOR" /><meta content="20050926;16592900" name="CREATED" /><meta content="20060109;17115900" name="CHANGED" /> nicolas@grimm:~$ wget<br /> <style type="text/css"><!-- @page { size: 21cm 29.7cm; margin: 2cm } P { margin-bottom: 0.21cm } -</style> <p><font face="Verdana, sans-serif"></font><font size="2" style="font-size: 9pt"><a href="http://linux.dell.com/files/aacraid/afaapps-2.6-0.tar.gz"><font size="2">http://linux.dell.com/files/aacraid/afaapps-2.6-0.tar.gz</font></a></font></p> <p><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif">nicolas@grimm:~$ sudo tar -Pvzxf afaapps-2.6-0.tar.gz</font></p> <p><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"># Normally you should have a device afa0 created. If not, check that devicename = “aac” in the file /dev/</font><font face="Verdana, sans-serif"></font><font size="2">MAKEDEV.afa then do</font> the following step :</p> <p><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif">cd /dev</font></p> <p><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font size="2">./MAKEDEV.afa afa0</font></style> </blockquote> <p><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif">More information on the Dell website : <a title="Dell Linux" href="http://linux.dell.com/storage.shtml">http://linux.dell.com/storage.shtml</a></font></p> <p><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif"></font><font face="Verdana, sans-serif">Now that we have a running <strong>AFACLI </strong>we can do a little script for checking hourly our raid status. Main thing to know with afacli - as lot of other CLI tool - is the <strong>HELP</strong> commande. First you will have to open your device with the command <strong>open afa0</strong> then you could check your container list, disk list etc. Due to network constraint I could only use an SSH connection, so for schedule this script I use a special user named “xfertuser” on each box for doing an<strong> SSH KEY AUTH</strong>. When my user is logon the remote box, I execute some command on <strong>AFACLI</strong> and use a simple <strong>diff</strong> on last afacli log file and the new afacli log file. Then in last step Icheck the messages log for <strong>AACRAID</strong> error messages.</font></p> <p> <a href="http://www.shell-tips.com/2006/10/02/raid-5-monitoring-on-dell-power-edge-2650-with-afacli/#more-7" class="more-link">Read the rest of this entry »</a></p> </div> </div><!--/post --> <div class="navigation"> <span class="previous-entries"></span> <span class="next-entries"><a href="http://www.shell-tips.com/2006/10/page/2/">Next Entries</a></span> </div> </div><!--/content --> <div id="sidebar"> <ul> <li id="search" class="widget widget_search"> <form id="searchform" method="get" action="http://www.shell-tips.com"> <div> <input type="text" name="s" id="s" size="15" /><br /> <input type="submit" value="Search" /> </div> </form> </li><li id="categories-1" class="widget widget_categories"><h2 class="sidebartitle">Categories</h2> <ul> <li class="cat-item cat-item-8"><a href="http://www.shell-tips.com/category/bash-gnu-shell/" title="View all posts filed under Bash - GNU Shell">Bash - GNU Shell</a> (14) </li> <li class="cat-item cat-item-19"><a href="http://www.shell-tips.com/category/benchmarks/" title="View all posts filed under Benchmarks">Benchmarks</a> (3) </li> <li class="cat-item cat-item-5"><a href="http://www.shell-tips.com/category/case-study/" title="View all posts filed under Case Study">Case Study</a> (6) </li> <li class="cat-item cat-item-15"><a href="http://www.shell-tips.com/category/cheat-help-sheets/" title="View all posts filed under Cheat & Help Sheets">Cheat & Help Sheets</a> (3) </li> <li class="cat-item cat-item-14"><a href="http://www.shell-tips.com/category/common-cli/" title="View all posts filed under Common CLI">Common CLI</a> (1) </li> <li class="cat-item cat-item-9"><a href="http://www.shell-tips.com/category/dos-windows-batch-files/" title="View all posts filed under DOS - Windows Batch Files">DOS - Windows Batch Files</a> (9) </li> <li class="cat-item cat-item-20"><a href="http://www.shell-tips.com/category/mac-os-x-shell/" title="View all posts filed under Mac OS X - Shell">Mac OS X - Shell</a> (1) </li> <li class="cat-item cat-item-16"><a href="http://www.shell-tips.com/category/news/" title="View all posts filed under News">News</a> (2) </li> <li class="cat-item cat-item-1"><a href="http://www.shell-tips.com/category/tips/" title="View all posts filed under Tips">Tips</a> (22) </li> </ul> </li><li id="archives" class="widget widget_archive"><h2 class="sidebartitle">Archives</h2> <ul> <li><a href='http://www.shell-tips.com/2008/07/' title='July 2008'>July 2008</a> (1)</li> <li><a href='http://www.shell-tips.com/2008/02/' title='February 2008'>February 2008</a> (1)</li> <li><a href='http://www.shell-tips.com/2008/01/' title='January 2008'>January 2008</a> (2)</li> <li><a href='http://www.shell-tips.com/2007/11/' title='November 2007'>November 2007</a> (2)</li> <li><a href='http://www.shell-tips.com/2007/09/' title='September 2007'>September 2007</a> (1)</li> <li><a href='http://www.shell-tips.com/2007/08/' title='August 2007'>August 2007</a> (2)</li> <li><a href='http://www.shell-tips.com/2007/04/' title='April 2007'>April 2007</a> (1)</li> <li><a href='http://www.shell-tips.com/2007/02/' title='February 2007'>February 2007</a> (2)</li> <li><a href='http://www.shell-tips.com/2007/01/' title='January 2007'>January 2007</a> (3)</li> <li><a href='http://www.shell-tips.com/2006/11/' title='November 2006'>November 2006</a> (4)</li> <li><a href='http://www.shell-tips.com/2006/10/' title='October 2006'>October 2006</a> (9)</li> <li><a href='http://www.shell-tips.com/2006/09/' title='September 2006'>September 2006</a> (4)</li> </ul> </li><li id="linkcat-18" class="widget widget_links"><h2 class="sidebartitle">CLI, Shell & Batch</h2> <ul> <li><a href="http://www.gnu.org/software/bash/" title="Bash is the GNU Project’s Bourne Again SHell, a complete implementation of the IEEE POSIX and Open Group shell specification.">BASH - GNU Project</a></li> <li><a href="http://www.ss64.com/" title="Command line reference for Windows, Bash, Oracle, and SQL Server">ss64.com</a></li> <li><a href="http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ntcmds.mspx?mfr=true" title="Windows XP - Command Line reference A-Z">Windows XP - CLI reference</a></li> </ul> </li> <li id="linkcat-17" class="widget widget_links"><h2 class="sidebartitle">Other links</h2> <ul> <li><a href="http://www.chateau-de-martainneville.info/?rm=hp-en" rel="me" title="The castle of Martainneville is from the XVII century. Its architecture is typical of the ">Castle of Martainneville</a></li> <li><a href="http://www.linkedin.com/in/nicolasbrousse" rel="me" title="View Nicolas Brousse’s profile on LinkedIn" target="_blank"><img src="http://www.linkedin.com/img/webpromo/btn_viewmy_160x33.gif" alt="My LinkedIn Profile" title="View Nicolas Brousse’s profile on LinkedIn" /></a></li> <li><a href="http://nicolas.brousse.info" rel="me" title="This is my Quick & Dirty Personnal Home Page">My Personnal Home Page</a></li> <li><a href="http://www.sur-habitat.fr" rel="friend" title="Sûr Habitat : Prévention des accidents domestiques">Sûr Habitat</a></li> </ul> </li> <li id="akismet" class="widget widget_akismet"> <h2 class="sidebartitle">Spam Blocked</h2> <div id="akismetwrap"><div id="akismetstats"><a id="aka" href="http://akismet.com" title=""><div id="akismet1"><span id="akismetcount">19,129</span> <span id="akismetsc">spam comments</span></div> <div id="akismet2"><span id="akismetbb">blocked by</span><br /><span id="akismeta">Akismet</span></div></a></div></div> </li> </ul> </div> <!--/sidebar --> <div id="footer"> <!--recent comments start --> <div class="footer-recent-posts"> <h4>Recent Posts</h4> <ul> <li> <strong><a href="http://www.shell-tips.com/2008/07/13/using-losetup-and-dd-to-secure-sensitive-data-encrypted-block-device/" rel="bookmark" title="Permanent link to Using losetup and dd to secure sensitive data (encrypted block device)">Using losetup and dd to secure sensitive data (encrypted block device)</a></strong><br /> <small>07-13-2008</small> </li> <li> <strong><a href="http://www.shell-tips.com/2008/02/13/mysql-impact-of-multiple-column-indexes-misuse/" rel="bookmark" title="Permanent link to MySQL - Impact of multiple column indexes misuse">MySQL - Impact of multiple column indexes misuse</a></strong><br /> <small>02-13-2008</small> </li> <li> <strong><a href="http://www.shell-tips.com/2008/01/14/printing-a-sequence-of-letters-or-numbers/" rel="bookmark" title="Permanent link to Printing a sequence of letters or numbers">Printing a sequence of letters or numbers</a></strong><br /> <small>01-14-2008</small> </li> <li> <strong><a href="http://www.shell-tips.com/2008/01/03/how-to-flush-dns-cache-in-mac-os-x/" rel="bookmark" title="Permanent link to How to flush DNS Cache in Mac OS X ?">How to flush DNS Cache in Mac OS X ?</a></strong><br /> <small>01-03-2008</small> </li> <li> <strong><a href="http://www.shell-tips.com/2007/11/26/cheap-secure-backup-solution-for-small-sized-platform/" rel="bookmark" title="Permanent link to Cheap SAN and secure backup solution for small-sized platform">Cheap SAN and secure backup solution for small-sized platform</a></strong><br /> <small>11-26-2007</small> </li> </ul> </div> <!--recent comments start --> <!--recent comments start --> <div class="footer-recent-comments"> <h4>Recent Comments</h4> <ul> <li><strong><a href="http://www.shell-tips.com/2007/09/04/bind-950-patch-dlz-mysql-5-for-auto-reconnect/#comment-14160" title="on Bind 9.5.0 - Patch dlz MySQL 5 for auto reconnect">Tobias Scherbaum</a></strong>: Thanks for this patch, Nicolas :) I included the patch as a ...</li> <li><strong><a href="http://www.shell-tips.com/2006/09/27/delete-old-files-by-last-access-date/#comment-13570" title="on Delete old files by the last access date">chris</a></strong>: Hi, I made some minor modifications...works well but one ...</li> <li><strong><a href="http://www.shell-tips.com/2006/11/25/fine-tuning-a-linux-apache-mysql-php-lamp-server/#comment-13398" title="on Fine tuning a Linux Apache MySQL PHP (LAMP) server">Claudio Izzi</a></strong>: I run a public and free linux shell service for anyone who w...</li> <li><strong><a href="http://www.shell-tips.com/2008/02/13/mysql-impact-of-multiple-column-indexes-misuse/#comment-13261" title="on MySQL - Impact of multiple column indexes misuse">alex</a></strong>: I made a couple graphs like this using RRD, but yours look w...</li> <li><strong><a href="http://www.shell-tips.com/2008/01/14/printing-a-sequence-of-letters-or-numbers/#comment-12840" title="on Printing a sequence of letters or numbers">Naaman Campbell</a></strong>: Cheers, thanks for "putting it" into Google ;)...</li> </ul> </div> <!--recent comments end --> <!--about text start --> <div class="footer-about"> <h4>About</h4> <p>This template is built with validated CSS and XHTML, by <a href="http://www.ndesign-studio.com">N.Design Studio</a>. Icons used here are from <a href="http://www.ndesign-studio.com/stock-icons/web-2-mini/">Web 2 Mini</a> pack. To download more <a href="http://www.ndesign-studio.com/resources/wp-themes/">WordPress Themes</a>, please visit www.ndesign-studio.com.</p> <p>Open "about_text.txt" file in the theme folder to edit this text.</p> </div> <!--about text end --> <hr class="clear" /> </div><!--/footer --> </div><!--/page --> <!--credits start --> <div id="credits"> <div class="alignleft"><a href="http://www.ndesign-studio.com/resources/wp-themes/">WP Theme</a> & <a href="http://www.ndesign-studio.com/stock-icons/">Icons</a> by <a href="http://www.ndesign-studio.com">N.Design Studio</a></div> <div class="alignright"><a href="http://www.shell-tips.com/feed/" class="rss">Entries RSS</a> <a href="http://www.shell-tips.com/comments/feed/" class="rss">Comments RSS</a> <span class="loginout"><a href="http://www.shell-tips.com/wp-login.php">Log in</a></span></div> </div> <!--credits end --> <script src="http://www.google-analytics.com/urchin.js" type="text/javascript"> </script> <script type="text/javascript"> _uacct = "UA-619094-3"; urchinTracker(); </script> <script src="http://trackmon.itor.us/res/mon_analytics.php?v=1" type="text/javascript"></script> <script type="text/javascript"> monitorus_account_number ="215393298"; start_mon_counter(); </script> </body> </html>