Unix - delete file with hyphen at beginning

Accidentally one of my script created a file named '-1264924755.done' (i.e. filename starting with a hyphen).

When I tried to remove this file:

$ rm -1264924755.done

rm: invalid option -- 1
Try `rm ./-1264924755.done' to remove the file `-1264924755.done'.
Try `rm --help' for more information.

This is because the command line parser treats the filename as a command line switch (because of the hyphen). So commands like mv, rm etc treats the file name itself as a parameter(option) to the command.

The ways to delete such files:

1) Referencing the filename to the command via its path location (so that the special character is not the first character after the white space)

$ rm ./-1264924755.done

or

$ rm ~/work/temptest/-1264924755.done

2) A special way:

$ rm -- -1264924755.done

Related posts:

- Find last modified directory in UNIX
- Grep and print control character characters in UNIX
- Find process running time in UNIX
- Bash - save command in history without executing it
- Display file permissions in octal in UNIX

Sed - save changes to same file

Sed receives text input, either from stdin or from a file, performs certain operations on specified lines(or all lines) of the input, one line at a time, then outputs the result to stdout or to a file. Today I am going to show how we can use sed to do some operation on a file (mainly substitute, which is the most popular with sed) and write back the results to the same file.

Input file:

$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active

Lets try to replace the word 'VERSION' in the above file with '8.04'

$ sed 's/VERSION/8.04/' file.txt
port:9903
os-version:8.04
codename:hardy
status:active

So, be default sed outputs the result to 'stdout'.

Append or redirection to the same filename will be wrong !!

$ sed 's/VERSION/8.04/' file.txt > file.txt

Newer sed versions (e.g sed version 4.1.4), there is a useful command line option:

-i[SUFFIX], --in-place[=SUFFIX]

Description: edit files in place (makes backup if extension supplied)

Lets try this option:

$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active

$ sed -i 's/VERSION/8.04/' file.txt

$ cat file.txt
port:9903
os-version:8.04
codename:hardy
status:active

It worked; the result is printed to the same filename.

We can also mention the backup extension like this:

$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active

$ sed -i.bak 's/VERSION/8.04/' file.txt

$ cat file.txt
port:9903
os-version:8.04
codename:hardy
status:active

The original content of the input file is backed up here:

$ cat file.txt.bak
port:9903
os-version:VERSION
codename:hardy
status:active

With older version of 'sed' editor (where this -i option is absent), we can write the result to a temporary file and then in the next step we can move the temporary file back to the original file like this:

$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active

$ sed 's/VERSION/8.04/' file.txt > file.txt.tmp
$ mv file.txt.tmp file.txt

And to work with more number of files (say perform the same replacement as above in all the .cfg files in current directory, including sub-directory)

for file in $(find . -name "*.cfg")
    do
        echo "Replacing on : $file"
        sed 's/VERSION/8.04/' $file > $file.tmp
        mv $file.tmp $file
        echo "Replacement done on : $file"
done

Related posts:

- Add Change Insert lines to file using sed
- Substitute character by position using sed
- Case insensitive search and replace using sed
- Accessing external variables in sed and awk
- Delete next few lines using sed

Embedding shell command in sed - bash

'sed' newbies might find this post useful !!

Input file:

$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active

The content of '/etc/lsb-release' file on my ubuntu desktop:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=8.04
DISTRIB_CODENAME=hardy
DISTRIB_DESCRIPTION="Ubuntu 8.04.3 LTS"

Lets extract the 'DISTRIB_RELEASE' version from the above file:

$ awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release
8.04

Required: Replace the text 'VERSION' in the input file 'file.txt' with the output of the above command (i.e. 'DISTRIB_RELEASE' version)

First way of doing this:

$ myvar=$(awk '/DISTRIB_RELEASE/ {print $2}' FS=\= /etc/lsb-release)
$ sed "s/VERSION/$myvar/g" file.txt

Output

port:9903
os-version:8.04
codename:hardy
status:active

** Important to see that we have used double quotes (instead of regular single quote) in the above sed statement. Using single quote is not going to expand the content of the variable 'myvar'.

Other ways :

$ sed "s/VERSION/`awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release`/g" file.txt

Which is same as:

$ sed "s/VERSION/$(awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release)/g" file.txt

And another way of quoting (This will allow you to use the single quote with sed statement):

$ sed 's/VERSION/'"$(awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release)"'/g' file.txt

same as

$ sed 's/VERSION/'"`awk -F '=' '/DISTRIB_RELEASE/ {print $2}' /etc/lsb-release`"'/g' file.txt

Related post:

- Accessing external variable in awk and sed

Prevent exit of shell with Ctrl d - IGNOREEOF

The function of 'Ctrl-d'(^D) key is to exit the shell. If IGNOREEOF variable does not exist in your shell a single ^D will exit from the shell (this is the default behavior).

What is this $IGNOREEOF ?

It controls the action of the shell on receipt of an EOF character as the sole input.

If set, the value is the number of consecutive EOF characters typed as the first characters on an input line before bash exits.

If the variable exists but does not have a numeric value, or has no value, the default value is 10.
(Source)

Now if you set the following:

export IGNOREEOF=1

This will prevent the shell to exit with a single press of 'Ctrl d' and will require two consecutive 'Ctrl d' (^D) to exit the session. You can also add it to your '~/.bashrc'.

Related bash tips:

- Remove path from PATH variable in bash
- Linux command line history with timestamp
- Bash - save command in history without executing it

Replace values in XML using sed and awk

Even though sed or awk is not suggested to work with XMLs (generally a programing language like Perl or Python is more preferred), there can be simple cases where we can make use of sed or awk to perform certain search and replace in simple XML s. Here is an simple example.

Input XML file:

$ cat test.xml
<?xml version="1.0" encoding="UTF-8"?>
<StudentInfo Version="1">
    <Student>
        <StudentId>INS469</StudentId>
        <ClassId>21</ClassId>
        <Amount>90</Amount>
        <Location>AA</Location>
    </Student>
    <Student>
        <StudentId>CSI150</StudentId>
        <ClassId>71</ClassId>
        <Amount>82</Amount>
        <Location>AX</Location>
    </Student>
    <Student>
        <StudentId>E687</StudentId>
        <ClassId>12</ClassId>
        <Amount>30</Amount>
        <Location>AR</Location>
    </Student>
</StudentInfo>

Required: Add value 20 t0 the existing 'Amount' values for each student entry in the above XML, i.e. the required output will be something like this:

<?xml version="1.0" encoding="UTF-8"?>
<StudentInfo Version="1">
    <Student>
        <StudentId>INS469</StudentId>
        <ClassId>21</ClassId>
        <Amount>110</Amount>
        <Location>AA</Location>
    </Student>
    <Student>
        <StudentId>CSI150</StudentId>
        <ClassId>71</ClassId>
        <Amount>102</Amount>
        <Location>AX</Location>
    </Student>
    <Student>
        <StudentId>E687</StudentId>
        <ClassId>12</ClassId>
        <Amount>50</Amount>
        <Location>AR</Location>
    </Student>
</StudentInfo>

The awk solution:

$ awk '
BEGIN { FS = "[<|>]" }
{
        if ($2 == "Amount") {
                sub($3,$3+20)
        }
        print
}
' test.xml

And to replace the 'Amount' value in the input XML with a constant number say '100' here is a sed solution:

$ sed 's#\(<Amount>\)[0-9]*\(</Amount>\)#\1'100'\2#g' test.xml

Output:

<?xml version="1.0" encoding="UTF-8"?>
<StudentInfo Version="1">
    <Student>
        <StudentId>INS469</StudentId>
        <ClassId>21</ClassId>
        <Amount>100</Amount>
        <Location>AA</Location>
    </Student>
    <Student>
        <StudentId>CSI150</StudentId>
        <ClassId>71</ClassId>
        <Amount>100</Amount>
        <Location>AX</Location>
    </Student>
    <Student>
        <StudentId>E687</StudentId>
        <ClassId>12</ClassId>
        <Amount>100</Amount>
        <Location>AR</Location>
    </Student>
</StudentInfo>

The \1 and \2 (Specifying which occurrence) used above can be well understood from few of the related posts I am mentioning below. Also there is a useful description of the same on grymoire.com

Related posts:

- Printing single quote with awk in bash
- Printing column of file using sed
- Use sed to replace part of a file in bash
- Convert date format using sed and awk
- Format lines using sed

Randomize lines using Linux rl command

Input file:

$ cat leader.txt
Mr B
Mrs C
Mrs A
Mrs X
Mr Y

Question: How to read a random line from the above file in Unix command line?

A few solutions:

Using UNIX/Linux shuf command:

$ shuf -n 1 leader.txt
Mr B

Using sed with bash RANDOM variable:

$ sed -n $((RANDOM%$(wc -l < leader.txt)+1))p leader.txt
Mrs C

Recently I came across UNIX/Linux rl command (randomize-lines) and the solution using 'rl' will be:

$ rl -c 1 leader.txt

'rl' is similar to UNIX/Linux 'shuf' utility which can be used to randomizes the lines in a file.

rl was developed by Arthur de Jong

On my ubuntu desktop, I installed rl utility in this way:

$ sudo apt-get install randomize-lines

From rl(1) man page:

rl  reads  lines  from  a  input file or stdin, randomizes the lines and outputs a
specified number of lines.
It does this with only a single pass over the input while trying to use as little
memory as possible.

Few of its important command lines options are:

-c, --count=N
Select the number of lines to be returned in the output.
If this argument is omitted all the lines in the file will be returned in random order.
If the  input  contains  less lines than specified and the --reselect option below is not
specified a warning is printed and all lines are returned in random order.

-o, --output=FILE
Send randomized lines to FILE instead of stdout.

-d, --delimiter=DELIM
Use specified character as a "line" delimiter instead of the newline character.

-n, --line-number
Output lines are numbered with the line number from the input file.

 -r, --reselect
When  using  this option a single line may be selected multiple times.
The default behavior is that any input line will only be selected once.
This option makes it possible to specify a --count option with more lines than
the file actually holds.

The  program uses the rand() system random function.
This function returns a number between 0 and RAND_MAX,
which may not be very large on some systems.
This will result in non-random results for files containing more lines than RAND_MAX.

Few of 'rl' common uses:

Randomize lines of file 'leader.txt':

$ rl leader.txt
Mr Y
Mrs C
Mrs X
Mr B
Mrs A

With -n option:

$ rl -n leader.txt
2: Mrs C
1: Mr B
3: Mrs A
5: Mr Y
4: Mrs X

Sending stdout to a file using -o option:

$ rl -o /tmp/leader.txt.sfl -n -c 3 leader.txt
$ cat /tmp/leader.txt.sfl
1: Mr B
4: Mrs X
2: Mrs C

Shuffle the words of a sentence/string:

$ echo -n "A,B,C,D,E," | rl -d ","

And a practial use inspired from 'rl' man page:

Play a random .mp3 song (from ~/lovesongs/ directory) after 5 minutes:

$ sleep 300 ; play $(find ~/lovesongs/ -name "*.mp3" -print | rl -c 1)

Vim copy paste indentation issue

While pasting something into a file opened in vim editor, sometime you will see kind of a "staircase" effect where each line is progressively spaced farther outward.

e.g.

I was doing to copy paste the following lines into a file opened in vim



And it appeared like this



This happens because Vim assumes that I am actually typing and not pasting, so it indents the lines again (I already had autoindent set in my .vimrc), and this results in additional white spaces at front.

To avoid this, I ran the following command in vim command mode (before pasting)

:set paste

Then pressed 'i' to switch to insert mode; pasted the lines. Worked!!

One may disable this by using:

:set nopaste

Vim paste:

Put Vim in Paste mode. This is useful if you want to cut or copy some text from one window and paste it in Vim. This will avoid unexpected effects. Setting this option is useful when using Vim in a terminal, where Vim cannot distinguish between typed text and pasted text.

This can also be achieved by setting:

:set noai

i.e. set 'no auto-indent'. And 'auto-indent' can be set by

:set ai

You might also like:

- Using tabs in vim editor, a small tutorial