Sed to replace a part of file : bash scripting

$ cat idfile
id='1' dsadsad adsad
id='31' dsadsad adsad 32432
id='231' dsadsad adsad 3234
id='123' 2332 dsadsad adsad
id='124' 2332


Output required:
Subtract 1 from the id value from each line of the above file. i.e. final output should like this:

id='0' dsadsad adsad
id='30' dsadsad adsad 32432
id='230' dsadsad adsad 3234
id='122' 2332 dsadsad adsad
id='123' 2332

The script:

$ cat idfile | while read line
> do
> R=`echo $line | sed "s/id='\([0-9].*\)'.*/\1/"`
> ((R-=1))
> echo $line | sed "s/\(id='\)\([0-9].*\)\('.*\)/\1$R\3/"
> done

Note:
If
((R-=1))
complians, then use
R=`expr $R - 1`


Some references:

$ echo "id='1' dsadsad adsad" | sed "s/\(id='\)\([0-9].*\)\('.*\)/\1/"
id='

$ echo "id='1' dsadsad adsad" | sed "s/\(id='\)\([0-9].*\)\('.*\)/\2/"
1

$ echo "id='1' dsadsad adsad" | sed "s/\(id='\)\([0-9].*\)\('.*\)/\3/"
' dsadsad adsad

Printing combination of lines using awk in bash

Input file:

$ cat file1
11
12
13
14

Output required: Print the combination of above lines so that output looks like:

11,12
11,13
11,14
12,11
12,13
12,14
13,11
13,12
13,14
14,11
14,12
14,13

Awk solution:

$ awk '
{ a[$0] }
END {
       for (i in a){
               for (j in a){
                       if (i != j)  print (i "," j)
               }
       }
}' file1

.

vi editor - join lines using "J" in command mode

shift j or "J" joins lines in command mode.
This will join the current line with the next line in your file.
Simple but useful.

Delete next few lines using sed

Title
____________

Delete next few lines where the pattern is found.

Input file
____________

$ cat para.txt
#Element1.blk
{
(Type= Fan)
(Name = KOL)
(Description = )
}
#Element2.blk
}
(Type= PPlug)
(Name = NIC)
(Description = )
}
#Element3.blk
}
(Type= Fan)
(Name = RIP)
(Description = )
}


Output required:
________________

If I search for "Element2.blk", the line containing "Element2.blk" and next 5 lines should be deleted.

#Element1.blk
{
(Type= Fan)
(Name = KOL)
(Description = )
}
#Element3.blk
}
(Type= Fan)
(Name = RIP)
(Description = )
}


sed solution
___________________________

$ sed '/#Element2.blk/{N;N;N;N;N;d;}' para.txt

Count and print using awk associative array

Input file:

$ cat results.txt
Anil:Awk:Pass
Rakesh:Sed:Fail
Anil:Expect:Pass
Bau:Bash:Fail
sbnak:ksh:Pass
Rakesh:Awk:Pass
Anil:Bash:Fail
Neo:zsh:Fail

Required: Count the number of Pass and Fails for each of the names(1st field) and print in the following order. i.e.

Name Num_fail_entries Num_pass_entries

Awk solution:

$ awk '
BEGIN {FS=":"; print "Name Num_fail Num_pass"} {
 n[$1]++;
 F_[$1] += ($3 == "Fail" ? 1 : 0)
 P_[$1] += ($3 == "Pass" ? 1 : 0)
}
END {
   for (i in n) {
      print i,F_[i],P_[i]
   }
}' results.txt

Output:

Name Num_fail Num_pass
Neo 1 0
Rakesh 1 1
Anil 1 2
Bau 1 0
sbnak 0 1

Listing file names in a numeric range - bash

This looks to be simple, but very important.

In a directory, I had 1000 files, something like:


$ ls -l | awk 'NR!=1{print $NF}' | sort -t "." -nk2


Output:
nsplog.1.log
nsplog.2.log
nsplog.3.log
nsplog.4.log
nsplog.5.log
nsplog.6.log
..
..
nsplog.12.log
nsplog.13.log
nsplog.14.log
..
..
nsplog.997.log
nsplog.998.log
nsplog.999.log
nsplog.1000.log


To list the files from nsplog.500.log to nsplog.599.log

$ ls nsplog.5[0-9][0-9].log


To list the files from nsplog.550.log to nsplog.599.log

$ ls nsplog.5[5-9][0-9].log

.

Match an IP range using egrep - bash

The input file:

$ cat myhosts.txt
172.22.21.123 mickwe
172.22.21.11 tests1
172.22.21.38 tests3
172.22.21.34 tests3
172.22.21.13 devenv3
172.22.21.9 dennet5
172.22.21.20 lic4
172.22.21.50 tests6

Required: Match IP range from 172.22.21.1 to 172.22.21.35 from the above file.

The solution using egrep is:

$ egrep '172\.22\.21\.([1-9]|(1[0-9]|2[0-2]|3[0-5])) ' myhosts.txt

Output:

172.22.21.11 tests1
172.22.21.34 tests3
172.22.21.13 devenv3
172.22.21.9 dennet5
172.22.21.20 lic4

Merge previous line using sed

Input file:

$ cat file.txt
1 AA 2
2 BB 3
3 DD 4
5 CC 12
7 ZZ 12

Required output:
If I search for line with first field as "3" , th line with first field 3 should be merged with the previous line to it.
i.e

1 AA 2
2 BB 3 3 DD 4
5 CC 12
7 ZZ 12

Awk code:

$ awk 'NR==1{printf $0;next}
/^3\>/{printf " " $0;next}
{printf "\n" $0}
END{print ""}
' file.txt

or if you just want to print that affected line only:

$ awk '/^3\>/{print l,$0}{l=$0}' file.txt
2 BB 3 3 DD 4

Break a line into multiple lines - awk, sed

Input line:

ASX,23,PILE2.3,MNP,54,NEW123,LELO,67,PNP2


Output required: 3 entries in one line i.e.

ASX,23,PILE2.3
MNP,54,NEW123
LELO,67,PNP2

awk and sed solution:

$ LINE="ASX,23,PILE2.3,MNP,54,NEW123,LELO,67,PNP2"

$ echo $LINE | awk -F, '{for(i=1;i<=NF;i++){printf("%s%s",$i,i%3?",":"\n")}}'

$ echo $LINE | sed 's/\([^,]*,[^,]*,[^,]*,\)/&\
/g'


$ echo $LINE | sed 's/\([^,]*,\)\{3\}/&\
/g'

Similar post: split a line into multiple lines using or sed

Merging lines using awk - bash

Input file:

$ cat myfile.txt
unix,bash
,scripting
awk is powerful
u1,u2
,u3,u4,u5


Objective: Merge the lines starting with comma "," with the previous line.
i.e. the output required.

unix,bash,scripting
awk is powerful
u1,u2,u3,u4,u5


Awk solution:

$ awk 'END { print r }
r && !/^,/ { print r; r = "" }
{ r = r ? r $0 : $0 }
' myfile.txt

Find the character immediately after a word - sed


$ echo "unix:bash/scripting:12"
unix:bash/scripting:12


Purpose: find the character after the word "bash"


$ echo "unix:bash/scripting:12" | sed 's/.*bash\(.\).*/\1/'
/

.

pause before copy : bash script newbie

I already discussed how to write pause function in bash scripting in one of my older post.

Now a simple script to show how that function can be used.

Situation: Your application say "hxes" can process one file at a time (files under its input dir /opt/hxes/input/). You have to copy a set of *.toprocess files to its incoming folder /opt/hxes/input/ in a way such that before copying it waits till you press enter.


#!/bin/sh

f_pause(){
read -p "$*"
}

for file in `ls *.toprocess`
do
echo "copying $file"
f_pause "Hit Enter key to continue."
cp $file /opt/hxes/input/
done


Is my computer/ operating system is 64-bit or 32 bit?

How can I determine if my computer/operating system is 64-bit?

A nice link to share

Article: How to determine that your OS is 64-bit compliant (for AIX, SUN solaris, Linux and Windows)

http://www.stata.com/support/faqs/win/64bit.html

Delete one or more space using sed- bash

Input file: 

$ cat test.txt

one two three       four

1   2 3  4

i     ii   iii           iv

Sed code to delete/remove one or more space from a file. (Basically combine one or more space to a single space)

$ sed 's/[ \t]\+/ /g' test.txt > test.txt.1

$ cat test.txt.1

one two three four

1 2 3 4

i ii iii iv

Convert date format using sed, awk, perl

Here we will be discussing two examples:

1) Convert mmddyy to dd-mm-yy format using sed and perl
2) Convert yyyymmdd format to mm/dd/yyyy using awk and sed


1) Convert mmddyy to dd-mm-yy format

Input file:

$ cat file2.txt
090607:awk sed expect
091207:bash ksh csh zsh
091207:bash perl python

Output Required:
- Convert the first field(delimiter :) which is in "mmddyy" format to "dd-mm-yy format". i.e.

06-09-07:awk sed expect
12-09-07:bash ksh csh zsh
12-09-07:bash perl python

sed solution:

$ sed -e "s/^\(..\)\(..\)\(.*\)/\2-\1-\3/g" file2.txt

Perl solution:

$ cat frmt.pl
#!/usr/bin/perl
open(FILE, "$ARGV[0]");

while(<FILE>){
$_=~s/^(\d{2})(\d{2})(\d{2})/$2-$1-$3/;
print $_;
}

Executing the perl script:

$ ./frmt.pl file2.txt
06-09-07:awk sed expect
12-09-07:bash ksh csh zsh
12-09-07:bash perl python

2) Convert yyyymmdd format to mm/dd/yyyy


$ echo "YYYYMMDD" | awk '
BEGIN {OFS="/"}
{print substr($1,5,2),substr($1,7,2),substr($1,1,4)}'


Output:
MM/DD/YYYY


or

$ echo "YYYYMMDD"|sed -n -e "s_\(....\)\(..\)\(..\)_\2/\3/\1_p"

Output:
MM/DD/YYYY