Happy New Year to all Bash shell lovers

Wish all my readers a Great, Prosperous and Healthy New Year 2010.

jaduks@unstableme:~$ sed 's/6/_/g' /tmp/hny2010
____________________________________________________________________________________________________
___99_____99__99__99999999__99_____99_______99______99__99999999__99____99______________9___________
___99_____99__99__99999999__99_____99________99____99___99999999__99____99_____________999__________
___99_____99__99__99________99_____99_________99__99____99____99__99____99____________99_99_________
___99__9__99__99__99999999__999999999__________9999_____99____99__99____99___________99___99________
___99_999_99__99__99999999__999999999___________99______99____99__99____99__________999999999_______
___9999_9999__99________99__99_____99___________99______99____99__99____99_________99999999999______
___999___999__99__99999999__99_____99___________99______99999999__99____99________99_________99_____
___99_____99__99__99999999__99_____99___________99______99999999__99999999_______99___________99____
____________________________________________________________________________________________________
____________________________________________________________________________________________________
____________________________________________________________________________________________________
__99_____99_________9_________99999999__99999999__99______99________99_____99__99999999__99_____99__
__99_____99________999________99999999__99999999___99____99_________999____99__99999999__99_____99__
__99_____99_______99_99_______99____99__99____99____99__99__________9999___99__99________99_____99__
__999999999______99___99______99999999__99999999_____9999___________99_99__99__99999999__99__9__99__
__999999999_____999999999_____99999999__99999999______99____________99__99_99__99999999__99_999_99__
__99_____99____99999999999____999_______99____________99____________99___9999__99________9999_9999__
__99_____99___99_________99___99________99____________99____________99____999__99999999__999___999__
__99_____99__99___________99__99________99____________99____________99_____99__99999999__99_____99__
____________________________________________________________________________________________________
____________________________________________________________________________________________________
____________________________________________________________________________________________________
___99______99__99999999_________9_________99999999________9999999__99999999___999____99999999_______
____99____99___99999999________999________99999999________99999999_99____99____99____99____99_______
_____99__99____99_____________99_99_______99____99______________99_999___99____99____999___99_______
______9999_____99999999______99___99______99999999_____________99__99_9__99____99____99_9__99_______
_______99______99999999_____999999999_____99999999____________99___99__9_99____99____99__9_99_______
_______99______99__________99999999999____99_99_____________99_____99___999____99____99___999_______
_______99______99999999___99_________99___99__99__________99999999_99____99____99____99____99_______
_______99______99999999__99___________99__99____99________99999999_99999999___9999___99999999_______
____________________________________________________________________________________________________

Linux dialog utility short tutorial

dialog - display dialog boxes from shell scripts.

'dialog' is a utility for building console-based 'front ends' in UNIX like operating systems.
In this brief tutorial I am mentioning the usage of few important basic controls available with this 'dialog' utility and later I have created a very simple front end application in UNIX bash scripting using dialog.

To install 'dialog' on your ubuntu:

$ apt-get install dialog

Box options available with dialog: (Do a 'man' of dialog to know the usage of each control box)

--calendar
--checklist
--dselect
--editbox
--form
--fselect
--gauge
--infobox
--inputbox
--inputmenu
--menu
--mixedform
--mixedgauge
--msgbox
--passwordbox
--passwordform
--pause
--progressbox
--radiolist
--tailbox
--tailboxbg
--textbox
--timebox
--yesno

Checklist box:

A checklist box allows you to present a set of choices to the user and the user can toggle each one on or off individually using the space bar.
A sample one:

$ dialog --checklist "Choose OS:" 15 40 5 \
1 Linux off \
2 Solaris on \
3 'HP UX' off \
4 AIX off

Radiolist box:

The 'radiolist' control box is same as 'checklist' box.

$ dialog --backtitle "OS infomration" \
--radiolist "Select OS:" 10 40 3 \
 1 "Linux 7.2" off \
 2 "Solaris 9" on \
 3 "HPUX 11i" off

Inputbox:

The 'inputbox' allows the user to enter a string.

$ dialog --title "Inputbox - Example" \
--backtitle "unstableme.blogspot.com" \
--inputbox "Enter your favourite OS here" 8 50

Menu box:

$ dialog --title "A dialog Menu Example" \
--menu "Please choose an option:" 15 55 5 \
1 "Add a record to DB" \
2 "Delete a record from DB" \
3 "Exit from this menu"

A message-box:

$ dialog --title "Example Dialog message box" \
--msgbox "\n Installation Completed on host7" 6 50

A yesno box:

    
$ dialog --title "Confirmation"  --yesno "Want to quit?" 6 20

Infobox:

$ dialog --infobox "Processing, please wait" 3 34 ; sleep 5

Textbox:
It is a simple file viewer

$ dialog --textbox ~/work/conf.txt 10 4

From the sample application I have created at the end of this post, you will easily learn how to program these dialog boxes i.e. how to capture what user has entered/pressed. The dialog program writes its output to the standard error by default. In most of the dialog controls we redirect the choice user has selected to a tempfile and then process return value of dialog and contents of the tempfile.

Gauge Box:

#!/bin/sh
#A gauge Box example with dialog
(
c=10
while [ $c -ne 110 ]
    do
        echo $c
        echo "###"
        echo "$c %"
        echo "###"
        ((c+=10))
        sleep 1
done
) |
dialog --title "A Test Gauge With dialog" --gauge "Please wait ...." 10 60 0

Calendar Box:

#!/bin/sh

dat=$(dialog --stdout --title "My Calendar" \
                   --calendar "Select a date:" 0 0 25 12 2009)

case $? in
0)
 echo "You have entered: $dat"   ;;
1)
 echo "You have pressed Cancel"  ;;
255)
 echo "Box closed"   ;;
esac

Time Box:

#!/bin/sh

tim=$(dialog --stdout --title "A TimeBox" \
                    --timebox "Set the time:" 0 0 10 13 59)

case $? in
0)
  echo "You have set: $tim"   ;;
1)
  echo "You have pressed Cancel"  ;;
255)
  echo "Box closed"   ;;
esac

A sample application:

Suppose:

$ cat /home/user9/work/conf.txt
port:3322
threads:2
logdir:/opt/user6/logs/
confdir:/opt/user6/etc/

The following bash script using dialog utility will facilitate a simple interface to view or edit the content of the above config file.

#!/bin/sh
#http://unstableme.blogspot.com/
#A sample application using UNIX/Linux dialog utility
#Auto-size with height and width = 0 of the dialog controls

file='/home/user9/work/conf.txt'
tempfile1=/tmp/dialog_1_$$
tempfile2=/tmp/dialog_2_$$
tempfile3=/tmp/dialog_3_$$

trap "rm -f $tempfile1 $tempfile2 $tempfile3" 0 1 2 5 15

_edit () {
   items=$(awk -F\: '{print $1,$2}' $file)
   dialog --title "A Sample Application" \
          --menu "What you want to change :" 0 0 0 $items 2> $tempfile1

   retval=$?
   parameter=$(cat $tempfile1)

   [ $retval -eq 0 ] && tochange=$parameter || return 1

   val=$(awk -F\: -v x=$tochange '$1==x {print $2}' $file)
   dialog --clear --title "Inputbox - Test" \
          --inputbox "Enter new value($tochange)" 0 0 $val 2> $tempfile2

   dialog --title "Confirmation"  --yesno "Commit ?" 0 0
   case $? in
       0) newval=$(cat $tempfile2)
          awk -v x=$tochange -v n=$newval '
               BEGIN {FS=OFS=":"}$1==x {$2=n} {print}
               ' $file > $file.tmp
          mv $file.tmp $file
       ;;
       1|255) dialog --infobox "No Changes done" 0 0
              sleep 2
   ;;
   esac
   dialog --textbox $file 0 0
}

_main () {
   dialog --title "A sample application" \
           --menu "Please choose an option:" 15 55 5 \
                   1 "View the config file" \
                   2 "Edit config file" \
                   3 "Exit from this menu" 2> $tempfile3

   retv=$?
   choice=$(cat $tempfile3)
   [ $retv -eq 1 -o $retv -eq 255 ] && exit

   case $choice in
       1) dialog --textbox $file 0 0
          _main
           ;;
       2) _edit
          _main ;;
       3) exit ;;
   esac
}

_main

Dialog utility home page

Vim tip - open file under cursor

In order to open a file whose 'name' (path) is under the cursor

gf : open in the same window ("goto file")
Ctrl-w f : open in a new window
Ctrl-w gf :  open in a new tab

And to return to the previous file or buffer : Ctrl-o

To learn more about vi 'gf' command , here is a good resource

Related posts:
- Pull word under cursor - vi editor tip
- Using tabs in vim editor - a short tutorial

Bash cat command space issue explained

Input file contains some 4 student names like this:

$ cat file.txt
Alex C M
Peter S
Dhiren K
Prahlad G N

Required: I was trying to produce the following output:

1) Alex C M [3]
2) Peter S [2]
3) Dhiren K [2]
4) Prahlad G N [3]

i.e. a serial number, Name of the student, number of words in his name.
Lets try with bash for loop like this:

$ cat lp1.sh
#!/bin/sh

c=0
for line in $(cat file.txt)
    do
        ((c+=1))
        numfields=$(echo $line | awk '{print NF}')
        echo "$c) $line [$numfields]"
done

And the output it produced !

$ ./lp1.sh
1) Alex [1]
2) C [1]
3) M [1]
4) Peter [1]
5) S [1]
6) Dhiren [1]
7) K [1]
8) Prahlad [1]
9) G [1]
10) N [1]

So what went wrong ?
I tried echo "$line" as well, same output.

In the above example, we need to take care of the Bash IFS environmental variable. From Bash man page:

IFS:
The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read built in command.
The default value is
<space><tab><newline>.

And since the lines in the input file got lines with spaces in between, above script is behaving in that way.
We can temporarily change the IFS in the shell script like this:

$ cat lp3.sh
#!/bin/sh

OLD_IFS=$IFS
IFS=$'\n'
c=0
for line in $(cat file.txt)
    do
        ((c+=1))
        numfields=$(echo $line | awk '{print NF}')
        echo "$c) $line [$numfields]"
done
IFS=$OLD_IFS

Output:

$ ./lp3.sh
1) Alex C M [3]
2) Peter S [2]
3) Dhiren K [2]
4) Prahlad G N [3]

Bash 'while loop' used in the below way also works without changing the IFS:

$ cat lp2.sh
#!/bin/sh

c=0
while read line
    do
        ((c+=1))
        numfields=$(echo $line | awk '{print NF}')
        echo "$c) $line [$numfields]"
done < "file.txt"

Output:

$ ./lp2.sh
1) Alex C M [3]
2) Peter S [2]
3) Dhiren K [2]
4) Prahlad G N [3]

Note: The above example is taken mainly to show the use of Bash IFS variable. Using awk, the above can be done easily like this:

$ awk '{print NR")",$0,"["NF"]"}' file.txt
$ awk '{++c}{print c")",$0,"["NF"]"}' file.txt

Some related posts:

- Bash script while loop sum issue explained
- Bash script for sequential subtraction of numbers
- Use of until loop in bash scripting

Sum numbers in each row - awk

Input file contains the scores of few students in certain rounds of a game in the following format.

id,Name,score1,score2,score3 etc

$ cat file.txt
id9,Mohit Kishore,19,13,14,10
id2,Niraj Kumar,13,8,23,8
id8,Kate Nil,19,18,15
id4,Rashi S,19,28,65,10,19

Required output:

Calculate the sum and average score of each student (notice that the number of rounds played by each student is not constant, few played 3 rounds, few 4 etc)

The awk script:

$ awk '
BEGIN {FS=OFS=","}
{
sum=0; n=0
for(i=3;i<=NF;i++)
     {sum+=$i; ++n}
     print $0,"sum:"sum,"count:"n,"avg:"sum/n
}' file.txt

Output:

id9,Mohit Kishore,19,13,14,10,sum:56,count:4,avg:14
id2,Niraj Kumar,13,8,23,8,sum:52,count:4,avg:13
id8,Kate Nil,19,18,15,sum:52,count:3,avg:17.3333
id4,Rashi S,19,28,65,10,19,sum:141,count:5,avg:28.2

Related posts:

- Calculate sum and average of multiple lines - bash
- Sum of numbers in a file - UNIX
- Calculate percentage using awk in bash
- Compute simple average using awk in bash
- Calculate sum of two times of hh-mm-ss format using awk
- Compute sum with awk substr function

Awk - Print next column after pattern

Input file 'results.classV.txt' contains the results of class V students in the School annual sports meet.

$ cat results.classV.txt
Chess:Alex,First,Naveen,Second,Lee,4th
Carrom:Naveen,First
billiards:Lee,First,Nill,Second
foosball:Nill,First,Naveen,4th
badminton:Lee,Second

i.e. the name of a student precedes his rank in a game.

Required:

Extract out the details of student "Naveen" in each game.

The awk solution:

$ awk '
BEGIN {FS="[:,,]"; OFS=":"; {print "Naveen"}}
{ for(j=0;j<=NF;j++)
    if ( $j == "Naveen" )
          print $1,$(j+1)}
' results.classV.txt

Output:

Naveen
Chess:Second
Carrom:First
foosball:4th

Related posts:

- Find blank columns in a file using awk
- Find column number based on field header using awk
- Print range of columns using awk
- Awk - find column number of a pattern

Awk - sum two times of hh-mm-ss format

2nd and 3rd field for input file 'file.txt' is in hh:mm:ss format.

$ cat file.txt
#slno   round1      round2
505     01:54:15    00:24:05
509     01:34:39    00:23:03
503     01:51:55    00:22:55
503     01:45:10    00:12:15

Required:

- Sum $2 and $3 of the above file and print it as the 4th field in 'seconds' format.


The awk solution:

$ awk '
function convert(t) {

    split(t,Arr,":")
    return Arr[1]*3600+Arr[2]*60+Arr[3]

    }

/^#/ {print $0,"\ttotal(sec)"; next}
{print $0,"\t",convert($2)+convert($3) }
' file.txt

Output:

#slno   round1      round2      total(sec)
505     01:54:15    00:24:05     8300
509     01:34:39    00:23:03     7062
503     01:51:55    00:22:55     8090
503     01:45:10    00:12:15     7045

Related posts:

- Convert seconds to hh:mm:ss format in Bash script
- Find time between two dates using Bash
- More of Linux/UNIX date command
- Expand entries in file using awk in Bash

Replace first 5 characters - awk and sed

Input file:

$ cat file.txt
sl909678342-slp10
sl919678362-slp11
sl929678322-slp12
sl929678382-slp12

Required output:

Replace first 5 characters in each line of the above file with 'XXXXX'. i.e. the required output:

XXXXX678342-slp10
XXXXX678362-slp11
XXXXX678322-slp12
XXXXX678382-slp12

Awk solution:

$ awk '{
    first=substr($0,1,5)
    gsub(/./,"X",first)
    end=substr($0,6)
    print first end
}' file.txt

Read about awk substr function here

Sed solution:

Using 'extended regular expressions' with sed (-r option)

$ sed -r "s/^(.{0})(.{5})/\1XXXXX/" file.txt

Related posts:

- Substitute character by position using sed
- Replace text based on position using awk
- Print or remove first few characters in bash
- Print first character in a field using awk
- Insert text after certain characters using awk and sed

Awk - numbering lines ignoring blank lines

In one of my earlier post I have already discussed on numbering lines in a file using awk, here is a similar post to number lines ignoring 'blank lines' present.

Input file 'file.txt' has got first 2 lines fixed as HEADER line, followed by a number of record lines (^k).

$ cat file.txt
h1|456|v1|1
h2|190|-|5
k|rn|90.67|12|90
k|rn|90.43|22|35
k|rn|90.62|71|90
k|rn|90.51|16|96
k|rn|90.37|18|71

Required: In the above file, replace the 2nd field in the record lines (i.e. ^k) with the serial record number (starting with zero i.e. 0).
i.e. required output:

h1|456|v1|1
h2|190|-|5
k|0|90.67|12|90
k|1|90.43|22|35
k|2|90.62|71|90
k|3|90.51|16|96
k|4|90.37|18|71

The solution using awk NR variable:

$ awk '
    BEGIN {FS=OFS="|"}
    $1=="k" {$2=NR-3} {print}
' file.txt

i.e. for the lines("|" delimited) where first field is "k", replace 2nd field with "NR-3" and then print the new output.
NR is the ordinal number of the current record. A post describing awk NR variable can be found here

Now if the input file contains certain blank lines, something like this:

$ cat file.txt
h1|456|v1|1
h2|190|-|5
k|rn|90.67|12|90
k|rn|90.43|22|35


k|rn|90.62|71|90

k|rn|90.51|16|96
k|rn|90.37|18|71

Executing the above awk one liner:

$ awk '
    BEGIN {FS=OFS="|"}
    $1=="k" {$2=NR-3} {print}
' file.txt

Output:

h1|456|v1|1
h2|190|-|5
k|0|90.67|12|90
k|1|90.43|22|35


k|4|90.62|71|90

k|6|90.51|16|96
k|7|90.37|18|71

As you can see the record numbers are not in serial order, as using NR will also count for the blank lines present.
A different solution:

$ awk '
    BEGIN{FS=OFS="|"}
    !/^$/ {++c}
    $1=="k" {$2=c-3} {print}
' file.txt

Output:

h1|456|v1|1
h2|190|-|5
k|0|90.67|12|90
k|1|90.43|22|35


k|2|90.62|71|90

k|3|90.51|16|96
k|4|90.37|18|71

Related posts using Awk:

- Replace a field with different values using awk
- Subdividing a file into sub-files using awk and bash
- Finding blank columns in a file - awk
- A practical example using awk

Use tab newline with echo command - bash

$ echo "HMM\tThis line got a TAB"
HMM\tThis line got a TAB

So it did not work !

To use tabs, new-line, and other formatting characters with UNIX/Linux echo command, specify the -e option:

$ echo -e "HMM\tThis line got a TAB"
HMM     This line got a TAB

Similarly:

$ echo -e "A B\nC\nD"
A B
C
D

From ECHO(1) man page:

-e enable interpretation of backslash escapes

If -e is in effect, the following sequences are recognized:

\\     backslash

\a     alert (BEL)

\b     backspace

\c     suppress trailing newline

\f     form feed

\n     new line

\r     carriage return

\t     horizontal tab

\v     vertical tab

Similar posts:

- Grep and print control characters in a file - UNIX
- Highlight match with color in UNIX grep command
- Print only matched string, not line in grep command

Awk - Print first occurrence of a set

Input file 'order-pref.txt' contains two sql UPDATE commands for each item.

$ cat order-pref.txt
computer networking book
UPDATE orders SET quantity = '5' WHERE id = '9';
UPDATE orders SET quantity = '5' WHERE id = '8';
puzzle and skill games
UPDATE orders SET quantity = '5' WHERE id = '99';
UPDATE orders SET quantity = '5' WHERE id = '98';
arithmetic sequence
UPDATE orders SET quantity = '5' WHERE id = '55';
UPDATE orders SET quantity = '5' WHERE id = '56';

Required:
For each item find and print the first UPDATE sql command (i.e. first occurrence of each set of sql commands for each item).

i.e. required output:

UPDATE orders SET quantity = '5' WHERE id = '9';
UPDATE orders SET quantity = '5' WHERE id = '99';
UPDATE orders SET quantity = '5' WHERE id = '55';

Awk solution:

$ awk '/^UPDATE/{++c;if(c%2==1)print}' order-pref.txt > o.sql

Output:

$ cat o.sql
UPDATE orders SET quantity = '5' WHERE id = '9';
UPDATE orders SET quantity = '5' WHERE id = '99';
UPDATE orders SET quantity = '5' WHERE id = '55';

Related posts:

- Print first or last occurrence of pattern range using awk
- Print lines with more than two occurrence of a pattern using sed, awk, grep
- Count total occurrence of a pattern using awk
- Replace a field other than the first occurrence using awk
- Find pattern with maximum occurrence using awk
- Find nth occurrence of pattern in vi editor