Counting number of files in the current Directory/sub Directory

Ah, this is very simple, one might think why I am putting this in my blog.
Could be a help for a BASH newbie :-)

$ ls -RF1 | grep -v $"[/,:]" | grep -v ^$ | wc -l
11

$ find . -type f | wc -l
11

Array in Shell Script - Shell Beginner

Some of the important operations with array in BASH.

$ cat array.sh
#!/bin/sh

array=(jerry gen glan rahim)
len=${#array[*]} #Number of elements of the array

echo "The array has $len members. They are:"

i=0
while [ $i -lt $len ]; do
echo "$i: ${array[$i]}"
let i++
done

#Some operations
echo "Adding "jack" to the end of the array"
array=( "${array[@]}" "jack" )
echo "Listng all the elements in the array"
echo "${array[@]}"
echo "Listng all the elements in the array"
echo "${array[*]}" #One more way
echo "Deleting \"gen\""
unset array[1] #Same as array[1]=
echo "Now the array is"
echo "${array[@]}"
echo "length of 3rd element in the array"
len1=${#array[2]}; echo $len
echo "Deleting the whole array"
unset array
echo "${array[@]}" #Array is empty

$ ./array.sh
The array has 4 members. They are:
0: jerry
1: gen
2: glan
3: rahim
Adding jack to the end of the array
Listng all the elements in the array
jerry gen glan rahim jack
Listng all the elements in the array
jerry gen glan rahim jack
Deleting "gen"
Now the array is
jerry glan rahim jack
length of 3rd element in the array
4
Deleting the whole array

A more about Array of BASH can be found here:
http://tldp.org/LDP/abs/html/arrays.html

Available disk space using AWK - Shell Beginner

Some calculations using AWK to find out the available disk space.

The AWK file:

$ cat df.awk
{ sum += $4 }
END { mb = sum / 1024
gb = mb / 1024
printf "%.0f MB (%.2fGB) of available disk space\n", mb, gb
}
EOF

Executing:
$ df -k / | awk -f df.awk
5023 MB (4.90GB) of available disk space

RPM Commands by Jeff Hunter

(Taken from http://www.idevelopment.info/data/Unix/Linux/LINUX_RPMCommands.shtml)

This document contains an overview of the principal RPM commands for installing, uninstalling, upgrading, querying, listing, and checking RPM packages on your Red Hat Linux system.
________________________________________
# rpm -ivh foo-2.0-4.i386.rpm
# rpm -i ftp://ftp.redhat.com/pub/redhat/RPMS/foo-1.0-1.i386.rpm
# rpm -i http://oss.oracle.com/projects/firewire/dist/files/kernel-2.4.20-18.10.1.i686
.rpm
Used to install a RPM package. Note that RPM packages have file naming conventions like foo-2.0-4.i386.rpm, which include the package name (foo), version (2.0), release (4), and architecture (i386). Also notice that RPM understands FTP and HTTP protocols for installing and querying remote RPM files.
________________________________________
# rpm -e foo
To uninstall a RPM package. Note that we used the package name foo, not the name of the original package file foo-2.0-4.i386.rpm above.
________________________________________
# rpm -Uvh foo-1.0-2.i386.rpm
# rpm -Uvh ftp://ftp.redhat.com/pub/redhat/RPMS/foo-1.0-1.i386.rpm
# rpm -Uvh http://oss.oracle.com/projects/firewire/dist/files/kernel-2.4.20-18.10.1.i686
.rpm
To upgrade a RPM package. Using this command, RPM automatically uninstall the old version of the foo package and install the new package. It is safe to always use rpm -Uvh to install and upgrade packages, since it works fine even when there are no previous versions of the package installed! Also notice that RPM understands FTP and HTTP protocols for upgrading from remote RPM files.
________________________________________
# rpm -qa
To query all installed packages. This command will print the names of all installed packages installed on your Linux system.
________________________________________
# rpm -q foo
To query a RPM package. This command will print the package name, version, and release number of the package foo only if it is installed. Use this command to verify that a package is or is not installed on your Linux system.
________________________________________
# rpm -qi foo
To display package information. This command display package information including the package name, version, and description of the installed program. Use this command to get detailed information about the installed package.
________________________________________
# rpm -ql foo
To list files in installed package. This command will list all of files in an installed RPM package. It works only when the package is already installed on your Linux system.
________________________________________
# rpm -qf /usr/bin/mysql
mysql-3.23.52-3
Which package owns a file? This command checks to determine which installed package a particular file belongs to.
________________________________________
# rpm -qpl kernel-2.4.20-18.10.1.i686.rpm
# rpm -qpl ftp://ftp.redhat.com/pub/redhat/RPMS/foo-1.0-1.i386.rpm
# rpm -qpl http://oss.oracle.com/projects/firewire/dist/files/kernel-2.4.20-18.10.1.i686
.rpm
List files in RPM file. This command allows you to query a (possibly) uninstalled RPM file with the use of the the "-p" option. You can use the "-p" option to operate on an RPM file without actually installing anything. This command lists all files in an RPM file you have in the current directory. Also note that RPM can query remote files through the FTP and HTTP protocols.
________________________________________
# rpm --verify mysql
To verify an installed package. This command will list all files that do NOT pass the verify tests (done on size, MD5 signature, etc). Where a file does NOT pass, the output is listed using the following codes that signify what failed:
S File size
M Mode (includes permissions and file type)
5 MD5 sum
L Symlink
D Device
U User
G Group
T Mtime
Take for example the following:
# rpm --verify mysql
S.5....T c /etc/my.cnf
This example indicates that file /etc/my.cnf failed on:
File size
MD5 Sum
Modified Time
However, the "c" tells us this is a configuration file so that explains the changes. It should still be looked at to determine what the changes were.
________________________________________
# rpm --checksig foo
To check a RPM signature package. This command checks the PGP signature of specified package to ensure its integrity and origin. Always use this command first before installing a new RPM package on your system. Also, GnuPG or Pgp software must be already installed on your system before you can use this command.

Setting DISPLAY variable

How about exporting the DISPLAY variable in a generic way? (instead of hardcoding it to specific IP address in .bash_profile e.g DISPLAY=172.22.21.21:0.0) , usually ip address is the address of the workstation on which the GUI(windows) from linux are to be displayed.

If you are in an working environment where you used to telnet to linux boxes (using putty or cmd line from windows boxes), and you want the linux application GUIs to be displayed in your local windows box(from where you are telneting), I feel this is the best way of doing that.

in .bash_profile/.profile

DISPLAY=`who am i | awk '{print $6}' | cut -d"(" -f2 | cut -d")" -f1`:0.0; export DISPLAY

or

DISPLAY=`who am i | awk '{print $NF}' | sed -e 's/(//g' -e 's/)//g'`:0.0; export DISPLAY

So its more generic, will take the IP address of the local box from where you are telnetting to the Linux box

Special Parameters - BASH beginner

These are the special parameters immensely used in Bash Scripting. Here I am describing the fundamental uses of all of them

$$ :: PID of current shell
$? :: exit status of last executed command
$! :: PID of last background process
$# :: Total number of positional parameters.
$0 :: Name of command being executed.
$* :: List of all shell arguments(can't yield each parameter separately)
$@ :: Same as $*(yields each argument separately when enclosed in double quotes)

Some small scripts to show their uses:

$$ and $0

$ cat apl.sh
#!/bin/sh

tmpfile=/tmp/$0.$$
date >> $tmpfile
cat $tmpfile

$ ./apl.sh
Sat Mar 22 11:08:18 IST 2008

$ cat /tmp/apl.sh.4596
Sat Mar 22 11:08:18 IST 2008

so,
$0 in the script became "apl.sh" (the name of the command i.e. script that got executed)
and
$$=4596 was the PID of the shell where the script got executed.

$?

$ cat exitst.sh
#!/bin/sh

NAME=$1
grep "$NAME" ./names.txt
[ $? == 0 ] && echo "found" || echo "Sorry, not found"

$ ./exitst.sh Alex
Alex Cian
found

$ ./exitst.sh Leo
Sorry, not found


$* and $@
$ cat count1.sh
#!/bin/sh
c=1
while [ "$*" != "" ]
do
echo "$c)$1"
shift
((c+=1))
done

$ ./count1.sh unix bash scripting 1995
1)unix
2)bash
3)scripting
4)1995

Differnece between $@ and $*
Following script will tell you the difference between $@ and $*

$ cat diff.sh
file "$*"
echo "++"
file "$@"

$ ./diff.sh a.txt b.sh
a.txt b.sh: cannot open `a.txt b.sh' (No such file or directory)
++
a.txt: empty
b.sh: Bourne shell script text executable

"$*" can not expand the command line args, so expects for a file name "a.txt b.cpp"
"$@" takes both the file names separately.

$#

$ cat count.sh
#!/bin/sh
c=1
echo "Number of command line args: $#"
tot=$#
while [ $c -le "$tot" ]
do
echo "$c)$1"
((c+=1))
shift
done

$ ./count.sh 12 Alex India
Number of command line args: 3
1)12
2)Alex
3)India

Yes/No - BASH beginner

Some of the ways to write yes/no or confirmation code in BASH:

#Usual way:
echo -n "Do you want to exit ? (y/n)"
read ans
[ $ans == "y" ] && echo "Exiting ......" && sleep 2 && exit 0 || echo "Good"

#In a function:
f_yesno()
{
echo -n "Do you want to exit ? (y/n)"
read ans
[ $ans == "y" ] && return 0 || return 1
}

f_yesno && echo "exiting" && exit || echo "Not exiting"
echo "This is next line"

#One more way:
f_confirm() {

echo -n "$1 ? (n) "
read ans
case "$ans" in
y|Y|yes|YES|Yes) return 0 ;;
*) echo Exiting; return 1 ;;
esac
}

f_confirm "Delete?" && rm "/tmp/bashscript.out"

Points to digest:
1) Use of &&
2) return 0 implies success and non zero return value implies failure
3) Writing case in BASH
4) echo -n (do not output the trailing newline)
5) How to write a function in BASH

All users of a group

/tmp/passwwd is a demo passwd file.

$ cat /tmp/passwwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:2:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:2:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:12:news:/etc/news:

Objective:
Merge all users(username,$1) of the same group(groupid,$4) together.

Awk Solution:
$ awk -F ":" '{Arr[$4]=sprintf("%s,%s",Arr[$4],$1)} END {for ( i in Arr) {printf("gid=%s(users=%s)\n",i,Arr[i])}}' /tmp/passwwd | sed 's/=,/=/'

gid=7(users=lp)
gid=12(users=mail,news)
gid=0(users=root,sync,shutdown)
gid=1(users=bin)
gid=2(users=daemon,adm,halt)

AWK: Report Generation

summary.txt contains the details of the employee's working hours.

$ cat summary.txt
A|Jan|clerk|02:45
B|Jan|Salesman|02:12
C|Jan|Accountant|03:12
A|Feb|clerk|01:10
B|Feb|Salesman|11:10
B|March|Salesman|3:10
C|Feb|Accountant|3:34

The output is required in a report format, like this

A = 235 mins
B = 992 mins
C = 406 mins

or

A = 3:55
B = 16:32
C = 6:46

This is how it can be achieved:

#First format
$ awk 'BEGIN{FS="[|,:]"} {arr[$1]+=$(NF-1)*60+$NF} END {for (i in arr) {print i,"=",arr[i],"mins"}}' summary.txt

#Second format
$ awk 'BEGIN{FS="[|,:]"} {arr[$1]+=$(NF-1)*60+$NF} END {for (i in arr) {printf "%c = %d:%d\n",i,arr[i]/60,arr[i]%60}}' summary.txt

Points to digest:
1) FS="[|,:]" - how to specify multiple FS in AWK
2) NF=number of fields in a line, $NF prints the last field.
3) Use of associative array in AWK.

FIRST occurence of a pattern - BASH beginner

In order to find the line number of first occurrence of the word "bash" in the following file

$ cat shells
korn is excellent
c is superb
bash is best
bash is handy
all are best

$ grep -n "bash" shells | sed '1!d' | awk -F : '{print $1}'
3

$ awk -F : '/bash/ {print NR}' shells | sed '1 !d'
3

While Vs Until

A very very important academic question, difference between while loop and until loop

* While: executes the body of the loop so long as the conditional expression is true
* Until: executes the body of the loop so long as the conditional expression is false

Two simple scripts to illustrate the use of both:

$ cat while.sh
#!/bin/sh

count=1
while [ "$*" != "" ]

do
echo "Argument number $count): $1"
shift
count=`expr $count + 1`
done

$ cat until.sh
#!/bin/sh

count=1
until [ "$*" = "" ]
do
echo "Argument number $count): $1"
shift
count=`expr $count + 1`
done

$ ./while.sh one 2 Three 4
Argument number 1): one
Argument number 2): 2
Argument number 3): Three
Argument number 4): 4

* Similar output for until.sh

Same task, different ways of doing

Here I am discussing some small operations with their alternative ways of doing.

A) Removing Blank Lines
$ cat test.out
this is line 1
this is line 2
some lines here


there are 2 blank lines above
this is last line

$ grep -v '^$' test.out
$ grep '.' test.out
$ sed '/^$/d' test.out
$ sed -n '/^$/!p' test.out
$ awk NF test.out
$ awk '/./' test.out

$ cat test1.out
this is line 1
this is line 2
some lines here
this is last line

B) Lines that doesn't contain a particular regexp
$ grep -v 'this' test1.out
$ awk '!/this/' test1.out
$ sed '/this/d' test1.out
$ sed -n '/this/!p' test1.out

will print
some lines here

Printing last Line
$ tail -1 test1.out
$ sed '$!d' test1.out
$ sed -n '$p' test1.out
$ awk 'END {print}' test1.out

will print
this is last line

C) Printing first line
$ head -1 test1.out
$ sed q test1.out
$ awk 'NR>1 {exit};1' test1.out

will print
this is line 1

D) Number of lines in a file
$ wc -l test1.out | cut -d " " -f1
$ sed -n '$=' test1.out
$ awk 'END {print NR}' test1.out

E) Printing first 2 lines
$ awk 'NR<3'>
$ head -2 test1.out
$ sed '1,2 p' test1.out

$ cat names.out
Lehe
Alex
Themo
Demo
Saik

F) Print the line immediately before a regex, but not the line containing the regex

$ awk '/Alex/ {print x};{x=$0}' names.out
$ sed -n '/Alex/{g;1!p;};h' names.out
$ S=`grep -n Alex names.out | awk -F : '{print $1}'`; S=`expr $S - 1`; sed -n "$S p" names.out

will print
Lehe

G) print the line immediately after a regexp, but not the line containing the regexp

$ sed -n '/Lehe/{n;p;}' names.out
$ awk '/Lehe/{getline;print}' names.out
$ S=`grep -n Lehe names.out | awk -F : '{print $1}'`; S=`expr $S + 1`; sed -n "$S p" names.out

will print
Alex