Bash read command timed out feature

In bash scripting if you have a situation where you don't want to wait for user response forever, bash read command given an option (-t) which cause read command to time out in "number of seconds" specified with this -t option.

From read command man page:

-t timeout : Cause read to time out and return failure if a complete line of input is not read within timeout seconds. This option has no effect if read is not reading input from the terminal or a pipe.

-p prompt : Display prompt, without a trailing newline, before attempting to read any input. The prompt is displayed only if input is coming from a terminal.

Lets try a small one liner:

$ ans="y"
$ read -t 5 -p "Want to proceed ?(y/n)" ans; echo "You entered $ans"

Also notice the "$?" (exit status of last executed command) value in both the cases

a) When user has responded within that time
b) When its timed out

Its useful !!

Evaluate expression inside another evaluation - bash script

As we know, $( ) is used to to evaluate sub-expressions in bash shell (same as of back-tick in bash shell)

i.e.

$ var=`/sbin/ifconfig | grep eth0`

is same as

$ var=$(/sbin/ifconfig | grep eth0)

Lets discuss an example to see the good use of $( ) in bash shell.

#Suppose 'offsetno' variable is set to 9
$ offsetno=9

#Number of *.sh files under my directory (/tmp/mydir) is 29
$ ls /tmp/mydir/*.sh | wc -l
29

#Now to add the values of 'offsetno' with 'total number of .sh files' in my directory
$ expr $offsetno + `ls /tmp/mydir/*.sh | wc -l`
38

#Now if you need to store this output value in a variable called 'eff_num_file'
$ eff_num_file=`expr $offsetno + `ls /tmp/mydir/*.sh | wc -l``

Output:
expr: syntax error
64

#Oh what happened!! Lets echo 'eff_num_file' variable content; alas!! nothing got evaluated and stored.

$ echo $eff_num_file

#Now try this way:
$ eff_num_file=$(expr $offsetno + $(ls /tmp/mydir/*.sh | wc -l))

#See whats stored in eff_num_file variable, correct!!
$ echo $eff_num_file
38

What we noticed here is that back-tick within back-tick does not work, whereas $( ) inside $( ) work.

** The above example is just an example to explain the trick.

Find largest files in Linux

Must be very useful for the Linux newbies.

If you want to find and print the top 10 largest files names (not directories) in a particular directory and its sub directories

$ find . -printf '%s %p\n'|sort -nr|head

To restrict the search to the present directory use "-maxdepth 1" with find.

$ find . -maxdepth 1 -printf '%s %p\n'|sort -nr|head

And to print the top 10 largest "files and directories":

$ du -a . | sort -nr | head

** Use "head -n X" instead of the only "head" above to print the top X largest files (in all the above examples)

Related post:
Sort files and directories by size in Linux

Colorful man pages in Linux - ubuntu tips

By default, the man pages are formatted using Linux 'less' utility which displays the texts in blank and white colors. Using 'most' utility to view man pages gives a color formatted man page, something like below



Now the ways to install 'most' utility and making it default for viewing the man pages in your Linux box:

Install most
$ sudo apt-get install most

Make it default for man page viewer:
$ sudo update-alternatives --config pager

The above command will list the alternatives which provide 'pager'. You can make 'most' as default by pressing the corresponding selection number for it.

And you are done. Now try to view a man page of a command.

Give a try and enjoy colorful man pages on your ubuntu.

For the newbies:
apt-get - APT package handling utility -- command-line interface
most - browse or page through a text file
update-alternatives - maintain symbolic links determining default commands

Related post:
- Enable color support for Linux ls command

Linux command line history with timestamp - HISTTIMEFORMAT

Typically when one type history command, it displays the command number and the command. For auditing purpose, it would be helpful to display the timepstamp along with the command. To do so we need to set the environmental variable HISTTIMEFORMAT.

HISTTIMEFORMAT supports format strings of strftime.

Some important format strings:

%T Replaced by the time ( %H : %M : %S )
%F Equivalent to %Y - %m - %d

Get the full list here

$ export HISTTIMEFORMAT='%F %T '

Now execute

$ history

it will print the command line history with corresponding timestamp when the command was executed.

Diff remote files using ssh in Linux

I have already discussed how we can edit a remote file using vi and scp in one of my previous post ; today we will see how we can find or show differences between a local file and a remote file using ssh.

Suppose we have to find the differences between local file "/tmp/filepurge.sh.old" and remote file "/root/scripts/filepurge.sh" located in remote host 172.21.16.11.

This is how can do it:

$ ssh root@172.21.16.11 "cat ~/scripts/filepurge.sh" | diff - /tmp/filepurge.sh.old

And using vimdiff:

$ vimdiff scp://root@172.21.16.11//root/scripts/filepurge.sh /tmp/filepurge.sh.old

** We would need ssh to work using public key authentication so that we can do remote commands execution without being prompted for passwords.

Create a file with given size - Linux dd command

Linux dd command (convert and copy a file) can be used for this purpose.

e.g.

To create a 10 MB (10*1024*1024=10485760 bytes) size file named testfile_10MB

The command is:
$ dd if=/dev/zero of=testfile_10MB bs=10485760 count=1

1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 0.312 s, 33.6 MB/s

To confirm the size:

$ ls -lrth testfile_10MB
-rw-r--r-- 1 iam mygp 10M Jan 20 21:29 testfile_10MB

Where:

/dev/zero is a special file that provides as many null characters (ASCII NUL, 0x00) as are read from it.

if=FILE (read from FILE instead of stdin)
of=FILE (write to FILE instead of stdout)
bs=BYTES (force ibs=BYTES and obs=BYTES)
ibs=BYTES (read BYTES bytes at a time)
obs=BYTES (write BYTES bytes at a time)
count=1 (copy only 1 input block)

Count non empty fields in a file - awk

Input file: "4w_prod_u_1.txt" is the attendance report for production unit1 of a factory for a particular week.
If the employee was present for a day there will be a "y" for that day, if absent nothing will be put.

$ cat 4w_prod_u_1.txt
#Name,Designation,attendance details
Naveen A,Unit Mngr,y,y,,y,y,y
S Puri,Driller,,,y,,y,y
N Nandan,Factory Mngr,y,y,y,y,y,
K Asif,Unit Mngr,,,y,y,y,y

Required:
We need to count total days an employee was present in that week, basically to count total number of non empty fields after the first two fields in each line (except the first hear line) or to be more specific counting total number of "y" for each line (after the 1st two fields)

The awk solution:

$ awk > 4w_prod_u_1.out '
BEGIN {print "Name[Num Days Present]"}
!/^#/ {cnt=0; for (i=3; i<=NF; i++)
{if ($i != "") {cnt++} } {print $1,"["cnt" days]"} }
' FS=, 4w_prod_u_1.txt

Output file:
$ cat 4w_prod_u_1.out
Name[Num Days Present]
Naveen A [5 days]
S Puri [3 days]
N Nandan [5 days]
K Asif [4 days]

or the

if ($i != "")

in the above solution can be

if ($i == "y")
will produce the same result.

Linux paste command good examples & uses

From paste(1) man pages:

paste - merge lines of files

Re-discussing some of the usual uses that I posted in my previous post

$ cat f1.txt
a
b
c

$ cat f2.txt
1
2

Now:

$ paste f1.txt f2.txt
a       1
b       2
c

$ paste f2.txt f1.txt
1       a
2       b
        c

Using -d (delimiter) and -s (serial) options:

$ paste -s -d : f1.txt f2.txt
a:b:c
1:2

Now some additional examples:

Example 1:
We can specify a delimiters list instead of a single delimiter with -d option.

Suppose you have 3 files fa.txt, fb.txt and fc.txt.

$ cat fa.txt
20
60
90
12

$ cat fb.txt
60
90
12
14

$ cat fc.txt
70
90
80
12

Required:
The requirement is to add the individual lines from fa.txt with fb.txt and then subtracting the corresponding line from fc.txt.

Lets construct the expression first:

$ paste -d"+-" fa.txt fb.txt fc.txt
20+60-70
60+90-90
90+12-80
12+14-12

Then use bc to evaluate each line of expression like this:

$ paste -d"+-" fa.txt fb.txt fc.txt| while read line
do
echo -n "($line)="
echo $line | bc
done

Output:
(20+60-70)=10
(60+90-90)=60
(90+12-80)=22
(12+14-12)=14

Awk will be more simple in this case:

$ paste fa.txt fb.txt fc.txt | awk '{print "("$1"+"$2"-"$3")="$1+$2-$3}'

Output:
(20+60-70)=10
(60+90-90)=60
(90+12-80)=22
(12+14-12)=14

Things to learn from above example:

1) -n option with echo (do not output the trailing newline)
2) Specifying a delimiters list with -d option in paste

Example 2:
Input file: element.txt has the element name followed by the rpm in the following format.

$ cat element.txt
Element E1:
rpm=2300
Element E5:
rpm=8900
Element E3:
rpm=5000
Element E4:
rpm=1200

Required: Only list the elements which have rpm > 3000

$ paste - - < element.txt | awk '$NF>=3000 {print $1}' FS=[:,=]
Element E5
Element E3

Things to learn from above example:

1) Specifying multiple field separator (FS) with awk, my previous post on that
2) paste - - ( see example 3 below for more clarification)

Example 3: Again one more example using paste - -

Input file: details.txt has some records like this.

$ cat details.txt
Jweis
1982
awk
Lalit
1983
sed
Annie
1983
Perl

Required: We need to make a line out of 3 rows from the above file. i.e. merging 3 rows to 1 row

$ < details.txt paste - - -

Jweis 1982 awk
Lalit 1983 sed
Annie 1983 Perl

Awk ?: is also can be used here.

$ awk '{printf("%s",NR%3 ? $0"\t":$0"\n")}' details.txt

Jweis 1982 awk
Lalit 1983 sed
Annie 1983 Perl

Things to learn from above example:

1) awk ?: operator: If the first pattern is true then the pattern used for testing is the second pattern, otherwise it is the third. Only one of the second and third patterns is evaluated.

Few more posts based on Linux paste command
- Replace column with column of another file using awk click
- Merge alternate lines of files using awk click
- Sequence subtraction one liner click

Difference between awk NR and FNR variable

From awk(1) man pages, the definition of awk NR and FNR variables are:

NR ordinal number of the current record
FNR ordinal number of the current record in the current file

For a single file, FNR is nothing but NR (line number)

e.g.

$ cat main.txt
ID1:2.45
ID2:21.5
ID4:1.32
ID3:89.2
ID7:12.4
ID9:13
ID5:19.8

$ awk 'BEGIN {print "NR\tFNR\tline"}
{print NR"\t"FNR"\t"$0}
' FS=: main.txt

NR      FNR     line
1       1       ID1:2.45
2       2       ID2:21.5
3       3       ID4:1.32
4       4       ID3:89.2
5       5       ID7:12.4
6       6       ID9:13
7       7       ID5:19.8

And for more than one file, NR and FNR will be equal for the first processed file, but on the first line of the second and subsequent files FNR will start from 1 again.

Lets have one more file named filter.txt

$ cat subfile.txt
ID9
ID3
ID4

Now, lets print the same NR, FNR and the lines with both the files combined.

$ awk 'BEGIN {print "NR\tFNR\tline"}
{print NR"\t"FNR"\t"$0}
' FS=: subfile.txt main.txt

NR      FNR     line
1       1       ID9
2       2       ID3
3       3       ID4
4       1       ID1:2.45
5       2       ID2:21.5
6       3       ID4:1.32
7       4       ID3:89.2
8       5       ID7:12.4
9       6       ID9:13
10      7       ID5:19.8

You might be wondering, what can be the use of awk NR and FNR combination. I already did a few posts using NR==FNR functionality.

Lets discuss one of them.

$ cat main.txt
ID1:2.45
ID2:21.5
ID4:1.32
ID3:89.2
ID7:12.4
ID9:13
ID5:19.8

$ cat subfile.txt
ID9
ID3
ID4

Requirement: We need to delete those rows from main.txt for which the id field (1st field) has an entry in subfile.txt.

so, required output:

ID1:2.45
ID2:21.5
ID7:12.4
ID5:19.8

The solution is:

$ awk 'NR==FNR{A[$1];next}!($1 in A)' FS=: subfile.txt main.txt

So, associative array named A is created with the Ids of 1st processed file (subfile.txt, for which NR==FNR, as it is first in the sequence "subfile.txt main.txt" above).

To confirm,

$ awk 'NR==FNR {print $0,FILENAME}' FS=: subfile.txt main.txt

ID9 subfile.txt
ID3 subfile.txt
ID4 subfile.txt

So associative array A will contain the elements ID9 ID3 and ID4.

We proceed to the next lines and printing only those of remaining rows, which does not (!) have 1st filed as any one of the elements of the associative array A.

Related posts:

- awk FNR variable usage example
- update file based on another file in awk
- match words, bash newbie

Redirecting MAN pages to file - linux

If we try normal bash redirection to redirect a man page to a file, the output file will produce many unreadable** characters.

So in order to redirect we need to use col command (which filter reverse line feeds from input)

$ man ls | col -bx > /tmp/ls.man

Where:

col - filter reverse line feeds from input
-b Do not output any backspaces, printing only the last character written to each column position.
-x Output multiple spaces instead of tabs.

** In my Ubuntu 7.10, normal redirection of man pages work fine, but in my debian 3.0 desktop, or in the cygwin shell, normal redirection does not work.

Related post:

- How to open man pages in vi editor - Linux
- Redirect top command output to a file

Test or check ssh connection in bash script

Many times I have come across situations where I need to test a SSH connection to a remote server before performing some operation using ssh in that box. Here is a bash script function to test a ssh connection.

....
....
_TestSSH() {
local user=${1}
local host=${2}
local timeout=${3}

ssh -q -q -o "BatchMode=yes" -o "ConnectTimeout ${timeout}" ${user}@${host} "echo 2>&1" && return 0 || echo "Make sure you have access to ${host}"

}

_TestSSH root 172.24.0.102 5 && _somefunction

...
...

Where:

From man pages of ssh command:

-q (Quiet mode) - Causes all warning and diagnostic messages to be suppressed. Only fatal errors are displayed. If a second -q is given then even fatal errors are suppressed.

echo 2>&1 - I have used this command, but this can be any command to execute in the remote host as a test command.

Convert seconds to hour minute seconds format - bash and awk

Lets see how using bash printf command we can format seconds to h:m:s format.

The bash script "convert.sh"

#!/bin/sh
#Convert seconds to h:m:s format

[ -z ${1} ] && echo "Usage: $(basename $0) <seconds>" && exit||secs=${1}
_hms()
{
 local S=${1}
 ((h=S/3600))
 ((m=S%3600/60))
 ((s=S%60))
 printf "%dh:%dm:%ds\n" $h $m $s
}

_hms ${secs}

Executing:

$ ./convert.sh
Usage: convert.sh <seconds>

$ ./convert.sh 456
0h:7m:36s

The bash one liner for the above script will be:

$ secs=456
$ printf ""%dh:%dm:%ds"\n" $(($secs/3600)) $(($secs%3600/60)) $(($secs%60))

0h:7m:36s

The awk one liner for this:

$ echo - | awk -v "S=456" '{printf "%dh:%dm:%ds",S/(60*60),S%(60*60)/60,S%60}'

0h:7m:36s

Sort files and directories by size - Linux bash

In my current directory, I have the following files/sub-directories:

$ file *
conf: directory
main.dat: ASCII text
temp.txt: ASCII text
tmplog: directory

i.e.
2 files and 2 directories

Now to list the files and directories (in the current directory) in the descending order of file size.

$ du -sk * | sort +0nr
4260 tmplog
2980 conf
1100 main.dat
968 temp.txt

Listing only the directories (in the current directory) in descending order of size:

$ du --max-depth=1 . | sort -nr
9308 .
4260 ./tmplog
2980 ./conf


Since the sub-directories also contains files; lets list all the files (not directories) in the descending order of file sizes

$ find . -type f -exec du -sk {} \; | sort -nr
2196 ./tmplog/main1.dat
2064 ./conf/pi.dat
1100 ./tmplog/huo.txt
1100 ./main.dat
968 ./temp.txt
964 ./tmplog/as.txt
916 ./conf/lis.txt

And if you want to restrict find to the current directory only and not the sub-directories, use maxdepth=1. i.e.

$ find . -maxdepth 1 -type f -exec du -sk {} \; | sort -nr
1100 ./main.dat
968 ./temp.txt

Round functionality with awk - bash shell

As there is no standard "round" function available in awk, some of the ways to round float values to the rounded integer.

$ echo "23.54" | awk '{printf("%d\n",$1 + 0.5)}'
24

$ echo "23.49" | awk '{printf("%d\n",$1 + 0.5)}'
23

Lets make this a function named round.

$ echo "23.54" | awk '
function round(A) {
return int( A + 0.5 )
}
{
printf("%d\n",round($1));
}'
24

Another way:

$ echo "23.54" | awk '{printf("%d\n",$0+=$0<0?-0.5:0.5)}'
24

Related post:
round float values using sprintf in awk

Insert, append, change lines using sed

Input file:
$ cat order789.txt
title:
PO for vessel unit1 const.
items:
fan:F34539
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar

# Add the line "heater:M21789" after the line "items"
$ sed '
/items/ a\
heater:M21789
' order789.txt

Output:
title:
PO for vessel unit1 const.
items:
heater:M21789
fan:F34539
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar

# Add the line "heater:M21789" after line number 3 (output would be same as above)
$ sed '
3 a\
heater:M21789
' order789.txt

# Insert a line "heater:M21789" before the line beginning with "fan"
$ sed '
/^fan/ i\
heater:M21789
' order789.txt

Output:
title:
PO for vessel unit1 const.
items:
heater:M21789
fan:F34539
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar

# We can insert or add more than one line also.
$ sed '
4 i\
heater:M21789\
newitem:YYYYY
' order789.txt

Output:
title:
PO for vessel unit1 const.
items:
heater:M21789
newitem:YYYYY
fan:F34539
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar


# One can change a line as well, e.g. to change the line beginning with "fan" with the line "BigFAN:F5757"
$ sed '
/^fan/ c\
BigFAN:F5757
' order789.txt

Output:
title:
PO for vessel unit1 const.
items:
BigFAN:F5757
tube:L1245
driller:M4545
Description:
PO signed and verified by factory manager S K Lp
Date: Fri Jan 2 17:26:44 UTC 2009
Author: M Kumar

In all the above cases one can mention the line numbers instead of the pattern before or after which one or multiple line(s) needs to be added/inserted or changed.