Bash - binary hexadecimal conversion using wcalc

wcalc - a natural-expression command-line calculator

Just came through wcalc, which is a command-line calculator designed to accept all valid mathematical expressions.

From man pages of wcalc:
It supports all standard mathematical operations, parenthesis, brackets, trigonometric functions, hyperbolic trig functions, logs, and boolean operators.

If no mathematical expression is given at the command line, wcalc enters "interactive" mode. Interactive mode has more features. Also, files may be piped to wcalc, and they will be interpreted.

Install wcalc in your Ubuntu or Debian:
$ sudo apt-get install wcalc

The first thing I tried with wcalc is hexadecimal, binary, octal, decimal conversion. It gives ready made option for the same, which I am going to explain in this post. Later I will put a detail post on its other usages.

In wcalc output:

- Number starting with 0x are interpreted as hexadecimal
- Number starting with 0b are interpreted as binary
- Number starting with 0 are interpreted as octal

Some of the sample conversions:

#Hexadecimal(base 16) of 13
$ wcalc -h 13
= 0xd

#Binary(base 2) of 5
$ wcalc -b 5
= 0b101

#Octal(base 8) of 11
$ wcalc -o 11
= 013

#Decimal(base 10) of binary 111
$ wcalc -d 0b111
= 7

#Option -q(quiet), Toggles whether the equals sign will be printed before the results.
$ wcalc -q -b 5
0b101

Related post:
- Bash desk calculator dc explained
- Hex to decimal conversion in bash shell

Print only matched string not line in grep

Suppose in your sample file the word 'bash' is present in different cases (some are BASH,some bash, some Bash etc etc)

Now if you want to see all the occurrences of the string bash, you might not be interested to see the lines containing the different format of the word 'bash'.

So you can use -o option along with -i option with GNU grep

From GREP(1) man pages

-o, --only-matching
Show only the part of a matching line that matches PATTERN.

-i, --ignore-case
Ignore case distinctions in both the PATTERN and the input files.

$ cat new.txt
help with bash
bashscripting 0.23
Index and match
last line

So,

$ grep -oi "bash" new.txt
Bash
bash
BASH

Bash function to compare multiple numbers equality

Initially I just made a simple function this way:

_isEqual () {
   local f1=$1
   local f2=$2
   [ $f1 -eq $f2 ] && echo "OK" || echo "Not"
}

Which basically check for equality of two numbers.

Then I thought of making it a function which can accept more than two arguments (numbers, float or integer) and check if all are equal.
The idea is to assign the list of numbers to a bash array, and then comparing each number (other than first one) with the first number (1st element in the array).
If they are equal then setting a variable (here k) to 1 else to 0 and performing this operation for all the elements on the array. At the end, if the value of the variable is 1, it means all the numbers are equal else not.

_isEqual () {
arr=("$@")
first=${arr[0]}
len=${#arr[*]}
i=0
k=0

while [ $i -lt $len ]
   do
      val=$(echo "${arr[$i]}")
      #[ $val -eq $first ] && k=1 || k=0
      #Support to compare float values as well
      [ $(echo $val == $first|bc) -eq 1 ] && k=1 || k=0
      let i++
   done
[ $k -eq 1 ] && echo "All equal" || echo "NOT"
}

#Some of the calls of the function _isEqual
_isEqual 2.3 4
_isEqual 90 90 90
_isEqual 2.34 2.34 2.34 2.34

Related post:
- Float comparison in bash script using bc and array
- Bash array introduction and examples

Print characters before and after a pattern - awk

e.g.

$ var="abcdefghij0123456789"

$ echo $var
abcdefghij0123456789

Now to print 3 characters before the pattern "h" in $var
$ echo $var | awk 'match($0,"h"){print substr($0,RSTART-3,3)}'

Output:
efg

And to print 4 characters after the pattern "h" in $var
$ echo $var | awk 'match($0,"h"){print substr($0,RSTART+1,4)}'

Output:
ij01

Combining, i.e. to print 3 characters before "h" and 4 characters after "h"
$ echo $var | awk 'match($0,"h"){print substr($0,RSTART-3,3),substr($0,RSTART+1,4)}'

Output:
efg ij01

Some awk terms:

match(string, regex): Returns the position of the first match for the regular expression regex in string, or 0 if no matches are found. Sets RSTART and RLENGTH variables.

substr(string, start [,length]: Return length characters from the specified string, starting from start. If length is not specified, return rest of record.

RSTART: Index of first character matched by a successful call to the match() function.

RLENGTH: Length of string matched by a successful call to the match() function.

Related post:
- Awk substr function explained
- Replace digit with serial number - awk
- Check for presence of a pattern in a line - awk

Find files that do not contain a pattern - linux

Requirement:
Find files which do not contain a particular pattern or text in one of its lines.

Well, when I first heard of this requirement; I just thought I will use "grep -vl pattern" piped with xargs to a regular find command (i.e. "find . -type f | xargs grep -vl pattern")

Then I realized that "grep -v" only works in line level, i.e. a file which contain the "pattern" in some line(s) may also contain some line(s) which "do not" contain that "pattern"; so as a whole that file will also be in the list of files which "do not" contain the "pattern" even though it contains the pattern.

So the ways would be:

e.g. To find all files which do not contain the pattern "void"

$ find . -type f \! -exec grep -q "void" {} \; -print

From man page of grep command:

-q, --quiet, --silent : Quiet; do not write anything to standard output. Exit immediately with zero status if any match is found, even if an error was detected.

-l : Suppress normal output; instead print the name of each input file from which output would normally have been printed.

-v : Invert the sense of matching, to select non-matching lines.

One more way would be:

$ find . -type f | while read file
    do
        grep "void" $file > /dev/null
        [ $? -ne 0 ] && echo $file
    done

Related post:
- Linux find command and logical operations
- Exclude directory from Linux find command

Remove characters using linux colrm command

colrm is a column removal filter.

If only one parameter is specified, the characters of each line will be removed starting from that specified column number.

and if called with two parameters (range of character position to remove) e.g.

$ colrm x y < file.txt

the columns/characters from character position x to character position y will be removed.

Some examples:

# Remove characters from 2nd character position till end
$ echo "abcdefghij" | colrm 2
a

# Remove characters from 2nd to 5th column position
$ echo "abcdefghij" | colrm 2 5
afghij

#The following command deletes the characters in column 4-8 from the file file.txt
$ colrm 4 8 < file.txt

Related post:
- Print or remove first some characters of a string - bash

Handling argument list too long - bash

I have nearly 200,000 files in one of my log directory out of which number of files of the name format "ka.log.*" is 120,000. So whenever I try to do apply some command such as rm, ls or cp etc on those big set of "ka.log.*" files, I used to get

$ ls ka.log.*
bash: /bin/ls: Argument list too long

$ cp ka.log.* new/
bash: /bin/cp: Argument list too long

$ mv ka.log.* new/
bash: /bin/mv: Argument list too long

$ rm ka.log.*
bash: /bin/rm: Argument list too long

"Argument list too long" error for the above commands is due to the limitation of the command (rm, mv, ls, cp) to handle large number of files(arguments).

Linux 'find' command is useful to perform these operations (ls, cp, mv or rm etc) on such big set of files/arguments.

e.g.

To copy those "ka.log.*" to directory /somedir

$ find . -name "ka.log.*" -exec cp {} /somedir/ \;

Looping through while:

find .  -name "ka.log.*" | while read FILE
    do
    ...
    <some operation on $FILE>
    ...
done

Another way is to assign the file names to a variable, e.g.

FILES=$(echo /mydir/ka.log.*)

for FILE in $FILES
    do
    ...
    <some operation on $FILE>
    ...
done

Break line in fixed width - linux fold command

Today I came to know about Linux 'fold' command, using which we can wrap each input line to fit in specified width.

If width is not specified using -w option, by default, 'fold' breaks lines wider than 80 columns. The output is split into as many lines as necessary.

Lets have one example to see how fold can be useful in breaking long lines in a file.

Print the alphabets from A-Z

$ echo "A B C D E F G H I J K L M N O P Q R S T U V W X Y Z"
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

** The above can be output using Linux jot command i.e.
$ jot -c 26 A | tr '\n' ' '

If we need to break the above one line into 8 alphabets per line, we need to specify the width as 16.

$ echo "A B C D E F G H I J K L M N O P Q R S T U V W X Y Z" | fold -w 16

Output:
A B C D E F G H
I J K L M N O P
Q R S T U V W X
Y Z

Sed and Awk alternatives for the above task will be:

$ echo "A B C D E F G H I J K L M N O P Q R S T U V W X Y Z" | sed -e "s/.\{16\}/&\n/g"

$ echo "A B C D E F G H I J K L M N O P Q R S T U V W X Y Z" | awk 'BEGIN{n=1}{while(substr($0,n,16)){print substr($0,n,16);n+=16}}'

Also from man pages of 'fold' command, other useful options with 'fold' are:

`-b'
`--bytes'
Count bytes rather than columns, so that tabs, backspaces, and
carriage returns are each counted as taking up one column, just
like other characters.

`-s'
`--spaces'
Break at word boundaries: the line is broken after the last blank
before the maximum line length. If the line contains no such
blanks, the line is broken at the maximum line length as usual.

Count total occurrence of a pattern - awk

Lets see how we can use awk FS variable to count the total occurrence of a pattern (word or character) in a file.

Input file:
$ cat file.txt
bash scripting bash
,bash, awk
sed,expect, python|awk

Required: Count how many times the word "bash" is occurred in the above file.

Awk solution:

$ awk -F "bash" '{print NF-1}' file.txt
2
1
0

Now adding them:

$ awk -F "bash" '{print NF-1}' file.txt | awk '{s+=$0} END {print s}'
3

So combining:

$ awk -F "bash" '{s+=(NF-1)} END {print s}' file.txt
3

So the word "bash" has occurred 3 times in the above file.

The default field-separator (FS) of awk is a tab or spaces. Using the word or character as the FS, we can count the total occurrence of that word or character in the above way.

Linux - copy file and preserve timestamp, ownership, mode

If you want to copy files in Linux and also want to keep or preserve the original mode or timestamp or ownership (or all) , cp command gives an option (--preserve).

From cp command man pages:

--preserve[=ATTR_LIST]
preserve the specified attributes (default: mode,ownership,timestamps) and security contexts, if possible additional attributes: links, all

Lets discuss this with some small examples.

I am logged in as user 'jk'

$ id
uid=32321(jk) gid=700(staff)

The example file tre.sh is having the following details:

$ ls -l tre.sh
-rw-r--r-- 1 jk staff 476 2009-01-13 16:20 tre.sh

Lets copy tre.sh to /tmp/tre.sh

$ cp tre.sh /tmp/tre.sh

So the timestamp is changed to the present timestamp

$ ls -l /tmp/tre.sh
-rw-r--r-- 1 jk staff 476 2009-02-05 15:10 /tmp/tre.sh

Now copy using "--preserve=timestamps" option.

$ cp --preserve=timestamps tre.sh /tmp/tre.sh.1

The original timestamp is preserved here

$ ls -l /tmp/tre.sh.1
-rw-r--r-- 1 jk staff 476 2009-01-13 16:20 /tmp/tre.sh.1

Now I just switched to root user

$ id
uid=0(root) gid=0(root) groups=0(root)

Copy tre.sh to /tmp/tre.sh.2

$ cp tre.sh /tmp/tre.sh.2

Notice the ownership and timestamp of the /tmp/tre.sh.2

$ ls -l /tmp/tre.sh.2
-rw-r--r-- 1 root root 476 2009-02-05 15:13 /tmp/tre.sh.2

You can preserve the ownership like this:
$ cp --preserve=ownership tre.sh /tmp/tre.sh.4

So /tmp/tre.sh.4 is still owned by user jk" (copied by root though)
$ ls -l /tmp/tre.sh.4
-rw-r--r-- 1 jk staff 476 2009-02-05 15:14 /tmp/tre.sh.4

Also we can specify "--preserve=ownership,timestamps" and also preserve the mode(permission) of the file with "--preserve=mode"

The cp command -p option is equivalent to --preserve=mode,ownership,timestamps

I am still 'root'; now copy using -p option

$ cp -p tre.sh /tmp/tre.sh.5

All the original attributes (mode,permission,ownership) of tre.sh is preserved.

$ ls -l /tmp/tre.sh.5
-rw-r--r-- 1 jk staff 476 2009-01-13 16:20 /tmp/tre.sh.5