Print all duplicate lines using awk

Input file:

$ cat cnr.txt
HIDDENHAUSEN|99.60
FIEBERBRUNN|99.07
MELLENDORF|99.04
HERBSTEIN|99.02
ACHTERWEHR|98.82
GOLM|98.82
PARA|98.82
BOGEN|98.61
SAINTANDRE|98.55
OLSZTYN|98.61
HYDERABAD|99.02

Output required: Print those lines for which 2nd field has occurred more than once. i.e. required o/p:

HERBSTEIN|99.02
ACHTERWEHR|98.82
GOLM|98.82
PARA|98.82
BOGEN|98.61
OLSZTYN|98.61
HYDERABAD|99.02

Awk solution:

$ awk 'NR==FNR && a[$2]++ {b[$2];next} $2 in b' FS="|" cnr.txt cnr.txt

Related posts:
- Difference between awk NR and FNR variables
- Posts on awk NR==FNR
- Awk FNR variable usage example
- Remove duplicates based on field using awk
- Remove duplicates from file without sorting using awk

Split a string to characters in Bash

Input string:

1234 5678

o/p required:

1
2
3
4

5
6
7
8

Using -w option of Linux fold (wrap each input line to fit in specified width)

$ echo "1234 5678" | fold -w1

Awk solution : Make field separator (FS) as NULL so that each character represents a field

$ echo "1234 5678" | awk 'BEGIN{FS=""}{for(i=1;i<=NF;i++)print $i}'

Using bash:

#!/bin/sh

str="1234 5678"
while [ -n "$str" ]
        do
                printf "%c\n" "$str"
                str=${str#?}
done

Using sed (Using & as matched string)

$ echo "1234 5678" | sed 's/[0-9 ]/&\n/g'

Close all windows in vimdiff - vi tip

vimdiff
a powerful file comparison tool (edit two or three versions of a file with Vim and show differences)

Two important window commands to close windows while performing vimdiff on multiple files. (Also applicable when you open multiple files using :split or :vsplit)

ctrl-w c - Close this window
ctrl-w o - close all Other windows (mnemonic - Only)

Two of vimdiff important commands:

do - (diff-obtain) obtain differences from other window into this one
dp - (diff-put) push differences in current window to the other window

Remove duplicate lines from vi editor

In order to remove all duplicate lines (rows) from a file which is opened in vi editor,

Use the following command

:sort u

which will sort all lines and remove duplicates and will keep only unique lines.
Other solutions for removing duplicate lines from vi editor can be found here

Related posts:
- Remove duplicate consecutive fields or lines from file
- Remove duplicate blank lines using awk in bash
- Remove duplicates based on fields using awk
- Remove duplicate lines without sorting file using awk

Print last section using awk RS variable

This post is mainly intended for the awk newbies.
Input file:

$ cat grn.log
Sat Jun 27 20:56:36 IST 2009 : Init
Sat Jun 27 20:56:39 IST 2009 : Phase1,mval=45
Sat Jun 27 20:56:46 IST 2009 : End
---
Sat Jun 27 21:06:15 IST 2009 : Phase1.4
Sat Jun 27 21:06:39 IST 2009 : Phase4,kval=23
Sat Jun 27 21:06:59 IST 2009 : Phase5,kval=29
Sat Jun 27 21:07:36 IST 2009 : End
---
Sat Jun 27 21:15:36 IST 2009 : Init
Sat Jun 27 21:16:29 IST 2009 : Phase1,mval=42
Sat Jun 27 21:16:46 IST 2009 : Cont

Output Required: Print the last section of the above file.

In general AWK reads one line at a time, and breaks up the line into fields.
We can print the last line this way:

$ awk 'END{print}' grn.log

o/p:
Sat Jun 27 21:16:46 IST 2009 : Cont

Now setting awk RS variable to "---", we can tell awk to treat a whole section as a line and then we can print the last section in the same way as above.

$ awk 'BEGIN{RS="---"}END{print}' grn.log

o/p:
Sat Jun 27 21:15:36 IST 2009 : Init
Sat Jun 27 21:16:29 IST 2009 : Phase1,mval=42
Sat Jun 27 21:16:46 IST 2009 : Cont