Awk - Print The Field After a Regex Match on a Field

awk '/regex2match/{for(i=1;i<=NF;++i)if($i~/regex2match/){print $(i+1);break}}'

Thanks to PHV at: http://www.tek-tips.com/viewthread.cfm?qid=1459823

List All Installed Packages via /var/log/dpkg*

alias lspkgs='cat <(ls -f1 /var/log/dpkg* | grep -v "gz\$") <(cat $(ls -f1 /var/log/dpkg* | grep "gz\$") | gunzip) | sort -t "." -k 3 -n | tac | grep installed | grep -v "half-installed" | grep -v "not-installed" | awk "/installed/{for(i=1;i<=NF;++i)if(\$i~/installed/){print \$(i+1);break}}" | sort | uniq' 

Sed Inserts via Process Substitution w/out backslashes

Sometimes its useful to be able to insert text wherever you would like in a file using a function that generates a multiline output. Sed is useful for the task but not without the gotcha of backslashes, all but the last backslash will normally remain unless you work to prevent sed from leaving them in.

#!/bin/bash 

declare -r TST_FILE="/tmp/__${0##*/}__$$" 

function print_header() { 
cat << EOF
-------------------------------------------------------------------
-------------------------------------------------------------------
-------------------------------------------------------------------
$(date '+%B %d, %Y @ ~ %r')  ID:$RANDOM
EOF
} 

function gen_test_file() { 
cat << EOF
line 1
line 2
line 3
line 4
line 5
EOF
}

gen_test_file > "${TST_FILE}"

[[ -z ${1} ]] && read -p "Enter line # to insert the multiline header (0-5):" -u 0 response || response=${1}  
[[ ! -z ${response} ]] && grep -q -o '^[0-9]*$' <<< "${response}" && ((${response} > -1)) && ((${response} < 6)) && : || 
{ echo "*** Out of range line ($response) using line zero.." && response=0; }
  ## If its before the first line (line 0) we need to use insert i because we can't replace address 0 in sed. 
((${response}==0)) && sed "1 i $(print_header | sed 's/$/\\/g;$s/$/\x01/')" "${TST_FILE}" | tr -d '\001' ||
  ## Otherwise we can use a slightly easier on the eyes replacement r.  
echo -en "$(print_header)\n" | sed ${response:-1}r/dev/stdin "${TST_FILE}"
rm -f "${TST_FILE}"
exit 0 

Thanks to Triplee for the sed r/dev/stdin idea.

Creating RDBMS-Style Views on your Fs w/ROFS-Filtered

Sometimes its useful to abstract a segment of a filesystem for variety of reasons: some applications don't offer the luxury of filtering filetypes. Some do yet have no clue what you want, such as Music Players when there are multiple formats of the same song if you add the directory the multiply formatted songs are duplicated.

#!/bin/bash 

declare -r TEST_DIR="/tmp" 
 ## If you update the following var, update gen_rofs_inverted_rc 
 ## accordingly. 
declare -r TEST_DATA_DIR="/tmp/rofs_data" 
declare -r TEST_MNT_POINT="/tmp/mnt/rofs"
declare -r FILE_FILTER="jpg"
declare -r ROFS_RC="rofs_only_jpg.rc" 
declare    ROFL_FILTERED_DIR=${1:-$(pwd)} 
 ## Add the default build location for fetches, just in case. 
PATH="${PATH}:${ROFL_FILTERED_DIR}:${TEST_DIR}/rofs-filtered" 

function preliminary_checks() { 
 if ! which rofs-filtered &>/dev/null; then     
  echo -en "\nCommand rofs-filtered not found in current PATH, try passing in the \
  directory on the cmdline\nE.g. ${0##*/} /path/to/built/software\n\n"   
  fetch_and_build_rofs_filtered
 fi 
} 

function fetch_and_build_rofs_filtered() { 
 read -p "Enter f to fetch/build the latest rofs-filtered from git, anything else to continue:" -u 0 response 
 case ${response} in 
  f)   
    cd "${TEST_DIR}" 
    git clone https://github.com/gburca/rofs-filtered.git
    cd rofs-filtered 
    ./autogen.sh 
    ./configure
    make
    ROFL_FILTERED_DIR="${TEST_DIR}/rofs-filtered" 
    PATH="${PATH}:${ROFL_FILTERED_DIR}"    
    cd .. 
    ;;
  *)     
    echo "You need to find rofs-filtered and put in the current PATH." 
    ;;
 esac 
} 

function gen_test_data() { 
 cd "${TEST_DATA_DIR}"
 ls -f1 | grep -E 'jpg$|mp3$|mp4$' | xargs -I {} rm {}
 touch fav_media_{1..100}.jpg
 touch fav_media_{1..100}.mp3
 touch fav_media_{1..100}.mp4
 cd - &>/dev/null
} 
 
 ### Adjust this to reflect any change in the location of the 
 ### test data directory. 
function gen_rofs_inverted_rc() { 
cat <<EOF
\/\$
\/\.\$
\/\.\.\$

tmp\$
rofs_data\$
./.+\.${FILE_FILTER}\$

EOF
} 

function unmount_rofs() { 
 mount | grep -q "${TEST_MNT_POINT}" && sudo umount "${TEST_MNT_POINT}"  
}

function clean_up() { 
 unmount_rofs
 rm -rf "${TEST_DIR}/${ROFS_RC}"
 rmdir "${TEST_MNT_POINT}" 2>/dev/null 
} 

function test_ls() { 
 echo "ls ${TEST_MNT_POINT} ..."
 ls "${TEST_MNT_POINT}/${TEST_DATA_DIR}" | tail -10 
} 
 
function main() { 
gen_rofs_inverted_rc > "${TEST_DIR}/${ROFS_RC}"
mkdir -p "${TEST_MNT_POINT}" "${TEST_DATA_DIR}"
gen_test_data 
preliminary_checks
which rofs-filtered &>/dev/null || exit -1 
unmount_rofs
rofs-filtered "${TEST_MNT_POINT}" -o invert -o source=/ -o config="${TEST_DIR}/${ROFS_RC}"
test_ls
} 

function what_likely_happened() { 
cat <<EOF

Here is what *likely* just happened:  
 
 Test data directory (${TEST_DATA_DIR}) was created and 
 loaded with empty files. 
 
 The following files were generated: 

  ROFS Filtered configuration file was created: ${TEST_DIR}/${ROFS_RC}
  The data from the test data directory above is now mounted and filtered 
  at ${TEST_MNT_POINT}. 
  
  If you traverse to ${TEST_MNT_POINT}/${TEST_DATA_DIR} you should see 
  only files with the .$FILE_FILTER extension. 
  
  To clean it up, just run it with a second argument on the commandline. 
  E.g. ${0##*/} "/path/to/rofs-filtered/bin/dir" cleanupthismess
   
EOF
} 

main 
what_likely_happened 
 ## Any second argument attempts a cleanup. 
 ## If you put rofs-filtered in your path and don't want to pass it in to 
 ## clean it up just put $(which rofs-filtered). 
(($# > 1)) && echo "Cleaning up ... (except for /tmp/mnt)" && clean_up

exit 0 

Sed Global Substitution - Ordering Matters!

I noticed a little nuance while programmatically generating some code segments (programming programming). It seems obvious, when I program by hand I never have the issue since I'm fairly tuned into the logical flow, but in the case of programming programming it makes sense to keep in mind that the order of global substitutions matters.

Test string: "trouble_%here_(_if_order_i$_wrong.html"

\x25 is the hexcode for percent %, if the substitution is on the end of the righthand side of the sed global substitutions list there is trouble since subsequent substitutions have their change applied.

E.g.

sed 's/\x20/%20/g;s/\x21/%21/g;s/\x22/%22/g;s/\x23/%23/g;s/\x5c\x24/%24/g;s/\x25/%25/g'  
<<< "trouble_%here_(_if_order_i$_wrong.html"

trouble_%25here_(_if_order_i%2524_wrong.html

Since the replacement of percents is all the way on the righthand side (generated by sequence {20..25}), 
the sed replacements of other special characters incorrectly have the resulting %'s from their
respective substitutions replaced.  

sed 's/\x25/%25/g;s/\x20/%20/g;s/\x21/%21/g;s/\x22/%22/g;s/\x23/%23/g;s/\x5c\x24/%24/g'
<<< "trouble_%here_(_if_order_i$_wrong.html"

trouble_%25here_(_if_order_i%24_wrong.html

Since the percent substitution is now on the lefthand side, sed parses the string of any
existing %'s and leaves the subsequent substitutions alone.

Mapping PID to Window ID / For Focusing and More!

CMD_PID=${1} && while IFS= read -r -d '' var; do 
if grep -q "^WINDOWID=" <<< "$var"; then 
 winid=$(printf '0x%x\n' $(sed 's/WINDOWID=//' <<< "${var}"))
 child_cnt=0 && IFS=$(echo -en "\n\b") && 
for a in $(xwininfo -root -tree | tac | sed -n "/${winid}/,\$p"); do 
 grep -q -i "children" <<< "${a}" && let child_cnt+=1
 ((${child_cnt} == 2)) && real_winid=$(grep -o '0x[^ ]*' <<< "${last_line}") && break
 last_line="${a}"
 done 
 xdotool windowraise ${real_winid} 
 break 
fi 
done < /proc/${CMD_PID}/environ

Parsing Substrings w/Sed, Grep

Sometimes its not enough to have a simple substitution possibility on a regex match, and it would be useful if you could match a regex in a string and then perform a further regex / substitution. This would probably be a useful and welcome improvement to sed, if it were incorporated. For example if you want to make a replacement only within quoted text in a string, in lets say you have a csv file named test.csv that looks like:

test.csv:
2015-03-23 08:50:22,Jogn.Doe,1,1,Ineo 4000p,"Microsoft, Word, Document1"
2015-03-23 09:34:11,"John,Doe",3,4,Canon 2000,"Further, comma, trouble"

When it's parsed as a csv the commas within in the quotes cause trouble, its necessary to get rid of those commas at least temporarily. It would be useful if we could do it with sed in one pass.

Wishful (non-existant) sed command: sed 's/"[^"]*"/{s/,/ /g}/g'

Instead we can use bash and grep to loop over each line and then search and replace the text with our parsed versions.

FILE=test.csv && i=1 && IFS=$(echo -en "\n\b") && for a in $(< "${FILE}"); do 
 var="${a}"
 for b in $(sed -n ${i}p "${FILE}" | grep -o '"[^"]*"'); do 
  repl="$(sed "s/,/ /g"  <<< "${b}")" 
  var="$(sed "s#${b}#${repl}#" <<< "${var}")" 
 done 
 let i+=1
 echo "${var}" 
done    

When run, all commas from the quotes are removed:
2015-03-23 08:50:22,Jogn.Doe,1,1,Ineo 4000p,"Microsoft Word Document1"
2015-03-23 09:34:11,"John Doe",3,4,Canon 2000,"Further comma trouble"