Bash - Auto Tailing new files in a directory

Post date: Dec 23, 2015 5:7:2 PM

I had a simple problem, I had multiple log files I needed to monitor and have scripts act on the results. But tail does stop looking at a file if it "disappears" and doesn't look for new files in a Dir once it starts. And after looking at the alternatives xtail and multitail i decided I was better off coming up with my own way to dealing with these issues.

New version no grep inline with tail (little bit better)

Version 2

#!/bin/bash

logs_name="*-sshd-*"

logs_path=/var/log

notify_hashs=/tmp/.notify

if [[ ! -d "${notify_hashs}" ]]; then

mkdir -p "${notify_hashs}"

if [[ ! -d "${notify_hashs}" ]]; then

echo "Could NOT create hash storage : ${notify_hashs}"

exit

fi

(

while [ 1 ]; do

## We store the hashs in a flat file. The Hashs are just flags for lines that

## we have already send out notfies about.

hashs="${notify_hashs}/$(date +%Y%m%d).hash"

## Why check if the file exists since touching is harmless... right?

touch "${hashs}"

(

## This sub process watches for newer/deleted versions of the listed files.

## And if any are found it will exit thus forcing the tail process to also

## exit and restart.

found_first=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")

repeat=1

## Loop forever unless we are told otherwise.

## During each loop we look for any changes in the files we are looking for.

while [ ${repeat} -eq 1 ]; do

sleep 0.5

## Find will return all the found files in one (possibly) LONG string.

## Hashing would shorten the string and reduce memory memory usage abit but

## it would also require launching other process which would use more CPU

## and temporary memory until the hash is made... EVERY loop!

found_current=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")

if [[ $found_current != $found_first ]]; then

repeat=0

fi

done

) &

## Here is where we tail the files in question, sending erros to /dev/null and only grabbing the lines we care about.

tail --pid="$!" -q -F ${logs_path}/${logs_name} 2>/dev/null | while read line; do

## Skip all lines that start with "==" OR blank

if [[ ! "$line" =~ ^\=\= ]] && [[ ! -z "$line" ]]; then

if [[ "$line" =~ :\ (Accepted|Failed) ]]; then

if [[ ! "$line" =~ ((www.bbhcsd.org|files).*10.80.16.10) ]]; then

hash=$(echo "${line}" | md5sum -)

if [ $(grep -c -m 1 "${hash}" "${hashs}") -eq 0 ]; then

echo $line

echo "${hash}" >> "${hashs}"

/scripts/notify_with_slack.bash -t "${line}"

fi

done

) &

Old Version with grep & tail

logwatcher.sshd.bash

#!/bin/bash

logs_name="*-sshd-*"

logs_path=/var/log

notify_hashs=/tmp/.notify

if [[ ! -d "${notify_hashs}" ]]; then

mkdir -p "${notify_hashs}"

if [[ ! -d "${notify_hashs}" ]]; then

echo "Could NOT create hash storage : ${notify_hashs}"

exit

fi

(

while [ 1 ]; do

hashs="${notify_hashs}/$(date +%Y%m%d).hash"

touch "${hashs}"

(

## This sub process watches for newer versions of the listed files.

## And if any are found it will exit thus forcing the tail process to also exit.

found_first=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")

repeat=1

while [ ${repeat} -eq 1 ]; do

sleep 0.5

found_current=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")

if [[ $found_current != $found_first ]]; then

repeat=0

fi

done

) &

tail --pid="$!" -q -F ${logs_path}/${logs_name} 2>/dev/null | grep --line-buffer -E ": (Accepted|Failed)" | grep --line-buffer -v -E "((www.nomadcf.com|files).*10.80.16.10)" | while read line; do

## Skip all lines that start with "==" OR blank

if [[ ! $line =~ ^\=\= ]] && [[ $line ]]; then

hash=$(echo "${line}" | md5sum -)

if [ $(grep -c -m 1 "${hash}" "${hashs}") -eq 0 ]; then

echo "${hash}" >> "${hashs}"

/scripts/notify_with_slack.bash -t "${line}"

fi

done

) &