Bash - Auto Tailing new files in a directory

Post date: Dec 23, 2015 5:7:2 PM

I had a simple problem, I had multiple log files I needed to monitor and have scripts act on the results. But tail does stop looking at a file if it "disappears" and doesn't look for new files in a Dir once it starts. And after looking at the alternatives xtail and multitail i decided I was better off coming up with my own way to dealing with these issues. 

New version no grep inline with tail (little bit better)

Version 2

#!/bin/bash

logs_name="*-sshd-*"

logs_path=/var/log

notify_hashs=/tmp/.notify

if [[ ! -d "${notify_hashs}" ]]; then

 mkdir -p "${notify_hashs}"

 if [[ ! -d "${notify_hashs}" ]]; then

  echo "Could NOT create hash storage : ${notify_hashs}"

  exit

 fi

fi

(

while [ 1 ]; do

 ## We store the hashs in a flat file. The Hashs are just flags for lines that

 ## we have already send out notfies about.

 hashs="${notify_hashs}/$(date +%Y%m%d).hash"

 ## Why check if the file exists since touching is harmless... right?

 touch "${hashs}"

 (

  ## This sub process watches for newer/deleted versions of the listed files.

  ##  And if any are found it will exit thus forcing the tail process to also

  ##  exit and restart.

  found_first=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")

  repeat=1

  ## Loop forever unless we are told otherwise.

  ## During each loop we look for any changes in the files we are looking for.

  while [ ${repeat} -eq 1 ]; do

   sleep 0.5

   ## Find will return all the found files in one (possibly) LONG string.

   ## Hashing would shorten the string and reduce memory memory usage abit but

   ## it would also require launching other process which would use more CPU

   ## and temporary memory until the hash is made... EVERY loop!

   found_current=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")

   if [[ $found_current != $found_first ]]; then

    repeat=0

   fi

  done

 ) &

 ## Here is where we tail the files in question, sending erros to /dev/null and only grabbing the lines we care about.

 tail --pid="$!" -q -F ${logs_path}/${logs_name} 2>/dev/null | while read line; do

  ## Skip all lines that start with "==" OR blank

  if [[ ! "$line"  =~ ^\=\= ]] && [[ ! -z "$line" ]]; then

   if [[ "$line" =~ :\ (Accepted|Failed) ]]; then

    if [[ ! "$line" =~ ((www.bbhcsd.org|files).*10.80.16.10) ]]; then

     hash=$(echo "${line}" | md5sum -)

     if [ $(grep -c -m 1 "${hash}" "${hashs}") -eq 0 ]; then

      echo $line

      echo "${hash}" >> "${hashs}"

      /scripts/notify_with_slack.bash -t "${line}"

     fi

    fi

   fi

  fi

 done

done

) &

Old Version with grep & tail

logwatcher.sshd.bash

#!/bin/bash

logs_name="*-sshd-*"

logs_path=/var/log

notify_hashs=/tmp/.notify

if [[ ! -d "${notify_hashs}" ]]; then

 mkdir -p "${notify_hashs}"

 if [[ ! -d "${notify_hashs}" ]]; then

  echo "Could NOT create hash storage : ${notify_hashs}"

  exit

 fi

fi

(

while [ 1 ]; do

 hashs="${notify_hashs}/$(date +%Y%m%d).hash"

 touch "${hashs}"

 (

  ## This sub process watches for newer versions of the listed files.

  ##  And if any are found it will exit thus forcing the tail process to also exit.

  found_first=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")

  repeat=1

  while [ ${repeat} -eq 1 ]; do

   sleep 0.5

   found_current=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")

   if [[ $found_current != $found_first ]]; then

    repeat=0

   fi

  done

 ) &

 tail --pid="$!" -q -F ${logs_path}/${logs_name} 2>/dev/null | grep --line-buffer -E ": (Accepted|Failed)" | grep --line-buffer -v -E "((www.nomadcf.com|files).*10.80.16.10)" | while read line; do

  ## Skip all lines that start with "==" OR blank

  if [[ ! $line  =~ ^\=\= ]] && [[ $line ]]; then

   hash=$(echo "${line}" | md5sum -)

   if [ $(grep -c -m 1 "${hash}" "${hashs}") -eq 0 ]; then

    echo "${hash}" >> "${hashs}"

    /scripts/notify_with_slack.bash -t "${line}"

   fi

  fi

 done

done

) &