Scripting‎ > ‎

Bash - Auto Tailing new files in a directory

posted Dec 23, 2015, 9:07 AM by Chris Franklin   [ updated Dec 23, 2015, 9:56 AM ]
I had a simple problem, I had multiple log files I needed to monitor and have scripts act on the results. But tail does stop looking at a file if it "disappears" and doesn't look for new files in a Dir once it starts. And after looking at the alternatives xtail and multitail i decided I was better off coming up with my own way to dealing with these issues. 

New version no grep inline with tail (little bit better)

Version 2

#!/bin/bash

logs_name="*-sshd-*"
logs_path=/var/log
notify_hashs=/tmp/.notify

if [[ ! -d "${notify_hashs}" ]]; then
 mkdir -p "${notify_hashs}"
 if [[ ! -d "${notify_hashs}" ]]; then
  echo "Could NOT create hash storage : ${notify_hashs}"
  exit
 fi
fi

(
while [ 1 ]; do

 ## We store the hashs in a flat file. The Hashs are just flags for lines that
 ## we have already send out notfies about.
 hashs="${notify_hashs}/$(date +%Y%m%d).hash"

 ## Why check if the file exists since touching is harmless... right?
 touch "${hashs}"

 (
  ## This sub process watches for newer/deleted versions of the listed files.
  ##  And if any are found it will exit thus forcing the tail process to also
  ##  exit and restart.
  found_first=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")
  repeat=1

  ## Loop forever unless we are told otherwise.
  ## During each loop we look for any changes in the files we are looking for.
  while [ ${repeat} -eq 1 ]; do
   sleep 0.5
   ## Find will return all the found files in one (possibly) LONG string.
   ## Hashing would shorten the string and reduce memory memory usage abit but
   ## it would also require launching other process which would use more CPU
   ## and temporary memory until the hash is made... EVERY loop!
   found_current=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")
   if [[ $found_current != $found_first ]]; then
    repeat=0
   fi
  done
 ) &

 ## Here is where we tail the files in question, sending erros to /dev/null and only grabbing the lines we care about.
 tail --pid="$!" -q -F ${logs_path}/${logs_name} 2>/dev/null | while read line; do
  ## Skip all lines that start with "==" OR blank
  if [[ ! "$line"  =~ ^\=\= ]] && [[ ! -z "$line" ]]; then
   if [[ "$line" =~ :\ (Accepted|Failed) ]]; then
    if [[ ! "$line" =~ ((www.bbhcsd.org|files).*10.80.16.10) ]]; then
     hash=$(echo "${line}" | md5sum -)
     if [ $(grep -c -m 1 "${hash}" "${hashs}") -eq 0 ]; then
      echo $line
      echo "${hash}" >> "${hashs}"
      /scripts/notify_with_slack.bash -t "${line}"
     fi
    fi
   fi
  fi
 done

done
) &





Old Version with grep & tail

logwatcher.sshd.bash

#!/bin/bash


logs_name="*-sshd-*"
logs_path=/var/log
notify_hashs=/tmp/.notify

if [[ ! -d "${notify_hashs}" ]]; then
 mkdir -p "${notify_hashs}"
 if [[ ! -d "${notify_hashs}" ]]; then
  echo "Could NOT create hash storage : ${notify_hashs}"
  exit
 fi
fi

(
while [ 1 ]; do

 hashs="${notify_hashs}/$(date +%Y%m%d).hash"
 touch "${hashs}"

 (
  ## This sub process watches for newer versions of the listed files.
  ##  And if any are found it will exit thus forcing the tail process to also exit.
  found_first=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")
  repeat=1

  while [ ${repeat} -eq 1 ]; do
   sleep 0.5
   found_current=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}")
   if [[ $found_current != $found_first ]]; then
    repeat=0
   fi
  done
 ) &

 tail --pid="$!" -q -F ${logs_path}/${logs_name} 2>/dev/null | grep --line-buffer -E ": (Accepted|Failed)" | grep --line-buffer -v -E "((www.nomadcf.com|files).*10.80.16.10)" | while read line; do
  ## Skip all lines that start with "==" OR blank
  if [[ ! $line  =~ ^\=\= ]] && [[ $line ]]; then
   hash=$(echo "${line}" | md5sum -)
   if [ $(grep -c -m 1 "${hash}" "${hashs}") -eq 0 ]; then
    echo "${hash}" >> "${hashs}"
    /scripts/notify_with_slack.bash -t "${line}"
   fi
  fi
 done

done
) &


Comments