I had a simple problem, I had multiple log files I needed to monitor and have scripts act on the results. But tail does stop looking at a file if it "disappears" and doesn't look for new files in a Dir once it starts. And after looking at the alternatives xtail and multitail i decided I was better off coming up with my own way to dealing with these issues. New version no grep inline with tail (little bit better) Version 2#!/bin/bash logs_name="*-sshd-*" logs_path=/var/log notify_hashs=/tmp/.notify if [[ ! -d "${notify_hashs}" ]]; then mkdir -p "${notify_hashs}" if [[ ! -d "${notify_hashs}" ]]; then echo "Could NOT create hash storage : ${notify_hashs}" exit fi fi ( while [ 1 ]; do ## We store the hashs in a flat file. The Hashs are just flags for lines that ## we have already send out notfies about. hashs="${notify_hashs}/$(date +%Y%m%d).hash" ## Why check if the file exists since touching is harmless... right? touch "${hashs}" ( ## This sub process watches for newer/deleted versions of the listed files. ## And if any are found it will exit thus forcing the tail process to also ## exit and restart. found_first=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}") repeat=1 ## Loop forever unless we are told otherwise. ## During each loop we look for any changes in the files we are looking for. while [ ${repeat} -eq 1 ]; do sleep 0.5 ## Find will return all the found files in one (possibly) LONG string. ## Hashing would shorten the string and reduce memory memory usage abit but ## it would also require launching other process which would use more CPU ## and temporary memory until the hash is made... EVERY loop! found_current=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}") if [[ $found_current != $found_first ]]; then repeat=0 fi done ) & ## Here is where we tail the files in question, sending erros to /dev/null and only grabbing the lines we care about. tail --pid="$!" -q -F ${logs_path}/${logs_name} 2>/dev/null | while read line; do ## Skip all lines that start with "==" OR blank if [[ ! "$line" =~ ^\=\= ]] && [[ ! -z "$line" ]]; then if [[ "$line" =~ :\ (Accepted|Failed) ]]; then if [[ ! "$line" =~ ((www.bbhcsd.org|files).*10.80.16.10) ]]; then hash=$(echo "${line}" | md5sum -) if [ $(grep -c -m 1 "${hash}" "${hashs}") -eq 0 ]; then echo $line echo "${hash}" >> "${hashs}" /scripts/notify_with_slack.bash -t "${line}" fi fi fi fi done done ) & Old Version with grep & tail logwatcher.sshd.bash#!/bin/bash logs_name="*-sshd-*" logs_path=/var/log notify_hashs=/tmp/.notify if [[ ! -d "${notify_hashs}" ]]; then mkdir -p "${notify_hashs}" if [[ ! -d "${notify_hashs}" ]]; then echo "Could NOT create hash storage : ${notify_hashs}" exit fi fi ( while [ 1 ]; do hashs="${notify_hashs}/$(date +%Y%m%d).hash" touch "${hashs}" ( ## This sub process watches for newer versions of the listed files. ## And if any are found it will exit thus forcing the tail process to also exit. found_first=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}") repeat=1 while [ ${repeat} -eq 1 ]; do sleep 0.5 found_current=$(find ${logs_path} -maxdepth 1 -mindepth 1 -print0 -iname "${logs_name}") if [[ $found_current != $found_first ]]; then repeat=0 fi done ) & tail --pid="$!" -q -F ${logs_path}/${logs_name} 2>/dev/null | grep --line-buffer -E ": (Accepted|Failed)" | grep --line-buffer -v -E "((www.nomadcf.com|files).*10.80.16.10)" | while read line; do ## Skip all lines that start with "==" OR blank if [[ ! $line =~ ^\=\= ]] && [[ $line ]]; then hash=$(echo "${line}" | md5sum -) if [ $(grep -c -m 1 "${hash}" "${hashs}") -eq 0 ]; then echo "${hash}" >> "${hashs}" /scripts/notify_with_slack.bash -t "${line}" fi fi done done ) & |
Scripting >
Bash - Auto Tailing new files in a directory
posted Dec 23, 2015, 9:07 AM by Chris Franklin [ updated Dec 23, 2015, 9:56 AM ]I had a simple problem, I had multiple log files I needed to monitor and have scripts act on the results. But tail does stop looking at a file if it "disappears" and doesn't look for new files in a Dir once it starts. And after looking at the alternatives xtail and multitail i decided I was better off coming up with my own way to dealing with these issues. New version no grep inline with tail (little bit better) Old Version with grep & tail |