VMWare copy and keep thin provisioning scripts & register the VM on a backup VM server

Post date: Mar 31, 2017 1:54:59 PM

I use ghettovcb for backing up my VM Hosts to my file servers via NFS. I do this as I can't quickly and easily copy them directly to my backup VM host(s). As VMWare refuses to acknowledge that it's users want and haven been asking for remote access to datastores since the beginning of time. So here is my stop gap solution that only uses the tools built into VMWare (like ghettovcb) to copy from a NFS mount to the backup VM host, Register the latest good backup & unregister the old copy of that same server

            VM Host ( ghettovcb via cron )=> NFS Share => Backup VM Host (My Scripts Via Cron)

This script does the fallowing.

• 1. Look for all STATUS.ok files found in the directory structure that is using the default ghettovcb setup (aka VM_Hostname/VM_Hostname-DATE ).

1. Copy over all files in that directory to a corresponding location on the "Backup VM Host"

2. Edit the Display names of all backups to read " Date - VM Hostname "  

   1. Copy the vmdk using vmkfstools to keep or covert the backup to a thin provisioned version

   2. verify and log everything along the way

   3. only keep X copies on the backup VM HOST

   4. Auto removes Moved copied from NFS host

   5. Only copys the STATUS.ok file the backup VM Host after everything else is done and verified aka no reported errors)

copy_vm_to_backup.sh

## config vars

max_copies=10

vmstore_backup="backups"

vmstore_restore="datastore1"

log="/vmfs/volumes/${vmstore_backup}/scripts/logs/esxi-backups.$(date +%s).log"

debug=1

vmkfstools="$(which vmkfstools)"

## functions

logit() {

 if [ $debug -eq 1 ]; then

  echo "$1"

 fi

 echo "$1" >> "${log}"

}

logit "========= Starting Copy ========="

if [ ! -d "${log}" ]; then

 mdkir -p "${log}"

 if [ $? -eq 0 ]; then

  debug=1

  logit "ERROR : mdkir -p '${log}'"

 else

  logit "Created Logs dir : '${log}'"

 fi

fi

## the work

find /vmfs/volumes/${vmstore_backup}/ -type d -maxdepth 1 -mindepth 1 | while read vmdirs; do

 # Reset skip this vm

 vmskip=0

 if [ -d "${vmdirs}" ]; then

  vmname=$(basename "${vmdirs}")

  logit "Found dir : ${vmdirs}"

  logit "VM name : ${vmname}"

  if [ "${vmdirs}" != "scripts" ]; then

   find "${vmdirs}" -type f -iname "STATUS.ok" | while read vmbackup; do

    logit "Found Completed backup file : ${vmbackup}"

    vmbackupdir=$(dirname "${vmbackup}")

    vmrestoredir=$(dirname "${vmbackup/${vmstore_backup}/${vmstore_restore}}")

    logit "vmbackupdir  : ${vmbackupdir}"

    logit "vmrestoredir : ${vmrestoredir}"

    if [ -e "${vmrestoredir}/BACKUP.err" ]; then

     logit "Removing old ERROR file"

     rm -f "${vmrestoredir}/BACKUP.err"

    fi

    logit "Looking into : ${vmbackupdir}"

    find "${vmbackupdir}" -type f | while read vmbackupfiles; do

     vmbackupfilesdir=$(dirname "${vmbackupfiles}")

     vmbackupfilesname=$(basename "${vmbackupfiles}")

     vmbackupfilesextension="${vmbackupfilesname##*.}"

     vmrestorefiles="${vmbackupfiles/$vmstore_backup/$vmstore_restore}"

     logit "Found file : ${vmbackupfiles}"

     if [ ! -d "${vmrestoredir}" ]; then

      logit "Creating dir : ${vmrestoredir}"

      mkdir -p "${vmrestoredir}"

      if [ $? -ne 0 ]; then

       vmskip=1

       logit "ERROR : mkdir -p '${vmrestoredir}'"

      fi

     fi

     if [ $vmskip -eq 0 ]; then

      case "${vmbackupfilesname}" in

       "STATUS.ok") ## Ignore the STATUS.ok file 

        ignore=1

        ;;

       *"-flat.vmdk") ## Ignore any -flat vhd files

        logit "checking if flat file is an orphan. : '${vmrestorefiles}' or '${vmrestorefiles/-flat.vmdk/.vmdk}.converted'"

        if [ -e "${vmrestorefiles}" ]; then

         logit "Found restored flat file"

         if [ ! -e "${vmrestorefiles/-flat.vmdk/.vmdk}.converted" ]; then

          logit "Found no converted flag though : '${vmrestorefiles/-flat.vmdk/.vmdk}.converted'"

          if [ -e "${vmrestorefiles/-flat.vmdk/.vmdk}.convert.log" ]; then

           if [ $(grep -c '100% done.' "${vmrestorefiles/-flat.vmdk/.vmdk}.convert.log") -eq 1 ]; then

            logit "Found convert.log & it reads the conver was 100%. Creating converted flag file : '${vmrestorefiles/-flat.vmdk/.vmdk}.converted'"

            touch "${vmrestorefiles/-flat.vmdk/.vmdk}.converted"

           else

            logit "ERROR : No converted flag found, convert file does NOT read 100% done. Leaving file"

           fi

          else

           logit "Found no converted log : '${vmrestorefiles/-flat.vmdk/.vmdk}.convert.log'"

          fi

         fi

         if [ -e "${vmrestorefiles/-flat.vmdk/.vmdk}.converted" ]; then

          logit "Found left over flat file"

          if [ ! -e "${vmbackupfiles/-flat.vmdk/.vmdk}" ]; then

           logit "Removing left over flat (orphan)"

           rm -f "${vmbackupfiles}"

           if [ $? -ne 0 ]; then

            logit "ERROR : Removing left over flat (orphan) : rm -f '${vmbackupfiles}'"

           fi

          fi

         fi

        fi

        ;;

       *".vmdk") ## Convert to thin (or at least keep thin provisioning)

        vmrestorefilesflat="${vmrestorefiles%.*}-flat.vmdk"

        if [ ! -e "${vmrestorefiles}.converted" ]; then

         logit "Looking for any preveous convert attemps : '${vmrestorefiles}' or '${vmrestorefilesflat}'"         

         if [ -e "${vmrestorefiles}" ] || [ -e "${vmrestorefilesflat}" ]; then

          if [ -e "${vmrestorefiles}" ]; then

           logit "Removing preveous convert attemp : '${vmrestorefiles}'"

           rm -f "${vmrestorefiles}"

           if [ $? -ne 0 ]; then

            logit "ERROR : removing preveous convert attemp : rm -f '${vmrestorefiles}'"

           fi

          fi

          if [ -e "${vmrestorefilesflat}" ]; then

           logit "Removing preveous convert attemp (flat) : '${vmrestorefilesflat}'"

           rm -f "${vmrestorefilesflat}"

           if [ $? -ne 0 ]; then

            logit "ERROR : removing preveous convert attemp : rm -f '${vmrestorefilesflat}'"

           fi

          fi

         fi

         logit "Convert : Start : $(date)"

         logit "Converting : '${vmbackupfiles}' => '${vmrestorefiles}'"

         $vmkfstools -d thin -i "${vmbackupfiles}" "${vmrestorefiles}" > "${vmrestorefiles}.convert.log"

         if [ $? -eq 0 ] && [ $(grep -c '100% done.' "${vmrestorefiles}.convert.log") -eq 1 ]; then

          logit "Convert : End : $(date)"

          touch "${vmrestorefiles}.converted"

          logit "Created : ${vmrestorefiles}.converted"

          logit "Looking for vmdk file : '${vmbackupfiles}'"

          if [ -e "${vmbackupfiles}" ]; then

           echo "Removing Converted VHD : ${vmbackupfiles}"

           rm -f "${vmbackupfiles}"

          fi

          logit "Looking for flat file : '${vmbackupfilesdir}/${vmbackupfilesname%.*}-flat.vmdk'"

          if [ -e "${vmbackupfilesdir}/${vmbackupfilesname%.*}-flat.vmdk" ]; then

           echo "Removing Converted Flat VHD : ${vmbackupfiles%.*}-flat.vmdk"

           rm -f "${vmbackupfilesdir}/${vmbackupfilesname%.*}-flat.vmdk"

          fi

         else 

          logit "Convert : End : $(date)"

          logit "ERROR : $vmkfstools -d thin -i '${vmbackupfiles}' '${vmrestorefiles}'"

          vmskip=1

         fi

        else

         logit "Already Convereted found : '${vmrestorefiles}.converted'"

         if [ $(grep -c '100% done.' "${vmrestorefiles}.convert.log") -eq 1 ]; then 

          logit "Converted log reports 100% complete : grep -c '100% done.' '${vmrestorefiles}.convert.log'"

          logit "Looking for left over vmdk file : '${vmbackupfiles}'"

          if [ -e "${vmbackupfiles}" ]; then

           echo "Removing Converted VHD : ${vmbackupfiles}"

           rm -f "${vmbackupfiles}"

          fi

          logit "Looking for left over flat file : '${vmbackupfilesdir}/${vmbackupfilesname%.*}-flat.vmdk'"

          if [ -e "${vmbackupfilesdir}/${vmbackupfilesname%.*}-flat.vmdk" ]; then

           echo "Removing Converted Flat VHD : ${vmbackupfiles}"

           rm -f "${vmbackupfilesdir}/${vmbackupfilesname%.*}-flat.vmdk"

          fi

         fi

        fi

        ;;

       *".vmx") ## Copy and modify base config

        vmbackupdate="${vmrestoredir%%_*}"

        vmbackupdate="${vmbackupdate##*/}"

        vmbackupdate="${vmbackupdate/${vmname}-/}"

        cp -f "${vmbackupfiles}" "${vmrestorefiles}.org"

        if [ $? -eq 0 ]; then

         logit "Copied Config : ${vmrestorefiles}.org"

         cat "${vmrestorefiles}.org" | sed -e 's/^displayName\ \=\ \".*"$/displayName\ \=\ \"'"${vmbackupdate} ${vmname}"'"/' > "${vmrestorefiles}"

         if [ $? -eq 0 ]; then

          logit "Mod'ed Config : ${vmrestorefiles}"

          rm -f "${vmbackupfiles}"

         else

          logit "ERROR : Mod'ing Config : ${vmrestorefiles}"

          vmskip=1

         fi

        else

         logit "ERROR : Copying Config : ${vmbackupfiles} => ${vmrestorefiles}.org"

         vmskip=1

        fi

        ;;

       *) ## Default copy any non known file

        if [ ! -d "${vmrestoredir}" ]; then

         mkdir -p "${vmrestoredir}"

         if [ $? -ne 0 ]; then

          logit "ERROR :  mkdir -p '${vmrestoredir}'"

          vmskip=1

         fi

        fi

        if [ -d "${vmrestoredir}" ]; then

         cp -f "${vmbackupfiles}" "${vmrestorefiles}"

         if [ $? -eq 0 ]; then

          logit "Copied MISC file : ${vmbackupfiles}"

          rm -f "${vmbackupfiles}"

         else 

          logit "ERROR : Coping MISC file : cp -f '${vmbackupfiles}' '${vmrestorefiles}'"

          vmskip=1

         fi

        else 

         logit "ERROR : Coping MISC file : no dir '${vmrestorefiles}'"

         vmskip=1

        fi

        ;;

      esac

     fi

     if [ $vmskip -eq 1 ]; then

      touch ${vmrestoredir}/BACKUP.err

     fi

    done

    if [ $(find "${vmbackupdir}" -type f | wc -l) -eq 1 ] && [ ! -e "${vmrestoredir}/BACKUP.err" ]; then     

     logit "Found only remaning file : $(find '${vmbackupdir}' -type f)"

     touch "${vmrestoredir}/STATUS.ok"

     if [ $? -eq 0 ]; then

      logit "Created STATUS.ok file : ${vmrestoredir}/STATUS.ok"

      rm -f "${vmbackupdir}/STATUS.ok"

      if [ $? -ne 0 ]; then

       logit "ERROR : unable to delete backup source STATUS.ok : rm -f '${vmbackupdir}/STATUS.ok'"

      fi

     else

      logit "ERROR : Creating STATUS.ok : touch '${vmrestoredir}/STATUS.ok'"

     fi 

     if [ $? -eq 0 ]; then

      rm -f "${vmbackupdir}/STATUS.ok"

     fi

     vmbackupdir="${vmbackupdir//../}"

     case "${vm2delete}" in

      *[!/]*/)

       vmbackupdir=${vmbackupdir%"${vmbackupdir##*[!/]}"}

      ;;

     esac

     if [ -d "vmbackupdir" ]; then

      if [ "${vmbackupdir}" != "/vmfs/volumes/${vmstore_backup}" ] && [ "${vmbackupdir}" != "/vmfs/volumes" ] && [ "${vmbackupdir}" != "/vmfs" ]; then

       rm -rf "${vmbackupdir}"

       if [ $? -eq 0 ]; then

        logit "Removed : '${vmbackupdir}'"

       else

        logit "ERROR : rm -rf '${vmbackupdir}'"

       fi 

      fi 

     fi

    else 

     logit "ERROR : Found remaning files : $(find ${vmbackupdir} -type f)"

    fi

   done

  fi

 else

  logit "Found misc file : ${vmdirs}"

 fi

done

logit "======= Start Old Removal ======="

find /vmfs/volumes/${vmstore_restore}/ -type d -maxdepth 1 -mindepth 1 -not -iname ".*" | while read vmbase; do

 foundvms=$(find "${vmbase}" -type f \( -name "STATUS.ok" -o -name "BACKUP.err" \) | wc -l)

 logit "Old Copies Count : ${foundvms} for ${vmbase}" 

 if [ $foundvms -gt $max_copies ]; then

  logit "Over Max Copies : ${vmbase} by $(($foundvms - $max_copies))"

  ls -1cr "${vmbase}" | head -n $(($foundvms - $max_copies)) | while read vmdelete; do

   vm2delete="${vmbase}/${vmdelete}/"

   vm2delete="${vm2delete//../}"

   case "${vm2delete}" in 

    *[!/]*/) 

     vm2delete=${vm2delete%"${vm2delete##*[!/]}"}

     ;; 

   esac

   logit "Removing Old Copy : ${vmbase}/${vmdelete}"

   if [ "${vm2delete}" != "/vmfs/volumes/${vmstore_restore}" ] && [ "${vm2delete}" != "/vmfs/volumes" ] && [ "${vm2delete}" != "/vmfs" ]; then

    echo rm -rf "${vm2delete}"

    if [ $? -eq 0 ]; then

     logit "Removed : ${vm2delete}"

    else

     logit "ERROR : Removing Old Copy : '${vmbase}/${vmdelete}'"

    fi

   fi

  done

 fi

done

logit "======= Completely Done ========="

This script does all the registering and un-registering of the backups.

register_guests.sh

vmstore_backup="datastore1"

find "/vmfs/volumes/${vmstore_backup}/" -type d -maxdepth 1 -mindepth 1 -not -iname "scripts" | while read vmname; do

 ls -1r "${vmname}"/*/STATUS.ok | head -n 1 | while read vmstatusfile; do

  ## Setup some vars

  vmpath=$(dirname "${vmstatusfile}") 

  vmversion=${vmpath##*/} 

  vmname=${vmpath%/*}

  vmname=${vmname##*/} 

  vmdate=${vmpath##*/}

  vmdate=${vmdate/${vmname}-/}

  vmdate=${vmdate%_*} #2017-03-27

  vmdisplayname="${vmname}_${vmdate}"

  # See if this VM is already reg.

  if [ $(vim-cmd vmsvc/getallvms | grep -c -i "${vmname}") -eq 0 ]; then

   # if not reg it

   echo "Adding : ${vmversion}"

   find "${vmpath}" -iname "*.vmx" -exec vim-cmd solo/registervm '{}' \;

  else

   # test is this version is already reged, if not unreg the existing and reg this opne

   if [ $(vim-cmd vmsvc/getallvms | grep -c "${vmversion}") -eq 0 ]; then

    # get vmid of exiting vm guest

    vmid=$(vim-cmd vmsvc/getallvms | grep -i "${vmname}" | awk '{print $1}')

    # chec kthat it is not on. if it is, we must be using it and SHOULD NOT remove it

    if [ $(vim-cmd vmsvc/power.getstate ${vmid} | grep -c "Powered off") ]; then

     vim-cmd vmsvc/unregiste ${vmid}

     # check that it was removed

     if [ $(vim-cmd vmsvc/getallvms | grep -c -i "${vmname}") -eq 0 ]; then

      # add the new one

      echo "Adding : ${vmversion}"

      find "${vmpath}" -iname "*.vmx" -exec vim-cmd solo/registervm '{}' \;

     fi

    fi

   else

    echo "Skipping : ${vmversion}"

   fi

  fi

 done

done