Scripting‎ > ‎

Bash Zimbra - Export All users Complete Mailboxs (Calendar,Emails,Folders, Etc) individually

posted Jan 16, 2018, 7:13 AM by Chris Franklin   [ updated Jan 16, 2018, 7:25 AM ]
I we wanted to start backing up our users complete mailboxes individually. But we had two issues. First was that zmmailbox is a single threaded JAVA "program" and secondly we need this job done during our "off hours" (who really has off hours any more). In order to over come this I needed to have multiple but limited copies of zmmailbox running at the same time. Quickly we found that just feeding a list of users to zmmailbox resulted in a backup taking 13+ hrs.... 13 hrs to backup 983 users WTF. At any rate this script reduces that 13+ hrs down to just 3.10 hrs (still not the hour to hour and a half I was hopping for). While using only the built into tools on a standard box & not over whelming the box... provided you don't over provision the number of jobs that can one at one time.

This was  written on a Red Hat Enterprise Linux Server release 6 box. 
  • I assume /dev/shm exists and is a ram drive
  • ZM Config settings are acquired from the zimbra config files. So there is no need to hard code them into the code.
  • I do try to "multi-thread" to speed things up up. But I wouldn't start more than 1 job per 2 cores, of course don't count a hyper-threaded core.
  • I exclude all archive mailboxes
  • I backup all domains & all user accounts 
  • The "ok" log contains how to restore the backup copy.
  • It can be run via the console OR a cron. If you run it via a cron you'll want to redirect the output to a log file of your choosing. ( zimbra_backup_mailboxes.bash > /var/log/zimbra_backup_mailboxes.log)
  • Our system only has 983 users with an average of  only a few gigs each. With 12 real cores (24 with HT) I run 8 jobs at time. This ends up taking 3.10 hrs to run and reducing our over all backup size down to just 220GB


zimbra_backup_mailboxes.bash

#!/bin/bash

MAXJOBS=8 # 8&10=same, 6=slower

LOG=/var/log/zimbra.user-backup.log
BACKUPS=/var/opt/zimbra/backup/mailboxes
ZMBACKUPCMD=/var/opt/zimbra/bin/zmmailbox
ZMCONFIG=$(whereis zmlocalconfig | cut -d ' ' -f2)
LDAPSEARCH=$(whereis ldapsearch | cut -d ' ' -f2)
LDAP_SEARCH="${LDAPSEARCH} -x -h $($ZMCONFIG ldap_host | cut -d '=' -f2) -D $($ZMCONFIG zimbra_ldap_userdn | awk '{print $3}') -w$($ZMCONFIG -s zimbra_ldap_password | cut -d ' ' -f3) -LLL -o ldif-wrap=no "

## Look for left over runtime files
if [ ! -z "$(find /dev/shm -type f -name "zimbra.backup.id.*")" ]; then
 echo "Found existing RUNTIME files, ending."
 echo "Found existing RUNTIME files, ending." >> "${LOG}"
 exit
fi

## Functions
function _thejob() {
 THISID=$1
 EMAIL=$2
 USERN=${EMAIL%%@*}

 ## Create runtime file
 touch /dev/shm/zimbra.backup.id.$THISID

 ## Remvoe old status log files
 find "${BACKUPS}/" -name "${USERN}.ok" -o -name "${USERN}.bad" -delete

 $ZMBACKUPCMD -z -m "${EMAIL}" -t 0 getRestURL '/?fmt=tgz' > "${BACKUPS}/${USERN}.tgz"
 if [ $? -eq 0 ]; then
  cat <<EOF > "${BACKUPS}/${USERN}.ok"
## Simple restore: (Skips dups, does not recreate deleted calendar events)
${ZMBACKUPCMD} -z -m '${EMAIL}' -t 0 postRestURL '/?fmt=tgz&resolve=skip' '${BACKUPS}/${USERN}.tgz'
## Full Restore:
${ZMBACKUPCMD} -z -m '${EMAIL}' -t 0 postRestURL '/?fmt=tgz&resolve=reset' '${BACKUPS}/${USERN}.tgz'
EOF
 else
  echo > "${BACKUPS}/${USERN}.bad"
 fi

 ## Backup LDAP info (although we can't restore it ... yet)
 $LDAP_SEARCH "(&(objectClass=zimbraAccount)(mail=${EMAIL}))" > "${BACKUPS}/${USERN}.ldif"

 ## Remove runtime file
 rm /dev/shm/zimbra.backup.id.$THISID
}

date

date > "${LOG}"

(
## List all users
/var/opt/zimbra/bin/zmprov -l gaa | grep cfranklin  | while read EMAIL; do
 ## Ignore Archive accounts
 if [[ ${EMAIL} != *".archive" ]]; then
  echo "${EMAIL}"
 fi
done
) | while read EMAIL; do
 echo "Working on : $EMAIL" >> "${LOG}"
 RUNNING=0
 while [ $RUNNING -eq 0 ]; do
  CURJOBS=0
  while [ $CURJOBS -lt $MAXJOBS ]; do
   if [ ! -e /dev/shm/zimbra.backup.id.$CURJOBS ]; then
    echo "Started : $EMAIL : $CURJOBS"  >> "${LOG}"
    ( _thejob $CURJOBS "${EMAIL}" ) &
    sleep 1
    if [ -e /dev/shm/zimbra.backup.id.$CURJOBS ]; then
     RUNNING=1
     CURJOBS=$MAXJOBS
    fi
   else
    ((CURJOBS++))
   fi
  done
  if [ $RUNNING -eq 0 ]; then
   if [ $CURJOBS -eq $MAXJOBS ]; then
    echo -n "Max Jobs running" >> "${LOG}"
   elif [ $CURJOBS -gt $MAXJOBS ]; then
    echo -n "."
   fi
  fi
  sleep 1
 done
 echo
done

## Waiting for jobs to finish
RUNNING=0
while [ ! -z "$(find /dev/shm  -type f -name "zimbra.backup.id.*")" ]; do
 if [ $RUNNING -eq 0 ]; then
  RUNNING=1
  echo -n "Waiting for jobs to finish."
 else
  echo -n "."
 fi
 sleep 1
done
echo
date
date >> "${LOG}"


Comments