Bash Zimbra - Export All users Complete Mailboxs (Calendar,Emails,Folders, Etc) individually
Post date: Jan 16, 2018 3:13:24 PM
I we wanted to start backing up our users complete mailboxes individually. But we had two issues. First was that zmmailbox is a single threaded JAVA "program" and secondly we need this job done during our "off hours" (who really has off hours any more). In order to over come this I needed to have multiple but limited copies of zmmailbox running at the same time. Quickly we found that just feeding a list of users to zmmailbox resulted in a backup taking 13+ hrs.... 13 hrs to backup 983 users WTF. At any rate this script reduces that 13+ hrs down to just 3.10 hrs (still not the hour to hour and a half I was hopping for). While using only the built into tools on a standard box & not over whelming the box... provided you don't over provision the number of jobs that can one at one time.
This was written on a Red Hat Enterprise Linux Server release 6 box.
I assume /dev/shm exists and is a ram drive
ZM Config settings are acquired from the zimbra config files. So there is no need to hard code them into the code.
I do try to "multi-thread" to speed things up up. But I wouldn't start more than 1 job per 2 cores, of course don't count a hyper-threaded core.
I exclude all archive mailboxes
I backup all domains & all user accounts
The "ok" log contains how to restore the backup copy.
It can be run via the console OR a cron. If you run it via a cron you'll want to redirect the output to a log file of your choosing. ( zimbra_backup_mailboxes.bash > /var/log/zimbra_backup_mailboxes.log)
Our system only has 983 users with an average of only a few gigs each. With 12 real cores (24 with HT) I run 8 jobs at time. This ends up taking 3.10 hrs to run and reducing our over all backup size down to just 220GB
zimbra_backup_mailboxes.bash
#!/bin/bash
MAXJOBS=8 # 8&10=same, 6=slower
LOG=/var/log/zimbra.user-backup.log
BACKUPS=/var/opt/zimbra/backup/mailboxes
ZMBACKUPCMD=/var/opt/zimbra/bin/zmmailbox
ZMCONFIG=$(whereis zmlocalconfig | cut -d ' ' -f2)
LDAPSEARCH=$(whereis ldapsearch | cut -d ' ' -f2)
LDAP_SEARCH="${LDAPSEARCH} -x -h $($ZMCONFIG ldap_host | cut -d '=' -f2) -D $($ZMCONFIG zimbra_ldap_userdn | awk '{print $3}') -w$($ZMCONFIG -s zimbra_ldap_password | cut -d ' ' -f3) -LLL -o ldif-wrap=no "
## Look for left over runtime files
if [ ! -z "$(find /dev/shm -type f -name "zimbra.backup.id.*")" ]; then
echo "Found existing RUNTIME files, ending."
echo "Found existing RUNTIME files, ending." >> "${LOG}"
exit
fi
## Functions
function _thejob() {
THISID=$1
EMAIL=$2
USERN=${EMAIL%%@*}
## Create runtime file
touch /dev/shm/zimbra.backup.id.$THISID
## Remvoe old status log files
find "${BACKUPS}/" -name "${USERN}.ok" -o -name "${USERN}.bad" -delete
$ZMBACKUPCMD -z -m "${EMAIL}" -t 0 getRestURL '/?fmt=tgz' > "${BACKUPS}/${USERN}.tgz"
if [ $? -eq 0 ]; then
cat <<EOF > "${BACKUPS}/${USERN}.ok"
## Simple restore: (Skips dups, does not recreate deleted calendar events)
${ZMBACKUPCMD} -z -m '${EMAIL}' -t 0 postRestURL '/?fmt=tgz&resolve=skip' '${BACKUPS}/${USERN}.tgz'
## Full Restore:
${ZMBACKUPCMD} -z -m '${EMAIL}' -t 0 postRestURL '/?fmt=tgz&resolve=reset' '${BACKUPS}/${USERN}.tgz'
EOF
else
echo > "${BACKUPS}/${USERN}.bad"
fi
## Backup LDAP info (although we can't restore it ... yet)
$LDAP_SEARCH "(&(objectClass=zimbraAccount)(mail=${EMAIL}))" > "${BACKUPS}/${USERN}.ldif"
## Remove runtime file
rm /dev/shm/zimbra.backup.id.$THISID
}
date
date > "${LOG}"
(
## List all users
/var/opt/zimbra/bin/zmprov -l gaa | grep cfranklin | while read EMAIL; do
## Ignore Archive accounts
if [[ ${EMAIL} != *".archive" ]]; then
echo "${EMAIL}"
fi
done
) | while read EMAIL; do
echo "Working on : $EMAIL" >> "${LOG}"
RUNNING=0
while [ $RUNNING -eq 0 ]; do
CURJOBS=0
while [ $CURJOBS -lt $MAXJOBS ]; do
if [ ! -e /dev/shm/zimbra.backup.id.$CURJOBS ]; then
echo "Started : $EMAIL : $CURJOBS" >> "${LOG}"
( _thejob $CURJOBS "${EMAIL}" ) &
sleep 1
if [ -e /dev/shm/zimbra.backup.id.$CURJOBS ]; then
RUNNING=1
CURJOBS=$MAXJOBS
fi
else
((CURJOBS++))
fi
done
if [ $RUNNING -eq 0 ]; then
if [ $CURJOBS -eq $MAXJOBS ]; then
echo -n "Max Jobs running" >> "${LOG}"
elif [ $CURJOBS -gt $MAXJOBS ]; then
echo -n "."
fi
fi
sleep 1
done
echo
done
## Waiting for jobs to finish
RUNNING=0
while [ ! -z "$(find /dev/shm -type f -name "zimbra.backup.id.*")" ]; do
if [ $RUNNING -eq 0 ]; then
RUNNING=1
echo -n "Waiting for jobs to finish."
else
echo -n "."
fi
sleep 1
done
echo
date
date >> "${LOG}"