Migrating opensource Zimbra 8.6.0 on Centos 6.8 to Zimbra 8.7.1 on Centos 7 safely and with no downtime !

Zimbra to Zimbra (zcs to zcs) migration

A bit of history for the context

After running a Zimbra mail server in a 500Gb Virtual Machine, for about 4 years, the server started feeling a bit crowded and with  #df -h reporting less than 50Gb of space left, it was time to move to a larger machine.

The version I was (and still am) running is the open source version, there are no migration tools available as part of the package, although you can find plenty of tutorials on the web forums about how to rsync stuff between the old and the new server. I was not comfortable with that. For starters, there would be down time involved, but apart from that I would have to rsync between two identical servers, meaning that the new server would still need to be zimbra 8.6 and running on an identical Centos 6 machine.

There was also the fact that some time ago, the server did a very bad shutdown (due to a power failure), and the database had been corrupted, and every so often a problem or two would crop up in the logs. This became very evident when an upgrade to 8.7.1 failed miserably, and the only thing that saved the day was the backup from the previous night ! I was afraid that most likely, an rsync migration would also transfer the problematic data, and that it would be back to haunt me down the road.

Since I was going to have to go through all the aches and  pains of  a server migration  I wanted to end up not only with more space,  but also to move to a newer O.S., and a newer release, so rsync was out of the question.

zmmailbox and cli to the rescue

If you have spent any time at all administering a Zimbra mail server, you are definitely familiar with the zmprov command. You might have used it to reset a forgotten admin password, or to list all the accounts using a forwarding address or to really do any other thing you could do from within the web interface ! You might be a bit less familiar with zmmailbox.

Like zmprov, zmmailbox is also a very powerful command, and can be used to do something as thrivial as creating a mail folder, but it can also export whole mailboxes in “tgz” format, including any folder structure the user might have …. AWESOME !! The only downside is that each mailbox needs to be exported separately, and you cannot use “*” or any wildcards. We are running on Linux however, so we can write  bash scripts to do all the heavy lifting for us :).

Preparing the new server

You can follow my guide about how to install Zimbra 8.7.1 on Centos 7 here. If you already know how to do it, just go ahead and install the new server. Basically you will need to install a server with the same settings you used for your old server, but with a different I.P. Address. Remember to setup named, and the zone file accordingly, along with the /etc/hosts file. You do not need to create all the domains inside the server. Just create your main one, we will import the others automatically. Remember that if you intend to install a “letsencrypt” certificate later on, your server name needs to be the same as your http://webmail name. Normally I like to use webmail.domain.com

Switching over to the new server

In order to have virtually zero down time, we will proceed as follows :-

1. Set the DNS TTL Entries pertinent to the mail server to the shortest possible time (Ideally this is done a day before to make sure the ttl propagates accordinly)
2. Prepare a fully working new server
3. Import all existing domains from the old server.
4. Import all existing accounts, passwords, distribution lists, and aliases from the old server
5. Move all DNS Pointers and firewall port forwards to the new server (or leave the DNS Pointers as they are, and simply swap the servers’ I.P. Addresses old to new, and new to old. (More about this later)
6. Make sure that new mail is arriving  on the new server.
7. Make sure users are able to connect and use the new server.
8. Export Mailbox data from the old server, and import it to the new while the new server is running

Before you go any further …

Most of the commands executed during export, and more importantly during import, may take hours. These should be run directly from  a console session. If you have absolutely no choice then to run the commands remotely, make sure you use the “screen” command, so that if the connection gets interrupted, you can connect back to your screen, without disrupting any running scripts

The export part

Before we begin the export part, we need  to make sure we have enough storage space, which can be accessed from both “old” and “new” servers. The old server did not have any space left but the new server had vitually 1Tb of free space. The way to go, in my case was to remote mount an NFS share from the new server, on the old server and use it as the intermediate storage. Other ways include external usb drives, a network attached storage etc. The choice is really up to you. The steps that follow are generic, and assume the intermediate storage is located at /migration/zimbra. If your path is different you will need to modify the scripts accordingly. To keep things organised, we will create separate folders for all the different migration files. Also make sure the path is readable / writable to Zimbra user. In this case I would recommend

# chmod 777 /migration
# chmod 777 /migration/zimbra
# chown zimbra:zimbra /migration
# chown zimbra:zimbra /migration/zimbra

Step 1. Export all domains

# su - zimbra
# cd /migration/zimbra
# mkdir domains
# cd domains
# zmprov gad | tee -a domains.txt

The command should last only a few seconds, depending on the number of domains on the server. A quick cat domains.txt will show us the available domains

# cat domains.txt
domain1.com
domain2.com

Step 2. Export all accounts

Go back to /migration/zimbra

Create a new directory called accounts and export the admin accounts

# cd /migration/zimbra
# mkdir accounts
# cd accounts
# zmprov gaaa | tee -a admins.txt

Again a quick “cat admins.txt” should list the domain admins, the same ones you see during execution of the “tee – a” command.

Next we export all the users

# zmprov -l gaa | tee -a users.txt

Step 3. Export all account details

Go back to /migration/zimbra

Create a new directory called account_details, and export all account details from Zimbra.

# cd /migration/zimbra
# mkdir account_details
# cd account_details
# for user in `cat ../accounts/users.txt`; do zmprov ga $user  | grep -i Name: | tee -a $user.txt ; done

Step 4. Export all account passwords

Go back to /migration/zimbra

Create a new directory called passwords, and export all shadow passwords

# cd /migration/zimbra
# mkdir passwords
# cd passwords
# for user in `cat ../accounts/users.txt`; do zmprov -l ga $user userPassword | grep userPassword: | awk '{ print $2}' | tee -a $user.shadow; done

Step 5. Export all distribution lists

Go back to /migration/zimbra

Create a new directory called distribution_lists, and export all distribution lists

# cd /migration/zimbra
# mkdir distribution_lists
# cd distribution_lists
# zmprov gadl | tee -a distribution_lists.txt
# for list in `cat distributinlist.txt`; do zmprov gdlm $list > $list.txt ;echo "$list"; done

Step 6.  Export all aliases

Go back to /migration/zimbra

Create a new directory called aliases and export all mail aliases

# cd /migration/zimbra
# mkdir aliases
# cd aliases
# for user in `cat ../accounts/users.txt`; do zmprov ga  $user | grep zimbraMailAlias | awk '{print $2}' | tee -a $user.txt ;echo $i ;done

This process will take some time, depending on the number of accounts. Since most accounts won’t actually have aliases, we will end up with a lot of unnecessary empty text files, which we will delete using “find . -type f -empty | xargs -n1 rm -v”

# find . -type f -empty | xargs -n1 rm -v

So far we have exported everything from our old server, except the mailboxes themselves and any filters the users might have created. We also have the following backup folder structure :-

Step 7. Restore all domains to the new server

The first step in the restoration process starts by restoring all domains. If you have already created the main domain on the new server, you can delete it from /migration/zimbra/domains/domains.txt

The next part assumes that the data we just just exported in the above section is available or copied (in the same directory structure) on the new server. If you have to copy the data, ensure that you issue a chown  -R zimbra:zimbra in /migration/zimbra to make sure, user zimbra has full acess to all the files and directories.

Let me also remind you again that any interruption during the the execution of these scripts could result in database corruption, these commands should be run either from a console or via a screen session. You have been warned.

Let’s begin. Login as zimbra user and issue the following commands :-

# cd /migration/zimbra/domains
# for domain in `cat domains.txt `; do zmprov cd $domain zimbraAuthMech zimbra ;echo $domain ;done

A quick visit to https://youserver-ip:7071 should confirm that the domains have been imported successfully.

Step 8. Restore all accounts and passwords

Restoring accounts and passwords requires the use of a small script.  Restoring each account will entail creating the account with a temporary password, and restoring the old password in another step.

Go to /migration/zimbra

create a directory called “scripts”, and cd to it, and open your favourite editor to create a file called restore_accounts.sh

# cd /migration/zimbra
# mkdir scripts
# cd scripts
# vim restore_accounts.sh

Paste this script to your file and save it

#!/bin/bash
PASSWDS="../passwords"
ACCOUNT_DETAILS="../account_details"
USERS="../accounts/users.txt"
for i in `cat $USERS`
 do
 givenName=$(grep givenName: $ACCOUNT_DETAILS/$i.txt | cut -d ":" -f2)
 displayName=$(grep displayName: $ACCOUNT_DETAILS/$i.txt | cut -d ":" -f2)
 shadowpass=$(cat $PASSWDS/$i.shadow)
 zmprov ca $i "TeMpPa55^()" cn "$givenName" displayName "$displayName" givenName "$givenName"
 zmprov ma $i userPassword "$shadowpass"
done

Make the script executable chmod 777 restore_accounts.sh, and execute it

# chmod 777 restore_accounts.sh
# ./restore_accounts.sh

The user accounts, and their relative passwords are now being imported into the new server. You might get an error every now and again saying :-

ERROR: account.ACCOUNT_EXISTS (email address already exists: [email protected], at DN: uid=admin,ou=people,dc=domain,dc=com)

This basically means that the account which is currently being imported already exists. This is true for admin, anti spam and anti virus accounts (you may wish to delete these accounts from /migration/zimbra/accounts/users.txt. Also keep in mind that after the script runs, the admin password of the new server will now be the same as the admin password of the old server . The password you have created during the installation will be overwritten (unless you delete [email protected] from users.txt)

Step 9. Restore distribution lists

Go to /migration/zimbra and execute the following command to re-create the distribution lists

for lists in `cat  distribution_lists/distribution_lists.txt`; do zmprov cdl distribution_lists/$lists ; echo "$lists -- done " ; done

Next we need to populate the lists, using a very short bash script

Go to /migration/zimbra/distribution_lists and copy and paste the script below to restore_dist_lists.sh

# cd /migration/zimbra/distribution_lists
# vim restore_dist_lists.sh

paste the following script, save and make it executable, and run it.

#!/bin/bash
for list in `cat distribution_lists.txt`
do
    for mbmr in `grep -v '#' ./$list.txt | grep '@'`
    do
        zmprov adlm $list $mbmr
        echo " $mbmr has been added to $list"
    done
done

# chmod 777 restore_dist_lists.sh
# ./restore_dist_lists.sh

You will see members being added to the various distribution lists, as the script goes thru the lists. Depending on the number of distribution lists, and the number of members in each list, this script could take a few minutes to finish

Step 10. Restore all aliases

Next we restore the user aliases.

You guessed it go to /migration/zimbra/aliases

create a new script called restore_aliases.sh paste the script below into it, and make it executable

# cd /migration/zimbra/aliases
# vim restore_aliases.sh

#!/bin/bash
echo "Processing User accounts"
for user in `cat ../accounts/users.txt`
do
    echo $user
    if [ -f "./$user.txt" ]; then
        for alias in `grep '@' ./$user.txt`
        do
            zmprov aaa $user $alias
            echo "$user ALIAS $alias - Restored"
        done
     fi
done

echo "Processing Admin accounts"
for user in `cat ../accounts/admins.txt`
do
    echo $user
    if [ -f "./$user.txt" ]; then
        for alias in `grep '@' ./$user.txt`
        do
            zmprov aaa $user $alias
            echo "$user ALIAS $alias - Restored"
        done
    fi
done

# chmod 777 restore_aliases.sh
# ./restore_aliases.sh

Step 11. Bring the new server online, and take the old server offline

At this point, we have migrated all the user accounts, their passwords, all the distribution lists, and account aliases. For all intents and purposes, we have a fully working server.

For good measure, let’s restart zimbra services

# zmcontrol restart

Wait for the services to restart, and issue a zmcontrol status

#zmcontrol status
Host domain.com
 amavis Running
 antispam Running
 antivirus Running
 ldap Running
 logger Running
 mailbox Running
 memcached Running
 mta Running
 opendkim Running
 proxy Running
 service webapp Running
 snmp Running
 spell Running
 stats Running
 zimbra webapp Running
 zimbraAdmin webapp Running
 zimlet webapp Running
 zmconfigd Running

Login to the web interface, and check that the number of accounts on the new server is the same as the number of accounts on the old server. There could be a minor discrepancy, if you have used a different domain name when you created the new server, this is due to the spam, galsync, and antivirus accounts which will lead to the number of accounts on the new server to be slightly higher than on the old one.

Once happy with what you see, login using a normal user account, and check to see if you can send emails to your domain, using the test server. Since it is running its own DNS server, and it believes it is the mail server for your domain.com, emails to any account on the server should be delivered immediately.

Check /var/log/zimbra.log for any anomalies.

If all seems well, it is time to take server live.

There are two ways to do this.

• Change all DNS records and port forwarding on your firewall to the new server I.P
• Wait for the DNS records to propagate before continuing with the migration

or

• Disconnect the new server from the network
• Change its I.P. Address to that of the old server
• Modify its /etc/hosts, /etc/named.conf and /var/your_zone_file  accordingly
• Restart the zimbra services (this is necessary if you mess around with DNS records)
• Make sure the server is still able to send / receive emails to and form its own domain
• Disconnect the Old mail server from the Network
• Connect the new server to the network (which will now be live)
• Change the IP address of the Old server to an unused I.P. Address
• Modify its /etc/hosts, /etc/named.conf and /var/your_zone_file  accordingly
• Restart zimbra services
• Connect it back to the network, to continue the migration of the mailbox data

The second method, seems more complicated, but it will guarantee ZERO downtime. Believe it or not, it could be the fastest method to switch over to the new server.

Once the switch over is complete, any users logging in to the server using IMAP or Webmail will be greeted with an empty mailbox, but any new emails arriving to the domain, will be delivered to the new server.

In the final few steps we will export the mailbox data from the old server, and import it to the new server.

Step 12. Migrating mailbox data

The next process will be the most time consuming of all the processes. Depending on the amount of data, exporting the data could take as long as a day if not more. Data import will take just as long.

Now that our users can login to the new server and send and receive emails, the pressure is a bit lower. Also if a very important email needs to be accessed, this can be done via the web interface of the old server, using https://old-server-i.p.

Let’s start the data export.

Login to the old server, via console or via a “Screen” session. I am assuming that the backup repository we were using in /migration/zimbra is mounted again on the “old” server using the same path

Login as Zimbra user and goto /migration/zimbra

# su - zimbra
# cd /migration/zimbra 
# mkdir mailbox_data
# cd mailbox_data
# for user in `cat ../accounts/users.txt`; do echo "Exporting mailbox $user" ; zmmailbox -z -m $user getRestURL '/?fmt=tgz' > ./$user.tgz ; done

You will start seeing mailboxes being exported. This is going to take a while, so you might go endeavor in other projects.

In the event that any mailboxes report errors during export, make a note of the problematic accounts, and try to re export them later, by creating a text file :

/migration/zimbra/accounts/problematic_accounts.txt 

and running the following command. The format of the file needs to be exactly like the format inside /migration/zimbra/accounts/users.txt. Use it as reference.

# for user in `cat ../accounts/problematic_accounts.txt`; do echo "Exporting mailbox $user" ; zmmailbox -z -m $user getRestURL '/?fmt=tgz' > ./$user.tgz ; done

Once the mailbox migration complete, let’s also export any user filter settings. We need a bash script for this.

Create a new directory called filters, create a file called export_filters.sh and paste the script below into it.

# mkdir /migration/zimbra/filters
# cd /migration/zimbra/filters
# vim export_filters.sh
# chmod 777 export_filters.sh

#!/bin/bash
mkdir tmp
set -x
clear
for user in `cat ../accounts/users.txt`; 
do
    filter=`zmprov ga $user zimbraMailSieveScript > ./tmp/$user`
    sed -i -e "1d" ./tmp/$user
    sed 's/zimbraMailSieveScript: //g' ./tmp/$user > ./$user;
    rm ./tmp/$user
    echo "Export filter for $user"
done
\rm -rf tmp

Mount, move or copy /accounts/zmigrate to the new server, and go to the /migration/zimbra/mailbox_data directory

# cd /migration/zimbra/mailbox_data
# for mailbox in `cat ../accounts/users.txt`; do zmmailbox -z -m $mailbox postRestURL "/?fmt=tgz&resolve=skip" ./$mailbox.tgz ; echo "$mailbox - done "; done

This process will also take quite a while to complete, the users may use the server freely during the import. When their turn arrives, they will start seeing their mailbox starting to get populated with the emails they left behind on the old server.

If you get any errors, you can re-export, and re-import the problematic accounts by manipulating /migration/zimbra/accounts/users.txt accordingly and re-running the export command. If you get any “broken pipe” type errors it probably means that the tar archive got corrupted during the transfer for some reason or other, and is essentially incomplete.

Finally let’s import the user filters. Go to /migration/zimbra/filters, and create a file called import_filters.sh. Paste the script below into it, save and make it executable.

# cd /migration/zimbra/filters
# vim import_filters.sh
# chmod 777 import_filters.sh

#!/bin/bash
for filter in ./*
do
    if [ "$filter" == "./import_filters.sh" ] ; then
        continue;
    fi


    if [ "$filter" == "./export_filters.sh" ] ; then
        continue;
    fi

    if [ "$filter" == "./tmp" ] ; then
        continue;
    fi

    Filter_String=`cat "$filter"`
    Account=$filter
    zmprov ma $(echo $filter | grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b") zimbraMailSieveScript '$Filter_String'
    echo "Process filter $Account"
done
echo "All filter has been import successfully"

Conclusion

That’s it,  your users should be happy using your newly migrated server. The only thing they will find missing are their account signatures. To date I have not yet found an automated reliable way to copy them automatically. What I did in this case was, made the old server available on a temporary sub-domain (old-webmail.domain.com), so that the users could login in to the old server, and essentially copy and paste their signature to the new server. If you feel extremely generous, you could also do it via the admin interface, but you would have to do the process account by account manually, so it could take some time.