Better Backups With Tar

Many professional users still lose information due to lack of making good computer file backups, yet Linux provides powerful tools that can help you make backups easily and automatically.

The example script on this page is an improvement of Chapter33.3. Automating backups with tar which shows a basic, yet, effective backup script (Chapter 33 has several other good ideas too).

If you understand how scripts work and are confident with being able to modify one, then you can modify the one shown below to meet your particular circumstances and create an effective, automated backup for yourself and/or custom backups for other users.

The improvements in this script are:

It shows you how you can backup more than one directory using the same backup.cron file.
In addition to a weekly backup, you have a rotating backup for the last and previous weeks therefore allowing you to go back in time more than just one week (considered a nice feature for an office server setting).
Ownership of backup files can be given to another user and group which may hold lower privileges than root access (this way, the administrator or user does not need root access to get the backup files to store elsewhere).
Fewer Setup steps required to enable this script.
Send error messages to your syslog file or another error message file of your choice (useful if you encounter errors since nobody is around at night to read error messages when creating backups, and therefore you end up unknowingly with incomplete backups).

As is mentioned in Chapter33.3. Automating backups with tar, it is also recommended to run this script at the beginning of the month for the first time, then insert the script in the /etc/cron.daily/ directory afterwards and let it run for a month automatically before making any major changes.

STEPS REQUIRED TO SETUP AND AUTOMATE BACKUPS

In the script example shown below, we backup 3 different directories to a backup directory on a local server.

First, create the backup script backup.cron file as shown below so that you can test it out manually. Make sure to use directories 1, 2, 3 according to what you need (this example backs up the directories /home/bird, /home/cat, /home/dog

#!/bin/sh
# full and incremental backup script
# created 07 February 2000
# Based on a script by Daniel O'Callaghan <danny@freebsd.org>
# modified by Gerhard Mourani <gmourani@videotron.ca>
# modified by Jose Da Silva <Digital@JoesCat.com> 05/feb/08
# modified by Jose Da Silva <Digital@JoesCat.com> 08/sep/10
#
# This script was originally based on above script shown on
# <http://www.faqs.org/docs/securing/chap29sec306.html> and was
# modified to backup several directories plus keep a "lastweek"
# "prevweek" and "priorweek" backups too, and report errors.
#
# If you need to backup directories which contain active files,
# it is recommended that you do a snapshot of those directories
# and then do a backup of the snapshot, this way, you have less
# problems dealing with file-locks and files in transition.
# If you need to run a snapshot first, just insert a call to the
# appropriate script at the beginning/end of the main routine below.
#
# Change the variables below to fit your computer/backup system

COMPUTER=genesis		# Name of this computer
DIR1="/home/bird"		# Directories to backup daily
DIR2="/home/cat"		# (only 3 examples shown here, but
DIR3="/home/dog"		#  more can be added DIR4, DIR5...)

BKDIR=/backups			# Where to store all the backups
TIMEDIR=/backups/last-full	# Where to store time of full backup
ERRFILE=/var/log/syslog		# Send error messages to this file
BKOWNER=root.adm		# Owner and Group for backup files
CHMODCHOWN=1			# This file system can use chmod+chown

# Reduce list of paths to trusted directories (most->least trusted)
PATH=/bin:/usr/bin:/usr/local/bin

# Name and location of commands so you don't need to search paths
TAR=/bin/tar			# Name and location of commands
BASENAME=/bin/basename		# Use "which" to find commands, but
CAT=/bin/cat			# try to check /bin before using
DATE=/bin/date			# commands located elsewhere such as
ECHO=/bin/echo			# in /usr/local/bin or in /usr/bin
CHMOD=/bin/chmod
CHOWN=/bin/chown
HOSTNAME=/bin/hostname
MKDIR=/bin/mkdir
MV=/bin/mv
WHOAMI=/usr/bin/whoami

DOW=`$DATE +%a`			# Day of the week e.g. Mon
DOM=`$DATE +%d`			# Date of the Month e.g. 27
MD=`$DATE +%b%d`		# Month and Date e.g. Sep27
MDT=`$DATE +"%b %d %T"`		# Month Day Time Sep 27 12:00:00
CMD=`$BASENAME "$0"`		# Command Line Program
EM="$MDT `$HOSTNAME` $CMD[$$]:"	# Error Message info

errortmp=0			# Temporary error accumilator
errors=0			# 0 if no errors found at all

# On the 1st of the month, do permanent full backups.
#
# Every Sunday, prevweek's backup is pushed to priorweek's backup,
# lastweek's backup pushed to prevweek's backup, and then Sunday's
# backup is pushed to lastweek's backup before creating a Sunday
# full backup. This creates 4 weeks worth of rollover backups.
#
# Monday to Saturday, an incremental backup is made based on Sunday
# so that you you have daily backups for new files until next week.
#
# if NEWER = "", then tar backs up all files in the directories
# otherwise it backs up files newer than the NEWER date. NEWER
# gets its date from the file written every Sunday.

ErrorTest () {
  # Check exit status of last command for any errors and set flag
  if [ "$?" -ne 0 ]; then
    errortmp=1;
  fi
}

ErrorSet () {
  # Set errors if errortmp=1 and send error message to $ERRFILE
  if [ $errortmp -eq 1 ]; then
    $ECHO "$EM Error $1" >> $ERRFILE;
    errors=1
  fi
}

UpdateTheDate() {
  # Update full backup date so increments happen after this
  errortmp=0;
  NOW=`$DATE +"%Y-%m-%d %X"`; ErrorTest;
  $ECHO "$NOW" > "$TIMEDIR/$COMPUTER-full-date"; ErrorTest;
  if [ $CHMODCHOWN -eq 1 ]; then
    $CHMOD 640 "$TIMEDIR/$COMPUTER-full-date"; ErrorTest;
    $CHOWN $BKOWNER "$TIMEDIR/$COMPUTER-full-date"; ErrorTest;
  fi
  ErrorSet "with time stamp $TIMEDIR/$COMPUTER-full-date";
}

MakeFullMonthlyBackup() {
  # Make a full monthly backup based on given directories
  errortmp=0;
  $TAR -cpzf "$BKDIR/$COMPUTER-$MD-$1.tar.gz" "$2"; ErrorTest;
  if [ $CHMODCHOWN -eq 1 ]; then
    $CHMOD 640 "$BKDIR/$COMPUTER-$MD-$1.tar.gz"; ErrorTest;
    $CHOWN $BKOWNER "$BKDIR/$COMPUTER-$MD-$1.tar.gz"; ErrorTest;
  fi
  ErrorSet "with tar file $BKDIR/$COMPUTER-$MD-$1.tar.gz";
}

MakeFullWeeklyBackup() {
  # Move previous week's backups into prior week's backups
  errortmp=0;
  if [[ -f "$BKDIR/$COMPUTER-$DOW-prevweek-$1.tar.gz" ]]; then
    $MV "$BKDIR/$COMPUTER-$DOW-prevweek-$1.tar.gz" \
        "$BKDIR/$COMPUTER-$DOW-priorweek-$1.tar.gz"; ErrorTest;
    if [ $CHMODCHOWN -eq 1 ]; then
      $CHMOD 640 "$BKDIR/$COMPUTER-$DOW-priorweek-$1.tar.gz";
      ErrorTest;
      $CHOWN $BKOWNER "$BKDIR/$COMPUTER-$DOW-priorweek-$1.tar.gz";
      ErrorTest;
    fi
    ErrorSet "moving $BKDIR/$COMPUTER-$DOW-prevweek-$1.tar.gz";
  fi
  # Move last week's backups into previous week's backups
  errortmp=0;
  if [[ -f "$BKDIR/$COMPUTER-$DOW-lastweek-$1.tar.gz" ]]; then
    $MV -f "$BKDIR/$COMPUTER-$DOW-lastweek-$1.tar.gz" \
           "$BKDIR/$COMPUTER-$DOW-prevweek-$1.tar.gz"; ErrorTest;
    if [ $CHMODCHOWN -eq 1 ]; then
      $CHMOD 640 "$BKDIR/$COMPUTER-$DOW-prevweek-$1.tar.gz";
      ErrorTest;
      $CHOWN $BKOWNER "$BKDIR/$COMPUTER-$DOW-prevweek-$1.tar.gz";
      ErrorTest;
    fi
    ErrorSet "moving $BKDIR/$COMPUTER-$DOW-lastweek-$1.tar.gz";
  fi
  # Then move this week's full backups into last week's backups
  errortmp=0;
  if [[ -f "$BKDIR/$COMPUTER-$DOW-$1.tar.gz" ]]; then
    $MV "$BKDIR/$COMPUTER-$DOW-$1.tar.gz" \
        "$BKDIR/$COMPUTER-$DOW-lastweek-$1.tar.gz"; ErrorTest;
    if [ $CHMODCHOWN -eq 1 ]; then
      $CHMOD 640 "$BKDIR/$COMPUTER-$DOW-lastweek-$1.tar.gz";
      ErrorTest;
      $CHOWN $BKOWNER "$BKDIR/$COMPUTER-$DOW-lastweek-$1.tar.gz";
      ErrorTest;
    fi
    ErrorSet "moving weekly file $BKDIR/$COMPUTER-$DOW-$1.tar.gz";
  fi
  # Then create a new weekly backup for this day-of-week
  errortmp=0;
  $TAR -cpzf "$BKDIR/$COMPUTER-$DOW-$1.tar.gz" "$2"; ErrorTest;
  if [ $CHMODCHOWN -eq 1 ]; then
    $CHMOD 640 "$BKDIR/$COMPUTER-$DOW-$1.tar.gz"; ErrorTest;
    $CHOWN $BKOWNER "$BKDIR/$COMPUTER-$DOW-$1.tar.gz"; ErrorTest;
  fi
  ErrorSet "with weekly file $BKDIR/$COMPUTER-$DOW-$1.tar.gz";
}

MakeIncrementalWeeklyBackup() {
  # Make an incremental backup based on date in NEWER file
  errortmp=0;
  $TAR --newer="$1" -cpzf "$BKDIR/$COMPUTER-$DOW-$2.tar.gz" "$3";
  ErrorTest;
  if [ $CHMODCHOWN -eq 1 ]; then
    $CHMOD 640 "$BKDIR/$COMPUTER-$DOW-$2.tar.gz"; ErrorTest;
    $CHOWN $BKOWNER "$BKDIR/$COMPUTER-$DOW-$2.tar.gz"; ErrorTest;
  fi
  ErrorSet "with incremental file $BKDIR/$COMPUTER-$DOW-$2.tar.gz";
}

#----- Main program starts here -----
if [ "`$WHOAMI`" != "root" ]; then
  $ECHO "$EM Sorry, you must be root!";
  exit 1
fi

# Verify backup directory exists, otherwise create it.
if [[ ! -d "$BKDIR" ]]; then
  $MKDIR "$BKDIR"
  if [[ ! -d "$BKDIR" ]]; then
    $ECHO "$EM Error, cannot make $BKDIR!" >> $ERRFILE;
    $ECHO "$EM Error, no backup files made!" >> $ERRFILE;
    exit 2
  else
    if [ $CHMODCHOWN -eq 1 ]; then
      errortmp=0;
      $CHMOD 740 "$BKDIR"; ErrorTest;
      $CHOWN $BKOWNER "$BKDIR"; ErrorTest;
      ErrorSet "setting permissions on directory $BKDIR";
    fi
  fi
fi

# Verify time directory exists, otherwise create it.
if [[ ! -d "$TIMEDIR" ]]; then
  $MKDIR "$TIMEDIR"
  if [[ ! -d "$TIMEDIR" ]]; then
    $ECHO "$EM Error, cannot make $TIMEDIR!" >> $ERRFILE;
    $ECHO "$EM Error, no backup files made!" >> $ERRFILE;
    exit 3
  else
    if [ $CHMODCHOWN -eq 1 ]; then
      errortmp=0;
      $CHMOD 640 "$TIMEDIR"; ErrorTest;
      $CHOWN $BKOWNER "$TIMEDIR"; ErrorTest;
      ErrorSet "setting permissions on directory $TIMEDIR";
    fi
  fi
fi

# Verify time file exists, otherwise create it.
if [[ ! -f "$TIMEDIR/$COMPUTER-full-date" ]]; then
  UpdateTheDate;
  if [[ ! -f "$TIMEDIR/$COMPUTER-full-date" ]]; then
    $ECHO "$EM Error, cannot find $TIMEDIR/$COMPUTER-full-date!" \
	  >> $ERRFILE;
    $ECHO "$EM Error, no backup files made !" >> $ERRFILE;
    exit 4
  else
    if [ $CHMODCHOWN -eq 1 ]; then
      errortmp=0;
      $CHMOD 640 "$TIMEDIR/$COMPUTER-full-date"; ErrorTest;
      $CHOWN $BKOWNER "$TIMEDIR/$COMPUTER-full-date"; ErrorTest;
      ErrorSet "setting permissions, $TIMEDIR/$COMPUTER-full-date";
    fi
  fi
fi

# Create Monthly Backups on 1st day of each month
if [ $DOM = "01" ]; then
  MakeFullMonthlyBackup "1" "$DIR1";
  MakeFullMonthlyBackup "2" "$DIR2";
  MakeFullMonthlyBackup "3" "$DIR3";
fi

if [ $DOW = "Sun" ]; then
  # Create Full Weekly Backups on Sundays
  MakeFullWeeklyBackup "1" "$DIR1";
  MakeFullWeeklyBackup "2" "$DIR2";
  MakeFullWeeklyBackup "3" "$DIR3";

  UpdateTheDate;
else
  # Make incremental backups - overwrite last weeks

  # Get date of last full backup
  NEWER="`$CAT $TIMEDIR/$COMPUTER-full-date`"

  MakeIncrementalWeeklyBackup "$NEWER" "1" "$DIR1";
  MakeIncrementalWeeklyBackup "$NEWER" "2" "$DIR2";
  MakeIncrementalWeeklyBackup "$NEWER" "3" "$DIR3";
fi

#if [ $errors -eq 1 ]; then
#  # Errors were found while doing a backup, warn someuser!
#  $ECHO "$EM Error creating backups!" >> \
#	 /home/someuser/Desktop/warning.txt
#  $CHMOD 777 /home/someuser/Desktop/warning.txt
#fi

Test your backup.cron file to make sure it runs. You will need to make the backup.cron file executable so that you can test it out manually on the command line.

[dog@genesis tmp]$ su
Password: ***
[root@genesis tmp]# chmod 750 backup.cron
[root@genesis tmp]# chown root.adm backup.cron
[root@genesis tmp]# ls -l backup.cron
-rwxr-x---  1 root adm 6161 Aug  8 21:00 backup.cron*
[root@genesis tmp]# ./backup.cron

Below is a quick look at the backup directory after a few test runs (the script was modified to use "08" and "Tue" as the first dates so that you can see these results shown here).

[root@genesis tmp]# ls -l /backups/
-rw-r-----  1 root adm     2307 Aug  8 21:21 genesis-Aug08-1.tar.gz
-rw-r-----  1 root adm   367504 Aug  8 21:21 genesis-Aug08-2.tar.gz
-rw-r-----  1 root adm 32094115 Aug  8 21:22 genesis-Aug08-3.tar.gz
-rw-r-----  1 root adm      138 Aug  8 21:47 genesis-Tue-1.tar.gz
-rw-r-----  1 root adm     1289 Aug  8 21:47 genesis-Tue-2.tar.gz
-rw-r-----  1 root adm     8383 Aug  8 21:47 genesis-Tue-3.tar.gz
drw-r-----  2 root adm     4096 Aug  8 21:21 last-full/

[root@genesis tmp]# ls -l /backups/last-full/
-rw-r-----  1 root adm  20 Aug  8 21:21 genesis-full-date

Backups is something you will do often, but you should also verify that your backups are indeed working correctly because restores are something that is seldom done yet very important to do when the time comes to restore a file (or many files). Please check Chapter 33, mentioned above, for steps on how to restore files and directories.

Once you are satisfied that the script appears to run well from the command line, then the next step is to install backup.cron in the cron directory and let your computer run it automatically. You should check the /backups/ directory the next day to verify it holds backups that were made at night.

[root@genesis tmp]# cp backup.cron /etc/cron.daily/backup.cron
[root@genesis tmp]# chmod 750 /etc/cron.daily/backup.cron
[root@genesis tmp]# chown root.adm /etc/cron.daily/backup.cron
[root@genesis tmp]# ls -l /etc/cron.daily/backup.cron
-rwxr-x---  1 root adm 6145 Aug  8 23:05 /etc/cron.daily/backup.cron*

NOTES TO THINK ABOUT

Important things to think about when using backup.cron

You may note that doing backups with tar is a disk intensive activity, so if your computer is doing little else at one o'clock in the morning, it's perhaps best to let it finish quickly.

If your computer is a server with additional duties, you may want to add the nice -2 command to make the backup more of a background task to give other processes more priority. This will make backups take longer, but it does allow other processes to run in better priority.

You may note that both this script and the basic script mentioned in Chapter33.3. Automating backups with tar have trouble backing up files that are still in use. If this is the case, you need to add to your backup.cron script a shutdown of the process(es) that cause trouble before you do a backup of that particular directory, then restart the process(es) after tar finishes those directories (for example SQL and other live services). A quicker method to shorten the shutdown time involves taking a snapshot of the troublesome directories, restart the process(es) and making a backup of the snapshot.

Some file systems and some versions of tar are limited to output file sizes of about 2 gigabytes. Make sure you use a recent version of tar and if you find that the output file will be approaching or passing 2 gigabytes, then you should think of using the tar --multi-volume option or you can use split to split the output to multiple files. If you use split then remember to use cat to join the files for restoring.

COMMENTS, ERRORS OR FEEDBACK

It is great if you find this backup.cron script useful, but if you can think of improvements or fixes, please send suggestions using the comment section here so that the script can be improved:

Links To Sections Of This How To
Top Of Page Setup Steps Notes Comments

Other Pages On This Web Site
Home Page Disk Image Backup BOINC On Linux Java Install Ship Arcade PicDis Disassembler

Links To Sections Of This How To
Top Of Page	Setup Steps	Notes	Comments

Other Pages On This Web Site
Home Page	Disk Image Backup	BOINC On Linux	Java Install	Ship Arcade	PicDis Disassembler

Your Name:
Your Email: