support new duplicity versions:

- general cleanup of dup handler
 - support remove-older-than command
 - also run "duplicity cleanup" before anything else, for safety reasons
 - migrated full/incremental backup switch to new syntax
ChangeLog: removed trailing space
This commit is contained in:
intrigeri 2008-01-14 20:32:19 +00:00
parent 18f6031a72
commit ef9e8e0dc2
2 changed files with 199 additions and 135 deletions

View File

@ -2,9 +2,17 @@ version 0.9.6 -- unreleased
backupninja changes
handler changes
rdiff:
dup:
. General cleanup
. Better support for new duplicity (>= 0.4.4) command line syntax:
run remove-older-than when $keep is not set to yes (Closes: #458816),
and run "duplicity cleanup" before any other duplicity command; both
only trigger a warning on failure, since they should not stop backups
from being done. Also migrated full/incremental backup switch to the
new syntax.
rdiff:
. Fixed ignore_version default value missing
. Add patch from Matthew Palmer to rdiff handler to incorporate sshoptions
. Add patch from Matthew Palmer to rdiff handler to incorporate sshoptions
into options via remote-schema not already specified (Closes: #424639)
wget:
. New handler from rhatto designed to incrementally pull content from
@ -38,7 +46,7 @@ version 0.9.5 -- December 2, 2007
to be correct, also set TLS to be the default over SSL (Closes: Trac#13)
maildir:
. Added an examples file (Closes: Trac#23)
. Applied patch from Anarcat that fixes the cp/mkdir calls to not use GNU
. Applied patch from Anarcat that fixes the cp/mkdir calls to not use GNU
coreutils options, as well as some bashisms (Closes: Trac#24)
. Fix test mode (Closes: Trac#25)
mysql:
@ -48,8 +56,8 @@ version 0.9.5 -- December 2, 2007
the data from a backup, but still backup the table structure. This is very
useful in cases where tables contain large amounts of cache data. See the
example.mysql for options, thanks Daniel Bonniot (Closes: #408829)
. Enhance code for selecting databases by asking MySQL not to give us the
header (-N), to not draw pretty boxes around the output (-B), send the query
. Enhance code for selecting databases by asking MySQL not to give us the
header (-N), to not draw pretty boxes around the output (-B), send the query
via -e instead of a pipe and ensure MySQL listens to -B. Thanks to
Matthew Palmer (Closes: #452039).
pgsql:
@ -83,7 +91,7 @@ version 0.9.5 -- December 2, 2007
. Fixed example in example.sys to detail the __star__ in partitionsfile and
note why its necessary (Closes: #409192)
. Force C locale for sfdisk to ensure english words are found in grep
. Make directory where output is placed configurable, and create the parent dir
. Make directory where output is placed configurable, and create the parent dir
if it doesn't exist (Closes: Trac#1)
ninjareport
. Added first draft of method to aggregate reports from many servers into
@ -94,7 +102,7 @@ version 0.9.5 -- December 2, 2007
fixed 'make install' bug that failed if /etc/backup.d already existed
changed spaces to tabs in Makefile.am
updated redhat spec file (thanks Adam Monsen)
version 0.9.4 -- October 6th, 2006
backupninja changes
. Fixed bug in toint(), and thus isnow(), which caused it
@ -103,7 +111,7 @@ version 0.9.4 -- October 6th, 2006
. Add admingroup option to configuration to allow a group that can
read/write configurations (instead of only allowing root). Checks
and complains about group-readable files only when the group differs
from the one in the configuration file (default is root as before).
from the one in the configuration file (default is root as before).
Thanks to Martin Krafft for the patch (Closes: #370396).
. When determining which backup actions to make, find now follows
symlinks for $configdirectory
@ -117,7 +125,7 @@ version 0.9.4 -- October 6th, 2006
Added rsnap handler:
. rotated rsync snapshops
. code from paulv@bikkel.org
Added rub handler:
Added rub handler:
. alternative to rsnap
. code from rhatto@riseup.net
mysql:
@ -133,7 +141,7 @@ version 0.9.4 -- October 6th, 2006
. Fixed shell expansion, thanks Thomas Kotzian (Closes: #363297)
. postgres user UID is now the one from inside the vserver if necessary
. Compress now happens in-line to save some disk space (Closes: #370778)
. $PGSQLUSER is used instead of hardcoding user 'postgres' (although this is the default)
. $PGSQLUSER is used instead of hardcoding user 'postgres' (although this is the default)
svn:
. Fixed inversed vsname emptiness check
rdiff:
@ -176,7 +184,7 @@ version 0.9.4 -- October 6th, 2006
vserver:
. init_vservers: fixed Debian bug #351083 (improper readlink syntax)
. found_vservers: escaped special grep repetition character +
. forced mktemp to use a template with a name to be more compatible with
. forced mktemp to use a template with a name to be more compatible with
different versions of mktemp, thanks anarcat
ninjahelper changes
. Recursively ignore subdirs in /etc/backup.d (Closes: #361102)
@ -193,7 +201,7 @@ version 0.9.4 -- October 6th, 2006
changed /etc/backup.d permissions to 0770 (for admingroup)
minor documentation fixes
improved RPM build process allowing 'make rpm-package' and 'make
srpm-package' targets, also fixes permissions on man directories,
srpm-package' targets, also fixes permissions on man directories,
cleans up RPM-related files during distclean, and adds default
EDITOR for "autogen.sh -f" if none is set. (thanks Robert Napier)
@ -242,7 +250,7 @@ version 0.9.3 -- February 1st, 2006
. init_vservers: warn if vservers are enabled but no vserver is found
. new function: vservers_exist
known bugs:
easydialog:
easydialog:
. formDisplay does not return exit status.
version 0.9.2 -- December 29, 2005
@ -256,7 +264,7 @@ version 0.9.2 -- December 29, 2005
trac:
. mkdir subdirectory problem fixed
duplicity:
. globbing support fixed in include and exclude options
. globbing support fixed in include and exclude options
. different signing and encrypting key support added
. fixed erroneous comments in example.dup about the way
GnuPG-related options are used
@ -270,12 +278,12 @@ version 0.9.2 -- December 29, 2005
rdiff-backup:
. used to expand '*' in default source directories
. the "Cancel" buttons used to have a weird behaviour
. updated to include Vserver selection
. updated to include Vserver selection
pgsql:
. forbid the user to choose an empty database set
. forbid the user to choose an empty database set
. "Cancel" button now does what it is meant to do
mysql:
. enhanced for vserver support
. enhanced for vserver support
. now able to select databases and dump directory
duplicity:
. new handler added (with Vserver support)
@ -291,20 +299,20 @@ version 0.9.1 -- November 05 2005
changed subversion handler to use svnadmin hotcopy instead of
the unsupported hot-copy.py script, which was moved in Debian
update rdiff ninjahelper
now detects and auto-install rdiff-backup on the remote
now detects and auto-install rdiff-backup on the remote
machine if possible, also tests the remote backup directory
and offers to create it if it doesn't exist
version 0.9 -- October 19 2005
*** IMPORTANT CHANGE, UPGRADE AT ONCE ***
fixed insecure temporary file creation
fixed insecure temporary file creation
*****************************************
removed erroneous magic file marker in pgsql handler
fixed incorrect find positional
changed direct grep of /etc/passwd to getent passwd.
rdiff helper has much better information on failed ssh attempt
(patch from cmccallum@thecsl.org).
rdiff handler now supports remote source and local dest.
rdiff handler now supports remote source and local dest.
(patch from cmccallum@thecsl.org).
man pages are greatly improved.
@ -353,7 +361,7 @@ version 0.6 -- June 16 2005
slapcat), restart, passwordfile and binddn. Default backup method
is set to ldapsearch as this is safer
******************************************************************
NOTE: to get the previous default behavior with the ldap handler,
NOTE: to get the previous default behavior with the ldap handler,
you must set "method = slapcat". The new default is ldapsearch.
******************************************************************
implemented fix so that the main script will echo fatal errors rather
@ -369,10 +377,10 @@ version 0.6 -- June 16 2005
added "testconnect" option
added "sshoptions" option (will be passed to ssh/scp)
added "bandwidthlimit" option
example.dup example config file for duplicity handler
example.dup example config file for duplicity handler
added trac (http://trac.edgewall.com/) environment handler (thanks Charles Lepple!)
added configfile option to mysql handler
the default is /etc/mysql/debian.cnf. with this,
the default is /etc/mysql/debian.cnf. with this,
sqldump doesn't need dbusername. (hotcopy still does).
fixed bug in mysql handler which caused some passwords to not work.
(.my.cnf files now have double quotes around password)
@ -380,15 +388,15 @@ version 0.6 -- June 16 2005
version 0.5 -- April 12 2005
rdiff handler works when remote sshd has a banner
rdiff handler supports local dest
rdiff handler supports local dest
logfile is created if it doesn't exist
added "when = hourly"
added "when = hourly"
added optional 'nicelevel' to rdiff handler
fixed bug where actions were not run in numeric order.
improved 'when' parsing.
version 0.4.4 -- March 18 2005
results of handlers are now read line by line.
results of handlers are now read line by line.
changes to rdiff handler: added "options", and "keep" is
not necessarily days now (ie, it will pass straight through to
rdiff-backup if the keep has a unit on it, otherwise it adds the 'D').
@ -396,7 +404,7 @@ version 0.4.4 -- March 18 2005
added maildir handler (very specialized handler)
added --run option (runs the specified action file)
improved sys handler, now uses hwinfo
added subversion hotbackup handler, svn.
added subversion hotbackup handler, svn.
added PATH to cron.d file, which fixes file not found errors.
version 0.4.2 -- Jan 6 2005
@ -409,7 +417,7 @@ version 0.4.1 -- Jan 3 2005
fixed major bug, 'when' actually works now.
replaced debug function with debug, info, warning, error, fatal.
added --now option to force all actions to be performed now.
version 0.4 -- Dec 26 2004
added "when" option, so that all configs can specify when
they are to be run.
@ -417,7 +425,7 @@ version 0.4 -- Dec 26 2004
added .sys handler (hardware, packages, partitions).
version 0.3.4 -- Dec 8 2004
fixed numerical variable quoting compatibility with older wc
fixed numerical variable quoting compatibility with older wc
fixed stderr redirect bug
some comments in example.rdiff
@ -440,7 +448,7 @@ version 0.3.1 -- Sept 5 2004
force only root can read /etc/backup.d/*
fixed missing equals symbols in example.rdiff
changed backupninja executable to be /usr/sbin rather than /usr/bin
version 0.3 -- Oct 20 2004
** IMPORTANT ** all config files are now ini style, not apache style
rewrote all scripts in bash for portability
@ -452,7 +460,7 @@ version 0.2 -- Oct 14 2004
fixed bug: removed printr of excludes (!)
added support for changing the user/group in rdiff sources.
added support for .mysql config files.
version 0.1 -- Oct 8 2004
initial release
version 0.1 -- Oct 8 2004
initial release

View File

@ -31,11 +31,13 @@ getconf destdir
getconf destuser
destdir=${destdir%/}
[ "$destdir" != "" ] || fatal "Destination directory not set"
[ "$include" != "" ] || fatal "No source includes specified"
### SANITY CHECKS ##############################################################
### vservers stuff ###
[ -n "$destdir" ] || fatal "Destination directory not set"
[ -n "$include" ] || fatal "No source includes specified"
[ -n "$password" ] || fatal "The password option must be set."
### VServers
# If vservers are configured, check that the ones listed in $vsnames do exist.
local usevserver=no
if [ $vservers_are_available = yes ]; then
@ -54,125 +56,95 @@ else
[ -z "$vsinclude" ] || warning 'vservers support disabled in backupninja.conf, vsincludes configuration lines will be ignored'
fi
### see if we can login ###
### See if we can login on $desthost
if [ "$testconnect" == "yes" ]; then
debug "ssh $sshoptions -o PasswordAuthentication=no $desthost -l $destuser 'echo -n 1'"
if [ ! $test ]; then
result=`ssh $sshoptions -o PasswordAuthentication=no $desthost -l $destuser 'echo -n 1'`
if [ "$result" != "1" ]; then
fatal "Can't connect to $desthost as $destuser."
else
debug "Connected to $desthost as $destuser successfully"
fi
fi
debug "ssh $sshoptions -o PasswordAuthentication=no $desthost -l $destuser 'echo -n 1'"
if [ ! $test ]; then
result=`ssh $sshoptions -o PasswordAuthentication=no $desthost -l $destuser 'echo -n 1'`
if [ "$result" != "1" ]; then
fatal "Can't connect to $desthost as $destuser."
else
debug "Connected to $desthost as $destuser successfully"
fi
fi
fi
### COMMAND-LINE MANGLING ###
### COMMAND-LINE MANGLING ######################################################
### initialize $execstr*
execstr_command=
execstr_options="$options --no-print-statistics"
execstr_source=
execstr_serverpart="scp://$destuser@$desthost/$destdir"
### duplicity version
duplicity_version="`duplicity --version | @AWK@ '{print $2}'`"
duplicity_major="`echo $duplicity_version | @AWK@ -F '.' '{print $1}'`"
duplicity_minor="`echo $duplicity_version | @AWK@ -F '.' '{print $2}'`"
duplicity_sub="`echo $duplicity_version | @AWK@ -F '.' '{print $3}'`"
# 1. duplicity >= 0.4.2 needs --sftp-command (NB: sftp does not support the -l option)
# 2. duplicity >= 0.4.3 replaces --ssh-command with --ssh-options, which:
# - is passed to scp and sftp commands by duplicity
# - has a special syntax we can not directly feed the command line with
# (e.g. "IdentityFile=/root/.ssh/id_dsa_backupninja Port=2222", without the -o)
# so we don't use it: since this version does not use the ssh command anymore,
# we keep compatibility with our previous config files by passing $sshoptions to
# --scp-command and --sftp-command ourselves
### ssh/scp/sftp options
# 1. duplicity >= 0.4.2 needs --sftp-command
# (NB: sftp does not support the -l option)
# 2. duplicity 0.4.3 to 0.4.9 replace --ssh-command with --ssh-options, which is
# passed to scp and sftp commands by duplicity. We don't use it: since this
# version does not use the ssh command anymore, we keep compatibility with
# our previous config files by passing $sshoptions to --scp-command and
# --sftp-command ourselves
scpoptions="$sshoptions"
[ "$bandwidthlimit" == 0 ] || scpoptions="$scpoptions -l $bandwidthlimit"
execstr="$options --no-print-statistics "
# < 0.4.2 : only uses ssh and scp
if [ "$duplicity_major" -le 0 -a "$duplicity_minor" -le 4 -a "$duplicity_sub" -lt 2 ]; then
execstr="$execstr --scp-command 'scp $scpoptions' --ssh-command 'ssh $sshoptions' "
execstr_options="${execstr_options} --scp-command 'scp $scpoptions' --ssh-command 'ssh $sshoptions'"
# >= 0.4.2 : also uses sftp, --sftp-command option is now supported
else
sftpoptions="$sshoptions"
# == 0.4.2 : uses ssh, scp and sftp
if [ "$duplicity_major" -eq 0 -a "$duplicity_minor" -eq 4 -a "$duplicity_sub" -eq 2 ]; then
execstr="$execstr --scp-command 'scp $scpoptions' --sftp-command 'sftp $sftpoptions' --ssh-command 'ssh $sshoptions' "
execstr_options="${execstr_options} --scp-command 'scp $scpoptions' --sftp-command 'sftp $sftpoptions' --ssh-command 'ssh $sshoptions'"
# >= 0.4.3 : uses only scp and sftp, --ssh-command option is not supported anymore
else
execstr="$execstr --scp-command 'scp $scpoptions' --sftp-command 'sftp $sftpoptions' "
execstr_options="${execstr_options} --scp-command 'scp $scpoptions' --sftp-command 'sftp $sftpoptions'"
fi
fi
# deal with symmetric or asymmetric (public/private key pair) encryption
### Symmetric or asymmetric (public/private key pair) encryption
if [ -n "$encryptkey" ]; then
execstr="${execstr}--encrypt-key $encryptkey "
debug "Data will be encrypted with the GnuPG key $encryptkey."
execstr_options="${execstr_options} --encrypt-key $encryptkey"
debug "Data will be encrypted with the GnuPG key $encryptkey."
else
debug "Data will be encrypted using symmetric encryption."
debug "Data will be encrypted using symmetric encryption."
fi
# deal with data signing
### Data signing (or not)
if [ "$sign" == yes ]; then
# duplicity is not able to sign data when using symmetric encryption
[ -n "$encryptkey" ] || fatal "The encryptkey option must be set when signing."
# if needed, initialize signkey to a value that is not empty (checked above)
[ -n "$signkey" ] || signkey="$encryptkey"
execstr="${execstr}--sign-key $signkey "
debug "Data will be signed will the GnuPG key $signkey."
# duplicity is not able to sign data when using symmetric encryption
[ -n "$encryptkey" ] || fatal "The encryptkey option must be set when signing."
# if needed, initialize signkey to a value that is not empty (checked above)
[ -n "$signkey" ] || signkey="$encryptkey"
execstr_options="${execstr_options} --sign-key $signkey"
debug "Data will be signed will the GnuPG key $signkey."
else
debug "Data won't be signed."
fi
# deal with GnuPG passphrase
[ -n "$password" ] || fatal "The password option must be set."
if [ "$keep" != "yes" ]; then
if [ "`echo $keep | tr -d 0-9`" == "" ]; then
keep="${keep}D"
fi
execstr="${execstr}--remove-older-than $keep "
debug "Data won't be signed."
fi
### Incremental or full backup mode
# If incremental==yes, use the default duplicity behaviour: perform an
# incremental backup if old signatures can be found, else switch to
# full backup.
# If incremental==no, force a full backup anyway.
if [ "$incremental" == "no" ]; then
execstr="${execstr}--full "
# before 0.4.4, full was an option and not a command
if [ "$duplicity_major" -le 0 -a "$duplicity_minor" -le 4 -a "$duplicity_sub" -lt 4 ]; then
execstr_options="${execstr_options} --full"
else
execstr_command="full"
fi
fi
execstr_serverpart="scp://$destuser@$desthost/$destdir"
execstr_clientpart="/"
### SOURCE ###
set -o noglob
# excludes
for i in $exclude; do
str="${i//__star__/*}"
execstr="${execstr}--exclude '$str' "
done
# includes
for i in $include; do
[ "$i" != "/" ] || fatal "Sorry, you cannot use 'include = /'"
str="${i//__star__/*}"
execstr="${execstr}--include '$str' "
done
# vsincludes
if [ $usevserver = yes ]; then
for vserver in $vsnames; do
for vi in $vsinclude; do
str="${vi//__star__/*}"
str="$VROOTDIR/$vserver$str"
execstr="${execstr}--include '$str' "
done
done
fi
set +o noglob
### deal with tmpdir ###
### Temporary directory
precmd=
if [ -n "$tmpdir" ]; then
if [ ! -d "$tmpdir" ]; then
@ -184,24 +156,108 @@ if [ -n "$tmpdir" ]; then
precmd="${precmd}TMPDIR=$tmpdir "
fi
### EXECUTE ###
### Cleanup old backup sets (or not)
if [ "$keep" != "yes" ]; then
if [ "`echo $keep | tr -d 0-9`" == "" ]; then
keep="${keep}D"
fi
# before 0.4.4, remove-older-than was an option and not a command
if [ "$duplicity_major" -le 0 -a "$duplicity_minor" -le 4 -a "$duplicity_sub" -lt 4 ]; then
execstr_options="${execstr_options} --remove-older-than $keep"
fi
fi
execstr=${execstr//\\*/\\\\\\*}
### Source
debug "$precmd duplicity $execstr --exclude '**' / $execstr_serverpart"
set -o noglob
# excludes
for i in $exclude; do
str="${i//__star__/*}"
execstr_source="${execstr_source} --exclude '$str'"
done
# includes
for i in $include; do
[ "$i" != "/" ] || fatal "Sorry, you cannot use 'include = /'"
str="${i//__star__/*}"
execstr_source="${execstr_source} --include '$str'"
done
# vsincludes
if [ $usevserver = yes ]; then
for vserver in $vsnames; do
for vi in $vsinclude; do
str="${vi//__star__/*}"
str="$VROOTDIR/$vserver$str"
execstr_source="${execstr_source} --include '$str'"
done
done
fi
set +o noglob
### EXECUTE ####################################################################
execstr_source=${execstr_source//\\*/\\\\\\*}
### Cleanup commands (duplicity >= 0.4.4)
# cleanup
if [ "$duplicity_major" -ge 0 -a "$duplicity_minor" -ge 4 -a "$duplicity_sub" -ge 4 ]; then
debug "$precmd duplicity cleanup $execstr_options $execstr_serverpart"
if [ ! $test ]; then
export PASSPHRASE=$password
output=`nice -n $nicelevel \
su -c \
"$precmd duplicity cleanup $execstr_options $execstr_serverpart 2>&1"`
exit_code=$?
if [ $exit_code -eq 0 ]; then
debug $output
info "Duplicity cleanup finished successfully."
else
debug $output
warning "Duplicity cleanup failed."
fi
fi
fi
# remove-older-than
if [ "$keep" != "yes" ]; then
if [ "$duplicity_major" -ge 0 -a "$duplicity_minor" -ge 4 -a "$duplicity_sub" -ge 4 ]; then
debug "$precmd duplicity remove-older-than $keep $execstr_options $execstr_serverpart"
if [ ! $test ]; then
export PASSPHRASE=$password
output=`nice -n $nicelevel \
su -c \
"$precmd duplicity remove-older-than $keep $execstr_options $execstr_serverpart 2>&1"`
exit_code=$?
if [ $exit_code -eq 0 ]; then
debug $output
info "Duplicity remove-older-than finished successfully."
else
debug $output
warning "Duplicity remove-older-than failed."
fi
fi
fi
fi
### Backup command
debug "$precmd duplicity $execstr_command $execstr_options $execstr_source --exclude '**' / $execstr_serverpart"
if [ ! $test ]; then
export PASSPHRASE=$password
output=`nice -n $nicelevel \
su -c \
"$precmd duplicity $execstr --exclude '**' / $execstr_serverpart 2>&1"`
code=$?
if [ $code -eq 0 ]; then
debug $output
info "Duplicity finished successfully."
else
debug $output
fatal "Duplicity failed."
fi
fi
export PASSPHRASE=$password
output=`nice -n $nicelevel \
su -c \
"$precmd duplicity $execstr_command $execstr_options $execstr_source --exclude '**' / $execstr_serverpart 2>&1"`
exit_code=$?
if [ $exit_code -eq 0 ]; then
debug $output
info "Duplicity finished successfully."
else
debug $output
fatal "Duplicity failed."
fi
fi
return 0