Rsnapshot
by Carl Fitch
Info
Rsnapshot is a script to automate backups using rsync. The script attempts to automate the backup process
This knowledge was gleaned for the most part from:
Using Rsnapshot and SSH (http://troy.jdmz.net/rsnapshot/)
Using Rsync and SSH (http://troy.jdmz.net/rsync/index.html)
rsnapshot /home (http://www.rsnapshot.org/)
How to
Backup Server - core
Create the key pair directory
The first thing is to generate a public/private key pair specifically for the machine to back up. We will generate a different key pair for each machine we will connect to and name it with the target machines hostname. In this particular case we will be setting up backups for vnc.omnitec.net so I named the file vnc_key.
Log on to the backup server to start the process. Here we will create the public/private key pair to connect to the machine to back up.
1. We will su to root and create the .cron directory to keep the keys in. The .cron directory could be named whatever you wish
su - mkdir /root/.cron chmod 700 /root/.cron cd /root/.cron
Generate ssh key pair
2. Here we create the dsa key pair for vnc.omnitec.net. There is a lot of material on how to do this on the web. Here I only have how I have created it. When asked for a passphrase I just hit enter to bypass having a passphrase. This should be considered a security risk, however the key will only be used on one specific machine and only with a specified forced-command.
ssh-keygen -t dsa -b 1024 -f /root/.cron/vnc_key
This creates two files, a private key that remains on the server and the public key which is placed in the remote system in /root/.ssh/authorized_keys. In this example those keys are named vnc_key which is the private key and vnc_key.pub is the public key.
3. Verify that the permissions are set correctly:
-bash-3.1# ls -al total 10 drwxrwxr-x 2 root wheel 512 Aug 16 18:14 . drwx------ 12 root wheel 1536 Aug 16 17:42 .. -rw------- 1 root wheel 672 Aug 16 18:13 vnc_key -rw-r--r-- 1 root wheel 611 Aug 16 18:13 vnc_key.pub
4. vnc_key is the private key, keep it secure by setting permissions to 600.
chmod 600 /root/.cron/vnc_key
vnc_key.pub, the public key will be placed on vnc.omnitec.net
Create root's config file
The config file is a custom configuration file for a user used to override the system wide settings in /etc/ssh/ssh_config for the ssh program. Note that this is only for ssh (client) not sshd (server). The path by default is ~/.ssh/config
Create config
The file config will need to be created if it does not exist
cd /root/.ssh touch config
Contents of config
Each host you need to back up will have it's own section. The hosts section starts with the hash pair
Host hostname
as output by the hostname command. In practice this will require the FQDN as it has to match the command line passed to ssh when connecting to the remote machine.
There are three hash's specific to each host.
Host vnc.omnitec.net #FQDN hostname Hostname vnc.omnitec.net or Hostname 192.168.1.251 IdentityFile /root/.cron/vnc_key #'The path to and name of the private key file
The resulting stanza in /root/.ssh/config would be:
# ## # Host vnc.omnitec.net Hostname 192.168.1.251 IdentiyFile /root/.cron/vnc_key
Make sure config is secure:
chmod 600 /root/.ssh/config
Check permissions:
ls -l /root/.ssh/config -rw------- 1 root wheel 74 Aug 16 18:17 /root/.ssh/config
Machine to be backed up
Adding To known_hosts
By some means you need to get the public key generated in Generate_ssh_key_pair onto the remote machine. in this case vnc.omnitec.net. Since backups need to be done as root this should be added to /root/.ssh/authorized_keys. You can create another user and change their UID to 0 (zero) which will give them root's ability but with another name. We will stick with root during this session, finding another user with root's UID can be exciting at the least.
Modify sshd_config
One of the first things you want to check is /etc/ssh/sshd_config (note the "d").
We need to temporarily make ssh insecure by allowing root to login and allowing logins with passwords.
Check that password logins are allowed by setting PasswordAuthentication to yes.
Next, look for the line that starts with
PermitRootLogin
We will be modifying this later to forced-commands-only but for now set it to yes to test the configuration.
I replaced the current PermitRootLogin with these four lines so that it would be easy to modify during the next steps. Just comment out all of these lines except for the one needed at each step. At this point take the # off of PermitRootLogin yes
#PermitRootLogin no PermitRootLogin yes #PermitRootLogin without-password #PermitRootLogin forced-commands-only
Then restart sshd (as root) by looking for the pid for /usr/sbin/sshd and sending the HUP signal.
ps ax | grep sshd 958 ? Ss 0:00 /usr/sbin/sshd kill -HUP 958
On Redhat based systems you could use the service command
service ssh restart
Other systems may use this:
/etc/init.d/ssh restart
For now we are as far as we can go with the remote machine, we'll return to the back up server, install and configure the software "rsnapshot".
rsnapshot
Rsnapshot is a script to automate backups using rsync. It provides a simplified configuration and set up so you don't need to learn all the settings for rsync. It also creates snapshots of a machine.
install
Rsnapshot is a perl file so it should work just about everywhere. There are some dependicies that may need to be installed. In OpenBSD the package is available so installation was a simple.
pkg_add rsnapshot
On Debian it is just as simple:
apt-get install rsnapshot
config
Rsnapshot is controlled, by default, by the config file /etc/rsnapshot.conf. Since we will be running a specific config file for each host we will create a directory in /etc/ and move the config file to it.
mkdir /etc/rsnapshot mv /etc/rsnapshot.config /etc/rsnapshot/
There are several things that need to be customized in rsnapshot.conf. To preserve the original file I copied it.
cp rsnapshot.conf rsnapshot.conf.sample
Next open the file in your favorite editor and change or set the following values to the defaults you will need: (diff shown)
diff rsnapshot.conf.default rsnapshot.conf 27c27 < snapshot_root /.snapshots/ --- > snapshot_root /u/
The snapshots for this install will be relative to /u instead of the default /.snapshots
57c57 < #cmd_ssh /usr/bin/ssh --- > cmd_ssh /usr/bin/ssh
We will be using ssh, check location of ssh executable with which ssh, uncomment cmd_ssh and set the value to the result.
132c132,133 < #ssh_args -p 22 --- > ssh_args -p 2012
Add whatever ssh arguments needed here, in this instance the remote machine uses port 2012 for ssh.
Leave rsnapshot.conf as a base and copy it to a file named for the machine
cp rsnapshot.conf vnc_rsnapshot.conf
In the machine specific config files set snapshot_root to the correct path. For vnc.omnitec.net I wanted the backups in /u/VNC/ so I set:
snapshot_root /u/VNC/
exclude
Additionally there are several other config lines that can prove useful. If the machine has a proc filesystem such as Linux has, you can exclude that directory with
exclude /proc/
Several things to keep in mind:
- separate "exclude" and the value with a tab
- only one value per exclude. If you need to exclude more than one value, use multiple "exclude" statements
- "*.o" would exclude all filenames matching *.o
- "/foo" would exclude a file (or directory) named foo in the transfer-root directory
- "foo/" would exclude any directory named foo
- "/foo/*/bar" would exclude any file named bar which is at two levels below a directory named foo in the transfer-root directory
Always run configtest after any changes
backup line
This is where you set what machine and what directorys are backed up:
#keyword source destination rsync parameters(optional) backup root@vnc.omnitec.net:/ vnc/
I placed that under the section labeled:
############################### ### BACKUP POINTS / SCRIPTS ### ###############################
The rules for these entrys:
- All lines start with the backup keyword
- All parameters are seperated by tabs (not spaces)
- The second parameter is user@what_machine:/path
- Third parameter is what sub directory of /u (or /.snapshot) to put backups
- Fourth parameter if supplied is rsync rules such as exclude=path
lock file
Uncomment and change the the line:
lockfile /var/run/rsnapshot.pid
This will keep the program from stepping on itself if it doesn't finish before it is called again. If you have multiple machines each with their own config file, set the lockfile name to something unique such as:
lockfile /var/run/vnc.rsnapshot.pid
This way you can be running several backups but a second backup of a particular machine will not start.
Back to Backup Server
Test connection
When you have the key in the remote machine you should test that it works. This also gets the remote machines signature into the known_hosts file for root on the back up server. Try to connect. To verify it is working correctly, or see what is causing a problem, use -v in the ssh command line to print more details to the screen.
ssh -v root@vnc.omnitec.net
You should connect without problems. Look for this section, it tells you where and what key pair it used.
debug1: Reading configuration data /root/.ssh/config debug1: Applying options for vnc.omnitec.net debug1: Reading configuration data /etc/ssh/ssh_config debug1: Connecting to 192.168.1.251 [192.168.1.251] port 2106. debug1: Connection established. debug1: permanently_set_uid: 0/0 debug1: identity file /root/.cron/vnc_key type 2
Once the keys are in known_hosts here is an excerpt for a successful session.
debug1: Host '[192.168.1.251]:2106' is known and matches the RSA host key. debug1: Found key in /root/.ssh/known_hosts:21 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Next authentication method: publickey debug1: Offering public key: /root/.cron/vnc_key
testing
After the mix is together test it out with
rsnapshot -c /etc/rsnapshot/vnc_rsnapshot.conf configtest
It should return Syntax OK
Next try running a test with hourly
rsnapshot -t -c /etc/rsnapshot/{machine name}_rsnapshot.conf hourly
Which outputs:
rsnapshot -t -c /etc/rsnapshot/vnc_rsnapshot.conf hourly
mv /u/VNC/hourly.1/ /u/VNC/hourly.2/
mv /u/VNC/hourly.0/ /u/VNC/hourly.1/
/usr/local/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
--exclude=/proc/* --rsh=/usr/bin/ssh -p 2106 \
--link-dest=/u/VNC/hourly.1/vnc/ root@vnc.omnitec.net:/ \
/u/VNC/hourly.0/vnc/
touch /u/VNC/hourly.0/
Check that --link-dest is going to place the files where you want them.
first run
If all goes well try an actual hourly backup
rsnapshot -c /etc/rsnapshot/vnc_rsnapshot.conf hourly
First time through will take some time as it has to suck everything down so run it at a time when things are quiet. While this is running go to the remote machine and find the command line passed. Do this by running
ps axw | grep rsync
This is the command= line you will add to the authorized_keys file
command="rsync --server --sender -logDtprR --numeric-ids . /"
Once you have this line, and the backup has succeeded now go to #do_the_tighten_up
do the tighten up
If this is the first time setting up everything return to this section after succesfully setting up rsnapshot
Several changes need to be made to ssh configuration. Set these options only after all configurations are working and at least two backups have completed successfully.
While one of the backups are running you need to get the command line rsnapshot is sending when connecting. To do this run on the remote machine (vnc):
ps ax | grep rsync
Example: the command line sent is:
rsync --server --sender -logDtprR --numeric-ids . /
Take this line and add it to the begining of the key in /root/.ssh/authorized_keys. As an added step put a from line in also. The from line specifies that this key is only valid from that IP address.
Before:
ssh-dss AAAAB3NzaC1aseRuPziVRkq9URQ+oZ3nyQNGSQwaDpCfZaLvdJfNiH7w6yMaseRuPziVRkq9URQ+oZ3nyQNGSQwaDpCfZaLv ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ mkkKJID8EOgzEskWFMxumoHGH6PytMzjNBI0+8Fp9qecZgiNw root@core.omnitec.net
After:
from="192.168.1.251",command="rsync --server --sender -logDtprR --numeric-ids . /" ssh-dss AAAAB3NzaC1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ mkkKJID8EOgzEskWFMxumoHGH6PytMzjNBI0+8Fp9qecZgiNw root@core.omnitec.net
This will only allow root to connect from the specified IP address and then only the specified command would be run.
There are no spaces between keywords, "=", and paramters. The parameter is enclosed by double quote marks. And there is a space separating the directives from the key.
Back ups below /
If you alter the config to backup other than root remember that that path will be sent along with the command.
To see exactly what command is being sent put this in as the command= statement in authorized_keys
command="echo $SSH_ORIGINAL_COMMAND > /tmp/echo_cmd"
This will put the command sent into the file /tmp/echo_cmd. Adjust the location of the file as needed if working on a hosted machine if you don't have access to /tmp/.
Limit PermitRootLogin
In /etc/ssh/sshd_config remove the comment for the line PermitRootLogin forced-commands-only and comment out the other 'PermitRootLogin lines. This will require that only the command specified in authorized_keys can be run.
Cron and Intervals
Rsnapshot needs to be run on a regular schedule by cron. These lines are added to root's crontab.
# ## Added to start rsnapshot 0 */4 * * * /usr/local/bin/rsnapshot -c /etc/rsnapshot/vnc_rsnapshot.conf hourly 30 23 * * * /usr/local/bin/rsnapshot -c /etc/rsnapshot/vnc_rsnapshot.conf daily 30 01 * * 7 /usr/local/bin/rsnapshot -c /etc/rsnapshot/vnc_rsnapshot.conf weekly
Several things to consider here. The rsnapshot.conf contains the lines
######################################### # BACKUP INTERVALS # # Must be unique and in ascending order # # i.e. hourly, daily, weekly, etc. # ######################################### interval hourly 6 interval daily 7 interval weekly 4 #interval monthly 3
Each line has the keyword interval, the interval name such as hourly, then the number of those intervals to keep. The names are what is called as part of the invocation of rnapshot. The number of intervals to keep should match the number of times that interval is called by cron.
All backups in a set should be staggared so that they are not stepping on each other
With this scheme, the further you go back in time the less changes are saved. As a result yesterdays backup only has the filesystem as it was at the end of the day. Same thing for weekly, you only have a snapshot of the filesystem at the end of the week. There was a suggestion for high precission backup intervals, where you would eliminate all the intervals except hourly. Then set the interval value to some high number such as 720. With a cron job set to run every hour this would result in saving the last 30 days of filesystem state down to a resolution of one hour.
Logging
There are several options for logging. all of which are configured via the machine name.rsnapshot.conf file.
The default is to use syslog through the logger program this writes the output to /var/log/messages
Aug 17 20:08:45 core rsnapshot[10634]: WARNING: Some files and/or directories in root@mx1.omnitec.net:/ vanished during rsync operation Aug 17 20:08:46 core rsnapshot[855]: WARNING: /usr/local/bin/rsnapshot -c /etc/r snapshot/mx1_rsnapshot.conf hourly: completed, but with some warnings Aug 17 23:30:02 core rsnapshot[3833]: /usr/local/bin/rsnapshot -c /etc/rsnapshot /vnc_rsnapshot.conf daily: completed successfully Aug 17 23:35:01 core rsnapshot[12267]: /usr/local/bin/rsnapshot -c /etc/rsnapsho t/mx1_rsnapshot.conf daily: completed successfully Aug 18 00:01:22 core rsnapshot[12160]: /usr/local/bin/rsnapshot -c /etc/rsnapsho t/vnc_rsnapshot.conf hourly: completed successfully Aug 18 00:08:25 core rsnapshot[24290]: /usr/local/bin/rsnapshot -c /etc/rsnapsho t/mx1_rsnapshot.conf hourly: completed successfully
In the sample above we see that there was a problem to begin with. The next step in his case was to run the hourly at the command line. I added the -v switch, also could have used the -V switch for even more information.
/usr/local/bin/rsnapshot -v -c /etc/rsnapshot/vnc_rsnapshot.conf hourly
The solution in this case was to add an exclude line to /etc/rsnapshot/vnc_rsnapshot.conf
#exclude ??? exclude /proc/*
Another way to have debugged this problem would to have changed the logging output via the conf file. Look for this section
############################################ # GLOBAL OPTIONS # # All are optional, with sensible defaults # ############################################ # Verbose level, 1 through 5. # 1 Quiet Print fatal errors only # 2 Default Print errors and warnings only # 3 Verbose Show equivalent shell commands being executed # 4 Extra Verbose Show extra verbose information # 5 Debug mode Everything verbose 2 # Same as "verbose" above, but controls the amount of data sent to the # logfile, if one is being used. The default is 3. # loglevel 3 # If you enable this, data will be written to the file you specify. The # amount of data written is controlled by the "loglevel" parameter. # #logfile /var/log/rsnapshot