Using rsync
to Deploy Enhydra Web Sites
Bill Karwin, Lutris Technologies Engineer
You're developing a web site in a multi-server environment, with
separate servers for development, staging, and production. After
spending all morning trying to debug your Javascript rollover code,
you find that you forgot to copy the latest images from the development
server to the staging server. There must be a better solution. You
need a tool to help keep all your deployed files synchronized across
multiple servers.
Rsync is a command-line tool to help transfer collections
of files incrementally from one host to another. Rsync is an open
source software package (GNU Public License) available at http://rsync.samba.org/.
Below are some of the capabilities of rsync:
-
File collections:
Group together files, links, devices, and directories recursively.
Exclude specified files, or CVS-related files.
Preserve file ownership and permissions.
-
Copying collections:
Copy collections over a network to another host.
Don't transfer files that are identical on both hosts.
Compress data during transfer.
-
Security:
Support anonymous or authenticated connections.
Encrypt data during transfer using ssh.
Don't require root privileges to perform a transfer.
There are other ways to use rsync, for instance to copy files from
one location to another on the same machine, or to get a list of
files on a remote machine. I will describe ways that I have used
rsync in web site projects.
Syntax:
Below is an example of using rsync to copy an Apache htdocs
directory from a staging server to a production server.
rsync -avuzCb /usr/apache/htdocs webadmin@www.myfirm.com:/usr/apache/htdocs
You would execute this command within a shell login on your staging
server. I'll separate it so I can describe the elements.
rsync
|
The command rsync, assuming you have installed it.
|
-a
|
The option for "archive mode," which specifies recursive
copying and preservation of all file attributes.
|
-v
|
The option for verbose messages.
|
-u
|
The option for updating files, without overwriting newer
files.
|
-z
|
The option for data compression using zlib.
|
-C
|
The option for ignoring CVS-related files.
|
-b
|
The option for backing up old files that would be overwritten.
|
/usr/apache/htdocs
|
The source directory for the files.
|
webadmin@www.myfirm.com:/usr/apache/htdocs
|
The destination location for the files.
The name "webadmin" is a valid login ID on the remote host.
The name "www.myfirm.com" is an example of a production server
hostname.
The path of the destination directory is the same as the source
path in this example, but it doesn't have to be.
|
The example above uses the syntax "login@host:/path",
which indicates that the rsync client connects to the remote host
using standard rsh or ssh. Rsync uses rsh by default, but you can
set the environment variable RSYNC_RSH to "ssh", for example, to
use that mechanism for secure connections.
You can also configure a specialized rsync service on the remote
host if you need anonymous access, more control over authentication,
or an option to restrict rsync activity to predefined collections
of files. That's a topic for another article.
Enhydra Application Files
Using rsync to deploy Enhydra applications is a little different.
You have to shut down and restart a running instance of the application
if you replace its .jar file. You also need to restart the application
to take advantage of changes in configuration files.
-
Deploy application binaries and configuration files using
rsync.
rsync -avuzCb ./output/ webadmin@www.myfirm.com:/usr/apache/apps/
-
Open the Enhydra Multiserver Admin console interface for the
deployment server.
http://www.myfirm.com:8001/
-
Shut down the currently running instance of the application.
-
Start the application; this should use the new version of
your application and its config files.
-
Run regression tests and test the new application changes.
(This goes without saying!)
You should use the -b option to make a backup of files
on the destination server that are overwritten during the transfer
of new files. Backup files are given a suffix character "~".
This allows you to revert to the previous version of your application
(manually, by renaming the backup files and restarting the application)
after deploying an update.
Customizing Configuration Files
Your application configuration file might have host-specific entries,
such as a JDBC URL or a pathname to an error log file. If entries
in the configuration files differ between environments, you have
to maintain different edited files on each host. Usually, making
these edits is a manual task and keeping the files under source
control is complex.
To solve this, I keep several config files under source control,
named with host-specific suffixes. For example, I might name these
files myApp_dev.conf, myApp_stage.conf, and myApp_www.conf. On each
server, I create a symbolic link from myApp_<host>.conf
to myApp.conf:
ln -s myApp_stage.conf myApp.conf
The link myApp.conf therefore points to a different file on each
server, even though all three config files reside in the source
control database and on all three servers. When I deploy application
files, I exclude myApp.conf from the copying, to avoid overwriting
the myApp.conf link on each server. I use the --exclude=myApp.conf
option of rsync.
rsync -avuzCb --exclude myApp.conf ./output/ webadmin@www.myfirm.com:/usr/apache/apps/
You might also find it convenient to create a Makefile rule to
deploy the finished application to another server after building
it. For example:
dist: jar
@echo "Copying finished $(JARNAME) application to deployment server..."
rsync -avuzCb --exclude myApp.conf \
$(OUTPUT)/ webadmin@www.myfirm.com:/usr/apache/apps/$(JARNAME)/
Copying Files
Rsync is efficient at transferring files. First, it filters files
by comparing them with the contents of the destination locations.
If the destination copy of the file is newer or identical to the
local file, then rsync doesn't copy that file. Also, rsync compresses
file data before transferring it across the network. These steps
reduces overall transfer time when deploying. There are options
to override this behavior, but usually it's desirable. If
you want to watch the progress and get some statistics on the transfer,
use the options -v --progress --stats.
Security
In secure environments, rsh and NFS should be disabled outside
a firewall. Ssh is a secure method of connecting to a host. Rsync
can use the ssh protocol to establish an encrypted connection for
copying files. Set the RSYNC_RSH environment variable to "ssh".
You can set the environment variables USER and RSYNC_PASSWORD to
automate authentication, but it's not recommended to put a plaintext
password in your environment.
Availability
Rsync is open source (GPL), and you can download the source from
http://rsync.samba.org/rsync/download.html.
Volunteers have contributed binaries for selected versions of Linux
i386/PowerPC, Solaris SPARC/x86, BSDi, AIX, HP-UX, SCO, and SunOS.
If you use another platform, you have to build rsync using a C compiler.
Rsync works best in most UNIX/Linux environments, and it reportedly
can work in Microsoft Windows 9x/NT environments too.
I now view rsync as a valuable tool to help automate copying collections
of files from development or staging environments to production
servers. I can relieve myself of many repetitive details of deployment,
reduce the risk of mistakes, and perform site updates more quickly.
SEE ALSO: By coincedence, this month in Webtechniques (November
2000) Bryant Durrell contributes an article about using rsync for
web content distribution. Great minds think alike.
http://www.webtechniques.com/archives/2000/11/durrell/
|