I’ve seen some people upset about the fact that scp has been deprecated. If you’ve never used scp, it basically tries to be a version of cp that works between computers, via an SSH connection. I haven’t used it in a long, long time, because I use rsync instead. While there will eventually be something that works like scp but using the SFTP protocol, I’d like to suggest that the far superior option is to learn rsync and switch to that, as I did a long time ago.

Here are some key benefits of rsync:

  • By default, it only transfers files it needs to, and only transfers the parts of those files that aren’t already present at the destination. When dealing with things like huge log files and directories with thousands of files, this can save a lot of time.
  • If it fails part way through because of a network error, you can just run it again, and it will pick up where it left off.
  • It can verify that files are byte-for-byte identical after transfer.
  • It will preserve permissions, timestamps, and other file metadata.
  • It transfers all the data securely via SSH, just like scp.

However, the rsync manual page is extremely long and has a ridiculous number of options listed, and I think a lot of people look at it and decide they don’t have time for all that. So, here’s my summary of everything you need to know to handle 99% of day-to-day file transfer tasks using rsync.

Move one file from here to there

Here’s an example incantation:

rsync -a path/to/file example.com:remote/path/to/file

The -a option is a shortcut for a bunch of individual options that you almost certainly want to ensure that the remote file ends up the same as the local file in terms of permissions, timestamps and ownership.

You might guess that you can add a -v flag to get verbose output, and you’d be right. Almost all my usage of rsync starts with rsync -av. (You can also do rsync -a -v of course.)

Either path can start with / if you want to specify an absolute path from the root directory. Otherwise, the local path starts from the current directory, and the remote path starts from your SSH login directory, as you’d expect:

rsync -a /home/steve/log.txt example.com:/tmp/steve-log.txt

Move one file from there to here

Exactly the same, but with the source and destination in the opposite order:

rsync -av example.com:Music/startup.mp3 ~/Music/start-me-up.mp3

This general principle works for all these examples, you can flip source and destination to move data the other way.

Move any number of files into a remote directory

Now comes the part that trips people up. The rsync program will also work on directories, but to get the results you probably expect, you’ll want to remember to put a trailing / on any path that points at a directory. For example:

rsync -av *.jpeg somemachine.local:/tmp/

Going the other way, you’ll need to use some quoting so that your shell doesn’t try to expand the wildcard:

rsync -avs "remote.example.com:Pictures/*.jpg" Pictures/

Copy everything from one directory to another

This is why I love rsync. I can do this:

rsync -av Software/ newbox.local:Software/

Everything from the source directory will be transferred to the destination directory. Note the destination slash symbols on both directory paths. (The fish shell autocompletes the directories with the trailing slashes, by the way.)

If there are already files in the destination which aren’t in the source, they’ll be left alone. If you don’t want this, and want the destination to end up exactly like the source, the --delete flag will do the trick:

rsync -av --delete Software/ newbox.local:Software/

After this, the destination will hold exactly the same contents as the source.

SSH tweaks

You might wonder how to handle it if you need to specify a username for the SSH connection. The answer is you can specify the SSH command to use via -e. For example:

rsync -av -e 'ssh -l myusername' local/*.jpg remote.example.com:Photos/

If you use that remote system regularly, though, it’s probably easier to create an entry in your ~/.ssh/config file so you don’t need to do that. You can also specify a port and IP address override the same way:

Host remote.example.com
    HostName 10.34.1.40
    User myusername
    Port 663

Additional tips

If you know that rsync is going to need to transfer entire files because they’re totally different from what’s in the destination, and if the computers and disks involved in the transfer are slow, you can save time by telling it that. The -W option (or --whole-file) says to copy whole files rather than using binary diffing to find the changes.

For the opposite situation, where the network is slow and the computers are fast, the -z option will compress all the data in transit and decompress it again at the far end.

By default, rsync uses a temporary file on the destination system, and then renames it into place when it’s fully transferred. This is to try to prevent other programs from seeing incomplete files, but if you don’t need that feature you can tell rsync to just go ahead and use the final filename rather than a temporary file, using the --inplace flag.

If the destination path has spaces in, the shell on the destination system can mess up the transfer. To avoid this, use the -s option to bypass the shell on the destination system. (This feature is missing from the default version of rsync supplied with macOS.)

If you add the --dry-run flag (-n), rsync won’t actually transfer or delete anything, it’ll just tell you what it would do.

If you don’t need all the details from -v but do want to know what it did, the -i or --itemize-changes option will make it print a concise list of updates.

Not using SSH at all

If you omit any hostname from the rsync command, no SSH connection is used. Instead, rsync works as a local copy command. This means you can use it for directory synchronization between local disks, for example making backups:

rsync -av ~/Photos/ /mnt/backup/Photos/

You can also use it with --dry-run to compare two directories and display which files are different.

And more

As I said at the start, rsync has a large number of options. Once you’ve mastered the basics, you’ll probably find that for any weird file transfer problem you have, there’s an rsync option that can help.

For example, one time I had a bunch of files on an MS-DOS FAT filesystem on a very slow USB device. I wanted to copy on some more and merge them into place, but I really wanted to avoid unnecessary data transfer. The solution involved telling rsync to trust that files were identical if their timestamps and sizes were identical, and to compare timestamps with 2 second resolution to allow for lack of accuracy on FAT filesystems.