I have an rsync job that has been added to a crontab and when it's running, I can only check that there's a rsync PID and confirm with htop that it's eating up n amount of CPU and RAM.

What I'd like to do is to monitor what files are actually being rsync'ed in realtime...when I want to. FYI I haven't passed any verbose option to the command nor have I added some logging. I really just want to check what's being rsync'ed on demand.

Any idea how I could achieve that?

6 Answers

You can do (in POSIX shells, including bash):

strace -e open,openat $(ps -o lwp= -LC rsync | sed 's/^/-p/')

Or in zsh:

strace -e open,openat -p${^$(ps -o lwp= -LC rsync)}

Or in fish

strace -e open,openat -p(ps -o lwp= -LC rsync)

Or in rc/es:

strace -e open,openat -p`{ps -o lwp= -LC rsync}

To see what it's doing, or

lsof -ad3-999 -c rsync

to see what files it currently has opened.

3

The most simple solution would be redirecting the output of rsync to a logfile.

rsync -avz /something /somwhere >> ~/rsynclog

1

Another way you could is if you know the rough directory you'll be syncing (ie, we'll use the directory 'movies' for example) you can use a combination of lsof and grep:

lsof | grep rsync | grep movies

lsof will list your open files, pipes the output to grep to find any opened by rsync, pipes that output to grep to find the directory/file which is open.

Here are two ways

With screen:Attach screen session to your cron job:

screen rsync --progress src dst

this will allow you to re attach to the rsync jobb anytime you want to check what files its currently processing (just be sure to be the same user as the one that launched the rsync job) with

screen -x

With loggingadd logging to your rsync job:

rsync --log-file=/tmp/rsync-status.txt src dst

then follow the log in real time with:

 tail -f /tmp/rsync-status.txt

As Király István suggested, I am running

rsync -ravz /Users/jkirby/Music/iTunes/* .

which gives output like

Jeffs-MBP-2:2016-08-15 jkirby$ rsync -ravz /Users/jkirby/Music/iTunes/* .
building file list ... done
Temp File 1.tmp
Temp File.tmp

From that output I can see what directory is being copied.

In the case where rsync is copying a lot of large files slowly, I monitor that directory using watch like so. This way I can see the temp file that rsync creates and I can see the size growing on the file currently being copied.

watch -n1 "~/Music/iTunes"

Monitor running process under GNU/Linux

Monitor rsync transfert on both ends

By running Linux kernel, you could query /proc pseudo filesystem. There is no need to resort to strace or lsof, (could be useful on small configuration where this tools could even not be installed).

ps -C rsync fw PID TTY STAT TIME COMMAND 8645 ? Ss 0:00 /usr/bin/rsync --daemon --no-detach
13763 ? S 0:01 \_ /usr/bin/rsync --daemon --no-detach
13764 ? S 1:30 \_ /usr/bin/rsync --daemon --no-detach
ps -C rsync fwo pid,lstart,stat,cmd PID STARTED STAT CMD 8645 Sat Jul 10 14:34:12 2021 Ss /usr/bin/rsync --daemon --no-detach
13763 Sat Jul 10 15:51:23 2021 S \_ /usr/bin/rsync --daemon --no-detach
13764 Sat Jul 10 15:51:23 2021 S \_ /usr/bin/rsync --daemon --no-detach

The 1st process is the daemon, looking for start time show two process started by remote rsync.

So, looking for the last one:

mapfile -t allrsyncpid < <(ps -C rsync fho pid)
rsyncpid=$((${allrsyncpid[@]: -1}))

Then with this $rsyncpid variable:

ls -l /proc/$rsyncpid/fd
total 0
lr-x------ 1 root root 64 jui 10 15:51 0 -> /dev/null
lrwx------ 1 root root 64 jui 10 15:51 1 -> socket:[106279330]
lrwx------ 1 root root 64 jui 10 15:51 2 -> socket:[106279330]
lrwx------ 1 root root 64 jui 10 15:51 3 -> socket:[106279331]
lrwx------ 1 root root 64 jui 10 15:51 4 -> /path/to/some/file.IxOWxh
lrwx------ 1 root root 64 jui 10 15:51 5 -> socket:[106323027]
lrwx------ 1 root root 64 jui 10 15:51 6 -> socket:[106320690]

Then

ls -l $(readlink /proc/$rsyncpid/fd/4)
-rw------- 1 root root 41024 Jun 10 15:59 /path/to/other/file.ZIjEDk

If you repeat this, you could see wich file are growing and how.

while [ -d /proc/$rsyncpid ] ;do ls -l $(readlink /proc/$rsyncpid/fd/4) read -t .5 foo && break
done
-rw------- 1 root root 0 Jul 10 16:16 /path/to/some/file.nMM3Lz
-rw------- 1 root root 0 Jul 10 16:16 /path/to/other/file.H2b61i
-rw------- 1 root root 0 Jul 10 16:16 /path/to/some/bigfile.B0Xgg7
-rw------- 1 root root 2097152 Jul 10 16:16 /path/to/some/bigfile.B0Xgg7
-rw------- 1 root root 42359296 Jul 10 16:16 /path/to/some/bigfile.B0Xgg7

Until you press return or the end of rsync process.

Of course, as you are watching for /proc entries, created and deleted very quickly by another process, there are some chance ls is executed too late after readlink. So this kind of message is not harmful:

ls: cannot access '/path/to/some/file.fKSZss': No such file or directory

Sender side vs reciever side.

Previous ls command will let you see file grown, because file doesn't exist before.

On sender side. As files are read, there are another nice /proc subdir: /proc/PID/fdinfo. Entries in this dir hold information about current pointer:

ps -C rsync w PID TTY STAT TIME COMMAND
3762 pts/13 D+ 3:18 rsync -ax /path/to/source/. rsync://rsyncserv/target
ls -l /proc/13762/fd
total 0
lrwx------ 1 root root 64 jui 10 16:27 0 -> /dev/pts/13
lrwx------ 1 root root 64 jui 10 16:27 1 -> /dev/pts/13
lrwx------ 1 root root 64 jui 10 16:27 2 -> /dev/pts/13
lrwx------ 1 root root 64 jui 10 16:27 3 -> socket:[106321578]
lr-x------ 1 root root 64 jui 10 16:27 4 -> /path/to/some/file

Then

cat /proc/13762/fdinfo/4
pos: 2883584
flags: 0100000
mnt_id: 131

Where pos is current position in file, flag represent how this file is accessed (0100000 mean LARGEFILE, see asm-generic/fcntl.h in your source tree ;-) and mnt_id indicate which filesystem file is located on.

So:

rsyncpid=$(($(ps -C rsync ho pid)))
while [ -d /proc/$rsyncpid ] ;do read -r _ pos </proc/$rsyncpid/fdinfo/4 crtfile=$(readlink /proc/$rsyncpid/fd/4) filesize=$(stat -c %s "$crtfile") printf "Crtfile: %s (%d), crtpos: (%d)\n" "$crtfile" "$filesize" "$pos" read -t .5 foo && break
done
Crtfile: /path/to/some/bigfile.iso (17985276), crtpos: (2097152)
Crtfile: /path/to/some/bigfile.iso (17985276), crtpos: (6553600)
Crtfile: /path/to/some/bigfile.iso (17985276), crtpos: (11272192)
Crtfile: /path/to/some/bigfile.iso (17985276), crtpos: (15990784)

Same as before: until you press return or the end of rsync process.

Server vs client side.

In this sample. rsync command was initiated for an external system, to send/write datas to the server.

In case rsync command is used to read/recieve datas from server, the method stay sames but on the other side:

• To determine rsync PID

  • server side:

    mapfile -t allrsyncpid < <(ps -C rsync fho pid)
    rsyncpid=$((${allrsyncpid[@]: -1}))

  • client side:

    rsyncpid=$(($(ps -C rsync ho pid)))

• To follow in file progress

  • sender side:

     while [ -d /proc/$rsyncpid ] ;do read -r _ pos </proc/$rsyncpid/fdinfo/4 crtfile=$(readlink /proc/$rsyncpid/fd/4) filesize=$(stat -c %s "$crtfile") printf "Crtfile: %s (%d), crtpos: (%d)\n" "$crtfile" "$filesize" "$pos" read -t .5 foo && break done

  • reciever side:

     while [ -d /proc/$rsyncpid ] ;do ls -l $(readlink /proc/$rsyncpid/fd/4) read -t .5 foo && break done

And for other tools...

This work for any tool accessing files. You could watch for tar, sha1sum, gzip, cat, dd, ....

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy