Using scp to transfer a .txt file list of files
I have 1.6 million files on server A, and only about 20k of them need to get to server B. The destination, server B, is on GoDaddy shared hosting, so I'm just about limited to scp for transferring many files at once.
I'd like to generate a .txt file of those 20k+ files from an SQL query, then feed that list into scp. Are there any options to do so?
cat /proc/version gives me Linux version 2.6.32-531.23.3.lve1.2.65.el6.x86_64 () (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Tue Aug 19 10:37:27 EDT 2014
5 Answers
if you have the text file already created you do the following
cat /location/file.txt | xargs -i scp {} user@server:/locationthis will go line by line of the output of the file.txt and run the scp comamand per line; I hope this helps.
2Consider rsync as an alternative, which has ssh support built-in and supports reading from a list of files using --files-from.
Example:
rsync -av --progress --partial-dir=/tmp --files-from=file_list.txt . :/destAdvantages over scp:
- Avoid cat/xargs and command line length limit, so only enter password once
- Resume progress after being interrupted
- Delta transfer algorithm avoids copying files that already exist
- Generally more robust and featureful
- Further reading: How does
scpdiffer fromrsync?
See also:
I wanted to do the same for list of folders and none of the answers using xargs worked for me. In the end I solved this simply with:
scp -r user@server:/location/{"$(cat folder_list.txt | tr '\n' ',' | sed 's/,$//')"} .Where I pass my list of files to shell expansion. I think this may be helpful to anyone looking for alternative solution.
The answer provided works great. I used xargs -a to send the file to standard out. I then combined that with the original answer of cat /location/file.txt | xargs -i scp {} user@server:/location
My Answer:
xargs -a /location/file.txt | xargs -i scp {} user@server:/locationThe above answers require you to authenticate for every file transferred. SCP runs on top of SSH and GoDaddy provides access to other commands over SSH such as tar. As such, the best method is to use tar with ssh via:
tar -T <file> -cf - | ssh <user>@<host> "tar -C <dir> -xvf -"If the list of files is on STDIN you can do the following instead: Note: cat is used here for generating STDIN only
cat <file> | tar -T - -cf - | ssh <user>@<host> "tar -C <dir> -xvf -"For reduced transfer time at the cost of additional processing time, you can add -z or -j to both tar commands to enable gzip or bzip compression respectively.
