Scripting parallel FTP downloads (multiple connections) on Windows
I need a scripting solution to download several (more then 100) websites by FTP for backup purposes. For each website I want a scripting solution which supports 10 parallel connections to only one website.
- Each website has his own FTP address.
- Splitting the complete task in 10 different batches is ok but still there will be 1 connection to 1 server which cause very a very slow download. Create tars on the server is not a option and is out of this question :-)
I'm using Windows 8.1, 64-bit.
I tried Windows FTP but that supports only 1 parallel connections to 1 server.
Also WinSCP is not supporting more then 1 parallel connection to 1 FTP server command line mode. I tried several options and maybe I'm doing something wrong here???
download.bat > this file is started with admin rights
md n:\backupftp\ithaka.im\public_html
winscp.exe /script="ithaka.txt" ithaka.txt
open ftp://xxx:xxxxx@
lcd n:\backupftp\ithaka.im\public_html
get *The winscp.exe and winscp.com are portable executables in the same directory as the download.bat and ithaka.txt
As a result of this I don't see the GUI interface. Only a task in the task manager. If I turn on logging I see only 1 connection to 1 FTP server at a time. No parallel downloads from 1 FTP server.
If I change the download.bat into (notice that I change .exe to .com):
md n:\backupftp\ithaka.im\public_html
winscp.com /script="ithaka.txt" The result will be:
modules | 0 B | 0,0 KB/s | binary | 0%
php | 0 B | 0,0 KB/s | binary | 0%
php.test | 4 KB | 6,9 KB/s | binary | 100%
php.module | 7 KB | 6,6 KB/s | binary | 100%
php.install | 1 KB | 1,8 KB/s | binary | 100%
php.info | 274 B | 1,3 KB/s | binary | 100%
block | 0 B | 1,3 KB/s | binary | 0%etc etc etc.. these downloads are not parallel and not supported by WinSCP.
So:
- Do I something wrong with WINSCP
- Is there a FTP scripting solution which support this on Windows? I looked into several alternatives but they all have graphical user interfaces which I don't want..
1 Answer
WinSCP scripting does not support parallel transfers on its own.
But you can run multiple WinSCP scripts in parallel.
You just have to split the set of files/directories to batches.
There's actually a ready-made PowerShell script based on WinSCP .NET assembly that does this.
See Automating download in parallel connections over SFTP/FTP protocol.
You can run the script (say parallel_download.ps1) like:
powershell.exe -File parallel_download.ps1 -sessionUrl ftp://xxx:xxxxx@ ^ -remotePath /domains/ ^ -localPath n:\backupftp\ithaka.im\public_html\ ^ -batches 3The above command can be directly used in a batch file (.bat).
Or executed from a command-line, if merged into a single line with all ^ removed.
