diff --git a/README.md b/README.md index 2636b38..5f37160 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ The script can be run without the second part but the server will quickly fill i
  • Gzip
  • Axel
  • Parallel
  • +
  • Rsync or WinSCP
  • Pre-Setup Steps: @@ -27,8 +28,18 @@ Run "prerequisites.sh" to setup the stack. 1. Run url_extractor.py against the directory of txt.gz files you are wanting to process. The script will ask for you to enter the location of the files, where you want to store the output and the concurrency to run the script at. 2. Once the script has completed running, verify the output of the file. +Data Transfer: + +There are two methods of transferring data from the server running the scripts back to a local machine being rsync or FTP. + +From testing I have found that while rsync is robust and reliable it is quite slow and can only move a single file at a time. When running multiple instances at a time it seems that no additional bandwidth is consumed and instead the files fight each other for bandwidth. + +I have developed a powershell script that uses WinSCP and the FTP protocol. Running multiple instances of the script significantly speeds up the data transfer rate. I strongly recommend using this method to download files from the server. + Notes: \ No newline at end of file