ArchiveTeam/CommonCrawl_URL_Processor
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
datechnoman 2024-03-31 11:40:22 +00:00
parent c2d5ad43b8
commit 511629b012
1 changed files with 1 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
README.md
View File

@ -12,7 +12,7 @@ The script can be run without the second part but the server will quickly fill i
<li>Gzip</li>
<li>Axel</li>
<li>Parallel</li>
<li>Rsync or WinSCP</li>
<li>WinSCP</li>
<li>PowerShell</li>
<li>Zstd</li>
</ul>
@ -32,10 +32,6 @@ Run "prerequisites.sh" to setup the stack.
<b>Data Transfer:</b>
There are two methods of transferring data from the server running the scripts back to a local machine being rsync or FTP.
From testing I have found that while rsync is robust and reliable it is quite slow and can only move a single file at a time. When running multiple instances at a time it seems that no additional bandwidth is consumed and instead the files fight each other for bandwidth.
I have developed a powershell script that uses WinSCP and the FTP protocol. Running multiple instances of the script significantly speeds up the data transfer rate. I strongly recommend using this method to download files from the server.
<b>Notes:</b>