ArchiveTeam/CommonCrawl_URL_Processor
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update urlextractor_archiveteam.sh

Browse Source
This commit is contained in:
datechnoman 2023-12-05 10:20:47 +00:00
parent 805b133dba
commit 0d5a54df43
1 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
urlextractor_archiveteam.sh
View File

@ -21,4 +21,7 @@ for file in "$directory"/*_urls.txt; do
done done
# Wait for all gzip processes to finish # Wait for all gzip processes to finish
wait wait
# Move compressed files to /opt/commoncrawl
mv "$directory"/*.gz /opt/commoncrawl/