ArchiveTeam/Migrated_Keyword_URL_Extractor
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
datechnoman 2023-12-12 12:09:37 +00:00
parent bc9e5971a6
commit 4930a9418d
1 changed files with 20 additions and 20 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
README.md
View File

@ -1,21 +1,21 @@
<b>Overview:</b>
Use the following scripts to extract urls from .txt.gz files and output to a txt file.
<b>Requirements:</b>
<ul>
<li>Python3</li>
</ul>
<b>Steps:</b>
1. Git Clone the Repository.
2. Run url_extractor.py against the directory of txt.gz files you are wanting to process. The script will ask for you to enter the location of the files, where you want to store the output and the concurrency to run the script at.
3. Once the script has completed running, verify the output of the file.
<b>Notes:</b>
<ul>
<li>The script is hardset to stream the txt files from the location and process it line by line migitating the need for large files to be loaded into RAM.</li>
<li>Running the script over the network is fine and performance appears to not be impacted. When running against CommonCrawl WAT files it has been identified that a 1Gbit link can be saturated with 12 concurrent processes on an i7. The CPU still has additional capacity.</li>
<b>Overview:</b>
Use the following scripts to extract urls from .txt.gz files and output to a txt file.
<b>Requirements:</b>
<ul>
<li>Python3</li>
</ul>
<b>Steps:</b>
1. Git Clone the Repository.
2. Run url_extractor.py against the directory of txt.gz files you are wanting to process. The script will ask for you to enter the location of the files, where you want to store the output, the keyword/url you are wanting to export and the concurrency to run the script at.
3. Once the script has completed running, verify the output of the file.
<b>Notes:</b>
<ul>
<li>The script is hardset to stream the txt files from the location and process it line by line migitating the need for large files to be loaded into RAM.</li>
<li>Running the script over the network is fine and performance appears to not be impacted. When running against CommonCrawl WAT files it has been identified that a 1Gbit link can be saturated with 12 concurrent processes on an i7. The CPU still has additional capacity.</li>
</ul>