From 65757f8cc464ac998144a1f312314ddce7928a82 Mon Sep 17 00:00:00 2001
From: datechnoman <datechnoman@hotmail.com>
Date: Tue, 12 Dec 2023 11:04:52 +0000
Subject: [PATCH] Update README.md

---
 README.md | 66 +++++++++++++++++++++++++++----------------------------
 1 file changed, 33 insertions(+), 33 deletions(-)

diff --git a/README.md b/README.md
index bd4832b..2636b38 100644
--- a/README.md
+++ b/README.md
@@ -1,34 +1,34 @@
-<b>Overview:</b>
-
-Use the following set of scripts to extract urls from CommonCrawl WAT files and output to a compressed txt file. 
-
-This script stack compromises of two processes running concurrently (One on the URL Extractor Server downloading, processing and zipping the results and the other stack pulling the files via rsync to a destination of your choice.)
-
-The script can be run without the second part but the server will quickly fill if large data sets are being processed (eg; CommonCrawl WARC/WAT Files). 
-
-<b>Requirements:</b>
-<ul>
-  <li>Python3</li>
-  <li>Gzip</li>
-  <li>Axel</li>
-  <li>Parallel</li>
-</ul>
-
-<b>Pre-Setup Steps:</b>
-
-Before running the scripts, there are some steps required to setup the stack. 
-
-A script has been devised to automate the steps.
-
-Run "prerequisites.sh" to setup the stack. 
- 
-<b>Steps:</b>
-
-1. Run url_extractor.py against the directory of txt.gz files you are wanting to process. The script will ask for you to enter the location of the files, where you want to store the output and the concurrency to run the script at.
-2. Once the script has completed running, verify the output of the file.
-
-<b>Notes:</b>
-
-<ul>
-  <li>None at this time.</li>
+<b>Overview:</b>
+
+Use the following set of scripts to extract urls from CommonCrawl WAT files and output to a compressed txt file. 
+
+This script stack compromises of two processes running concurrently (One on the URL Extractor Server downloading, processing and zipping the results and the other stack pulling the files via rsync to a destination of your choice.)
+
+The script can be run without the second part but the server will quickly fill if large data sets are being processed (eg; CommonCrawl WARC/WAT Files). 
+
+<b>Requirements:</b>
+<ul>
+  <li>Python3</li>
+  <li>Gzip</li>
+  <li>Axel</li>
+  <li>Parallel</li>
+</ul>
+
+<b>Pre-Setup Steps:</b>
+
+Before running the scripts, there are some steps required to setup the stack. 
+
+A script has been devised to automate the steps.
+
+Run "prerequisites.sh" to setup the stack. 
+ 
+<b>Steps:</b>
+
+1. Run url_extractor.py against the directory of txt.gz files you are wanting to process. The script will ask for you to enter the location of the files, where you want to store the output and the concurrency to run the script at.
+2. Once the script has completed running, verify the output of the file.
+
+<b>Notes:</b>
+
+<ul>
+  <li>A text file with all of the CommonCrawl links named "urls_to_download.txt" is required to be located in /opt/CommonCrawl_URL_Processor.</li>
 </ul>
\ No newline at end of file