Commit Graph

20 Commits

Author SHA1 Message Date
3151fce353 Update warc_wat_url_processor.py 2024-02-08 00:39:09 +00:00
486a68a796 Removed multithread compression and added force overwrite for compression files 2024-01-28 22:50:09 +00:00
ebc07a6974 Update zstd to overwrite conflicts 2024-01-28 11:39:00 +00:00
29d24e9826 Reverting 2024-01-28 11:36:17 +00:00
6d591ef0d0 Added in error logging 2024-01-28 11:29:53 +00:00
54747b64f6 Update warc_wat_url_processor.py 2024-01-28 11:23:27 +00:00
b6a9c68140 Update warc_wat_url_processor.py 2024-01-28 09:14:12 +00:00
d0fa7c84f4 Update warc_wat_url_processor.py 2024-01-28 09:10:17 +00:00
4f46c841b3 Updated to include disk space checking 2024-01-27 03:02:54 +00:00
8244b9241a Update concurrency 2024-01-26 08:01:39 +00:00
5661ce44b2 Updated loop issue 2024-01-26 07:14:25 +00:00
2bee6db85a Updated error processing for zstd compression errors 2024-01-25 01:33:11 +00:00
295c3daba4 Update warc_wat_url_processor.py 2024-01-23 10:45:32 +00:00
6edffba451 Update warc_wat_url_processor.py 2024-01-23 04:43:15 +00:00
513b32e80a Update warc_wat_url_processor.py 2024-01-20 12:33:28 +00:00
06e3399861 Update warc_wat_url_processor.py 2024-01-20 12:28:23 +00:00
e98f80aec4 Updated to use command line zstd 2024-01-20 11:29:57 +00:00
78f6b69cdf Updated with higher compression 2024-01-20 11:25:59 +00:00
17b4ce6077 Update warc_wat_url_processor.py 2024-01-20 03:26:03 +00:00
99c2f07498 Add warc_wat_url_processor.py 2024-01-20 03:25:50 +00:00