21 Commits

Author	SHA1	Message	Date
datechnoman	7c157c5a48	Update warc_wat_url_processor.py	2024-03-31 11:45:33 +00:00
datechnoman	3151fce353	Update warc_wat_url_processor.py	2024-02-08 00:39:09 +00:00
datechnoman	486a68a796	Removed multithread compression and added force overwrite for compression files	2024-01-28 22:50:09 +00:00
datechnoman	ebc07a6974	Update zstd to overwrite conflicts	2024-01-28 11:39:00 +00:00
datechnoman	29d24e9826	Reverting	2024-01-28 11:36:17 +00:00
datechnoman	6d591ef0d0	Added in error logging	2024-01-28 11:29:53 +00:00
datechnoman	54747b64f6	Update warc_wat_url_processor.py	2024-01-28 11:23:27 +00:00
datechnoman	b6a9c68140	Update warc_wat_url_processor.py	2024-01-28 09:14:12 +00:00
datechnoman	d0fa7c84f4	Update warc_wat_url_processor.py	2024-01-28 09:10:17 +00:00
datechnoman	4f46c841b3	Updated to include disk space checking	2024-01-27 03:02:54 +00:00
datechnoman	8244b9241a	Update concurrency	2024-01-26 08:01:39 +00:00
datechnoman	5661ce44b2	Updated loop issue	2024-01-26 07:14:25 +00:00
datechnoman	2bee6db85a	Updated error processing for zstd compression errors	2024-01-25 01:33:11 +00:00
datechnoman	295c3daba4	Update warc_wat_url_processor.py	2024-01-23 10:45:32 +00:00
datechnoman	6edffba451	Update warc_wat_url_processor.py	2024-01-23 04:43:15 +00:00
datechnoman	513b32e80a	Update warc_wat_url_processor.py	2024-01-20 12:33:28 +00:00
datechnoman	06e3399861	Update warc_wat_url_processor.py	2024-01-20 12:28:23 +00:00
datechnoman	e98f80aec4	Updated to use command line zstd	2024-01-20 11:29:57 +00:00
datechnoman	78f6b69cdf	Updated with higher compression	2024-01-20 11:25:59 +00:00
datechnoman	17b4ce6077	Update warc_wat_url_processor.py	2024-01-20 03:26:03 +00:00
datechnoman	99c2f07498	Add warc_wat_url_processor.py	2024-01-20 03:25:50 +00:00