15 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
datechnoman	51c127601a	Removing the tar process and now deleting cdx.json files older than 24 hours	2024-01-09 05:45:25 +00:00
datechnoman	49a9bc4710	Add pastebin_automated_cdx_processor.py	2024-01-09 05:15:31 +00:00
datechnoman	457f32ed61	Added tar support for files older than 24 hours to keep cdx file count low	2024-01-09 00:19:22 +00:00
datechnoman	f5928594a5	Changed time delta to 1 day to minimize cdx file count	2024-01-08 23:50:17 +00:00
datechnoman	85319ba390	Updated code to move files to /opt/cdxfiles/mediafire	2024-01-08 11:09:35 +00:00
datechnoman	42be737ee6	Code fix	2024-01-08 11:05:07 +00:00
datechnoman	2bf2e02275	Update mediafire_automated_cdx_processor.py	2024-01-08 10:44:33 +00:00
datechnoman	d56d7bb81f	Added in support to compare already existing files to skip them being reprocessed	2024-01-08 10:37:24 +00:00
datechnoman	61f5a4ddbf	Update concurrency of downloads	2024-01-08 08:27:15 +00:00
datechnoman	2e938624f9	Add mediafire_automated_cdx_processor.py	2024-01-08 08:23:28 +00:00
datechnoman	b2a9c44c9e	Added in a cdx filesize count statistic for the Grafana Dashboard to report on sizing	2023-12-27 08:58:44 +00:00
datechnoman	2dddda28bf	Add into a "cdxcount": 1, count for metric reporting of processed CDX files	2023-12-27 08:52:23 +00:00
datechnoman	182c58f1ce	Implemented subprocess for running multiple json extractions at once	2023-12-27 08:46:36 +00:00
datechnoman	8496391064	Update to remove tophosts output	2023-12-26 11:59:44 +00:00
datechnoman	ae515fb425	Upload files to "/"	2023-12-23 04:39:00 +00:00