24 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
datechnoman	81bd092284	Update pastebin_automated_cdx_processor.py	2024-01-10 00:42:57 +00:00
datechnoman	0c062e7326	Update mediafire_automated_cdx_processor.py	2024-01-10 00:31:54 +00:00
datechnoman	d3d110789a	Updated bug in code for previous change	2024-01-09 22:48:26 +00:00
datechnoman	43d55a27fd	Set ia path variable for cron execution	2024-01-09 21:35:25 +00:00
datechnoman	6c16ee45bd	Updated cdx file retention to 2 days	2024-01-09 21:15:18 +00:00
datechnoman	1e44917569	Updated cdx file retention to 2 days	2024-01-09 21:14:23 +00:00
datechnoman	8e1912cb9b	Updated cdx file retention to 2 days	2024-01-09 21:13:48 +00:00
datechnoman	4d4dc9c81c	Add imgur_automated_cdx_processor.py	2024-01-09 05:51:41 +00:00
datechnoman	08dc0c0bf6	Removing the tar process and now deleting cdx.json files older than 24 hours	2024-01-09 05:49:56 +00:00
datechnoman	51c127601a	Removing the tar process and now deleting cdx.json files older than 24 hours	2024-01-09 05:45:25 +00:00
datechnoman	49a9bc4710	Add pastebin_automated_cdx_processor.py	2024-01-09 05:15:31 +00:00
datechnoman	457f32ed61	Added tar support for files older than 24 hours to keep cdx file count low	2024-01-09 00:19:22 +00:00
datechnoman	f5928594a5	Changed time delta to 1 day to minimize cdx file count	2024-01-08 23:50:17 +00:00
datechnoman	85319ba390	Updated code to move files to /opt/cdxfiles/mediafire	2024-01-08 11:09:35 +00:00
datechnoman	42be737ee6	Code fix	2024-01-08 11:05:07 +00:00
datechnoman	2bf2e02275	Update mediafire_automated_cdx_processor.py	2024-01-08 10:44:33 +00:00
datechnoman	d56d7bb81f	Added in support to compare already existing files to skip them being reprocessed	2024-01-08 10:37:24 +00:00
datechnoman	61f5a4ddbf	Update concurrency of downloads	2024-01-08 08:27:15 +00:00
datechnoman	2e938624f9	Add mediafire_automated_cdx_processor.py	2024-01-08 08:23:28 +00:00
datechnoman	b2a9c44c9e	Added in a cdx filesize count statistic for the Grafana Dashboard to report on sizing	2023-12-27 08:58:44 +00:00
datechnoman	2dddda28bf	Add into a "cdxcount": 1, count for metric reporting of processed CDX files	2023-12-27 08:52:23 +00:00
datechnoman	182c58f1ce	Implemented subprocess for running multiple json extractions at once	2023-12-27 08:46:36 +00:00
datechnoman	8496391064	Update to remove tophosts output	2023-12-26 11:59:44 +00:00
datechnoman	ae515fb425	Upload files to "/"	2023-12-23 04:39:00 +00:00