Commit Graph

24 Commits

Author SHA1 Message Date
81bd092284 Update pastebin_automated_cdx_processor.py 2024-01-10 00:42:57 +00:00
0c062e7326 Update mediafire_automated_cdx_processor.py 2024-01-10 00:31:54 +00:00
d3d110789a Updated bug in code for previous change 2024-01-09 22:48:26 +00:00
43d55a27fd Set ia path variable for cron execution 2024-01-09 21:35:25 +00:00
6c16ee45bd Updated cdx file retention to 2 days 2024-01-09 21:15:18 +00:00
1e44917569 Updated cdx file retention to 2 days 2024-01-09 21:14:23 +00:00
8e1912cb9b Updated cdx file retention to 2 days 2024-01-09 21:13:48 +00:00
4d4dc9c81c Add imgur_automated_cdx_processor.py 2024-01-09 05:51:41 +00:00
08dc0c0bf6 Removing the tar process and now deleting cdx.json files older than 24 hours 2024-01-09 05:49:56 +00:00
51c127601a Removing the tar process and now deleting cdx.json files older than 24 hours 2024-01-09 05:45:25 +00:00
49a9bc4710 Add pastebin_automated_cdx_processor.py 2024-01-09 05:15:31 +00:00
457f32ed61 Added tar support for files older than 24 hours to keep cdx file count low 2024-01-09 00:19:22 +00:00
f5928594a5 Changed time delta to 1 day to minimize cdx file count 2024-01-08 23:50:17 +00:00
85319ba390 Updated code to move files to /opt/cdxfiles/mediafire 2024-01-08 11:09:35 +00:00
42be737ee6 Code fix 2024-01-08 11:05:07 +00:00
2bf2e02275 Update mediafire_automated_cdx_processor.py 2024-01-08 10:44:33 +00:00
d56d7bb81f Added in support to compare already existing files to skip them being reprocessed 2024-01-08 10:37:24 +00:00
61f5a4ddbf Update concurrency of downloads 2024-01-08 08:27:15 +00:00
2e938624f9 Add mediafire_automated_cdx_processor.py 2024-01-08 08:23:28 +00:00
b2a9c44c9e Added in a cdx filesize count statistic for the Grafana Dashboard to report on sizing 2023-12-27 08:58:44 +00:00
2dddda28bf Add into a "cdxcount": 1, count for metric reporting of processed CDX files 2023-12-27 08:52:23 +00:00
182c58f1ce Implemented subprocess for running multiple json extractions at once 2023-12-27 08:46:36 +00:00
8496391064 Update to remove tophosts output 2023-12-26 11:59:44 +00:00
ae515fb425 Upload files to "/" 2023-12-23 04:39:00 +00:00