datechnoman
  • Joined on 2023-03-10
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 22:48:29 +00:00
d3d110789a Updated bug in code for previous change
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 21:35:28 +00:00
43d55a27fd Set ia path variable for cron execution
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 21:15:20 +00:00
6c16ee45bd Updated cdx file retention to 2 days
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 21:14:25 +00:00
1e44917569 Updated cdx file retention to 2 days
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 21:13:52 +00:00
8e1912cb9b Updated cdx file retention to 2 days
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 05:51:43 +00:00
4d4dc9c81c Add imgur_automated_cdx_processor.py
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 05:49:58 +00:00
08dc0c0bf6 Removing the tar process and now deleting cdx.json files older than 24 hours
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 05:45:26 +00:00
51c127601a Removing the tar process and now deleting cdx.json files older than 24 hours
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 05:15:34 +00:00
49a9bc4710 Add pastebin_automated_cdx_processor.py
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 00:19:23 +00:00
457f32ed61 Added tar support for files older than 24 hours to keep cdx file count low
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 23:50:19 +00:00
f5928594a5 Changed time delta to 1 day to minimize cdx file count
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 11:09:37 +00:00
85319ba390 Updated code to move files to /opt/cdxfiles/mediafire
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 11:05:10 +00:00
42be737ee6 Code fix
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 10:44:34 +00:00
2bf2e02275 Update mediafire_automated_cdx_processor.py
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 10:37:25 +00:00
d56d7bb81f Added in support to compare already existing files to skip them being reprocessed
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 08:27:18 +00:00
61f5a4ddbf Update concurrency of downloads
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 08:23:31 +00:00
2e938624f9 Add mediafire_automated_cdx_processor.py
datechnoman created repository ArchiveTeam/Ignored_ArchiveOrg_URL_Processor 2023-12-30 01:48:12 +00:00
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2023-12-27 08:58:47 +00:00
b2a9c44c9e Added in a cdx filesize count statistic for the Grafana Dashboard to report on sizing
datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2023-12-27 08:52:25 +00:00
2dddda28bf Add into a "cdxcount": 1, count for metric reporting of processed CDX files