datechnoman

0 Followers · 0 Following

• Joined on 2023-03-10

Repositories Projects Packages Public Activity Starred Repositories

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 22:48:29 +00:00

d3d110789a Updated bug in code for previous change

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 21:35:28 +00:00

43d55a27fd Set ia path variable for cron execution

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 21:15:20 +00:00

6c16ee45bd Updated cdx file retention to 2 days

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 21:14:25 +00:00

1e44917569 Updated cdx file retention to 2 days

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 21:13:52 +00:00

8e1912cb9b Updated cdx file retention to 2 days

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 05:51:43 +00:00

4d4dc9c81c Add imgur_automated_cdx_processor.py

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 05:49:58 +00:00

08dc0c0bf6 Removing the tar process and now deleting cdx.json files older than 24 hours

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 05:45:26 +00:00

51c127601a Removing the tar process and now deleting cdx.json files older than 24 hours

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 05:15:34 +00:00

49a9bc4710 Add pastebin_automated_cdx_processor.py

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-09 00:19:23 +00:00

457f32ed61 Added tar support for files older than 24 hours to keep cdx file count low

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 23:50:19 +00:00

f5928594a5 Changed time delta to 1 day to minimize cdx file count

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 11:09:37 +00:00

85319ba390 Updated code to move files to /opt/cdxfiles/mediafire

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 11:05:10 +00:00

42be737ee6 Code fix

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 10:44:34 +00:00

2bf2e02275 Update mediafire_automated_cdx_processor.py

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 10:37:25 +00:00

d56d7bb81f Added in support to compare already existing files to skip them being reprocessed

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 08:27:18 +00:00

61f5a4ddbf Update concurrency of downloads

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-01-08 08:23:31 +00:00

2e938624f9 Add mediafire_automated_cdx_processor.py

datechnoman created repository ArchiveTeam/Ignored_ArchiveOrg_URL_Processor 2023-12-30 01:48:12 +00:00

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2023-12-27 08:58:47 +00:00

b2a9c44c9e Added in a cdx filesize count statistic for the Grafana Dashboard to report on sizing

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2023-12-27 08:52:25 +00:00

2dddda28bf Add into a "cdxcount": 1, count for metric reporting of processed CDX files