datechnoman

0 Followers · 0 Following

• Joined on 2023-03-10

Repositories Projects Packages Public Activity Starred Repositories

datechnoman pushed to main at ArchiveTeam/CommonCrawl_URL_Processor 2024-02-13 22:35:12 +00:00

722838b24a Updated to include zstd package

datechnoman pushed to main at ArchiveTeam/CommonCrawl_URL_Processor 2024-02-08 00:39:12 +00:00

3151fce353 Update warc_wat_url_processor.py

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-02-05 08:44:51 +00:00

f0a70605da Update telegram_automated_cdx_processor.py

datechnoman pushed to main at ArchiveTeam/Migrated_ArchiveOrg_CDX_Stats_... 2024-02-05 08:40:32 +00:00

a82c328401 Add telegram_automated_cdx_processor.py

datechnoman pushed to main at ArchiveTeam/Migrated_Keyword_URL_Extractor 2024-02-05 02:19:17 +00:00

9e6e5190d3 Updated to support zst compressed files

datechnoman pushed to main at ArchiveTeam/Migrated_Keyword_URL_Extractor 2024-02-04 22:14:30 +00:00

95abf80bd1 Updated script to stream compressed files

datechnoman pushed to main at ArchiveTeam/Migrated_CommonCrawl_WAT_Path_... 2024-02-02 04:07:20 +00:00

0fd62a7391 Updated script to support .zst instead of .gz

datechnoman pushed to main at ArchiveTeam/CommonCrawl_URL_Processor 2024-01-28 22:50:13 +00:00

486a68a796 Removed multithread compression and added force overwrite for compression files

datechnoman pushed to main at ArchiveTeam/CommonCrawl_URL_Processor 2024-01-28 11:39:03 +00:00

ebc07a6974 Update zstd to overwrite conflicts

datechnoman pushed to main at ArchiveTeam/CommonCrawl_URL_Processor 2024-01-28 11:36:19 +00:00

29d24e9826 Reverting

datechnoman pushed to main at ArchiveTeam/CommonCrawl_URL_Processor 2024-01-28 11:29:56 +00:00

6d591ef0d0 Added in error logging

datechnoman pushed to main at ArchiveTeam/CommonCrawl_URL_Processor 2024-01-28 11:23:29 +00:00

54747b64f6 Update warc_wat_url_processor.py

datechnoman pushed to main at ArchiveTeam/CommonCrawl_URL_Processor 2024-01-28 09:14:16 +00:00

b6a9c68140 Update warc_wat_url_processor.py

datechnoman pushed to main at ArchiveTeam/CommonCrawl_URL_Processor 2024-01-28 09:10:22 +00:00

d0fa7c84f4 Update warc_wat_url_processor.py

datechnoman pushed to main at ArchiveTeam/ArchiveOrg_Convert_Items_to_Do... 2024-01-28 02:10:38 +00:00

ddbe2a34eb Update README.md

datechnoman pushed to main at ArchiveTeam/ArchiveOrg_Convert_Items_to_Do... 2024-01-28 02:04:04 +00:00

7a78a05c3e Update ia_metadata_to_download_links.py

datechnoman pushed to main at ArchiveTeam/ArchiveOrg_Convert_Items_to_Do... 2024-01-28 02:03:33 +00:00

3bdbb023d5 Delete ia_metadata_to_download_links.py

datechnoman pushed to main at ArchiveTeam/ArchiveOrg_Convert_Items_to_Do... 2024-01-28 02:02:16 +00:00

a87ce2476a Upload files to "/"

datechnoman pushed to main at ArchiveTeam/ArchiveOrg_Convert_Items_to_Do... 2024-01-28 02:01:57 +00:00

13c3a8ff30 Update README.md

datechnoman pushed to main at ArchiveTeam/ArchiveOrg_Convert_Items_to_Do... 2024-01-28 01:59:17 +00:00

d426ca189d Add README.md