ArchiveTeam/CommonCrawl_URL_Processor
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Updated script to keep regenerating a list of files to download

Browse Source
This commit is contained in:
datechnoman 2023-12-21 09:35:39 +00:00
parent 50e89b9de2
commit bfc13cb6ef
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
commoncrawl_transfer.ps1
View File

@ -52,7 +52,7 @@ try
$remoteFilePath = [WinSCP.RemotePath]::Combine($remotePath, $fileInfo.FullName) $remoteFilePath = [WinSCP.RemotePath]::Combine($remotePath, $fileInfo.FullName)
# Construct the local file path # Construct the local file path
$localFilePath = [System.IO.Path]::Combine("E:\CommonCrawl\", $fileInfo.Name) $localFilePath = [System.IO.Path]::Combine("E:\CommonCrawl", $fileInfo.Name)
# Download the file # Download the file
$transferResult = $session.GetFiles($remoteFilePath, $localFilePath, $True) $transferResult = $session.GetFiles($remoteFilePath, $localFilePath, $True)