I was recently tasked to build a script that is as light as possible but still able to send files via SFTP securely and quickly. This script has error checking and reporting, and will sleep if the files are not currently available for a set amount of time.
Can you make this script more efficient? Yes, but it will carry more requirements. We needed to be light and simple to allow the best compatibility.
Notes:
– Will detect for default values and files
– Will error locally and via ftp if it can connect
– Waits for file complete.%DATE%.date to signal script that all files are ready to be uploaded
– Uploads all files with specified extension (csv is default)
– Archives processed files, compresses them, and deletes the sources
– Purges old archives after specified time
Source files needed:
– 7z.exe
– psftp.exe
You will also need to create the config.xml file which saves the required information:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
<?xml version="1.0" encoding="UTF-8"?> <FTP> <username>USER</username> <password>PASS</password> <ftpUrl>URL</ftpUrl> <fileLocation>FILE</fileLocation> <days>30</days> </FTP> <!-- Instructions: 1) Enter your full username 2) Enter your password 3) Enter ftp url without protcol (eg. secure.example.com) 4) Enter file location without a trailing backslash (eg. C:\usr\test) 5) Alter days field if more or less than 30 days retention are required. --> |
PowerShell:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 |
#Set main variables $username = $null $password = $null $dir = $null $url = $null $date = date #cd to working directory cd $pwd #Check if config file exists $testxmlpath = test-path config.xml if ($testxmlpath -ne $true) { echo "Error: Config.xml file is missing, please make sure this file exists in $pwd" > log.txt exit } else { #Get data from config file and set variables [xml][/xml]$xml = Get-Content config.xml $username = $xml.FTP.username $password = $xml.FTP.password $dir = $xml.FTP.fileLocation $url = $xml.FTP.ftpUrl $days = $xml.FTP.days } #Check if config file has been altered from default if ($username -eq "USER" -or $password -eq "PASS" -or $url -eq "URL" -or $dir -eq "FILE") { echo "Error: Default values in config.xml have not been changed!" > log.txt exit } #build ftp expression $com = "./psftp.exe $url -l $username -pw $password -batch -b batch.scr" #test file path $pathtest = Test-Path $dir\*.csv #Set correct date format $Fdate = (get-date).ToString("MMddyyy") #loop to look for complete file, if not there sleep for 10 min 5 times while($i -le 5) { $completed = Test-Path $dir\complete.$Fdate.date if($completed -eq $true) { break } else { Start-Sleep -s 6 #00 $i++ if($i -eq 5) { echo "Error: Completed file could not found in $dir!" > log.txt exit } } } if($pathtest -ne $true) { #file does not exist, build error log echo "Error: File could not be found at $dir on $date" > error.log $errorlog = "error.log" #build batch script "put $errorlog" | set-content batch.scr -Encoding Ascii "quit" | add-content batch.scr -Encoding Ascii #run command invoke-expression $com | Out-Null if ($lastexitcode -eq 0) { echo "Warning: FTP transmit success on $date, but no CSV could be found!" > log.txt remove-item error.log remove-item batch.scr } else { echo "FTP transmit failure on $date" > log.txt echo "Error: File could not be found at $dir on $date" >> log.txt remove-item error.log remove-item batch.scr } } else { #build batch script "mput $dir\*.csv" | set-content batch.scr -Encoding Ascii "quit" | add-content batch.scr -Encoding Ascii #run command invoke-expression $com | Out-Null #test for failure if ($lastexitcode -eq 0) { echo "FTP transmit success on $date" > log.txt remove-item batch.scr #build 7z expression and archive csv files $arcDate = (get-date).ToString("MM-dd-yyy") $arcName = "Archived-$arcDate" $archive = "./7z.exe a $dir\Archives\$arcName.zip *.csv" invoke-expression $archive | Out-Null #delete orginal files get-childitem $dir -include *.csv -recurse | foreach ($_) {remove-item $_.fullname} #purge archived zips $now = get-date $lastWrite = $now.AddDays(-$days) $arcFiles = Get-Childitem $dir\Archives -Include *.zip -Recurse | Where {$_.LastWriteTime -le "$LastWrite"} foreach ($arcFile in $arcFiles){ if ($arcFile -ne $NULL){ echo "Deleting archive $arcFile due to age older than $days" >> log.txt Remove-Item $arcFile.FullName | out-null } else{ echo "No archives older than $days to delete" >> log.txt } #remove completed files remove-item $dir\complete.$Fdate.date } } else { echo "Error: CSV was found but, FTP transmit failed on $date" > log.txt remove-item batch.scr } } |