Parsing data is something that anyone writing code will eventually have to do. The annoying part is its hard to practice unless you have a real example, data dumps can be really strange and its sometimes really hard to get the data how you want it. Recently we needed to parse an output of some f5 pools and nodes, we were using this output to run some REST calls for maintenance work and building the json file we needed for that maintenance work manually was just not feasible.
If you have not seen my post on building json objects with powershell I recommend you give that a read first as it explains how to build json from powershell objects: https://zeleskitech.com/2016/03/10/building-json-powershell-objects/
Lets dig in. The data we got from our f5s looks like this:
|
1 2 3 4 5 6 7 8 9 10 11 |
Ltm::Pool: subA.domain.com_pool | Ltm::Pool Member: subA1.domain.lan:443 | Ltm::Pool Member: subA2.domain.lan:443 Ltm::Pool: subB.domain.com_pool | Ltm::Pool Member: subB1.domain.lan:443 | Ltm::Pool Member: subB2.domain.lan:443 Ltm::Pool: subC.domain.com_pool | Ltm::Pool Member: subC1.domain.lan:443 | Ltm::Pool Member: subC2.domain.lan:443 | Ltm::Pool Member: subC3.domain.lan:443 | Ltm::Pool Member: subC4.domain.lan:443 |
Nothing crazy here, but lets see what kind of json format we needed:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
{ "f5Data": [ { "name": "subA.domain.com_pool", "members": [ "~Common~subA1.domain.lan:443, 1", "~Common~subA2.domain.lan:443, 2" ] }, { "name": "subB.domain.com_pool", "members": [ "~Common~subB1.domain.lan:443, 1", "~Common~subB2.domain.lan:443, 2" ] } ] } |
This is where things start to get a little tricky. We need separate objects for each pool, and we also need to group the nodes into those pools. The members have a comma separated value of 1 or 2 which tells our maintenance tool to pick which group to run.
How do we do this? Well lets take a look at the whole script and then walk through it explaining why everything was done the way it is:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
[cmdletbinding()] param( [Parameter(Mandatory=$true)]$inputFile, [Parameter(Mandatory=$true)]$outputFile ) # Get file and set som vars $file = get-content $inputFile $poolData = @() $everyOther = $true # get each line foreach ($item in $file) { #This if allows us to first if the line is a pool, if it is we also check if there was a previous loop that needs to be saved off #If there is we save it, clear the vars, and start writing the new pool data while also cleaning up the data. if ($item -like "Ltm::Pool:*" -and $item -like "*_pool") { if ($tempData.name -ne $null -and $tempMembers[0] -ne $null) { $poolData += [pscustomobject]@{ name = $tempData.name members = $tempMembers } } $tempData = @{} $tempMembers = @() $everyOther = $true $pool = $item.replace("Ltm::Pool: ","") #Write-output "$pool" $tempData.name = $pool } #This if checks if the line is a member, if it is we clean up the line and use the if else to swap between odd and even groups #this method is dumb and dirty but it works for now. if ($item -like " | Ltm::Pool Member:*") { $node = $item.replace(" | Ltm::Pool Member: ","") #Write-output "$node" if ($everyOther){ $tempMembers += "~Common~" + $node + ", 1" $everyOther = $false } else { $tempMembers += "~Common~" + $node + ", 2" $everyOther = $true } } } #used to send last loop of data to array $poolData += [pscustomobject]@{ name = $tempData.name members = $tempMembers } #add parent object $jsonDoc = [pscustomobject]@{ f5Data = $poolData } #convert to json and output to file $jsonFile = $jsonDoc | convertto-json -depth 5 $jsonFile | Out-File $outputFile |
Lines 1 – 7 are setting up some default features in powershell (if you dont know what cmdletbinding does you should) as well as setting our input and output variables. Lines 10 – 12 we get the contents of our input file and store them, and then we initialize two values, poolData will be the object that we store our array indexes in, and everOther allows us to control that group number at the end of the node (note that this is a dirty trick, if you want true odd even a bit of regex is needed).
With that lets dig into the foreach loop:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
foreach ($item in $file) { #This if allows us to first if the line is a pool, if it is we also check if there was a previous loop that needs to be saved off #If there is we save it, clear the vars, and start writing the new pool data while also cleaning up the data. if ($item -like "Ltm::Pool:*" -and $item -like "*_pool") { if ($tempData.name -ne $null -and $tempMembers[0] -ne $null) { $poolData += [pscustomobject]@{ name = $tempData.name members = $tempMembers } } $tempData = @{} $tempMembers = @() $everyOther = $true $pool = $item.replace("Ltm::Pool: ","") #Write-output "$pool" $tempData.name = $pool } #This if checks if the line is a member, if it is we clean up the line and use the if else to swap between odd and even groups #this method is dumb and dirty but it works for now. if ($item -like " | Ltm::Pool Member:*") { $node = $item.replace(" | Ltm::Pool Member: ","") #Write-output "$node" if ($everyOther){ $tempMembers += "~Common~" + $node + ", 1" $everyOther = $false } else { $tempMembers += "~Common~" + $node + ", 2" $everyOther = $true } } } |
As I said before we need to be able to break up our nodes and pools into different objects, so how do we handle this? First we obviously need a loop, a foreach to be exact. This allows us to get each line individually.
On line 4: “if ($item -like “Ltm::Pool:*” -and $item -like “*_pool”)” allows us to decide if that line is a pool line.
Then on line 5: “if ($tempData.name -ne $null -and $tempMembers[0] -ne $null)” this is important because we need to be able to detect if we are done with a pool and need to save off that data, this line means we saved data off to our temp vars and we hit the next pool, we then save off that data into the poolData var which is an array of pscustomobjects.
Lines 11 – 16 clear our temp vars, set our everyOther var back to true so the group number starts at 1, removes some of the unneeded data and saves that off to our temp var.
On line 20: “if ($item -like ” | Ltm::Pool Member:*”)” allows us to get the pool member, we clean up the data, and then add that data to our temp var based on what our everyOther var is currently set to.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
#used to send last loop of data to array $poolData += [pscustomobject]@{ name = $tempData.name members = $tempMembers } #add parent object $jsonDoc = [pscustomobject]@{ f5Data = $poolData } #convert to json and output to file $jsonFile = $jsonDoc | convertto-json -depth 5 $jsonFile | Out-File $outputFile |
Now as I said we check for the next pool during the beginning of the loop, if we were to exit the loop on the very last pool and convert our results we would lose the last entry as it wouldn’t be saved. Lines 2 – 5 are exact duplicates from the loop, we use this to capture the very last entry in our input. The remaining lines of code add the child objects to the parent object, and then finally convert our nested pscustomobject to json and output it to a file.
At this point we now have our data in our required format needed to run some automation tasks! Hope this helps!