Using MOVEit 2018's REST API to Create and Monitor File Transfers with PowerShell

We've already covered Swagger UI and how to use PowerShell to authenticate to MOVEit Automation's REST API. In this article, we're going to take what we've learned and get into some real-world use cases using MOVEit Automation's REST API— creating and monitoring tasks.

For this use case, I have a bunch of files in a folder and I'd like to upload all of these files to an Amazon S3 Bucket, one of MoveIt Automation's new available host types. On the webadmin Tasks page, it looks like this:

That was created via the webadmin page, let's see how we can do it via the REST API!

Authentication

Using our previous article on how to authenticate to the MOVEit Automation REST API, I'm going to quickly build some PowerShell to get an access token.

$cred = Import-CliXml -Path 'C:\MOVEitAutomationCred.xml'
 $apiConfig = @{
     UserName = $cred.UserName
     Password = $cred.GetNetworkCredential().Password
     GrantType = 'password'
     MoveItHost = '<yourmoveitautomationservernamehere>'
 }
 $authEndpointUrl = "https://$($apiConfig.MoveItHost)/webadmin/api/v1/token"
 $authHttpBody = "grant_type=$($apiConfig.GrantType)&username=$($apiConfig.UserName)&password=$($apiConfig.Password)"
 [System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }
 $token = Invoke-RestMethod  -Uri $authEndpointUrl -Method 'POST' -Body $authHttpBody
 $token = $token.access_token
 $authParam = @{
     Headers = @{ 'Authorization' = "Bearer $token" }
 }

Creating the Task

Since I'm assuming that you're not a MOVEit Automation REST API expert and don't have all of the API and methods memorized, it's always a good bet to use the Swagger UI. The Swagger UI will allow us to interactively play around and see what APIs are needed and how to call them.

As I look over the available APIs via Swagger, I spot an operation that looks exactly like what I need. The description of "adds a new task" is pretty self-explanatory.

As I click into it, I see that I'm going to have to provide the API with some JSON representing the task I'd like to create. Using the example JSON provided, I've managed to come up with some JSON representing the task I'd like to do.

{
     "Info": {
         "Description": "This task moves some files locally to an AWS S3 bucket on demand."
     },
     "steps": [
         {
             "Source": {
                 "HostID": "0",
                 "Path": "C:\\Temp",
                 "Type": "FileSystem"
             }
         },
         {
             "Destination": {
                 "HostID": "730111199",
                 "Type": "S3"
             }
         }
     ],
     "Name": "Copy Files to S3 Bucket",
     "Active": 1
 }

Now that I have the JSON created, I'll copy this JSON into a file called C:\DemoTask.json on my local computer and pass the appropriate parameters to Invoke-RestMethod. To pass all of the JSON data to the API, I'm using Get-Content -Raw and specifying the path to the file. Also, the ContentType parameter is required else the API will not be able to understand that the body is, in fact, JSON.

$taskEndpoint = "https://$($apiConfig.MoveItHost)/webadmin/api/v1/tasks"
 Invoke-RestMethod @authParam -Uri $taskEndpoint -Method POST -Body (Get-Content -Raw -Path 'C:\DemoTask.json') -ContentType 'application/json'
 
 Info               : @{Description=This task moves some files locally to an AWS S3 bucket on demand.; Notes=}
 NextActions        : @{NextAction=System.Object[]}
 Schedules          : @{Schedule=System.Object[]}
 steps              : {@{Source=}, @{Destination=}}
 Group              : {}
 ID                 : 475167498
 Name               : Copy Files to S3 Bucket
 Active             : 1
 AR                 : 0
 CacheNames         : random
 NextEID            : 13
 TT                 :
 UseDefStateCaching : 1

Starting the Task

Now that the task is created, let's start it. To do that, I first need to find the ID of the task I just created. If I were doing this in the same script, I could just pull the ID from the output when a new task is created but this isn't usually the case. Most of the time, you will have another script that starts the task.

To find the task ID, I'll need to query the Tasks API API again but this time I'll be using a GET method to pull existing tasks. Since I don't want to see all of the tasks but rather just that single task we're working with, I can use the name HTTP parameter and specify the name of the task.

$taskName = 'Copy Files to S3 Bucket'
 $httpQueryString = "?name=$taskName"
 $getUri = "{0}{1}" -f $taskEndpoint,$httpQueryString
 $task = Invoke-RestMethod @authParam -Uri $getUri -Method GET

I now have the task properties represented in the $task variable and can easily find the ID by looking at the $task.items.ID value.

Once I have the task ID, I can now pass this ID to the Tasks API again referencing the start attribute. Once started, you'll be returned a property called nominalStart which represents the time the task started.

$taskId = $task.items.ID
 $startTaskUri = '{0}/{1}/start' -f $taskEndpoint,$taskId
 $startTaskResp = Invoke-RestMethod @authParam -Uri $startTaskUri -Method POST -ContentType 'application/json'
 $startTaskResp
 
 nominalStart
 ------------
 2018-03-16 18:53:00.05

Monitoring the Task

Depending on how many files we have in the folder will depend on how long the task will take. As-is, we'll start the task and the PowerShell script will continue running but what if we have some kind of dependency to the result of this task. In this instance, perhaps we have some further automation in this script that does something to those files in the S3 bucket. In that case, we need to monitor the task as it runs and hold further script execution until the task finishes. We can do this by using a PowerShell while loop and the MOVEit Automation's Reports API.

We can get the status of running tasks using the taskruns attribute. Again, let's assign variables to the URIs, build some queries and using Invoke-RestMethod, kick off another API call to start a taskruns report.

$reportsEndpoint = 'https://{0}/webadmin/api/v1/reports' -f $apiConfig.MoveItHost
 $taskRunsUri = '{0}/taskruns' -f $reportsEndpoint
 
 $predicate = "taskid==$taskId; nominalStart=='$($startTaskResp.nominalStart)'"
 $query =  @{ 'maxCount' = 15; 'orderBy'="!StartTime"; 'predicate' = $predicate }
 
 $report = Invoke-RestMethod @authParam -Uri $taskRunsUri -Method POST -Body ($query | ConvertTo-Json) -ContentType 'application/json'
 
 $report.items
 
 
 ScheduledTime : 2018-03-16T19:14:05.46
 TaskID        : 642900281
 TaskName      : Copy Files to S3 Bucket
 Node          : 0
 NominalStart  : 2018-03-16 19:14:05.46
 StartedBy     : MIAdmin
 LogStamp      : 2018-03-16T19:14:05.46
 StartTime     : 2018-03-16T19:14:05.46
 RecType       : start
 RunID         : 70
 QueryType     : TaskRuns

We were able to get some good information here but this is just at one point in time. We'll need to check the state of this task over and over again until it's complete.

You can see that the API returned some information but one of them wasn't the status of the task. We need this to determine what state the task is in to see if it is complete or not. Unfortunately, the status attribute doesn't actually exist until the task stops running but that's not a problem with PowerShell!

$reportParams = $authParam + @{
     Uri = $taskRunsUri
     Method = 'POST'
     Body = ($query | ConvertTo-Json)
     ContentType = 'application/json'
 }
 while ('Status' -notin (Invoke-RestMethod @reportParams).items.PSObject.Properties.Name) {
     Write-Host 'Waiting for task to complete...'
     Start-Sleep -Seconds 5
 }

When this script is kicked off, it will then wait for the task to either fail or succeed and will let us know the task is still running.

Waiting for task to complete...
Waiting for task to complete...