Video: How to Download Files From Azure Blob Storage with MOVEit Automation

 One of the new features of Ipswitch's MOVEit Automation 2018 product is the introduction of REST APIs. In this video from Ipswitch contributor Adam Betram , we'll cover how to use PowerShell to first authenticate to a MOVEit Automation server and then we'll cover how to create and monitor a file transfer job using a PowerShell script.

In this 10-minute clip from TechSnips, you'll learn how to use a PowerShell script to download files from Azure Blog Storage with MOVEit Automation. Want to know more about working with Azure and MOVEit? Check out our previous posts, Getting Started with Microsoft Azure's Cloud Shell and How to Copy Files from an Azure App Service with PowerShell. 

Big thanks to Adam Bertram's awesome new site, TechSnips, for the video. TechSnips is a new (and totally free) IT career development platform that offers video how-to's on dozens of IT technologies and techniques. Adam and co. have covered tons of technologies, like cloud computing, programming, system administration, and a whole lot more. Check it out!

About Adam:

Adam Bertram is a 20-year veteran of IT. He’s an automation engineer, blogger, consultant, freelance writer, Pluralsight course author, and a Microsoft Cloud and Datacenter Management MVP. Adam focuses on DevOps, system management, and automation technologies as well as various cloud platforms mostly in the Microsoft space. Catch up on Adam’s articles at adamtheautomator.com or follow him on  GitHub or Twitter at @adbertram.

Transcript:

All right in this step we're going to go over how to download files from Azure Blob Storage with Ipswitch’s MOVEit Automation 2018.

So there's a view prereqs you're going to need to fill if you're going to follow along.

First thing, you're obviously going to need MOVEit Automation 2018 and you're going to need the web admin install so that's a separate feature you want to make sure to install that we're going to be using a lot here.

You’re going to need rights in Automation, obviously, you’re going to need a storage account and you need that service principle set up in Azure so that Automation can authenticate to Azure. That's done by Azure credentials and there is a good snip that I refer to which is How to Create and Authenticate to Azure with a Service Principal using PowerShell. That’s another good snip.

There's another good Snip that’s in the library about How to Set Up an Azure Virtual Machine with WhatsUp Gold, that kind of does the same thing so there’s some good suggestions for you.

 And the scenario is going to be we're going to create a PowerShell script and this PowerShell script is just going to download some files from a storage container in your Azure subscription to a local file on the MOVEit Automation server.

 So first thing we have to do is we have to create that PowerShell script. Now this PowerShell script is created just for MOVEit. Most of it is just the general PowerShell, but there are a few variables in here with methods on various objects that you are going to need to know because they are MOVEit specific.

But starting from the top here, 21 through 36, that's downloading the AzureRM PowerShell module only if it's required so this is a good condition of logic since we are running this on the MOVEit Automation server itself, this just does a check to make sure that the module is there and if it's not then it goes ahead and installs the prerequisite NuGet version and gets it all downloaded and all good.

Task Parameters

The next piece of this script is we ensure that these required task parameters are there, so we will be creating a task in MOVEit Automation that will have various parameters in it. Now we don't necessarily have to have parameters we could statically put in this stuff in the script itself, but it's that's never a good idea in case we want to change it up to a different service account or a container, we can do that.

But in this region I'm essentially defining all of the parameters that I want so I'm going to have three different parameters for our MOVEit Automation tasks and here I'm checking to make sure they're there. I'm doing that by running the miaclient variable and then referencing the miaclient object and then the MIGetTaskParam method.

The miaclient is a MOVEit Automation specific object, PowerShell knows what that is when it runs within MOVEit Automation, so if I were to just run this now it would just blow up because it would have no idea what miaclient is.

So MIGetTaskParam is the one that I'm using and I'm setting those instead of just running MIGetTaskParam to get the value for all those parameters, I'm using set variable here to actually create variables in my PowerShell scripts so I can access those easily.

Authenticating to Azure

In the next region I’m authenticating to Azure. First checking to see if we are authenticated on 51, and then I'm setting up a credential, and the credential is your ServicePrincipal ID and password so that's going to be your enterprise application ID which you should have already set up. And then on 55 that's where I finally do that connection authenticate to my subscriptions I want, specify my last subscription ID and my tenant ID and I'm good to go.

Scripting the Download Action

Next up is when I actually do the downloading. So first up I have to create the storage context here's where I use the value of the Azure Storage Account and the Azure Storage Key—those task parameters that I brought in earlier. Then on line 61 through 65, that's where I have that foreach loop. This is where I list out all of the files inside of the Azure container that I provided via my task parameters. And then I'm going through here, down to 62 there, here's where I have another miaclient method. The MINewCacheFilename. This creates a temporary file in cache internally inside of MOVEit Automation so it understands that we're working with files.

Line 63 is where I run Get-AzureStorageBlobContent, that's where I actually go out and download it. And then a 64 I have another miaclient object reference, this time MIAddFile. This adds the file… this essentially converts the file and Azure to a local MOVEit Automation cached file so it can understand that we're working with those files again.

Catch Block

And then on 67 through 72  I had the catch block so if you notice up here on 38 it starts out with try I have a try catch block, and in PowerShell if the script errors in that try block if anything errors out it's gonna run into this catch block and then I'm using the miaclient reference again and this time SetErrorCode and SetErrorDescription.  It makes MOVEit Automation understand that PowerShell threw an error if not it would just say success all the time because it doesn't know how to interpret a PowerShell error. So those lines do that.

Adding the PowerShell Script to MOVEit

Alright, so now that we have the PowerShell script all created now we can actually put it into MOVEit Automation. There's a couple ways to do that we can upload an entire script or we can copy and paste. Because I'm lazy—I don't want to create another script—I will copy and paste. You could do it any different way. So to do that I go up here to scripts and then I go to add script and then I will call this, oh, I don't know, “Download Azure Blob Storage” it's going to be a PowerShell script and we have script templates in here that Ipswitch provides, but we're not going to need those this time. So here's where we copy and paste, we can just paste that in or we could do import script file which allows us to import an entire file. One thing I like about MOVEit up Automation is that it gives you the nice syntax highlighting so notice that we have the nice syntax highlighting here in PowerShell, so it doesn know the PowerShell syntax so that's good.

Creating a new task

Alright so now that we have the script we can go over here to tasks—we need to create a new task. Go over to add task new task give it a friendly name… Traditional we're not going to do anything fancy with loops or conditional blocks you can do all kinds of stuff but we’re not going to now. Add tasks—alright here's where we set these steps, since this is just going to be a single script—we could have broken this out into lots of different scripts we can do this lots of different ways but just to get you started we just have that one script. So to invoke a PowerShell script we need to create a step here and we're not going to have a source because technically the script is the source we are gathering all those files within these the script itself. So we need to create a Process Step. Here's where we define the script that we created so we choose our Azure blogs s—uh oh, you can tell I've been blogging too much—and Azure blob storage, oh well we'll leave it.

Task Parameters

All right so now here is where we provide the task parameters, this is where I was telling you about in the script. Here on line 40 here's where are those required MIparams are so we need to provide these or it's going to error out. Alright, so first off we need the Azure storage account parameter name and the Azure storage account, let's go over here and it I'm already in the portal here and it is going to be TechSnips so I will go ahead and put in TechSnips, add parameter that's the next one we need— we need Azure storage key so you will copy and paste that one out. And then to find your storage key you go to the Azure storage account here TechSnips scroll down until you see access keys here in settings and you want to pick the first key here. So I will copy this out and then I will go back to MOVEit paste that in. One more parameter:  Azure container. That is the azure container name. Click on our storage account here and go down to containers and is going to be MOVEit demo. Going in here you see we just have one file thumbnail.PNG. That's going to be the file that we're going to download.

Define Destination

So now that we add that Azure container, the azure container is MOVEit demo. Alright now we have all of the parameters defined. So we hit save and then now we are good to go. We add our process step and now we need to define the destination: where is it actually going to go. So notice that in here, we didn't actually define where it was going to go. We were downloading it would get Azure blob content but we said miaclient, MIAddFile($cacheFilename) here, we didn't actually specify where was going. We just had a file name. So it's inside of MOVEit at the moment and we're going to move it where it's supposed to go. So to do that we can add a destination step. So we go at the step and then go to destination and we're just going to save it to a local folder here. We can choose any kind of different host we want—that may be another snip for another day. But for now this is just going to be on the local file system of the server itself. So click Next and let's just save it to the root of C:\. That's just telling us the folder where the file is going to go. So now we can add destination and now we have run a custom script and save it to the local file system. So now we have our test setup.

Running the Task

So now we need to run it, so we go out here and then click on the ellipsis here next to our TechSnips demo tasks we just created, then click run tasks now. And then you see it's going to run the scripts, it’s going to go through a few things, authenticate run our script and then it should drop it into the root of C:\. So if this works, notice that the files that are processed were one and the file size was 49.9kb. So it looks like it actually did work so let's verify that since we dropped it on the root of C:\. I'm on the server right now and now notice thumbnail.png is here and it round it up to 50k. But just to keep me honest I'll delete it and I will run it again and you pop up—voila! And it popped up so it's downloading it from Azure every time and putting it in there. Alright so that has been a snip on how to download files from Azure blob storage using Ipswitch MOVEit Automation 2018.