Making File Transfer Safe and Easy: Just Say No to Scripting

File transfers are too important to trust to just any old script. Ensure speed and secure transfers with automation. 

Many IT pros love to script. They have mastered their language of choice and can quickly write code to do exactly what they want. For many things scripting is great, but it is often far from the best answer – especially for file transfers. Here are a bunch of reasons why.  

What happens if your scripter leaves the company? Who understands the script – does anybody even use that scripting language? And if you are using scripting to automate, you must write a script for every single new thing you want to automate. That means any new file transfer scenario needs a new or updated script. 

There Must Be a Better Way

Manually transferring numerous files may seem like small potatoes – but can easily chip away at your time, especially if you’re doing it every day. Take back your time using file transfer automation, a powerful solution that gets you back to those mission-critical projects.  

Managed File Transfer (MFT) replaces scripts and centralizes those automation processes through a single pane of glass, making these processes easy to perform through an intuitive Web-based management portal. We’ll get back to this topic a little later. 

Watch the free on-demand webinar, "How to Automate & Integrate with Managed File Transfer".

More Reasons Not to Script

When automating file transfer processes, you really don’t want to have to script. Simply put, scripting does not scale, scripts are hard to support and are far from a full solution to secure file transfer problems. Scripts miserably fail the ease-of-use test.  

Meanwhile, these scripts end up dispersed among many systems throughout your infrastructure. As these script-based transfers with low levels of automation tend to be dispersed, IT doesn't know where the scripts are running from or what files they are sending. This is also a problem if the person that manages or created that process leaves the organization, as there is likely no support for these implementations and no way to train new users once the script’s creator is gone. Finally, if changes need to be made to the process, who understands the script enough to do that?   

There are a myriad of scripting languages. What happens if the script was written in a language no one in the company understands? How do you make sure the underlying scripting solution is up to date and secure – and that the script itself is up to date? If it is an open-source scripting module that has not been updated, that tool can be breached, meaning your sensitive files can be compromised.     

And of course, IT has zero visibility into what the process does. Where are these files being sent?   

“Scripting has become a popular method for automating file transfer because it's included at no extra cost as a part of most operating systems. Commercial and open-source operating systems offer built-in FTP client and server capabilities with scripting languages, such as bash, Shell, VBScript, and Windows PowerShell. You can then use a tool like Windows Scheduler to execute your scripts,” according to the Dataflog blog 17 Ways MFT is Superior to Scripting for Automating File Transfers. But while you don't have to pay for scripting languages, you stand to incur a number of hidden costs from using them.  

“Because they are not full-fledged applications, scripting tools often offer limited functionality. For example, your organization will be able to create scripts that can transfer files. But those scripts may be unable to: 

• Encrypt files at rest 

• Execute custom logic 

• Parse files and route them to different locations 

• Trigger your script to monitor a file or check a record in a database based on an event 

• Or perform integration in a flexible manner 

• These core, basic automation capabilities prove critical for many organizations.

When Scripts Fail - What Do You Do Then?

Scripts can start out fine, then get flaky over time. “It starts with a requirement for a simple yet regular file transfer between two points, for which a script is created and run as either a cron job or Windows scheduler task. Over time, both the organizations reliance on this script grows as the importance of the task increases; and the number of files being transferred increases,” explained in the Advanced Cyber blog, Automate File Transfers: Ditch the Scripts. “One-day the script fails and with the mist of time, staff having left and knowledge has been lost, nobody knows how the script functions nor how to fix it. A major risk has been created.” 

But there are other reasons not to script, or at least rely 100% on scripts for file transfers: 

• It is hard to write scripts that are reusable for new and even slightly different file transfer tasks. 

• It is hard to manage, modify and troubleshoot scripts especially with multiple authors and usually little to no documentation. 

• Scripts lack features such as encryption and auditing, and therefore lack security. 

The Benefits of File Transfer Automation

If you have gotten to the stage where you're automating file transfers, you're probably using scripting such as PowerShell. While true PowerShell gurus easily create these types of scripts, that’s only part of the process. Chances are you have no idea where these files wound up. Did they make it to their destination? Were they processed? What files got processed?     

Scripts have other limitations. Often with scripts, you write out to a flat file. If you have a more built-out environment, you may be writing to a database to record that information, but that's uncommon. Most often, that scenario means there is no logging whatsoever. That process just runs – with absolutely no record. Occasionally, if there is an issue, the database might spot an error but there’s no real tracking of the file’s journey.   

Here are the benefits of automation, according to the Automate File Transfers: Ditch the Scripts blog: 

• “Both administrator actions and individual file movements are logged for auditability. 

• Dual configurations for high-availability mean that even if one solution instance fails, another will complete the transfer. 

• Security features such as file encryption and secure shredding once a file has been transferred. 

• A graphical user interface which supports the creation of file transfer workflows with no scripting knowledge.” 

Luckily, MOVEit Automation from Progress works with MOVEit Transfer (as well as other hosts like SFTP, FTPS, UNC, Azure Blob, S3 and others) to consolidate and automate file transfer activities. MOVEit can easily:  

• Quickly automate new workflows and reduce the likelihood of error in repetitive processes.  

• Ensure compliance and security.  

• Design advanced workflow automation without scripting.  

• Integrate seamlessly with your existing environment.

MOVEit Automation: Creating a Workflow

Scripts have other limitations. Often with scripts you write out to a flat file. If you have a more built-out environment, you may be writing to a database to record that information, but that's uncommon. The most common scenario is there is no logging whatsoever. That process just runs – with absolutely no record. Occasionally, if there is an issue, the database might spot an error but there’s no real tracking of the file’s journey.   

In MOVEit Automation, workflows are called tasks. Here, IT determines what processes look like, such as where files come from and where they need to go.

“Let’s assume we are pulling files from a MOVEit Transfer server – though you can specify any host types we previously defined by adding a step such as source or destination. That process could include using PGP and performing file encryption, decryption, zip, unzip, as well as additional custom processes.

As far as the actual file movement, you select the source by picking from the hosts defined, then navigate out to see folders,” said Andrew Lorandos, CISSP​, Principal Solutions Engineer and MOVEit Expert. “Say you only need to collect new files. Here, automation keeps track of files that were previously processed. That prevents the deletion of originals or renaming of originals on the source.” 

When to Say Yes to Scripting

There are definitely times what scripting for file transfers still makes sense, and MOVEit’s integration with PowerShell and other scripting languages makes it work like a charm. “PowerShell with MOVEit can be a great combo. PowerShell can run processes against files you are transmitting. MOVEit Automation, meanwhile, can create custom scripts and one of those languages is PowerShell. The other is VBScript. You can run processes in line in your task by creating a custom script in PowerShell. There is a script manager within the MOVEit Automation web interface,” said Lorandos. 

“For advanced tasks including conditional logic, you can pass variables between the MOVEit Automation service and PowerShell script and vice versa. This is handy for when, say, you are parsing a file line by line for data validation and you need to pass it back to ensure it ran through successfully,” Lorandos explained. “MOVEit Automation handles internal transfers just between internal systems and external transfers. You can reach out to a vendor or trading partner externally, and pick up or deliver files to their system, then pull them in internally or vice versa.”  

Learn to Master and Automate Secure File Transfers

Learn more about managed and secure file transfer in our free on-demand webinar: Automate and Integrate Managed File Transfer.