Powerful Data Validation and Transformation With Corticon New Operators

In recent versions of Corticon, we have added a bunch of new operators to various data types, based on customers’ requests.

Of particular interest, are the new operators using regular expressions. Regular expressions are so rich that a very large set of new use cases is now possible. So, in this blog we will see what regular expressions are, and we will dive into use cases and examples for using the new String.matches operator. What you will learn for this operator directly applies to the new String.regexReplaceString operator as they are both using regular expressions as input.

Here is the full list of added operators:

• Integer.random
• Decimal.random
• String.replaceString
• String.regexReplaceString 
• String.matches

What Are Regular Expressions (Also Called Regex)?

They are a language of string patterns and are used for powerful text search, manipulation and replacement in many programming languages. For example, with a single expression, you can validate that a string contains an email address. Or that the string contains an email address of a specific company, for example progress.com. Or that a field contains a state name in either full form or abbreviated form (Ma or Massachusetts or Mass) or validate that a field contains the name of countries that belongs to the EEC.

They are commonly used to validate fields like phone numbers, social security numbers, emails, etc.

But you may have heard that regular expressions are confusing, as their syntax is relatively complex.  There is some truth to this, in particular for non-programmer users of Corticon (After all, regexs have to cater to a huge array of possibilities so their relative complexity provide a wide range of capabilities).

However, the good news is that many cases have already been solved for you and a Google search will usually get you immediately what you need, and if not exactly what you need, you won’t be far away for the final solution. For example, to match valid dates: https://www.regular-expressions.info/dates.html

There are also lots of tools for creating regexs. Here is a very useful online tool https://regexr.com/ that among other things provides explanations on expressions as well as a searchable community submitted regular expression catalog.

Use Cases

The operators String.matches and String.regexReplaceString can be used for data validation as well as rules driven data transformation in general.

Below are two examples:

Example 1: Validate a field has a valid identifier.

To illustrate, let’s say, a company's items are uniquely identified using the following pattern:

1. First five characters of the Item no. will be alphabet.
2. Next four characters will be digits only.
3. Last character will be an alphabet again.

For example, COMps739B

Here is how the operator would be used to validate the input string is valid:

Ent1.isValid = Ent1.MyString1.matches ( '[A-Z,a-z]{5}[0-9]{4}[A-Z,a-z]{1}' )

Example 2: Does a string contain a valid email address?

Ent1.emailAddressCorrect = Ent1.MyString1.matches ( '[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$' )

Example 3: Cleanup Strings 

Using the String.regexReplaceString operator, you can use the same regular expressions to cleanup strings. For example, you could hide email addresses returned in a payload for privacy reasons. This is easily achieved with this new operator as in the example below: The following expression will replace any email address with a comment string

Ent1.MyString2 = Ent1.MyString1.regexReplaceString ( '[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$',  '<email address hidden>' )

Conclusion

With these new operators you can now create rules that do powerful data validation and transformation easily and productively.

They are both available with Corticon classic Java and .Net as well as with the Serverless and frontend Corticon.js offering.

If you feel that certain regular expressions are so commonly used that they could be elevated to their own operators, please let us know. For example, one could imagine operators like isValidEmail or isValidUSASocialSecurityNumber?

Also, drop us a note if you have encountered the need for any other operators.

For reference, here is a detailed syntax for each operator:

Integer.random

Syntax: random (minRange, maxRange)

Description: This function returns a random integer between minRange and maxRange. If maxRange is less than minRange, maxRange becomes minRange and minRange becomes maxRange.

Decimal.random

Syntax: random (minRange, maxRange)

Description: This function returns a random decimal between minRange and maxRange. If maxRange is less than minRange, maxRange becomes minRange and minRange becomes maxRange.

String.replaceString

Syntax: replaceString( stringToBeReplaced, replacementString )

Description: Returns a new String where the instances of string to be replaced are replaced by the replacement string.

String.regexReplaceString

Syntax: regexReplaceString (regularExpression, replacementString)

Description: Returns a new String where the strings matching the regular expression are replaced by the replacement string.

Regular expression examples:

regexReplaceString("[0-9]", "X"): Will replace all instances of digits with a letter X

regexReplaceString(" {2,}", " "): Will replace all instances of multiple spaces with a single space

regexReplaceString("[aeiou]", "X"): Will replace all vowels with a letter X

regexReplaceString("[^aeiou]", "X"): The opposite of previous expression. That is it will match all non-vowel characters, thus it will replace all consonants with a letter X

regexReplaceString("[a-m]", "1"): Use of range expression. It will match all characters from a to m and replace them with character 1

String.matches

Syntax: matches ( regExp: String ) 

Description: If the regular expression matches the string, the function will return true, otherwise it will return false.

Learn more about Corticon