CEP as XTP, in SOA Environments

I have been tracking caching based application platforms such as Gigaspaces for sometime now, and I was surprised to see a new spin on these - as XTP (Extreme Transaction Processing) complimenting SOA. They are extending the caching capabilities to SOA solutions. Labeling this stateful SOA environment as a SOA Grid, and projected as an enabler to build next generation applications are in the areas of fraud detection, trade resolution, and risk management calculations away from the mainframe to low cost commodity hardware. Now, this is exactly the pitch from complex event processing (CEP) environments as well. And even CEP is seen as extending SOA infrastructure to support the capabilities to handle large volume event streams such a the banking or credit-card transaction events, needed for fraud detection.

At some level, the notion of "stateful" SOA grids proposed in the above paper is a contradiction of sorts. SOA is by definition discrete and loosely coupled services environment that are brought together in an orchestration environment to realize required business functions. Products such as Progress Sonic ESB do extend the orchestration capabilities to a more distributed environment (itineraries). Here one can see some value for a distributed state information (that caching products such as Giga and Terracota provide). But expecting this to provide the required capabilities for large volume event stream processing is a real stretch!

For starters, CEP is more than (surely, different from) a fast data-access/query mechanism, which is what caching solutions are. Caching solutions essentially extend the database processing model by providing a faster access to data. While CEP is less about data and much much more about quickly 'reacting' to events and detecting temporal patterns that occur in and across these large volume event streams, combining this within an SOA that can emit the events (based on the biz processing occurring within), or provide the required biz processing after the complex-event is detected, is in itself an XTP environment. It is capable of processing a very large volume of event streams and it needs to detect complex patterns in these streams. And, largely due to the unique approach used in the Progress Apama Correlator engine, it flips the conventional store-and-query approach (which is where caching products come in) to a look-ahead-and-react mode. This has already been proven in some of our successful Apama CEP deployments, such as Algorithmic Trading and Fraud Detection (the very same cases referred in above blog post). See the diagram below to see how Apama CEP works: