In many respects, I believe this argument is rooted in how some CEP vendor's have architected their product. Many vendors have a focus of event stream processing as a variant of database processing. They see streaming data as just a database in motion and therefore have designed and built their products myopically around that paradigm. By doing so those vendors (possibly inadvertently) have plotted a course where the fit-for-purpose of their products is focused on use-cases that are data-intake dominate. They can consume the fire-hose of data, filter, aggregate and enrich it temporally but little else. What is missing in this equation is the definition of the semantics of the application. Whether that is a custom application such as algo trading or monitoring telco subscribers or some business intelligence (BI) dashboard. To those vendors, that is viewed as adjunct or external (i.e. the client) and solved with the typical complement of technologies; java, C++, .NET and considered outside of the direct domain of CEP.
While this paradigm clearly does work, it incites the (CEP) critics; "where's the distinguishing characteristics? why can't I just do this with traditional data processing technologies?".
A challenging question when so many CEP products are stuck with that look and feel of a database, even a few of the academic projects I've reviewed seem to be myopically centered on this same paradigm. It reminds of that television commercial with the tag line: "Stay Within the Lines. The Lines Are Our Friends." (I think it was for an SUV). Quite frankly such thinking does not represent the real world. Time to think outside the box (or table as the case may be).
Yes, the real world is full of in motion entities, most often interacting with each other in some way. Cars and trucks careening down the freeway zigzag from one lane to another at high speed with the objective of reaching a destination in the shortest possible time without a collision. Would be an interesting CEP application to track and monitor the location and movement of such things.
In fact, location tracking is beginning to show signs of being a common use-case with the Apama platform. Not long ago we announced a new customer, Royal Dirkzwager that uses Apama to track ship movements in sea lanes. My colleagues Matt Rothera and David Olson recently published a webinar on maritime logistics. This webinar follows the same basic premise as the Royal Dirkzwager use-case, that of tracking the location of ships at sea. In fact, we aren't the only one seeing activity in location tracking, here's a similar use-case for CEP in location-based defense intelligence. The basic idea is the ability to track the movement of some entity, typically in relation to other entities, are they getting closer together (i.e. collision detection) or moving further apart (i.e. collision avoidance), are they moving at all? at what speed? will they reach a destination at an appropriate time? A CEP system needs, at it's core the ability to have both temporal and geospatial concepts to easily support this paradigm. Here's a handful of examples where this applies:
- Tracking ship movements at sea (as I mentioned with Royal Dirkzwager, and the Apama webinar on maritime logistics)
- Airplanes moving into and out of an airspace
- Baggage movement in an airport
- Delivery trucks en route to destinations
- Service-enabled mobile phones delivering content as people move through shopping and urban areas
- Men, machines and material moving on the battlefield
These are just a handful of location tracking use-cases for which the Apama platform is well suited.
Another colleague, Mark Scannell has written a location tracking tutorial that is part of the Apama Studio product. This is a great example that exemplifies the power of the Apama EPL for building location tracking CEP applications. The tutorial provides a narrative description explaining it's purpose and the implementation. Below I've included a small snippet of that example to highlight the elegant simplicity, yet powerful efficiency of the Apama EPL. If you're in need of a brief introduction to the Apama EPL, you can find that here, the beginning of a three part series on the language.
Location Tracking in the Apama EPL.
| // Track me - the tracked entity action trackMe() { // Call self again when new location is detected on LocationUpdate( id = me.id ):me { trackMe(); } // Detect a neighbor in our local area - // Increment number of neighbors that have been spotted // Decrement count of spotted neighbors after one second |
As a brief introduction, the Location Tracker tutorial is designed to track the movement of Entities (i.e. cars, ships, trucks, planes, or any of those things I listed above) in relation to other Entities within a coordinate system or grid. An entity is considered a neighbor if it is within 2 grid units (-1,+1) of any other entity. The grid and the units within the grid are largely irrelevantly for the syntactic definition of entity tracking. Their semantic meaning on the other hand, is within the context of a specific use-case (i.e. a shipping harbor, air space, battlefield, etc.).
From the tutorial I pulled a single action, trackMe, it contains the heart and soul of the tracking logic. As entities move they produce LocationUpdate events. The event contains the entities unique id and the X,Y coordinate of the new location. This trackMe action is designed to track their movement by monitoring LocationUpdate events. For each unique entity there is a spawned monitor instance (a running micro-thread so to speak) of this trackMe action.
The idea is that when an entity moves its new location is instantly compared against all other tracked entities (except of course itself, witness the recursive call to trackMe when id's match (id = me.id)) to determine if it has become a neighbor (remember the 2 grid units). This is elegantly implemented with the listener "on all LocationUpdate( x in [me.x - 1.0 : me.x + 1.0], ...". In a narrative sense, this can be read as "If the X,Y coordinate of this entities new location is within 2 grid units (-1.0, + 1.0) of me then identify it as a neighbor and update an alarm condition" (via a call to updateAlarm()).
This small bit of code (about 20 lines) exhibits an immensely powerful geospatial concept, the ability to track the movement of 100's, 1000's even 10,000's of entities against each other as they move, and of course this is accomplished with millisecond latency.
This small example demonstrates a few of characteristics of the Apama EPL, specifically that it is an integrated well-typed procedural language with event expressions. It allows you to code complex event conditions elegantly expressed in the middle of your procedural program. This allows you to focus on the logic of your application instead of just selecting the right event condition.
However to get a clear picture, the language of CEP is just one aspect of an overall platform. The Apama strategy has also been focused on a whole product principle, one where the whole is greater than the sum of the parts. As a mandate to our vision we have outlined four key defining characteristics: 1) A scalable, high performing Event Engine. 2) Tools for rapid creation of event processing applications supporting business and IT users. 3) Visualization technologies for rich interactive dashboards and 4) An Integration fabric for the rapid construction of high performance, robust adapters to bridge into the external world.
The possibility exists that CEP will struggle to break out as a truly unique technology platform when so many just see a variant of database technology. It's time to break out of the box, drive over the lines and succinctly answer the critics questions. CEP is not about tables, rows and columns but events. Events that are often artifacts of the real world. A world that is in constant motion, be it ships, planes, car, trucks, phones, or you and me. Information flows from it all in many forms but that does mean we have squeeze it into the database paradigm.
Once again thanks for reading,
Louie