Who Needs DW? Welcome, Real-time BI !

Last week I got a last minute invite to speak at the Silicon India BI conference (last minute for a good reason I guess... me & BI? Hmmm... :-) My database background notwithstanding!). Initially I resisted as couldn't think of anything of value, especially because the participants were expected to be architects and senior tech folks. As Pradeep (heads Silicon India) nudged, a thought crossed. I recently also came across a BAM use case where the kind of questions they wanted the solution to answer bordered on BI. And, in this solution they needed both SOA and Complex Event Processing (CEP).

Isn't using CEP for BAM in some form real-time BI? I deliberated for a few minutes - the CEP SOA synergies seemed to really make sense, so I agreed to speak. When asked for the title, I submitted "Design Approaches for Real-time BI".

BAM (business activity monitoring) is an extension of traditional business intelligence; adding event monitoring to scheduled, batch-based reporting. As I see it, BI can be of a few different types (when seen from the tainted SOA lens :-) ):

• Simple BAM is watch ‘as-is’. Most BAM software supports simple aggregation and watches for thresholds, et al.
• Complex BAM can watch for more complex patterns.

At some level isn't complex BAM BI? It provides insight and information that traditional OLTP systems cannot give. For BAM or BI, the discrete steps in the decision cycle are the same: Track/monitor; Analyze; Hypothesize/ model; Decide; and Act . The only difference being that BAM (however sophisticated) may not be able to answer adhoc queries and what-ifs. It can only answer pre-defined and pre-modeled situations. Nevertheless, this is BI. And it is real-time!

SOA and CEP form a good basis for realizing this. Conventional (read 'simple') BAMs may not suffice. BI is more than just thresholds or SLA violation alerts. One may need to detect a pattern involving a series of biz failures that could potentially cause a much serious eventual failure, which if detected and alerted in time, can be averted. This is where more complex pattern recognition come into play. The CEP engine. Given that we are talking about business events from all over the enterprise as the inputs, SOA infrastructure fits right in as the potential vehicle to deliver these events to the CEP engine.

SOA+CEP for real-time BI! An interesting proposition! And surely something traditional DW solutions cannot handle. DW & real-time? Nah. Now, can we extend this basic real-time BI to also include adhoc querying and such? Something to ponder about.