Joe McKendrick's fairly recent posting pointed me to another installment of Greg the Architect talking about the analyst's "Mystic Grid" on SOA Management. Very funny.
Too bad what Greg the Architect is saying about SOA Management, while funny, is true. There are vendors in our space that have never once allowed their platform to be publicly benchmarked -- and with good reason: time after time we've seen their products fall over in anything but the simplest proof of concept.
Unfortunately, this leaves you, the customer, having to do your own benchmark comparison. Without independent analysis (your own, or that of others), you can't trust a vendor's word on their scalability. The scenarios they test against may be far different than the real-world scenarios you face -- if you don't benchmark vendors in your environment you could be making a big mistake.
Just one simple example of how you can go wrong... Let's say you know you are going to have 100 transactions per second that you need audited (e.g. for SOX compliance). The vendor might assure you that yes, they can audit 100 transactions per second. That should meet your needs, right?
Wrong. Let's say your transactions are composed of (on average) 6 separate tiers or services (e.g. two services each call three other services). In this case, for SOX compliance, you need to ensure that the vendor can audit 600 transactions per second since you actually need to audit every step of the transaction.
SOA What? The moral of the story is that if you are evaluating SOA management or runtime governance vendors, see if they can produce 3rd party objective evidence of how their product performs versus others. If they can't, make sure you budget enough time in your PoC to ensure that the vendor can meet both your current and future performance needs.