ESBs Cohabitating in SOA: Controversial?

I had a talk with Rich Seeley from SearchSOA a couple weeks ago, and his article: "SOA Complicated by ESB Proliferation" reflected how our discussion turned to what we're finding is an inevitable reality of SOA in large enterprises: the need for a team to manage and validate the Intermediation of different ESB and integration platforms.

Then this from our peer David Linthicum: "Are ESBs Hurting SOA?". Well usually we have seen eye-to-eye on many issues, especially since we are in the "platform neutral" space of needing to govern & validate the leading platforms on the market. In fact, we just sponsored a paper Dave authored "Testing & Validating Heterogeneous SOA" coming out soon that talks about how to design and validate these kinds of heterogeneous platforms. Dave is likewise not beholden to one ideal platform or solution.

Was I wrong this time on so many levels?
And who knew there was a controversy around ESBs in SOA?

That article spurred several comments for sure - and I had to put my own in there. We have also heard from a few more experts weighing in on the topic - Jeff Schneider didn't really see why it was a big deal in "Linthicum on ESBs," after all that kind of reality is likely what he is solving every day in his practice, while Loraine Lawson had a very good post "Multiple ESBs Causing Problem, but Nobody's Listening," supporting Dave's point that we need to do much better architecturally, and not just accept it as a reality.

Now to caveat, I was actually talking about it as a challenge -- I was never recommending multiple ESBs as an ideal approach - but it is certainly not out of reach to manage well with good intermediation practices. Companies putting all of their eggs in the basket of mediating strictly at the web services layer also run into a proliferation of different standards and metadata, or the "JBOWS" conundrum -- I addressed this recently in a brief -- and it is hard to dictate that compliance unless you have a very tight grip on standards, and for most large companies, the standards around services aren't consistent. So if you try to enforce too strict a compliance, you end up with poor reuse.

With multiple ESB platforms, you are still providing a very good way to bring underlying business applications and transaction systems to bear with an integration and messaging framework, sometimes with business process management as well. Yes they are different systems, but they can be pulled together effectively as long as the validation is there, and as long as the multiple teams have a means to virtualize their dependencies and continue developing and testing new functionality.

Obviously you would like to start with a clean slate and architect the system from the ground up with well-defined semantics, proper levels of service granularity, get partners delivering on the same standards, and realize the reuse and agility benefits you expect from having a very modular system. But, given the reality of acquisitions, partnerships, and the need to support ongoing businesses and their heterogeneous technology at consistent service levels, most larger companies need to find a way to get these systems to play together nicely.

We have to reconcile ourselves to this fact: the likelihood that there will only be one platform participating in business process execution will decrease, when there is a broader distribution of the business process. Departments may mandate homogeneity, but inter-enterprise transactions couldn't possibly do so. SOA is about abstractions that make disparate things work together.  Let's leverage this pattern to bring these systems together.

Now Dave just came back with a rebuttal in Real World SOA: "Madness I say!" - and yes, I do understand where he's coming from. An ESB does not a SOA make, and, yes, more centralized control would improve the situation. Still, I'm right on so many levels as well though... : ) We often take the tack of "SOA Federation," which is akin to the idea of Federal vs. State authority: a central (Federal) authority that dictates some ground rules and standards for interacting, but leaves much of the specific technology decisions such as which ESB to use, to the parties (or States) that provide and consume the shared services.

When you take SOA as a multi-enterprise workflow of a consortium of partners working together, centralized control is practically unattainable. You can't just dictate that partners follow your policies and your architectural decisions.

I’m a big fan of both leverage-and-extend, and iteration as a means to an end. I'd submit that the greatest IT value of a service-based architecture is that it enables these things. With them, I can be more responsive to business needs faster. I'm also too much of a realist to take a 'puritanical' view of how to build out an SOA.  We have to try and refine over time.  So I'd rather "grade" SOA on its level of maturity, not call it something else.  For example, ungoverned web services used within a tightly coupled team is an immature SOA; broadly distributed services leveraged by a variety of consumers with increasing development responsiveness is a mature SOA... 

If I am a business, I want the benefit of being uninterested in the implementation technologies deployed under the service interfaces I consume, so long as the governance and testability are there. Why preclude any technology from making its contribution to SOA, especially one that is doing fairly well at the job like ESBs?

I love a good back-and-forth on these kind of issues. I've talked to a few other bloggers/columnists since then, and I'm sure we haven't seen the last of this debate. To me it says that the practices and methods for SOA are still evolving, and there are a lot of good people passionate about working on these issues. We may never arrive at an "optimal" SOA but I do believe that IT will get better at addressing business needs if we keep signing up for solving the hardest problems while ensuring continuity today.

More comments on the ESB and SOA hullaballoo...

• Joe McKendrick is a little surprised ESB is this big a story as well... who knew?
• "So how SOA are ESBs?" Michael Meehan in Search SOA puts it in perspective.
• “ESBs on trial?” - David's third post this week on the topic (He's taking some fire, but maybe liking the attention?)
• “Businesses aren't machines, and enterprise architecture can't make them so” - Neil Ward-Dutton's commentary puts the business first...
• Dave makes his... Last ESB/Architecture Post, For Now
• Tony Baer in OnStrategies apparently missed the whole debate while on vacation...

Comments

[Neil Ward-Dutton]

Thanks for the trackback John - and nice post btw! I know where Dave is coming from, but from what I've seen the world just doesn't work like that apart from in the 1% edge cases.