The ITKO Blog: Service Virtualization and Application Lifecycle Optimization - CA Technologies

New Paper on SOA Design & Validation Out from Linthicum

Our previous post referred to a paper that SOA analysts ZapThink just put out by David Linthicum. The paper (or "manual") is now downloadable from iTKO for free as well as from ZapThink .

Now, if you read last week's flurry of blogs and posts with Dave and John, et.al. about whether ESB is the bane of SOA, that is a philosophical question that makes you wonder if we see eye-to-eye with him -- but you need to vet out those kinds of practices in discussions to get to useful advice. That dialogue is what we love about blogging. We were talking about the challenges of leveraging multiple ESBs that will never go away in the largest companies and multi-vendor relationships, while Dave was simply stating that wouldn't be a best practice if you had a choice.

The manual/whitepaper: "Design & Validation in a Heterogeneous SOA" sounds like a mouthful but I found the content to be quite a good primer on service composition, and some of the ground rules to consider before using, or reusing, any particular piece of technology. Both SOA architects and the business stakeholders that are driving the practice should at least have a level set of what they are trying to accomplish, BEFORE talking shop about the merits of specific technology.

We couldn't agree more with this direction... it offers a good primer for teams who are growing their skills in SOA architecture or governance practices. Don't get locked in to one notion of SOA by a vendor or specific technology -- first compose and design around the business, plan to validate your approach as early as possible and test throughout the process, THEN rationalize what to service-enable or buy just what you really need for development and integration.

If you read this one, let us know what you "zap" think of it!

ESBs Cohabitating in SOA: Controversial?

I had a talk with Rich Seeley from SearchSOA a couple weeks ago, and his article: "SOA Complicated by ESB Proliferation" reflected how our discussion turned to what we're finding is an inevitable reality of SOA in large enterprises: the need for a team to manage and validate the Intermediation of different ESB and integration platforms.

Then this from our peer David Linthicum: "Are ESBs Hurting SOA?". Well usually we have seen eye-to-eye on many issues, especially since we are in the "platform neutral" space of needing to govern & validate the leading platforms on the market. In fact, we just sponsored a paper Dave authored "Testing & Validating Heterogeneous SOA" coming out soon that talks about how to design and validate these kinds of heterogeneous platforms. Dave is likewise not beholden to one ideal platform or solution.

Was I wrong this time on so many levels?
And who knew there was a controversy around ESBs in SOA?

That article spurred several comments for sure - and I had to put my own in there. We have also heard from a few more experts weighing in on the topic - Jeff Schneider didn't really see why it was a big deal in "Linthicum on ESBs," after all that kind of reality is likely what he is solving every day in his practice, while Loraine Lawson had a very good post "Multiple ESBs Causing Problem, but Nobody's Listening," supporting Dave's point that we need to do much better architecturally, and not just accept it as a reality.

Now to caveat, I was actually talking about it as a challenge -- I was never recommending multiple ESBs as an ideal approach - but it is certainly not out of reach to manage well with good intermediation practices. Companies putting all of their eggs in the basket of mediating strictly at the web services layer also run into a proliferation of different standards and metadata, or the "JBOWS" conundrum -- I addressed this recently in a brief -- and it is hard to dictate that compliance unless you have a very tight grip on standards, and for most large companies, the standards around services aren't consistent. So if you try to enforce too strict a compliance, you end up with poor reuse.

With multiple ESB platforms, you are still providing a very good way to bring underlying business applications and transaction systems to bear with an integration and messaging framework, sometimes with business process management as well. Yes they are different systems, but they can be pulled together effectively as long as the validation is there, and as long as the multiple teams have a means to virtualize their dependencies and continue developing and testing new functionality.

Obviously you would like to start with a clean slate and architect the system from the ground up with well-defined semantics, proper levels of service granularity, get partners delivering on the same standards, and realize the reuse and agility benefits you expect from having a very modular system. But, given the reality of acquisitions, partnerships, and the need to support ongoing businesses and their heterogeneous technology at consistent service levels, most larger companies need to find a way to get these systems to play together nicely.

We have to reconcile ourselves to this fact: the likelihood that there will only be one platform participating in business process execution will decrease, when there is a broader distribution of the business process. Departments may mandate homogeneity, but inter-enterprise transactions couldn't possibly do so. SOA is about abstractions that make disparate things work together.  Let's leverage this pattern to bring these systems together.

Now Dave just came back with a rebuttal in Real World SOA: "Madness I say!" - and yes, I do understand where he's coming from. An ESB does not a SOA make, and, yes, more centralized control would improve the situation. Still, I'm right on so many levels as well though... : ) We often take the tack of "SOA Federation," which is akin to the idea of Federal vs. State authority: a central (Federal) authority that dictates some ground rules and standards for interacting, but leaves much of the specific technology decisions such as which ESB to use, to the parties (or States) that provide and consume the shared services.

When you take SOA as a multi-enterprise workflow of a consortium of partners working together, centralized control is practically unattainable. You can't just dictate that partners follow your policies and your architectural decisions.

I’m a big fan of both leverage-and-extend, and iteration as a means to an end. I'd submit that the greatest IT value of a service-based architecture is that it enables these things. With them, I can be more responsive to business needs faster. I'm also too much of a realist to take a 'puritanical' view of how to build out an SOA.  We have to try and refine over time.  So I'd rather "grade" SOA on its level of maturity, not call it something else.  For example, ungoverned web services used within a tightly coupled team is an immature SOA; broadly distributed services leveraged by a variety of consumers with increasing development responsiveness is a mature SOA... 

If I am a business, I want the benefit of being uninterested in the implementation technologies deployed under the service interfaces I consume, so long as the governance and testability are there. Why preclude any technology from making its contribution to SOA, especially one that is doing fairly well at the job like ESBs?

I love a good back-and-forth on these kind of issues. I've talked to a few other bloggers/columnists since then, and I'm sure we haven't seen the last of this debate. To me it says that the practices and methods for SOA are still evolving, and there are a lot of good people passionate about working on these issues. We may never arrive at an "optimal" SOA but I do believe that IT will get better at addressing business needs if we keep signing up for solving the hardest problems while ensuring continuity today.

More comments on the ESB and SOA hullaballoo...

• Joe McKendrick is a little surprised ESB is this big a story as well... who knew?
• "So how SOA are ESBs?" Michael Meehan in Search SOA puts it in perspective.
• “ESBs on trial?” - David's third post this week on the topic (He's taking some fire, but maybe liking the attention?)
• “Businesses aren't machines, and enterprise architecture can't make them so” - Neil Ward-Dutton's commentary puts the business first...
• Dave makes his... Last ESB/Architecture Post, For Now
• Tony Baer in OnStrategies apparently missed the whole debate while on vacation...

How Can You Implement SOA without an Architecture?

Michael Meehan writes, in "Having trouble with SOA? Pay attention to the architecture," that he has never seen a successful SOA implementation story that did not feature an architecture.  He goes on to write, “If you can’t produce a representation of your architecture and then demonstrate that you’ve actually built systems in accordance with that architecture, then you can’t claim to be doing SOA.” As one commenter on Michael's blog wrote, it is asking the right question.

I would extend that you can substitute just about any enterprise technology for SOA in this statement, whether that is an ESB or even a Java server. I think it needs to be said, even if it is a "duh-yeah, of course" statement.   

I am constantly amazed that people think about implementing any complex technology without creating an architecture, but they do. At the risk of being corny, SOA without the architecture is SO? Michael goes on to say that the top SOA development challenge listed in their surveys is the lack of skills.

The demand for architects who can design SOA is greater than the supply.  In this kind of environment, you see a lot of both premium outsourcing, and training of the necessary skills for handling both changing business requirements and complex integration processes. We try to instill this with customers by training them on the paths of least risk and resistance in the architecture phase, so there is a better chance of delivered quality, while the skills are cultivated.

Unfortunately most architects don't get the luxury of designing an ideal solution, and they are inheriting "the sins of the father..." in their own heterogeneous environment. One of our goals is to try and make the testing and validation process simpler within the tools and offer virtualization to eliminate the constraints of dependency from much of the lifecycle. If those phases move faster, the architect has more time to focus on optimizing new features. However, an SOA tool does not reduce the need for strong architecture skills -- as it is in design and definition where the costliest errors will occur due to bad architectural decisions. And no toolset or platform is going to architect that requirement away from us.

We'll have plenty more to say on this topic. Your comments welcome.

Gartner SOA Design Patterns Call for Complex Testing and Validation Requirements

While we find many enterprise customers don't make the distinction between design patterns -- they either call it "enterprise integration" or SOA, yet there are some interesting ways to break down the patterns so you can look at your own unique situation in that light.

Tony Cook recently posted Gartner: SOA Design Patterns, on the service oriented enterprise blog.  He covers the five design patterns observed by Gartner analysts and comments on each.  They are:
1. Multichannel Applications
2. Composite Applications
3. Business Process Orchestration
4. Service Oriented Enterprise
5. Federated SOA

It strikes me how each of these raises the complexity bar for testing and validation. As Tony writes, Multichannel Applications separate back-end business logic from front-end logic and deliver full application functionality to a maximum number of users from various channels. This multi-channel delivery ups the validation requirements.  Composite applications are often drawing from multiple data sources that the company might tightly control, but those components are often created for very different business reasons. This is an area that requires ongoing validation to make sure the multiple sources stay on course - even if they are centrally managed.

Business process management (BPM) uses a metadata repository to manage the proceedings of the business process model at runtime. Implementation requires solid business logic testing. Tony writes that the SOA-based enterprise model is a step beyond composite applications, as all applications are perceived as components of one integrated whole. This integrated whole is created from reusable components, so the same ongoing validation requirements for composite applications come into play. Federated SOA logically splits the enterprise into semi-independent SOA domains. These domains are then federated to enable inter-domain sharing of services. Here is another scene for the necessity of ongoing validation. In this realm, we have found that actual US federal agencies are leading the charge -- they call it a "Net-Centric" approach of sharing and instilling Trust across agencies.

You can buck classification, but face it, if you are integrating multiple enterprise IT assets, you fall somewhere along the axis of either highly disparate (composite) use of controlled assets, or highly controlled (federated) integration of disparate assets. Either way, you want to increase your ability to deliver functionality better and faster amidst complexity and change.

Therefore, comprehensive testing and validation is a requirement -- otherwise, trust between the teams that build and control those Services is hard to come by.  As some of the contributing services may be in various stages of development, virtualization of service behaviors can also help speed things along by making the constrained or unavailable service available for this activity.

Software AG - Fourth in a Series on Software Community Blogs

This post on Software AG is the fourth in our awareness series on the community blogs of enterprise software firms who have products that iTKO customers are looking to test and validate.

First off, the company is part of an SOA community and accompanying platform we support, the Centrasite initiative, which consists of a SOA Registry/Repository, and other management tools. We offer LISA to CentraSite customers as a way to validate Policy so it can be directly enforced, as well as publishing some whitepapers and advice to the community at large.

Software AG has a community site with an invitation to join their Software AG blog. The blog says “Spanning a range of topics from emerging technologies to the latest trends in SOA, BPM and BAM, Software AG blogs chronicle the unvarnished thoughts of Software AG employees. Find out what company leaders, topic experts and guest bloggers think about the technology landscape of today - and tomorrow. Become a part of the conversation by reading and sharing your comments with us!”  While they do not allow non-employee blogs, like SAP and Oracle, you can make comments and join the conversation. They blog was just started in April 2008 and there are 6 posts on SOA governance so far and 5 on SOA. There are nine bloggers so far and their pictures are found in the right column.

The top post when I looked was WOA + SOA + POA = SOA, posted by (our associate since his days when Infravio was merged into webMethods, and then webM into Software AG), Miko Matsumura, Deputy CTO, Software AG with 294 visits. Miko quotes one of our favorite bloggers, Joe McKendrick, who cites Anne Thomas-Manes in defense of SOA success stories, "The issue, Anne points out, appears to be that SOA is still siloed within the IT organization, and the transformative effects have not yet spread to the business. Line of business managers, she observes, are not ready to share services, the crux of SOA value. Anne says so far, she has only come across one company that has been able to connect SOA success to business success, through the establishment of strong positive and negative incentives that encourage people to adopt a better attitude toward sharing."

Miko agrees that SOA is currently overly siloed in IT, and writes that there have to be ways to get the value expressed in business. He puts the answer in the larger context by discussing that SOA is really all of the following: Component Patterns (SOA), Process Patterns (POA), and Human Patterns (WOA). He concludes, “Since SOA is Seismological, consider that it is an inevitability, but only if you look at it in the geological time scale. WOA is sociological and therefore has a greater chance of establishing momentum in the short term. What remains to be seen is if WOA, SOA and POA emerge as a postmodern rethinking of the role of technology in business.”

Here's to breaking down the silos that Joe and Miko describe, and getting all teams to open up about the value they expect to receive, as well as the value they acheive.

I enjoyed browsing the Software AG blog posts. Their efforts are just starting but the quality and depth of posts is high.

SOA Governance is a Starting Point, not an End Game

Joe McKendrick recently reported in All Too Often, Governance is 'Retrofitted' into SOA that “surveys conducted here at ebizQ, most early SOA efforts do not have governance of any kind in place -- typically, organizations hold off until they have some type of critical mass of services before they consider it worth investing in governance.” We just talked about some useful governance thoughts from Brenda Michelson’s Does your SOA lack social skills?  Here she and Hub Vandervoort articulated some useful governance concepts but not doing it was not one of them (see The People Side of SOA post).

It is concerning that Joe’s surveys found that that only one out of seven companies currently have active governance efforts underway. Joe added that, “even among the most advanced sites surveyed, two out of three companies do not yet have comprehensive governance programs in place.” Joe also points to a Butler Group report, Firms That Under-Invest in SOA Governance Won't Reap SOA Rewards. The title speaks for itself.

There is a lot of effort going on to assemble services, but not enough to validate them, and that is the key to having any sense of trust or reliability. We recently talked about this in the Wring ROI out of your UDDI Registry/Repository by plugging in SOA Validation post. Why is this happening? I wrote, “a lot of people like the concept of Governance and having Policies, but it’s not necessarily practical all by itself for the way businesses construct and leverage applications. This happens because SOA Governance without Validation doesn’t provide any assurance that it’s actually meeting the business requirements set forth in the Policies. In essence, it is like posting a speed limit, but not having the radar gun to ensure that drivers are obeying the rules of the road.”

Being able to connect a UDDI registry/repository, where you store all of the metadata of your services, their locations, and their policies for use in one place, with a strong Validation strategy can be a big advantage in this environment. But to make it work, that validation process has to be continuous and ingrained into every phase of the lifecycle of your services.

Progress buys Mindreef: More SOA vendor consolidation?

If you are tracking the SOA vendors, you probably recently heard about Progress Software buying Mindreef (right after picking up IONA the day before as well). So let's take a minute to talk about iTKO's own view of the rapid moves and what it means for the market for software companies like us who are committed to making SOA a reality.

 

Clear message: SOAP testing isn't SOA testing

In one sense this represents consolidation, true, but in another sense, this proves that focusing on testing a single technology, such as web services doesn't provide enough coverage for the kinds of heterogeneous technologies that are inherent in today's SOA architectures, at least not enough for a company to survive on WSDL/SOAP testing alone. Mindreef was early to the market with web services testing, but over the past 5 years, we've seen that for every major enterprise, you also need to test different flavors of ESB, rich Web apps/RIAs, databases, EJB, CORBA, etc.

Dana Gardner blogged on the details of this acquisition recently, and talked a bit about his view of Mindreef in the light of WS-* testing alone:

"I took my first briefing with Mindreef, given their neighborly proximity, about three years ago. The seasoned team had a hit on their hands with SOAPscope, and their timing in the SOA market was great. But I’m not sure the company grew as was hoped, and perhaps the fast evolution of SOA beyond a WS-* emphasis played a role. SOAP hasn’t blossomed to quite the degree some people had forecast."

In fact we have always asserted that SOA Testing isn't a viable space on its own, unless it is tied in with Integration, Governance and Virtualization practices. So this kind of endgame is no different than what we've seen with Solstice for instance (which specialized in functionally testing webMethods ESB only), going to GreenHat (which tests TIBCO ESB); or Agitar (which bet on automating Java developer unit tests only) winding down operations. You need to be able to test and validate across every stage of the software lifecycle, and have the integrations and access in place to do so, across every technology layer to keep the lights blinking for critical SOA applications.

As SOA expert Joe McKendrick noted in yesterday's "Progress makes another SOA acquisition: Mindreef," Mindreef has a better home as a part of the broader solution that Progress (Actional) offers: "With IONA and Mindreef, Progress is clearly aiming to develop and offer an end-to-end suite of SOA products — from integration to management."

 

The open source market also commoditized some aspects of web services testing - as web services standards start to solidify, an open source tool like SOAPUI meets the needs of developers who don't mind coding some functional tests if they only concerned with testing WSDL/SOAP for compliance checks. This is also why we gave away a free web services only testing version of LISA for a year. And you know what we found? If a company is serious about SOA they will need to go deeper than web services.

 

Are times hard for iTKO with all this consolidation? Are you getting acquired?

• First of all, here at iTKO we are quite comfortable with our market leadership, awareness and growth. We are coming off a record quarter, record year-over-year growth, and we are reinvesting that in continued research and support, so LISA can be the most complete SOA testing, validation and virtualization framework on the market.
• Second, a BIG part of enabling quality and trust in SOA is in the ability to work with all the leading SOA platforms. Yes, we've seen a lot of consolidation lately, however our CUSTOMERS still have a heterogeneous mix of new and existing technologies to integrate and validate, and we are committed to helping them. So we will continue to partner with and drive quality and virtualization into all the leading SOA platform vendors - including TIBCO, SoftwareAG, SOA Software, SAP, IBM MQ, Oracle/BEA and more. Variety is a good thing for the agility and reuse we expected of SOA, but it presents a big risk. If we can help companies mitigate the risk of all that complexity at a lower cost, that is a good thing. 
• Third, we will support Progress customers - for instance we are still the only native Sonic ESB/integration testing solution out there, and we can interact with Actional SOA governance tools within the same framework. We can also Virtualize service environments for Progress customers, so they have a ready SOA test and dev environment without the constraints of access and dependency on a complex SOA environment - whether they use LISA to test, or try Mindreef tools. This coexistence is nothing new for us - many of our customers are also using HP/Mercury and Rational tools and processes and seek iTKO's help for SOA-specific challenges we address.

That's all for now - of course, we'll still keep up with Jeff Schneider's "SOA Consolidation" hit list to see if anything is happening we don't know about ;)

http://schneider.blogspot.com/2008/06/mindreef-acquired-by-progress-software.html