The ITKO Blog: Service Virtualization and Application Lifecycle Optimization - CA Technologies

Comparing the State of Consumer Web and Enterprise Applications

Here is an interesting post by Jeremy Chone in Bits and Buzz on Enterprise Web vs Consumer Web [2.0]: Top Six Differences. I was struck by this comment on one differences and thought of similar things we've seen in the field. Especially the fifth factor.  Here it is in Jeremy’s words.

"5) Integration (Loose vs Strict) ... In the consumer space, application integration happens very organically. Application providers (e.g., Twitter) or application container providers (i.e., Facebook) define  interfaces with which everybody can integrate. The result is that thousands if not hundreds of thousands of applications will potentially integrate with some of these interfaces. Consequently, interfaces tend to be simpler, lighter (e.g., REST), and often even client-centric (e.g., OpenSocial and iGoogle). In other words, consumer application integration is about breadth rather than depth. In the enterprise space, on the other hand, application integration has to be thoughtfully designed and managed and often has an impact in all layers of the application (i.e., presentation, logic, and data). Heavy and expensive technology, such as ETL, SOA, JMS, and Portal, are often an effective way to establish strong integration between applications."

Many of the current enterprise application efforts have been influenced by the consumer web (and you can call it 3rd party services, Cloud, SaaS, what have you).  However, success depends on recognizing the challenges and differences, as much as the opportunities and similarities. Much of the consumer web involves preparing for the unknown such as traffic volumes, as well as integration from multiple and unanticipated applications that you may not have control over, or be able to enforce an expected service level on.

While more is often known about the architecture in the enterprise world, the level complexity can go way up due to much deeper layers of system-to-system interaction.  There is also less tolerance for lapses in performance and data integrity.  This is where the edge between Consumer Web and enterprise applications gets particularly jagged, and where we are seeing the most innovation.

For instance, the need for software testing to be truly stateful and asynchronous as you cross over the edge between the enterprise and consumer web is a necessary thing (and quite a mouthful) -- if we are going to consume the best of both in our distributed apps.

How do you validate a transaction, if the response may move around among several third parties and decision points, and maintain its state when (and if) an appropriate response occurs? This kind of leap is going to take a lot of automation and innovation to allow the two to achieve scale.

Gartner Sees Expanding Role for SOA

I like the way this post, Gartner Says SOA Is Evolving Beyond Its Traditional Roots, starts out: "Despite the economic downturn, Gartner analysts said they do not anticipate a dramatic drop in SOA adoption."

It goes on to quote Jess Thompson, a Research VP we've enjoyed talking to at Gartner: 
"SOA underpins and enables initiatives aimed at cost reduction, such as SaaS or BPM, and we have noticed a growing interest around SOA from vertical industries like government and healthcare, which have been waiting for the technology and best practices to mature."

The post goes on to acknowledge that there are horror stories about SOA projects failing that have created a perception that the era of SOA is over. However, they add that the lack proper SOA governance is, and will be through 2010, the most common reason for SOA failures. This is something we have discussed a bit on this blog (see Avoiding SOA Disillusionment).

They conclude that quite contrary to going away SOA is actually emerging from the Trough of Disillusionment within Gartner's Hype Cycle and climbing the Slope of Enlightenment. It is good to see this trend in the field. Indeed, their previous commentary on SOA talked about fewer enterprises starting new SOA projects in 2008 - which is true - when you consider that any expansive IT department is already leveraging elements of an SOA model and it is quietly being used to create value.

Perfecting IT Strategies in a Down Economy

In today's tough economy, you can't even read an article that doesn't start out: "In today's tough economy..." This one is no different. Fortunately when it comes to IT, the message does not need to be accompanied with knee-jerk cost-cutting measures.

Last month Todd Biske offered some solid advice on what to do in today's down economy in his post, "Don’t Go On an IT Diet, Change Your Behavior". Needless to say, his post remains relevant. I especially liked his comments:

“The whole point of Enterprise Architecture, SOA, and many of these other strategic IT initiatives is to allow IT to be more agile- to respond more quickly to changes in the business objectives. Guess what? We’re in the middle of a big unprecedented change in our lifetime. My guess is that the best survivors of this meltdown will be organizations that don’t go on a starvation diet, but instead simply recognize that their priorities and goals have changed and execute without significant disruption to the way they utilize IT.”

Todd also wrote that cost cutting efforts to respond to the economy should not mean that organizations drop their SOA initiatives, or other long-term initiatives, especially when most of these efforts are trying to reduce costs cost by eliminating redundancy in the enterprise. It would be ironic as these efforts are exactly what are needed today.

SOA efforts should be more closely managed and held to their ROI objectives in any economy - otherwise, they will not have a chance to create the value you expected. It may be that closer review will actually improve the number of SOA efforts that are successful, as people will be more highly motivated to get them right. There is no time for poor execution now but there is a lot of room for innovation. As Jonathan Schwartz, the Sun COO, wrote in "Innovation Loves a Crisis," now is the time to make changes that will make you most adaptive to the new economy and best positioned to come out of the downturn on top.

McKinsey also supported this strategic thinking in their report, "Leading through uncertainty": “The task of business leaders must be to overcome the paralysis that dooms any organization, and to begin shaping the future.” Transforming to SOA - not because it's "cool" but because we have real value expectations, is one way to become more flexible and cost effective at the same time.

Accompanying that, we are also sure to mention Service Virtualization as a key means of increasing the availability of SOA assets in the lifecycle, at a small fraction of the environmental cost. It is money well spent not just in terms of cost savings, but more so in coming out of a lean time with a more agile and efficiently delivering dev and test organization.

ebizQ Discussion on SOA & Cloud Computing - Everyone's got a point...

The eBizQ site recently hosted a discussion is based on JP Morgenthal's feature article, Differing Views on SOA & Cloud Computing, and the question, which seemed to cause a bit of disagreement between the old "gray hair" and new "newbie" schools of IT, is: SOA & Cloud Computing new and unique or a continuation of the same old stuff?

Occasionally you come across a topic like this that touches off a flood of responses from the community. There are a range of insights from a "who's who" of SOA and IT pundits on this particular topic worth looking at on the ebizQ discussion site: SOA & Cloud Computing: New & Unique or Continuation of the Same Old Stuff?

Here was my response:

The future risk of the cloud is the current risk of SOA. I agree with the above statements about the hype factor of cloud computing, however there's still one more challenge that both strategies will share -- dependency. In SOA engagements, we encouraged the importance of good architecture and governance, but often it was just too easy for companies to hook up a couple web services to a legacy system for a quick ROI.

Dependency didn't rear its ugly head in SOA until the services started to be used by multiple customers, each with their own needs, and then those services were connected with the services of business partners, and used in a variety of unintended ways. A lack of access to dependent services and systems throughout the development lifecycle created a constraint on the agility we expected from SOA.

Similarly, the true cost and complexity of using cloud-based solutions will become evident only at the point when companies rely on them for critical needs. The low cost of entry of the Cloud's easy provisioning and pay-as-you-use can end up costing more than expected once the solution reaches business volume. If I pay-per-use for a cloud-based solution, I am establishing recurring costs, and increasing dependencies in my testing, development and release of new functionality that need to be addressed. Hopefully this shared concern of dependency will encourage enterprises not to sacrifice good architecture, governance, and virtualization practices for short term ROI gains.

We see many companies getting past the hype of SOA to generate real value today by applying the approach with more discipline, and many of the same lessons will apply to cloud computing.

Can You Lend Me A Shovel So I Can Start My SOA?

Joe McKendrick (aka, Joe the SOA guy), put on his economist hat and recently commented on stimulus ideas for the downturned economy in his post, Shovel-ready SOA. He writes, "perhaps we should look at service orienting as a shovel-ready 'stimulus' program for corporate growth. Because it works both ways — the current rough-and-tumble economy may be good for SOA, and SOA may be good for the economy."

Good idea Joe. As we wrote earlier, on one hand SOA is not dead or going away. We see success stories quietly happening all the time. On the other hand, like the shovel, SOA isn't glamorous anymore, but it's being used to create real value. Not only that, but we recently were musing on how SOA virtualization can help cut carbon emissions while it is saving money, hitting another aspect of the current stimulus suggestions.

Joe takes this discussion of SOA and the economy a step further as he comments that it can also help in "revitalizing business growth as well ... Consider the advantages properly implemented SOA can deliver: Increase confidence through greater transparency of applications and data, especially at a time when markets have been quaking from fear of what they don’t know; help businesses collaborate better through more rapid integration… help organizations expand or contract pieces of the business, as needed, without fuss and muss."

In tough times, nothing boosts your chances to survive, and be positioned to thrive, like enhanced confidence, collaboration, and agility through SOA efforts. We'll just have to roll up our sleeves.

Of course, before you start digging, a little upfront architecture and planning goes a long way -- you don't want to hit a wire or gas line! OK, enough, we'll put away the shovel metaphor for now. Each of these themes can be further enabled by the confidence, speed, and agility that automated testing, ongoing validation and Service Virtualization can bring to SOA.

More on SOA & Cloud Governance from HP via Dana Gardner

Dana Gardner recently posted on Enterprises can leverage cloud models and manage transition risks using service governance, says HP.

Dana had a conversation on the role of cloud computing, services oriented architecture (SOA) and governance in exploring cloud opportunities with Rebecca Lawson, Director of Service Management and Cloud Solutions at HP; Scott McClellan, Vice President and Chief Technologist of Scalable Computing and Infrastructure in HP’s Technology Solutions Group, and Norman Lindsey, Chief Architect for Flexible Computing Services at EDS, an HP company. Dana provides highlights in his post and you can also read a complete transcript of the discussion.

The panel pointed out that virtualization and the associate cost savings are not dependent on cloud computing, but they are an enabler of it. The panel focused more on the cloud than SOA, but they had a number of useful thoughts. I liked the panel’s comment that -- “In the longer term, the more overarching impact of cloud comes when your IT department can deliver value back to the business, rather than just taking cost out."

That is more than a novel idea - for the past few years we've talked too much about driving cost out of IT, instead of driving value into the business.

Some examples? "Using aspects of social networking and other aspects of cloud computing, and the fact that cloud is delivered over ubiquitous media, the Internet, to increase 'wallet share,' increase market share, maybe bring higher margin to a business, and build ecosystems, and drive user communities for a business."

This allows cloud computing to be strategic and not simply tactical. As cloud computing becomes more strategic, it becomes even more important to have a solid governance model to make the decision on whether to move from the traditional deployment model to a cloud-delivery model. In the process of getting to a service-centric IT governance model, organizations are going to have to deal with the governance model for deploying new services (see also – Dana’s post on IDC research shows enterprise SOA adoption deepens based on certain critical practices).

These governance models can better sort through the risk/reward trade-offs on cloud computing. Of course a good testing and validation strategy takes out some of the risk side of the equation. The panel added that tougher economic conditions would heighten the acceleration of cloud computing, and not just because of the opportunity to save cost (see also Direct Insights Analysts identify IT winners and losers in global economic downturn). The panel said that opportunities for the cloud to bring value are high when it can drive user communities with users and consumers of whatever it is your business produces, using techniques of social networking and other web-based capabilities. In these strategic cases that directly touch your customers, having continuous validation of services is especially critical.

For SOA Quality - there's just no free beer (Live from Gartner AADI)

Just exited a breakout session from Gartner's Tom Murphy, who is one of the main analysts covering the Testing side of our marketspace, as it edges into Agile development and ALM. His talk, "Application Testing and Quality in the Age of SOA" revisited a theme we haven't touched on in a while here: that in order to motivate and ensure good Reuse, you need Compensation.

Look at it this way: If I am a producer of a service - unless I'm a ramen-eating college student who just likes producing services for the glory of making the next cool widget, I am going to need some motivation to make that service as readily consumable and usable by my end customers as possible. So it falls to the Producer of a service to build quality into their service from the beginning.

Tom mentioned that SOA "offers a platform with a shifting foundation" so a Center of Excellence around SOA testing and quality is needed. The reward for good quality (and useful design as well) is Reuse, and that is compensated with real or "company store" budgetary dollars.

In SOA, quality is not as granular as checking the code for structural bugs or clicking on a UI to test at the end. It requires discipline and specialized tools that directly invoke and verify the services so they meet defined policies - structural, behavioral, performance, and that this process is carried out continuously throughout development and into runtime. That's a lot of responsibility if you have a well-used or critical service.

The part we often forget is the responsibility of the Consumer in this equation - If I am consuming a service, and its underlying architecture, there is no "free beer" here. I should be expected to reward the Producer that makes a service that has the most utility to my business. And in addition, I need to declare exactly how I am intending to use the service so I am not going to impact the promised service levels of other Consumers.

Check out this post from a couple years ago - "Consumption is not Free" is still one of our favorite themes and it's good to hear it again here at Gartner AADI.

Sizing the Opportunity for IT to Cut Carbon Emissions

A recent McKinsey report, How IT can cut carbon emissions, reported that the carbon footprint associated with IT, including laptops and PCs, data centers and computing networks, mobile phones, and telecommunications networks, could make them among the biggest greenhouse gas emitters by 2020. IT now accounts for 0.86 metric gigatons of emissions a year, or about 2 percent of the emissions added to the atmosphere globally. However, the McKinsey report predicted that the increasing need for IT could significantly drive up that figure.  They write, “by 2020, they (IT) will account for about 3 percent of all emissions: 1.54 metric gigatons, or twice what the United Kingdom produces today.” The report adds that, “the adoption and use of information and communications technologies in China, India, and other developing economies will account for much of this growth.”

As a Baylor alum I recently had a chance to attend a BBN talk with HP's CEO Mark Hurd, and he put it in an interesting light. Our hunger for accessing, using and storing data is increasing exponentially every year - and it is moving faster than Moore's law - the exponential increase in how much data our processing systems can effectively manage. That puts a lot of attention on innovating to reduce heat in high-powered processors, and the energy consumption associated with trying to keep it down. We will need to continue to innovate in reducing the energy cost of all of that data in the data center with more efficient systems.

On the good news side the McKinsey report said that there is the potential for IT products to help abate far more emissions in the general economy than their own production and use generates. The authors looked at things like smarter use of energy in buildings and greater efficiency in motors used in manufacturing. However, they also studied the possibility of reducing emissions by “dematerializing” physical goods and processes through telecommuting, video conferencing, Internet shopping, and downloading content rather than physical objects using paper, CDs, DVDs and found that there are great opportunities here also.  They concluded that information and communications technologies that promote abatement will “create attractive growth opportunities for those companies.”

This is good news as Virtualization (another way to say dematerializing) is a major opportunity for SOA to help with the global warming crisis. We have written about the opportunities for greater business efficiency with virtualization (e.g., Outlook Interview Part Four: Virtualization and Testing and  Can Virtual Environments take Performance & Load Testing?  This all leads to energy efficiency as less power is used by factors of ten or more when you can virtualize aspects of testing and validation. Here is another positive impact that may increase market demand for more environmentally friendly technologies.  Joe McKendrick recently discussed the cost saving side of SOA in his post, Service Oriented Architecture on $10 a day, and how it might do well in today's down economy. "Web services were forged in the dot-com era, but were, quite simply, pressed into service as a very low-cost way to work around very complex integration projects with reusable code."

Joe goes on to provide some good advice on SOA in a down market. This all can start by starting “with the smallest business process you can find that is the most inefficient, where the inefficiency is caused by an aspect of continuous change (a lack of agility) and the business is nevertheless forced to continue to invest in that inefficient business process.”

Of course all of this is an adjunct to the things we can personally do to reduce environmental impact. Especially in business - companies are now more compatible than ever with telecommuting, and "virtual meetings" which can have a dramatic impact on our carbon footprint.

SOA Governance Panel Recap: Federated SOA for the Day

iTKO's John Michelsen recently served on ebizQ's SOA Governance panel for a spirited online discussion. Joe McKendrick moderated the panel which also included: Anne Thomas Manes, vice president and research director with Burton Group, and former Chief Technology Officer at Systinet; Ron Schmelzer, managing partner with ZapThink; Frank Martinez, senior vice president, product strategy for SOA Software; David Bressler, SOA Evangelist for the Actional products at Progress Software; and Ed Horst, vice president of marketing and product strategy for AmberPoint. Joe provided both a summary, Panel: SOA Governance Answers 'What's in it for Me?' Question, and the full transcript: see Transcript: SOA Governance Panel.

Joe charged the panelists to comment on how far along we really are with SOA, and what is needed to take things to the next level. He also wanted them to be practical, and explore the tools and techniques that are essential to getting started with SOA governance. In addition the discussion focused on the implications of SOA governance for the "Web 2.0-ish" activities (like Cloud and WOA) that are often well under the radar of corporate IT management. Lastly, he wanted to cover one of our favorite topics, virtualization and SOA.

As Joe said in the summary, setting up proper governance seems like the logical approach to introduce SOA to the business. However, in the real world, SOA proponents run up against a series of obstacles. Many stakeholders have to be signed on to the SOA effort. These include developers who are expected to reuse someone else's code, departments that are expected to share assets and timelines, functional groups that are expected to provide more funding than usual to the IT underlying their processes with the hope for future cost savings and greater agility.

In response to these challenges John suggested that a Federated approach may be essential to SOA governance. In effect, people may be more willing to accept governance at  more decentralized levels, rather than more centralized approaches:

"We found that it is very analogous to the governance analogy in the physical world. That federating or layering of the governance policy is how it happens. If you think about it, my homeowners association, it wants to know where I'm going to put my flag. But at the Federal level.. that would be an absurd notion for them to decide such a thing. So we are more tolerant of greater control closest to us, than we are far away. So if there’s this very distant group of people who decide on how we are supposed to build and consume services, then we will tend to react negatively toward that. But if there is even.. a fairly onerous set of policies, but it’s shared, and its owned by me and my close constituents, I tend to be more willing to do it."

Later John added that since SOA has been around for a while, we know in advance a number of things that require planning, including aspects of the governance requirements. The solutions require more that installing infrastructure, they require an understanding of the desired business behaviors and how this infrastructure will support these behaviors.  He also said that there are two types of governance: what operates during design and what operates during run time as the transition to the new SOA services takes place and new complexities come up. 

While the panel was not able to cover virtualization like we hoped in the hour, they did manage to cover  other topics. The panel’s responses provide a rich resource on SOA Governance and make a good read (or listen). Here are the participants. They are left to right: Anne Thomas Manes,  Ronald Schmelzer, Frank Martinez, David Bressler, John Michelsen, Ed Horst, and Joe McKendrick.

Thanks for the questions and comments, and those of you who stopped by our "virtual booth" for more questions. We're looking forward to our next one at the "SOA in Action" virtual conference Nov 19th 2008, which has a who's who of SOA experts throughout the day. John is participating in the "Avoiding SOA Disillusionment" panel at 2:00 PM Eastern with Joe and a new cast.

Business and IT Alignment Regains Top Spot for CIO Concerns

The TotalCIO blog highlighted some of the results of the  recent Society for Information Management  survey of top CIOs. IT and business alignment, the Number 1 issue for four of the past 5 years, reclaimed the top spot after dropping to Number 2 last year. The IT talent issues that dominated year ago — attracting IT professionals and retaining IT professionals, both ranked Number 1 last year– slipped: retaining IT pros fell to the 8 spot, attracting talent to No. 4. IT strategic planning, 8th a year ago, is now up in the 3 spot this year, suggesting that hard times call for better planning.

The survey of 291 companies — 350 company executives — was done in June. In the TotalCIO blog post Linda Tucci wrote that, “it would be natural to conclude that economic realities have intruded on the profession.” However they quoted SIM’s Jerry Luftman, author of the survey, who cautioned against reading too much into the changes, noting that in the broad survey “there is no sense the sky is falling.”

It is nice to know that [at the time of this survey...] there is not panic at the CIO level and great to see a stronger concern with IT and business alignment and strategic planning. Of course, given some of the current grim news on financial shakeups and mismanagment, a little panic may intrude on the CIO's prospects, depending on their affiliation. However, this straw poll shows that many are thinking long term, and this should further open the door for better understanding the alignment of business rules with their validation in existing and planned software.

Thanks to Linda for giving us a preview of the result that will be fully released in November at the annual Society for Information Management meeting.

Outlook Interview Part One: Current Business Trends and SOA Testing

I was recently interviewed on the program:  Outlook Series with host Mike Lippis, and we covered a variety of SOA-related topics, from Testing -- all the way to Virtualization.

You can listen to the 45 minute podcast interview here, but our team has also broken the interview into five parts for easier reading. Here is part one -- which focuses on SOA Governance and how companies are attempting to use this discipline to meet  business goals.

Mike:  “Can you share with the audience what are some business trends that might be driving any kind of shift in IT environments currently?”

John: “The business is why we are here.  So what we need to do is to align our initiatives, as an IT organization, toward what the business is trying to accomplish.

“Among these (initiatives) are to increase the agility that we have to their customers in terms of responding to regulatory pressure…new market competitive pressure, etc.; to reduce cost…to go global…to either be acquiring or to become a part of larger organizations (M & A is becoming quite common, especially right now), and then, of course, compliance is over-riding all of this in that -- even though we’re increasing the speed of the train, we need to be more careful about how we execute along the path.”

Mike:  “What kind of IT initiatives would be driving that besides things like SOA and virtualization for example?”

John:  “Those are certainly two of them, as the business is trying to reduce cost, we need to increase our leverage and reuse, and so Service-Oriented is certainly one way to do that.  To increase our speed of development we need to take a more agile process, we need to leverage and extend technologies more often than bringing in new siloed systems.  Reducing costs often is reuse, but it is also outsourcing and off- shoring and virtualizing things.  So there is quite a bit going on in IT in order to accomplish these business objectives.”

Mike:  “Sounds like there’s a new IT environment driven a lot by SOA and Virtualization.  Could you describe some of the downsides associated with these new IT environments?”

John: “For all of the goodness that we are trying to bring about by increasing inter-connectedness to leverage and extend -- reusing existing assets, and the improvements we’re trying to provide by increasing our pace of change, (we’re releasing software updates to our business users more frequently than we were in the past) -- those changes are more interconnected with lots of different systems than they were in the past. While those are good things; the unfortunate outcome is that we’re increasing our risk of failure every time we increase the number of times we do something and every time we increase the complexity in which we do something, we necessarily have an increase in the possibility of failing, so we have to address that.”

Mike:  “So, it sounds like you’re really looking at minimizing business risk in terms of what your overall mission is at iTKO.  Could you kind of drill down into that a little bit for us to kind of see how you are addressing some of those issues?”

John:  “You bet - and that’s a good point Mike because we frequently get asked: ‘If you’re in the virtualization space why do you also offer testing?’  Well, because the real business we’re in is helping to resolve the risk associated with service-based architectures and agile processes, and more generally increasing complexity and increasing paper chains.  So what we try to do is:  if we can help enable these positive effects without the risk; we’re doing so to shorten the time to market, to reduce the failure rate in production, to increase our recovery time when there is a failure, and in general then to reduce the total cost of maintaining our own ESB systems that we produce.”

Mike:  “When it comes to SOA testing which is a big part of what you do, what are some of the challenges surrounding SOA testing?”

John:  “There are a number; the one that’s most common is that SOA increases the likelihood that we just can’t manually test -- we have an inadequate level of testing in many organizations from an automated standpoint and this becomes the governor on how frequently we can make changes to our system.  If I have to manually revalidate an application any time anything changes, then I can’t change it fast enough.”

Mike:  “I would think that today, based on what we’ve discussed so far, in terms of all the complexity, you really have a lot of heterogeneous environments and or most IT environments are very heterogeneous in terms of the technology that‘s now being used.  So how does this come into play with the SOA and what you are doing?”

John:  “Well it definitely scratches that problem quite a bit.  The reason it does is because:  Instead of thinking: my application is this completely self-contained box that I deliver to my users and they use it directly; rather, many teams are now building components of an integrated system and those components are, (as you say yourself) in heterogeneous technology, where there may be existing legacy CORBA applications and main frame applications and even mid-80’s mainframe messaging architectures, and modern messaging architectures, and all of these things being brought together for one application.

“So, we have a real challenge here when we try to test a system like this, we can’t just test each individual box independent of each other (although we should do some of this certainly) we must be able to test across the work flow as the business executes, so we must be able to test across these heterogeneous boundaries.”

Mike:  “You had mentioned earlier-on that IT initiatives, such as outsourcing and off- shoring for example, are very common in today’s global economy, how does this come into play with collaboration and to what extent does this frustrate collaboration; what are you doing in that area to make that a more seamless exercise?”

John:  “So the short version of a very long best practices story is that it necessitates collaboration, and it really does because:  what we can’t do is to have completely disjointed applications being developed -- we need collaboration.  Collaboration comes in a variety of ways.  One is we need visibility, the ability to touch all of these different technologies and view through all of these business processes all the way down into the implementation technology.  That’s just one simple way.

“Another way is we need governance in place that better creates a structure in which we all can predict how we are behaving.  The more reasonable governance is in place, the better we will all function and the better the trust there will be in this very disconnected world that we’re building and maintaining with that.  So there are a variety of things that collaboration encompasses but that’s certainly two of them.”

[After setting a little context, the next post picks up with a discussion of SOA testing best practices, our three C's. Stay tuned.]

Blowing Smoke? 3 Tests for the Cloud's Future

We couldn't help but notice how fast Cloud Computing has gone to the top of the hype cycle over the last month. You could say the Cloud is "the internet" in general and it has been there all along, and in fact it may have been referenced 10 or more years ago. This time it is new and improved, and it is really getting attention as a philosophy that is broader than SOA - you can't click twice in your IT news feed without sniffing this version of the Cloud.

Alas, this is probably the time that the big SaaS and internet platform providers like eBay and Amazon, Google and Salesforce, really come of age. Joe McKendrick recently put out some good coverage on this on his FastForward blog, "Cloud Computing: Uh Oh, Now It’s Getting Serious".

To us, this is not a mind-blowing concept -- it is a natural move for services, and applications to "live" in the cloud. However, there are 3 tests that Cloud Computing will need to overcome if it is indeed going to become the air we breathe:

1. There is no "free beer" for consumption of the Cloud.
Just like we saw a few years ago in SOA, there a perception that Cloud resources will be freely available for use in designing and building your architectures. But that is not a sustainable model. And no it's not going to be supported by advertising (on your iPhone for Facebook, etc.). If someone is offering an application that we rely on, they will need to ensure that they cover those support costs, and that the consumers are using the applications in ways that can be supported without unduly impacting other users of the cloud. Strong Validation of new offering and intended uses puts responsibility for reliability on both the producer, and consumer of Cloud resources. If there is healthy competition in the Cloud Computing market, the most reliable applications will win the most subscription dollars or usage fees.

2. There is still dependency, in fact, dependency may increase.
So if you are relying on the Cloud to provide data, transactions and other services, how can you test and validate your own business workflows and continue to evolve, if the Cloud may either not be available or ready to take your team's test and development transactions? Lack of ability to validate your own business applications against Cloud resources creates a huge bottleneck to productivity and agility. This is where behavioral Virtualization of the Cloud becomes of primary importance. If you could capture and simulate your expected interactions with the Cloud, that would eliminate a lot of the dependency and bring back the agility you expected.

3.  Fad resistance. (Uh, what were we talking about again?)
When the Cloud hype dissipates, how will this IT approach be driven by business, and not by the "coolness" of the technology? The world's leading companies will be taking more cautious wing-flaps into the Cloud - after all, they are still working on integrating and consolidating their most critical live applications for end customers. Vendors will continue to attempt to assert dominance over the definition of Cloud Computing - you might call it a shared Grid for a variety of apps, or a suite of apps running as a SaaS on a hosted provider's servers. Interoperability definitions and standards will be in flux. Nobody wants to be left behind when the definition and practice continues to change, and it certainly will continue to evolve.

We have been asked about this a lot lately, in terms of how to test, validate and virtualize Cloud Computing environments. And what we've found is that most of the same concepts that we saw in SOA apply to the Cloud - if you are going to rely on these services and apps for business, the risks have to be mitigated with continuous testing, and you need to isolate your own teams from dependency on the Cloud by virtualizing those behaviors. In this way, you can reap the benefits of ubiquitous availability of Cloud resources (services and data) as a platform for your apps, while still being able to productively design, test and develop (and keep working) when the cloud isn't available.

Anyway, this is just an initial take on the topic. We will be there at SYS-CON's first "cloud computing expo" in San Jose in November, and talking much more about how cutting edge companies are making the cloud work for business in the future.

Progress buys Mindreef: More SOA vendor consolidation?

If you are tracking the SOA vendors, you probably recently heard about Progress Software buying Mindreef (right after picking up IONA the day before as well). So let's take a minute to talk about iTKO's own view of the rapid moves and what it means for the market for software companies like us who are committed to making SOA a reality.

 

Clear message: SOAP testing isn't SOA testing

In one sense this represents consolidation, true, but in another sense, this proves that focusing on testing a single technology, such as web services doesn't provide enough coverage for the kinds of heterogeneous technologies that are inherent in today's SOA architectures, at least not enough for a company to survive on WSDL/SOAP testing alone. Mindreef was early to the market with web services testing, but over the past 5 years, we've seen that for every major enterprise, you also need to test different flavors of ESB, rich Web apps/RIAs, databases, EJB, CORBA, etc.

Dana Gardner blogged on the details of this acquisition recently, and talked a bit about his view of Mindreef in the light of WS-* testing alone:

"I took my first briefing with Mindreef, given their neighborly proximity, about three years ago. The seasoned team had a hit on their hands with SOAPscope, and their timing in the SOA market was great. But I’m not sure the company grew as was hoped, and perhaps the fast evolution of SOA beyond a WS-* emphasis played a role. SOAP hasn’t blossomed to quite the degree some people had forecast."

In fact we have always asserted that SOA Testing isn't a viable space on its own, unless it is tied in with Integration, Governance and Virtualization practices. So this kind of endgame is no different than what we've seen with Solstice for instance (which specialized in functionally testing webMethods ESB only), going to GreenHat (which tests TIBCO ESB); or Agitar (which bet on automating Java developer unit tests only) winding down operations. You need to be able to test and validate across every stage of the software lifecycle, and have the integrations and access in place to do so, across every technology layer to keep the lights blinking for critical SOA applications.

As SOA expert Joe McKendrick noted in yesterday's "Progress makes another SOA acquisition: Mindreef," Mindreef has a better home as a part of the broader solution that Progress (Actional) offers: "With IONA and Mindreef, Progress is clearly aiming to develop and offer an end-to-end suite of SOA products — from integration to management."

 

The open source market also commoditized some aspects of web services testing - as web services standards start to solidify, an open source tool like SOAPUI meets the needs of developers who don't mind coding some functional tests if they only concerned with testing WSDL/SOAP for compliance checks. This is also why we gave away a free web services only testing version of LISA for a year. And you know what we found? If a company is serious about SOA they will need to go deeper than web services.

 

Are times hard for iTKO with all this consolidation? Are you getting acquired?

• First of all, here at iTKO we are quite comfortable with our market leadership, awareness and growth. We are coming off a record quarter, record year-over-year growth, and we are reinvesting that in continued research and support, so LISA can be the most complete SOA testing, validation and virtualization framework on the market.
• Second, a BIG part of enabling quality and trust in SOA is in the ability to work with all the leading SOA platforms. Yes, we've seen a lot of consolidation lately, however our CUSTOMERS still have a heterogeneous mix of new and existing technologies to integrate and validate, and we are committed to helping them. So we will continue to partner with and drive quality and virtualization into all the leading SOA platform vendors - including TIBCO, SoftwareAG, SOA Software, SAP, IBM MQ, Oracle/BEA and more. Variety is a good thing for the agility and reuse we expected of SOA, but it presents a big risk. If we can help companies mitigate the risk of all that complexity at a lower cost, that is a good thing. 
• Third, we will support Progress customers - for instance we are still the only native Sonic ESB/integration testing solution out there, and we can interact with Actional SOA governance tools within the same framework. We can also Virtualize service environments for Progress customers, so they have a ready SOA test and dev environment without the constraints of access and dependency on a complex SOA environment - whether they use LISA to test, or try Mindreef tools. This coexistence is nothing new for us - many of our customers are also using HP/Mercury and Rational tools and processes and seek iTKO's help for SOA-specific challenges we address.

That's all for now - of course, we'll still keep up with Jeff Schneider's "SOA Consolidation" hit list to see if anything is happening we don't know about ;)

http://schneider.blogspot.com/2008/06/mindreef-acquired-by-progress-software.html

 

Enterprise 2.0 and Testing, Validation, Virtualization

Had a good long conversation with Bill Ives of the FastForward blogs a little while ago. Apart from sharing a lot of information on what is happening with LISA's capabilities these days, he took a little time to enlighten me (and also blog himself) on what Enterprise 2.0 is supposed to mean, and how SOA quality and service virtualization fits into that concept.

I like the mindset - E2.0 isn't a technology, however it encompasses most of the advances we've seen over the last 10 years - SaaS, RIAs, mashups, and the underpinnings like SOA and virtualization. But beyond that it is really about a shift in organizational structure around information. Better leverage and control over information, better collaboration and sharing, more innovation, more agility.

Anyway, interesting topic and worth a read, to see how all of this IT change is being driven by the business need for Enterprise 2.0-ready applications.

IT Predictions for 2008 -- Are Some Too Late?

Yes, there are many pundits proffering predictions on the coming year, but these are ours. In fielding these over lunch at iTKO, we couldn't help but notice a few of them would have been last year's predictions as well... but, even in IT, not all change happens as fast as you might have hoped or feared.

1. SOA and Virtualization will come together at last. In 2007, Virtualization really came of age for the enterprise, thanks to the immediate hardware and configuration costs it saves. The two disciplines have traditionally been considered separate disciplines, as it seems impossible to Virtualize a bunch of distributed Services and mainframe apps that don't live in a centralized data center. However, there will be new ways to start Virtualizing these components and business logic in 2008.

2. BPM and BAM will continue to become more intertwined, and BPM will finally emerge to become the linchpin behind enterprise SOA applications, as compliance to standards and integration issues start to be answered for leading enterprises.

3. Believe it or not, the US Government will emerge as a leader in SOA innovation. Many American firms will lag far behind Uncle Sam on the adoption curve come next New Year. However, on the business side, Europe will lead the enterprise charge for SOA in 2008.

4. Offshoring will continue to increase to meet a growing technical labor and talent crunch, but only the most savvy and collaborative SI firms will succeed and grow to meet this demand. The age of sending work to a "body shop" will fade as businesses realize that without strategy and quality, outsourcing doesn't generate much net benefit. (A holdover from 2007 that is still happening.)

5. Applications as a service, such as SalesForce Apex and Force.com, will emerge as a truly viable alternative for enterprise applications.

6. Many enterprises will stall on SOA efforts, until they give up on the strictly "top-down" architectural approach. (How about this year?)

7. Integration platforms will continue to consolidate, but their customers will have even more heterogeneous systems to manage, as they will be merging and consolidating disparate IT infrastructures as well. (Here's another one from last year, but it's still going on.)

8. Services will start to go mobile as "micro-services" residing on thousands of devices, with light status-based updates feeding into, and out of, a centralized architecture. (Not quite sure what Jason meant here...)

9. RIAs and SaaS will drive further innovation in how multiple enterprises collaborate at a semantic level across heterogeneous systems.

10. NetWeaver-based SOA infrastructures will start to become mainstream in a certain segment of the market.

Agree or disagree? We'd love to hear your thoughts and we'll see how many "I told you sos" we can collect out of 10 come next December!

Practice Makes Less Perfect? Not with SOA Best Practices

I’m a professional, likely you are too. We’d like to think we get better at our discipline over time. Gartner’s Allie Young is predicting that this will not be the case.

Our practice, like many other professions, must adjust to the changing dynamics of our field of endeavor. For us, the increasing interconnectedness of SOA and SOA-like applications creates challenges that are not new to the readers of this blog but are in the market as a whole.

Due to these new dynamics, Ms. Young’s predictions include the following:
- Overall application downtime is expected to increase significantly, and that increase is expected to come from application errors (as opposed to infrastructure issues for example).
- Through 2011, 75% of all IT organizations will have quality problems predominantly due to silo-based suboptimization.

Here’s one time when I think it would be a good idea to let some professional pride get in the way of the status quo. My hope is that we can help so many customers achieve what Ms.Young called “holistic, integrated quality management” that the numbers actually improve over time. Call me a nut, but that’s why I spend so much of my time on the road with practitioners and in conduits like this blog.

I am convinced, and we have more data show every day, that adopting a set of best practices around our SOA Quality Three C’s and taking a practical look at SOA Governance will allow you to become one of those who Ms. Young says will actually be distinguishing themselves as the top-tier with their process, quality, and service improvement capabilities.

See, she left us an out. We can be the leaders and still keep her honest ;)

SOA as a Marketplace

Consider that there is a cost reduction or profit motive to an SOA initiative. Also consider that I’ve already seen a financial incentive to those who successfully create reusable services that are leveraged throughout the organization. Before long, you realize that SOA is likely to create a competitive marketplace for services within an organization.

This marketplace will have most or all of the common attributes of any other marketplace, i.e.: promotion, profit incentive, and a winner and loser type mentality to those participants in the marketplace.

This is a good thing. The alternative is hope that altruism reigns throughout organizations and people make corporate decisions with no financial incentive to do so. Good luck!

Let’s walk through a scenario: Let’s say that I create a service that I wish to offer for reuse across the enterprise. In order to do so I must meet certain objectives, certain criteria that make it available for reuse. In doing so, I have to create additional investment before there’s ever any reuse possible. To justify this, my consumers are going to end up either directly or indirectly compensating me for the additional burden of creating that reusable service and at the very least maintaining their behavioral expectations on my services over time.

As soon as this occurs, (as soon as the producer is getting financially compensated for creating the reusable service) I’ve just reduced the cost of producing the service. There goes the marketplace.

(now I’m going to put on a divisional development director kind of hat…)

My goal becomes: To see how many globally reusable services I can mine from my requirements. I will begin promoting my development of those services. In fact, I’ll make sure that my teams will do the best job in producing those kinds of services so that I can cover the long term needs of the enterprise, thereby defraying the cost of producing these applications that have caused the construction of those services. If I’m better at it than the other guys across the organization, my services will tend to get leveraged more than theirs, and my budget indirectly grows.

The phenomenon here is simple enough: all it takes is to incent SOA is money. Since we can’t expect SOA to survive long-term with the concept of “free beer” reuse, or “anarchy,” we’re going to have more pay-for-use kinds of consumption of services.

Now this doesn’t mean that there’s likely to be some kind of transaction meter deployed in the infrastructure, or that consumers in one division will have to negotiate a price to buy those services, that’s not what I’m saying (this could be, but I doubt that it will happen in all but the rarest of cases). What I’m saying is that budget allocated to Division “A” to build an application that is at least partially consumed by using other services in Division “B” will create a horizontal shift in that budget.

If project “A” gets a million dollars to build a project, but $200,000 of that million goes to the producer in a different division to maintain an existing service, that is a net decrease in the budget of that project and a net increase in that other division’s budget. This will create that competitive marketplace.

Those who produce the best services and get them leveraged through reuse will become more critical to the business overall, and will be able to demand more budget; and will, in fact will be getting more budget. Those who do not over time will get smaller and smaller budgets. Even if the real dollars don’t change, the net dollars do - because there’s more shifting across the organization instead of direct build and maintenance of their own services.

One more way to look at it: If one division does a great job and three divisions do very poor jobs of producing reusable services, those other three divisions will ultimately become more business process level development teams while that first division will not only build their own applications but be the primary authors of all the core services. Thus the budgetary allocations will reflect that.

So when I mentioned this to Jason Bloomberg some time ago, he equated it to a “survival of the fittest” SOA mentality. I don’t think of it that way; I think it’s good old market dynamics. It’s healthy within an organization for those who do the best job to be empowered to do more and for those who don’t to reap their reward as well.