The ITKO Blog: Service Virtualization and Application Lifecycle Optimization - CA Technologies

Employing Virtual Environments for Manual Testers: It Still Doesn't Suck

Here's a brand new article in Virtualization Journal: Employing Virtual Environments for Manual Testers from one of our expert architects, Andy Nguyen.

Here at iTKO, we evangelize virtualizing and automating the entire software development lifecycle. While manual testing is a topic we don't cover as often - Virtualization of Manual Testing labs is a really cool thing that can add a lot of value in any large IT organization. For every development and test automation expert in an enterprise, there can be a whole team of on-site, partner, outsourced and seasonal testers sitting idle - waiting for access to a valid, configured test environment to run their scenarios against.

This article contained a cartoon - you might remember our distant history when we advertised the claim that "Your Testing Sucks" - and nobody ever disagreed with us. So here's the comic by Sal Monella:

Service Virtualization is the best way to solve the above constraint for Larry Lamprey, not buying more hardware or conducting laborious, expensive custom setups of a multi-tier environment behind the UI for testers.

Manual testing is never going to disappear, as someone will always be expected to test application UIs for user experience, etc. However, to eliminate downtime from testing, and conflicts over limited available test environments, ALL testers can use a Virtual Service Environment (VSE) to stay productive. Virtualization of those setups and data scenarios also delivers a huge productivity boost.

Why Stubs Don’t Work: Part 1 – Developers & Consumers

I’m frequently asked what the difference is between what we call a Virtual Service, and what developers have done for years, which is “stubbing” and/or “mocking out” a particular service. There are a number of ways I can address this, depending on who I’m talking to.

 First, for the Developer, I’m going down the path that Virtualization is now taking on the task of industrializing or productizing the practice of stubbing itself. Some 20 years ago, when I first started development, I wrote my own database, wrote my own development tools, my own middleware to get components to talk to each other – in fact, I used to code all kinds of things that today, I would never even dream of building anymore.

The requirements for proper databases and middleware became advanced, so that it was no longer feasible for developers to roll their own. Instead, those requirements became a category of software, and vendors started delivering solutions for that space, thus we saw the rise of a database market, and a middleware market, and so on. We see that the requirements for simulating components have become too steep to roll your own. So in that light, we see Service Virtualization as essentially the “space” that is coming out of the requirements of stubbing and mocking.

The second answer I would give, more often to the Consumer of a stub or mock is just how brittle these stubs or mocks have to be in order to be efficient. The whole reason why a team would construct their own stub is summarized as: “I don’t need a product for that, because it is something I can make quickly and easily by myself.”

Well if the goal is to do something quick and easy, the best you will be able to do with this artifact, is do a “round trip” to the stub, to prove connectivity of a particular application to that stub. In reality, there is so much more detail needed by the consumer of a service, that calling it “something you can mock up quickly” means you don’t understand the true nature of the problem.

Let me explain: If every customer response for a stub has the exact same profile, and the exact same address, account balance, etc. all with hard coded values and dates, then that may help the consumer of that stub with ONE data scenario. But, what about customers with high account balances? What about old invoices? What about transactions that occurred yesterday – and will they work the same tomorrow when that date becomes 2 days ago?

The problem with a stub that has static data, is that it won’t support the real variety of scenarios that are needed for a real world application. And you can’t just randomize the data – that would be even worse!  Consumers need predictability and control over the data they’re seeing. And they need to integrate that data not only from the simulated application, but from all the other services that might be associated with that particular application they are building.

A simple example: Say you have customer lookup in one service, while you have account management in another service. So if you have Account IDs and Product Codes and Balances that are not aligned properly between the two systems, you will have an application that fails, either at the consumer level, or somewhere in the downstream systems.

When you think about data required to support development in this scenario, stubbing a system simply doesn’t hold up. Considering all the things you would have to do to make that stub more real, you would need a pretty sophisticated framework to build those stubs. So you might start out building stubs, thinking “it’ll only take a few hours,” but you end up taxing your whole project plan with the hidden cost of building the first generation of a stub, and then modifying it over and over again. And by the way, they don’t even work very effectively.

This is part one of John's discussion on "Why Stubs Don't Work." Read part 2 of the blog, where he will discuss Performance Lab uses of stubs, and the "Tipping Point" for moving from stubs to Virtual Services.

Question: Are Pre-Prod Environments Balooning while Production Infrastructure Shrinks?

I recently posed a question to eBizQ and got some good answers, both reasonable and unexpected perspectives. See my eBizQ SOA forum question here: Are Servers Bloating in Pre-Production, but Shrinking in Production due to SOA?

K. Scott Morrison and Joe McKendrick discussed how Virtualization is a tester's best friend in reducing this phenomenon - and yes we have seen virtualization of hardware reducing some infrastructure needs. What about the ones you can't virtualize? I vote the best answer (received by 5:00PM CST) goes to Avi Rosenthal, whose comments highlight why this phenomenon is happening:

"In distributed Service Based, Virtualized and Cloud based environments the number of other environment servers is skyrocketing vs. the number of Production servers. This is true as well for virtualized server instances. The reason for that phenomenon is that Production environments are more controlled because no one can allow uncontrolled Production. The risks of non controlled Development or Test environments are less obvious. As the number of instances grows the number of servers in the non-controlled environments grows more rapidly than the number of servers included in the controlled environment. However, logically the number could be reduced at least to a ratio of 1:3 or 1:4 and not 1:20 if the non-Production environments will be managed and virtualized similar to the Production environment."

Really good point Avi.

However HP's Kelly Emo then came up with the gist of an issue I wanted to address further: the huge rift between measuring costs and effort for Production vs. Pre-production environments. It seems these activities are usually owned by two completely different departments and cost centers within the enterprise as she notes:

"In a majority of customers I speak with, their organization, culture and funding models are no where near having this happen... different silos are managing hardware and software resources and different SLAs are associated with them. In fact, in some cases, SOX compliance keeps organizations from being able to mix and mingle their networks and data between pre-production and operations. That said, I see one place where this can happen.. in the cloud..."

That's a next-generation view. Indeed the cloud is a big piece of the puzzle. However, many of the companies who get excited about the value of deployment savings using VMs and Cloud-based infrastructure to replace servers, simply do not have a reason to care about the number of groups maintaining and consuming dev, test and integration environments.

 The same goes vice-versa for pre-production - when you break up development into smaller, agile dev & test teams, those teams each provision and set up their own hardware and software environments in a largely unregulated fashion - and often find out that there are many dependencies they can't copy or replicate via conventional means. This leaves a lot of cost and effort unaccounted for in the wash of the software lifecycle, or as a cost of per-employee infrastructure or support.

You will see a lot more on this topic from us - while we aren't in the business of virtualizing servers and hardware, we are particularly curious about why companies aren't recognizing the value they get by optimizing Production vs. optimizing their infrastructure at every step along pre-production. Is a ballooning number of servers a side-effect of distributed, agile approaches, or is the symptom we are reporting simply a matter of perception? We'll keep you posted as we do more research in this area.

Webinar: No "free beer" in Service-Oriented and Cloud-based Applications?

We have harnessed the flexibility of readily available services, which can be assembled into business processes, and provisioned on Cloud-based, pay-as-you-need infrastructure. But despite our best expectations, there still is no "free beer" available in service-oriented and cloud-based applications for the enterprise.

What's raining on our parade? Someone has to pay to make sure that as we increase the number of distributed software components and processes, that the combined slurry of services will perform and function as expected by the business.

Face it, we are caught in a conundrum. While we can develop and assemble new business processes faster than ever from components, we also must deal with the fact that the rate of change and complexity of business software is only increasing. Every new service we connect to has the potential for failure. With that, the quality assurance effort and infrastructure cost of assuring reliable delivery of enterprise applications is also ballooning exponentially.

Let's turn the lights on, and discuss why the advent of SOA, BPM and Cloud Computing approaches, while a good thing on so many levels, also increases risk and cost. And maybe along the way, we will find some practices like robust automation and Service Virtualization that may get us closer to AFB (Almost Free Beer) -- readily available Virtual Test Environments, and increased reuse, so we can improve performance and quality with a much smaller incremental cost.

We are excited to present an upcoming "dual continent" webinar March 3rd for Europe and North America, featuring our partner Accenture's John McEvoy, head of EMEA testing, and our founder and Chief Geek, John Michelsen.

Webinar: "No Free Beer" in Modern and Cloud-Based Application Performance and Quality - http://www.itko.com/resources/webinar_accenture_mar2010.jsp.

Hope to see you on the webinar, and bring your questions for John and John.

Chalk Talks: Validation 101 & 102

For this week's Chalk Talk installment, founder John Michelsen revisits what Validation means to today's modern application lifecycle. It is a crucial part of any well-governed software architecture, one that is still too often neglected, once the project test and release cycle is complete.

This first installment covers the basics of Validation ("Validation 101"):

Next, John talks about how we apply this practice using LISA Validate:

We are always looking for more topics, customer stories and problems we can address, so if you have a question for us to answer in our upcoming Chalk Talks, please comment here or contact us.

Chalk Talk: Service Virtualization 101

Time for the next installment of our iTKO Chalk Talk Series. What is Service Virtualization and how is it unique from existing forms of hardware virtualization? iTKO's John Michelsen provides an updated primer on the practice of virtualizing the behavior of constrained systems.

iTKO Chalk Talk - Service Virtualization 101 - John Michelsen (3:30)

We have found a greater understanding in the IT community over the last year of how Service Virtualization is a next logical step after the value companies already achieve with Hardware and OS Virtualization. Service Virtualization solves for the kinds of applications you can't easily replicate and drop on a VM for use while developing your own apps - they could be third party Services that you don't have control or access to, "big iron" systems like mainframes and systems of record, and even components that are under development.

 

We're thankful for IT change and complexity in the 00's.

The challenges of change and complexity are what we are thankful for this season here at iTKO - because without them, we frankly wouldn't have much to talk about. Likely, if you are reading this blog, I wager that these two factors might be keeping you up at night. But the source of this change and complexity might not be what you expect.

Simplicity and agility in development, is quite literally becoming the cause of change and complexity in IT, especially in ensuring that a reliably functioning software environment is delivered on time. We have been covering this theme for years - but for some reason it has really turned on the light bulb for IT listeners this year, when pressures to deliver more functionality with less resources have been at their greatest. 

Enterprises tried to reduce release cycles and costs by focusing on one part of the lifecycle: development. They've brought in new development tools, techniques, methodologies, to make software development more agile, done in smaller, reusable units, more distributed, more iterative, with ever-shortening release cycles.

Companies have spent billions, in fact, on buying platforms to enable SOA, BPM, ESB, and legacy modernization approaches. It created a new problem however... development is getting faster, but the time and cost savings is more than eaten up by QA and maintenance -- and the overall release timeline is getting longer. This imbalance has continued to grow, and organizations can not afford to keep going down this path.

To resolve it, you have to focus on test AND delivery, to wit: How do I optimize the larger lifecycle to keep up with Complexity and Change in my software? We can't afford the support nightmare of software failures in production, but if we throw resources and infrastructure at testing to keep up with all the changes, our costs and timelines, will balloon.

This is why companies are turning to Service Virtualization as a way to alleviate the huge cost of having a valid system environment available for testing, validation and performance purposes. We are thankful to be here, working to solve this problem, at a time of so much change and complexity.

Hear what some other musing friends feel thankful for on the eBizQ forum here. I personally have also been thankful to see less top 10 predictions, and more things to be thankful for, as we finally head out of what has been a tough decade for many businesses. Here's to a very successful 10's!

Thoughts from Gartner AADI 2009 Summit - SOA's still on.

I recently returned from the Gartner AADI conference in Las Vegas, and had the chance to enjoy dozens of good conversations with IT execs who were there primarily to explore strategies for the coming year. Some thoughts from the conference and sessions I was able to catch:

• Budget optimism... Overall it seemed that there was a guarded optimism for re-invigorated architecture and integration projects and budgets in 2010, many people said they were part of a new group or new reporting structure specifically tasked with defining a strategy.
• SOA Won't Die - in fact any lead architect or CIO I caught up with says they are quietly going about service-orientation and getting real value from it, even if it's not sexy anymore. After the age of boundless service proliferation earlier in the decade, it seems that much of the talk is now about authority - who owns and manages these services, and how are they compensated for doing so. Recently a lot of the "SOA elite" posted some good discussion on this topic (check out this eBizQ SOA forum here).
• Heterogeneity is king. The optimism for integration is tempered by a big dose of skepticism about "one stop shop" vendor claims - even more so than you normally hear from the analysts. Companies are dealing with the realities of multiple acquired systems from different vendors, and looking for ways to modernize their applications.
• People understand the distinction of service virtualization. Another pleasant surprise was the understanding shown for hardware/network virtualization vs. the practice of Service or behavioral virtualization of constrained assets that simply can't be captured or dropped on a VM. I believe the analyst community has talked to application layer customers about how these assets can be brought to bear earlier into the lifecycle, even if this practice doesn't exactly fit under defined research fields like software quality, app dev, virtualization and IT ops.
• Cloud Computing still uncommon in large-scale enterprises - There was a great deal of content about Cloud-based approaches at the summit, and it is certainly important to get some forward insight. In practical terms however I did not see these $500M+ company IT execs mention Cloud in their basket of must-do items for 2010, other than a notable exception of specifically wanting trading partners to use the Cloud more to ease cost and access issues. (This reminds me of public transportation, "it would be great if everyone else would do it to free up the road...") Certainly a lot of curiosity and predictions around this. I am sure at a less enterprise software heavy conference you would see more cloud practitioners and this will continue to grow.

In any case it was a productive time at Caesar's, and we will discuss some topics we touched on this week at a later time.

Virtual Data in the Clouds - Enabling Remote Testing Teams

We had a chance to briefly touch on this topic in our founder John's latest webinar with Infosys' Aparna Sharma on "Accelerating Modern Application Quality" -- archive available now.

A question came in during the show that was particularly pertinent to the topic at hand. How can these virtualized services and data models work in the Cloud to allow collaboration with distributed teams, especially offshore test teams? Indeed, without a good approach for virtualizing data in a Cloud-based development environment, you simply cannot get there without wasting about 80% of your effort and cost in testing today's service-based applications. 

You expected ubiquitous, flexible access to needed resources with Cloud Computing. So why can testing teams end up sitting on the bench instead of getting involved as early as possible? Much of the challenge lies within the test data itself.

In short there are "Wires hanging out of your Test Cloud" as we illustrate here - especially when it comes to Data. You can't just take a 4TB database or an external SaaS-type data provider and plop them into a hypervisor and provision them in the Cloud. Test Data is a frontier because it is not easy to replicate via known means - it is either too bulky or too dynamic to reliably move to the Cloud, but we absolutely must provision teams with good test data, in order to do enough testing to rely upon these systems for business.

Primarily there is the problem of valid data access needed for thorough testing. Most often, this is locked up in some mainframe or live service that is simply too critical to open up to the testing team. Since business continuity usually trumps new product testing, IT Ops guys will crack down on test access time windows to the live systems and the data within them. So you might have a 2 hour period of access - barely enough time to test an on-premise system, much less a Cloud-based system that is talking to other data providers.

Second, there is the problem of data sensitivity. Even if we have contracts and a good working relationship in place, we don't want to be putting data that needs to stay private in the hands of any third party. So we do need to mask or "desensitize" the data so it follows the expected structure, without BEING someone's social security number or bank account. This has always been a concern in healthcare for instance (see HIPAA guidelines), but the level of certification, compliance and controls around data privacy and security in every industry is only getting tighter. We need to be able to give these remote teams obfuscated data so they can continue testing efficiently.

Third, data setup and teardown continues to be difficult in Cloud environments. We touched on one of our favorite examples in the webinar, a Telco that literally spends 2 hours running tests, and 2 whole DAYS resetting data. Many of these problems wont go away when the Cloud-provisioned system is talking to external dependencies and systems of record that need to be accessed and later cleaned out of running systems. Quite often this is a main point of contention, as a test could corrupt the tests of other teams or even the live systems if not contained correctly.

These are just three concerns. There are many other open issues of how Test Data will be handled when we move to Cloud-based environments. I thought Judith Hurwitz nailed this as one of her observations in her "What are the Unanticipated Consequences of Cloud Computing" post:

"Data will increasingly be seen as a reusable resource that can be used in lots of different situations. There will continue to be strategic line of business applications but they will be more systems of record that keep track of the final result of actions that take place dynamically in the cloud. The value of data is not in its tight packaging as we have been used to for decades but it the flexibility to move, transform, and leverage data. The watch word for data in this new model will be Trusted Data in the Cloud."

Couldn't agree more with this observation. As we move to ever more distributed, flexible computing models, we are supporting these with very distributed development and testing teams. The data needed for the software lifecycle must be shared among business partners, on shore and off-shore teams, so success will all come back to our level of Trust that the Cloud will give us the data we need, and not give us the side effects of data we don't need.

Of course, here at iTKO, we've been stretching out our LISA Virtualize to better support Test Data Management (TDM) solution efforts with the capture, manipulation, dynamic "desensitization" and Virtualization of data used in Cloud-based development environments. Look for more discussion and research from us soon on how expert testing teams are taking advantage of Virtual Test Data in the cloud.

Webinar 10/27 w/Infosys & ITKO - Accelerating Modern Application Quality

You might say we are between a pillow and a hard place right now when it comes to modern applications. On one side, we have killer new development approaches such as service orientation, BPM, and SaaS that provide us with a much greater amount of flexibility and speed in assembling reusable components and services. Development time and costs have indeed gone down, while functionality gets ever more sophisticated.

However the unintended consequence of being able to build and change applications so quickly comes at a price. Testing and validating these systems has become harder than ever. So many interconnected, interdependent components means an exponential increase in potential points of failure, and an inability to have access to a stable and affordable test environment. So much so, that testing environment and error repair costs are outstripping development cost in many cases.

There's gotta be a way out of this situation. Fortunately, we have some experts ready to help shed some light on these new costs and track down the quality issues in modern applications. We are privileged to have our founder/"Chief Geek" John Michelsen, presenting with Infosys' head of Client Services, Aparna Sharma on this upcoming session:

Webinar with Infosys and iTKO: "Accelerating Modern Application Quality: Testing Strategies for Reducing Risk and Cost in Today's Distributed Application Environment"
Tuesday, October 27th, 2009 – 1:00 PM Eastern (US) Time

Between these two, you have a great deal of experience leading and advising test and development teams in hundreds of engagements with the most complex types of software environments. So I expect nothing less than finding out what works today (and what didn't work) for real enterprises. After all, time-to-market is measured by delivery success, not the rate at which developers can string together new (but untested) functionality.  Hope to see you there, and bring your questions!

Finding Hidden Value in Test Data Virtualization

As you've seen in recent posts here, most of our discussions around Service Virtualization center around the lack of availability of systems, environments, etc. -– i.e. the "Over-utilized" problem. A Virtual Service Environment (VSE) is a solution that helps eliminate these constraints and therefore can dramatically reduce the time to market, and/or infrastructure costs. Also, once these constraints are removed, testing (functional and performance) can be done much earlier, and therefore greatly improve the quality of the released software.

We also talk about the use of VSE to enable parallel development. Being able to simulate dependencies that don't exist, and allowing testing and development activities to happen in parallel. Again, this can greatly reduce time to market, costs and improve overall quality.

However, there is a value of VSE that we don't talk as much about. An issue that has applicability in almost every enterprise environment, is one related to Test Data challenges. Let me explain.

There are at least 4 Test Data related challenges that Service Virtualization helps solve or assuage:

1. Data Scenarios are difficult to create. In the modern application, data required to support scenarios will have to be synchronized across multiple systems. Not only is that highly time consuming, but due to access constraints, or skill constraints, or various other reasons, creating and synchronizing that data could become a near impossible task. This may either elongate the testing cycle or, even worse, encourage the testers to reduce the number of scenarios that the system is tested under.
2. Data is Volatile. This is partially related to (1). In a lot of cases, the systems being tested go directly against systems that are either in-production or are being shared by other teams. This causes the data to be volatile (changing randomly and often). This makes the testing process impossible, because any results of the testing against these volatile systems cannot be validated.
3. Data needs to be Desensitized. In a lot of cases, data challenges could be reduced by using production data. However, production data may have sensitive information like credit card numbers, social security numbers, etc. Not only is this a problem for the direct employees, the challenge is greatly magnified when offshore or outsourcing contractors are used.
4. Data should not impact production systems. Some test scenarios result in transactions being created that can mess up a production system. For instance, if a test scenario requires a new order getting placed, the order once generated will have to be traced and deleted from the production systems. This is a major pain for the testers, and for the operations personnel.

As you can see from the above examples, Test Data Management challenges represent a huge constraint on the productivity of almost any enterprise software development project of significant scope.

Here's how we have found customers can approach these challenges using Service Virtualization:

• Reduce the number of points that need to be synchronized, by capturing relevant transactions, chaining them together into scenarios, and creating a virtual model that can be manipulated without programming.
• Stabilize the data so that the test results are actually valid, for instance, maintaining the relevance of variables like changing dates and varying types of customer data entry.
• Desensitize the data so it does not violate security and privacy policies.
• Substitute a Virtual Service for the production system, so there is no need to impact the critical live system of record to retrieve data. This can be done alongside ALM tools and Service Registries for SOA - essentially routing a known test data request to the VSE instead of the real thing.

In this arena of virtualization, we have been amazed by how widespread these shortcomings are in test data management, with little recognition of any available solution for them. We believe our LISA Virtualize uniquely supports these practices, with automation for capturing, modeling, desensitizing and maintaining virtual test data.

Webinar 9/23 with Perficient's Eric Roch: Addressing SOA Quality Gaps

Many of our initial customer discussions focus on assessing the "health" of your SOA and Integration efforts. How do you measure health in the distributed software applications like we see in every major enterprise? It's not just about measuring application performance and response time.

There are many details such as functional accuracy, time per release, and availability for testing that directly impact how healthy the IT environment will remain over time.

You may remember last year's blockbuster "ESB-Test-Fest" webinar with a leader from one of our key integration partner leaders and the author of "The SOA Blog," Perficient's Chief Technologist Eric Roch. Well now we have another one coming up that takes the next step into testing and validating the entire SOA application lifecycle with Eric and iTKO's co-Founder and CTO Ruston Vickers - two guys with as much practice experience solving the deepest integration quality challenges as you'll find.

Webinar: "Overcoming SOA Integration Testing Challenges and Other Critical Gaps to SOA Quality and Efficiency" on iTKO.com

September 23, 2009 - 1:00-2:00 PM Central
with Eric Roch, Chief Technologist, Perficient, and Ruston Vickers, CTO & Founder, iTKO.

Bring along your questions to this one-hour session and hope to see you there.

Virtualizing Over-Utilized Systems: Part 3 - Common Bottlenecks

This is the third post in our series based on the new whitepaper "The Next Frontier for Virtualization: Over-Utilized Systems" by iTKO (Ahrens, English). In this installment, we will break off one of the big 3 challenges of Over-Utilization: the "Common Bottleneck" that constrains multiple teams attempting to develop, test and integrate enterprise software. Conventional virtualization can break under the strain of attempting to eliminate these bottlenecks, as they are often too large or too complex to be replicated on a given VM.

[Nifty picture I took in Tierra del Fuego, Argentina - constant 50mph west wind]
Well-respected enterprise application design patterns are actually causing software development lifecycle bottlenecks where numerous teams rely on a key system or technology.  These systems become new constraints to development and testing, and push the original goal of time and cost savings further out of reach. Here's the key points where we find Common Bottlenecks:

Design of Core Services – In some Enterprise IT organizations, a few key services are so fundamental that they are used in a majority of projects.  From the standpoint of re-use this is exactly what we were looking for, but while designing, testing, and running our applications, these core services create dependencies which slow teams down, because unlimited access for software teams is often considered too risky.

Examples of core services include:

o        Customer/Account – An example of one of the most commonly created core services, so there is a single place to get customer records.

o        Billing – Customers don’t want to receive multiple bills from the same vendor, so this is also a common core service to create.

o        ERP System – Especially for internal purposes (HR, Financials, etc.) companies adopt a single platform and require everyone to integrate.

Customer Example:

A large North American Telco had issues in replicating their core SAP system across a myriad of environments.  Only 3 of 7 non-production environments have an SAP instance they can use. Also the training system does not have a copy of SAP, so they cannot train their Customer Service Reps on applications that integrate with SAP until they are rolled into production. The result is that bugs are not found until it is too late to repair them and still deliver projects on time.

Data Warehouse – Numerous services depend upon the same set of data, especially for account information. In order to make data available for development purposes, this data usually comes from a production “refresh” as needed, and is put into pre-production.

• Data setup & teardown – Managing relevant, stable and secure data in the environment can become a huge drain on team productivity, sometimes consuming 60% or more of testing time.
  
• Data volatility - When data is treated as community resource for multiple development and test teams, other teams can pollute the test data (for instance deleting a key user required for your regression test suite).
• Secure data - In addition, live data must be suitably scrubbed or “desensitized” to protect unauthorized tester access to account information, which is a time-consuming process.

Cloud Services – Saving money by using external service vendors on a pay-per-use basis is great, but not when they get in the way of being able to do testing and development. This means we have another team we need to coordinate with from a data standpoint (test accounts, etc.) and potentially we are charged a usage fee when calling on these services.

Incremental fees are fine if they are a part of a revenue-generating customer activity in the live application, but they are a problem when they become a development cost to the business. For instance, how can we avoid skyrocketing fees for load tests that include a cloud service, or analyze negative ramifications if the cloud service doesn’t meet their SLA?

Customer Example: Insurance in the Cloud

A regional insurance company needed to be cautious about how they run their performance tests for adding new auto insurance customers. Among other things, the new client application performs a credit check and runs an automobile history report.  During one of their routine load tests against the new app, they sent actual requests for automobile history reports to a third party, resulted in a $15,000 access charge. After this unexpected charge they simply chose not to test any third-party service calls until they reach production.

Next installment: Limited Capacity and Other Inefficiencies

 

On-Demand App Dev Strategies with Compuworld's Mark Hall

I recently had a chance to catch up with Mark W. Hall, an analyst and writer for Compuworld. Mark was especially interested in the concept of bringing development and testing environments into readiness in his blog "On Demand App/Dev? Not yet, but closer."

Mark talked to several vendors that are in application quality related fields, as this can often be an underdeveloped aspect of development. In other words, we are now take an extremely "agile" approach to developing new functionality, and the quality or ability to test all of the components and interconnectedness in an application's environment just hasn't kept up.

Here's an excerpt of Mark's article:

"At iTKO Inc. in Dallas the company's LISA product lets you tests services (in the cloud or in your data center) virtually. Chief Geek John Michelsen says LISA watches how your app interacts with a given service over time, then it creates a virtual instance of the service with all of its behavioral quirks and features that you can test as often as you like. Michelsen claims it is unrealistic for a developer to test against a live service."

[Just as a clarification - I believe we definitely must enable developers to test live services for checking implementation - however I do believe developers and testers cannot get access to the production environment to do so 95% of the time they really need it, especially if it is already handling live customer transactions...]

Mark also asked about hosting these virtual environments in the Cloud, and a resounding YES, we already see Cloud providers and partners offering up Virtual Services capabilities, playing nicely alongside other virtual servers and machines that make up a well-rounded test environment. Look for more from us on this topic soon.

Whitepaper: Demystifying Cloud-Based Testing and the Related Benefits

You might have noticed a lot of commentary about Cloud Computing recently on this blog, and it is not a coincidence. In our conversations with analysts, peers and of course customers in the field, we are frequently asked about how this approach can add value to the testing and virtualization practices we support.

We have just put out an iTKO whitepaper on this topic, titled "Demystifying Cloud-Based Testing and the Related Benefits." The PDF is free to download.

There are many hard lessons and new techniques learned from industry-wide attempts to decouple and flexibly assemble enterprise software around key business processes. In order to reduce risk as we move into Cloud environments, we need systematic testing that gets around the roadblocks of inaccessibility and unexpected costs. We need to make use of automation and virtualization of the extended environment as much as possible. Cloud-based services need to be governed and validated as part of a continuous business process.

At this stage of the Cloud's development, there are many vendors and service providers that can contribute value to a comprehensive deployment and development environment solution. This short paper does not seek to recommend a comprehensive solution set, rather it focuses on gaining value within the software testing lifecycle. It outlines the key challenges and the top 4 strategies in play - first steps for virtualizing and testing distributed assets in the Cloud. We hope you find the information useful, and invite your feedback.

Testing Cloud Performance Strategies: Mike Kavis

Read a recent article from a very active pundit and practitioner in the Cloud Computing space, Mike Kavis, titled "Architecting the Cloud: Testing performance in your cloud application."

In the article, Mike advocates a layered approach to testing cloud-deployed applications that makes a lot of sense, in terms of the kinds of performance testing you can actually do while the design and development lifecycle is going on. To summarize, this involves testing from the system roots up while the solution is firmed up in this order:

1. Infrastructure. Test performance of the cloud provider(s) to ensure they can sustain the needed load.
2. Data. "it is time to analyze the impact of encryption, transformation, data replication, and the various ways that data is being manipulated to address security , compliance, reliability, and scalability requirements."
3. Business Logic. Interactions of services and processes.
4. User experience. Does the application meet user needs?

[Image Source: Mike Kavis]

This is a solid approach for testing to happen while a "build from scratch" cloud-based project is being designed/developed. Testing the raw performance level of the infrastructure first is something worth validating early on, and so on up the stack, finally testing the business logic and application when they come together.

Mike talks about how covering the core performance of the cloud infrastructure early can mitigate risk at deployment: "... It would be very expensive to find performance issues from the lower levels of the architecture at this point. That is why I recommend the layered approach to performance testing. What I also like about this approach is that you can start testing very early in the life cycle."

To expand on this discussion, one way to shorten this cycle is through Service Virtualization, which is the practice of simulating and modeling the behaviors and performance characteristics of components, business logic and even UIs in the cloud that are not yet available for live testing. This kind of virtualization can go a long way toward eliminating service and 3rd party system dependencies and bringing performance testing to bear much earlier in the process.

Then, after that rollout, there remains a continuous validation need. (Only so much you can cover in a blog, so this is a topic we'd like to see Mike address in a future post... por favor... : )

For instance, the infrastructure itself may behave differently under the kinds of data and usage variability in ways that you could never predict with the raw-traffic or user interface kinds of performance benchmarking. New versions of services can be introduced, the Cloud provider can do something that affects their performance -- the list of risks goes on and on. Continuously validating going forward is a way to ensure that there are no unintended consequences of change at any one of the above layers.

All in all, it is a nice development to see practitioners showing real architectures and methodology around Cloud Computing as it is being applied for the business, as opposed to continuously defining the space, so I'd recommend keeping up with future posts from Mike.

Podcast: Virtual Strategy Magazine on VSE Benefits

Virtual Strategy Magazine's Carryl Roy recently recorded a podcast briefing with iTKO about the new features of LISA 4.6, with a special emphasis on the Virtual Service Environment (VSE) capability of the product, and why a more expansive vision of Virtualization is needed for today's heterogeneous, ever-changing software environments.

Virtual Strategy: "iTKO Rolls with the Punches, Releases New Version"

Virtualization of hardware has already made great strides in reducing costs and increasing flexibility in the staging and deployment of software in the data center, but there is much more work to be done in optimizing the software lifecycle before production.

We are seeing companies pressured to keep IT budgets down, while the complexity and volume of data accounted for in applications keeps growing exponentially. There is a need to prune away many of the costly dependencies that can drag down the productivity of software development and testing teams. You can listen to or download the 15-minute podcast here.

Webinar 6/18: Automated ALM & Dynamic SQA with iTKO & Forrester's Visitacion

Wanted: A way to tie together the software delivery advantages of Application Lifecycle Management (ALM), with Dynamic SQA, which provides the in-depth quality coverage required for today's complex and constantly changing enterprise apps. What practices and tools are required to make this connection happen?

We are excited about an upcoming webinar on Integration Developer News that should amply address both. iTKO Founder/Chief Geek John Michelsen is joined by Forrester VP Margo Visitacion, a leading expert in ALM and Software Quality. Ensuring predictable results and on-time delivery of enterprise software requires testing, validation and virtualization across a myriad of changing technologies and services. Customer examples and current methodologies will be explored.

Here's the official description and signup link:

iTKO and Forrester Webinar 6/18: "Removing ALM Obstacles for Greater Agility, Cost Savings, and Lower Risk"
Experts from iTKO and Forrester Research present trends / technologies behind automated testing and validation to streamline ALM for higher productivity, accuracy and agility. Attend and Learn how Testing, Validation and Virtualization are being employed to support the best practices of Dynamic SQA (Software Quality Assurance). With with strong Dynamic SQA in place, enterprise IT teams are poised to deliver predictability, patterns and reuse to better manage complex software projects.

Our last webinar on SOA and Integration Efficiency (view archive) had more than 500 attendees, and lots of great questions we were happy to address (either on the session or afterward). We hope you will join us for this one!

How to be Fast and Agile with Increasing Services Complexity Without Falling on your Face

Agile development is spreading in the SOA world to rapidly decrease development times, but if testing is not done properly, you might lose all the efficiency gains through having to conduct a massive cleanup effort afterwards. Like building bridges, fix it as you go is a much better approach than fix it later with massive traffic flowing over the structure. InfoWorld's Eric Knorr writes about this in his recent Modernizing IT article, "How to be agile without falling on your face."

Eric reports on a conversation with iTKO’s Derick Townsend. Derick said that companies within an M&A cyclone -- particularly in times of high volatility like we are seeing today in financial services markets -- are desperate to accelerate application development projects.

"They get 80 percent through a project, and they think the end is in sight, but the last 20 percent really stymies them, and that has to do with the test and delivery part. They're dealing with so much complexity. Telcos are experiencing this too. The components and underlying systems are changing constantly underneath," Townsend said.

Eric goes on to write about how iTKO provides what we call Service Virtualization, which is different than traditional hardware virtualization. Instead, iTKO LISA absorbs the behavior of all relevant components, services and data in an extended IT environment and simulates them, This capability allows developers to determine how an application based on those services will behave under various circumstances in the real world. Developers can become more agile using Virtualization practices to support increases in complexity, without a corresponding increase in testing and development risks and cost.  

Role of IT Architects and Architecture in BPM – MWD Survey

MWD advisors is running a survey on their blog (see "Is there room for architects and architecture in BPM?") exploring their audience's views on the role of IT architects and architecture in BPM. We plan to take the survey and encourage others to do this, as we are interested in seeing the results.

We feel that BPM exists as a discipline outside of any particular IT strategy, but as we have often mentioned in this blog, BPM can be a major driver of service-orientation, as well as governance and quality initiatives.

As the MWD blog points out, there is a lot of talk within the BPM technology vendor community about enabling customers to "scale up" their BPM initiatives. So they rightly suggest that IT architect involvement is likely to be a key factor in shaping how that happens in practice.

They have recently built relationship with the not-for-profit International Association of Software Architects (IASA), and asked them if they would help explore this question. This also want broader input, so we encourage you to participate in this survey and look forward to the results.

Minimize IT Outsourcing Risk with Collaborative Quality - Part Four

This is the fourth in a four part series that summarizes the key points in Minimize IT Outsourcing Risk with Collaborative Quality. You can click on the title to get the complete white paper. In the previous post in this series we looked at how to handle changing components and dependencies and achieve validation on an ongoing basis, as well as how to enable parallel development and testing without constraints.

Too often, companies allow the success of their outsource efforts to hinge on the performance of the services partner alone. In this post, we summarize best practices and responsibilities for the outsourcing consumer, after all, successful collaboration must be a two-way street.

• Reward Outsource Partners for Delivery Success... Not Just Delivering. – Do not prematurely sign off on delivered code assets just to meet a deadline, before adequate testing to ensure requirements are met.
• Trust, But Verify... Continuously - Distributed teams need to document component dependencies and test them as part of their deliverable. The consumer then needs to add tests at a business process level to ensure that expected behaviors are verified in the context of the consuming workflow.
• Enable Greater Productivity by Removing Constraints - Service Virtualization can enable multiple teams to work freely in parallel, rather than at odds due to the constraint of having access to live or incomplete systems. For instance the VSE in LISA Virtualize your partners with a way to decouple themselves from dependencies, by modeling test resources as Virtual Services that behave just like the rest of the application for testing and development purposes.

In summary, a partnership between outsource consumers and outsource providers based on shared objectives for collaborative quality is the best approach to maximize the benefits of IT outsourcing. Success depends upon a combination of people, processes and enabling technology to foster greater productivity, agility and quality across distributed teams. Key capabilities for this success include a collaborative framework for testing functional requirements, continuous test and validation of delivered components within the context of larger systems, and virtualization of constrained resources for greater productivity and improved time-to-market for distributed teams.

This ends our brief series on Outsourcing with Collaborative Quality - however we will continue to post on this important topic as we work with an ecosystem of customers and partners with the shared goal of delivering successful software releases on time and under budget. Your comments and ideas for future topics welcomed.

Minimize IT Outsourcing Risk with Collaborative Quality - Part Two

This is the second of a four part series that summarizes the key points in Minimize IT Outsourcing Risk with Collaborative Quality. You can click on the title to get the complete white paper.

In the last post, we looked at the risks that can come from outsourcing IT development. Solving for these risks, not just through contractual negotiations, but in every day collaboration, can unlock the benefits of outsourced IT and address larger distributed development issues. In fact, if the appropriate IT infrastructure is available to better manage accountability and quality across distributed teams, while eliminating dependencies and constraints that restrict the agility and productivity of these teams, then it follows that these benefits could apply not just to outsourcing situations, but to any distributed team environment.

The first step in meeting the challenges of distributed teams is to gain a commitment across all teams to deliver components against well-defined requirements, in an auditable form, with the ability to accommodate the highly interdependent nature of today’s applications. This requires more than the right attitude, it requires the right tools.  Key testing, validation and virtualization capabilities will help firms mitigate these risks, and foster more collaborative quality across interdependent teams. These capabilities include:

1. A collaborative testing framework that directly verifies all technology layers with a high level of automation, so the team can better meet defined requirements at lower cost.
2. Continuous Validation so both parties can assure that each software component supports all upstream and downstream dependencies, in an auditable form.
3. A Virtual Service Environment (or VSE) which simulates the behavior of the highly interdependent and critical systems the teams must build and test against, even when they may not have access to live or completed components.

With these capabilities, the consumer of services should not just receive code assets, but also the deliverables they need to rapidly validate them - collaborative testing is matched to functional requirements. This is important, since resolving a misunderstanding early in the requirements phase can be orders of magnitude less expensive than fixing it later in deployment. So how can we move testing earlier into the lifecycle, to save costs and improve quality?

If the offshore partner’s testing is not tied to functional requirements, there is no way to know if there is any value to the test. The key to achieving collaboration is to demand that each component a team delivers is also comes with a suite of working tests that prove its functional requirements have been met. That way the next team, whether local or offshore, can iterate on testing from there.

For instance LISA tests are saved as .XML documents, then the business team can augment these processes by combining them with traditional requirements docs as executable test assets in ALM and Test Management tools. From this foundation, the tests become part of a continuous build and validation process, where regressions are proven to match defined requirements in later testing cycles. In our next post, we look at how you can handle the changing components and dependencies and achieve this kind of validation on an ongoing basis.

Minimize IT Outsourcing Risk with Collaborative Quality - Part One

This is the first of a four part series that summarizes the key points outlined in our paper on Minimizing IT Outsourcing Risk with Collaborative Quality. You can click on the title to get the complete white paper.

In the past ten years we have seen explosive growth in the practice of outsourcing IT functions to SIs (System Integrators) and services firms around the world. One of the fastest growing areas was the practice of outsourcing software testing to overseas teams. How as enterprise systems change from huge monolithic applications to more componentized, services-based solutions, the risk involved in outsourcing testing rises. This series looks at these risks and offers a solution.

First let’s set the stage by looking at four common IT outsourcing models:

1. Single Outsource Vendor where the buyer outsources all IT development and testing functions to a single SI partner. It offers the benefit of a single source for delivery of business applications, and one vendor to call if something goes wrong. However, it can also create high switching costs if you decide to change vendors later.
2. Outsourced Testing appears to be the easiest route if you decide to keep software development close to home. Since testing is by nature the process of “breaking” an application, many organizations find that a removed third party tester can be very valuable and impartial. This model was very easy to adopt when testing was a manual, point-and-click-on-the-screen endeavor that was best done by less costly staff. But times have changed, forcing testing offerings to evolve in sophistication.
3. Development Support. The many companies following this strategy are seeking both cost efficiency for non-differentiating development tasks, and access to developers with specialized skills needed to carry out specific tasks, such as migration between two different integration platforms that they may not have the expertise to pull off in-house.
4. Multi-Vendor is most common when development and testing activities are outsourced to different SIs. In this model, the company hopes to keep costs down, while the multiple providers keep each other "in check" to ensure project success. While you can mitigate some risks associated with a single outsource vendor, it can also be harder to manage multiple firms and get them to collaborate as efficiently.

Now let's look at the risks, as managing risk can be more important than obtaining upfront savings, if these cost reductions come back to haunt you later. After all, what good is a lower cost project if it takes 3 times as long to repair, or if your app fails in front of customers? Your outsourcing partner needs to be incented to build quality into the application, and not simply cut costs and initial project time.

This balance of quality and speed gets more complex when multiple teams are involved. While many larger companies have a multi-vendor strategy of procuring IT services, the more companies and teams that are involved in a project, the more opportunities there are for the information sharing and collaboration to break down.

What is needed is greater team accountability for shared deliverables all the way across the software lifecycle, which includes visibility into problems early, versus later in the deployment where they become significantly more costly. There is also the risk generated by dependencies on unavailable or inaccessible systems. Interdependent teams working together can be constrained by limited access to live applications and components that are not yet completed or available for testing. We see this issue often in the field as reported in our series.

When there is increased contention for shared resources, agility suffers as teams are forced to queue up for access, introducing scheduling delays and project risks. In some cases, teams may attempt to maintain their own test environments by replicating the components they need in a testing lab or custom test harness.

These team-to-team constraints on environments can become very costly to manage, requiring a high level of configuration, licensing costs and maintenance to keep current, even if it is running on virtualized hardware (which also has incremental licensing costs). And of course, some sensitive systems such as ERP mainframes are simply too big and have too much overhead to be replicated by conventional hardware virtualization means. In our next post in this series, we start to look at ways to meet these challenges and make more effective use of IT outsourcing.

New Outsourced IT Paper - Collaborative Quality to Minimize Risk

Though we are in the testing and virtualization business, over the last 5 years we have had a lot of requests from customers for our advice about working with one or more outsourcing firms. Since we have encountered virtually every kind of working arrangement in our engagements, we do have some experience encountering both successful - and unsuccessful - teams comprised of multiple services partners.

Given the current market conditions for IT outsourcing, it seems like there is more uncertainty than ever in this realm. Outsourcing is still a must-do activity for at least some of any large enterprise IT budget.

For the most part we've kept our advice to the realm of working with our SI partners - as part of working with joint end customers, we need to cover how to better collaborate on quality, validate that business requirements like accuracy and performance benchmarks are met, and improve agility to deliver new features ahead of schedule.

There are many up-front aspects of properly selecting, negotiating contracts and defining requirements that will make or break a project - too much to cover here - but in our experience once the partners are selected, what defines the back-end of delivering a successful project?

The number one post-agreement issue we've found when companies have deployment issues and cost overruns in working with services firms, is that partners must be rewarded not just on delivery, but delivering with quality. We're not talking about simply handing down a set of requirements and enforcing them. Good collaboration is a two-way street - both the consumer and provider of services must be responsible for defining and delivering things that are tested, and testable.

Anyway, we wrote what we think is a pretty good primer on the topic - you can get the new paper here: "iTKO Whitepaper : Minimize IT Outsourcing Risk with Collaborative Quality" - http://www.itko.com/site/resources/collaborativequality.jsp. We can't tell you who to work with, but we can provide some advice we've seen on what works!

We'll be posting further thoughts on collaboration between multiple on-shore and offshore teams here - your thoughts and questions on the topic are welcome.

Agile SOA Across the Lifecycle - Part Five: IT and SOA Governance

This is the fifth of a six part series of posts on the Agile SOA life cycle. Here we will at look at IT and SOA Governance.

With the introduction of agile, spiral, and scrum development methodologies, the traditional waterfall development approach of testing a near-finished app at the end of many Agile development cycles won't be agile at all, as the elements of the application are constantly changing.  Traditional models of IT governance will also not work.

To aggravate testing, the service-oriented architecture (SOA) design pattern is used to make IT  more responsive to changes requested by business. New process tooling has been introduced to specifically assist in the cataloging of service assets, and organization of policies governing SOA.

This new set of tooling created to support SOA revolves around governance platforms like HP Systinet / S2, SAG Centrasite, SOA Software, TIBCO ActiveMatrix and Oracle Fusion. These SOA Governance tools manage the collection and cataloging of metadata about services, and organize the interdependencies they have with each other, along with the documentation of SOA Policies to define how combined services should meet business requirements. 

Within the Governance toolset, SOA Registry/Repositories provide a new platform for LISA to automate testing and validation.  The traditional system requirement and functional testing will still occur, but in a SOA there are more opportunities to automate the validation and enforcement of policies.

The current thinking around SOA Governance is largely siloed and specialized around design-time WSDL validation, runtime performance, and security policy enforcement.  As SOA Governance matures to support robust and widely diverse SOA initiatives, this thinking must expand dramatically. One of the greatest values a SOA Governance platform must provide is the proof that policies written in human or business terms are in fact implemented in the system, on a continuous basis.

Validation with our LISA solution takes the form of positive and negative tests on the use of these policies, which are executed continuously at change and runtime, as services and their underlying technologies and data are constantly changing and evolving. The validation of the LISA test becomes a demonstrable artifact for exposing and enforcing SOA Policy, within the changing workflow of the SOA Governance framework.

Example Scenario of Validating SOA Policies

Let's look at a sample deployment, a series of web service definitions, as WSDL files, are loaded into the UDDI registry.  The relationships of the services to each other, and the documentation of how they can be tied together to create a business process is documented.  For a new employee introduction, there may be a series of services from the human resources package to identify the employee's location and managers, there are services that provision IT resources like computer and e-mail, and services to a external partner for insurance and benefits enrollment.  These services would be documented via their WSDLs and stored in the UDDI registry.  A HR-to-payroll business process will be defined utilizing these assets.

This all seems simple as long as each participant in the business process provides a well documented, functioning, reliable, and secure service.  The policies around the structural, behavior, and performance of these individual services are documented the governance tool.  For example, the business analyst will write policies in a tool like CentraSite ActiveSOA, describing the behavior of the services. 

The IT service will accept a new employee's name and id number and return an e-mail address, domain login, and default password.  The security service will accept the manager and role and provision the appropriate access control to systems for the new employee. The security manager will specify that the services must only run if encrypted and signed data is sent to them.  The signature must be validated against the companies master certificate provider to prove that the request is from only authorized HR personnel.  Lastly, the IT operations team will provide policies around how many transactions per second (TPS) the service should handle, and the maximum response time the service will take, so the component will be a good participant in a decomposed transaction.

That's far too many steps to carry out if it's a strictly manual validation process. The execution of the tests needs to live within the SOA Governance process.

Testing, Validation and Policy Enforcement Solutions

Given all of the processes enterprises must employ to manage the design, construction, integration and management of the IT environment, it makes sense to have a common, reusable way to apply validation and enforcement across these processes. SOA Validation becomes “the long arm of the law” with reusable and rich testing that merges into the workflow of each of these processes, and the tools that support them.

The Validation arm shown above (with our LISA as the "long arm") provides capabilities to run test cases to verify different rules and policies defined in leading IT process tools. The policing action is accomplished by tying automatic running of tests to the expected behaviors and policies in the workflow of the process tool.  The invocation of  tests from a governance platform to ensure SOA policy is similar to the policing action done for test management solutions, the biggest difference is the context and stage in the lifecycle of the component that needs to be verified. 

In a traditional test of a waterfall development there is a specific point in time in which the system is tested and deemed ready for production.  In today’s services-based applications, there is no longer one point in time in which we can identify a test run to ensure quality of the application.

In a SOA lifecycle, we think of design time, run time, and change time as the three stages of a service or business process being constructed within these loosely coupled systems.  In the example above some of the services are interactions with packaged applications (HR system), some from home grown systems (IT user and security provisioning), and other from external partners (benefits).  The development and release cycles of these three areas will be different and will not be coordinated for a single big bang release.  One advantage of SOA is the ability to decouple systems and utilize services created by different parties instead of building everything yourself.

At design time, a test will be run to make sure that WSDLs are WS-I compliant, that they follow an RPC or doc-literal call structure, and that a customer-specific XML tag is used for service identification.  Based the on passing of a compliance test, the WSDL will then be made available in the Repository for developers to code the functionality behind it. 

Once the developers have created the necessary code to implement the service and its underlying technology components, the behavior and security of the service needs to be validated.  The only way to verify if the code behind a service is working properly, since there are no hard-coded responses in the real world, and that security is respected, is to invoke the service, then verify that the underlying systems of record are updated properly.

Again, a test case is used to invoke and verify that the business requirements and the behavioral and security policies have been implemented. If the candidate Service can pass these Policy tests, it can now be published in the registry for a consumer to use.  If these tests fail, the service should not be promoted for availability to consumers in the registry.

As a best practice, the individual services should undergo load testing once they are made available. Far less risk is introduced if we test for load while the components are being built, when something can be done to improve performance issues, rather than waiting until the system is deployed, when issues are much more costly to repair. Progressive development shops start their load testing once the service functionality has been verified, so that developers can make code changes, and optimize logic and data access before consumers start utilizing the service.

In run time, the service is now promoted to the Registry, along with a test that demonstrates its functionality, and made available for consumers to leverage as part of their own application workflows. These consumers also bear responsibility for validating that their intended use of the service is actually supportable. The best way to do this is to create an executable test asset to validate that the structural, behavioral and performance aspects of the overall consumer’s workflow are supported when the service is leveraged. This will pay big dividends when the test is run continuously as part of the governance process.

The final stage is change time. This is when the true advantages of SOA and agile development come into play. If a new service needs to be added, or a modification to a existing service needs to be made to respond to changing business requirements, the policies for the service must be re-verified.  Not only do we want to update any behavioral policy with the new expected behavior, but the regression of existing policies must occur.

Put yourself in the place of the consumer of a service. How can you trust that changes will not create unintended consequences (e.g. failures) for a service you depend upon, and break your existing business processes?  Each consumer needs to place their expected behavior as a policy in the registry, with an accompanying test, so that it can be automatically validated before that change is made.  The automated structural and behavior policies in the test represent the consumer’s rights – and their responsibility for using the services as defined. When these tests run automatically as part of SOA Governance, trust between service consumers and producers is achieved. For more on SOA governance see Joe McKenrick’s posting of the full transcript of a recent eBizQ SOA governance panel that John Michelsen participated in.

In our next post, we will provide our conclusions. You can download this complete series as the "Agile SOA Across the Lifecycle" at our ITKO LISA resources page. 

Agile SOA Across the Lifecycle - Part Four: IT Operations, Monitoring & Performance

This is the fourth of a six part series of posts on how SOA works within the Agile life cycle. Here we will look at IT operations, monitoring and performance. Once an enterprise application is in deployment and under use, ensuring its continued availability, performance and accuracy becomes critical to the business. IT Operations teams may be part of the overall test and development team, or grouped as a separate function in many enterprises to offer a measure of impartiality to ensuring that required Service Levels are met within the live application environment.

Operations Management and Monitoring

The leading monitoring frameworks can be supported by directly testing and validating the systems under management, and feeding rich metrics and test output directly into the team’s IT Management dashboard, such as Wily/CA Unicenter, TIBCO Hawk, HP OpenView, IBM Tivoli, and others. As an example, our LISA tests can be run continuously at runtime and act as a listener for both functional errors, and server metrics from components including JMX, SNMP, Windows Perfmon and LISA’s own types of metrics.

The Operations team can set thresholds or boundary conditions on these continuous tests and have the Testing framework alert them if an issue occurs. For instance, if performance is too slow, or memory usage or a certain database table increases in an unexpected way over time, the test should report a failure to the management dashboard, and send an SMS message to the admin, along with the test case for root cause analysis and correction. LISA can also output this data as reports into the team’s management tools or data store to provide the ability to perform audits.

Performance & Load Testing with Leading Tools and a Virtual Service Environment (VSE)

Also key to the Operations team is ensuring that delivered systems will meet expected Service Level Agreements (SLAs) for customers. For today’s highly interconnected applications, the leading Test and Performance Lab solutions on the market were left out of the lifecycle of applications until near the end, when an interface or a complete system environment was available to test. A Virtual Service Environment (or VSE) provides teams using Performance Testing tools like LoadRunner and SilkTest the ability to get involved in ensuring performance much earlier in the lifecycle, at up to a 90% reduced cost per test environment.

The VSE captures and simulates the expected behavior and response times all of the dependent systems within your target environment. As component load tests are conducted earlier, teams can optimize their use of system resources, find memory leaks and fix the root causes of errors, at a much earlier stage in the design and development lifecycle, without needing access to the live system or all of the interdependent components needed in the test environment. The ability for multiple teams to conduct their own performance testing processes in parallel, prior to solution delivery, pays huge dividends in productivity and quality in deployment.

Behavioral Service Virtualization in this sense is by no means a replacement for virtualizing hardware and network resources to save costs in IT operations. But it does represent a huge way to reduce the dependencies and costs that can't be touched by virtualizing a given set of servers - after all, there are many third party or shared Services in the cloud, or heavy transaction systems and partner systems that can't be virtualized by conventional means.

Michael Vizard of eWeek writes about the role of virtualization in his post, Network Management Gets Virtual. He writes that many vendors are saying that, “it's become critical to extend existing network and systems management tools into the realm of virtualization. They argue that IT organizations don't need to invest in learning separate management tools for physical and virtual environments and the impact of decisions in one environment can be devastating on another unless the management infrastructure for both are tightly coupled.”

This tight coupling can occur under LISA. Using a VSE, teams can either model incomplete components, or capture live services, and simulate their behavior on a running server. Functionally and under load, these Virtual Services hosted in LISA’s VSE act and respond just like other services, databases and systems you connect to in the live deployment, including expected response and transaction times. Without requiring access to critical live systems, the team can connect their component to the rest of the virtual architecture in LISA Virtualize, and conduct performance and load testing with their tools of choice.

It is important to note that most companies have significant investments in test lab tools, and supporting them with robust target test beds makes those labs much more productive within the software lifecycle. Of course, our LISA Test & Validation solutions can also be that load and pefromance testing engine, and LISA test suites are executable and interoperable with the Test Management, ALM and SOA Governance processes mentioned in this series.

In our next post, we will look at IT and SOA governance. You can download this complete series in the Whitepaper on "Agile SOA" at our ITKO LISA resources page. 

LISA Eclipse Edition released - TVV in the IDE

Developers are increasingly being asked to provide more test coverage, and design more testability into the components they are delivering to the business. So why not provide a side-by-side way to code and build apps - and - create and execute tests against them, within the same development environment?

Tomorrow we are officially releasing LISA Eclipse Edition - it's a real edition of our LISA 4.5x product running inside the Eclipse UI. We've had a lot of customer demand for LISA as a first-class-citizen of Eclipse, and it joins our cross-platform Java standalone version of LISA that runs in Windows, Mac, and Linux/Unix client flavors.

I had a chance to talk with SD Times' Jeff Feinman in advance of the release, about some of the benefits of having the IDE of Eclipse working hand-in-hand with LISA's live testing and validation capabilities. Jeff posted a good article on the topic today here: "iTKO integrates Eclipse into LISA, allows for more testing"

Enough of my product marketing hat for now... Many kudos to the LISA development team and here's the official press release: "iTKO Releases LISA Eclipse Edition for Complete Test Coverage within the Application Lifecycle"

Making IT More Green and Saving Money at the Same Time Through SOA

Saving money though reducing the environmental impact of IT has gained increased urgency in our down economy. Green computing even has a Wikipedia entry. Fortunately, many of the Green IT efforts also result in cost reduction. One of our business partners, Software AG, has looked at ways that SOA can help here – see Eco-computing: Software AG and the Hasso Plattner Institute Demonstrate SOA’s Green Potential.

Software AG and the Potsdam-based Hasso Plattner Institute for Software Systems Engineering (HPI) completed a one-year research project examining the impact of service-oriented architecture (SOA) governance on Green IT practices. They developed methods and tools for optimizing resource sharing of IT services, thereby reducing energy consumption and lowering hardware costs. Using these methods, administrators of complex IT landscapes can determine exactly how much computing power is needed, as well as the maximum amount of energy each service might use. With proper SOA governance you can regulate and more tightly control energy costs by taking advantage of the more efficient distribution of services that SOA can bring. This makes sense and is one more reason to make sure your SOA governance is carefully conceived.

There are other ways SOA can have a positive eco-impact, as well as a bottom line impact. For example, virtualization of hardware saves the materials and energy consumption of so many servers - as much as 10 to 1. Most companies have already gone that far, but what about the kinds of applications that are too distributed (i.e. shared services and Cloud-based apps) or too monolithic (i.e. mainframes, major enterprise apps) to be replicated via hardware virtualization?

We call this Service Virtualization and it allows the behavior of complex SOA applications to be captured and simulated - which is key to achieving the productivity and efficiency you expect from SOA. Therefore, if you could save the cost of attempting to replicate several distributed applications and services, that's saving a lot more processing power throughout the lifecycle and not just servers in your own data center. In addition, it allows you to scale down hardware and processing power in your test lab, and in deployment, as you no longer have to try to make a live instance available to so many teams, or scale up the hardware of the live instance to handle a bunch of extra capacity.

There are more savings to be harvested. When you break up project teams into smaller "agile" teams focused on enabling certain Services for use and reuse, you gain additional efficiencies besides Agility and labor cost savings (see our recent paper: "Agility Across the SOA Lifecycle" for more).

Think about the reduced need for travel, and for dozens of people to occupy an office configuring and maintaining separate test environments per partner and per test/development team. The more we can do to decouple our dependencies on constrained servers and applications, and decouple our interdepency on other teams' lifecycles through service virtualization, the bigger the positive ripple effect between the interaction of technology and the environment.

We welcome any discussion and ideas about how Virtualization in a Service-Oriented World can make a difference in the world.

SOA Governance Panel Recap: Federated SOA for the Day

iTKO's John Michelsen recently served on ebizQ's SOA Governance panel for a spirited online discussion. Joe McKendrick moderated the panel which also included: Anne Thomas Manes, vice president and research director with Burton Group, and former Chief Technology Officer at Systinet; Ron Schmelzer, managing partner with ZapThink; Frank Martinez, senior vice president, product strategy for SOA Software; David Bressler, SOA Evangelist for the Actional products at Progress Software; and Ed Horst, vice president of marketing and product strategy for AmberPoint. Joe provided both a summary, Panel: SOA Governance Answers 'What's in it for Me?' Question, and the full transcript: see Transcript: SOA Governance Panel.

Joe charged the panelists to comment on how far along we really are with SOA, and what is needed to take things to the next level. He also wanted them to be practical, and explore the tools and techniques that are essential to getting started with SOA governance. In addition the discussion focused on the implications of SOA governance for the "Web 2.0-ish" activities (like Cloud and WOA) that are often well under the radar of corporate IT management. Lastly, he wanted to cover one of our favorite topics, virtualization and SOA.

As Joe said in the summary, setting up proper governance seems like the logical approach to introduce SOA to the business. However, in the real world, SOA proponents run up against a series of obstacles. Many stakeholders have to be signed on to the SOA effort. These include developers who are expected to reuse someone else's code, departments that are expected to share assets and timelines, functional groups that are expected to provide more funding than usual to the IT underlying their processes with the hope for future cost savings and greater agility.

In response to these challenges John suggested that a Federated approach may be essential to SOA governance. In effect, people may be more willing to accept governance at  more decentralized levels, rather than more centralized approaches:

"We found that it is very analogous to the governance analogy in the physical world. That federating or layering of the governance policy is how it happens. If you think about it, my homeowners association, it wants to know where I'm going to put my flag. But at the Federal level.. that would be an absurd notion for them to decide such a thing. So we are more tolerant of greater control closest to us, than we are far away. So if there’s this very distant group of people who decide on how we are supposed to build and consume services, then we will tend to react negatively toward that. But if there is even.. a fairly onerous set of policies, but it’s shared, and its owned by me and my close constituents, I tend to be more willing to do it."

Later John added that since SOA has been around for a while, we know in advance a number of things that require planning, including aspects of the governance requirements. The solutions require more that installing infrastructure, they require an understanding of the desired business behaviors and how this infrastructure will support these behaviors.  He also said that there are two types of governance: what operates during design and what operates during run time as the transition to the new SOA services takes place and new complexities come up. 

While the panel was not able to cover virtualization like we hoped in the hour, they did manage to cover  other topics. The panel’s responses provide a rich resource on SOA Governance and make a good read (or listen). Here are the participants. They are left to right: Anne Thomas Manes,  Ronald Schmelzer, Frank Martinez, David Bressler, John Michelsen, Ed Horst, and Joe McKendrick.

Thanks for the questions and comments, and those of you who stopped by our "virtual booth" for more questions. We're looking forward to our next one at the "SOA in Action" virtual conference Nov 19th 2008, which has a who's who of SOA experts throughout the day. John is participating in the "Avoiding SOA Disillusionment" panel at 2:00 PM Eastern with Joe and a new cast.

Outlook Interview Part Three: Performance Management and Testing

Here is part three of the interview with host Mike Lippis of the Outlook Series. We have broken the 45 minute podcast interview into five parts for easier reading. This section covers performance management and testing.

Mike:  “When it comes to really achieving business value from IT investments, lots of folks will really look at performance.  Can you give us some background on some current performance management strategies that are out there?”

John:  “I can compare that to what our traditional approach to performance management has been and again, if in the old world, entire applications were produced and then they were performance-tested in a lab and released for production.  This worked okay when the thing we built was a 100% self-contained thing. 

“But now back to an analogy of a service-based architecture, where I am producing applications that consume services from you, Mike.  What do you do to performance test or to manage the performance of your service?  You are no longer a complete application.  I can’t ask you to produce your service, and send it individually into the performance lab if the performance lab hasn’t retooled?  I’ve got to think about performance testing at the services layer.  I’ve got to think about the performance testing even at the orchestration layer.  By that I mean there may be two or three services that collaborate to make one business process or even one sub-process of the business function. 

“Once we start to think about performance management, now that I’ve broken my big box application into a number of components; I’ve got to do the same thing with performance.  I’ve got to break my run-time SLA into the component-level SLA  (Service Level Agreement) that’s basically my performance and resource expectations that I’ve put on my applications.  So, Mike, I might need to do something that we call decompose the SLA where my application must run at 2 seconds or less response time for the user.   

“If 2 seconds is what I have -- I need to get your performance under 2 seconds so that I can meet my SLA.  Your response time is a component of my response time, because I am invoking you -- I am dependent on your service.  What I need to do is negotiate (for lack of a better phrase) your SLA with you--we call that decomposing the SLA.  You need .5 seconds, then my application can consume 1.5 seconds, and therefore, I can meet my overall SLA.  This is in many ways a new way of thinking about performance than previously.   We used to just think about 2 seconds -- we dropped the application in a lab to see if we had 2 seconds.  Now Mike’s team has to go to the lab and has to meet .5 seconds, and my team has to go to the lab to meet not 2 seconds but 1.5 seconds. I’ve got to make sure that my 1.5 seconds will combine with your .5 seconds.
Make sense?

Mike:  “Once you have an application completed, a lot of times they are difficult to tune for performance.  What are you specifically doing to address that?”

John:  “This notion of taking our SLA apart into components actually helps us do this.  It does for a couple of reasons the biggest is:  if we can performance test within the life cycle, if we can do our testing as a part of our own development than we can find and resolve our performance issues much more quickly.  In my experience, and I bet many people in the performance field will nod in agreement, so many times if we build an app that does not scale and then try to tune it into scalability, the best we’ll do is 10 or 20% improvement.  You can’t add hardware and make a product that is un-scalable all of a sudden scalable--it just doesn’t work.  If you had known earlier where those scalability issues are--than you can build in the scalability along the way a whole lot easier.

This is essentially the same reason why we iterate from a requirements definition standpoint, because the more we iterate thru the scope of our work the better we are at delivering on it and the faster we are   The same is true with performance, we have to iterate and we have to do performance testing at the component level in order to get that performance up and the scalability up.”

Mike:  “Are you doing anything to address the challenge of discontinuous development cycles?” 

John:  “Yes.  Because of the fact that you may be building your component at a different lifecycle pace than my application even though I am a consumer of your component, (usually we call that service) our challenge is that I got to have access to your service maybe at a time you don’t even have it ready for me.  You might have down stream dependencies of your own.  As soon as I think of you as “Order Management” -- you might turn around and say: Yes, I do order management but I sit on top of the customer database that I don’t own, and I sit on top of the ERP system that I don’t own either.  So, you have your own set of dependencies.  So if we actually fit it on a white board, most large-scale enterprises would literally fill a white board with very small little boxes all interdependent on each other and they are all iterating on their own life cycle.  It drives me nuts just thinking about how anything actually works!

“So in order for us to support this kind of distributed lifecycle, we’re doing a lot more virtualizing of the components underneath each other. 

“What we mean by this is:  since I am invoking your service, I am dependent on your service, but what if your service isn’t available to me yet, what if I need you to change your service and your changes aren’t ready yet?  What if I need to do performance testing of my application on an interim basis but I don’t have scalable production quality hardware to run your service on?  Or, maybe worse, maybe your service talks to the mainframe, but I don’t have availability of mainframe resources to do this?  All these are what we call constraints.  These are real world constraints which we are dependent upon. 

“What we do is:  we virtualize the behavior of these things.  What I will do is construct a model (our LISA Virtualize product is really good at doing this) -- it constructs a model of the behavior of your service, and once that model exists I can run that model in a virtual environment, it doesn’t have to have the real world constraints; it’s a virtual thing.  So, if I need it early, I’ve got it. If I need it to scale to 500 transactions a second, I can do that. Even though I don’t have production type hardware for your real service, I can virtualize it and get the kind of performance that I’m looking for.” 

Mike:  “It sounds like you would be able to provision the right hardware and platform for a given solution then.”

John:  “Right, because what we do is that we dramatically reduce what that requires.  A classic example for one of our customers is:  I’ve got an application that is a portal-type application that invokes a set of services.  Some of those services talk to SAP.  SAP is a great ERP system and many large organizations use it; but of course we don’t want a lot of performance and load testing type transactions against that system.  We don’t always have availability when we need it.  We don’t always have the data in the database when we want to -- SAP testing is a very complex field -- so there are all kinds of real world constraints.  What we do is we virtualize the behavior of SAP and in doing so, your portal has access to what it needs 24/7 and I don’t have to provision the entire SAP environment to do it.  All I have to do is provision my own application, we can’t virtualize away you --but we can virtualize away your dependency. and once we do that, we decouple you away from a lot of the challenges that those constraints have.”

Mike:  “When it comes to performance labs, it seems like a lot of them really rate data management as a very big problem.  What can you tell us about dealing with that challenge in leveraging LISA?”

John:  “One of the great challenges with data in performance labs … is really two challenges. One is that you can never talk to just one database in a performance test. You are usually dealing with multiple data sources.  For example (as I mentioned a minute ago), order management is dealing with a customer database in one system and with the orders database, let’s say, underneath SAP -- so here’s two completely independent systems that are now interdependent on each other with regard to data.  Now here I am in a performance lab and I’m trying to support hundreds of scenarios of real world use of my application.  In order to do that, I need the data between those two databases populated in a certain way and synchronized between each other and whatever changes are made in one database, I need those changes to be reflected appropriately in the other database.  Do you know how often this happens? Much, much less frequently than Christmas! 

“What we have is a dynamic that we see all the time, you’ll go into the performance lab wanting to do some 100 different scenarios of use, these are all the use cases you want to be sure are performing and scalable in your application; but by the time you figure out just how to get all the data to synchronize between these two systems, the best you’ll come out with is about 20.  Now the problem is that you’ll performance test just 20 scenarios, but you’ll go to production with all 100.  You’ll find out the hard way whether the other 80 are a problem or not, because your real world users will be doing those other 80 (the one’s you couldn’t performance test) and the problem is:  if you have scalability or performance or resource issues in those other 80 … then you have a problem.”

Mike:  “When it comes to best practices in the area of performance and load testing, what can you tell us about decomposing performance expectations?”

John:  “I gave a quick example of this earlier, I’ll elaborate on this a little bit. 

“If the production SLA for a particular application; let’s say mine is 2 seconds…that’s the example that I gave previously … where my response time to my user is 2 seconds … but one of the things that my application does is to consume services from your team, Mike, I need to negotiate your response time as a part of my response time.  So what I do is decompose that SLA.  What I’ll do is say, ‘okay, I’ve got to make sure that I can get Mike’s service calls in a short enough amount of time so that my application overall’ -- once you add up all of the time consumed – it doesn’t trip over our expected response time. 

“So we help by giving the customer the ability to do two things.  The first is:  we’ll help them do performance testing at the component level.  Mike, you can performance test your service.  You can say, ‘hey John, I can hit my half-second response time metric independent of you.  I owe you the right functionality at the right performance.  I can functionally test this system to make sure it works and I can performance test my service (independent to the rest of the world) at .5 seconds so I have done my job.’

“Now, I come into the picture and I can do performance testing on my application and make sure I hit my SLA as well.  In order for me to hit my SLA, I have to take you out of the mix. That’s why we talk a lot about virtualization in the performance lab. 

“So second, I can simulate your service’s behavior and I can even simulate your behavior at .5 seconds.  I can say, ‘alright Mike’s virtual service; I want you to behave like Mike’s virtual service, and I want you to take .5 seconds whenever you respond to me.’  By the way, I can even say ‘what if.’  What if your response time takes .6 seconds, what does that do to my overall response time? What if you’re actually running a little faster at .4 seconds …what does that do to my response time? 

“Sometimes the fastest way for me to get better scalability is to negotiate you a little faster so that my application hits its SLA:  stuff like that.  So that’s why performance testing as a component and virtualizing married together really give you a solid foundation for performance management in SOA.”

Mike:  “We had mentioned provisioning before, how does that come into play in terms of optimizing provisioning relative to managing performance; should one be done before the other and how is that relevant?”

John:  “The relevance is that we need to be able to capture virtual services as a part of the provisioning activity.  Where in the past, I needed the entire SAP system to be available for provisioning, now I just need the virtual services that encapsulate the behavior of SAP; I need that to be the provision item instead of the whole SAP system. 

By the way, when you think about these large-scale applications, often there are not only many servers with a very specific and complex kind of configuration, but there are huge databases underneath these big systems.  So very frequently we have customers tell us that they can’t do development work against this massive system because it’s got a terabyte-sized database underneath it.  In order for me to get access to it, I have to wait till midnight to 2 a.m. because that’s the only time they will give me time on the system. When I try to do performance testing, they refuse to let me because in order to avoid impacting production, they limit the amount of non-production activity on that particular system.  All of these kinds of constraints are those very same kinds of things. 

“So, back to provisioning, if I need access to this particular database or this particular large application that spans many servers, I encapsulate all that behavior into one virtual service and I can provision that one unit to represent that entire set of systems.”

This concludes part three. In the next post we cover virtualization and testing.

Agile Commons – "A Community to Support Software Agility"

I have added a new blog to the list of blogs we read. Agile Commons is run by Rally Software as a means to share information about Agile development.  Rally provides Agile lifecycle management solutions. As their site states, “Rally unites Agile project management with tracking of requirements, tests and defects, so everyone has a real-time picture of your project’s features and quality, priorities, roadblocks and risks.” There is a lot of great content to support Agile development is available through their site and through Agile Commons.

The Agile Commons blog is a collective effort of the community and you can request an invitation to join.  There are plenty of comprehensive and useful posts. Luke Hohman just did a three part series on Why Prioritizing Your Product Backlog for ROI Doesn’t Work. Luke starts with the premise that pundits tell developers to prioritize their backlogs to generate the best possible ROI. But he has not found an Agile teams that does this. Then he explains why using ROI for prioritization does not work in three comprehensive posts.  He basically feels that it is hard to get good data and is thus a fool’s errand.

Instead of ROI, Luke suggests gathering stakeholder preferences, making sure efforts align with corporate strategy, and look for initiatives that drive profit. These do seem more concrete than ROI.  There are many other excellent bloggers contributing to this conversation. Building a knowledgeable community of Agile Coaches and experts is a good strategy for Rally.  Luke’s post and a number of the others, such as Rachel Westin and Evan Campbell,  are related to topics explored at the Agile 2008 conference, August 4-8 in Toronto.

Through the Agile Commons blog you also gain access to the Rally Community, Agile University and Agilistas – a collaborative community dedicated to evangelizing and advancing Agile practice. All of these resources are worth checking out. 

European SOA: Ahead of the Curve?

I'm in Germany right now, looking forward to a session at ZapThink's Practical SOA in Frankfurt event tomorrow. I'll be joining some peers from companies like T-Mobile, Swisscom, Novartis and SwissLife who are going to be sharing their own SOA success stories.

Why does SOA seem to be moving forward a little faster in Europe than in North America? We've posed these kinds of questions in our surveys and forums, and often it seems that stateside, the term "SOA" can polarize some IT teams - it's an "either/or" decision at the architectural level.

Talking to our EMEA director, Wilfred, he says part of the reason it seems adoption is high in Europe is that it SOA is often driven by developers working in smaller teams. Service-orientation can be started on a much smaller scale, and tested pragmatically before rollout to the larger organization. So, SOA there doesn't need to be an enterprise-level initiative in all cases, it can be something the company dabbles in before making a full commitment.

The upside of this pragmatic approach is that the SOA project can leverage some small wins to pay for a bigger commitment. The downside? Companies still need to beware of publishing services without a higher level of architectural coordination between teams.

This isn't to say that there aren't also companies in Europe starting from a strong enterprise architecture perspective and defining a top-down approach to SOA. I look forward to meeting both the pragmatists and the strategists this week!

Global SOA - Quality = A World of Trouble?

We have a customer who was a highly advanced and ambitious early adopter of Services-based applications. Prior to adopting some of our best practices and our product for continuous testing, they had come to the point where their Services Group (a shared services team that was the producer of a set of services for an entire division of a company) had to go to an annual release basis.

Much worse, because of the fact that there were so many applications built on top of the shared services platform, and they were so distributed globally, they really couldn’t keep their hands around everyone’s use of these shared services.

They had to make a hard stop on November 1^st of every year, and go into a 6-week procedural test cycle, where any failure anywhere within the entire division would stop the rest of the division from being able to complete the test cycle. And, upon completion (hopefully in that six weeks) to find what could run for that month’s test cycle, finally release all the upgrades to not only the shared services that were built that year, but all of the applications that were depending upon that shared services platform.

Now, hopefully, that gives you some insight into what happens when you try to go to a services approach, but you don’t marry it with very rigorous and continuous automated testing. At the heart of the matter for these guys, SOA lifecycle quality was the missing piece. When you have a shared set of services being leveraged by a variety of consumers across the globe, the changes that are seemingly irrelevant almost, at the shared services level, can have a catastrophic effect on the applications that depend upon them. The worst part is that the developers of those shared services may not even be aware of the potential catastrophic effects of their application cycles on the service consumers.

All it takes is once or twice of having those unintended consequences show up as huge performance issues, or as huge production outages, or, my worst nightmare, actual bad functional integrity where the business is actually getting the wrong answers (I don’t mind slow answers or no answers -- but wrong answers are worse). No one is sitting with a calculator beside their computer verifying all of the transactions that the system is doing. So if the global system is doing them wrong -- you are in a world of trouble.

So what they had to do was coordinate the release of every one of these applications within this very large division of a large company. With our solution, taking some good SOA best practices, and using LISA from a continuous testing/continuous integration kind of basis, they are now going to be able to start dismantling this annual release cycle. As they increase the level of automated coverage of shared services and of the applications that depend on them; they have the freedom to change without the unintended consequences catching them off-guard.

So, in a nutshell, continuous SOA Lifecycle Quality testing to prove functional integrity has huge returns for the business. They see the value of going from an annual Big Bang release, to more quarterly releases, then monthly releases or “as needed” releases so that they can realize the agility they expected from SOA, with the reliability required for business. John Michelsen, Founder and Chief Architect, iTKO