Introduction
We have all felt the pain. We upgrade something on our PC or laptop and suddenly something else stops working. You are not alone: a Google search returns 48 million hits for 'upgrade problems' and 24 million for 'computer crash'. While this is inconvenient for us as users of personal computers, this problem can cause significant disruption when it occurs on an enterprise scale.
Imagine a typical development community for a bank or government department. There are, perhaps, a couple of hundred developers using a diverse collection of technologies designed to assist them in the timely delivery of quality applications. These technologies will include Requirements Management tools, Issue and Defect Tracking tools, Project Management tools, Integrated Development Environments (IDEs), Software Configuration Management tools, Build and Release Management tools, Automated Testing tools, Deployment tools and many others. Invariably enterprises will select the best-in-class tools to meet their needs in each of these categories. However, modern development communities are constantly striving to optimise the effectiveness of their developers. Hence, they need to have the tools they have selected tightly integrated together.
How productive would developers be if they were constantly switching between these applications to get anything done? This is why it is essential that these tools are tightly integrated together. As a result, we spend countless hours crafting scripts, writing macros and adjusting configuration settings so that our development community can effortlessly transition from one tool to another in pursuit of the perfect coded solution.
But, we all know that when we upgrade even the most minor part of our development infrastructure, we are putting at risk the whole thing. We do it nonetheless and the results are legendary. Every developer has a tale of technical tragedy: whether it is not being able to check code any longer, losing all the changes they have made in the last week, or suddenly production builds failing to pick up the development libraries.
When we upgrade and these integrations fail our 200, formerly highly productive developers, they may now be unable to do any work at all at worst and are severely limited in what they can do at best. The consequent lost productivity, impact to schedules and delayed delivery of mission-critical applications is expensive and very disruptive.
Caption: Fig. 1: Vendor-to-vendor integrations do not scale, are inconsistent, and are compromises that meet few customer needs
Are the Vendors to Blame?
So why does it happen? To answer this, we have to get into the minds of the software vendors. Each software vendor exists to deliver the most competitive solution that demonstrates their expertise in some technology domain. Customers select software from these vendors based on a large number of selection criteria such as price, features, functions, ease of use, performance etc.
In research conducted by Market Focus, several large enterprise development teams were polled and asked about their software selection process. Every one of the respondents placed Integration-With-Other-Vendor’s-Tools in their top five criteria. One respondent even remarked, “We have been living in a tool silo world; now we need more integration between tools”. Another commented, “We like integrated, end-to-end solutions here; that is integrated, best-in-breed solutions”.
Hence, vendors are required to integrate their tools with other vendors’ tools but they are driven to make their tools’ features the most advanced and compelling expression of their domain expertise. These two goals are in conflict and it reflects in the choices they make when investing their development resources. We constantly see stunning improvements in a tools features, functions, performance and ease of use but the integrations that we rely upon and which were once stable and reliable break all too frequently.
For the vendor there is money to be made from integrations but it is usually to close the deal to acquire a new customer. We have all used the leverage of a new software purchase to get the maximum from our selected software vendor. This often includes requiring them to integrate the tool with some obscure, or not so obscure, technology we have previously selected or even something that we have handcrafted ourselves. The vendors do this to close the sale but the code they develop is often done in a rush, to meet a set of minimum requirements, will probably not be reused, and all too often is a reverse engineering achievement of another vendors published (and unpublished) API. We might ask why do we sign off on these technologies that are nothing more than bailing wire, band-aids and bubble gum? Perhaps, we are partly to blame for the current state of affairs.
When it is time to renew the tool, how much effort goes into renewing the integrations? How many of the integrations you have in place today were originally developed as a consulting services offering in order to close the deal with you. Do your vendor’s developers know anything about the solutions developed by the consulting team?
In short, it is no surprise that integrations fail: it is far more surprising that they do not fail more often.
A Glimmer of Hope or False Dawn?
To be fair many vendors are acutely aware of this problem and some have put in the effort to try and resolve this. Enter the “framework vendors”. These vendors have begun to develop an integration framework to enable all their tools to work seamlessly, effortlessly, flawlessly (insert your marketing hyperbole of choice) together.
These vendors though are still missing the point. Developing a proprietary framework for your own tools helps no one but the vendor, and it helps them in more ways than one. Of course, it facilitates good integration amongst the vendors’ own tools but no one vendor has all the tools we need in today’s modern development environment. Thus, we are still faced with having to get one vendor’s framework to integrate with another’s. But, the more insidious aspect of the proprietary framework is that many vendors attempt to use this to lock a customer into that vendor’s proprietary stack of tools, claiming that all their tools will work better together: of course, they will! But if you want to pick a best-in-class tool from some other vendor, will it work just as (insert hyperbole) as the framework’s vendors tool? These framework vendors usually have a good selection of tools but rarely all of them are world class, even if they are, there are always some compromises of platform or architecture that are forced upon us, the consumers.
The Answer is Staring Us in the Eyes
What would a world look like if the tool vendors, who provide our development infrastructure, had an interoperability framework? Vendors would have to write and maintain one integration, to the framework, rather than dozens. The framework would insulate each tool from upgrade issues of another tool allowing us to upgrade parts of our infrastructure without worrying if the other tools were “compatible” with this latest release. We could start to standardise our tooling by plugging.
Caption: Fig. 2: A hub-and-spoke model with defined standards for interoperability ensures quality, robust and rich integrations designed to meet customer’s business needs
This Changes Everything
It is often thought that standards bring conformity, stifle innovation and limit expression. However if we are able to create interoperability standards for the tool vendors in the Application Lifecycle Management (ALM) space, we can change all that.
No longer will a vendor be able to try and lock a customer into their tools because they have a proprietary framework.
Customers will be able to select best-in-class technologies, without having to compromise, knowing that, however small or specialised their chosen vendor, all the tools they select will be able to work together.
Vendors will be able to liberate engineering resources to concentrate on making their tools the best-in-class instead of paying lip-service to integrations and disappointing everyone.
The framework will become a platform for innovation where new solutions to ALM challenges are no longer limited by the existence of tools.
Will it Ever Happen?
Back in April of 2006, Serena Software proposed to the Eclipse Open Source Community exactly this approach to the problem. After a period of global review, the Eclipse Management Organisation approved the project. In the months that followed, many vendors have signed up to support the development of this solution; they all realise that it is in their best interest to develop a common shared interoperability framework and so vendors like Compuware, Catalyst, Cognizant, Segue, Secure and many others are participating in this project.
The project is called the Application Lifecycle Framework (ALF) and is designed to solve, once and for all, the challenge of integrating (actually interoperating) vendors’ tools.
The project is well underway and will release its first “Proof of Concept” version in late January or early February. This code will demonstrate the effectiveness of a common framework for interoperability amongst five separate tools. The plan is for the full version to be delivered in the third quarter of 2006.
What is very interesting is the rapid adoption of this idea by the community at large. Already over a dozen vendors have signed up for the project. Even the giants like IBM and Microsoft are aware of the project and are keeping a watching brief over the outcome before deciding if they too are going to participate.
Caption: Fig. 3: Through simple orchestration technology, ALF enables each customer to organise the technology they have chosen into tool workflows that enforce the customer’s business processes.
Integration is Inadequate
Perhaps the most surprising aspect of this project is that it intends to leave behind the idea of integration and, instead, drive the industry towards interoperability.
For most of the tool-to-tool integrations vendors develop, they are really just automating cut-and-paste with, perhaps, the exchange of some status or semaphore. In today’s software development environment, this is simply not enough. Tools need to be fully aware of each other’s capabilities and needs. Let us look at an example.
Today, we are all familiar with the “Release Build” stage of the development lifecycle. But in reality, this phase comprises of many separate sub-tasks often executed by a combination of human and multiple tool interactions. However, the process is entirely repeatable, or should be, and would benefit from more automation.
Let us look at the steps involved in the Release Build. We need an event to trigger the process, maybe this is recorded in our task management tool. Next, we need to collect the source code from the configuration management tool (2). However, we may want to check that all code changes were completed and approved, this would be in the defect and issue management tool (3). Once we have collected all the sources we need to build them with our build management tool (4). The resulting build will need to be scanned for viruses and other kinds of security issues by our security scanning software (4). If this passes we will run the final automated test scripts against the production image (5). Finally, if everything passes, we will move the resulting production build image to the staging server, and it is now ready for deployment (6).
Today, in most shops, each of these steps is executed by the Build and Release Engineer. However, a lot of these steps are entirely repeatable and predictable, so why are they not already fully automated? Indeed, in some shops, they are: Build and Release Engineers spend an inordinate amount of time creating scripts to automate as much of this process as they can because the diverse set of tools they are working with are not already integrated.
Orchestration Makes the Big Band Sound
One of the primary design features of ALF is the orchestration layer. Its purpose is to enable the interoperability of not just two tools together but a whole host of enterprise technologies.
When we look at how different enterprises use their technology infrastructures, we realise that one-size does not fit all. Every enterprise has its own way of doing things. When vendors integrate, they aim their integration at a lowest common denominator which satisfies few and infuriates many.
The approach in ALF is to allow end-users to customise how individual tools will collaborate within the interoperability framework. We call this process, in ALF, orchestration.
Every vendor’s tool can be ALF enabled by the creation of web service wrapper around their existing API. This will expose events that can be called by other ALF-enabled tools according to the rules specified by the end-user. In this way, each integration (or rather each “interoperability”), can be custom developed to match the business processes of each enterprise.
In our Release and Build Management example, we would have web services for each of the tools involved and when the “Build the Production Release” event arrived at the ALF framework. These tools, once installed or configured, would register their events with the framework. The end-user would then orchestrate those events into the sequence that meets their development processes for a Production Build event. It might go like this:
-
Launch Production Build Orchestration
-
Launch Issue and Defect Tracking web service with Validate Approved Changes
-
If not “Approved”: Exit
-
Launch Configuration Management tool web service with Get Source Code event
-
Launch Build tool web service with Build event
-
Launch Automated Test tool web service with Execute All Scripts
-
If not “Passed”: Exit
-
Launch Deployment tool web service with Stage event
In reality, this whole orchestration is designed graphically and executed through the embedded BPEL engine within the ALF framework.
A further, critical, aspect of the orchestration capability is the mapping of the data payload from one web service to another. From the outset, ALF has been designed with extensibility in mind. Each ALF-enabled web services’ results payload can be mapped to another web services’ input payload. In this way, the user can determine how richly and deeply they want the data integration to go. It also means that vendors can open up their tools without worrying about how the data gets to their API or indeed what happens to the resulting data once the input data has been processed.
You Can Get Involved
The ALF project will be publishing its first code at the end of January. The project calls this first version the Proof of Concept. It is a fully functional version of the framework and several vendors products orchestrated together in a production build scenario. The code, and a self-running demo, can be downloaded from the Eclipse ALF website from early February at http://www.eclipse.org/alf.
Over the next few months, the project is completing the code and making it production ready. The plan is for release 1.0 of ALF to be made available in the third quarter of 2006. Between now and publication of version 1.0, the most critical task in the project is the building of the ecosystem. It is essential that the vendors of Application Lifecycle Management tools start to ALF-enable their products. The specification for how to do this is already posted on the ALF website.
This is where the end-user community can become very influential – by lobbying your vendors and encouraging them to ALF-enable their products.
The great thing about Eclipse, and other open source projects, is that they are open and transparent. This means that you can get involved in the project in another way. By participating in the weekly project meetings, you can give your input on the project requirements, framework architecture or help with the planning and project management. You can even help with contributions of code to the framework.
The World According to ALF
There is an episode of the 1980s situation comedy ALF: Alien Life Form where he is introduced to a jigsaw puzzle for the first time. He lets the pieces cascade through his fingers and says to Willie, “It is broken!” Willie answers, “That is the object: you are supposed to put it together”. To which ALF replies, “Why? I did not break it!”
This is exactly the problem we have today. We have a jigsaw puzzle of technology pieces, all provided by different vendors. The pieces do not fit together, they are from different puzzles, and there is no picture to guide us. The vendors have placed the burden on us to solve the puzzle, but to put it in ALF’s words – “we did not break it”.
The Application Lifecycle Framework hopes to solve this technology jigsaw puzzle. ALF makes sure all the pieces are developed to a common standard so that they fit together. ALF ensures that the pieces all come from the same puzzle and, while the vendors will not actually put the pieces together for you, you will be able to arrange the pieces to match the picture of your technology infrastructure.
Related Reading
 Kevin Parker is the Eclipse Application Lifecycle Framework (ALF) Project evangelist. He was born and educated in the United Kingdom and now lives and works in California. Kevin has over 25 years of software industry experience as developer, designer, consultant and teacher. He works for Serena Software, promoting the ALF project and building the ecosystem of ALF-Enabling vendors.
|
