Business Process Excellence for Financial Services 2011 Conference

I will be presenting two sessions at the IQPC Business Process Excellence for Financial Services 2011 conference. The first one is a pre-conference workshop on Service Oriented Architecture And Business Process Management: A Comprehensive Approach To Business Performance. The second one is the first session of Day 1, Embracing Truly Integrated BPM: A Holistic Approach. This should be very exciting!

SOA Misconceptions

I continue to see a number of common misconceptions about SOA. There have been a number of articles written about this when SOA was first introduced. They primarily concentrated on dispelling the myths that SOA = Web Services. Current misconceptions run deeper and a lot more general.

1. SOA is Expensive
SOA doesn't have to expensive. There is an abundance of open source tools and technologies that can be used to build a truly state of the art SOA platform.

2. SOA is Inefficient
SOA is as inefficient as you make it. Not all the problems and organizations require a complete SOA stack but, at a certain level, it becomes necessary. Otherwise, you will have too much complexity and, in fact, breed inefficiency.

3. SOA is Non-productive
A well established SOA program can run like a well-oiled machine. I am sure a lot of people can cite numerous examples of lost productivity because of unnecessarily complex SOA environments and processes. However, in large and complex corporate IT departments well defined organizational structures and processes are a must. They actually make things more efficient and increase productivity. Every situation requires a different approach. However, SOA, as a general pattern for building software, has been shown to dramatically improve productivity. Think about it -- if several projects can reuse an already existing, well tested service, this results in tremendous productivity improvements and costs savings!

4. SOA is Unnecessary
This is true, with a caveat. SOA has been employed as an IT and business strategy to make organizations more efficient, productive, and agile. If you want your company to achieve these goals, SOA becomes a part of the answer. Otherwise, you are stuck in the world of point-to-point integrations, Just a Bunch of Services, and an unmitigated integration mess in general. Obviously, smaller companies can get away without employing SOA for a much longer period of time but larger ones will feel the pain much sooner.

5. SOA is Too Complicated
Yes, SOA is complicated. But what major effort isn't? Enterprise Architecture is complex, so we shouldn't do it?! EAI was complex but companies did it out of necessity. Master Data Management is complex, so companies should forget about managing their data?! SOA can be as simple or complex as you need to make it. Create a roadmap for your SOA program and follow it. It should guide you in your quest to achieve SOA maturity, however simple or complex you need to make it.

SOA is a program. You can make it into whatever you need. You can use whatever technologies and approaches you like. As long as you keep the goal of increasing agility and saving money as a result of creating reusable business capabilities in mind, you should be successful.

SOA and the Trough of Disillusionment

All the signs point to the fact that SOA has entered the trough of disillusionment. Why? I am seeing a lot of companies de-funding or stopping SOA efforts altogether. Many executives don't see the hype anymore and thus no longer understand the value of SOA. Many companies have tried SOA and failed, so they don't want to try it again. Plus, the economy is not helping. Budgets are getting slashed, jobs are lost, people are afraid to innovate. The first type of spending that companies cut is strategic. They want to get through the tough economic times with minimal required spending -- just to keep the lights on. Thus, SOA program funding gets cut and the executive support goes downhill with it.

What can we do to make sure that SOA moves onto the slope of enlightenment and further to the plateau of productivity? Continue to educate your organizations and executives about the value of SOA! Even though SOA is not in vogue anymore, it doesn’t mean that it lost its value. The benefits are still there. You just have to work harder to make them visible. Document your successes and shift into a marketing mode. Promote the SOA program and the benefits you achieved as much as possible. Show real value. This will certainly get executives’ attention and guarantee their support. Don’t give up. Continue to fight negative perceptions and concentrate on delivering value.

SOA, like any other enterprise wide initiative, is a differentiator. Companies that can successfully implement SOA will become a lot more successful than those that can’t. When the economic crisis ends, organizations that continued to invest into SOA will come out on top. Those that didn’t will be left behind and will need to spend a lot more money and efforts to catch up. SOA can produce tangible business benefits. Don’t let the disillusioned minority silence its value proposition.

SOA Funding Models

One of the primary reasons SOA efforts fail in many companies is simply due to inadequate or inappropriate funding models. Costs are typically at the core of every problem and SOA programs are not exempt. We hear horror stories all the time – the initial investment to establish an SOA environment was too high, so the effort was cancelled; there are many services created in the company but they are hardly reused; etc. Establishing a funding model that is right for your company is the key to moving the SOA program forward.

Any SOA initiative is comprised of two parts – infrastructure and services. Both need to have a separate funding model established in order to successfully support SOA program’s goals.

SOA Infrastructure Funding

Infrastructure funding requires a pretty straight forward approach. When discussing SOA infrastructure, I am referring to shared platforms that are used by a number of services across the organization. Some companies host services on the same platforms whose functionality is being exposed. However, even if this is the case, some shared infrastructure components like ESBs, service management technology, Registry/Repository, etc. must exist to support SOA program’s needs. Thus, it is safe to assume that some form of shared SOA infrastructure exists. There are two possible ways to provide effective funding to build it out.

1. Fund all the SOA infrastructure centrally
2. Identify appropriate projects to acquire / extend new / existing SOA infrastructure

Central funding is probably the easiest and most effective approach. It allows the organizations to establish an independent roadmap for introducing and upgrading SOA infrastructure. It also makes the SOA program operate more efficiently as the cost, scaling, and availability issues will no longer be relevant to individual projects. If central funding option is selected, several approaches for recouping the initial and ongoing investment can be utilized.

• Do not recoup the investment
• Place an entry fee to use any SOA infrastructure component
• Charge a small fee for each usage instance

Since all the SOA infrastructure is provided centrally, not recouping the initial investment is a real option. If the organization’s fiscal model does not call for IT recouping all its costs from the business groups using their products, this option works well. If this is not the case, however, you have a choice between placing a predefined entry fee that each application / project must pay to use the specific SOA infrastructure platform and charging end users based on the total usage.

The per-use-fee scenario is a little tricky as each SOA infrastructure component needs to define what a transactional unit is and how much to charge for it. Transactional units can be different for each SOA platform. For example, an ESB transactional unit can be a service call, Registry/Repository – an individual user and/or a UDDI request, etc. In this case, total usage amount based on predefined transactional units would be calculated, multiplied by the unit cost, and charged to the business units. The most effective way to determine a unit cost is to divide the total investment made in the platform by the total transactional units being consumed. The obvious effect is that unit costs would decrease with increased usage. Here are all the formulae discussed above.

Usage charges per platform:
Unit = Different per Platform
Unit Cost = Total Platform Investment / Total Amount of Units Consumed
Line of Business Usage = Units Used by Line of Business * Unit Cost

Some companies have chosen to grow their SOA infrastructure gradually, without a central program or funding. A typical approach in this scenario has been to attach SOA spending to the most appropriate projects. Thus, the projects would purchase new SOA infrastructure platforms or upgrade existing ones to suit their needs. There are several problems with this approach.

1. Typically, the projects purchasing the infrastructure don’t want to share it with other potential consumers unless there is significant pressure from above. The platforms don’t end up being reused or, if so, only minimally. The projects do not have any incentive to sharing their investments with anyone else, especially since they are seen as critical to projects’ success.
2. Projects often get cancelled due to over-inflated budgets. SOA infrastructure is expensive and cost conscious enterprises do not want to invest into what looks like excessive infrastructure for project’s needs.
3. Demand to extend a platform based on project’s needs typically comes without enough lead time to accommodate project’s timelines. Thus, projects face a tough decision – to extend their delivery date or use alternative infrastructure.

Funding the SOA infrastructure centrally is more effective in delivering service-oriented solutions faster, moving the enterprise more efficiently towards a higher level of SOA maturity, and addressing the project needs. Project-based funding will most likely spell doom to the SOA program as a whole.

Service Funding

As discussed earlier, funding for the SOA infrastructure should come from a central source. Where the money comes to build individual services, however, presents a bigger challenge. Since projects are the primary drivers behind demand for services, special consideration should be given to project needs and budgets. However, service design and implementation can incorporate additional requirements that fall outside of the project scope. Another typical project-related problem stems from the shared nature of services. It is unfair to burden a project with the full cost of a service that will be utilized by a number of other consumers.

There are three possible ways to address the service funding concerns.

1. Make the first project to build a service provide the complete funding
2. Establish a central funding source that will cover all service design and construction expenses
3. Provide supplementary funding to projects building services

If option1 is selected, several strategies for recouping the initial investment can be used.

• Do not recoup the investment
• Place a surcharge on each instance of service leverage
• Charge a small fee for each service call

As mentioned above, it is unfair for the project to carry the complete costs of the service build-out, especially if it includes additional requirements. Thus, unless the project implements one of the options to recoup its initial investment, funding option #1 is not going to be viable. Not recovering the funds is not a realistic option either as it does not incent the projects to build truly reusable services. The other cost recovery strategies may work but require detailed metrics to be captured on the service leverage and/or transactional volume.

Establishing a central funding source for all projects to use when building reusable services is probably the ideal approach. Few companies, however, would be willing to write what in essence would be a blank check for the projects to use in their service delivery efforts. The opportunity for abuse and misappropriations would be too tempting. Unless strong governance and control mechanisms are in place, this funding method will most likely end up costing the company more money and provide unrealistically small return on investment.

Providing supplementary funding to projects building services is probably the most realistic approach. A central fund needs to be established to cover the efforts falling outside of the project scope. Since shared services would typically incorporate other projects’ and enterprise requirements, the actual cost ends up being higher than what projects budgeted for their needs. Thus, the easiest way to distribute supplementary funding is to allow the projects to pay for functionality already included in their budgets and cover all the additional costs through the central fund.

Whatever the funding approach is used, it needs to be carefully administered. A party not involved in day-to-day project work is best suited to play the administrative role. There could be a couple different groups managing the infrastructure and service funding and chargeback mechanisms. Overall, however, this should fall under the SOA Governance umbrella and managed centrally as part of the SOA Program.

EA & SOA in Down Economy

Many articles, discussions, and presentations have been made on how the recent economic downturn affects the companies’ EA and SOA efforts. Some, like Joe McKendrick of ZDNet, believe that slow economy can spell boon for SOA but subprime SOA will suffer. Dave Linthicum published the results of his survey indicating the same. Beth Gold-Bernstein of ebizQ suggests that we bet the farm on Complex Event Processing. Ronald Schmelzer of ZapThink advocates that companies should continue to invest in SOA in a down economy. I would go a step further. EA and SOA investment is critical to the companies’ success – it will enable them to stay competitive, achieve significant efficiencies, and potentially even gain market share.

Pretty bold statement, some would say. I don’t think so. Let’s consider the facts.

When the times are tough, the first thing most companies do is slash budgets. IT budget gets reduced just like everyone else’s. The focus shifts from the strategic initiatives to simply keeping the lights on and completing projects as quickly as possible. Enterprise Architecture efforts are usually the first ones to be eliminated or significantly reduced. Point solutions become the norm resulting in duplication of software, hardware, and overall efforts. Smokestack applications rise up from the ashes of the Enterprise Architecture. Everyone becomes more concerned about keeping their jobs rather than doing the right thing for the company. IT managers shift to the aggressive empire building mode in order to protect their jobs and eliminate their own risks. (The old mentality of “I own more than you, therefore I am more important than you” is still alive and well, unfortunately. IT managers also think that if they can “own” and control every piece of their application, it will reduce their risk and allow them to deliver results faster.) Governance becomes unenforceable and largely forgotten.

Through this chaos, interesting trends emerge. While the initial IT budget is reduced through a series of staff reductions and some technology rationalization efforts, the costs begin to creep back up in subsequent years. When the economy finally turns around and the pressure to keep the budget low eases, the IT budget suddenly becomes larger than what it was prior to the cuts. Why? The explanation is simple. The empire building and unfettered decision making by IT management finally bears fruit. There are more software, licenses, hardware, and code in the data center, all of which requires more people to support. There is very little reuse and sharing because each group has built silo applications residing on their own unique platforms. Costs increase, efficiencies decrease, and it takes longer to deliver new capabilities especially if they require several applications to integrate with each other.

Enterprise Architecture and SOA can help reverse these trends and, in fact, keep the IT budgets low. Most companies have a number of redundant systems, applications, and capabilities that have grown through the type of uncontrolled behavior described above. EA, through an effective discovery and governance mechanisms, can eliminate these redundancies while maintaining the same capacity and level of operational responsiveness. Additionally, EA groups can influence or implement new architecture approaches to help consolidate resources and gain efficiencies. Examples of this could be virtualization, green technologies, cloud computing, etc. SOA, as a subset of EA, provides much the same benefits. Encapsulating key business functions as reusable services will help achieve more consistency, save money, and enable faster project delivery. An effective EA program can protect companies’ IT budgets from ballooning by establishing and enforcing standards, promoting reuse opportunities, and ensuring transparency across all IT systems.

The bottom line is that companies can not afford not to invest in EA and SOA. These programs will make organizations more efficient through the economic downturn and help achieve the necessary savings. On the long run, EA and SOA will keep the costs down while increasing business agility. Effective EA and SOA programs are a competitive advantage, not an overhead. They will easily pay for themselves and, what’s more important, enable organizations to avoid uncontrolled spending in the future. Enterprise Architecture and SOA is a must, not an option!

Starting on Your SOA Journey

Many industry leaders believe that SOA should be started small and evolve over time. The argument is that this approach gives companies an opportunity to implement SOA practices on a small scale, test them out in a controlled environment, and understand how everything would work within the organizational boundaries. This is not a bad idea. However, I believe that the most effective way to introduce SOA is to build out the whole infrastructure, introduce the necessary technology, and establish all the patterns, best practices, reference architectures, and governance mechanisms before creating a single service.

The reasoning for this is simple. SOA on a small scale is not SOA. It is just a bunch of services. SOA’s goal is to create and leverage services across the organization. A single project or a couple of services cannot achieve this. Furthermore, effective governance, best practices, and lifecycle processes cannot be established on a small scale. They need to be designed and implemented with the large scale in mind. Testing them on a single project is not only impractical – it doesn’t provide any knowledge of how SOA will truly work within the organization.

Any successful SOA implementation will eventually have all of its elements in place – infrastructure, technology, governance, practices, processes, and people. Consider the impact of growing all this organically. You will end up with a hodge-podge of services implemented on different platforms using different approaches and design patterns. The technology set will be inconsistent. Governance mechanisms that typically tend to be established late in the game will most likely allow inadequately designed and implemented services to go into production. All this would have to be remediated at some point of time. Imagine the effort required to clean up years of organic growth! Most companies simply move on and leave the mess behind.

Now imagine what will happen if all of the SOA elements are in place from the very beginning. No rework, re-platforming, or cleanup will be required. All of the services will reside on the right platform that can be scaled for future demands, all of the best practices will be followed, and the governance mechanisms will be able to catch most, if not all the subpar services. The company will be able to reap SOA benefits right away without having to do the costly cleanup or conversion.

Of course, waiting to complete all the preliminary work can take years. No company, regardless of how strong its commitment to SOA is, can wait that long to start seeing the benefits of something that will require a lot of upfront investment. Thus, the most pragmatic approach is to introduce as many SOA elements as possible that will provide the most complete and consistent SOA foundation for the future. This should be achieved within a reasonable timeframe, so that services can start to be built and benefits can be quickly shown. All the remaining strategic tasks should continue to be addressed in parallel with the ongoing tactical service implementations.

The prescription above will not cure all of your SOA ills but will introduce a dose of prevention for the future. Building services following a consistent set of standards, using a consistent set of tools, and deploying on a consistent platform from the very beginning will ensure the success of your whole SOA program, not just a few projects or services.