SOA and EDA - One and the Same

Event-Driven Architecture (EDA) has received a lot of coverage lately. Gartner’s analysts spent a lot of time last year talking about it and promoting their ideas on how it relates to the Enterprise Architecture. A lot of companies, lead primarily by the financial institutions, have taken the plunge into the EDA. Alerts offered by virtually every bank are a good example of this. However, I do not feel that the relationship between EDA and SOA has been fully explored yet.

EDA is based on the concepts of events, publishers, and subscribers. At the most basic level, the idea behind EDA is that publishers publish events and subscribers consume them. Of course, some logic and rules must be applied to properly route the events. Through this mechanism, systems become connected in a loosely coupled fashion. This makes the integrations a lot easier and eliminates the need for each publisher and subscriber to know the details of how to communicate with each other.

Sounds familiar? Absolutely! Change the terms events, publishers, and subscribers to services, providers, and consumers respectively and the paragraph above reads like an explanation of SOA. Why is this, you would ask? Because EDA is nothing more than an asynchronous version of SOA. The major difference between the two is that services are typically implemented as real time calls while events are published and consumed asynchronously. All the other concepts are virtually the same.

While the architectural approaches and design patterns for EDA are slightly different from SOA, the fundamental concepts are still the same. A central event handling infrastructure that knows how to receive, route, and transform the messages is required. It should be viewed as practically the same thing as the Enterprise Service Bus (ESB). In fact, generically, I would call it the Enterprise Eventing Bus (EEB). As events are published, they need to be translated into a common representation, so that a consistent set of rules and operations can be applied to them. A canonical model is the best solution to achieve this goal. Additionally, the same façade pattern should be used as described in the SOA Façade Pattern post to abstract the publishers from knowing and being tied directly to the Enterprise Canonical Model. Note that the logical EDA architecture presented below is very similar to the one introduced in the SOA Façade Pattern.

Events, just like services, should be registered and be discoverable via a central Registry mechanism. Subscribers looking to consume specific events should be able to discover them at design time and receive them from EEB at the run-time. Design of the events, their publication and consumption should also follow the same standards and patterns as service design. All of the related documentation should be stored in the same centralized repository. Governance mechanisms applied to events should also be very similar to those applied to services. EDA does not require establishing a completely new governance model but can effectively leverage existing SOA governance practices.

The bottom line is that EDA and SOA are virtually the same architectural approaches. If you think about EDA as an asynchronous version of SOA, everything else will fall in place.

SOA for the REST of Us

There has been a lot of discussions comparing REST with SOAP and presenting various points of view on how these technologies affect SOA. The opinions expressed are widely varied. Some believe that REST is the best thing since sliced bread because it reduces load on the network and eliminates the unnecessary complexity of the WS-* standards. Others argue that REST cannot yet be effectively used with SOA due to the protocol’s relative immaturity. More and more, however, we are hearing voices of moderation that consider both approaches complementary.

I believe there is place for both REST and SOAP in an SOA program. You cannot necessarily prohibit the use of one technology over the other with one notable exception – you cannot yet standardize solely on REST. SOAP should still be considered the preferred protocol with REST being utilized in very specific situations. There is a number of reasons why.

1. Enterprise applications require enterprise capabilities
   REST was designed for simple Web-based interactions, not for complex enterprise applications. A plethora of WS-* standards exist specifically to address the complexity of enterprise integration and interaction needs. Capabilities such as transactions, security, policy, guaranteed delivery, and many others are a must in any enterprise caliber system. REST does not yet support this level of standards and, most likely, never will due to its focus on simplicity and performance.


2. Security
   REST does not support any specific security standards. In fact, it relies on the infrastructure and middleware to secure end-to-end communications. If service consumers have more robust and complex security requirements than can be met with the underlying infrastructure alone, REST becomes insufficient.


3. Strong contracts
   Strong adherence to service contracts is the cornerstone of any SOA implementation. It ensures that services expose well-defined contracts and provide adequate information on how to access them. SOAP-based web services have built-in capability to validate their contracts. REST does not inherently support this type of verification. In fact, all the interactions between the consumers and RESTful services contain nothing more than a command and a list of parameters. It becomes the responsibility of the service to ensure the validity of the contract.

As a general rule, REST should be used when performance and simplicity are paramount, no special security requirements exist, and no complex interactions between the consumer and the service are necessary. The best uses for RESTful services should be considered simple extracts of small data sets, inquiries against open public data sources, calls to external vendors supporting the protocol, etc.

A number of SOA and middleware vendors are feverishly working on creating solutions that support both SOAP-based and RESTful services. Very soon, REST and SOAP will simply become some of the many communication protocol choices existing in the rich SOA toolbox. Most of the differences will be eliminated by a combination of server-side technologies and client-side frameworks. However, until this happens, strict standards should be established guiding the use and adoption of RESTful services.

SOA and OO

While SOA and OO are based on some very similar concepts, the approaches they utilize to achieve them are quite different. The biggest difference lies in the distributed nature of services. OO technology assumes local, in-memory calls while SOA assumes a distributed network of services. This fact drives a number of differences between the two technologies.

One of differences is the cost of traversing the distributed network boundaries. It is nontrivial in terms of complexity and performance. Service-oriented designs acknowledge these costs by putting a premium on boundary crossings. Because each cross-boundary communication is potentially costly, service-orientation is based on a model of explicit message passing rather than implicit method invocation. Compared to distributed objects, the service-oriented model views cross-service method invocation as a private implementation technique, not as a primitive construct — the fact that a given interaction may be implemented as a method call is a private implementation detail that is not visible outside the service boundary.

Another difference is autonomy of the code. Object-oriented programs tend to be deployed and act as a unit. They are not autonomous but rather embedded into a managing container. Service-oriented development departs from object-orientation by assuming that atomic deployment of an application is the exception, not the rule. Services are deployed, managed, and run as autonomous units. They are, in fact, the containers, in which OO code lives.

The final difference is the architecture. Since OO almost always deals with in-memory calls, it places no premium on the number of method invocations and the amount of data passed. In fact, the best OO interface contains many methods that are created for a single purpose. Under SOA, network overhead needs to be considered. This fact puts a premium on the amount of calls made from the client to the service. Thus, the number of service calls should be limited while the amount of data passed should be maximized.

Keep in mind that OO and SOA are not competing but rather complementary approaches. Think about service as an outer shell of an OO application that enables it to become sharable across the network.

Creating and Managing a Canonical Model

As I discussed in my previous post (http://leoshuster.blogspot.com/2008/05/soa-canonical-modeling.html), a canonical model is a critical element in any successful SOA program. However, given the fact that it should describe the business completely and in a standard way, the resulting model will most likely be very large and complex. Thus, a natural question begs to be asked: how do you create and maintain a canonical model?

There are basically three ways to create a canonical model.

1. Buy it or adopt an existing industry standard model
   There is a number of organizations that either developed a set of standard models targeting a specific vertical (e.g. IBM - http://www-03.ibm.com/industries/financialservices/doc/content/bin/fss_ifw_gim_2006.pdf) or maintain industry standard definitions (like MISMO for Financial Services or ACORD for insurance). You can adopt one of these models to serve as the canonical representation of all the business entities.




Pros:



• Most of the work is already done
  
• Another organization maintains the model for you and introduces changes as necessary
  
• The model is standard and should help with external partner integration
  


Cons:



• The specifics of your organization may not be complete captured, which require custom additions to be made
  
• Some changes or customizations may be needed that would make it harder to upgrade in the future
  
• The elements are too generic or unnecessarily complex
  
• High learning curve for canonical model consumers
  






2. Create it from scratch
   A canonical model is created from scratch and built out completely before any work utilizing it can begin. This would require at least 3-6 months of effort meeting with various groups across the organization, collecting and sorting the information, and validating the result with the potential users.




Pros:




• Would provide the most complete and targeted model
  
• Users will have innate knowledge of the model since they helped build it
  


Cons:




• Requires all projects that need to use the canonical model to stop until it is completed
  
• Not highly realistic or pragmatic approach
  
• Requires modifications to be made and managed internally
  






3. Build it incrementally
   A canonical model is built incrementally over the span of multiple projects. Only those elements that are required by the project are added or modified.




Pros:




• Does not require a lot of upfront effort to get started
  
• Efficient and demand-driven – model only what is needed
  
• Low learning curve – users have more opportunity to learn it as the model evolves
  


Cons:




• High propensity for change – the model is frequently refactored as new projects leverage it
  
• Requires a centralized team to own or govern it
  
• Frequent changes would require a large amount of testing and updates to the existing consumers
  

The best way to manage the changes to the canonical model is to establish a centralized team to own the whole thing or to provide governance over it. It would be responsible for making / tracking the changes, notifying consumers, performing compatibility testing, versioning, training, and communications. A comprehensive list of all the canonical model consumers needs to be maintained in order to notify them of all the relevant changes and understand the overall impact of modifications. Without a centralized team, there can never be a canonical model because there will be no one to synchronize or drive all the disparate efforts towards a single goal.

Regardless of the chosen approach, changes to the canonical model are inevitable. Therefore, the façade pattern described in one of my earlier posts must be utilized when using canonical models in the SOA context.

SOA Façade Pattern

In the last post, I discussed the value of the canonical modeling and described how to minimize the impact of canonical model changes on the service consumers. The solution was to use the façade pattern. I would like to elaborate on this topic since a more in-depth discussion is needed to define the pattern and understand its uses.

A good definition of the façade pattern can be found on Wikipedia: http://en.wikipedia.org/wiki/Facade_pattern. In general terms, it is described as “a simplified interface to a larger body of code”. This is exactly how it should be applied to SOA. A façade should be built in front of any service whose interface is based on the canonical model. Consumers would not access the service directly but rather through its exposed façade interface. In fact, the canonical interface should only be exposed for internal consumption. Each façade should be designed to be specific for each consumer or a group of consumers and not directly tied to the canonical model. The diagram below depicts the pattern details and its usage.

There are several distinct benefits of using the façade pattern.

1. Façade shields service consumers from the changes in the canonical model.
   If every consumer was dependent on the canonical model, even the smallest change could have disastrous effects. All of the services as well as potentially all of the service consumers would need to be re-tested. Lacking automated regression and functional tests already developed, this would be a major undertaking. Using the façade pattern would minimize the impacts of any canonical model changes. Since the facades are specific to each consumer and are not directly tied to the canonical model, the only thing that would need to change is internal mapping between the façade interface and the canonical model.
   
2. Façade hides the complexity of the canonical model.
   Modeling the whole business domain is not a simple task. Therefore, canonical models are usually large and complex. Service consumers do not typically want to know the entire canonical model and understand all of its intricacies. They want to get the data they need and continue performing their business functions. Exposing a consumer-specific interface via the façade prevents service consumers from having to know any canonical model details. Additionally, since canonical models are fairly generic, most of the data elements in the returned entity may not be relevant to the consumer. A façade simplifies the request and response data structures and ensures that only relevant information is returned.
   
3. Façade returns data representation understood by the consumer.
   A canonical model is generic. It is designed to describe the whole organization. However, service consumers typically operate in their own specific domains. Service façade that is designed to return data in a format that consumers understand simplifies the overall consumption experience and reduces the overall efforts. The consumer does not need to perform any translations and can start working with the data right away. Additionally, a façade can help representing the same entity differently for different consumers if so required. There may be instances, for example, when one Line of Business (LoB) thinks of a customer one way while another LoB views a customer completely differently. These views may even be largely incompatible but as long as they are represented in the canonical model, a façade can be created to address specific LoB needs.

The façade pattern introduces a small translation layer between the service consumer and the canonical service interface. It should not contain any generic business logic but can perform some consumer-specific operations. While it may cause reduction in performance and increased development costs, the negative impacts should be minimal and will be offset by the benefits described above.

SOA & Canonical Modeling

There has been a number of discussions in various articles and blogs on the topic of application and value of canonical modeling for SOA. Majority of industry experts support the use of canonical models as one of the key SOA patterns. However, there are some that consider canonical modeling as detrimental to SOA (http://service-architecture.blogspot.com/2006/08/single-canonical-form-not-for-soa.html). I would like to add my voice to those that consider canonical modeling a critical aspect in the success of the SOA program.

Primary goal of any SOA program is to introduce a variety of reusable services. Reusability typically implies that a service has a number of different consumers and data providers. The first and most obvious value of a canonical model is that it acts as an abstraction layer between all those consumers and providers. It is an old and well-known design pattern – when you have a number of data sources and their consumers that need to be integrated together, you introduce an abstraction layer, so that neither is aware of the internal details of the other. This way, any changes made to a consumer or a provider will have minimal impact on all of its integration points. This is the second benefit of using a canonical model. It minimizes the impact of internal service changes, modifications of data sources, or switch to a new backend data source on the service consumers. The canonical model should remain unchanged regardless of what happens inside the service, which, in turn, ensures that the contract between the service consumer and the service itself remains unaffected. The maximum possible impact on the service may be the need to change the mappings between internal service data structures and the canonical model.

Since the canonical model minimizes the impact of internal service changes on its consumers, it also reduces the need for regression testing. (This is the third benefit of canonical modeling.) If services did not provide a layer of abstraction between its internal implementation, backend data sources, and its consumers, any change inside the service or data provider would be reflected in its interface and thus would require a full regression test. Usage of a canonical model eliminates the need to perform rigorous regression testing since, as we discussed above, any such changes would not impact the service consumer. The only thing that would need to be done is perform a test validating that the service contract did not change.

Representing standard structures in a canonical model maximizes service reuse (fourth benefit). Consider all the entities that your business deals with every day. It could be customer, account, price, payment, etc. However, different parts of the organization may view these entities differently. One division, for example, may care about the customer household information while other about his/her geo-positioning. Without a canonical model, you would end up with multiple slightly different representations of the same entity. This results in services being built based on disparate models targeted for only specific audiences. Other groups trying to reuse these services would require changes to address their needs or would simply not be able to consume them. Representing all entities in a standard way eliminates this incongruence and allows different parts of the organization to speak the same language. This, in turn, maximizes the potential and real reuse of services built across the company.

Of course, some would argue that any changes to the canonical model would impact all of the service consumers and they would be right. In order to minimize this risk, a façade pattern needs to be used. Rather than exposing the canonical interface directly to the consumers, a façade would need to be build based on each consumer’s needs. It would expose data contracts specific to each consumer or a group of consumers. The service would never be called directly but only through one of its façade interfaces. This way, any changes made to the canonical model would not impact the service consumers directly. The façade would remain intact. The only change that would potentially need to be made is modification of mappings between the façade and canonical data structures.

Used together, the canonical modeling technique and the façade pattern will maximize the service reuse and minimize the impact of internal changes on service consumers. The approach will save costs and time on regression testing efforts. Regardless of what the opponents say, use of these techniques is critical to the overall SOA program success.

Links to some good SOA & canonical modeling articles:
http://www.it-eye.nl/weblog/2007/06/13/soa-best-practice-9-use-a-canonical-data-model/
http://www.ibm.com/developerworks/db2/library/techarticle/dm-0803sauter/
http://www.soapatterns.org/canonical_interface_expression.asp
http://reallifeserviceorientedarchitecture.blogspot.com/2007/11/how-to-create-canonical-form.html

SOA & Agile Software Development

Many industry articles and experts would like you to believe that SOA and Agile software development methodologies (let’s call them “ASDM” for simplicity sake) are made for each other. People like Gregor Hohpe of Google or Brenda Michelson of ebizQ would like you to believe that Agile and SOA are a powerful combination. I would like to set the record straight once and for all. ASDM, regardless of the flavor or the sort, is the worst thing you can apply to your SOA program.

Agile is created to build software quickly while requirements continue to shift thus allowing the end users to see the results in days rather than months. Refactoring (read “change”) is assumed as normal and is, in fact, welcomed. Software design is often organic and evolves with each story card or iteration.

SOA is an architectural style that requires rigorous planning, forethought, and discipline. Most experts will tell you that A (“Architecture”) is the important element in SOA. Services must be designed with an eye towards the future reuse, not immediate requirements. Contract-first development, which is inherent in any SOA approach, is largely alien to an ASDM. Same is true for comprehensive design cycles that focus on designing reusable, flexible, and architecturally sound services. Agile simply has no room for design. Code is self documenting, Agilists will tell you. Some Agile flavors that account for some design time still focus on a very narrow set of requirements and never take the big picture into account.

The most telling example of incompatibility between SOA and Agile is a project that needs to build a large number of shared services for a large number of consumers. Under an ASDM, each service would be built incrementally, over time as the story cards with new requirements come up. As new consumer’s requirements are satisfied, the service must change, which most likely triggers the need to test the impact on the existing consumers. The more consumers the service has, the more testing must be done. Using a model-driven design and contract-first development would undoubtedly solve this problem. Service interface would be modeled in its entirety and a complete contract presented to each consumer. This would eliminate the need to retest each subsequent consumer integration.

To take this example even further, imagine that this needs to be done for several, not just one project! And timelines are not evenly in synch. Amount of refactoring and testing would become enormous. Throw in the typical governance processes each organization has in place, a registry that keeps track of each service’s lifecycle and policies, service management platform that must be integrated with it, an ESB through which services must be exposed, etc. and you will get an even better picture. Without proper planning, design, and architecture, your SOA program will not succeed. No Agile methodology applies the level of rigor and depth required to build truly reusable services. Stop using Agile before it’s too late for your SOA program!