"Style is everything. I mean, the blues are three chords. But every guy that plays the blues plays differently, because that's their own style. Right?" —Skip Engblom

Introduction

The open source software community is building a wall to contain Microsoft and other monopolistic forces, and middleware is the mortar.

It should come as no surprise, then, that the most important feature a piece of middleware should provide is the ability to wire together disparate subsystems and to allow systems to be broken down, so that specialized communities of developers can each focus on building their bricks, their pieces of the whole.

With enough of these communities working together, there are very few limits to what we'll build. Open source Java will dominate the enterprise market. The days of per-processor licensing fees will be gone forever. Medium-sized businesses and cottage industries all over the world will spring into a new life. The global 2000 enterprises from the 20th century will meet a hundred thousand reinvigorated family enterprises in the 21st. The era of big ideas is ending, and the insects shall inherit the earth.

Okay, seriously — that was a bit dramatic, but you get the point. Competition from the bottom will force the companies at the top to either lower their prices or dramatically improve the quality of their products and services to remain competitive. And open source software will play a driving role in that shift.

The goals of a framework (which qualifies as middleware because of its role in lacing together disparate information systems) were neatly outlined by one Dr. Trygve Reenskaug in the "Administrative Controls in the Shipyard" essay he presented at the ICCAS conference in 1973. The goals and principles he identified are as fresh and relevant today as they were 30 years ago. Dr. Reenskaug suggested that the goals of a framework were to support:

1. Simplified manual override
2. Division into modular subsystems
3. Behavioral transparency of subsystems
4. Pluggable subsystem modules
5. The capability of continuous growth for the total system

But large systems are complex. The only hope for developers who wish to model them is to decompose the larger "total" system into subsystems, and then repeat the process for subsystems, breaking them down into role categories and objects that fulfill the requirements of those roles. In Java, the terms "interface" and "role" are practically interchangeable. Interfaces define the nature of an object but not its specifics, and can limit the scope of interactions between objects in the system according to their public roles. This both reduces the complexity of the system and makes it more flexible and extensible. Therefore, breaking down complex systems into families of interfaces is something we should at least attempt.

The Model, View, and Controller roles of the MVC architectural style have been offered as a means of categorizing components in web application frameworks, but in my previous article on this topic, I argued that the attempts to implement these distinctions have failed to show that this is a correct approach. The primary evidence of this failure is the absence of a single component framework that has managed to isolate the concerns of its components into Model, View, and Controller interfaces.

A correct approach would be to create a new hypothetical style that describes best practices already used in the problem space, instead of prescribing an architectural style from a distantly related problem space. Then the hypothetical new style should be tested through implementation. Any flaws uncovered in the style should be addressed in the refactoring stage, and the cycle repeats itself: hypothesize, implement, refactor.

If these principles are applied to the complex systems that drive dynamic web applications on the server side, we find a family of roles, which should be utterly unsurprising. The roles I outlined in my first article have themselves undergone a refactoring. I've renamed the DataSource interface to State because of confusion between it and javax.sql.DataSource, and also removed the Presentation interface entirely. Presentation components are officially beyond the scope of this problem space, but the representation of the application's state is not. At the suggestion of others in the community, I've added a Representation role, which I'll describe in more detail below. After incorporating these changes, we are left with:

1. Application State components that maintain the incoming data.
2. Action components that can alter the State.
3. Workflow components that decide the order in which Action objects execute.
4. A Representation subsystem to limit the complexity of the State for component developers by restructuring or "pre-chewing" the state to make it more intuitive for component developers.

Since writing the first article, I've been notified of a better term than "pattern" to describe what to call this collection of roles. It's just an "architectural style." Nothing more, and simply put. I call this architectural style WARS, which is very appropriate considering the use to which we put it in designing Java web apps.

I've yet to find a middleware/command-pattern/servlet/MVC (call it what you like) framework whose components can be neatly separated into implementations of Model, View, and Controller interfaces. That isn't to say MVC is a hoax — not at all. It's absolutely great for building user interfaces. It just isn't the best way to describe the architectural style of our middleware systems. On the other hand, there is at least one middleware framework whose components implement the WARS architectural style. Its name is Shocks, and I wrote it as a proof of concept for this architectural style.

But before we get to that, I'd like to note that other frameworks, most notably Apache Cocoon, have already been using this architectural style (or something very close to it) for some time. Since I developed it by studying how things were actually being done in popular systems like Struts and Webwork, and by disregarding the MVC marketing speak, this should not come as a surprise. WARS is a descriptive term for what is already common practice and satisfies the principle of least astonishment.

It's just that we hadn't hit on the right words to describe it. Until now — or so I hope. An insightful (if somewhat aged) letter by Berin Loritsch in the Cocoon archives shows that the Cocoon project was aware of very similar concepts over two years ago — it just never crystallized into our lingua franca. Struts has been decomposing toward the commons-chain package for some time now, so this idea reinforces their work in that area and vice versa. WebWork introduced an extremely sophisticated state machine (in their value stack concept) quite a long time ago. My guess is that the terminology surrounding these ideas prevented them from taking wing. "The difference", wrote Samuel Clemens, "between the almost-right word and the right word is really quite a large thing. It's the difference between the lightning bug and the lightning." Whatever the case, I look forward with renewed interest to all their projects.

Having said that, I want to turn now toward my proof of concept — the Shocks framework. It's a far cry from perfect. But like the first hot air balloon flight in Paris, it has exciting and unexpected implications. There will be better versions, better implementations, evolution — there always is.

After all, you know what they say ...

Any Implementation Today

... Is better than a perfect implementation tomorrow. Already, based on advice from Craig McClanahan of the Struts project, dIon Gillard from Codehaus, and dozens of others, I've made significant changes to the internal structure of Shocks. What I have to present today is a step beyond simply getting "there and back again;" from the web page to one or more action classes that perform some kind of business logic, and back out to the user by way of some kind of presentation subsystem.

The internal architecture of Shocks is very simple and straightforward. There's a shocks.wars package that contains the master interfaces. Figure 1 displays this arrangement.

Figure 1. Arrangement of shocks.wars package

As you can see, the Representation interface is quite shy. The idea for representation components was inspired by Greg Wilkins' recent blog entry about "contentlets" and Roy Fielding's Representational State Transfer architectural style.

A representation component will have access to the application's runtime state, but will not alter the state in any way. Instead, it'll harvest information from the state at runtime, restructure it for easy access, and make it available to a higher context — such as a data repository through JMX.

Representation components aren't action components, because they don't alter the state at runtime. They're not workflow components either, because they make no decisions about the execution of actions. And they're not really state components, because their job is not to passively store data — although the State interface would be their closest relative. If anything, the Representation interface serves as a protocol-neutral bridge between the application state and developers. If well-developed enough, it could completely shelter action and workflow components (and their developers) from the complexity of the state. So Representation gets its own seat at the table.

The State interface is also very minimal. I'm erring on the side of caution for the time being, and allowing other interfaces to extend the basic State requirements. Implementations of those objects can be recast inside of the different Action and Workflow implementations without cluttering up the method signatures with specific dependencies. A toMap() method will probably be in order eventually. Both of State's methods return java.lang.Object, which likewise must be recast. This flexibility comes with an accompanying degree of complexity, which we'll explore more later.

The Action interface is simple. Actions are given a reference to the State and they don't return anything. Their role is to alter the State in some way, or to trigger an external process as a result of the state. The other components in the framework make no assumptions about what actions actually do. A Workflow component can check the state before and after an action has fired, and use that information to figure out what to do next, but has no other way of knowing what goes on therein.

Workflow implementations accept a reference to the State object but, unlike Action objects, they also return a State reference. This is important, because it might not be the same State as the one they received. Right now, Workflow components examine the application state, decide what to do next, maybe fire off some actions, forward control to another Workflow component, and once they're done doing their thing, they give control back to whatever object called them. All they do is examine the state and decide whether or not to execute actions. By themselves, action objects aren't capable of deciding the order in which they execute. They can set a flag in the state, thereby suggesting the order, but ultimately, the workflow components make that decision.

The best part about Workflow components is that the brunt of their decision-making functionality can be replaced with a JSR-94-compatible rules engine. In the future, the role of Shocks' Workflow components will be to delegate decisions to that JSR-94 rules engine by way of JMX. The entire Workflow engine will therefore be completely modular, extensible, and manageable.

The Client API

For action developers who use the framework, we've set aside the shocks.client package. Its job is to abstract component developers from the internal complexities of the framework. At present, this package has three items in it. Figure 2 shows this design:

Figure 2. Arrangement of the shocks.client package

Shocks wouldn't be much of a framework if there weren't points of connection. Actions must have some way of getting at the internal state of the application at runtime. However, reducing the degree to which actions are dependent (or even aware) of the framework is one of our top priorities. People don't want their classes to be locked into a single framework — and who can blame them for that?

In the most absolute basic implementation, actions would only need to implement the Action interface. But the goal that prompted me to develop the framework in the first place was the ability to trade actions between classloaders using JMX. To this end, the framework creates an instance pool in which actions are stored and managed. In addition to cloning the action instances for pooling, the framework sorts the instances by name and version. Toward this end, the CachedAction interface extends Action and Cloneable, and also introduces two methods for getting and setting the action metadata.

The repository that manages CachedAction instances is available through JMX and universally accessible from any classloader in the VM. This allows us to trade actions between WAR files and provide tools for managing the actions (fulfilling the manual override requirement from above), and allows application developers to revert to a previous action versions at runtime. If a new version of an action brings negative side effects to the system, the manual override should allow a developer or admin to roll the system back to an earlier, more stable version. But that sort of functionality comes at a price — action developers must abide by the CachedAction interface or be willing to write a new repository mechanism with fewer interface requirements. Another great technique would be to implement a "dynamic action" that can map to a plain-old Java object (POJO). Of course, this could have negative effects on resources if the POJOs aren't pooled. Regardless, neither solution would be difficult to incorporate into the total system.

The clients package also contains my first feint toward a protocol-neutral state subsystem. Earlier versions passed the HttpServletRequest, HttpServletResponse, and ServletContext directly into the action object through its execute() method signature. In this implementation, the incoming State parameter can be cast to an EnvironmentContext, which provides access to commonly used Internet protocols. It certainly isn't the best way to do business, but it's good enough until we can write a better system.

A better system would aggregate all of the data in the RuntimeState (which the framework uses internally, and which presently implements EnvironmentContext) in a protocol-neutral manner, re-organize it all, and make it accessible to internal and external component developers. This is where the representation subsystem will come in. In other words, the role of Representation components is to manage the interaction between developers and the application state. At that point, a more sophisticated state subsystem can be built without adding complexity for component developers.

There is another area in which the state subsystem is currently lacking. Families of actions (or workflow components, for that matter) can be coupled through their reliance upon values cached in the RuntimeState. This is called "state-coupling," and it can be broken during development if one of the objects alters the key by which they refer to that mutually required value. When this occurs, it can result in runtime errors that are very difficult to track down and debug at compile time. Externalizing the application state and making it manageable could help with this, as could aggressive system-level unit testing. I'm laying out plans for a project to provide this capability.

The client package is completed by the CachedActionSupport object, which provides a bare-bones implementation of CachedAction, and handles the downcasting of State to EnvironmentContext so you don't have to. But you're certainly not required to use CachedActionSupport. You can easily provide your own implementation of CachedAction if you like, or write a new action repository mechanism (as mentioned above) if you don't.

Framework Architecture

The structure of the framework itself is very straightforward, as illustrated by Figure 3.

Figure 3. Arrangement of the shocks.framework package

The main entry point into the framework is the ShocksHttpServlet. Any sort of handler (like an SMTP handler or a SOAP handler) could also provide entry into the workflow system — this is just the default implementation. When initialized by the servlet container, the ShocksHttpServlet triggers a loading mechanism that parses the shocks-workflow.xml file in the WAR file's WEB-INF directory. All of the XML metadata from that file is loaded into a central repository, which binds it into MDBeans and manages the details of instance pooling. The loading mechanism then uses these MDBean objects to load Action and Filter objects, and combinations of the three to build Sequence objects.

At runtime, the ShocksHttpServlet pulls the command URI from the incoming request object, then packages it (along with the relevant protocol objects) into a new RuntimeState object. That RuntimeState object gets passed on to the WorkflowController, which figures out what to do with it.

The WorkflowController's job is made especially simple because of the way we pre-build and cache metadata and workflow sequences. It passes the command URI to a MetadataRepository and gets a MDBean containing all of the information about that command. The WorkflowController then passes the MDBean as a key to a SequenceRepository, and gets an appropriate Sequence object, which is an ordered set of workflow components. The WorkflowController can then forward to the Sequence, which is a series of filters and a target action. Eventually, the flow returns to the WorkflowController. When this happens, it puts the Sequence back into the repository, examines the RuntimeState, and decides what to do next.

Watching the framework hit the action and come back the first time was like watching a skateboarding legend like Tony Alva or Jay Adams hit the lip of a swimming pool. There and back is just the price of entry. The tricks you pull off while you're getting there and back, and your style — the way you pull those tricks off — that's what really matters.

As noted before, none of these tricks works quite the way I'd like, and a lot of the decision making could be farmed out to a rules engine like Drools. But this proves that the concept and the process are sound, that division of the entire system into a family of explicit roles can be accomplished.

Framework Checklist

Now, after working through this design, how well does it address the five challenges set forth by Dr. Reenskaug? I'm going to grade our progress in this, but not on a curve compared to other projects.

1. Manual override: B
   This is accomplished by making the workflow and action subsystems completely manageable through JMX. At present, it should be possible to override and direct the behavior of an application at runtime. When we plug the Drools engine in and write management hooks for that, we'll have even more fine-tuned control over the application workflow. The addition of state- and representation-management capabilities, as well as some really good UI tools, would drive the grade straight into "A" territory. Look for advances in those areas and a web-based application management suite sometime in the next year.

2. Modular subsystems: B
   The total system has successfully been divided into modular subsystems, but the decoupling could be improved upon. A better representation subsystem could lead to much more developer-friendly environment, and we'll be working in this direction.

3. Behavioral transparency: B-
   This is accomplished through the use of interfaces and access to a common RuntimeState. There is a certain degree of state-coupling, because objects are tied to one another through the values stored therein. If one developer switches the name of an attribute in one object, it could wreak havoc across the entire system. Aggressive unit testing and an external state-management subsystem could simultaneously reduce complexity and raise transparency here (I got the idea from the Apache Cocoon list — it is by no means a novel concept).

4. Pluggable modules: B+
   New features and functionality can be plugged into the framework as WAR files packed with useful Action objects. Entire feature platforms that provide common functionality like form validation, type conversion, and JSF-style command processing can all be added to extend the core framework's functionality. The addition of the Drools workflow system will allow new Petri nets, or "workflow modules" to be plugged into the framework, as well. All of the current instance-pooling mechanisms are replaceable, but this process could be made even easier so that it requires no manipulation of framework class files at all.

5. Continuous growth: B
   By reducing (and aiming to eliminate) the need for third-party subsystem developers to directly manipulate the framework source code, the framework allows the total system to grow with less friction. The core developers of the framework (no matter how good their intentions) represent a bottleneck that prohibits the introduction of functionality to the total system. We are actively working to reduce the need for framework developers to integrate changes into the total system. Eventually, this framework plus its third-party "feature platform" extensions plus your business logic extensions (all three comprise the total system) will be capable of continuous growth with a bare minimum of friction. The ability to break the application down into explicit subsystems is absolutely integral for this to even be possible.

Conclusion

Shocks is clearly not perfect, but that's alright. I just assumed I'd do everything wrong and that eventually every subsystem would need to be torn out and replaced with a better piece — most likely by a better developer. The challenge was to make that process easier, and also to explore new possibilities and expose myself to unexpected consequences. I think I've done that. There are no hard and fast rules in this game, and any absolute principles that might apply have not been identified. You can write something to cover the 80% use case, but that other 20% must be left to artists. If you give them hooks for incorporating their ideas into the whole, hopefully their jobs will be easier and everyone benefits.

The best we can hope for as designers is that every day we render obsolete the technology of the day before, or that we strike upon some guiding principles to help us along our way. With one another's help, I hope we continue doing just that. The tools we use in five years will not resemble the tools we use today. Hopefully, we can plug into each other's talents with better frameworks, and hopefully, this architectural style can help us more easily see how all the pieces fit together.

If not, that's alright too. We can always refactor and hone the style. Right?

N. Alex Rupp studies theoretical geometry and classical literature and is conducting ongoing research in declarative programming.