Skip to main content

App-Managed JDBC DataSources with commons-dbcp

November 17, 2005

{cs.r.title}









Contents
About the Sample App
Pooling Theory
Object Pooling with commons-pool
commons-dbcp at a High Level
Your Very Own DataSource:
Coding the commons-dbcp API
Binding It to JNDI
But Is It for Me?
Conclusion
Resources

In enterprise Java applications, it's considered poor form to
instantiate database connections yourself. The enlightened path
involves a database connection pool, in the form of a JDBC
DataSource, that is provided by your container
(Tomcat, Jetty, WebLogic). The container is responsible for
maintaining a number of connections to the database service, and
all your code has to do is look up the DataSource via
JNDI to get a connection. Simple beauty.

Sometimes, though, you want the efficiency and centralization of
a pool but the control of managing it yourself. Consider the
following scenarios:

  • Your app runs in a managed environment and you
    don't have access to the container's admin console. This is not
    uncommon in the corporate enterprise, or with some web hosts.
  • You want a container-neutral web app that can
    be deployed by third parties with minimal intervention on your
    part. Do you really want to provide DataSource how-to
    guides for all of your customers' containers?
  • Your app runs in a special environment, such
    as a demo CD-ROM, that requires it to be even more self-contained
    than a standard web app.

Whatever the case, there are ways to have your cake and eat it,
too. In this article, I'll share a technique that uses Apache's
commons-dbcp
(and, by way of dependency, "http://jakarta.apache.org/commons/pool/">commons-pool) to move
the DataSource into your web application. Managing a
DataSource yourself requires a some knowledge of
object pooling and JNDI, so I'll explain those along the way.

About the Sample App

I have provided a sample web application to help explain the
technique - see the resources section for
a link to the sample code. Two servlet plus JSP "http://www.martinfowler.com/eaaCatalog/pageController.html">page
controller
s are the front ends of simple data input and
retrieval operations. The servlets use a central object registry
(class ObjectRegistry) to look up a "http://java.sun.com/blueprints/corej2eepatterns/Patterns/DataAccessObject.html">
Data Access Object
(DAO, SimpleDAO). In turn, the
DAO is used to move data into and out of the database. The DAO
fetches database connections from a DataSource that's
bound to JNDI.

Since the DataSource is not provided by the
container, it's up to the web app to initialize it before any code
tries to use a database connection. A "http://java.sun.com/j2ee/1.4/docs/api/javax/servlet/ServletContextListener.html">
ServletContextListener
is a suitable candidate for
the task. Implementations of this interface, introduced in the 2.3
servlet spec, are registered in the web app's web.xml
deployment descriptor. The container calls each registered
listener's contextInitialized() method when its parent
web app is deployed, and calls contextDestroyed() when
the web app is removed from service.

Much of this article focuses on a single such listener,
SetupDataSourceContextListener. This class uses
commons-dbcp and commons-pool to create a DataSource
and bind it to JNDI.

The sample web app was tested under "http://jakarta.apache.org/tomcat/">Tomcat 5.5 and "http://jetty.mortbay.org/">Jetty 5.1. It uses an embedded,
in-memory HSQLDB database to make
it completely self-contained and ready for testing in your
environment.

Pooling Theory

Most DataSources are backed by pools of database
connections, so it's tough to describe the former without a brief
look at the latter. Pooling is a way to manage resources
that are expensive to create and in constant demand, but used only for
short periods at a time. Most Java developers are familiar with
pools of database connections, or even pools of threads. In
languages that don't provide garbage collection, such as C and C++,
it's not uncommon to use pools of raw memory.

The premise behind pooling is rather simple: instead of directly
instantiating a resource--calling new--code
borrows a resource from some central repository.
Client code briefly uses the resource and returns it to the
repository. Object pooling is not unlike using a rental car: the
client code doesn't know the resource's origin, nor what happens
when it returns it, nor whether it will get the same object each
time it asks to borrow one.

The repository is responsible for the resource objects'
lifecycle. It instantiates some number of resource objects and
makes them available for borrowing, and later cleans them up. The
pool is exhausted when all resources are loaned out. What
happens when code requests a resource from an exhausted pool is a
pool implementation detail: the pool's borrowing operation can
block (not return) until a resource is ready, return
null, or throw an exception.

Object Pooling with commons-pool

commons-dbcp uses commons-pool, Apache's generic pooling
framework, as its pool back end. ObjectPool is the core
of commons-pool. This interface represents a class that manages an
underlying object collection that comprises the pool. It has
methods to borrow objects (borrowObject()) and return them
(returnObject()), and to query the pool state
(getNumActive(), getNumIdle()):


public interface ObjectPool {

   Object borrowObject() ;
   void returnObject(Object obj) ;

   int getNumActive() ;
   int getNumIdle() ;

   void invalidateObject(Object obj) ;

   void clear() ;
   void close() ;

   void addObject() ;
   void setFactory(PoolableObjectFactory factory) ;
}

Two other methods, addObject() and
setFactory(), merit further discussion. An
ObjectPool has only indirect knowledge of the objects
it manages. The setFactory() method specifies the
factory that the ObjectPool will use to create those
objects. In turn, addObject() tells the
ObjectPool to call on its factory to create an object
and add it to the underlying pool.

The sample code uses the GenericObjectPool
implementation of ObjectPool (other implementations,
such as StackObjectPool and
SoftReferenceObjectPool, offer features above and
beyond basic pooling). This class offers several methods beyond its
base interface. Notably, setWhenExhaustedAction()
specifies what borrowObject() does if the pool is
exhausted:


GenericObjectPool pool = new GenericObjectPool(null);
pool.setWhenExhaustedAction(
   GenericObjectPool.WHEN_EXHAUSTED_BLOCK
) ;

Here, it blocks until another resource is available (a resource is
returned to the pool, or some other code calls
addObject() to add a new object to the pool). There
are also options to grow the pool
(WHEN_EXHAUSTED_GROW) or throw an exception
(WHEN_EXHAUSTED_FAIL). Which of these is appropriate
depends on the application.

The pool is initially empty. It's up to client code to call
addObject() a few times to populate the pool with
objects from the backing factory. Here, the pool created in
SetupDataSourceContextListener will have ten
objects:


for (int ix = 0 ; ix < 10 ; ++ix ) {
   pool.addObject() ;
}

It's good form to clean up whatever you create. The container
calls SetupDataSourceContextListener's
contextDestroyed() when the web app shuts down. In
turn, this method calls the pool's close() method to
free its resources:


if( null != _pool ){
   _pool.close() ;
}

commons-pool is only half of the story. Mix it with commons-dbcp
to create a connection pool.

commons-dbcp at a High Level

commons-dbcp is Apache's database connection pooling framework.
As you'll soon see, it works with pretty much any JDBC-compliant
database driver. If it's supported by the classic
DriverManager, you can plug it into commons-dbcp.

commons-dbcp uses a multilayered concept. Client code only sees
the DataSource object--here, the
PoolingDataSource--on which it calls
getConnection() to fetch a database connection. A lot
more happens behind the scenes.

The PoolingDataSource's
getConnection() calls its ObjectPool's
borrowObject() to fetch a JDBC
Connection. This returns a plain Object,
so getConnection() casts the result:


...
public Connection getConnection(){
   ...
   Connection result = (Connection) _pool.borrowObject() ;
   return( result ) ;
}

This is just pseudocode, of course.

In turn, the ObjectPool returns an object created
by is backing factory, here a
PoolableConnectionFactory. Remember, the
ObjectPool's addObject() called this
factory to populate the pool with objects.

To clarify, PoolableConnectionFactory is a factory
of PoolableConnection objects. These objects implement
the JDBC Connection interface, so they look and act
just like any other database connection. They even wrap other
Connection objects. The twist is that
PoolableConnection intercepts calls to the
close() method: instead of terminating the network
connection to the database service, it calls the parent pool's
returnObject() to return the wrapped
Connection to the pool.


public class PoolableConnection {

   ...
   
   // PoolableConnection delegates most methods
   // (such as createStatement()) to this object:
   private Connection _realConnection ;
   
   // the pool from whence this PoolableConnection
   // object came:
   private ObjectPool _parentPool ;
   
   ...
   
   // don't really close it; put it back in the
   // pool so someone else can use it
   void close(){
        _parentPool.returnObject( this ) ;
   }

}

This, by the way, is one reason connection pooling is considered
more efficient than direct connection use: the connections in the
pool are always open, so you don't waste time and resources logging
in to the database service for each query.

Digging deeper, PoolableConnectionFactory doesn't
create those wrapped JDBC Connections itself. It
proxies yet another factory, in this case a
DriverManagerConnectionFactory. In turn,
DriverManagerConnectionFactory wraps calls to the
ever-familiar DriverManager.getConnection() to create
a database connection.

Your Very Own DataSource: Coding the commons-dbcp API

The commons-dbcp API makes coding this nested "http://en.wikipedia.org/wiki/Matryoshka_doll">matryoshka
doll
easier than describing it. First, create the underlying
factory to furnish Connection objects:


ConnectionFactory factory = new DriverManagerConnectionFactory(
   ... JDBC URL ... ,
   ... login ... ,
   ... password ...
) ;

Next, back the pool with a
PoolableConnectionFactory:


PoolableConnectionFactory pcf =
        new PoolableConnectionFactory(

   factory , // ConnectionFactory
   _pool ,   // ObjectPool
   null ,    // KeyedObjectPoolFactory
   null ,    // String (validation query)
   false ,   // boolean (default to read-only?)
   true      // boolean (default to auto-commit?)
);

You've already seen the ConnectionFactory and
ObjectPool. The third parameter, the
KeyedObjectPoolFactory, is used to pool prepared
statements. Here it is null, which means the
connection factory will not pool prepared statements. The fourth
parameter is a simple query that commons-dbcp can call to confirm
that the connection is still valid. It uses this to avoid loaning out
problem connections. The last two booleans set whether the
connections are read-only and default to auto-commit, respectively.

Don't be fooled by the call to create pcf. It looks
like a no-op statement, but the variable pcf assigns
itself as the pool's factory. That means the
ObjectPool's addObject() calls
pcf.makeObject().

Finally, set the pool as the back-end Connection
provider for the DataSource.


PoolingDataSource result = new PoolingDataSource();
result.setPool( _pool ) ;

This completes the circuit: PoolingDataSource is a
DataSource like any other. Calling its
getConnection() method will call
_pool.borrowObject(), as described above, to return a
JDBC Connection.

For all of this to work, however, you're missing one last piece:
the JAR file(s) with your database drivers. Since these are no
longer managed by the container, you must include these JARs in
your web app's WEB-INF/lib directory along with your other
dependencies.

Binding It to JNDI

Code must be able to find the DataSource in order
to use it. One option is to make your shiny new
DataSource available via JNDI, where it will look
exactly like one that's provided by the container (at least, to
client code). Switching from a container-managed to an app-managed
DataSource would require changes to just the JNDI
lookup string used by client code. And if you've followed best
practices, this should be only one place!

JNDI is a directory service, often used in Java as an object
store: one piece of code instantiates an object and binds
(stores) it under some key; another fetches the object based on
that key. Services such as JNDI promote loose coupling because the
object creator and user are not directly related; they only have to
know of JNDI and the lookup key.

JNDI works on a tree model, not unlike a filesystem. Objects are
bound to (located at) contexts (paths). The
elements of the context--that is, the branches along the tree--are called subcontexts. One string representation of a
context looks like a Unix filename, complete with slashes
(/) to separate the subcontext elements.

SetupDataSourceContextListener's
bindObject() method takes an object to bind and a
string that represents the context. It first breaks down the string
and builds the subcontexts as needed:


Context currentContext = new InitialContext() ;
final String name = currentContext.composeName(
   fullPath ,
   currentContext.getNameInNamespace()
) ;

// get the subcontext pieces
final String[] components = name.split( "/" ) ;

for( /* elements of "components" */ ){
   
   final String nextPath = /* current element */

   try{

      // lookup the subcontext
      currentContext = (Context)
         currentContext.lookup( nextPath ) ;

   }catch( final NameNotFoundException ignored ){

      // not found; create subcontext
      currentContext = currentContext
         .createSubcontext( nextPath ) ;

   }

}

After that, binding the object to the JNDI tree is a simple call to

subContext.bind( name , objectToBind ) ;

Not all containers provide writable JNDI trees. Others forbid
binding to certain context paths, such as the standard
java:comp/env. You'll know about any problems right
away: the ContextListener will throw an exception if
it is unable to bind. If yours fits into this category, you'll have
to make the DataSource available some other way,
either via dependency injection (a la Spring) or a custom app-wide
object registry (similar to the sample code's
ObjectRegistry class).

Frankly, if your app already has its own object registry, JNDI
is redundant and otherwise of limited use. JNDI is more suitable
for loose coupling on a larger scale, such as between
applications.

But Is It for Me?

There are few, if any, one-size-fits-all solutions. Consider the
following when choosing between app-managed and container-managed
DataSources.

First of all, commons-dbcp and commons-pool have very wide
usage. For example, Tomcat's container-managed database
connectivity is backed by code similar to that used in the sample
app. Tomcat just wraps it in a neat XML config file. There are
occasional complaints that the format is too complex and
error-prone, but that's another story.

If you want to build a container-neutral app, this technique
puts you one step closer to that goal. You can combine a
commons-dbcp DataSource with an in-memory database
such as HSQLDB to deliver demo apps that run self-contained, out of
the box, even on read-only media.

Perhaps you run your web apps via a third-party hosting service.
App-managed database pooling gives you runtime control of the pool
size and configuration--for example, you can drop and recreate
pools while the app is still live. In a shared environment, hosting
providers (understandably) cannot always grant customers access to
the container's admin control panel.

Then again, container-provided DataSources exist
for a reason. The loose coupling of web apps and database
connectivity centralizes administration. Some containers offer a
GUI control panel, from which administrators can keep an eye on
managed services. App-managed database pools are invisible to these
tools.

Furthermore, you can't use app-managed database pooling to back
container-managed services, such as Tomcat's security realms. This
is because the DataSource exists in the web app's
classloader, making it invisible to the container's classloader.
That said, there's nothing to stop you from using both: app-managed
connection pools for your business code, and container-managed
pools for container-managed security.

Creating the DataSource within your app requires
more effort on your part. You have to manage and clean up any
associated resources (typically, via a
ServletContextListener). You also have to rebuild and
deploy your app if you change database drivers, since the JARs are
packaged inside of WEB-INF/lib. Finally, it's up to you to
create your own management interface from which to control the pool
at runtime. Either that, or you must rebuild and redeploy the
web app whenever you want to change the number of connections or
reset the pool.

Conclusion

Moving database connectivity from the container into your web app
addresses some compatibility problems and addresses niche issues,
such as creating demo applications. It's certainly not appropriate
for all situations; but you now have the theory and practice to
apply it when the need arises.

I'd be happy to hear your success stories of trying this
technique with other containers. Note that the sample app requires
a servlet spec 2.4 container, such as Tomcat 5, Jetty 5, or
WebLogic 9. It should work in a 2.3 spec container with a few
tweaks to the JSPs.

Resources

width="1" height="1" border="0" alt=" " />
Ethan McCallum is a freelance technology consultant.
Related Topics >> Databases   |