Skip to main content

Dynamic Load Balancing in GlassFish Application Server

{cs.r.title}







GlassFish is a fully
Java EE 5-compliant application server with enterprise-ready
features available under two OSI-approved licenses. Among many
other enterprise-level features, GlassFish provides a very good
self-management functionality extendable using the "http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/">
Java Management eXtension (JMX)
standard.

The GlassFish application server provides good facilities for
cluster management and load balancing. Still, sometimes we need to
have more fine-grained control over how our cluster nodes are
loaded with requests. One such condition can happen in a shared
environment when we are using the processing power to host several
applications, ranging from databases to batch job processing to
application servers serving requests that come from different
customers. Such conditions can lead us to change the load routed
toward each GlassFish instance during different time slices. We can
change the routed load manually, but having the ability to change
it automatically based on defined rules is something very
desirable.

This article will explain how we can use GlassFish self-management facilities, JMX, and the "https://glassfish.dev.java.net/javaee5/amx/">Application Server
Management eXtension (AMX)
APIs to change the load balancer
configuration dynamically. Sample code to illustrate these
processes is provided throughout the article and is also available
in the Resources section.

Aside from reviewing some background information, we will need
to perform three steps in order to complete the example to change
load routed toward each application server instance:

  1. Developing an MBean, which can change the load balancer
    configuration.
  2. Creating the management rule and deploying it.
  3. Testing the system with new features.

Fundamentals of JMX and AMX APIs

JMX refers to several JSRs that allow software vendors and
developers to have a common technology for exposing management and
monitoring capabilities of their Java-based applications.

Usually when we say JMX, we are referring to a set of at least two
JSRs: Java Management Extensions ( "http://jcp.org/en/jsr/detail?id=3">JSR 3) and the JMX Remote API
(JSR 160).
Although there are other specifications that complement JMX, like
the J2EE Management Specification ( "http://jcp.org/en/jsr/detail?id=77">JSR 77), the most utilized
JSRs are JSR 3 and JSR 160.

JMX has three important components:

  • MBean: MBeans are probe-level objects of the JMX specification.
    There are four kinds of MBeans for different use cases; the most
    popular MBean type is Dynamic.
  • MBeanServer: This is the access point for MBeans.
    All MBeans should be registered with an MBeanServer.
    It is the access point that allows a variety of operations such as
    invoking MBeans' operations, changing attributes of MBeans, event
    listener registration, and so on.
  • Connectors and Adapters: These allow different types of
    management consoles to create a connection to an
    MBeanServer from outside of the host JVM. Popular
    connectors are RMI and the JMX Messaging Protocol (JMXMP; defined by
    JSR 160). Popular adapters are the SNMP adapter defined in Java
    Platform Profiling Architecture ( "http://jcp.org/en/jsr/detail?id=163">JSR 163), and the web
    (HTML/HTTP) adapter.

Using JMX, we guarantee that any other software that knows JMX
can interact with our MBeans in order to monitor specific
attributes or change some others that are exposed by our MBeans, or
to listen for different types of JMX events published by our
MBeans.

As mentioned, accessing MBeans is only possible using
MBeanServer, and it is not a convenient way for many
users, as it will involve them with JMX, which not an easy API to
deal with. Calling an MBean operation will go through reflection,
so we will need to know which MBean we want to use, the method
name, the method parameters and signature, etc. We can find the
required MBean by using its ObjectName, which is the
name that we register our MBeans with. An ObjectName
core is a string that follows an schema to make a
queryable representation of registered MBeans. The
MBeanServer provides methods to query the registry
based on the tree-like representation formed by the
ObjectName instances.

AMX is present to interact with GlassFish application server
MBeans, free of the difficulties discussed above. AMX is a set of
APIs that expose management MBeans as client-side dynamic proxies
that implement AMX interfaces. These AMX interfaces are mostly
included in com.sun.appserv.management.config and some
of its sub-packages. We can use AMX APIs on the client side in
order to manage almost all JSR 77-approved Java EE objects like
servers, applications, resources, etc., without knowing anything
about JMX. However, all of these EE objects are also manageable through
plain JMX.

Using AMX we can:

  1. Change application server configurations--create resources,
    delete resources, enable or disable, etc.
  2. Manage servers, node agents, clusters, etc.
  3. Receive notifications for almost any event happening in the
    application server and react accordingly.
  4. Monitor the state of many objects that are hosted inside the
    application server. These include EJBs, web applications,
    enterprise applications, connection pools, etc.

When we install GlassFish in the cluster or enterprise profile,
we create a domain of GlassFish application servers that
initially has only one member. This first member is the Domain
Administration Server (DAS) and it is a single point of management
and administration for every cluster or instance that we may add to
this domain. All related configuration of every domain object (such as
instances, clusters, connection pools, etc.) is stored in DAS and
can be changed directly using DAS. AMX allows us to connect to DAS
and perform every required operation on application server MBeans
in order to change the configuration of mentioned domain objects. There
are some helper classes included in
com.sun.appserv.management.helper that further ease
working with AMX APIs. The following example shows how we can
change the weight of a clustered server instance using these APIs.
Weight is a number that indicates the percent of requests
that the load balancer sends to the instance.

[prettify]
AppserverConnectionSource ASConnection =
    Connect.connectNoTLS("127.0.0.1", 8686, "admin", "adminadmin"); 
DomainRoot dRoot = ASConnection.getDomainRoot();
Map<String, ClusterConfig> clusters = 
    dRoot.getContaineeMap(XTypes.CLUSTER_CONFIG);
ClusterConfig clusterConf= clusters.get("Cluster-1");
Map <String, ClusteredServerConfig< servers =
    clusterConf.getClusteredServerConfigMap();
ClusteredServerConfig instance1 = servers.get("instance-01");
instance1.setLBWeight("25");
[/prettify]

As you can see, we simply get the clustered server instance and
change its configuration--without knowing anything about JMX.
This is what the AMX APIs provide us with. In the AMX design, all
objects are branches of the domain root, which seamlessly maps to
DAS. In the next section, we will discuss what management rules are
and how we can use them to implement our defined requirements.

Basics of GlassFish Management Rules

A management rule is a set of:

  • Event: An event uses the JMX notification mechanism to trigger
    actions. Events can range from an MBean attribute change to
    specific log messages.
  • Action: Actions are associated with events and are triggered
    when related events happen. Actions can be MBeans that implement
    the NotificationListener interface.

Important types of events are as follows:

  • Monitor events: These type of events trigger an action based on
    an MBean attribute change.
  • Notification events: MBeans can implement
    NotificationBroadcaster in order to send notifications
    to all listeners that registered their interest on its event
    notifications.
  • System events: This is a set of predefined events that come from the
    internal infrastructure of GlassFish application server. These
    events include: lifecycle, log, timer, trace, and cluster
    events.

The focus of this article is on cluster events, which are
members of the system events family. There are three cluster events, fired when a cluster is started or stopped, or has
failed. Our sample code listens for the cluster start event and
upon receiving this event, it schedules and start a timer. This
timer calls methods in order to perform the logic for deciding
whether it is required to update instance weights or not. This
decision will be made based on the content of a configuration file
with the following elements:

  1. Name of the cluster whose instance weights we will update.
  2. Acceptable delay between entering a time slice and updating
    instance weights. This value is the interval on which the
    previously mentioned timer will execute the task that performs our
    logic to decide on updating instance weights.
  3. Some time slices, where each time slice is a set of start
    and end times and instance names with their weights during that
    slice of time.

A sample configuration file that can be used by the article
sample code is shown in the following snippet.


       
               
               
       

       
               
               

This configuration file indicates that there is a cluster named
Cluster-1 and it is required to reconfigure its instance
weights during two time slices. Acceptable delay between entering
the time slice and updating the instance weights is 1000 ms; i.e.,
one second.

We saw that each management rule has an event that will trigger
an action. An action is a MBean that implements
the NotificationListener interface, which has one to-be-implemented method named handleNotification. When
the management rule triggers the action it will call this
method.

By looking at the JMX tutorial and references we can find that
the most basic type of MBeans are standard MBeans. These MBeans are
composed of an interface that must follow a specific naming
schema and the class that implements that interface. The interface
must end with word MBean. In this article, the action
MBean is a custom MBean that, in addition to its own MBean interface
named ClusterWeightManagerMBean, implements the
NotificationListener interface in order to receive
the cluster's start notification. So the final action
MBean class implements the following two interfaces:

  • ClusterWeightManagerMBean: This interface includes
    public methods that our MBean makes available for a JMX agent or
    tools like JConsole.
  • NotificationListener: Implementing this interface
    allows our MBean to listen for a cluster start event after we
    register its interest for that kind of event by defining a
    management rule.

The action MBean discussed above needs a
configuration file to read time slices from and use them during its
operations, and the configuration file address is a property of
MBeans that we need to persist for future use. Here, GlassFish
comes to help us by providing some facilities for storing and
initializing the JavaBean-patterned property of MBeans. GlassFish
stores the property value in the domain.xml file and
initializes it each time an object of the MBean gets
constructed.

In order to let GlassFish read and write the property value, we
should provide setter and getter methods accessible from a management
agent; as we already saw, any method that needs to be accessible by
a management agent should be included in the MBean interface. The
following code snippet shows ClusterWeightManagerMBean
with a pair of setter and getter methods to read and write the
configurationFilePath attribute.

[prettify]
public interface ClusterWeightManagerMBean {
    public String getConfigurationFilePath();
    public void setConfigurationFilePath(String configurationFilePath);
}
[/prettify]

The ClusterWeightManager class implements this
interface to form an acceptable MBean. It should also implement the
NotificationListener interface that is required to
receive notifications from a management rule. Before getting into
details of ClusterWeightManager's implementation, let's
take a look at a helper class named TimeSlice, which
holds each time slice's information. The following listing shows
the TimeSlice class.

[prettify]public class TimeSlice {
        private Map<String, String> instanceWeights;
        private Date startDate;
        private Date endDate;
        public TimeSlice(Map<String, String> instanceWeights, Date startDate, Date endDate) {
                this.instanceWeights = instanceWeights;
                this.startDate = startDate;
                this.endDate = endDate;
        }
 
        public Map<String, String> getInstanceWeights() {
                return instanceWeights;
        }
        public void setInstanceWeights(Map<String, String> instanceWeights) {
                this.instanceWeights = instanceWeights;
        }

        public int compareTime(Date curDate) {
        int result = 0;
                if (curDate.compareTo(endDate) == 1) {
                result = 1;
                }
        if (curDate.compareTo(startDate) == -1) {
                result = -1;
        }
        return result;
        }
}
[/prettify]

Finally we need to implement the core functionalities that
reside in the ClusterWeightManager, which is the MBean
implementation class. The full code would be too long to include
here; see the sample code ZIP in the "#resources">Resources section. For now, let's focus on the
important parts.

Before investigating any other method, we should discuss
applyInstanceWeights, which applies the weight of each
instance based on the timeSlice configuration that we
pass to it. Here's the implementation of the
applyInstanceWeights method.

[prettify]
private void applyInstanceWeights(TimeSlice currectTimeslice) {
        String instanceNamePrefix = "amx:j2eeType=X-ClusteredServerConfig,name=";
        Map<String, String> instanceWeights = currectTimeslice.getInstanceWeights();
        try {
            Set instanceNames = instanceWeights.keySet();
            Iterator It = instanceNames.iterator();
            while (It.hasNext()) {
                String instanceName = (String) (It.next());
                String instanceWeight = instanceWeights.get(instanceName);
                ObjectName name = new ObjectName(instanceNamePrefix + instanceName);
                Attribute attrib = new Attribute("LBWeight", new Integer(instanceWeight));
                mBeanServer.setAttribute(name, attrib);
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        } 
    }
[/prettify]

By looking carefully at the above code snippet, you can see that
we pass an object containing the current time slice to this method
and it will use its instanceWeights property, which is
a map of instance names and instance weights, to update instance
weights. As you may have noticed, the
instanceNamePrefix value is not complete, and in the
next few steps we append the instance name to make it a complete
ObjectName in order to let the
MBeanServer locate its corresponding object (a
clustered instance). Then we use this ObjectName and
MBeanServer to change the value of the
LBWeight attribute of the located instance.

One other important method is handleNotification,
which is inherited from NotificationListener. It
starts the instance weight update process when it receives the
cluster start notification.

[prettify]
        public void handleNotification(Notification notification, Object handback) {
            initialize();
            this.startManager();
       }
    
[/prettify]

After receiving the notification, it calls the
initialize method, which parses the
XML file and creates an ArrayList of all time slices
included in the configuration file. The startManager
method schedules and starts the timer that we talked about
earlier.

You may ask, "What if our cluster is already running? Should we
restart it after we deploy the management rule?" In order to give an
answer to these questions, let's take a look at what JMX provides
for a similar situation and then to the solution that we used.
Usually when we need to perform some tasks after registering an
MBean, we implement the MBeanRegistration interface
and use its postRegister method to perform the
required logic. But in our case, we cannot do this. Why? Because we
have an XML file that contains the instance weight configuration, and
we put its address into one of our MBean variables;
postRegister will execute before setters, so we don't
have the configuration file address during the post-registration
phase to read the configuration. What we can do is use the
setConfigurationFilePath method to check the cluster
state and perform required actions if cluster is already
running.

In the setConfigurationFilePath method we simply use
the cluster name to create the complete ObjectName and
then check the state attribute of the corresponding
object using mBeanServer: 1 means that
the cluster is running, -1 indicates that the cluster
failed, and 0 means the cluster is stopped.

[prettify]
  public void setConfigurationFilePath(String configurationFilePath) {
        this.configurationFilePath = configurationFilePath;
        String clusterPrefix = "amx:j2eeType=X-J2EECluster,name=";
        initialize();
        try {
            ObjectName name = new ObjectName(clusterPrefix + clusterName);
            Integer state = (Integer) mBeanServer.getAttribute(name, "state");
            if (state == 1) {
                isTimerRunning = true;
                this.startManager();
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }
[/prettify]

Now, in the handleNotification method we add code to check an object-level property to see whether the
timer is already running before we start another instance of the
timer.

Creating the Management Rule and Deploying It

Assuming you've unzipped the sample code and compiled
JAR file, the class files package and the sample configuration file
are available for this step. Copy the the entire package hierarchy
(which is inside a folder named classes in the attached
sample code) into domain_dir/applications/mbeans/ in order
to make the classes available to your domain class finder. Start
the application server and use the following asadmin
command to deploy the MBean into the DAS instance, making sure that
you change the configuration file path according to where you
copied the configuration file.

[prettify]
create-mbean --host localhost --port 4848 --user admin --name ClusterInstanceWeightsManager --attributes ConfigurationFilePath=c\:/config.xml samples.glassfish.management.clustermanager.ClusterWeightManager
[/prettify]

Deploying the ClusterWeightManager MBean is equal
to including some configuration elements in the domain.xml file
and registering the ClusterWeightManager MBean with
the appropriate MBeanServer. During each startup, the
application server will register all MBeans included in
domain.xml with the MBeanServer. Now we have
our ClusterWeightManager MBean registered with
the management agent, and when it receives notifications it can start
the logic and update instance weights if required. But before it
can receive any notifications, it should announce its interest in receiving the cluster start notification. Creating a cluster
management rule is how we register our MBean's interest for one of
the cluster events.

We need to define the management rule, which will
automatically send the cluster start notification to our MBean when
the cluster starts. This management rule is the registration of our
MBean as a listener for cluster events. One additional task of the
management rule is filtering events, so our class will only receive
an event type if we define our interest in it. The following
asadmin command will create the required management
rule.

[prettify]
create-management-rule --eventtype cluster --host localhost --port 4848  --user admin --ruleenabled=true  --action ClusterInstanceWeightsManager --event loglevel INFO --recordevent=true  --eventproperties name=start ClusterManagementRule
[/prettify]

We can query our registered MBeans and management rules using
following asadmin commands.

[prettify]
>list-management-rules --user admin
>list-mbeans --user admin 
[/prettify]

Testing the System

In order to make this configuration automatically apply to your
load balancer, you need to do some configuration using the
GlassFish application server administration console. So locate your
load balancer in the navigation tree and make sure to check the
Automatically Apply Changes checkbox in the load balancer
configuration page, under the General tab. To test the system,
add two time slices close to each other and let the system run for
enough time to cross from one one time slice to the other, and then
open the GlassFish administration console. From the navigation tree
locate your cluster, check the Instances tab in the cluster
configuration page. You should notice that the instance weights are
changed accordingly.

Conclusion

This article shows an small sample of using GlassFish's self-management features to perform some administration tasks that can
be difficult and time-consuming to do manually. By looking at the
sample code and concepts introduced in the article, you should be
able to create your own MBeans and management rules to perform your
daily administration tasks. On the other hand, AMX can be a good
door for application server internal configuration and states. In the Resources section you can find many
good references for going deeper into each concept or API that
was introduced or used in this article.

Resources


width="1" height="1" border="0" alt=" " />
Related Topics >> Programming   |