| |
Dynamic Load Balancing in GlassFish Application ServerGlassFish is a fully Java EE 5-compliant application server with enterprise-ready features available under two OSI-approved licenses. Among many other enterprise-level features, GlassFish provides a very good self-management functionality extendable using the Java Management eXtension (JMX) standard.
The GlassFish application server provides good facilities for cluster management and load balancing. Still, sometimes we need to have more fine-grained control over how our cluster nodes are loaded with requests. One such condition can happen in a shared environment when we are using the processing power to host several applications, ranging from databases to batch job processing to application servers serving requests that come from different customers. Such conditions can lead us to change the load routed toward each GlassFish instance during different time slices. We can change the routed load manually, but having the ability to change it automatically based on defined rules is something very desirable.
This article will explain how we can use GlassFish self-management facilities, JMX, and the Application Server Management eXtension (AMX) APIs to change the load balancer configuration dynamically. Sample code to illustrate these processes is provided throughout the article and is also available in the Resources section.
Aside from reviewing some background information, we will need to perform three steps in order to complete the example to change load routed toward each application server instance:
JMX refers to several JSRs that allow software vendors and developers to have a common technology for exposing management and monitoring capabilities of their Java-based applications.
Usually when we say JMX, we are referring to a set of at least two JSRs: Java Management Extensions (JSR 3) and the JMX Remote API (JSR 160). Although there are other specifications that complement JMX, like the J2EE Management Specification (JSR 77), the most utilized JSRs are JSR 3 and JSR 160.
JMX has three important components:
MBeanServer: This is the access point for MBeans.
All MBeans should be registered with an MBeanServer.
It is the access point that allows a variety of operations such as
invoking MBeans' operations, changing attributes of MBeans, event
listener registration, and so on.MBeanServer from outside of the host JVM. Popular
connectors are RMI and the JMX Messaging Protocol (JMXMP; defined by
JSR 160). Popular adapters are the SNMP adapter defined in Java
Platform Profiling Architecture (JSR 163), and the web
(HTML/HTTP) adapter.Using JMX, we guarantee that any other software that knows JMX can interact with our MBeans in order to monitor specific attributes or change some others that are exposed by our MBeans, or to listen for different types of JMX events published by our MBeans.
As mentioned, accessing MBeans is only possible using
MBeanServer, and it is not a convenient way for many
users, as it will involve them with JMX, which not an easy API to
deal with. Calling an MBean operation will go through reflection,
so we will need to know which MBean we want to use, the method
name, the method parameters and signature, etc. We can find the
required MBean by using its ObjectName, which is the
name that we register our MBeans with. An ObjectName
core is a string that follows an schema to make a
queryable representation of registered MBeans. The
MBeanServer provides methods to query the registry
based on the tree-like representation formed by the
ObjectName instances.
AMX is present to interact with GlassFish application server
MBeans, free of the difficulties discussed above. AMX is a set of
APIs that expose management MBeans as client-side dynamic proxies
that implement AMX interfaces. These AMX interfaces are mostly
included in com.sun.appserv.management.config and some
of its sub-packages. We can use AMX APIs on the client side in
order to manage almost all JSR 77-approved Java EE objects like
servers, applications, resources, etc., without knowing anything
about JMX. However, all of these EE objects are also manageable through
plain JMX.
Using AMX we can:
When we install GlassFish in the cluster or enterprise profile,
we create a domain of GlassFish application servers that
initially has only one member. This first member is the Domain
Administration Server (DAS) and it is a single point of management
and administration for every cluster or instance that we may add to
this domain. All related configuration of every domain object (such as
instances, clusters, connection pools, etc.) is stored in DAS and
can be changed directly using DAS. AMX allows us to connect to DAS
and perform every required operation on application server MBeans
in order to change the configuration of mentioned domain objects. There
are some helper classes included in
com.sun.appserv.management.helper that further ease
working with AMX APIs. The following example shows how we can
change the weight of a clustered server instance using these APIs.
Weight is a number that indicates the percent of requests
that the load balancer sends to the instance.
AppserverConnectionSource ASConnection =
Connect.connectNoTLS("127.0.0.1", 8686, "admin", "adminadmin");
DomainRoot dRoot = ASConnection.getDomainRoot();
Map<String, ClusterConfig> clusters =
dRoot.getContaineeMap(XTypes.CLUSTER_CONFIG);
ClusterConfig clusterConf= clusters.get("Cluster-1");
Map <String, ClusteredServerConfig< servers =
clusterConf.getClusteredServerConfigMap();
ClusteredServerConfig instance1 = servers.get("instance-01");
instance1.setLBWeight("25");
As you can see, we simply get the clustered server instance and change its configuration--without knowing anything about JMX. This is what the AMX APIs provide us with. In the AMX design, all objects are branches of the domain root, which seamlessly maps to DAS. In the next section, we will discuss what management rules are and how we can use them to implement our defined requirements.
A management rule is a set of:
NotificationListener interface.Important types of events are as follows:
NotificationBroadcaster in order to send notifications
to all listeners that registered their interest on its event
notifications.The focus of this article is on cluster events, which are members of the system events family. There are three cluster events, fired when a cluster is started or stopped, or has failed. Our sample code listens for the cluster start event and upon receiving this event, it schedules and start a timer. This timer calls methods in order to perform the logic for deciding whether it is required to update instance weights or not. This decision will be made based on the content of a configuration file with the following elements:
A sample configuration file that can be used by the article sample code is shown in the following snippet.
<config interval="10000" cluster-name="Cluster-1">
<slice start-time="06:40:00 AM" end-time="10:40:00 AM">
<instance name="instance-01" weight="55"/>
<instance name="instance-02" weight="55"/>
</slice>
<slice start-time="03:00:00 PM" end-time="06:00:00 PM">
<instance name="instance-01" weight="38"/>
<instance name="instance-02" weight="38"/>
</config>
This configuration file indicates that there is a cluster named Cluster-1 and it is required to reconfigure its instance weights during two time slices. Acceptable delay between entering the time slice and updating the instance weights is 1000 ms; i.e., one second.
We saw that each management rule has an event that will trigger
an action. An action is a MBean that implements
the NotificationListener interface, which has one to-be-implemented method named handleNotification. When
the management rule triggers the action it will call this
method.
By looking at the JMX tutorial and references we can find that
the most basic type of MBeans are standard MBeans. These MBeans are
composed of an interface that must follow a specific naming
schema and the class that implements that interface. The interface
must end with word MBean. In this article, the action
MBean is a custom MBean that, in addition to its own MBean interface
named ClusterWeightManagerMBean, implements the
NotificationListener interface in order to receive
the cluster's start notification. So the final action
MBean class implements the following two interfaces:
ClusterWeightManagerMBean: This interface includes
public methods that our MBean makes available for a JMX agent or
tools like JConsole.NotificationListener: Implementing this interface
allows our MBean to listen for a cluster start event after we
register its interest for that kind of event by defining a
management rule.The action MBean discussed above needs a configuration file to read time slices from and use them during its operations, and the configuration file address is a property of MBeans that we need to persist for future use. Here, GlassFish comes to help us by providing some facilities for storing and initializing the JavaBean-patterned property of MBeans. GlassFish stores the property value in the domain.xml file and initializes it each time an object of the MBean gets constructed.
In order to let GlassFish read and write the property value, we
should provide setter and getter methods accessible from a management
agent; as we already saw, any method that needs to be accessible by
a management agent should be included in the MBean interface. The
following code snippet shows ClusterWeightManagerMBean
with a pair of setter and getter methods to read and write the
configurationFilePath attribute.
public interface ClusterWeightManagerMBean {
public String getConfigurationFilePath();
public void setConfigurationFilePath(String configurationFilePath);
}
The ClusterWeightManager class implements this
interface to form an acceptable MBean. It should also implement the
NotificationListener interface that is required to
receive notifications from a management rule. Before getting into
details of ClusterWeightManager's implementation, let's
take a look at a helper class named TimeSlice, which
holds each time slice's information. The following listing shows
the TimeSlice class.
public class TimeSlice {
private Map<String, String> instanceWeights;
private Date startDate;
private Date endDate;
public TimeSlice(Map<String, String> instanceWeights, Date startDate, Date endDate) {
this.instanceWeights = instanceWeights;
this.startDate = startDate;
this.endDate = endDate;
}
public Map<String, String> getInstanceWeights() {
return instanceWeights;
}
public void setInstanceWeights(Map<String, String> instanceWeights) {
this.instanceWeights = instanceWeights;
}
public int compareTime(Date curDate) {
int result = 0;
if (curDate.compareTo(endDate) == 1) {
result = 1;
}
if (curDate.compareTo(startDate) == -1) {
result = -1;
}
return result;
}
}
Finally we need to implement the core functionalities that
reside in the ClusterWeightManager, which is the MBean
implementation class. The full code would be too long to include
here; see the sample code ZIP in the Resources section. For now, let's focus on the
important parts.
Before investigating any other method, we should discuss
applyInstanceWeights, which applies the weight of each
instance based on the timeSlice configuration that we
pass to it. Here's the implementation of the
applyInstanceWeights method.
private void applyInstanceWeights(TimeSlice currectTimeslice) {
String instanceNamePrefix = "amx:j2eeType=X-ClusteredServerConfig,name=";
Map<String, String> instanceWeights = currectTimeslice.getInstanceWeights();
try {
Set instanceNames = instanceWeights.keySet();
Iterator It = instanceNames.iterator();
while (It.hasNext()) {
String instanceName = (String) (It.next());
String instanceWeight = instanceWeights.get(instanceName);
ObjectName name = new ObjectName(instanceNamePrefix + instanceName);
Attribute attrib = new Attribute("LBWeight", new Integer(instanceWeight));
mBeanServer.setAttribute(name, attrib);
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
By looking carefully at the above code snippet, you can see that
we pass an object containing the current time slice to this method
and it will use its instanceWeights property, which is
a map of instance names and instance weights, to update instance
weights. As you may have noticed, the
instanceNamePrefix value is not complete, and in the
next few steps we append the instance name to make it a complete
ObjectName in order to let the
MBeanServer locate its corresponding object (a
clustered instance). Then we use this ObjectName and
MBeanServer to change the value of the
LBWeight attribute of the located instance.
One other important method is handleNotification,
which is inherited from NotificationListener. It
starts the instance weight update process when it receives the
cluster start notification.
public void handleNotification(Notification notification, Object handback) {
initialize();
this.startManager();
}
After receiving the notification, it calls the
initialize method, which parses the
XML file and creates an ArrayList of all time slices
included in the configuration file. The startManager
method schedules and starts the timer that we talked about
earlier.
You may ask, "What if our cluster is already running? Should we
restart it after we deploy the management rule?" In order to give an
answer to these questions, let's take a look at what JMX provides
for a similar situation and then to the solution that we used.
Usually when we need to perform some tasks after registering an
MBean, we implement the MBeanRegistration interface
and use its postRegister method to perform the
required logic. But in our case, we cannot do this. Why? Because we
have an XML file that contains the instance weight configuration, and
we put its address into one of our MBean variables;
postRegister will execute before setters, so we don't
have the configuration file address during the post-registration
phase to read the configuration. What we can do is use the
setConfigurationFilePath method to check the cluster
state and perform required actions if cluster is already
running.
In the setConfigurationFilePath method we simply use
the cluster name to create the complete ObjectName and
then check the state attribute of the corresponding
object using mBeanServer: 1 means that
the cluster is running, -1 indicates that the cluster
failed, and 0 means the cluster is stopped.
public void setConfigurationFilePath(String configurationFilePath) {
this.configurationFilePath = configurationFilePath;
String clusterPrefix = "amx:j2eeType=X-J2EECluster,name=";
initialize();
try {
ObjectName name = new ObjectName(clusterPrefix + clusterName);
Integer state = (Integer) mBeanServer.getAttribute(name, "state");
if (state == 1) {
isTimerRunning = true;
this.startManager();
}
} catch (Exception ex) {
ex.printStackTrace();
}
Now, in the handleNotification method we add code to check an object-level property to see whether the
timer is already running before we start another instance of the
timer.
Assuming you've unzipped the sample code and compiled
JAR file, the class files package and the sample configuration file
are available for this step. Copy the the entire package hierarchy
(which is inside a folder named classes in the attached
sample code) into domain_dir/applications/mbeans/ in order
to make the classes available to your domain class finder. Start
the application server and use the following asadmin
command to deploy the MBean into the DAS instance, making sure that
you change the configuration file path according to where you
copied the configuration file.
create-mbean --host localhost --port 4848 --user admin --name ClusterInstanceWeightsManager --attributes ConfigurationFilePath=c\:/config.xml samples.glassfish.management.clustermanager.ClusterWeightManager
Deploying the ClusterWeightManager MBean is equal
to including some configuration elements in the domain.xml file
and registering the ClusterWeightManager MBean with
the appropriate MBeanServer. During each startup, the
application server will register all MBeans included in
domain.xml with the MBeanServer. Now we have
our ClusterWeightManager MBean registered with
the management agent, and when it receives notifications it can start
the logic and update instance weights if required. But before it
can receive any notifications, it should announce its interest in receiving the cluster start notification. Creating a cluster
management rule is how we register our MBean's interest for one of
the cluster events.
We need to define the management rule, which will automatically send the cluster start notification to our MBean when the cluster starts. This management rule is the registration of our MBean as a listener for cluster events. One additional task of the management rule is filtering events, so our class will only receive an event type if we define our interest in it. The following asadmin command will create the required management rule.
create-management-rule --eventtype cluster --host localhost --port 4848 --user admin --ruleenabled=true --action ClusterInstanceWeightsManager --event loglevel INFO --recordevent=true --eventproperties name=start ClusterManagementRule
We can query our registered MBeans and management rules using
following asadmin commands.
>list-management-rules --user admin
>list-mbeans --user admin
In order to make this configuration automatically apply to your load balancer, you need to do some configuration using the GlassFish application server administration console. So locate your load balancer in the navigation tree and make sure to check the Automatically Apply Changes checkbox in the load balancer configuration page, under the General tab. To test the system, add two time slices close to each other and let the system run for enough time to cross from one one time slice to the other, and then open the GlassFish administration console. From the navigation tree locate your cluster, check the Instances tab in the cluster configuration page. You should notice that the instance weights are changed accordingly.
This article shows an small sample of using GlassFish's self-management features to perform some administration tasks that can be difficult and time-consuming to do manually. By looking at the sample code and concepts introduced in the article, you should be able to create your own MBeans and management rules to perform your daily administration tasks. On the other hand, AMX can be a good door for application server internal configuration and states. In the Resources section you can find many good references for going deeper into each concept or API that was introduced or used in this article.
Masoud Kalali has been programming for the last eight years.
 |
 java.net RSS Feeds
|
