The directory ~/.MacOSX will need to be created. Apparently this file is read in at login time, so if you change it you will need to logout and log back in again for any change to make a difference. In my case I added the path to my Ant, my Findbugs, and /usr/local/bin which contains the Mercurial (hg) I want to have available in the PATH. I'm not sure NetBeans will actually use the version of Ant in the PATH, but that hasn't been an issue for me.

Why Does hg need to be in the PATH?

It was the need for /usr/local/bin (hg) in the path (/usr/local/bin) that got me started on all this because in my Ant file I wanted the build to automatically pull the version information out of the repository and make it available to the built product as a property setting. Effectively I need to run:

    # Get the last changeset
    hg tip --template '{node|short}\n'
    # Get the latest tag with a Version string in it
    hg log -l 1 --template '{desc|firstline}\n' -k "Version:"
    # Get the date of the last changeset
    hg tip --template '{date|shortdate}\n'

I used Ant rules something like this:

    <target name="hgpropfile" description="Create property file">
        <exec executable="hg" outputproperty="hg.tip.id">
            <arg value="tip"/>
            <arg value="--template"/>
            <arg value="{node|short}\n"/>
        </exec>
        <exec executable="hg" outputproperty="hg.last.tag.summary">
            <arg value="log"/>
            <arg value="-l"/>
            <arg value="1"/>
            <arg value="-k"/>
            <arg value="Version:"/>
            <arg value="--template"/>
            <arg value="{desc|firstline}\n"/>
        </exec>
        <exec executable="hg" outputproperty="hg.tip.date">
            <arg value="tip"/>
            <arg value="--template"/>
            <arg value="{date|shortdate}\n"/>
        </exec>
<!-- Indentation is critical here -->
<echo file="${dist.dir}/config-default.properties" append="false">
product.version=${hg.tip.date} ${hg.last.tag.summary} [${hg.tip.id}]
</echo>
    </target>

This property and it's value would need to be read in or made available at runtime, the actual Java code would just need to getProperty("product.version") to get the version string.

I used a tag to track the version code name, with the most recent tag containing "Version:" provides the product code name. The date and changeset id come from the tip, or most recent changeset.

Automating the creation of a new version code name tag can be done with a special Ant target used when needed. Effectively it needs to run:

    hg tag -f -m "Version: Name" TAG-YYYY-MM-DD

    <target name="new_version" description="Create new version tag">
        <tstamp>
            <format property="date.year.day" pattern="D"/>
        </tstamp>
        <tstamp>
            <format property="date.ymd" pattern="yyyy-MM-dd"/>
        </tstamp>
        <property name="versions.file" value="AllVersions"/> 
        <exec executable="wc" outputproperty="version.count.temp" input="${versions.file}">
            <arg value="-l"/>
        </exec>
        <exec executable="sed" outputproperty="version.count" inputstring="${version.count.temp}">
           <arg value="-e"/>
           <arg value="s@^[\ ]*\([1-9][0-9]*\)$@\1@"/>
        </exec>
        <exec executable="expr" outputproperty="version.selection">
            <arg value="${date.year.day}"/>
            <arg value="%"/>
            <arg value="${version.count}"/>
        </exec>
        <exec executable="sed" outputproperty="version.name" input="${versions.file}">
            <arg value="-e"/>
            <arg value="${version.selection},${version.selection}p"/>
            <arg value="-n"/>
        </exec>
        <exec executable="hg">
            <arg value="tag"/>
            <arg value="-f"/>
            <arg value="-m"/>
            <arg value="Version: ${version.name}"/>
            <arg value="TAG${date.ymd}"/>
        </exec>  
    </target>

Or in a Makefile you could do a similar thing:

DATE_YEAR_DAY:=$(shell date +%j)
DATE_YMD:=$(shell date +%Y-%m-%d)
VERSIONS_FILE=AllVersions
VERSION_COUNT:=$(shell cat $(VERSIONS_FILE) | wc -l | sed -e 's@^[\ ]*\([1-9][0-9]*\)$@\1@')
VERSION_SELECTION:=$(shell expr $(DATE_YEAR_DAY) '%' $(VERSION_COUNT))
VERSION_NAME:=$(shell cat $(VERSIONS_FILE) | sed -n -e "$(VERSION_SELECTION),$(VERSION_SELECTION)p" -n)
new_version:
        hg tag -f -m "Version: $(VERSION_NAME)" TAG$(DATE_YMD)

Now creating tags like this may not be advised or necessary with all repositories, but the basic principle can work in many situations. For example, with the OpenJDK project, the Release Engineering people will create the major milestone tags, and using those you could effectively identify a JDK version with a name (e.g. JDK 7 Build 23), and an exact changeset id. The trick is to get the version information from the repository, into the build tool (make or ant), into the product installation or baked into the product executable, and then available at runtime by the product plus easily seen when looking at an installation of the product. One issue I see is dealing with the situation where you are building a plain source tree without the Mercurial data or the Mercurial tools, somehow when the plain source tree is created the version data would need to be left in the source bundle.

Hope this is of some use to people. I'm sure there might be a better way, so if anyone has any ideas please add your comments. Ultimately I'd like a product to be able to provide enough details to a user so that the original source tree could be made quickly available. Given the changeset id, the exact and complete source could be re-created with hg clone --rev, of course that gets more complicated with a forest, but still pretty simple.

-kto

A Short java_crw_demo User Guide
2008-06-09 02:18:43

The java_crw_demo library is provided as a native code (C) demo of a BCI library that can instrument class files. It is just a demo, but an operating demo in that it has been used in the hprof VM agent and various JVM TI demo agents delivered in the OpenJDK JVM TI Demo Sources (typically the built versions of these demos are in the JDK installed on your system in the demo/jvmti directory). The complete code to java_crw_demo can be found in the OpenJDK Mercurial Repository. In particular, the #include file java_crw_demo.h is of primary interest.

A complete description of the class file format can be found in Chapter 4 of the Java Virtual Machine Specification (or look at the wikipedia entry on the class file format). Only part of the class file needs to be modified for basic instrumentation: the Code attribute (including the max-stack field), the constant pool, the Exception Table, the LineNumberTable, the LocalVariableTable, the LocalVariableTypeTable, the StackMapTable, and any StackMap attributes. The java_crw_demo does not add methods or fields and does not change the exception data except to adjust the pc offsets or byte offsets in the exception table. In fact the basic Table changes are just the pc or byteoffset adjustments. Any instrumentation more detailed than this would best be done with something other than a demo library like java_crw_demo, something more like ASM or BCEL would make much more sense.

Please keep in mind this is just a demo library, and fairly primitive in what it can do. It's functionality was driven by what hprof needed to do, e.g. instrument method entry and exit, and instrument memory allocations.

Basically there are just two functions in this library:

void java_crw_demo(
         unsigned              class_number,
	 const char *          name,
	 const unsigned char * file_image, 
	 long                  file_len,
	 int                   system_class,
	 char *                tclass_name,
	 char *                tclass_sig,
	 char *                call_name,
	 char *                call_sig,
	 char *                return_name,
	 char *                return_sig,
	 char *                obj_init_name,
	 char *                obj_init_sig,
	 char *                newarray_name,
	 char *                newarray_sig,
	 unsigned char **      pnew_file_image,
	 long *                pnew_file_len,
	 FatalErrorHandler     fatal_error_handler,
	 MethodNumberRegister  mnum_callback);

char * java_crw_demo_classname(
         const unsigned char * file_image, 
	 long                  file_len, 
	 FatalErrorHandler     fatal_error_handler);

The java_crw_demo_classname method is used to extract out the classname from a class file. In some cases classes are loaded into the VM without a name (see defineClass method in java.lang.ClassLoader.)

The java_crw_demo is the function you can call with a class file image and get back an instrumented class file image. The arguments are defined below:

• unsigned class_number
  A unique identifying number for this class in your agent (you get to define what this will mean). This number will be given back to you when the instrumentation code is executed for method calls and method returns. If you are not doing method call or method return instrumentation, this has little value. It is assumed that you would have some kind of table in the agent code that could map this class number to the class name and it's method tables if needed. is ess
• const char * name
  The name of the class in the form "java/lang/Object".
• const unsigned char * file_image
  The class file image.
• long file_len
  The number of bytes in the file_image.
• int system_class
  Set to non-zero if this class is one that is loaded very early in the VM startup. Great care needs to happen with modifying these classes during VM startup.
• char * tclass_name
  The name of the Tracker class that will have the static methods we will call as part of the instrumentation code.
• char * tclass_sig
  The class signature for the Tracker class.
• char * call_name
  The name of the static method in the Tracker class that will be used for method entries or indications of method calls.
• char * call_sig
  The method signature for the call_name method.
• char * return_name
  The name of the static method in the Tracker class that will be used for method exits or indications of method returns.
• char * return_sig
  The method signature for the return_name method.
• char * obj_init_name
  The name of the static method in the Tracker class that will be used for object allocations.
• char * obj_init_sig
  The method signature for the obj_init_name method.
• char * newarray_name
  The name of the static method in the Tracker class that will be used for array allocations.
• char * newarray_sig
  The method signature for the newarray_name method.
• unsigned char ** pnew_file_image
  If instrumentation happens, this will be a pointer to the new instrumented class file image, malloc() space.
• long * pnew_file_len
  The length of the new class file image returned in *pnew_file_image.
• FatalErrorHandler fatal_error_handler
  If non NULL, provides a function to call when fatal errors are encountered while parsing or creating the new class file image.
• MethodNumberRegister mnum_callback
  If non NULL, provides a callback function to get access to the method names and signatures in the class. This returns the class number you supplied plus arrays of method names and signatures plus a count of those methods. These method numbers (index into the arrays is the method number) are passed into the instrumented method calls, baked into the instrumentation.

It's assumed that the JVM TI agent code would request some kind of class load event, a good example is the heapTracker demo. When it gets a CLASS FILE LOAD HOOK event, it effectively passes in the class image to java_crw_demo:

newImage = NULL;
newLength = 0;
java_crw_demo(cnum, classname,
              class_data, class_data_len, systemClass,
              "HeapTracker", "LHeapTracker;",
              NULL, NULL,
              NULL, NULL,
              "newobj", "(Ljava/lang/Object;)V",
              "newarr", "(Ljava/lang/Object;)V",
              &newImage;, &newLength;,
              NULL, NULL);

Which only does instrumentation for object allocations and doesn't use the callbacks.

This demo doesn't fully use java_crw_demo like hprof, the hprof Tracker class is a complete Tracker class, while the heapTracker class is just a partial tracking class. The VM agent needs to implement and register the native methods for these Tracker classes.

The Tracker class doesn't have to use native methods, but since hprof was a native code agent, and most VM agents are native code, somehow the information captured via the class file instrumentation needs to get back into the native agent anyway.

A pure java VM agent via the java agent mechanisms is probably a better way to go, but at this time I don't have a simple demo of the java agent.

-kto

Why is NetBeans so slow? Here is a tip.
2008-06-09 02:18:43

I use NetBeans 6 all the time on my Mac laptop and have learned how to configure it to run more effectively for me. NetBeans, like any Java application needs room to work, and the default memory settings for Java and NetBeans are usually not what I consider ideal. Of course how would they know what is ideal, you need to tell it. Granted my solution may not be ideal for others, but it's worth playing with. I'm using jdk5 update 13 on my Mac, but these same setting should apply to jdk6 also. And it's not just for The Mac or even NetBeans 6, this should work on any platform.

KellyMacBook<3> cat ~/.netbeans/6.0/etc/netbeans.conf
netbeans_default_options="-J-Xms256m -J-Xmx512m -J-XX:PermSize=32m -J-XX:MaxPermSize=160m
-J-Xverify:none -J-Dapple.laf.useScreenMenuBar=true -J-XX:+UseConcMarkSweepGC
-J-XX:+CMSClassUnloadingEnabled -J-XX:+CMSPermGenSweepingEnabled"

Any change to this file will require a reboot (there should be one line in the file above, I had to break it up to get the blog to display it in a readable way).

Also, turn on the View->Toolbars->Memory to keep track of the NetBeans memory usage, clicking on it will force a GC, and you can get a feeling for how much memory your NetBeans session needs, making adjustments to the maximums above. And my recommendation is that the maximum (-J-Xmx) setting should never be more than the RAM on the machine minus 512m, or system memory thrashing may occur.

Additional information can be found at the NetBeans FAQ on GC pauses and the NetBeans FAQ on configuration.

The above settings make a world of difference in the way NetBeans performs for me.

-kto

Following The Rules
2008-06-09 02:18:43

With the recent release of the OpenJDK Developer Guide, I'm remembering Sister Dennis Anne from my grade school days. I never got the ruler slapped over the back of my hands, well, not very often, but my older brother practically had a ruler tatoo on his hands. Ouch. :^(

Now it's not all that bad, but I'm sure some of you have been thinking it is. Even Sister Dennis Anne didn't slap that ruler needlessly, well maybe my brother was a special case, he usually deserved it. ;^) Fortunately, the rules belong to us, not the Sisters.

In any case, please subscribe to the alias guide-discuss@openjdk.java.net if you'd like to make comments about the guide or discuss it.

-kto

P.S. Picture courtesy of listverse.com.

OpenJDK Bug States
2008-05-11 12:48:07

Until the OpenJDK project converts to Bugzilla, I thought some more information about how our internal bug tracking system works might help people watching the current situation.

The OpenJDK change requests (CRs) or bugs are currently visible at bugs.sun.com.

There are 11 states for a CR and the normal change of state is the following:

1. Dispatched: The initial state for a new CR.
2. Incomplete: Something critical is missing from the CR.
3. Accepted: CR was accepted, the first step in being investigated and fixed.
4. Defer: Work on this CR is on hold.
5. Cause Known: A basic understanding of the problem is known.
6. Fix Understood: A basic fix is now understood.
7. Fix in Progress: The assigned engineer is actively working on the fix or getting it integrated.
8. Fix Available: A changeset or the actual fix has been made available in a team area.
9. Fix Failed: Something went wrong with the fix.
10. Fix Delivered: The changeset or fix has been integrated into the master area and will show up in the next build promotion.
11. Closed: There are many reasons why a bug will be in the closed state. It might be verified as fixed, closed as a duplicate, closed as 'not a bug' or closed as 'will not fix'.

The states 1-9 are considered "unresolved", so until a bug becomes "10 - Fix Delivered" or "11 - Closed", it is still considered unresolved. So being "unresolved" may mean that a fix is available, just not in a position to be made part of a formal build promotion.

The bugs.sun.com interface is read-only and somewhat crude, but you can query the bug information, for example the Unresolved JDK Build Subcategory Bugs and RFEs. Hope this helps explain things.

-kto

Sun Studio Compilers on LINUX?
2008-05-07 12:48:42

With the magic of Mercurial, you can see changesets, like this one: http://hg.openjdk.java.net/jdk7/jdk7/hotspot/rev/485d403e94e1. Which Serguei Spitsyn integrated recently.

But wait, what does this changeset actually mean? Sun Studio on Linux? Does that make sense? YES! It does and it's true. Mind you, it's just Hotspot that can be built with the Sun Studio compilers on Linux right now, but it's an important piece, the Hotspot C++ code is not a trivial pile of code to compile and optimize correctly. My hat is off to the Sun Studio team in making this all happen. What should be interesting now is how well the rest of the tools work like dbx and the analyzer/collector.

More can be read about the Sun Studio Linux Compilers at the Sun Studio Site.

Humm, I guess I'm now on the hook to see if the rest of the OpenJDK can be setup to build with Sun Studio on Linux.... Back to work...

-kto

P.S. No, we are not abandoning gcc/g++, just providing choices on how the OpenJDK can be built.

InfoQ Article on Git, Mercurial, and Bzr
2008-05-07 12:19:38

A while back Sébastien Auvray asked me some questions about the OpenJDK Mercurial conversion. His article was recently published at http://www.infoq.com/articles/dvcs-guide.

Has some interesting information and stats about Git, Mercurial, and Bzr.

-kto

Windows, cygwin, mks, java, and a lump of coal
2008-04-30 18:48:04

I've been trying to setup a system that could easily switch between using MKS and CYGWIN, and also run from a Java program... what a pain.

Some of the issues I've run into:

• The Windows convention of using \ characters in pathnames is a major pain with \ being an escape character in most shells and in Java strings too. Most of the time you can get away with using / instead of \ on Windows, but the system seems to favor the \ character. If you use \ characters, depending on how many times the string is processed you might need \\ or \\\\ or even \\\\\\\\, how ugly is that?
• Java doesn't understand /cygdrive/ or /usr/bin paths, so you have to use cygpath -m to get the name into the 'mixed' style of C:/
• Java paths like CLASSPATH and properties with paths use the Windows convention of ";" separators and the C: or drive letter paths.
• When Java starts up it primes the java.library.path property with the current directory and the paths in the PATH variable. So depending on what is in PATH, you get a very different java.library.path setting. Also an old unfixed bug in java gives you duplicate entries in java.library.path. So a java process started from a cygwin shell could behave very different than being started in a non-cygwin shell.
• Spaces in paths become problematic, so using the MKS dosname -s or cygwin's cygpath -m -s is pretty essential. But those short non-space paths on Windows could become invalid if the original directory is deleted and re-created, so if a directory could come and go, those short paths have to be constantly recreated.
• The cygwin PATH variable is ':' separated cygwin paths, like any Unix system. On Windows, and MKS, the PATH variable used ';' as a separator and the cygwin paths must be transformed, using something like cygpath -m -p "${PATH}".
• Inside shells, it's always a good idea to quote the paths, and you have to quote the Windows PATH because it contains the ";" character, which is a shell command separator. Doing x=${PATH} won't work if the PATH has ";" characters in it or spaces. You end up quoting paths all the time, even using cygpath to transform a Windows PATH you should x=`cygpath -p "${PATH}"`
• Not sure exactly when and where but the shells seem to try and adjust and even add to the PATH settings sometimes. The cygwin shells have a -l option to adjust the environment as a login shell, but it's not entirely obvious where the profile file comes from. Comes down to "Where is HOME?" on a windows system running cygwin. :^(
• Both cygpath and dosname only work if the directory exists.
• Using the cygwin sshd doesn't guarantee the shell used when you ssh into the Windows machine. It appears to use whatever shell is found on the System PATH setting. Installing MKS adds MKS to the System PATH.

At this point, at least on Windows, I'm not too crazy about trying to get all these things to work well together. I may completely avoid shells on Windows from now on, it's like getting a lump of coal for XMAS. :^(

-kto

P.S. I see in a comment that I may have discouraged people from using cygwin, I did not mean to do that. Either MKS or cygwin on it's own, or mixed with Java is very possible, heck anything is possible. And I suspect there are other Windows Unix tool sets that would work just fine. The problem I was having was when they all got mixed together in a process chain. Creating a shell script that works with in either a MKS bash session and a cygwin bash session is quite possible, and not too messy. But having these two shell processes be connected is another story. I actually prefer cygwin even with the differences in PIDs and the pathname/PATH issues because it seems to behave like Unix, the ps and kill seem to be reliable. We have used MKS for a long time, and basically have been happy with it, our only issue is that it is not free for our OpenJDK users. So we are trying to convert to cygwin where we can, but bridging between the two at runtime is the most difficult.