The digital camera revolution leaves us with many images to
handle. I bought my Canon A95 camera over two years ago and have
taken some 13,000 photos since then. With such a deluge of images,
we need good software for quick and easy image viewing. What
software do you use to browse through images? Are you happy with
the way it works, and how it zooms an image and navigates around the
image to see enlarged parts of it?
My camera came with ZoomBrowser EX software that uses a
simple image viewer. Zooming only starts when a pull-down list with
zoom level values gains focus. The browser uses scroll bars for
image navigation, which is cumbersome and slow. It has arbitrary
lower and upper zoom limits and uses a "nearest neighbor"
interpolation that creates pixelation effects for large zoom
factors. Finally, the area of an image you are zooming in most
often disappears from the view, meaning you have to use scroll bars
in order to find it.
Wow, that is a long list of complaints! Wouldn't it be nice to
have an image viewer with fast, predictable zooming so that
magnified details of the image stay on the screen rather than
disappear from view, a viewer with good quality rendering, and
easy navigation around the zoomed image? The Navigable Image
Panel presented in this article, is my attempt at this.
Navigation
When an image is larger than its container's display area, a
scroll pane with scroll bars is commonly used to allow the user to
move the image around the container's view. Scroll bars also give
rough indication about the zoom level and how far away the
displayed area of the image is from the top, bottom, left, and right
edges of the image.
Scroll bars do not work well with zoomed images, especially at
large zoom levels. In most cases, the user needs to use both the
horizontal and vertical scroll bars to bring various areas of the
image into the view. Scroll bars are also of little value when it
comes to "having a larger picture": they say nothing about the
areas adjacent to the area currently in the view.
When I was looking for alternatives to scroll bars, I recalled a
different approach implemented in my Canon A95 digital camera. The
camera has four buttons to move the zoomed image around the view of
the LCD display, and the zoom lever for zooming in and out. It
shows a semi-transparent rectangle in the lower right-hand corner
of the LCD display, which represents the whole image (Figure 1).
Figure 1. Canon A95 navigation rectangle
There is also a smaller, solid rectangle inside the
semi-transparent rectangle, which represents the part of the image
currently displayed on the screen. The size of the solid rectangle
and its location within the semi-transparent rectangle are
proportional to the size and location of the displayed area of the
image. The smaller the visible, zoomed area of the image, the
smaller the solid rectangle. If we move the image towards the left
edge, the solid rectangle moves closer to the left edge of the
semi-transparent rectangle. Inspired by this solution, I dumped
scroll bars in favor of what I call a "navigation image."
The idea behind the proposed navigation is quite simple: we
display a smaller version of the image in the corner of the panel
and click it to show which part of the image should be displayed
(Figure 2).
Figure 2. Navigation image
Clicking the mouse anywhere within the navigation image causes
the panel to display that area of the image at the current zoom
level and centered around that point of the image where the mouse
click happened. In this way, we can quickly move around the image by
pointing with the mouse to the areas of interest, which is simpler and
faster than using scroll bars. In order to keep track of which part
of the image we are currently looking at, we draw a white rectangle
around that part of the navigation image.
Initially, the navigation image has the width of 15 percent of the
panel's width, but this can be easily changed. Move the mouse over
the navigation image and turn the mouse wheel in order to
increase/decrease the size of the navigation image.
The method of navigation described above is not only fast, but
also quite precise. And if this is not precise enough, you
can also drag the image with the mouse to adjust its exact position
in the view.
The Navigable Image Panel can also be navigated
programmatically, allowing the user to come up with a new, custom
GUI for navigation. In order to implement a custom navigation, turn
off the navigation image with the
setNavigationImageEnabled() method, and then use the
following methods to move the image around the panel:
getImageOrigin(), setImageOrigin(Point),
and setImageOrigin(int, int). The image origin
is the upper left corner of the scaled image in the panel
coordinate system. The reader might want to check the Coordinate
Systems section below to find out more about the various coordinate
systems used in Navigable Image Panel.
Zooming
Commonly, image viewers assume the center of the displayed image
is the zooming center. The zooming center is the point of an image
that remains stationary during zooming. Such a point stays at the
same location on the screen and other points move radially away
from it (zooming in) or towards it (zooming out). The further a
given point is from the zooming center, the faster it moves. As a
result, the more peripheral a detail of an image is, the greater
the chance that it will disappear off the screen before we can see it
at the desired magnification.
Navigable Image Panel is different in this regard. The zooming
center is not statically bound to the center of the panel. Instead,
it moves with the mouse pointer, or, in other words, is bound to
the mouse position. Navigable Image Panel assumes that the zooming
center is the point where the mouse pointer is. The
user moves the mouse pointer to the part of the image that he/she
is interested in, zooms in, and voila! that part remains
stationary, no matter how peripheral it is to the center of the
panel (Figures 3 and 4).
Figure 3. Mouse-bound zooming center: before zooming in
Figure 4. Mouse-bound zooming center: after zooming in
When an image is loaded into the panel, it is displayed in its
entirety with its aspect ratio preserved. The image stretches from the
top to bottom or left to right boundaries of the panel, depending
on its size, orientation, and the size of the panel. This is defined
as 100 percent of the image size and its corresponding zoom level is 1.0
(Figure 5).
Figure 5. Initial image size and location
The default zooming device is the mouse scroll wheel. Turning
the wheel by one position zooms the image by a zoom increment (the
default is 20 percent). Whether zooming is in or out depends on the mouse
wheel's direction. If the mouse does not have the scroll wheel, it
is easy to use two mouse buttons as a zooming device:
panel.setZoomDevice(ZoomDevice.MOUSE_BUTTON);
The left button zooms in and the right button zooms out.
Zooming can also be controlled programmatically, allowing
implementation of a custom method. In this case the user needs to
set the zoom device to "none" in order to disable both the mouse
wheel and buttons for zooming purposes:
panel.setZoomDevice(ZoomDevice.NONE);
Then, the user can use the setZoom() method to change the
zoom level. This method accepts a new zoom level as its parameter.
It is assumed that the zooming center is the center of the panel.
In order to be able to specify a different zooming center there is
an overloaded setZoom() method that accepts a new
zooming center as the second parameter:
setZoom(double newZoomLevel, Point newZoomingCenter)
For all zooming methods, the zoom increment value can be changed
with the setZoomIncrement() method.
High Quality Rendering
Before an image is rendered in the panel, it needs to be scaled.
The size of the image is different than the size of the panel and
some sort of interpolation/decimation is required to
increase/decrease the number of pixels to display the image for a
given zoom level. There are three possible interpolation algorithms
available in Java 5, each requiring different computational times
and producing results of different quality. The default
interpolation is nearest neighbor. Whenever an image is
drawn on the screen using Graphics.drawImage() method,
the nearest neighbor interpolation is applied by default, which
produces the fastest rendering and acceptable quality. The quality
is quite good up to a given zoom level. Beyond that level,
pixelation effects become visible and lines and boundaries become
jagged (Fig. 6).
Figure 6. Nearest neighbor interpolation
Figure 7. Bilinear interpolation
Bilinear interpolation is more time intensive, but
produces better results (Figure 7). Bicubic interpolation
requires even more computational time and its rendering quality is
the best of the three interpolation methods.
How fast is bilinear interpolation? It takes half a second
to render a 6-megapixel image on a PC with Windows XP, 1GB RAM,
and a 3.2GHz Pentium 4 CPU, using Java 5. Rendering time increases
with image resolution and reaches two seconds for 16-megapixel
images. These times are clearly too long for quick zooming in
Navigable Image Panel.
But wait a minute! Rendering time is proportional to the number
of pixels to be interpolated. When we zoom in, we decrease the
number of pixels in the original image that need to be interpolated
in order to fill up the screen. We could start with the nearest
neighbor interpolation initially, and then switch to bilinear
interpolation when the number of pixels to be processed is small
enough to ensure an acceptable speed of rendering. Let us try it
out by modifying the paintComponent() method in the
following way:
When the scaled image is smaller than the original image (scale
< 1.0) the Graphics.drawImage() method uses the
default, fast nearest neighbor interpolation. As the scaled image
becomes as large as the original image, we set a rendering hint to
instruct the rendering engine to use the bilinear interpolation
instead. When we run the code with the modified
paintComponent() method, everything works fine
initially. The moment the bilinear interpolation should start,
rendering is slow and does not show any signs of improvement, even
if we keep zooming in. This should come as no surprise, since Java
uses the immediate mode image buffer model. This model requires
processing of complete images, so interpolation is applied to the
whole image rather than to the part of it rendered on the screen. What we
need to do is to create a separate image that contains only the
pixels that will be used for interpolation and pass it to
drawImage(). The BufferedImage class has
a getSubimage() method that will do the job:
public void paintComponent(Graphics gr) {
super.paintComponent();
if (isHighQualityRendering()) {
Rectangle rect = getImageClipBounds();
BufferedImage subimage = image.getSubimage(rect.x,
rect.y, rect.width, rect.height);
Graphics2D gr2 = (Graphics2D)gr;
gr2.setRenderingHint(
RenderingHints.KEY_INTERPOLATION,
RenderingHints.VALUE_INTERPOLATION_BILINEAR);
gr.drawImage(subimage, ...);
}
}
Zooming now works well with bilinear interpolation in place.
Slightly slower image dragging at high zoom levels is the price we
pay for better rendering. The reader might want to experiment with
different threshold values for high quality rendering (the
HIGH_QUALITY_RENDERING_SCALE_THRESHOLD constant) in
order to get the best trade-off between responsiveness and
rendering quality. Those users who prefer top responsiveness over
better image quality for large zoom levels can turn off the high
quality rendering with the
setHighQualityRenderingEnabled() method.
Alternatively, they can upgrade to Java 6 and set the
INTERPOLATION_TYPE variable in the code to bicubic
interpolation. In this case, slight delay during image zooming and
dragging is similar when bilinear interpolation is set in Java
5.
Coordinate Systems
Throughout this article three different coordinate systems are
used (Figure 8).
Figure 8. Coordinate systems
Image coordinates refer to the original image. The image
coordinate origin is the IO point.
Screen image coordinates apply to a scaled image displayed in
the screen. The original image needs to be scaled before it is
rendered in the panel. Its width and height are multiplied by the
current zoom value (getScreenImageWidth() and
getScreenImageHeight()). The screen image coordinate
origin is the same IO point.
Panel coordinates refer to the rendering area of the panel.
Their origin (the originX and originY
variables in the image coordinates system) is the upper left corner
of the panel (the PO point).
All coordinate systems have x values increasing to the right and
y values increasing downward.
A number of methods translate coordinates from one coordinate
system to another. These are used by the zooming and image dragging
methods, making implementation easier and simpler. In order to
retain high accuracy when performing coordinate transformation
calculations, a custom Coords class is used instead of
Point, with double values rather than integers.
Memory and CPU Usage Considerations
NIP has been tested with a number of JPEG files, both large and
small, to assess memory and CPU usage. Most often a component like
NIP will be used in an application for browsing through a number of
JPEG files. At a minimum, two JPEG files will be loaded into memory
at any given point in time: one currently displayed in the image
panel, and the other to be read from a file (or any other
source), waiting to replace the first one. Therefore, all testing of
NIP has been done with two images in memory.
The most common digital cameras on the market today have resolutions of five to seven megapixels. They create 2-3MB JPEG files when the highest
quality is selected. NIP requires 33MB of RAM to run a simple test
program that displays two consecutive photos of this size. It needs
to be stressed that NIP itself requires around 22MB, leaving 10MB
for the two BufferedImages. With larger images, this proportion
changes and for two 16-megapixel images of 13 and 14 MB, NIP
requires 105MB of RAM. The
largest image I have found is 25MB image file from a 22-megapixel camera, and two of these requires 200MB of RAM. The amounts
of RAM above were declared using the -Xmx option of
the Java launcher.
Testing the largest 25MB image was interesting from a performance
point of view. Nearest neighbor interpolation was fast, but when
bilinear interpolation kicked in at a higher zoom level, the
performance ground to a halt. It took 25 seconds for the
Graphics.drawImage() to complete. I pressed Ctrl-Break
and analyzed a thread dump. It turned out that the test image I was
dealing with was not using the standard RGB color space, but
another one, and conversion to the standard RGB color space was
taking a long time. When I opened that image in GIMP and save it (GIMP can read images
with different color spaces and saves images in the standard RGB
color space), all the sluggish performance disappeared. In order to
test whether an image uses the standard RGB color space, the
isStandardRGBImage() method has been added to
Navigable Image Panel.
Conclusion
In this article I have presented an image panel that uses a
navigation image rather than scroll bars; sports powerful zooming
with a dynamic zooming center; and dynamic interpolation, providing
good quality rendering and satisfactory responsiveness. All
features of Navigable Image Panel can be controlled
programmatically, allowing the user to come up with new ways of
image zooming and navigation. The panel works well even with very
large images.
Hi chucked... It may be necessary to set the maximum size of the memory allocation pool of the Java interpreter in the IndexColorModel, using the -Xmx command line option. This is particularly important if you are attempting to work with large, untiled imagery.
* -Xmx<size>
o Sets the maximum size of the memory allocation pool (the garbage collected heap) to <size>. The default is 1 megabyte of memory. The size must be at least 1000 bytes.
By default, the size is measured in bytes. To specify the size in either kilobytes or megabytes, append "k" for kilobytes or "m" for megabytes.
* -Xms<size>
o Sets the startup size of the memory allocation pool (the garbage collected heap) to <size>. The default is 4 megabytes of memory. The size must be at least 1000 bytes and must be less than or equal to the maximum memory size (as specified by the tech decals -Xmx option).
By default, the size is measured in bytes. To specify the size in either kilobytes or megabytes, append "k" for kilobytes or "m" for megabytes.
Rendering speed in general
2007-08-19 01:27:41 chuckd
[Reply | View]
This was very interesting article and and darned cool application. Elegant OO design to boot.
It helped me to solve a problem I was having with some very slow repainting of large (order 10k x 10k pixel) raster images for a chart plotting application. The images are stored in a RLE encoding scheme with a small (4-bit or less) color table that makes them amenable to using an IndexColorModel. However these, like .gif images loaded into this app, while relatively efficient memory-use wise are dreadfully slow when doing any sub-image creation with .drawImage() or the other Image / BufferedImage methods to extract sub-images.
By loading the image rasters to a TYPE_3BYTE_BGR BufferedImage, rendering speed increases by up to a couple of orders of magnitude! ... though memory usage explodes from a few 10s of MB to several 100s of MB.
I'm curious if anyone has any insights into approaches that might preserve the memory benefits of using and IndexColorModel/MultiPixelPackedSampleModel while also producing the kind of blazing-fast performance seen with the PixelInterleavedSampleModel of the TYPE_3BYTE_BGR BufferedImage type.
I've been working on a picture visualisation tool some time ago, and you may find some interesting things in this work.
One of my goals was to experiment with a number of techniques to speed up the display and processing of images in java, while keeping the heap limit as low as possible.
Thanks for posting this. Java may have been in use for imaging for a long time but when I recently dove into it for the first time I found relatively few examples and articles on the subject.
I'm still learning about imaging but isn't Bicubic the best choise for scaling images? Even though it may also be the most demanding on the CPU.
A general weakness of Java images is just that; extreme performance demands that frequently lead to OutOfMemoryExceptions, which makes it more difficult to make good reliable end-user apps for advance imaging in Java - but we'll find ways to solve this, no worries!
Indeed, Bicubic interpolation gives the best looking results, but it is also time-intensive. It was my subjective opinion that Bicubic applied in Java 5 was a bit too slow for responsive dragging of an image at large zoom factors. As Chris from the Java2D team mentioned Java 6 has big scaling performance improvements, so big, that it justifies switching from Linear to Bicubic, which I suggest in the article.
is considered as a reference point for image scaling in the JAI community - I presume that this holds true also for Java2D, in any case I'm longing to see Chris' article.
For what concern allocation strategies and such, it's a complex stuff - I've written a specialized component for image manipulation (based on Java2D and JAI) and I think that multiple strategies should be offered to the programmer. For instance, a few years ago most of the non-Java professional imaging tool I remember (Photoshop, Nikon Capture Editor, ...) extensively used tiling (you could realize it when the computer was swapping on disk and the image was slowly rendered tile by tile). Most recent applications, such as Adobe Lightroom or Aperture IMO load the entire image in memory, in fact you can click on it and zoom in/out at light speed, even with some smooth animation.
Last but not least, the JDK has some infamous bugs... Slav mentioned the slowness with some color profiles (I believe that it's related to http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4886071 - in some cases can be worked around), but the speed can be dramatically different also depending on the samplemodel. Personally I've found that it's a good solution to convert it into the one that perform best in the target o.s. Some operations have very different speed on different o.s. (especially Mac OS X), so a lot of testing must be done.
Sorry - just to clarify "mature to do advanced image manipulation". Java has been doing advanced image manipulation for years in the scientific community. I was referring to the typical desktop applicatons dealing with photo for end-users.
Hi Slav: Nice article... I have just a couple comments/clarifications.
- Scaling performance was improved greatly in (Sun's implementation of) JDK 6. I didn't see this called out explicitly, but I guess you were alluding to it when you said "they can upgrade to Java 6 and set the INTERPOLATION_TYPE variable in the code to bicubic interpolation".
- It's best to avoid BufferedImage.getSubimage() whenever possible. Most people aren't aware that the subimage created will share the same underlying DataBuffer with the original, which can have unintended consequences if you're not careful (doesn't matter much in this case, but still worth mentioning; we're updating the javadocs to call this out more clearly). Even worse, prior to JDK 7, calling getSubimage() will actually defeat internal image management/acceleration mechanisms. And on top of all that, it will create garbage (a new BufferedImage object) everytime you call this method. So what's better? Use the variant of Graphics.drawImage() that takes a source and destination rectangle. For example:
public void paintComponent(Graphics gr) {
super.paintComponent();
if (isHighQualityRendering()) {
Rectangle rect = getImageClipBounds();
Graphics2D gr2 = (Graphics2D)gr;
gr2.setRenderingHint(
RenderingHints.KEY_INTERPOLATION,
RenderingHints.VALUE_INTERPOLATION_BILINEAR);
gr2.drawImage(subimage, dstx, dsty, dstw, dsth,
rect.x, rect.y, rect.width, rect.height, null);
}
}
Another benefit of using drawImage() directly for subimage scaling is that it opens the door to hardware accelerated scaling, when either the OpenGL or D3D pipeline is enabled.
I saw Figure 5 and this sentence "When the scaled image is smaller than the original image (scale < 1.0) the Graphics.drawImage() method uses the default, fast nearest neighbor interpolation."... I suspect the reason you're just using nearest neighbor interpolation in this case is that you didn't see any better quality with BILINEAR or BICUBIC in the downscaling case (much better than the quality in Figure 5). There are techniques for getting significantly better quality when downscaling, unfortunately they're not terrible obvious. I have an article that should be posted soon on java.net that's all about image scaling with Java 2D and goes into more detail about this and the other issues above. In the meantime, refer to Romain's image scaling code in the SwingX GraphicsUtilities class.
One other thing to mention regarding your "Memory and CPU Usage Considerations"... For applications like this, it is often unnecessary to load (and keep) the entire original image in memory. For the common cases where the user is just viewing at device resolution (scaling to fit the window), you can use ImageIO subsampling to reduce the size of the image that is read from disk. For large images (e.g. 6 MP images) you can cut memory consumption greatly by using this technique. Later, if the user needs to zoom in, you could consider scaling the lower-resolution image while the zoom is in progress, and then in the background reload the higher-resolution original image from disk.
I did notice and was very glad to see great scaling performance improvements in Java 6.
That is why I suggest users switch to the latest JDK 6 because then they can have Bicubic
interpolation without compromising scaling speed. Indeed, a big thank you is due to you
and the rest of the 2D team at Sun for your hard work regarding this performance boost.
So I take this opportunity and say, THANK YOU!
You are right about risks and disadvantages of using
BufferedImage.getSubimage().
In the context of this article it is not so crucial, but to promote good practices here is
a modified part of the paintComponent() method:
I've briefly tested the above code and it seems to work ok.
Nearest Neighbor interpolation was chosen for its fast scaling speed. It is possible
to improve image quality for zoom factors below 1.0, as you mention in your comment,
but I don't think an average user would cry for it. Navigable Image Panel is intended
for browsing through screen-big images and occasionally zooming in, but not for creating
polished icons. By the way, the Figure 5 is not intended to show the
image quality for zoom factors below 1.0. It is only for illustration purposes
to show that initially an image is rendered to fill up the screen.
I agree with you that subsampling would reduce memory consumption, but in this particular
case I opted for a simpler solution and chose to load whole images.
The digital camera market is dominated by 6-8 megapixel models at present. Two images
of this resolution and Java classes require less than 40MB. It is not a huge memory
requirement, so loading whole images makes sense, in my opinion.
A Navigable Image Panel
java.net RSS Feeds
