Category Archives: Java

Using XPath to pick data out of XML

This week I wrote a WattDepot sensor for the TED 5000 home energy meter. The TED 5000 gateway (a small Internet-connected embedded computer) provides a URI that generates XML showing the current power data. First, I needed to figure out what the XML meant. Once that was done, I wanted a quick and simple way to pick out the 2 pieces of data from the XML that I care about using Java.

WattDepot uses JAXB extensively for XML processing, but that was kinda heavyweight for my needs here. I had heard about XPath, and it sounded like the right type of tool for just grabbing a little data from XML. Turns out that Java 1.5 and later have XPath built-in, so there’s no additional dependencies.

IBM has a good tutorial on using XPath from Java by Elliotte Rusty Harold. Unfortunately, I was confused initially because all the XPath examples in the tutorial are for finding all XML nodes in a document that meet certain criteria, whereas I knew exactly where in the XML tree my data was lurking. Luckily, it turns out that XPath is really a lot like a path in a filesystem (duh), so traversing the tree is easy.

Say you have the following XML from TED (some parts elided):

<LiveData>
  ...
  <Power>
    <Total>
      <PowerNow>2995</PowerNow>
      ...
      <PowerMTD>515227</PowerMTD>
      ...
    </Total>
  ...
  </Power>
</LiveData>

The XPath that would pull out the value from PowerNow is /LiveData/Power/Total/PowerNow/text(), and for PowerMTD it is /LiveData/Power/Total/PowerMTD/text(). Simple!

Here a code fragment that extracts those two values from an XML file (stealing liberally from the XPath tutorial linked above):

public class XPathTest {

  public static void main(String[] args) throws ParserConfigurationException, SAXException,
      IOException, XPathExpressionException {
    if (args.length != 1) {
      System.out.println("Need XML filename arg.");
      return;
    }
    DocumentBuilderFactory domFactory = DocumentBuilderFactory.newInstance();
    domFactory.setNamespaceAware(true);
    DocumentBuilder builder = domFactory.newDocumentBuilder();
    Document doc = builder.parse(args[0]);

    XPathFactory factory = XPathFactory.newInstance();
    XPath powerXpath = factory.newXPath();
    XPath energyXpath = factory.newXPath();
    XPathExpression exprPower = powerXpath.compile("/LiveData/Power/Total/PowerNow/text()");
    XPathExpression exprEnergy = energyXpath.compile("/LiveData/Power/Total/PowerMTD/text()");
    Object powerResult = exprPower.evaluate(doc, XPathConstants.NUMBER);
    Object energyResult = exprEnergy.evaluate(doc, XPathConstants.NUMBER);

    Double power = (Double) powerResult;
    Double energy = (Double) energyResult;
    System.out.println("Power from TED 5K: " + power + "W");
    System.out.println("Energy from TED 5K month to date: " + energy + "Wh");
  }
}

It’s nice to have a quick and easy way to make use of XML from Java in my toolbox.

Advertisements

it’s electric: TED data storage and plotting

I was checking on the website for The Energy Detective the other day looking for API info, and found that their page of 3rd-party applications had been updated, and included an application called it’s electric. it’s electric is a Java web application that queries the TED gateway frequently for the 1 second resolution power data, and stores it in a Berkeley DB. That alone is useful, as the TED has a segmented data storage system, keeping the 1 second resolution data only for an hour (and so on for coarser grained data).

It also provides a graphing system based on Google’s Annotated Timeline visualization, with some enhancements like automatically changing the resolution of the displayed data depending on the time interval displayed. Here’s a screenshot:

Screenshot of graph produced by it's electric

There’s a Google group for support and discussion, and the author Robert Tupelo-Schneck seems quite responsive. A jar file is provided on the group page (which I won’t link to since you should download the latest version), which includes the Java bytecode as well as the source, which is released under the AGPL license. The application is not large, consisting of 5 class files.

Compared to WattDepot, it’s electric seems considerably snappier. Presumably this is due in part to using Berkeley DB for persistence instead of an SQL database. The code also stores data in byte form, rather than higher-level Java objects and XML. Also, it’s electric occupies a clear functionality niche: it provides long-term storage of the finest-grained TED data (which is otherwise lost every hour), and provides graphing of that data from locations outside the home network.

I experienced some problems when scrolling around the data on the live it’s electric website, sometimes the graph would not update, or I was unable to scroll to where I wanted to apparently because new data was being loaded in for the current location.

Overall it’s electric looks like it could be useful for TED owners that want to hold on to that fine grained data, and want more options for displaying that data outside the home.

Debugging Restlet connector problem

In the course of developing WattDepot, I ran into an annoying intermittent bug in my JUnit tests. I would sometimes get a failure in one particular test class, but not always in the same method of that class. The failure manifested as a 60 pause on the affected test, followed by the WattDepotClient method returning a 1001 miscellaneous failure status code. Maddeningly, it would only fail sometimes, making it much harder to track down (and making continuous integration comical). Further, running the test from within Eclipse would work fine every time, so I was unable to use the debugger to figure out what was going on.

Philip pointed out that this sounded like a classic deadlock problem between threads, perhaps in Derby which I’m using for persistence. He suggested that I use VisualVM to see if I could track down any deadlocks. Mac OS X comes with VisualVM installed as “jvisualvm”, and it’s pretty easy to use. Luckily, since the failure manifested as a 60 second pause, I could start the test, and then attach to the JUnit process and obtain thread dumps to see what was going on.

After a few thread dumps, I tracked it down to HTTP communication. The failure happens when the client is using PUT to send a new resource to the server, and the server is waiting for the end of the entity body from the client. This happens before any Derby call, so it looks like Derby is ruled out (at least for this bug).

WattDepot uses the Restlet framework to make it easier to implement the REST API, and to perform all the HTTP client and server work. Restlet provides a variety of connectors for both the client and server HTTP connections. In fact, there are enough options that it is somewhat confusing trying to pick one. Restlet has internal HTTP client and server connectors that come in the core Restlet jars. According to this email thread, the choice of connector is done automatically by scanning the classpath, with the first match winning.

When first setting up WattDepot, I based the set of Restlet jars I was using on Hackystat. Hackystat’s SensorBase includes org.restlet.jar (API classes), com.noelios.restlet.jar (reference implementation, including internal HTTP connectors), com.noelios.restlet.ext.net.jar (client connector based on JDK HTTP code), and com.noelios.restlet.ext.simple_3.1.jar (server connector based on Simple framework). So it appears that WattDepot is using the Net connector for client HTTP connections, and the Simple connector for server connections, both overriding the internal HTTP connections in the reference implementation.

Since my problem was taking place in the HTTP code, I decided to try experimenting with removing Net and Simple from the classpath, thereby allowing the appropriate internal HTTP connector to kick in. Since I’m using Ivy and Ivy RoundUp for dependency management, this turns out to be as easy as changing the configuration parameter in the Restlet Ivy config, deleting the project “lib” directory and rerunning the tests.

After trying all combinations (all internal connectors, internal server & Net client, Simple server & internal client, Simple server & Net client), I found that only the combination of the Simple server connector and the Net client connector leads to my unit test failure. I guess I’m just lucky that way. 🙂

The solution is then to stop using either the Net client or the Simple server. Since the WattDepot server is likely to be the more performance-sensitive aspect of WattDepot, I opted to keep the Simple server on the assumption that it is higher performance than the internal Restlet server. It would be nice to figure out which of the variety of client and server connectors is recommended as the best performing, but this will do for now.

In the future I plan to post something to the Restlet mailing list to see if anyone else has run into this problem so it can be tracked down and perhaps fixed.

Java 1.6 & Eclipse on Mac OS X

These are some notes on using Java 1.6 on Mac OS X with Eclipse.

Mac OS X lagged behind for some time on adopting Java 1.6, but it was finally released as Java for Mac OS X 10.5 Update 1. Unfortunately, this release has some limitations: it only runs in 64-bit mode and only on Mac OS X 10.5 (Leopard). This meant no Java 1.6 support on Mac OS X 10.4, no support for PowerPC systems, and no support on the early 32-bit Intel Macs released in 2006.

To make Java 1.6 the default Java execution environment, run the Java Preferences application (found in the Utilities subfolder of the Applications folder). There you can drag the Java versions into any order you wish for both applications and applets. So to default to using Java 1.6 for everything, you can just drag Java SE 6 to the top of both the application and applet lists. The Java Preferences application will change around symlinks to correspond to your new choices. Mac OS X 10.6 (Snow Leopard) appears to include only Java SE 6, but includes both 32 and 64-bit versions, so it should allow Java 1.6 to be run on the oldest Intel Macs (no PPC because Snow Leopard does not support PPC).

Some command line scripts expect the JAVA_HOME environment variable to be set to the directory that contains the Java distribution being used. Also, some scripts might require different versions of Java. To support this, Apple introduced a new utility called /usr/libexec/java_home. By default, java_home just returns the home value appropriate based on the selection in the Java Preferences utility, but command line arguments can request different versions (see the manual page linked earlier). The output of java_home is intended to be assigned to the JAVA_HOME environment variable in a shell initialization script (like .profile or .cshrc).

This brings us to Eclipse, the Java development environment. Eclipse can be used to develop for multiple Java versions. When creating a Java project, Eclipse prompts for the desired JRE version in the new project dialog box. To ensure that Eclipse tracks the language differences between versions, also select the desired version in Preferences->Java->Compiler->Compiler compliance level to match your project.

However, despite these changes, Eclipse itself does not necessarily run using the Java version being used for the project being developed. Initially, the latest version of Eclipse (3.5) was released only in 32-bit mode for both Carbon (an older and deprecated Mac OS API) and Cocoa (the modern Mac OS API). Since Leopard only supported Java 1.6 in 64-bit mode, this meant that Eclipse was always running under Java 1.5. With the release of Eclipse 3.5.1, there are now 64-bit Cocoa downloads available, and these will run under Java 1.6. For some users, this may not be important, but for those doing Eclipse tool development (such as the Hackystat Eclipse sensor) it is very helpful. I use JAXB in some of my Ant build scripts, and JAXB is built into Java 1.6 but not Java 1.5. When running Eclipse in Java 1.5, many of my Ant scripts report spurious errors about being unable to locate JAXBException, but these all vanish when running Eclipse under Java 1.6. I should note that at least some people disagree with this advice, and suggest using the 32-bit Cocoa version on Leopard instead of the 64-bit version. Everyone apparently agrees that on Snow Leopard you want the 64-bit Cocoa version of Eclipse (unless you are on a 32-bit Intel Mac).

So if you want to use Java 1.6 for your application and run Eclipse under Java 1.6 there are two options:

  • Use Mac OS X 10.5 Leopard (with all software updates) on a Core 2 Duo Intel Mac, with Java Preferences set to use Java SE 6 as the default Java version, and use the 64-bit Cocoa version of Eclipse 3.5.1.
  • Use Mac OS X 10.6 Snow Leopard on an Intel Mac, and use the 64-bit Cocoa version of Eclipse 3.5.1 [note, I have not specifically tested this configuration, but it should work. Confirmed, this works]

Loading images in Java

In the Java project I am working on I needed to load an image from a file (representing a unknown user profile) for display to the user. This is easy to do. However, in production the application runs from a jar file, so I want my app to find the image inside the jar file. There are a number of example web pages on how to load images from jar files.

But I wanted something that also worked when I was developing in Eclipse, and none of the examples I found worked unless the application was running from a jar file. Here’s my simple solution:


// Try to grab unknown profile icon from JAR file
URL picturePath = this.getClass().getResource("/images/unknown-profile.jpg");
// However, if we are running from Eclipse then no JAR file
if (picturePath == null) {
  try {
    picturePath = new URL("file://" + System.getProperty("user.dir")
        + "/images/unknown-profile.jpg");
  }
  catch (MalformedURLException e) {
    System.out.println(bogusErrMessage);
  }
}
if (picturePath != null) {
  this.picture = new ImageIcon(picturePath);
  this.profile.setPhotoUrl(picturePath.toString());
}

Of course you need to have an “images” directory at the top level of your project directory, and you need to write appropriate ant tasks to copy over your images directory to your build directory when you create the jar file.