Category Archives: WattDepot

Notes from SmartGridComm 2010

Last week I attended the first Smart Grid Communications conference. I was presenting our paper on WattDepot. The presentation went well and there were several questions afterward. One point that I got several questions about was our requirement to support rapid data collection (sub-minute), since that is much faster than most commercial meters support. In retrospect, I should have emphasized our application domain (the Kukui Cup) more in the presentation, which might have helped to explain that requirement.

The following are some of my notes from the conference:

  • There were 102 papers accepted at the conference (40% acceptance rate), and 441 registrations. The mix was roughly 44% academic, 31% industry, 20% R&D, and 5% government.
  • The conference was held at NIST, and throughout the building were NIST clocks that presumably were very accurate. :)
  • There is a Smart Grid Consumer Collective, which sounds like an organization we should keep an eye on.
  • There are apparently standard power network topologies that can be used for analysis and research, such as the IEEE 300 bus power flow test case.
  • Some power industry slang I was not aware of: “big wire” relating to electricity distribution, “little wire” relating to communication networks. As in “the Smart Grid is all about the little wire people working with the big wire people.”
  • Georgios Kalogridis presented work on “Privacy for Smart Meters: Towards Undetectable Appliance Load Signatures”. The basic idea being that if the utility has fine-grained data about energy usage, they can determine a lot about what the consumers are doing. Kalogridis et. al. propose a system where a rechargeable battery is located in the home, and can be used to mask the signatures of energy use by charging and discharging at appropriate times. They cite this interesting paper by Elias Quinn that lays out all the potential privacy issues related to fine-grained energy usage data.
  • Came across a reference to Stanford’s PowerNet project, which aims to measure the “energy consumption of enterprise-style computing infrastructures”. Looks cool, paper here.
  • Hironori Nakanishi from the Japanese government reported on the Japanese perspective on the smart grid. Apparently Japanese utilities were not that eager to pursue smart grid upgrades because on average a Japanese consumer has 16 min/year of power outage, compared to the US average of 162 min/year. However, the Japanese government set the target of 25% CO2 reduction by 2020, which will require 28 GW of new solar power.
  • I also had not heard about the Hawaii-Okinawa partnership on clean energy.
  • One of the reasons that demand-side management is being pursued is for “peak shaving” by shifting some appliance loads to off-peak periods. Unfortunately, if all the smart appliances decide to shift their loads to the start of the off-peak pricing period, then you get a rebound peak. This seems like a challenging problem, barring real-time pricing that takes the rebound peak into account.
  • Jay Taneja presented work from UCB on shifting appliance loads to make maximal use of fluctuating renewable energy.The idea is that certain thermostatically-controlled loads like a refrigerator store electrical energy as thermal energy, so they can be scheduled based on the availability of renewable energy (in their study, wind energy). So if there is excess wind energy available, the fridge compressor could run more frequently, and vice versa. Bringing data about grid renewable energy production is something we have been thinking about from the consumer human interface perspective (the carbon “traffic light” concept), but they are working on making actual appliances sensitive to renewable production. Very cool.
  • In total, I gave out 11 business cards, and 2 REIS brochures.

Next year the conference is in Brussels. Paper deadline is April 4, 2011.

Cross-origin data access from JavaScript

WattDepot features a RESTful API that emits XML, and supports the Google Visualization API using the Google-provided Java library. There is a Java client library that makes it easy to write WattDepot clients in Java, and Google provides a library for JavaScript clients using the Google Visualization API. A several Java clients have been written and several JavaScript clients have been written.

Yichi Xu is working on a GeoMap visualization for WattDepot, and it needs to use the REST API rather than the Visualization API. This means he had to query the REST API directly, and parse the resulting XML. He quickly ran into the same-origin policy in JavaScript. In short, scripts running from a particular domain can generally only access resources from that same domain. This makes sense, because you don’t want any random web page you browse to have access to all your email just because you have a Gmail window up. However, from a developer’s perspective, this is a big when you want to use an HTTP API. Yichi’s gadget is loaded either locally or from Google’s servers, so when it tries to perform an HTTP GET to the WattDepot servers, it fails due to the same-origin policy.

There are a few workarounds for this problem, and Nelson Minar provides a good introduction. You can use proxies, but that’s gross. The most common solution makes use of a loophole in the same-origin policy: you can load JavaScript code from anywhere (just not data). So if you wrap your data so that it looks like code, then you can violate the same-origin policy. If you are providing the data in the JSON format, once wrapped it becomes JSONP.

Right now, WattDepot doesn’t support JSON representations (though it may in the future), so the JSONP trick is not available to us. Luckily, Yichi came across this draft from W3C on Cross-Origin Resource Sharing. CORS allows servers to emit HTTP headers that indicate how browsers should restrict access to the resource. The header can open things up so anyone can use the resource, or only certain other domains. However, this requires browsers to act on the header, luckily Firefox 3.5 and the latest Safari both implement this (the Firefox page has an excellent discussion of CORS). For now, supporting Firefox 3.5 and Safari is acceptable, so WattDepot just emits “Access-Control-Allow-Origin: *” for all requests. In the future this will be cleaned up to only emit this for public sources, and also provide JSON and JSONP representations for easier use in JavaScript.

Using XPath to pick data out of XML

This week I wrote a WattDepot sensor for the TED 5000 home energy meter. The TED 5000 gateway (a small Internet-connected embedded computer) provides a URI that generates XML showing the current power data. First, I needed to figure out what the XML meant. Once that was done, I wanted a quick and simple way to pick out the 2 pieces of data from the XML that I care about using Java.

WattDepot uses JAXB extensively for XML processing, but that was kinda heavyweight for my needs here. I had heard about XPath, and it sounded like the right type of tool for just grabbing a little data from XML. Turns out that Java 1.5 and later have XPath built-in, so there’s no additional dependencies.

IBM has a good tutorial on using XPath from Java by Elliotte Rusty Harold. Unfortunately, I was confused initially because all the XPath examples in the tutorial are for finding all XML nodes in a document that meet certain criteria, whereas I knew exactly where in the XML tree my data was lurking. Luckily, it turns out that XPath is really a lot like a path in a filesystem (duh), so traversing the tree is easy.

Say you have the following XML from TED (some parts elided):


The XPath that would pull out the value from PowerNow is /LiveData/Power/Total/PowerNow/text(), and for PowerMTD it is /LiveData/Power/Total/PowerMTD/text(). Simple!

Here a code fragment that extracts those two values from an XML file (stealing liberally from the XPath tutorial linked above):

public class XPathTest {

  public static void main(String[] args) throws ParserConfigurationException, SAXException,
      IOException, XPathExpressionException {
    if (args.length != 1) {
      System.out.println("Need XML filename arg.");
    DocumentBuilderFactory domFactory = DocumentBuilderFactory.newInstance();
    DocumentBuilder builder = domFactory.newDocumentBuilder();
    Document doc = builder.parse(args[0]);

    XPathFactory factory = XPathFactory.newInstance();
    XPath powerXpath = factory.newXPath();
    XPath energyXpath = factory.newXPath();
    XPathExpression exprPower = powerXpath.compile("/LiveData/Power/Total/PowerNow/text()");
    XPathExpression exprEnergy = energyXpath.compile("/LiveData/Power/Total/PowerMTD/text()");
    Object powerResult = exprPower.evaluate(doc, XPathConstants.NUMBER);
    Object energyResult = exprEnergy.evaluate(doc, XPathConstants.NUMBER);

    Double power = (Double) powerResult;
    Double energy = (Double) energyResult;
    System.out.println("Power from TED 5K: " + power + "W");
    System.out.println("Energy from TED 5K month to date: " + energy + "Wh");

It’s nice to have a quick and easy way to make use of XML from Java in my toolbox.

it’s electric: TED data storage and plotting

I was checking on the website for The Energy Detective the other day looking for API info, and found that their page of 3rd-party applications had been updated, and included an application called it’s electric. it’s electric is a Java web application that queries the TED gateway frequently for the 1 second resolution power data, and stores it in a Berkeley DB. That alone is useful, as the TED has a segmented data storage system, keeping the 1 second resolution data only for an hour (and so on for coarser grained data).

It also provides a graphing system based on Google’s Annotated Timeline visualization, with some enhancements like automatically changing the resolution of the displayed data depending on the time interval displayed. Here’s a screenshot:

Screenshot of graph produced by it's electric

There’s a Google group for support and discussion, and the author Robert Tupelo-Schneck seems quite responsive. A jar file is provided on the group page (which I won’t link to since you should download the latest version), which includes the Java bytecode as well as the source, which is released under the AGPL license. The application is not large, consisting of 5 class files.

Compared to WattDepot, it’s electric seems considerably snappier. Presumably this is due in part to using Berkeley DB for persistence instead of an SQL database. The code also stores data in byte form, rather than higher-level Java objects and XML. Also, it’s electric occupies a clear functionality niche: it provides long-term storage of the finest-grained TED data (which is otherwise lost every hour), and provides graphing of that data from locations outside the home network.

I experienced some problems when scrolling around the data on the live it’s electric website, sometimes the graph would not update, or I was unable to scroll to where I wanted to apparently because new data was being loaded in for the current location.

Overall it’s electric looks like it could be useful for TED owners that want to hold on to that fine grained data, and want more options for displaying that data outside the home.

WattDepot going “real-time”

In the past week I have added a new REST API method to support near-real-time queries in WattDepot. The goal is to support user interface widgets that display the latest sensor data from a source, such as a smart meter in a home or dormitory. I have also written a command line monitoring client that shows how to use the new functionality. Both of these will be released as part of WattDepot 1.2 in the near future, hopefully with the addition of a sensor that collects data from TED 5000 home smart meters.

Speaking of sensors, I created a wiki page that explains how to write a WattDepot sensor. This should be helpful for anyone planning to write a sensor to support a new type of meter.

In other WattDepot news, there are three projects in ICS414 this semester that are related to WattDepot. The WattDepot Apps team are working on demonstration web applications for WattDepot. The first application is a visualizer that makes use of the Google Visualization API to make graphs of WattDepot data. It should be ready for a 1.0 release very soon. Next they will be moving on to create a web application that monitors the latest sensor data from a source using the new API method. In the future, hopefully they will be working on a browsing application that lets users look over the users and sources in a WattDepot repository.

The Stoplight Gadget team is working on a Google Visualization gadget that checks a data source for a value, and based on user-settable thresholds displays a traffic light as either red, yellow, or green. While this is a general-purpose visualization gadget, we expect to use it with WattDepot data as part of the UH dorm energy competition, though precisely how is yet to be determined.

Finally, the Energy Meter team is surveying power meters that can be used for the UH dorm energy competition. While they have been in a data gathering phase so far, they are now switching to implement a Modbus/TCP sensor for WattDepot. This sensor will be used to collect data from the floors of the dorms in the energy competition.

Debugging Restlet connector problem

In the course of developing WattDepot, I ran into an annoying intermittent bug in my JUnit tests. I would sometimes get a failure in one particular test class, but not always in the same method of that class. The failure manifested as a 60 pause on the affected test, followed by the WattDepotClient method returning a 1001 miscellaneous failure status code. Maddeningly, it would only fail sometimes, making it much harder to track down (and making continuous integration comical). Further, running the test from within Eclipse would work fine every time, so I was unable to use the debugger to figure out what was going on.

Philip pointed out that this sounded like a classic deadlock problem between threads, perhaps in Derby which I’m using for persistence. He suggested that I use VisualVM to see if I could track down any deadlocks. Mac OS X comes with VisualVM installed as “jvisualvm”, and it’s pretty easy to use. Luckily, since the failure manifested as a 60 second pause, I could start the test, and then attach to the JUnit process and obtain thread dumps to see what was going on.

After a few thread dumps, I tracked it down to HTTP communication. The failure happens when the client is using PUT to send a new resource to the server, and the server is waiting for the end of the entity body from the client. This happens before any Derby call, so it looks like Derby is ruled out (at least for this bug).

WattDepot uses the Restlet framework to make it easier to implement the REST API, and to perform all the HTTP client and server work. Restlet provides a variety of connectors for both the client and server HTTP connections. In fact, there are enough options that it is somewhat confusing trying to pick one. Restlet has internal HTTP client and server connectors that come in the core Restlet jars. According to this email thread, the choice of connector is done automatically by scanning the classpath, with the first match winning.

When first setting up WattDepot, I based the set of Restlet jars I was using on Hackystat. Hackystat’s SensorBase includes org.restlet.jar (API classes), com.noelios.restlet.jar (reference implementation, including internal HTTP connectors), (client connector based on JDK HTTP code), and com.noelios.restlet.ext.simple_3.1.jar (server connector based on Simple framework). So it appears that WattDepot is using the Net connector for client HTTP connections, and the Simple connector for server connections, both overriding the internal HTTP connections in the reference implementation.

Since my problem was taking place in the HTTP code, I decided to try experimenting with removing Net and Simple from the classpath, thereby allowing the appropriate internal HTTP connector to kick in. Since I’m using Ivy and Ivy RoundUp for dependency management, this turns out to be as easy as changing the configuration parameter in the Restlet Ivy config, deleting the project “lib” directory and rerunning the tests.

After trying all combinations (all internal connectors, internal server & Net client, Simple server & internal client, Simple server & Net client), I found that only the combination of the Simple server connector and the Net client connector leads to my unit test failure. I guess I’m just lucky that way. :)

The solution is then to stop using either the Net client or the Simple server. Since the WattDepot server is likely to be the more performance-sensitive aspect of WattDepot, I opted to keep the Simple server on the assumption that it is higher performance than the internal Restlet server. It would be nice to figure out which of the variety of client and server connectors is recommended as the best performing, but this will do for now.

In the future I plan to post something to the Restlet mailing list to see if anyone else has run into this problem so it can be tracked down and perhaps fixed.

Future publication venues

Update March 5, 2010: now maintained as a wiki page rather than a blog post

Updated Feb 2, 2010: added IEEE upcoming journals and IEEE Sensors conference.

So I’ve been thinking lately about where I might publish my research on WattDepot, and later on the UH dorm energy challenge. Here’s what I have come up with so far.


  • IEEE Smart Grid Conference. The deadline for papers is May 1, with the conference happening Oct 4-6 in Maryland. Philip has suggested this might be a good place for a paper on WattDepot, and I agree. The maximum page length is 6 pages.
  • Behavior, Energy, Climate Change 2010. Philip attended BECC 2009 and it seems like an ideal conference for the dorm energy competition results. The call for abstracts (presentations & posters) goes out in March, with a mid-May deadline to submit abstracts. The conference is November 14-17 in Sacramento. There is no paper required for the presentation, just slides, so we could potentially present some actual results in the presentation (which wouldn’t be available in May when the abstract is submitted).
  • Hawaii International Conference on System Sciences 44. This is happening on Kauai January 4-7 2011. The deadline for papers is June 15. The Jennifer Mankoff’s group has had a series of papers about StepGreen and related work at HICSS so this seems like a good venue, and the travel will be much easier. :)
  • CHI 2011. Apparently this is happening May 7-12 in Vancouver (BC I assume?). There have been a variety of papers on supporting green/sustainable behavior in CHI before, and the CHI community is a large and vibrant one.
  • Ubicomp 2010. There was plenty of sustainability work at Ubicomp 2008 (including a workshop I attended), so this is a possible venue. However, the submission deadlines are rather soon (March for papers) so it’s probably more realistic for 2011.
  • Pervasive 2011. This is similar to Ubicomp, but happens in May in Europe. Submission deadline is mid-October.
  • IEEE Sensors 2010. May 4 is the abstract submission deadline, with the conferencing happening November 1-4 at the Hilton Waikaloa. Maximum paper length is 4 pages. This is perhaps less relevant than the IEEE Smart Grid conference, but still worthy of consideration.


  • International Journal of Sustainability in Higher Education. This is where the Oberlin dorm energy contest paper was published, so it seems an obvious choice. Not the broadest appeal though.
  • Environment & Behavior.
  • IEEE Transactions on Smart Grid. Journal to be launched soon.
  • IEEE Transactions on Sustainable Energy. Another journal to be launched soon. The smart grid one looks more relevant, this looks to be focused on energy generation from renewables.

I’m sure more will come up as I read more, but this is a good starter list.

Prepping for screencast

Last week I focused mostly on non-programming tasks of reviewing the NSF proposal and preparing for the WattDepot screencast. I did close one refactoring issue that I generated when rolling out the Google Visualization support.

This week will be focused on the issues identified as critical for a WattDepot 1.0 release.

Plans from last week:

  • Review grant proposal
    • done
  • Plan script for 10 minute screencast
    • in progress
  • Close some additional WattDepot issues
    • closed refactoring issue 24

Other accomplishments from last week:

  • Created WattDepot issues for items discussed with Philip

Pointers to work products:

Plans for this week:

  • Create stress tests for WattDepot (issue 32)
  • Benchmark WattDepot API using stress tests
  • Optimize Derby implementation based on performance data
  • Finish script for 10 minute screencast
  • Make 10 minute screencast?
  • Start support for energy counters in Energy class
  • CSDL firewall reorg

Cool links:

WattDepot, now with Google Visualizations

After discussion with Philip, decided to prioritize the Google Visualization data source work over performance testing. WattDepot now provides a data source for the Google Visualization API that allows plotting multiple types of data on the same graph, and expanding virtual sources into their component subsources. I have written a wiki page about using WattDepot’s Google Visualization data source.

Plans from last week:

  • Test performance of Derby implementation
    • deferred for now
  • Make GVisualizationServlet work with virtual sources
    • done
  • Review grant proposal?
    • not done, working on it this week
  • Close some more WattDepot open issues
    • closed some issues, but opened more new ones

Other accomplishments from last week:

  • Registered for Spring 2010 classes
  • Tracked down CSDL’s copy of Snow Leopard Server from UHM Bookstore
  • Pushed out new WattDepot release and notified 413/613 students of visualization availability
  • Watched some crazy big-wave surfing in Waimea Bay without getting sunburned

Pointers to work products:

Plans for this week:

  • Review NSF proposal
  • Plan script for 10 minute screencast
  • Close some additional WattDepot issues

Cool links:

  • Google Goggles is an Android application that lets you take pictures of objects (only some are recognized) and does a search for relevant information. Will do OCR on text found if it doesn’t know about that type of object.

WattDepot has persistence

The Derby persistence implementation is now complete. This week I will start performance testing the Derby persistence implementation, and work on the GVisualizationServlet so we can visualize virtual sources. Loading default data into WattDepot using Derby for storage (no network traffic) is running at ~876 sensor data entries per second, which seems pretty good.

Plans from last week:

  • Remove last of HashMap code from DerbyStorageImplementation
    • done, along with some other cleanup (like running unit tests on both DB implementations)
  • Start performance testing of Derby storage implementation
    • just started

Other accomplishments from last week:

  • Ate lots of delicious food

Pointers to work products:

Plans for this week:

  • Test performance of Derby implementation
  • Make GVisualizationServlet work with virtual sources
  • Review grant proposal?
  • Close some more WattDepot open issues

Cool links: