Java Code Splitter

Struggling with Servlet 3.0

A couple of days ago, I wrote a blog post about a session which I had attended during JavaOne. It was a session about the upcoming Servlet 3.0 specification, also known as JSR-315. Back at work, I thought it might be fun to try some of the new features myself. We have this one application running in our production site, which stores events that occur during Poker games. The application uses a Service based on Hessian, which clients can call to store the events. The Hessian service is exposed using the HessianServlet from Caucho, so that it may be invoked from HTTP. The Servlet runs in Jetty 6 container. The application is heavily used and around 20 to 30 calls are made per second.

My idea was to rewrite some parts within the application, so that everything is based on Servlet 3.0. I have written a test case where I measure the execution time of 30 parallel Threads persisting 30 Poker events. I am hoping that using Servlet 3.0 asynchronous requests, I can see a performance improvement. I know that this is not the perfect use case where asynchronous requests can shine as there is nothing that can be optimized using parallel processing within a single Thread. Anyway, we will see how it goes. I am curious about the results anyway.

The first step towards Servlet 3.0 is to use a JSR-315 compliant Servlet container. At JavaOne everyone was talking about the upcoming Jetty 7 release and that it supports Servlet 3.0. This was my disappointment. Jetty 7 is not built on top of Servlet 3.0 but still only Servlet 2.5 compatible. The semiofficial reason is that Jetty has moved from Codehaus to Eclipse and JSR-315 is delayed anyways. My next pick would have been Tomcat 7 as I have seen a comparison matrix that indicated, that the next Tomcat would support Servlet 3.0 as well. Unfortunately I don't think development is that far. I found some source code in SVN but I don't know if it was official and it was also only Servlet 2.5 based. So my last resort was Glassfish v3 which is the reference implementation for Java EE6. The preview release of Glassfish v3 comes with a Servlet container that implements JSR-315. Perfect.

It was the first time I installed Glassfish. It was very easy. The application server ships with a web administration interface and is easy to maintain. Currently there are not so many tutorials and examples for Servlet 3.0 on the Web, so I looked forward to check some samples which ship with Glassfish. "After installation, samples are located in install-dir/glassfish/samples/javaee6" - well this directory just does not exist. Not in the standard preview nor in the web profile. Too bad, they were supposed to have a sample for asynchronous requests as well as adding Servlet's dynamically.

Anyway, I changed my application from a standalone jar distribution that starts a Jetty 6 container to a war distribution that is deployed in Glassfish v3. To deploy something in Glassfish, just copy it into the autodeploy directory of your domain in the glassfish directory. Since the old version uses the HessianServlet directly, I had to download the source from Caucho and modify it, so that it uses asynchronous requests from Servlet 3.0. Unfortunately the HessianServlet is not really built for extensibility, so I just got a copy of the whole file to play with. To use another Sevlet 3.0 feature, I decided to add the Servlet at runtime using the new ServletContext.addServlet method. I looked up a sample on how to write a ServletContextListener. Some old documentation about JSR-315 indicated that you had to annotate the Listener with the @WebServletContextListener annotation. This annotation does not exist anymore in the final draft of JSR-315. Instead you do it the oldschool way. Write a class that implements ServletContextListener and add it to the web.xml as a context-listener. Then in the contextInitialized method, I added my AsynchronousHessianServlet.

In the next posting I will write about asynchronous requests and if this really makes an existing application faster.

Update: the javaee6 samples will be downloaded and installed using the Glassfish updater tool. In my first install attempt, the updater would not work with my companies firewall, so I never got the samples folder. It works fine if your updater tools works. Would have been nice to mention on the Glassfish or Sun website.

Session of the day: Java NIO2 in JDK7

Today has been a good day at the JavaOne. I have seen quite a few great and useful talks. For the session of the day I have picked a talk by Alan Bateman and Carl Quinn from Netflix about the new IO API (JSR203) that will be available in JDK7.

In my own private projects I still use the old Java IO API and I guess that's perfectly fine if your application is not IO critical. At work however, we have multiple projects that make use of java.nio and are very much defendant on a good performance when it comes to files and directories. So what can JSR-203 do for us?

First of all there will be a class Path which is an abstraction to a physical file or directory resource. Path is basically what File used to be in plain Java IO. To create a Path instance you have a bunch of options. You can call FileSystems.getDefault().getPath("/foo/bar") or just Paths.get("/foo/bar"). One nice thing is that Path will implement java.lang.Iterable so that you can iterate over a physical path from root to current directory. If you want to know at which depth you currently are from the root just call Path.getNameCount(). Another nice thing, when you iterate over Path using the legacy Iterator idiom and you invoke iterator.remove, then the physical file gets deleted.

In the example code above you already saw something called FileSystem. This is the a of all Paths in NIO2. In JDK7 there will also be something called a Provider which you can leverage to create your own FileSystem. It will be possible to create a memory based FileSystem, a Desktop FS or a Hadoop FileSystem or anything else you can think of. You can even make your FileSystem the default FileSystem so that whenever your application calls FileSystems.getDefault() will return your custom FileSystem.

Another cool thing is the possibility to traverse a directory tree. NIO2 contains a Interface called FileVisitor. The Interface has a bunch of methods like preVisitDirectory, postVisitDirectory, visitFile, visitFileFailed etc. Each of the methods will be invoked at certain stages when traversing a file tree. For convinced JSR-203 ships with a bunch of implementations of FileVisitor like SimpleFileVisitor or InvokeFileVisitor. You can use one of these FileVisitor's and then only overwrite the methods that are interesting for you. To kick off traversal of the file tree you would call Files.walkFileTree(Path path, FileVisitor fileVisitor).

This becomes really handy when you use it in conjunction with another new class in JDK7 called PathMatcher. This is an Interface similar to FileFilter in old Java IO maybe. This is how to create a PathMatcher: FileSystems.getDefault().getPathMatcher("glob:*.log"). In this example it will select all files matching *.log. You can also use regular expressions instead of glob syntax.

If you look at the method signature of visitFile in the FileVisitor Interface you will notice that the second parameter is of type BasicFileAttributes, which are the attributes of the current file that visitFile is invoked with. So let's say you create your FileVisitor with a PathMather that selects *.log files. What you could do in the visitFile method, is to invoke PathMatcher.match and if it is a log file, check the file size attribute using the given BasicFileAttributes. If the file is bigger than a certain size, delete it. Pretty handy or? A piece of very short Java code that traverses a File tree and deleted logfiles of a certain size.




PathMatcher matcher = FileSystems.getDefault().getPathMatcher("glob:*.{java,class}");

Path filename = ...;
if (matcher.matches(filename)) {
    System.out.println(filename);
}

A entirely different use case can be covered with WatchService, Watchable, WatchEvent and WatchKey. These guys make it possible to sit, listen and react to changes that occur to Path objects. First you get yourself a WatchService using the default FileSystem. WatchService watcher = FileSystems.getDefault().newWatchService(). The next step is to get the Path like before Paths.get("/foo/bar/old.log"). Then you register the Path with the WatchService to get a WatchKey: WatchKey key = path.register(watcher, ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY). The last parameters in the register method are varargs. In the example you will be watching create, delete and modification events on the specified Path. Finally you need to create an invinite loop that is constantly polling the events out of the WatchKey. Your code can then react to these events.




for (;;) {

    //wait for key to be signaled
    WatchKey key;
    try {
 key = watcher.take();
    } catch (InterruptedException x) {
 return;
    }

    for (WatchEvent event: key.pollEvents()) {
 WatchEvent.Kind kind = event.kind();

        //This key is registered only for ENTRY_CREATE events,
        //but an OVERFLOW event can occur regardless if events are
        //lost or discarded.
 if (kind == OVERFLOW) {
     continue;
 }

        // do your stuff
    }

    boolean valid = key.reset();
    if (!valid) {
        break;
    }
}

One new feature that is particular interesting for our applications is the new DirectoryStream class. You use it to access the contents of a directory. Well you could do this before but DirectoryStream scales much better and uses less resources. This will not be an issue if you have a couple of hunded files in your directory but make a huge difference if there are hundreds of thousand files in the directory. Here is how to use DirectoryStream.




Path dir = new File("/foo/bar").toPath(); // new method on File in JDK7
DirectoryStream stream = null;
try {
    stream = dir.newDirectoryStream();
    for (Path file: stream) {
        System.out.println(file.getName());
    }
} catch (IOException x) {
    ....
} finally {
    if (stream != null) stream.close();
}

Okay this post was maybe a bit too theoretical. You can check out what nio2 feels like with the OpenJDK or this great tutorial.

Session of the day: JVM debugging

The second day at JavaOne was surprisingly just average. Kohsuke had a great talk about distributed Hudson builds and Hudson EC2 integration but the rest of the sessions was pretty normal. Then I am going into this session “Debugging your production JVM” from Ken Sipe and the guy blasts the roof off. He is showing all these nifty, cool tools that can help to get an insight what is going on in your JVM. Not just one tool but many. I am really having a hard time to write everything down. A lot of command-line tools that already come with Java like jstat, jps or jmap. They are ready to be used, you just need to know how to use them with the right parameters etc.

Then Ken starts to talk about something really fancy - BTrace. So how can BTrace help to debug your production VM. Essentially it is a little tool that you can use at runtime to at debugging “aspects” to your running Java bytecode. I use the word Aspect here because when I first saw it, BTrace felt quite similar to AspectJ. What BTrace does, it takes a little Script that your write in Java and dynamically injects it as tracing code into the running JVM.

What I called Script here is pure Java code. What you write looks almost exactly like a plain Java Classes with a lot of Annotations. The Code you can write as BTrace Script is really limited however. Ken had a slide in the presentation about all the Java stuff that is not doable. I only remember that you could not use the new keyword to create new objects. Luckily I found the other restrictions on the BTrace website:

can not create new
can not create new arrays.
can not throw exceptions.
can not catch exceptions.
can not make arbitrary instance or static method calls - only the public static methods of com.sun.btrace.BTraceUtils class may be called from a BTrace program.
can not assign to static or instance fields of target program's classes and objects. But, BTrace class can assign to it's own static fields ("trace state" can be mutated).
can not have instance fields and methods. Only static public void returning methods are allowed for a BTrace class. And all fields have to be static.
can not have outer, inner, nested or local classes.
can not have synchronized blocks or synchronized methods.
can not have loops (for, while, do..while)
can not extend arbitrary class (super class has to be java.lang.Object)
can not implement interfaces.
can not contains assert statements.
can not use class literals.

Your hands are tied. Well almost. So let's have a look at a sample from the BTrace website.




import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;

// @BTrace annotation tells that this is a BTrace program
@BTrace
public class HelloWorld {

// @OnMethod annotation tells where to probe.
// In this example, we are interested in entry
// into the Thread.start() method.
@OnMethod(
    clazz="java.lang.Thread",
    method="start"
)
public static void func() {
    // println is defined in BTraceUtils
    // you can only call the static methods of BTraceUtils
    println("about to start a thread!");
}
}

You can see, it is a standard Java class annotated with @BTrace. Then it says @OnMethod with two parameters which translates to – every time the start method is invoked in java.lang.Thread ... What it will do in that case is invoke the static function it will find within the @BTrace annotated class. It has to be a static method. I forgot if it had to follow a naming convention too. So the static method will be invoked every time a Thread is started. In the sample, it will just print out something fixed on the console. You could also count the number of Threads or other things.

Here is another example.




@BTrace public class Memory {
@OnTimer(4000)
public static void printMem() {
    println("Heap:");
    println(heapUsage());
    println("Non-Heap:");
    println(nonHeapUsage());
}
}

This will print out memory usage every 4 seconds. Awesome. Now something really huge.




@BTrace public class HistogramBean {
// @Property exposes this field as MBean attribute
@Property
private static Map histo = newHashMap();

@OnMethod(
    clazz="javax.swing.JComponent",
    method=""
)
public static void onnewObject(@Self Object obj) {
    ....
}

@OnTimer(4000)
public static void print() {
    if (size(histo) != 0) {
        printNumberMap("Component Histogram", histo);
    }
}
}

Don't worry about the details what it does right now. The important part is that you can annotate fields with @Property and have BTrace expose these fields as Mbean. All you need to do is inject the BTrace script a little bit different from the command line.

Some words on the @OnMethod, @OnTimer stuff. These Annotations are called probe points and there are more like @OnMemoryLow, @OnExit, @OnError etc. Another example is to use BTrace to monitor entering and leaving of synchronization blocks.

Unfortunately BTrace requires Java 6+, it will not work with 5. Now you have a good reason to step up your Java version.

Session of the day: Servlet 3.0

SVG and Canvas is really cool stuff and “Cross Browser Vector Graphics with SVG and Canvas” came really close to be my session of the day here at JavaOne, but the Servlet 3.0 stuff topped it all. There are so many great features in JSR-315. The final draft is now out and the guys said it will go live with Java EE6.

web.xml = history
First of all, the biggest difference is that you do not need a web.xml anymore. Servlet 3.0 fully relies on Java Annotations. If you want to declare a Servlet use the @WebServlet Annotation on your Servlet class. The class still has to inherit from HttpServlet, this remains unchanged. That means it is not possible to use other methods instead of doPost, doGet etc. and annotate them to mark them as Request handler methods. This is something you can do in the Jersey library (JSR-311). In theory I guess it would have been possible to not inherit from HttpServlet but the create a rule that for every class annotated with @WebServlet you have to have methods annotated with @PostMethod or @GetMethod. I guess there are good reasons not to do so in Servlet 3.0.

At the very minimum, you have to annotate specifying a URL under which to invoke your Servlet. If omitted, the full class name will be used as the Servlet name. Other parameters you can specify in the top-level @WebServlet Annotation are for instance if you want the Servlet to be usable in asynchronous requests. More on asynchronous requests later.




@WebServlet(url = "/foo", asynchronous = true)
public class SomeServlet extends HttpServlet {
 public void doGet(...) {
     ....
 }
}

So the web.xml is gone. Filters are added to the ServletContext using @WebFilter Annotation, Listeners are added using @WebListener Annotation. The deployment descriptor File web.xml is still useful though. It can be used to overwrite whatever you have specified using Class Annotations. So if you create a web.xml file, whatever you have in there has the final word when the Container starts up the ServletContext.

Servlets, Listeners and Filters can now also be added programatically. The ServletContext class has new methods like addServlet, addServletMapping, addFilter or addFilterMapping. On Container start up, you can hook in and add Servlets or whatever you want at Runtime.

Web Frameworks can plug in
Something that I think is really cool, is the possibility for Web Frameworks like Apache Wicket, Tapestry or Spring MVC to plug-in into the ServletContext creation. Remember that in the past, whenever you learned about a new web framework, there was this one section in the documentation where you had to add some Servlet, some Filter or some Listener to the web.xml?




<web-app>
 <display-name>Wicket Examples</display-name>
 <filter>
     <filter-name>HelloWorldApplication</filter-name>
     <filter-class>org.apache.wicket.protocol.http.WicketFilter</filter-class>
     <init-param>
       <param-name>applicationClassName</param-name>
       <param-value>org.apache.wicket.examples.helloworld.HelloWorldApplication</param-value>
     </init-param>
 </filter>
 <filter-mapping>
     <filter-name>HelloWorldApplication</filter-name>
     <url-pattern>/*</url-pattern>
 </filter-mapping>
</web-app>

This is now history in Servlet 3.0. Web Frameworks can supply something in their JAR-file deployables that is called web-fragment.xml. It is basically a light version of the web.xml and has almost the exact XML structure. What the Container does when it loads up, it will go into all the JAR files in WEB-INF/lib and scan for a web-fragment.xml File in the META-INF directory. If it finds one, the file will be used when creating the final ServletContext. Sometimes you want more control over how fragments are being pulled into the ServletContext creation. There are ways to control the ordering in which web-fragment.xml files are being put together. The library itself can specify a relative ordering in the web-fragment.xml file. For instance it can say, I am not interested in a particular order, load me last. It is also possible to define an absolute ordering in your own web.xml file which will have again the final word. One important thing of notice. Only the WEB-INF/lib directory will be used to scan for web fragments, not the classes directory.

Third party libraries can also add resources on their own. Whatever the web framework has in META-INF/resources becomes available from the context in which the library is loaded. For instance if you have META-INF/resources/foo.jsp then the resource is available from http://localhost:8080/foo.jsp. Kind of useful too.

New security, New Error Page

Security constraints can now also be expressed using Annotations. I forgot the correct Annotation names, I think it was something like @DenyRoles or something that you could put on for instance doPost methods to secure them. I vaguely remember also, that the standard security mechanism to display a Web Form (Form Based Authentication) is removed. What you now do instead is to properly annotate a method which will authenticate the User for you. You are therefore basically free to choose however you want to authenticate your Users. Unfortunately this is a feature in the Servlet Specification that I have almost never used in the last 10 years, so I did not pay too much attention when they talked about this and the security part was also kept short in the session.

Which brings me to the next improvement, default error pages. Remember back in the old Servlet 2.3+ days that you had to add a default error page for every error code? What a copy and paste mess. JSR-315 gives you the possibility to define default error pages. You can have for instance something like “show this page for all errors except 404”. Very handy.

New methods in Response
Remember that it was a pain in the ass to work with HttpResponse sometimes? It was not apparent in which phase the response was, what the status was. Servlet 3.0 will add new methods to HttpRepsonse that will make our lives easier. You can get Header name, Headers and the Response status now using API calls.

Asynchronous Requests
Finally to the most impressive new feature - asynchronous requests. This is huge! Imagine that you have HTTP POST method in a Servlet and what it does is to call out to a WebService. While the Service is doing it's work, the HttpRequest is being held by the Servlet Container. Standard Stuff. This however becomes problematic if your thread pool limit is reached. So lets say you have defined that 50 Threads should be in the Pool. You receive 30 Requests per second. If the web service call takes 2 seconds, you have a problem because 60 Requests are coming in and only 50 Threads are in the pool.

Here is what Servlet 3.0 does. Well, it is kind of hard to explain and I did not understand it fully. But here is what I think it does. You can specify on Servlet and Filters that they may be used for asynchronous requests. This is something you have to declare as an attribute of the @WebServlet or @WebFilter Annotation. The request comes in, it will look at the Filter chain and the Servlet. It will figure out if it can run asynchronously or not. The original request calls out and the Request is suspended, therefore freeing a Thread and returning it to the pool. A callback method is given along. The callback method is invoked when the external resource becomes available and the Container will use a new Thread to generate the Response. The Request therefore resumes it's work. New methods for resuming and suspending as well as querying the current status of the request have been added. There was also something called asynchronous Handle in the presentation slides but I forgot how it was used.

Anyway the guy at JavaOne had an example ready where he had written a Servlet that queried the Ebay Rest Interface for 3 keywords. He then had written the same using asynchronous Requests from Servlet 3.0 and it was like 3 or 4 times faster. This is because not only are the Threads returned to the pool while they are doing nothing (Thread starvation) but also can multiple Requests be run in parallel. This is a major improvement I think. Our current web applications can be made faster just by using features of a new Servlet specification.

I hope this was useful. I will experiment with Servlet 3.0 before the final release comes out. I think I might be able to do this today already using some experimental version of Jetty or Glassfish.

Welcome bag

I made it through the first day of the CommunityOne West. Unfortunately my drive back from Fresno to California took longer than I thought. I was only be able to attend two conference sessions in the afternoon. The first session was about database architectures for web 2.0 applications. Personally I found the talk a bit superficial. Too much high level stuff. I wrote down a few buzzwords like Memcache, Multi Statement Requests, Horizontal Partitioning and Hibernate Shards.

The second session was an introduction to Amazon EC2 and it was very interesting. F

inally I know what Cloud Computing stands for and how it works. I realized, that I was programming in the cloud already, as I had used Google App Engine. Chris Richardson, who did the session, sounded a bit negative every time he mentioned Google App Engine. At least I got these vibes. He did not like that GAE was not fully 100% Java compliant and that database access was limited to Appengine's own datastore. Well, at least it is free I'd say.

Before I went to the CommunityOne West, I checked in for tomorrows Java One and got my welcome bag. It contained a T-shirt, a jacket, some papers, an Open Solaris CD and something, you would have had a big question mark above your head for - antibacterial hand gel. Pig influenza is just around the corner.

Zeitgeist - what you can do as Java developer

Since JavaOne is in San Francisco, I used the chance that I am in the Bay Area and visited a friend, that I had not seen a few years, in Fresno. We ended up hacking around and watching documentaries on DVD. One particular documentary that made quite an impression on me was Zeitgeist Addendum. You can watch it here on the blog but beware it has full-movie length.

In Zeitgeist Addendum the author evaluates how things tick today by looking at our system and the society. He starts by looking at the global financial system and how it is doomed to fail sooner or later. In the second part, someone talks about how small countries are controlled and corrupted by organizations in the worlds leading countries. It becomes apparent that this system will never change by itself. The last two parts, which I found most interesting, talk about alternatives to our current society structures. There is the Venus Project as a different society experiment. It is shown how different things are handled in the Venus Project than they are in our current society.

One of the biggest problems today is that we live in a profit orientated society. You need money to survive. Companies need to make more money every year. A lot of bad things we see in our world today are directly connected to the profit orientated society. The destroying of our worlds environment. The wars that countries fight against each other. In Zeitgeist Addendum it is proposed that the profit based system must be weakened for mankind to move on. Today, environmental friendly technology is held back or killed by competition just to make more profit. The solution to a lot of problems are not politicians but technology. How can I help?

This is where I think even as a, maybe unimportant, Java developer you can help. Because this is what we create – technology. Start being a better developer. Educate yourself. Read a book at least once in three months. This will give you new ideas, this will motivate you, this will maybe help you to be more efficient, to create more in a shorter time. Help other developers around you. Do not think about why you should help or teach them. Do not be jealous to what they have and you don't. Do not be angry why they get away being inefficient every day. Give them something back instead, from what you have learned and therefore help others to become better and more efficient too.

Automate things you are doing repeatedly and slow you down. If you spent 3 hours every week, looking at various log files, investigating why something did not work as expected - create something that will make it easier and more efficient for you. First of all, you will learn something while you do it. Secondly, if you only need 30 minutes instead of 3 hours, you have gained 2 ½ hours in which you can work on something more challenging - maybe the next MapReduce algorithm.

Support the community. Participate in writing open source software. Give answers to other peoples questions. Write a blog. Publish the super-cool code snippet you have written, so that others save time. How can this help to weaken the profit based, monetary system? Well, maybe your answer to some question, your code, your open-source plugin will be used by someone else to create better technology. Technology that will makes it possible to travel faster using renewable energies, so that we don't need airplanes anymore. Or you indirectly help to create software that examines data intensive DNA patterns and cancer can be cured.

Start using Linux. Almost everything that you do on a Windows powered machine, you can do using free software and Linux too. By using Linux and showing others (developers, family) how easy it is to use, or that it even exists, you help weakening the influence of Microsoft. I do not say Windows is bad, it is really great even. It just costs too much and Microsoft has for a long time leveraged the fact that there was no alternative. Unfortunatly an OS is something every computer needs. You cannot sell it for a couple of hunded bucks. Come on. Support the idea of free software. Imagine a world in which software is free and everyone has access to it. So many people that could help creating new, helpful technology.

Support companies creating modern technology like Google. Even though Google is a profit orientated company, they have in the last years given so much great stuff out for free. Take Google App Engine. Isn't it great to have a place that does not cost you anything, where you can put your own Python slash Java applications and share it with others? Take the MapReduce paradigm I mentioned earlier. Google has shared it with others. It can now be leveraged by till example Health Researchers for data mining. There is a backside of the coin though. Google is dominating search. The world of search is paralyzed. People are searching for contents like they did in 1999. There cannot be new development in search if everyone uses Google. Think about alternatives, even though it hurts, I know :)

You work in IT. You are making probably more money than the average salesman in your town. Think about switching to energy providers offering energy from renewable sources. This will likely costs more. It can even be that the energy is not from a renewable source at all (some people call that cheating by the way). But it will create awareness if people start paying extra for nice energy. As more and more people will hop on, companies will fully shift renewable energy. They will invest into renewable energy and it will become cheaper. Start to turn off computers when you are not using them. I had mine always running during the night. Probably because I was lazy to turn them off or I wanted to save time turning them on again next day. Save yourself some money and some non-renewable energy.

Alright now it seems to be getting too far off. Just watch Zeitgeist Addendum and think about it. I am off for JavaOne now.

Spending time before Java One in San Francisco

I finally made it to San Francisco. My company sent me and ten colleagues to the Java One conference this year. This is supposed to be the biggest Java related conference in the world. We will see about that. It starts next Tuesday. In the meantime, I have some extra days I can spent in San Francisco. The plan is to travel to Fresno today and meet a friend which I have not seen for a couple of years.

Even before the Java One starts are there a lot of conferences about agile software development, Java based technologies or vendor specific stuff. Some of them are even for free. There is the CommunityOne West that starts next Monday. It is a side conference, also hosted by Sun. The Community One West goes from 1st to 3rd of June, also in Moscone Center. Since it is starting one day before the real JavaOne conference, it will be a great opportunity for me to get my head filled up with Java stuff even earlier. This is the CommunityOne program for Monday. There are a lot of sessions about Open Solaris and Cloud Computing. Since I will unfortunately only be coming back from Fresno on Monday, I will miss the morning and lunch sessions of the Community One West. However, there are some “pearls” I found during the afternoon like “Dynamic Data in a Web 2.0 World”, “Three Techniques for Database Scalability with Hibernate” or “What Do You Need to Know About Creating and Running a Scalable Web Site but Were Afraid to Ask?” Looking forward to go there.

Anyway, I want to quit this post with some practical tips for you guys entering the US. One of my colleagues was really hit hard this time from the US border control. I am not sure that is the correct name but they are the guys who will check your filled in papers from the plane and ask all these questions. Obviously they had found something in his profile or he just looked similar to someone they were looking for. He was asked to not proceed to the Exit but to another office called “Secondary”. In there, they asked a lot of detailed questions and this time even with a quite obvious background. Something like “Do you have family or friends in Saudi Arabia, Iran or somewhere else in the middle east?” I guess he was tempted to answer that our former System Owner migrated from Iran to Sweden some 20 years ago :)

Then the staff was really going into detail with questions about Java One, like what type of conference it is. How many years he had worked in our company. What his position is in the company. What exactly he was doing etc. Finally, he cleared secondary. Now everyone, even the Exit people, are in an area between the border security and the customs. This is where you pick up your luggage. I headed directly to the bathroom to get my hands washed. It is really smart, in times where US has the most Swine Influenza cases, to have everyone press their four finger and thumb of both hands on a fingerprint scanner!

Waiting for the luggage, some guys with beagle dogs went around, checking bags. These dogs are really great. They found a lot of food in peoples bags. It was fun to watch - snap, dog caught ya. After a couple of minutes, we got the bags and moved out. Guess what, my colleague was picked on again and he had to go someplace else. They asked him, if he wanted to change something in his customs declaration paper. Obviously that was not the case and they started searching his belongings. Of course they did not find anything.

Other stuff. Take it easy when leaving the plane. Border control usually starts off with only a few counters, so it looks like you have to wait forever. They will then open more and more counters. The passengers who were among the first, waited longest. It is better to come late I'd say. Make sure you have both sides on all forms filled in. Be prepared to answer detailed questions about the purpose of your trip and details about the place you are staying. Do not travel to the US if you do not speak English please. I had a Spanish lady in my queue, filling in a German I-94 form, who could not talk Any! English. It was a disaster. She did not even know what to fill in in which fields, not to talk about the questions they asked her. I know it is ignorant but thats the way it is, you have to be at least OK in English.

How to Google App Enginefy your existing Java application and fail

Last month Google added Java support to the Google App Engine. This means, that you can from now on host Java based applications on the Google infrastructure. Even better is the fact, that you can do it for free. Well, it is free for starters. There are quotas like 5 million page views per day, 6,5 hours CPU time per day, 1GB traffic per day etc. For the everyday "lets-do-something in Java" app, this is more than enough, so I decided to run a simple Wicket web-application on Googles appengine.

I am still a part-time student at Fernuni Hagen in Germany and in the current semester I am in a course called “Web 2.0 and social software”. The course is organized so that 3 students have to pick a topic from the web 2.0 bubble and prepare a presentation about it. Recently I have read the book “Collective Intelligence in Action” so I thought I might as well talk about Tag, Tagging and TagClouds. It is always better to not just talk in your presentations but show some real action. Therefore I created a little web-application, based on Satnam Alag's domain model from the book and powered with Apache Wicket and Spring to be fully functional.

The non Google App Enginefied version uses an in-memory HSQL database to persist Tags, Users and Items. This means, if you restart the application, everything will be gone. Everything is visualized using a TagCloud embedded in a Apache Wicket WebPage. The page will also contain entry forms to add new Users, Items and Tags. So here is the domain model which consists of five classes. The Entity class is an abstract base class for Item, Tag, TaggedItem and User.




public abstract class Entity implements Serializable
{
    private static final long serialVersionUID = 1L;

    public int id;

    public String name;

    public int getId()
    {
        return id;
    }

    public void setId(int id)
    {
        this.id = id;
    }

    public String getName()
    {
        return name;
    }

    public void setName(String name)
    {
        this.name = name;
    }

    @Override
    public boolean equals(Object o)
    {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        Entity entity = (Entity) o;

        if (id != entity.id) return false;

        return true;
    }

    @Override
    public int hashCode()
    {
        return new Integer(id).hashCode();
    }
}

public class User extends Entity
{
}

public class Tag extends Entity
{
}

public class Item extends Entity
{
}

public class TaggedItem extends Entity
{
    private int _userId;

    private int _itemId;

    private int _tagId;

    private int _weight;

    public int getUserId()
    {
        return _userId;
    }

    public void setUserId(final int userId)
    {
        _userId = userId;
    }

    public int getItemId()
    {
        return _itemId;
    }

    public void setItemId(final int itemId)
    {
        _itemId = itemId;
    }

    public int getTagId()
    {
        return _tagId;
    }

    public void setTagId(final int tagId)
    {
        _tagId = tagId;
    }

    public int getWeight()
    {
        return _weight;
    }

    public void setWeight(final int weight)
    {
        _weight = weight;
    }
}

The CRUD operations for this domain model are implemented using the DAO pattern (surprise, surprise). On top of the hierarchy sits the GenericDao interface which contains methods for read, write, delete. Then I have created interfaces for each individual DAO class that is connected to one entity in the domain model, ie. UserDao or ItemDao. These individual DAO interfaces extend the GenericDao, specifying the Entity to persist and the primary key type. The concrete implementation is now done in classes like UserJdbcDao or ItemJdbcDao. These are implementing the appropriate interface and extend a helper class called AbstractJdbcDao, which contains a reference to the Spring SimpleJdbcTemplate. Here is an example of the class structure for the User class.




public interface GenericDao<T, PK extends Serializable>
{

    /**
     * Persist the newInstance object into database
     */
    PK create(T newInstance);

    /**
     * Retrieve an object that was previously persisted to the database using
     * the indicated id as primary key
     */
    T read(PK id);

    /**
     * Retrieves all objects that were previously persisted to the database.
     */
    List<T> readAll();

    /**
     * Save changes made to a persistent object.
     */
    void update(T transientObject);

    /**
     * Remove an object from persistent storage in the database
     */
    void delete(T persistentObject);
}

public interface UserDao extends GenericDao<User, Long>
{
    /**
     * Returns the {@link User} whose name matches the given String.
     * @param name a users name
     * @return User or <code>null</code>
     */
    User readByName(final String name);
}

public abstract class AbstractJdbcDao<T extends Entity>
{
    private SimpleJdbcTemplate m_simpleJdbcTemplate;
    private String m_identityQuery;

    protected SimpleJdbcTemplate getSimpleJdbcTemplate()
    {
        return m_simpleJdbcTemplate;
    }

    public void setDataSource(DataSource dataSource)
    {
        m_simpleJdbcTemplate = new SimpleJdbcTemplate(dataSource);
    }

    public String getIdentityQuery()
    {
        return m_identityQuery;
    }

    public void setIdentityQuery(final String identityQuery)
    {
        m_identityQuery = identityQuery;
    }

    abstract ParameterizedRowMapper<T> getMapper();
}

public class UserJdbcDao extends AbstractJdbcDao<User> implements UserDao
{
    private static final ParameterizedRowMapper<User> MAPPER = new ParameterizedRowMapper<User>()
    {
        public User mapRow(ResultSet rs, int rowNum) throws SQLException
        {
            final User user = new User();
            user.setId(rs.getLong("user_id"));
            user.setName(rs.getString("user_name"));
            return user;
        }
    };

    public Long create(User newInstance)
    {
        final SimpleJdbcTemplate template = getSimpleJdbcTemplate();
        template.update("INSERT INTO user(user_name) VALUES(?)", newInstance.getName());
        return template.queryForLong(getIdentityQuery());
    }

    public User read(Long id)
    {
        final SimpleJdbcTemplate template = getSimpleJdbcTemplate();
        final List<User> users = template.query("SELECT * FROM user WHERE user_id = ?", getMapper(), id);
        return users.isEmpty() ? null : users.get(0);
    }

    public User readByName(final String name)
    {
        final SimpleJdbcTemplate template = getSimpleJdbcTemplate();
        final List<User> users = template.query("SELECT * FROM user WHERE user_name = ?", getMapper(), name);
        return users.isEmpty() ? null : users.get(0);
    }

    public List<User> readAll()
    {
        final SimpleJdbcTemplate template = getSimpleJdbcTemplate();
        return template.query("SELECT * FROM user", getMapper());
    }

    public void update(User transientObject)
    {
        throw new UnsupportedOperationException("Not implemented yet.");
    }

    public void delete(User persistentObject)
    {
        final SimpleJdbcTemplate template = getSimpleJdbcTemplate();
        template.update("DELETE FROM user WHERE user_id = ?", persistentObject.getId());
    }

    ParameterizedRowMapper<User> getMapper()
    {
        return MAPPER;
    }
}

Everything is wired together using Spring beans. This is where I define, that I want to use an in-memory database. All standard Spring stuff so far, no magic involved.




<?xml version="1.0" encoding="UTF-8"?>
<beans
        xmlns="http://www.springframework.org/schema/beans"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:aop="http://www.springframework.org/schema/aop"
        xmlns:tx="http://www.springframework.org/schema/tx"
        xsi:schemaLocation="
   http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
   http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd
   http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd">

    <!-- HSQL DS -->
    <bean id="dataSource" class="org.springbyexample.jdbc.datasource.InitializingBasicDataSource" destroy-method="close">
        <property name="driverClassName" value="org.hsqldb.jdbcDriver"/>
        <property name="url" value="jdbc:hsqldb:mem:."/>
        <property name="username" value="sa"/>
        <property name="password" value=""/>
        <property name="sqlScriptProcessor">
            <bean class="org.springbyexample.jdbc.core.SqlScriptProcessor">
                <property name="sqlScripts">
                    <list>
                        <value>classpath:/db-create.sql</value>
                    </list>
                </property>
            </bean>
        </property>
    </bean>

    <!-- DAO base -->
    <bean id="daoWithDataSource" abstract="true">
        <property name="dataSource" ref="dataSource" />
        <property name="identityQuery" value="CALL IDENTITY();" /> <!-- HSQL specific -->
    </bean>
    
    <!-- JDBC DAO's -->
    <bean id="userDao" class="com.fernuni.db.jdbc.UserJdbcDao" parent="daoWithDataSource" />
    <bean id="itemDao" class="com.fernuni.db.jdbc.ItemJdbcDao" parent="daoWithDataSource" />
    <bean id="tagDao" class="com.fernuni.db.jdbc.TagJdbcDao" parent="daoWithDataSource" />
    <bean id="taggedItemDao" class="com.fernuni.db.jdbc.TaggedItemJdbcDao" parent="daoWithDataSource" />

</beans>

Also note that I used some special DataSource which will always execute a database initializer script. To use this InitializingBasicDataSource, you have to reference the SpringByExample library. In Maven powered projects you would add this to your POM:




    <repositories>
        <repository>
            <id>springbyexample.org</id>
            <name>Spring by Example</name>
            <url>http://www.springbyexample.org/maven/repo</url>
        </repository>
    </repositories>

    <dependency>
        <groupId>org.springbyexample</groupId>
        <artifactId>spring-by-example-jdbc</artifactId>
        <version>1.0.3</version>
    </dependency>

Finally I have created a single page using Apache Wicket which will visualize the TagCloud and also contain form fields to create and delete Users, Items and Tags. The page class is called HomePage and contains references to all DAO beans. Later on, it will replace the DAO beans with the ones I have to use in a Google App Engine environment. Here is my HomePage. I have removed a lot of code for better readability as the class is complex. Get the source code to have a look at the full implementation.




public class HomePage extends WebPage
{
    private static final long serialVersionUID = 1L;

    private String m_newUserName = "";
    private String m_newItemName = "";

    private String m_newTagName = "";
    private User m_taggingUser = new User();
    private Item m_taggedItem = new Item();

    @SpringBean
    private UserDao m_userDao;

    @SpringBean
    private ItemDao m_itemDao;

    @SpringBean
    private TagDao m_tagDao;

    @SpringBean
    private TaggedItemDao m_taggedItemDao;

    public HomePage(final PageParameters parameters)
    {
        addUserFields();
        addItemFields();
        addTagFields();

        displayTagCloud();

        ...
    }

    private void displayTagCloud()
    {
        ...
    }


    private void addTagFields()
    {
        final Form tagForm = new Form("newTags");

        final RequiredTextField newTagName = new RequiredTextField("newTagName", new PropertyModel(this, "newTagName"));
        tagForm.add(newTagName);

        final IModel userChoices = new LoadableDetachableModel()
        {
            protected Object load()
            {
                return m_userDao.readAll();
            }
        };

        final IChoiceRenderer userChoiceRenderer = new IChoiceRenderer()
        {
            public Object getDisplayValue(Object object)
            {
                final User user = (User) object;
                return user.getName();
            }

            public String getIdValue(Object object, int index)
            {
                final User user = (User) object;
                return user.getId() + "";
            }
        };

        final ListChoice userListChoices = new ListChoice("taggingUser", new PropertyModel(this, "taggingUser"), userChoices, userChoiceRenderer);
        userListChoices.setRequired(true);
        tagForm.add(userListChoices);

        final IModel itemChoices = new LoadableDetachableModel()
        {
            protected Object load()
            {
                return m_itemDao.readAll();
            }
        };

        final IChoiceRenderer itemChoiceRenderer = new IChoiceRenderer()
        {
            public Object getDisplayValue(Object object)
            {
                final Item item = (Item) object;
                return item.getName();
            }

            public String getIdValue(Object object, int index)
            {
                final Item item = (Item) object;
                return item.getId() + "";
            }
        };

        final ListChoice itemListChoices = new ListChoice("taggedItem", new PropertyModel(this, "taggedItem"), itemChoices, itemChoiceRenderer);
        itemListChoices.setRequired(true);
        tagForm.add(itemListChoices);

        final Button saveNewTagButton = new Button("saveNewTagButton")
        {
         @Override
         public void onSubmit()
         {
                super.onSubmit();

                Tag existingTag = m_tagDao.readByText(m_newTagName);
                if (existingTag == null)
                {
                    final Tag newTag = new Tag();
                    newTag.setName(m_newTagName);
                    m_tagDao.create(newTag);

                    existingTag = m_tagDao.readByText(m_newTagName);
                    assert existingTag != null;
                }

                final Long taggingUserId = m_taggingUser.getId();
                final Long taggedItemId = m_taggedItem.getId();

                final TaggedItem taggedItem = new TaggedItem();
                taggedItem.setItemId(taggedItemId);
                taggedItem.setUserId(taggingUserId);
                taggedItem.setTagId(existingTag.getId());

                m_taggedItemDao.create(taggedItem);
         }
        };
        tagForm.add(saveNewTagButton);

        add(tagForm);

        // Existing Tags
        final Form existingTagsForm = new Form("existingTagsForm");

        final IModel taggedItemsModel = new LoadableDetachableModel()
        {
            protected Object load()
            {
                return m_taggedItemDao.readAll();
            }
        };

        final ListView existingTaggedItems = new ListView("existingTaggedItems", taggedItemsModel)
        {
            protected void populateItem(ListItem item)
            {
                final TaggedItem taggedItem = (TaggedItem) item.getModelObject();

                final User taggingUser = m_userDao.read(taggedItem.getUserId());
                final Item itemContainingTag = m_itemDao.read(taggedItem.getItemId());
                final Tag tag = m_tagDao.read(taggedItem.getTagId());

                final Label taggedText = new Label("taggedText", new PropertyModel(tag, "name"));
                item.add(taggedText);
                final Label taggedBy = new Label("taggedBy", new PropertyModel(taggingUser, "name"));
                item.add(taggedBy);
                final Label taggedAt = new Label("taggedAt", new PropertyModel(itemContainingTag, "name"));
                item.add(taggedAt);

                final Button deleteTaggedItem = new Button("deleteTaggedItem")
                {
                    @Override
                    public void onSubmit()
                    {
                        super.onSubmit();

                        m_taggedItemDao.delete(taggedItem);
                    }
                };
                item.add(deleteTaggedItem);

            }
        };
        existingTagsForm.add(existingTaggedItems);

        add(existingTagsForm);
    }


    ...

}

So far so good. Lets now Google App Enginefy this application. First of all, since this application uses Apache Wicket, you have to change some settings regarding Threads. Google Appengine does not support Threads today. Step 1: run the Wicket application in deployment mode to disable a thread checking for modifications in the background. Step 2: override the method newSessionStore() in your WebApplication class like this:




    @Override
    protected ISessionStore newSessionStore()
    {
        return new HttpSessionStore(this);
    }

This will eliminate another thread that is used by the default DiskPageStore class. Step 3: enable sessions in Google App Engine as they are turned off by default. Wicket uses Http Session heavily. I got all this from Alastair Maw’s blog, thanks a lot!

Now you have to add all the libraries that come with the Google App Engine SDK to your project. In my case the project is based on Maven, so I added the following to my pom.xml.




    <repositories>
        <repository>
            <id>appengine</id>
            <name>Google App Engine Libraries</name>
            <url>http://www.mvnsearch.org/maven2</url>
        </repository>
        <repository>
            <id>datanucleus</id>
            <name>Datanucleus Libraries</name>
            <url>http://www.datanucleus.org/downloads/maven2</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>jdo2-api</artifactId>
            <version>2.3-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>datanucleus-appengine</artifactId>
            <version>1.0.1.final</version>
        </dependency>

        <dependency>
            <groupId>org.datanucleus</groupId>
            <artifactId>datanucleus-core</artifactId>
            <version>${datanucleus.version}</version>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>org.datanucleus</groupId>
            <artifactId>datanucleus-jpa</artifactId>
            <version>${datanucleus.version}</version>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-api-1.0-sdk</artifactId>
            <version>1.2.1</version>
        </dependency>
     </dependencies>

     <properties>
        <datanucleus.version>1.1.0</datanucleus.version>
     </properties>

Unfortunately some dependencies cannot be resolved, even with the extra repositories I added manually. You have to add these jar file manually to your local Maven repository. I will not write about it here but check Dan Walmsleys Blog or Shalins Blog, they explain what you need to do.

Alright, now that we have all set up to Google App Enginefly our Java application - let's start. The App Engine SDK ships with a Servlet Container which you can start and test your application in an App Engine like environment. Download and unzip the SDK, then go into the bin folder. To enable remote debugging in the appengine environment, open the startup script (dev_appserver.sh in Linux) and make it look like this:




#!/bin/bash

# Launches the development AppServer

[ -z "${DEBUG}" ] || set -x  # trace if $DEBUG env. var. is non-zero

SDK_BIN=`dirname $0 | sed -e "s#^\\([^/]\\)#${PWD}/\\1#"` # sed makes absolute

SDK_LIB=$SDK_BIN/../lib

SDK_CONFIG=$SDK_BIN/../config/sdk

java -ea -cp "$SDK_LIB/appengine-tools-api.jar" \

  com.google.appengine.tools.KickStart \

  --jvm_flag=-Xdebug \

  --jvm_flag=-Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n \

  com.google.appengine.tools.development.DevAppServerMain $*

Next time you start your local appengine, you will be able to start a remote debugger on port 8000. Nice!

So at this point, the application would run on Google App Engine without problems. However, the data is not persistent and will be gone every time you restart. So let's use a real database. Each App Enginefied application can access something what Google calls a datastore. The datastore may be accessed using either JDO or JPA. Neither plain JDBC nor O/R mappers like Hibernate or Toplink will work at this stage. Appengine uses something called datanucleus for datastore access, which I think is another abstraction for different ways of persistence. So datanucleus will process JDO or JPA instrumented classes and do some persistence magic.

I decided to use JDO to access the appengine datastore from my application. In the first step let's add a PersistenceManagerFactory and some new DAO beans to the current Spring configuration.




    <bean id="persistenceManagerFactory" class="org.springframework.orm.jdo.LocalPersistenceManagerFactoryBean">
        <property name="jdoProperties">
            <props>
                <prop key="javax.jdo.PersistenceManagerFactoryClass">
                    org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory
                </prop>
                <prop key="javax.jdo.option.ConnectionURL">appengine</prop>
                <prop key="javax.jdo.option.NontransactionalRead">true</prop>
                <prop key="javax.jdo.option.NontransactionalWrite">true</prop>
                <prop key="javax.jdo.option.RetainValues">true</prop>
                <prop key="datanucleus.appengine.autoCreateDatastoreTxns">true</prop>
                <prop key="datanucleus.DetachOnClose">true</prop>
            </props>
        </property>
    </bean>

    <bean id="daoWithPersistenceManagerFactory" abstract="true">
        <property name="persistenceManagerFactory" ref="persistenceManagerFactory" />
    </bean>

    <bean id="userDao" class="com.fernuni.db.jdo.UserJdoDao" parent="daoWithPersistenceManagerFactory" />
    <bean id="itemDao" class="com.fernuni.db.jdo.ItemJdoDao" parent="daoWithPersistenceManagerFactory" />
    <bean id="tagDao" class="com.fernuni.db.jdo.TagJdoDao" parent="daoWithPersistenceManagerFactory" />
    <bean id="taggedItemDao" class="com.fernuni.db.jdo.TaggedItemJdoDao" parent="daoWithPersistenceManagerFactory" />

Next I wrote a set of new DAO classes based on the JdoTemplate from Spring. When compared to JDBC, it is the AbstractJdoDao this time, which contains most code. The concreted DAO classes like UserJdoDao are rather small and reuse almost everything in the AbstractJdoDao.




public abstract class AbstractJdoDao<T extends Entity>
{
    private JdoTemplate m_jdoTemplate;

    public JdoTemplate getJdoTemplate()
    {
        return m_jdoTemplate;
    }

    public void setPersistenceManagerFactory(final PersistenceManagerFactory persistenceManagerFactory)
    {
        m_jdoTemplate = new JdoTemplate(persistenceManagerFactory);
    }

    public T readEntityByName(final String name, final Class<T> clazz)
    {
        final JdoTemplate jdoTemplate = getJdoTemplate();
        final Collection found = jdoTemplate.find(
            clazz, "m_name == value", "String value", new Object[] {name}
        );
        return (found != null && !found.isEmpty()) ? (T) found.toArray()[0] : null;
    }

    public Long createEntity(final T item)
    {
        final JdoTemplate jdoTemplate = getJdoTemplate();
        final T persistentItem = (T) jdoTemplate.makePersistent(item);
        return persistentItem.getId();
    }

    public T readEntity(final Long id, final Class<T> clazz)
    {
        final JdoTemplate jdoTemplate = getJdoTemplate();
        return (T) jdoTemplate.getObjectById(clazz, id);
    }

    public List<T> readAllEntities(final Class<T> clazz)
    {
        final JdoTemplate jdoTemplate = getJdoTemplate();
        return new ArrayList<T>(jdoTemplate.find(clazz));
    }

    public void updateEntity(final T transientObject)
    {
        final JdoTemplate jdoTemplate = getJdoTemplate();
        jdoTemplate.refresh(transientObject);
    }

    public void deleteEntity(final T persistentObject)
    {
        final JdoTemplate jdoTemplate = getJdoTemplate();
        jdoTemplate.deletePersistent(persistentObject);
    }
}

public class UserJdoDao extends AbstractJdoDao<User> implements UserDao
{
    public User readByName(final String name)
    {
        return readEntityByName(name, User.class);
    }

    public Long create(final User user)
    {
        return createEntity(user);
    }

    public User read(final Long id)
    {
        return readEntity(id, User.class);
    }

    public List<User> readAll()
    {
        return readAllEntities(User.class);
    }

    public void update(final User transientObject)
    {
        updateEntity(transientObject);
    }

    public void delete(final User persistentObject)
    {
        deleteEntity(persistentObject);
    }
}

Using JDO requires that the bytecode in your domain model classes is instrumented with the persistence information. So first of all, lets put down this information. To be least intrusive, I decided to provide separate .jdo files instead of using annotations. Here is the package.jdo file which I put in the package along with my domain model classes.




<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jdo PUBLIC
    "-//Sun Microsystems, Inc.//DTD Java Data Objects Metadata 2.0//EN"
    "http://java.sun.com/dtd/jdo_2_0.dtd">
<jdo>
    <package name="com.fernuni.domain">
        <class name="Entity" detachable="true" identity-type="application">
            <inheritance strategy="complete-table"/>
            <field name="id" primary-key="true" value-strategy="identity">
                <column name="ENTITY_ID"/>
            </field>
            <field name="name">
                <column name="ENTITY_NAME"/>
            </field>
        </class>
        <class name="User" detachable="true" identity-type="application">

        </class>
        <class name="Item" detachable="true" identity-type="application">

        </class>
        <class name="Tag" detachable="true" identity-type="application">

        </class>
        <class name="TaggedItem" detachable="true" identity-type="application">
            <field name="m_userId">
                <column name="TAGGED_USER_ID"/>
            </field>
            <field name="m_itemId">
                <column name="TAGGED_ITEM_ID"/>
            </field>
            <field name="m_tagId">
                <column name="TAGGED_TAG_ID"/>
            </field>
        </class>
    </package>
</jdo>

Unfortunately datanucleus does not support primary keys of type Integer and all Id's in my JDBC based applications were Integers. So I had to change from Integer to Long to get it working. Quite intrusive but something I can live with. As a last step, Maven needs to instrument the domain classes using the JDO instructions from my package.jdo file. I added the maven-datanucleus-plugin in my Maven build process.




    <build>      
        <plugins>
            <plugin>
                <groupId>org.datanucleus</groupId>
                <artifactId>maven-datanucleus-plugin</artifactId>
                <version>${datanucleus.version}</version>
                <configuration>
                    <mappingIncludes>**/*.class</mappingIncludes>
                    <verbose>true</verbose>
                    <enhancerName>ASM</enhancerName>
                    <api>JDO</api>
                </configuration>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>enhance</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

Unfortunately it turned out the Google App Engine has huge difficulties when you have a hierarchy of persistence capable subclasses. In my case each User, Item, Tag and TaggedItem all derive from Entity. The Entity class contains the persistent fields id and name. For some reason in the current version of the datanucleus-appengine plugin, the fields from a persistence capable superclass are not accessible within the subclass. In other words, you cannot use inheritance in your domain model if you want to use JDO and Google App Engine. I have started two threads in the App Engine community and Max Ross from Google promised that Inheritance will receive a higher priority in upcoming versions of the datanucleus-appengine plugin.

Let's sum this up. Google App Engine is a really great thing Google came up with. It is also a Google example that Tim O'Reilly is right about the perpetual Beta, which is what I think we see with appengine right now. It has been put into public earlier than you would usually expect it from other Google products, probably to leverage the “power of the community” in making it better and more stable. On the other hand, I do not think that Google App Engine has reached a state where it can be used for real life enterprise applications.

Looking forward to JavaOne

Only one month left to this years JavaOne in San Francisco. About 8 to 10 Java geeks from my company will be heading to the US for the conference. We are leaving Stockholm May 29 to fly to San Francisco via Frankfurt. I am really happy I was selected to go to Java One this year. It is a great opportunity to get some fresh ideas from the Java Community. Also I am meeting a good friend which I haven't visited since 2002.

Today I received an email from JavaOne team that the schedule builder was ready and that it was recommended to put together a list of sessions I would like to attend. Unlike other software development conferences like OOPSLA, where everything is rather small, JavaOne is supposed to be quite crowdy and rooms fill up fast. So I spent some minutes today to build up my preliminary Java One session schedule.

Compared to OOPSLA 2008, the Java One is very Sun focused. A lot of talks are about Sun libraries, applications and tools. Since we are not using Glassfish application server or Java Server Faces, I decided to skip on these talks. The this years big hype seems to be Cloud Computing. There are tons of sessions having the word “cloud” in the title. Another big topic is REST and web services in general. Not sooo many talks are about dynamic languages this year I think. Unfortunately I skipped all the great Scala sessions at last years OOPSLA, as I did not knew what Scala was back then. There are now 3 talks about Scala and the Scala web framework Lift which I will be attending to. Unfortunately there are no sessions about Apache Wicket, my current favorite web framework.

Some of the session I really look forward to:

(TS-6802) Hadoop, a Highly Scalable, Distributed File/Data Processing System
Implemented in Java™ Technology

This is about an open source implementation of Google's Map Reduce algorithm, which comes in handy if you have to deal with gigantic sets of data. This would be the case if you harvest user behavior on the web to use it for Collective Intelligence, an area which fascinates me very much right now.

(BOF-3820) Lift: The Best Way to Create Rich Internet Applications with Scala

As Groovy went more or less by me, I decided to work myself in into Scala. There was a series about Scala in the German Java Magazin that woke my interested. Lift is the Scala webframework that supposingly “steals” the best features from other Java based web frameworks. They borrow the strict separation of layout and business logic from Apache Wicket, which is one of my favorite libraries for web development today.

(BOF-5105) Hudson Community Meet-Up

Writing Hudson plugins is fun. I wrote a plugin for Testability Explorer last Christmas and I am hoping to meet Kohsuke and some nice people I had mail contact with via the Hudson community like Ulli Hafner.

Finally some sessions focusing on scaleability and performance which I am hoping to put into practice in our company as we deal with a gaming application that need to be high performant.

(TS-4407) Best Practices for Large-Scale Web Sites: Lessons from eBay

(TS-4696) JDBC? We Don't Need No Stinkin’ JDBC: How LinkedIn Scaled with memcached, SOA, and a Bit of SQL

(TS-4588) Where’s My I/O: Some Insights into I/O Profiling and Debugging

My agenda also has some general purpose topics like Ajax Push and Comet as well as Central Event Processing (CEP) which might be useful in the future. So we will see about these.

Collective Intelligence in Action

I just finished one of the best books I have read in a long time. It is titled “Collective Intelligence in Action” from Manning Publications and is written by Satnam Alag. We were reading the book in a book circle, which my company runs twice a year.

Collective Intelligence is all about making web applications better by using intelligence gathered from user interactions and behavior. There are a lot of very successful web 2.0 applications out there, which harvest user intelligence and then use this data to improve the user experience. I liked this book because it was very different from other Java related books I have read. It is not focused on one particular technology or framework. It is not too code focused, even though quite mathematical sometimes. While I read it, I came up with all these cool new ideas how I could make my own websites better. I will start to implement a little bit of collective intelligence in the next few months. I can really recommend this book if you want to be inspired about some new advanced features for your web applications.

The book starts off by giving a brief overview about web 2.0 applications and collective intelligence. Then the author explains how users and items can be mapped to each other using either content based mapping or collaboration based mapping. It gets a bit mathematical here with some dot product computations and the cosine based similarity. Chapter 3 is all about tags, tagging and how to leverage tags in a web application. The chapters 4, 5 and 6 introduce some nifty tools that can be used to write a api based blog searcher and a web crawler. I have myself done this stuff when I build my web applications, so it was great to see alternative approaches. In the second and third part of the book the author introduces the different algorithms to make predictions or cluster users and items. You will learn about classification, regression, clustering etc. It is getting very theoretical and sometimes a bit hard to follow but very, very interesting. Since all the code examples in the book are in Java, Alag uses WEKA and JDM (Java Data Mining) to implement the algorithms.

The reader will also learn a bit Lucene, the popular text indexer and searching framework. Lucene is being used to do content based learning. The Lucene parts are pretty basic though and should be familiar to those of you, who have worked with Lucene before. The book ends with a practical example of how to build a recommendation engine similar to Amazon.

Who is the book for? I can recommend it to experienced Java developers who would like to try out some new things in their own web applications. There are tons of great ideas in this book. Having a website with a couple of hundred visitors per days is a plus if you practically want to unleash some collective intelligence.