My favorite bug in 2011

Well, the year is still young but I thought to post my favorite bug in 2011 so far. It was a Spring related piece of code which I had added to a legacy application. Can you spot the problem?


<bean id="deactivationDateFormat" class="java.text.SimpleDateFormat" scope="prototype">
<constructor-arg value="yyyy-MM-dd HH:mm:ss" />
</bean>

<bean id="gamePeriodStatsService" scope="prototype"
class="gameperiodstats.impl.GamePeriodStatsServiceImpl">
<property name="dao" ref="gamePeriodStatsDAO"/>
<property name="globalPropertyService" ref="globalPropertyService"/>
<property name="gamePeriodStatsUpdater" ref="gamePeriodStatsUpdater"/>
<property name="gpsDeactivationDate">
<bean factory-bean="deactivationDateFormat" factory-method="parse">
<constructor-arg value="${gps.deactivation.date}" />
</bean>
</property>
</bean>


The change I made to the existing application was to add the gpsDeactivationDate field into the GamePeriodStatsServiceImpl class. I decided to make this a field of type Date and use the SimpleDateFormat class inside the Spring configuration to parse a String value while the ApplicationContext is initialized. The String value has to be in the format yyyy-MM-dd HH:mm:ss and is read from a Java properties file (not shown). The property holding the date as String is called gps.deactivation.date.

We released the code into one of our integration test systems. A few days later, I got assigned a bug in Jira from one of the testers. He copied me a stacktrace from a logfile. As you can see below, there is a lot of noise in the Exception logging from JBoss and Spring. The information you are interested in, is on the last line.


2010-12-31 17:00:34,565 ERROR []
[foo.util.jboss.log.LogInterceptor]
RuntimeException in method: public abstract void foo.util.cron.ejb.CronInvokerLocal.invokeSpring
(java.lang.String,java.lang.String,java.lang.String) throws java.lang.ClassNotFoundException,java.lang.NoSuchMethodException,
java.lang.reflect.InvocationTargetException,java.lang.IllegalAccessException:
[] org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'networkStatsService' defined in class path resource [service/addons/networkstats.applicationContext.xml]: Cannot resolve reference to bean 'gamePeriodStatsService' while setting bean property 'gamePeriodStatsService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gamePeriodStatsService' defined in class path resource [service/addons/gameperiodstats.applicationContext.xml]: Cannot create inner bean 'deactivationDateFormat$created#335fbef1' while setting bean property 'gpsDeactivationDate'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'deactivationDateFormat$created#335fbef1' defined in class path resource [service/addons/gameperiodstats.applicationContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public java.util.Date java.text.DateFormat.parse(java.lang.String) throws java.text.ParseException] threw exception; nested exception is java.lang.NumberFormatException: multiple points


The system verification department was doing heavy-load testing using bots on several machines. The error showed up only two or three times during the entire day. I started googling for the java.lang.NumberFormatException: multiple points. It turned out it is a weird way of saying you are using SimpleDateFormat from multiple threads on a shared instance. There are tons of articles about SimpleDateFormat not being thread-safe, and I knew about it. I was a bit surprised. The way I had configured my Spring context invokes SimpleDateFormat.parse exactly one time only, at creation of the Application Context.

The problem however, is in the gamePeriodStatsService bean, which was set to scope="prototype" for reasons I don't know. Using this scope would actually always create a new instance of the GamePeriodStatsServiceImpl class every time someone uses applicationContext.getBean inside the code. I figured out that there were a lot of places in the code where getBean was used. Every time a new gamePeriodStatsService bean was created that way, it would use my shared deactivationDateFormat bean - invoking the parse method in the SimpleDateFormat class. The deactivationDateFormat bean is shared because I did not put it in the prototype scope - like you should do. Only stateful beans should use the prototype scope.

We decided to do a semi-lame fix and put the deactivationDateFormat bean also in prototype scope. The proper solution would probably be to remove the prototype scope from all other beans where it was needed. However, I was not brave enough to change this in a legacy systems having 20% test coverage :)

Keep-Alive query in HSQLDB

Today I was in the need for a good keep-alive statement that works with HSQLDB. For all the code running against MySQL we usually use:


SELECT 1


but the statement is incompatible with HSQLDB. The problem came up in one of our unit-tests, where we often use a in-memory database instead of depending on a externally running MySQL instance.

For me, a good keep-alive statement is one, that is not depending on a user defined schema and table structure. I came up with this one for HSQLDB:


SELECT * FROM INFORMATION_SCHEMA.SYSTEM_TABLES


which should work in every vanilla HSQLDB installation. I haven't looked thoroughly into right and permission handling but I kind of assume that every user can at least read from this schema. If you run HSQLDB in memory, you are most likely using the sa user anyways.

In our case it is semi-useful, to programmatically verify that the in-memory database is up during unit-tests. It will make more sense in production code against MySQL. You should also be able to use this as the validationQuery in a org.apache.commons.dbcp.BasicDataSource or as keep-alive statement for Quartz.

Riak, Protocol Buffers and Java in the Mix

Today I was using Riak for the first time, in order to compare it with Cassandra. I set up a Maven quickstart project to use Riak from Java using Google Protocol Buffers. The Riak clients come in two flavours - either using HTTP or Protobuf. The latter supposed to be a bit faster. Well, no surprise. HTTP is top level in the OSI model, so there is some overhead involved. However, if performance is not your biggest concern, I would almost always go for HTTP because it is easier to use in client applications. Anyway, let's continue with the Protocol Buffers version anyways (some people claim it is 10x faster).

If you are running Ubuntu like me, follow the install instructions using the binary packages. This is actually installing Riak Search, which includes Riak core. What the guide is not telling you, is how to start the database server afterwards. Well not too hard:


/etc/init.d/riaksearch start


To test if it is up and running execute:


curl http://localhost:8098/stats

curl -v http://127.0.0.1:8098/riak/test


Continue to set up a vanilla Java project using Maven:


mvn archetype:generate


Pick the maven-archetype-quickstart type (15) and enter the remaining stuff. Next make sure, that you have the compiler for Protocol Buffers installed:


protoc --version


If it is not installed run sudo apt-get install protobuf-compiler.

Next download the riakclient.proto file, which we will use to generate us a Java version of the Riak client. Before you do this, let's create 2 new folders which we need later. In the Maven project structure create src/main/proto. This is the default location for all your .proto files. Furthermore create src/main/java-gen which we will use to output generated Protocol Buffers files. Now download riakclient.proto and put it into src/main/proto.

The protobuf-compiler need to be told in which Java package you want to generate your source code files. Open riakclient.proto and add the package instructions before the first Message block:


...

** 24 - RpbMapRedResp{1,}
**
*/
option java_package = "my.package.riak.client";

// Error response - may be generated for any Req
message RpbErrorResp {
required bytes errmsg = 1;
required uint32 errcode = 2;
}

...


Ideally, you want to generate all source code based on the Protocol Buffers files every time you build your project. This is where the maven-protoc-plugin comes in handy. In order to use it, the pom.xml file needs to be tweaked. Add the plugin repository hosting the plugin:


<pluginRepositories>
<pluginRepository>
<id>dtrott</id>
<url>http://maven.davidtrott.com/repository</url>
</pluginRepository>
</pluginRepositories>


then hook in the plugin when running compile:


<plugin>
<groupId>com.google.protobuf.tools</groupId>
<artifactId>maven-protoc-plugin</artifactId>
<configuration>
<outputDirectory>src/main/java-gen/</outputDirectory>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
<phase>generate-sources</phase>
</execution>
</executions>
</plugin>


With this configuration, the maven-protoc-plugin will use the protoc compiler installed in your OS. This implies, that you use the same dependency version of Protocol Buffers in your project. So run protoc --version from the command line and set this version in your Maven dependencies section. If you don't do this, you might see errors like this:


Riakclient.java:[118,51] boolean cannot be dereferenced


Also note the outputDirectory defined in the plugin above. Unfortunately the maven-protoc-plugin cleans this directory before generating sources. This is why you cannot generate the protobuf sources into the same folder as your regular Java project files or they will be deleted.

Finally we are ready to write some Java code to run against Riak. Here is an example of a TestNG unit test that invokes the List Buckets operation:


/**
* List all of the buckets using protocol buffers. You need to have Riak
* up and running on port 8087.
*
* @author reiks, Jan 5, 2011
*/
@Test(groups = "all", sequential = true)
public class ListBucketsTest {

public void testListBuckets() throws IOException {
final Socket socket = new Socket("localhost", 8087);
DataOutputStream dout = null;
DataInputStream din = null;

try {
dout = new DataOutputStream(
new BufferedOutputStream(
socket.getOutputStream(), 1024 * 200
)
);

din = new DataInputStream(
new BufferedInputStream(
socket.getInputStream(), 1024 * 200
)
);

dout.writeInt(1);
dout.write(15); // 15 - RpbListBucketsReq
dout.flush();

final byte[] bytes = getData(16, din); // 16 - RpbListBucketsResp
assertNotNull(bytes);
assertTrue(bytes.length > 0);

System.out.println(Arrays.toString(bytes));

} finally {
IOUtils.closeQuietly(dout);
IOUtils.closeQuietly(din);
socket.close();
}
}

byte[] getData(final int expectedCode, final DataInputStream din) throws IOException {
int len = din.readInt();
int returnCode = din.read();

byte[] data = null;
if (len > 1) {
data = new byte[len - 1];
din.readFully(data);
}

if (expectedCode != returnCode) {
throw new IOException("Unexpected (" + returnCode + "), expected " + expectedCode);
}

return data;
}
}


Double check that you run against port 8087 which is for Protocol Buffers communication. I was using port 8098 from the curl examples above for a while and got totally strange results until I figured that this is actually for HTTP communication. Stupid me!

As you can see, this code is rather low-level. Not only do you need to know the different Message codes, which you can find in the riakclient.proto file, you also need to get the java.io stuff right. In a perfect world, this code should be hidden behind a facade. This is exactly what the riak-java-pb-client library is doing. A similar facade library for Cassandra is Hector. It let's you write application code against Riak in a convinient way and hides the low-level details. Unfortunately at this stage, the riak-java-pb-client library was only available as source files in github and they don't provide a way to build it out of the box. I have sent them a version of their code which is restructured and can be build with Maven. With a bit of luck, you can git clone and build riak-java-pb-client in a few days.