My favorite bug in 2011

Well, the year is still young but I thought to post my favorite bug in 2011 so far. It was a Spring related piece of code which I had added to a legacy application. Can you spot the problem?


<bean id="deactivationDateFormat" class="java.text.SimpleDateFormat" scope="prototype">
<constructor-arg value="yyyy-MM-dd HH:mm:ss" />
</bean>

<bean id="gamePeriodStatsService" scope="prototype"
class="gameperiodstats.impl.GamePeriodStatsServiceImpl">
<property name="dao" ref="gamePeriodStatsDAO"/>
<property name="globalPropertyService" ref="globalPropertyService"/>
<property name="gamePeriodStatsUpdater" ref="gamePeriodStatsUpdater"/>
<property name="gpsDeactivationDate">
<bean factory-bean="deactivationDateFormat" factory-method="parse">
<constructor-arg value="${gps.deactivation.date}" />
</bean>
</property>
</bean>


The change I made to the existing application was to add the gpsDeactivationDate field into the GamePeriodStatsServiceImpl class. I decided to make this a field of type Date and use the SimpleDateFormat class inside the Spring configuration to parse a String value while the ApplicationContext is initialized. The String value has to be in the format yyyy-MM-dd HH:mm:ss and is read from a Java properties file (not shown). The property holding the date as String is called gps.deactivation.date.

We released the code into one of our integration test systems. A few days later, I got assigned a bug in Jira from one of the testers. He copied me a stacktrace from a logfile. As you can see below, there is a lot of noise in the Exception logging from JBoss and Spring. The information you are interested in, is on the last line.


2010-12-31 17:00:34,565 ERROR []
[foo.util.jboss.log.LogInterceptor]
RuntimeException in method: public abstract void foo.util.cron.ejb.CronInvokerLocal.invokeSpring
(java.lang.String,java.lang.String,java.lang.String) throws java.lang.ClassNotFoundException,java.lang.NoSuchMethodException,
java.lang.reflect.InvocationTargetException,java.lang.IllegalAccessException:
[] org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'networkStatsService' defined in class path resource [service/addons/networkstats.applicationContext.xml]: Cannot resolve reference to bean 'gamePeriodStatsService' while setting bean property 'gamePeriodStatsService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gamePeriodStatsService' defined in class path resource [service/addons/gameperiodstats.applicationContext.xml]: Cannot create inner bean 'deactivationDateFormat$created#335fbef1' while setting bean property 'gpsDeactivationDate'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'deactivationDateFormat$created#335fbef1' defined in class path resource [service/addons/gameperiodstats.applicationContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public java.util.Date java.text.DateFormat.parse(java.lang.String) throws java.text.ParseException] threw exception; nested exception is java.lang.NumberFormatException: multiple points


The system verification department was doing heavy-load testing using bots on several machines. The error showed up only two or three times during the entire day. I started googling for the java.lang.NumberFormatException: multiple points. It turned out it is a weird way of saying you are using SimpleDateFormat from multiple threads on a shared instance. There are tons of articles about SimpleDateFormat not being thread-safe, and I knew about it. I was a bit surprised. The way I had configured my Spring context invokes SimpleDateFormat.parse exactly one time only, at creation of the Application Context.

The problem however, is in the gamePeriodStatsService bean, which was set to scope="prototype" for reasons I don't know. Using this scope would actually always create a new instance of the GamePeriodStatsServiceImpl class every time someone uses applicationContext.getBean inside the code. I figured out that there were a lot of places in the code where getBean was used. Every time a new gamePeriodStatsService bean was created that way, it would use my shared deactivationDateFormat bean - invoking the parse method in the SimpleDateFormat class. The deactivationDateFormat bean is shared because I did not put it in the prototype scope - like you should do. Only stateful beans should use the prototype scope.

We decided to do a semi-lame fix and put the deactivationDateFormat bean also in prototype scope. The proper solution would probably be to remove the prototype scope from all other beans where it was needed. However, I was not brave enough to change this in a legacy systems having 20% test coverage :)

Keep-Alive query in HSQLDB

Today I was in the need for a good keep-alive statement that works with HSQLDB. For all the code running against MySQL we usually use:


SELECT 1


but the statement is incompatible with HSQLDB. The problem came up in one of our unit-tests, where we often use a in-memory database instead of depending on a externally running MySQL instance.

For me, a good keep-alive statement is one, that is not depending on a user defined schema and table structure. I came up with this one for HSQLDB:


SELECT * FROM INFORMATION_SCHEMA.SYSTEM_TABLES


which should work in every vanilla HSQLDB installation. I haven't looked thoroughly into right and permission handling but I kind of assume that every user can at least read from this schema. If you run HSQLDB in memory, you are most likely using the sa user anyways.

In our case it is semi-useful, to programmatically verify that the in-memory database is up during unit-tests. It will make more sense in production code against MySQL. You should also be able to use this as the validationQuery in a org.apache.commons.dbcp.BasicDataSource or as keep-alive statement for Quartz.

Riak, Protocol Buffers and Java in the Mix

Today I was using Riak for the first time, in order to compare it with Cassandra. I set up a Maven quickstart project to use Riak from Java using Google Protocol Buffers. The Riak clients come in two flavours - either using HTTP or Protobuf. The latter supposed to be a bit faster. Well, no surprise. HTTP is top level in the OSI model, so there is some overhead involved. However, if performance is not your biggest concern, I would almost always go for HTTP because it is easier to use in client applications. Anyway, let's continue with the Protocol Buffers version anyways (some people claim it is 10x faster).

If you are running Ubuntu like me, follow the install instructions using the binary packages. This is actually installing Riak Search, which includes Riak core. What the guide is not telling you, is how to start the database server afterwards. Well not too hard:


/etc/init.d/riaksearch start


To test if it is up and running execute:


curl http://localhost:8098/stats

curl -v http://127.0.0.1:8098/riak/test


Continue to set up a vanilla Java project using Maven:


mvn archetype:generate


Pick the maven-archetype-quickstart type (15) and enter the remaining stuff. Next make sure, that you have the compiler for Protocol Buffers installed:


protoc --version


If it is not installed run sudo apt-get install protobuf-compiler.

Next download the riakclient.proto file, which we will use to generate us a Java version of the Riak client. Before you do this, let's create 2 new folders which we need later. In the Maven project structure create src/main/proto. This is the default location for all your .proto files. Furthermore create src/main/java-gen which we will use to output generated Protocol Buffers files. Now download riakclient.proto and put it into src/main/proto.

The protobuf-compiler need to be told in which Java package you want to generate your source code files. Open riakclient.proto and add the package instructions before the first Message block:


...

** 24 - RpbMapRedResp{1,}
**
*/
option java_package = "my.package.riak.client";

// Error response - may be generated for any Req
message RpbErrorResp {
required bytes errmsg = 1;
required uint32 errcode = 2;
}

...


Ideally, you want to generate all source code based on the Protocol Buffers files every time you build your project. This is where the maven-protoc-plugin comes in handy. In order to use it, the pom.xml file needs to be tweaked. Add the plugin repository hosting the plugin:


<pluginRepositories>
<pluginRepository>
<id>dtrott</id>
<url>http://maven.davidtrott.com/repository</url>
</pluginRepository>
</pluginRepositories>


then hook in the plugin when running compile:


<plugin>
<groupId>com.google.protobuf.tools</groupId>
<artifactId>maven-protoc-plugin</artifactId>
<configuration>
<outputDirectory>src/main/java-gen/</outputDirectory>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
<phase>generate-sources</phase>
</execution>
</executions>
</plugin>


With this configuration, the maven-protoc-plugin will use the protoc compiler installed in your OS. This implies, that you use the same dependency version of Protocol Buffers in your project. So run protoc --version from the command line and set this version in your Maven dependencies section. If you don't do this, you might see errors like this:


Riakclient.java:[118,51] boolean cannot be dereferenced


Also note the outputDirectory defined in the plugin above. Unfortunately the maven-protoc-plugin cleans this directory before generating sources. This is why you cannot generate the protobuf sources into the same folder as your regular Java project files or they will be deleted.

Finally we are ready to write some Java code to run against Riak. Here is an example of a TestNG unit test that invokes the List Buckets operation:


/**
* List all of the buckets using protocol buffers. You need to have Riak
* up and running on port 8087.
*
* @author reiks, Jan 5, 2011
*/
@Test(groups = "all", sequential = true)
public class ListBucketsTest {

public void testListBuckets() throws IOException {
final Socket socket = new Socket("localhost", 8087);
DataOutputStream dout = null;
DataInputStream din = null;

try {
dout = new DataOutputStream(
new BufferedOutputStream(
socket.getOutputStream(), 1024 * 200
)
);

din = new DataInputStream(
new BufferedInputStream(
socket.getInputStream(), 1024 * 200
)
);

dout.writeInt(1);
dout.write(15); // 15 - RpbListBucketsReq
dout.flush();

final byte[] bytes = getData(16, din); // 16 - RpbListBucketsResp
assertNotNull(bytes);
assertTrue(bytes.length > 0);

System.out.println(Arrays.toString(bytes));

} finally {
IOUtils.closeQuietly(dout);
IOUtils.closeQuietly(din);
socket.close();
}
}

byte[] getData(final int expectedCode, final DataInputStream din) throws IOException {
int len = din.readInt();
int returnCode = din.read();

byte[] data = null;
if (len > 1) {
data = new byte[len - 1];
din.readFully(data);
}

if (expectedCode != returnCode) {
throw new IOException("Unexpected (" + returnCode + "), expected " + expectedCode);
}

return data;
}
}


Double check that you run against port 8087 which is for Protocol Buffers communication. I was using port 8098 from the curl examples above for a while and got totally strange results until I figured that this is actually for HTTP communication. Stupid me!

As you can see, this code is rather low-level. Not only do you need to know the different Message codes, which you can find in the riakclient.proto file, you also need to get the java.io stuff right. In a perfect world, this code should be hidden behind a facade. This is exactly what the riak-java-pb-client library is doing. A similar facade library for Cassandra is Hector. It let's you write application code against Riak in a convinient way and hides the low-level details. Unfortunately at this stage, the riak-java-pb-client library was only available as source files in github and they don't provide a way to build it out of the box. I have sent them a version of their code which is restructured and can be build with Maven. With a bit of luck, you can git clone and build riak-java-pb-client in a few days.

Java Example with Cassandra 0.7

Last Friday release candidate 2 of the upcoming Cassandra 0.7 version saw the light of the day. It took a while to get some old Java test code working again using Thrift directly. Let's use this Maven project file:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>javasplitter</groupId>
<artifactId>cassandra-test</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>

<repositories>
<repository>
<id>riptano</id>
<name>Riptano Repository</name>
<url>http://mvn.riptano.com/content/repositories/riptano/</url>
</repository>
</repositories>

<dependencies>
<dependency>
<groupId>org.apache.cassandra</groupId>
<artifactId>apache-cassandra</artifactId>
<version>0.7.0-rc2</version>
</dependency>
<dependency>
<groupId>libthrift</groupId>
<artifactId>libthrift</artifactId>
<version>0.5</version>
</dependency>
</dependencies>
</project>


You want to use the Riptano repository as these guys have all the latest artifacts very fast. Here is a snippet of my new yaml based Cassandra configuration - yammi:


...
keyspaces:
- name: "javasplitter"
replica_placement_strategy: org.apache.cassandra.locator.SimpleStrategy
replication_factor: 1
column_families:
- name: author
compare_with: BytesType


Straightforward, no advanced use cases. Some Java code involving Thrift that works.


TTransport framedTransport = new TFramedTransport(new TSocket("localhost", 9160));
TProtocol framedProtocol = new TBinaryProtocol(framedTransport);

Cassandra.Client client = new Cassandra.Client(framedProtocol);
framedTransport.open();

client.set_keyspace("javasplitter");
String columnFamily = "author";

ColumnParent cp = new ColumnParent(columnFamily);
byte[] userIDKey = "nadine".getBytes();
client.insert(
ByteBuffer.wrap(userIDKey),
cp,
new Column(
ByteBuffer.wrap("lastname".getBytes("UTF-8")),
ByteBuffer.wrap("schatz".getBytes("UTF-8")),
System.currentTimeMillis()
),
ConsistencyLevel.ONE);


One new thing is the use of framed transports. Make sure your cassandra.yaml has this line (which it does by default):


# Frame size for thrift (maximum field length).
# 0 disables TFramedTransport in favor of TSocket. This option
# is deprecated; we strongly recommend using Framed mode.
thrift_framed_transport_size_in_mb: 15


Make really sure you are using the above code against a 0.7 Cassandra installation otherwise you will get a lot of weird errors which are not really self explaining. For instance running the Java code above against Cassandra 0.6.8 would just freeze on the line containing framedTransport.open(). It is still possible to use plain TSocket transport or even to mix. Cassandra.Client has a constructor that takes two TProtocol instances (in and out).

Another thing that has changed is the use of java.nio.ByteBuffer from the Thrift based Cassandra.Client. Everything expects a ByteBuffer now (I think it took byte[] before) so the the code becomes a bit more verbose. Also make sure you are running with libthrift 0.5 (see pom.xml) otherwise you will get errors like:


java.lang.NoSuchMethodError: org.apache.thrift.protocol.TProtocol.writeBinary(Ljava/nio/ByteBuffer;)


I think 0.7 beta 2 used another version of libthrift (959516) so be careful. Anyway, you should probably use Hector from production code anyway. Should work well with 0.7 of Cassandra. I was however interested in Thrift since apparently Cassandra is moving away from Thrift to Avro. No idea yet, why Avro is better. It can't be the performance.

Having Maven create a nice zip File and separate Configuration

A co-worker and I are preparing a presentation about Amazon EC2 for other developers in my company. To show some stuff in action, we decided to write two JMS powered applications. One is sending messages, the other is receiving messages and persisting them into a database. During the presentation we will roll this out onto 3 EC2 nodes. Each application is build using Maven. To have it as convenient as possible, I changed the Maven package build phase to produce a single zip-file. The zip-file can be copied over to the EC2 node, where it is extracted. The zip-file contains one big "uber-jar" (with all the third party dependencies included) and a single properties-file to be able to set host and port for the JMS communication. We use ActiveMQ as JMS vendor in our projects.

Once the big zip-file has been extracted on the EC2 nodes and the properties have been set, you can start the producer and the consumer from the Main class. (Note the little dot infront of .:consumer... - this is needed so that the properties-file is found)

java -cp .:consumer-1.0-SNAPSHOT-final.jar package.MessageReceiver


For the packaging of the big zip-files, I use the Maven Assembly plugin during the package phase. The configuration looks like this:


<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>final</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<unpack>true</unpack>
<scope>runtime</scope>
<useProjectArtifact>false</useProjectArtifact>
</dependencySet>
</dependencySets>
<fileSets>
<fileSet>
<directory>${project.build.outputDirectory}</directory>
<outputDirectory>/</outputDirectory>
<excludes>
<exclude>consumer.properties</exclude>
</excludes>
</fileSet>
</fileSets>
</assembly>
src/main/assembly/jar.xml

This explodes all third party dependency jar files and merges them into one big uber-jar file. The properties-file is excluded from the uber-jar (this name sounds hilarious if you are from Germany by the way).


<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>bin</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>${project.basedir}/src/main/resources</directory>
<outputDirectory/>
<includes>
<include>consumer.properties</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory/>
<includes>
<include>*-final.jar</include>
</includes>
</fileSet>
</fileSets>
</assembly>
src/main/assembly/zip.xml

This creates a zip-archive containing the "uber-jar" and the properties-file. Notice that the "uber-jar" has the suffix of "-final" equal to the id-attribute in the jar.xml file.


<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>package.MessageReceiver</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<descriptors>
<descriptor>src/main/assembly/jar.xml</descriptor>
<descriptor>src/main/assembly/zip.xml</descriptor>
</descriptors>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
pom.xml

This code snippet runs the maven-assembly-plugin during the package phase.

Everything went well when we finished the producer application last week. Today I ran into some weird errors when I worked on the consumer end. Trying to start the MessageReceiver main class gave me the following error:


Caused by: org.xml.sax.SAXParseException: cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'amq:broker'.


I was able to resolve this error with the help of the ActiveMQ XML reference page, only to stumble into the next problem:


Caused by: org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Unable to locate Spring NamespaceHandler for XML schema namespace http://www.springframework.org/schema/context


It did not really jump me at first why this was happening. There a unit tests which load the Spring Context, they run fine. Maven runs the test in the package phase, they passed fine. So this was odd. After doing some research, I found out that the maven-assembly-plugin is responsible for this. Apparently Spring needs the spring.handlers and spring.schemas files to be present in the META-INF directory of the "uber-jar". A lot of other people had already hit the same problem before me. Some of them recommend the use of the maven-shade-plugin with the following setup:


<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.3.1</version>

<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${artifactId}-${version}-final</finalName>
<transformers>
<transformer implementation="
org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>
com.sabre.newgermanrail.dbitool.DbiTool
</mainClass>
</transformer>
<transformer implementation="
org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
<transformer implementation="
org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
<resource>consumer.properties</resource>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
pom.xml

With the help of Transformers, the spring.handlers and spring.schemas files are added. If you are using Spring 3.x, there are many small spring-xyz.jar files and each of these comes with a spring.handlers and spring.schemas file. The maven-shade-plugin will append the content of each of these files using the AppendingTransformer. As you can see in the example above, I am again excluding my properties-file using the DontIncludeResourceTransformer. I decided to keep the zip-archiving part from the maven-assembly-plugin. Maybe this is something the maven-shade-plugin could do for me as well, not sure.

Interviewed at Google Pt.1

Today I want to blog about an on-site interview which I had at Google a while ago. Since I did not sign a non-disclosure agreement and only agreed on not taking photos in their office, I hope it is okay to write about this interview. Going to an on-site interview, is like reaching the next level, if Google is satisfied with your previous telephone interview(s). Due to summer-time and a lot of vacations, I had my on-site interview more than 1,5 months after doing the initial phone interview. This was very good, because it gave me extra time to brush up on some stuff. I read some books about data-structures and algorithms and practiced white-board coding a bit. I coded through a lot of sample questions from software interviews, which I was able to find on the net. However, all the questions during the interview were new to me, so I had to be spontaneous.

The job I was interviewed for, was Software Engineer in Test and the interview was held in Zürich, Usually the candidates are flying to the interview location one day in advance. That did not work for me, because I was already on another flight a day earlier. Instead Google booked two flights on the same day for me. which was OK. I arrived in time and signed in at the reception. As stated, you have to agree on not taking photos and you have to put some label identifying yourself on your shirt. Some minutes later I was picked up by my HR contact. She gave me a short office tour and set me up in one of the interview rooms. You could tell that Google is doing a lot for their employees. The Zürich office had a big fitness room with a personal coach. Free soft drinks everywhere, video gaming rooms, pool tables all that kind of stuff. My interview room was very small, maybe 3 times 3 meters and had a white-board. She handed me my interview schedule and I was surprised that it was six interviews, each 45 minutes long with a lunch break in between.

After some minutes the first interviewer came and we got into action right away. This first interview went really well. We talked a bit about continuous integration, my contribution to the Hudson project and about testing in general. Then the interviewer switched to a question about bash in Linux. We talked about how piping works under the hood. We spoke about the case where the first command is non-stopping one like "yes" or "tail -f". I learned that Linux is not executing the commands sequentially, like I thought it would. Rather it sends the output of a command to a buffer and the following command would use read and write blocks to work on the buffer. Time went and the next two interviewers came.

For some of the interviews, there are actually two people in the room. One of them being just a observer, that need to learn how to interview candidates. For the second interview, the interviewer directly put up a matrix on the white-board.


-4 -1 4 5
-3 0 6 10
1 8 11 15
17 19 22 30


I did not realize, that the matrix was set up in a way that each row and each column was ascending - from negative into positive numbers. My task then, was to come up with an algorithm that, given any number, would return true or false if the number was contained in the matrix. The first idea I had was looking at the upper left number, reading the right and lower neighbor and select the neighbor which would bring me closer to the target number. Surprisingly this worked out but the interviewer was able to find a negative example. For the next attempts I tried starting from the lower right corner, starting from the middle element looking at all 4 neighbors and even go row by row running binary search. The binary search algorithm would have worked but the interviewer indicated that this is not the best solution and that I should try a different starting location for my first approach. After some discussion we agreed that it would be possible to start lower left, so that the row would be ascending and the column would be descending. If the number, we were looking for, was bigger than the current item in the matrix (lower left at the start) I would go to the right neighbor. If it was lower I would go to the upper neighbor. This neighbor becomes the next item and the algorithm would again check right and upper neighbors until either the element was found or there was no way to go further. Finally I wrote some Java code for this and the second interview finished.

It was lunchtime and someone from Brazil picked me up for a lunch-date. We went to the Google cafeteria and I got a free lunch. We talked about different things and it was nice to relax a little bit from coding exercises.

Subversion Changes to Archive File

Just a simple one-liner that I want to share with you. Do you know this problem: you have made changes to one of your projects, some files have changed, some images were added. Now you need to move this to your live server. Most often I moved the files one by one based on their change dates. Today, just before committing the changes into Subversion, I had the idea to use the Subversion changes for creating a Tarball which I could copy over to and extract on my server.


svn st | grep -v '?' | awk '{ print $2 }' | xargs tar rvfz changes.tar


svn st - show Subversion changes (assuming that you have done all you svn add stuff before)
grep -v '?' - filter files which are not versioned, ie. IntelliJ project files
awk '{ print $2 }' - print only the file path and name
xargs tar rvfz changes.tar - add these files into changes.tar

All that is left is to transfer the file via SCP or FTP and extract it in the right location.

Google Phone Interview

A couple of months ago, I got contacted from a recruiter working at Google. It was about a Software Engineer in Test position in Stockholm. I was a bit surprised because this position has been on the net for a long time and it was more than a year ago since I sent them my CV. Even though I am not pro-actively looking for a new job, you cannot say no if Google is asking. I replied back that I was interested and she called me back two days later. Google's recruiting process is very different compared to other companies I applied at. First there will be a one or more interviews over the telephone, and if you are good at those, you will be invited to an on-site interview. The on-site interview is a whole day where you will meet 5 to 7 interviewers each for a 45 min interview. Each interviewer will then write together an evaluation of your interview. All of the evaluations are reviewed by the hiring committee and if they decide to hire you, as one of the last steps, your documents including CV is sent over to the US headquarter for the final go or no-go.

So when the recruiter called me, this was actually the first step in a long recruiting process. The only thing she was interested during the fist call, was the length of my notice period and if it was negotiable. After this she asked me two technical questions. First question for the worst case time complexity of Quicksort. Second question something about Radixsort complexity, but she could not read the question, so she asked what the difference between a HashMap and a HashSet was. The recruiter is not really a technical person. She was probably given a list of questions and answers to filter out the very bad candidates. My answers were okay and she told me that she would set up a telephone interview.

A week or two passed. Since I did not hear from her, I wrote a mail asking for the status of the telephone interview. Due to vacations it took a bit longer to set up the interview. My recruiter told me that a confirmation would be sent soon. To prepare myself, I should be looking at the Google Testing blog. Also she wrote, that the interviewer would be interested in my testing background and that he would be asking questions relevant to my coding and problem solving skills, algorithms, core computer science concepts, OOP and datastructures. It could also be, that he asked me to write a test plan or how to test an arbitrary Google product like Maps or how I would design a cache. Also we would be talking about my projects, problems I had found, tested and fixed and also what I would do with my Google 20% time. It would also be good to know some stuff about the company, like the founders, the products, the business model etc.

The interview however, went completely different. Someone else sent me a confirmation mail including a link to a Google Docs document. which I could share with the interviewer. It was also suggested, that I "warm" up a bit on topcoder.com in the "Software Competitions - Algorithms" arena, which I did. On the day of my phone interview, I was called exactly at 11am. The interviewer was very friendly and explained everything very carefully.

For the first task I was given two Lists of Integers, one sorted ascending the other sorted descending. I should write an algorithm (Java) in the Google docs document, which would take both Lists and return a combined List that is sorted. I remembered that in the merge phase of the Mergesort, you do a similar thing. Have a pointer on the first element of the first list and a pointer on the last element in the second list. Then copy back the smaller of the two elements and increase or decrease the appropriate pointer.

The second task was about anagrams. Given a list of words, I was asked for a datastructure in which you could store anagrams. I suggested to use a Hashtable. Each word is effectively a character array that could be sorted. So sort the characters and use this as the key for the Hash function. That way, elements having the same key are anagrams of each other.

In the last task I was introduced to a Skip List, which I never heard of before. My interviewer explained the Skip List a couple of times and I was asked to write an algorithm for finding an element in the List. I did okay I think and then he asked me if I had any questions. I asked a bit about the Stockholm office, what he was working with at Google, if they used Git for version controlling (they use Perforce) and some other questions. I asked about feedback but he said, he is never giving feedback directly. Instead the interviewer said, he would write an evaluation of the interview in the next couple of days. Also he said, if later on I was asked to do another phone interview, it did not mean I was good or bad in the first one.

One week or so passed. My recruiter called me and said that they liked my interview and want to meet me for an on-site interview. Because Stockholm was not big enough and not so many test managers worked there, they would like to have the interview in Zürich. I would be given further details and date suggestions later on. Google pays the flight upfront. The hotel must be paid by the candidate but the money can be reimbursed by sending an expense form to Google in Poland. For those candidates staying overnight, Google also pays the food (30 Euro, US-Dollar or Pound depending on where the interview was held). Rental car fees can also be reimbursed. I decided to pay for everything myself, since I was not staying over night. Sending some reimbursement form to Poland, seemed complicated. I was given 4 days to choose from for the interview. I picked one but they set up the time on another day anyways - weird. So I was flying on-site to Google in Switzerland, roughly two months after the initial contact.

Jetty7, Spring, Testing and Classloading

Currently working in a project where Jetty is configured in a Spring Context, that is started for unit tests. I was permanently hitting: "Context attribute is not of type WebApplicationContext". However, in the debugger I could see it was given a XmlWebApplicationContext - WTF? I figured it must be some classloading issue. Maybe something that was different when comparing Jetty 6 and 7? In previous Spring projects I have always used Jetty 6. This was the first time I tried Jetty 7 - which is still not final I think.

Anyway, what you want to do is this in your Spring applicationContext.xml file:









Previously I never had to use the parentLoaderPriority property but this made my tests working.

One (unrelated) question mark remains though. This is a Maven based project and the deploy artifact is a WAR file. There was one unit test that manually started a Jetty server in-process to do some testing. My initial idea was to use the maven-jetty-plugin to start Jetty before the test suite runs using the Maven command-line. However in that case it would not have been possible to run this test isolated in the IDE. I decided to make the test a Spring powered unit test and have the Spring Context start up Jetty. Unfortunately this required an existing WAR file, correctly set in the war property of the WebAppContext (see above). Maven builds the WAR file after running the test. To work around this, I told Maven to construct the WAR file before the tests are run:





org.apache.maven.plugins
maven-war-plugin


war-it
generate-test-resources

war








Anyone else has solved this scenario differently?

Spring Insight and Google Speed Tracer

I am sitting in a session here at Disruptive Code called "From Zero to Cloud" which is presented by Adam Skogman from SpringSource. The session was actually split in the middle, with lunch in between. While the talk itself did not give me much of a big WOW effect, simply because I have used the Amazon cloud services before, Adam mentioned a very cool tool called Spring Insight. He spoke about Spring Insight just under a minute. It sounded like an extension to the Spring tc server, which makes it possible to do performance analysis for web applications. It can measure and display information about the execution time of a HTTP request, a query execution in the database or even a single Spring bean invocation. Spring Insight can furthermore be integrated with Google Speed Tracer, which is apparently a Chrome plugin that I did not know of just a couple of minutes ago.

Spring Insight is very interesting project for me. A while ago a former colleague and I had a similar idea for an open-source project. We were planning to implement a language independent framework to measure execution time of web applications. In Java, the idea was to implement this using Annotations and Servlet Filters. In PHP, we were planning to have the same outcome using explicit method calls, which the PHP developer would have to add to the code manually. While the application was executed, it would then write a report file which users could then upload online to visualize the collected data. It is this part, that Google Speed Racer is doing now in the setup together with Spring Insight. Unfortunately we never had the time to work on our project, so it ended up just being an idea.

Anyway, I googled around a bit. The biggest point of criticism for Spring Insight is apparently that it is tightly coupled to SpringSource's tc Server. Even though there a millions of Spring Framework powered applications, only a fraction of them is running in Spring tc Server. Doing a quick research, it seems like the main reason to use Spring tc Server for Spring Insight, is the usage of custom Container deployers to enable the functionality. The good news is that someone else has already started to work on a third party library, to enable Spring Insight functionality without actually forcing the user to go with Spring tc Server. The library is called spring4speedtracer. Even though it is not as powerful as the original Spring Insight - for instance JDBC query execution times cannot be measured - it is a promising project. When I get home, I will have a closer look at this library, maybe I can contribute to the project (and put my own project idea to rest for good).