Having Maven create a nice zip File and separate Configuration

A co-worker and I are preparing a presentation about Amazon EC2 for other developers in my company. To show some stuff in action, we decided to write two JMS powered applications. One is sending messages, the other is receiving messages and persisting them into a database. During the presentation we will roll this out onto 3 EC2 nodes. Each application is build using Maven. To have it as convenient as possible, I changed the Maven package build phase to produce a single zip-file. The zip-file can be copied over to the EC2 node, where it is extracted. The zip-file contains one big "uber-jar" (with all the third party dependencies included) and a single properties-file to be able to set host and port for the JMS communication. We use ActiveMQ as JMS vendor in our projects.

Once the big zip-file has been extracted on the EC2 nodes and the properties have been set, you can start the producer and the consumer from the Main class. (Note the little dot infront of .:consumer... - this is needed so that the properties-file is found)

java -cp .:consumer-1.0-SNAPSHOT-final.jar package.MessageReceiver

For the packaging of the big zip-files, I use the Maven Assembly plugin during the package phase. The configuration looks like this:

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
    <id>final</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <unpack>true</unpack>
            <scope>runtime</scope>
            <useProjectArtifact>false</useProjectArtifact>
        </dependencySet>
    </dependencySets>
    <fileSets>
        <fileSet>
            <directory>${project.build.outputDirectory}</directory>
            <outputDirectory>/</outputDirectory>
            <excludes>
                <exclude>consumer.properties</exclude>
            </excludes>
        </fileSet>
    </fileSets>
</assembly>

src/main/assembly/jar.xml

This explodes all third party dependency jar files and merges them into one big uber-jar file. The properties-file is excluded from the uber-jar (this name sounds hilarious if you are from Germany by the way).

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
    <id>bin</id>
    <formats>
        <format>zip</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <fileSets>
        <fileSet>
            <directory>${project.basedir}/src/main/resources</directory>
            <outputDirectory/>
            <includes>
                <include>consumer.properties</include>
            </includes>
        </fileSet>
        <fileSet>
            <directory>${project.build.directory}</directory>
            <outputDirectory/>
            <includes>
                <include>*-final.jar</include>
            </includes>
        </fileSet>
    </fileSets>
</assembly>

src/main/assembly/zip.xml

This creates a zip-archive containing the "uber-jar" and the properties-file. Notice that the "uber-jar" has the suffix of "-final" equal to the id-attribute in the jar.xml file.

<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
    <archive>
        <manifest>
            <mainClass>package.MessageReceiver</mainClass>
        </manifest>
    </archive>
    <descriptorRefs>
        <descriptorRef>jar-with-dependencies</descriptorRef>
    </descriptorRefs>
    <descriptors>
        <descriptor>src/main/assembly/jar.xml</descriptor>
        <descriptor>src/main/assembly/zip.xml</descriptor>
    </descriptors>
</configuration>
<executions>
    <execution>
        <id>make-assembly</id>
        <phase>package</phase>
        <goals>
            <goal>single</goal>
        </goals>
    </execution>
</executions>
</plugin>

pom.xml

This code snippet runs the maven-assembly-plugin during the package phase.

Everything went well when we finished the producer application last week. Today I ran into some weird errors when I worked on the consumer end. Trying to start the MessageReceiver main class gave me the following error:

Caused by: org.xml.sax.SAXParseException: cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'amq:broker'.

I was able to resolve this error with the help of the ActiveMQ XML reference page, only to stumble into the next problem:

Caused by: org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Unable to locate Spring NamespaceHandler for XML schema namespace http://www.springframework.org/schema/context

It did not really jump me at first why this was happening. There a unit tests which load the Spring Context, they run fine. Maven runs the test in the package phase, they passed fine. So this was odd. After doing some research, I found out that the maven-assembly-plugin is responsible for this. Apparently Spring needs the spring.handlers and spring.schemas files to be present in the META-INF directory of the "uber-jar". A lot of other people had already hit the same problem before me. Some of them recommend the use of the maven-shade-plugin with the following setup:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.3.1</version>

<executions>
    <execution>
        <phase>package</phase>
        <goals>
            <goal>shade</goal>
        </goals>
        <configuration>
            <finalName>${artifactId}-${version}-final</finalName>
            <transformers>
                <transformer implementation="
org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                    <mainClass>
com.sabre.newgermanrail.dbitool.DbiTool
</mainClass>
                </transformer>
                <transformer implementation="
org.apache.maven.plugins.shade.resource.AppendingTransformer">
                    <resource>META-INF/spring.handlers</resource>
                </transformer>
                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                    <resource>META-INF/spring.schemas</resource>
                </transformer>
  <transformer implementation="
org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
                    <resource>consumer.properties</resource>
                </transformer>
            </transformers>
        </configuration>
    </execution>
</executions>
</plugin>

pom.xml

With the help of Transformers, the spring.handlers and spring.schemas files are added. If you are using Spring 3.x, there are many small spring-xyz.jar files and each of these comes with a spring.handlers and spring.schemas file. The maven-shade-plugin will append the content of each of these files using the AppendingTransformer. As you can see in the example above, I am again excluding my properties-file using the DontIncludeResourceTransformer. I decided to keep the zip-archiving part from the maven-assembly-plugin. Maybe this is something the maven-shade-plugin could do for me as well, not sure.

Interviewed at Google Pt.1

Today I want to blog about an on-site interview which I had at Google a while ago. Since I did not sign a non-disclosure agreement and only agreed on not taking photos in their office, I hope it is okay to write about this interview. Going to an on-site interview, is like reaching the next level, if Google is satisfied with your previous telephone interview(s). Due to summer-time and a lot of vacations, I had my on-site interview more than 1,5 months after doing the initial phone interview. This was very good, because it gave me extra time to brush up on some stuff. I read some books about data-structures and algorithms and practiced white-board coding a bit. I coded through a lot of sample questions from software interviews, which I was able to find on the net. However, all the questions during the interview were new to me, so I had to be spontaneous.

The job I was interviewed for, was Software Engineer in Test and the interview was held in Zürich, Usually the candidates are flying to the interview location one day in advance. That did not work for me, because I was already on another flight a day earlier. Instead Google booked two flights on the same day for me. which was OK. I arrived in time and signed in at the reception. As stated, you have to agree on not taking photos and you have to put some label identifying yourself on your shirt. Some minutes later I was picked up by my HR contact. She gave me a short office tour and set me up in one of the interview rooms. You could tell that Google is doing a lot for their employees. The Zürich office had a big fitness room with a personal coach. Free soft drinks everywhere, video gaming rooms, pool tables all that kind of stuff. My interview room was very small, maybe 3 times 3 meters and had a white-board. She handed me my interview schedule and I was surprised that it was six interviews, each 45 minutes long with a lunch break in between.

After some minutes the first interviewer came and we got into action right away. This first interview went really well. We talked a bit about continuous integration, my contribution to the Hudson project and about testing in general. Then the interviewer switched to a question about bash in Linux. We talked about how piping works under the hood. We spoke about the case where the first command is non-stopping one like "yes" or "tail -f". I learned that Linux is not executing the commands sequentially, like I thought it would. Rather it sends the output of a command to a buffer and the following command would use read and write blocks to work on the buffer. Time went and the next two interviewers came.

For some of the interviews, there are actually two people in the room. One of them being just a observer, that need to learn how to interview candidates. For the second interview, the interviewer directly put up a matrix on the white-board.

-4  -1   4   5
-3   0   6   10
 1   8   11  15
17   19  22  30

I did not realize, that the matrix was set up in a way that each row and each column was ascending - from negative into positive numbers. My task then, was to come up with an algorithm that, given any number, would return true or false if the number was contained in the matrix. The first idea I had was looking at the upper left number, reading the right and lower neighbor and select the neighbor which would bring me closer to the target number. Surprisingly this worked out but the interviewer was able to find a negative example. For the next attempts I tried starting from the lower right corner, starting from the middle element looking at all 4 neighbors and even go row by row running binary search. The binary search algorithm would have worked but the interviewer indicated that this is not the best solution and that I should try a different starting location for my first approach. After some discussion we agreed that it would be possible to start lower left, so that the row would be ascending and the column would be descending. If the number, we were looking for, was bigger than the current item in the matrix (lower left at the start) I would go to the right neighbor. If it was lower I would go to the upper neighbor. This neighbor becomes the next item and the algorithm would again check right and upper neighbors until either the element was found or there was no way to go further. Finally I wrote some Java code for this and the second interview finished.

It was lunchtime and someone from Brazil picked me up for a lunch-date. We went to the Google cafeteria and I got a free lunch. We talked about different things and it was nice to relax a little bit from coding exercises.

Subversion Changes to Archive File

Just a simple one-liner that I want to share with you. Do you know this problem: you have made changes to one of your projects, some files have changed, some images were added. Now you need to move this to your live server. Most often I moved the files one by one based on their change dates. Today, just before committing the changes into Subversion, I had the idea to use the Subversion changes for creating a Tarball which I could copy over to and extract on my server.

svn st | grep -v '?' | awk '{ print $2 }' | xargs tar rvfz changes.tar

svn st - show Subversion changes (assuming that you have done all you svn add stuff before)
grep -v '?' - filter files which are not versioned, ie. IntelliJ project files
awk '{ print $2 }' - print only the file path and name
xargs tar rvfz changes.tar - add these files into changes.tar

All that is left is to transfer the file via SCP or FTP and extract it in the right location.

Google Phone Interview

A couple of months ago, I got contacted from a recruiter working at Google. It was about a Software Engineer in Test position in Stockholm. I was a bit surprised because this position has been on the net for a long time and it was more than a year ago since I sent them my CV. Even though I am not pro-actively looking for a new job, you cannot say no if Google is asking. I replied back that I was interested and she called me back two days later. Google's recruiting process is very different compared to other companies I applied at. First there will be a one or more interviews over the telephone, and if you are good at those, you will be invited to an on-site interview. The on-site interview is a whole day where you will meet 5 to 7 interviewers each for a 45 min interview. Each interviewer will then write together an evaluation of your interview. All of the evaluations are reviewed by the hiring committee and if they decide to hire you, as one of the last steps, your documents including CV is sent over to the US headquarter for the final go or no-go.

So when the recruiter called me, this was actually the first step in a long recruiting process. The only thing she was interested during the fist call, was the length of my notice period and if it was negotiable. After this she asked me two technical questions. First question for the worst case time complexity of Quicksort. Second question something about Radixsort complexity, but she could not read the question, so she asked what the difference between a HashMap and a HashSet was. The recruiter is not really a technical person. She was probably given a list of questions and answers to filter out the very bad candidates. My answers were okay and she told me that she would set up a telephone interview.

A week or two passed. Since I did not hear from her, I wrote a mail asking for the status of the telephone interview. Due to vacations it took a bit longer to set up the interview. My recruiter told me that a confirmation would be sent soon. To prepare myself, I should be looking at the Google Testing blog. Also she wrote, that the interviewer would be interested in my testing background and that he would be asking questions relevant to my coding and problem solving skills, algorithms, core computer science concepts, OOP and datastructures. It could also be, that he asked me to write a test plan or how to test an arbitrary Google product like Maps or how I would design a cache. Also we would be talking about my projects, problems I had found, tested and fixed and also what I would do with my Google 20% time. It would also be good to know some stuff about the company, like the founders, the products, the business model etc.

The interview however, went completely different. Someone else sent me a confirmation mail including a link to a Google Docs document. which I could share with the interviewer. It was also suggested, that I "warm" up a bit on topcoder.com in the "Software Competitions - Algorithms" arena, which I did. On the day of my phone interview, I was called exactly at 11am. The interviewer was very friendly and explained everything very carefully.

For the first task I was given two Lists of Integers, one sorted ascending the other sorted descending. I should write an algorithm (Java) in the Google docs document, which would take both Lists and return a combined List that is sorted. I remembered that in the merge phase of the Mergesort, you do a similar thing. Have a pointer on the first element of the first list and a pointer on the last element in the second list. Then copy back the smaller of the two elements and increase or decrease the appropriate pointer.

The second task was about anagrams. Given a list of words, I was asked for a datastructure in which you could store anagrams. I suggested to use a Hashtable. Each word is effectively a character array that could be sorted. So sort the characters and use this as the key for the Hash function. That way, elements having the same key are anagrams of each other.

In the last task I was introduced to a Skip List, which I never heard of before. My interviewer explained the Skip List a couple of times and I was asked to write an algorithm for finding an element in the List. I did okay I think and then he asked me if I had any questions. I asked a bit about the Stockholm office, what he was working with at Google, if they used Git for version controlling (they use Perforce) and some other questions. I asked about feedback but he said, he is never giving feedback directly. Instead the interviewer said, he would write an evaluation of the interview in the next couple of days. Also he said, if later on I was asked to do another phone interview, it did not mean I was good or bad in the first one.

One week or so passed. My recruiter called me and said that they liked my interview and want to meet me for an on-site interview. Because Stockholm was not big enough and not so many test managers worked there, they would like to have the interview in Zürich. I would be given further details and date suggestions later on. Google pays the flight upfront. The hotel must be paid by the candidate but the money can be reimbursed by sending an expense form to Google in Poland. For those candidates staying overnight, Google also pays the food (30 Euro, US-Dollar or Pound depending on where the interview was held). Rental car fees can also be reimbursed. I decided to pay for everything myself, since I was not staying over night. Sending some reimbursement form to Poland, seemed complicated. I was given 4 days to choose from for the interview. I picked one but they set up the time on another day anyways - weird. So I was flying on-site to Google in Switzerland, roughly two months after the initial contact.

Jetty7, Spring, Testing and Classloading

Currently working in a project where Jetty is configured in a Spring Context, that is started for unit tests. I was permanently hitting: "Context attribute is not of type WebApplicationContext". However, in the debugger I could see it was given a XmlWebApplicationContext - WTF? I figured it must be some classloading issue. Maybe something that was different when comparing Jetty 6 and 7? In previous Spring projects I have always used Jetty 6. This was the first time I tried Jetty 7 - which is still not final I think.

Anyway, what you want to do is this in your Spring applicationContext.xml file:

Previously I never had to use the parentLoaderPriority property but this made my tests working.

One (unrelated) question mark remains though. This is a Maven based project and the deploy artifact is a WAR file. There was one unit test that manually started a Jetty server in-process to do some testing. My initial idea was to use the maven-jetty-plugin to start Jetty before the test suite runs using the Maven command-line. However in that case it would not have been possible to run this test isolated in the IDE. I decided to make the test a Spring powered unit test and have the Spring Context start up Jetty. Unfortunately this required an existing WAR file, correctly set in the war property of the WebAppContext (see above). Maven builds the WAR file after running the test. To work around this, I told Maven to construct the WAR file before the tests are run:

    
        
            
                org.apache.maven.plugins
                maven-war-plugin
                
                    
                        war-it
                        generate-test-resources
                        
                            war
                        
                    
                
            
        
    

Anyone else has solved this scenario differently?

Spring Insight and Google Speed Tracer

I am sitting in a session here at Disruptive Code called "From Zero to Cloud" which is presented by Adam Skogman from SpringSource. The session was actually split in the middle, with lunch in between. While the talk itself did not give me much of a big WOW effect, simply because I have used the Amazon cloud services before, Adam mentioned a very cool tool called Spring Insight. He spoke about Spring Insight just under a minute. It sounded like an extension to the Spring tc server, which makes it possible to do performance analysis for web applications. It can measure and display information about the execution time of a HTTP request, a query execution in the database or even a single Spring bean invocation. Spring Insight can furthermore be integrated with Google Speed Tracer, which is apparently a Chrome plugin that I did not know of just a couple of minutes ago.

Spring Insight is very interesting project for me. A while ago a former colleague and I had a similar idea for an open-source project. We were planning to implement a language independent framework to measure execution time of web applications. In Java, the idea was to implement this using Annotations and Servlet Filters. In PHP, we were planning to have the same outcome using explicit method calls, which the PHP developer would have to add to the code manually. While the application was executed, it would then write a report file which users could then upload online to visualize the collected data. It is this part, that Google Speed Racer is doing now in the setup together with Spring Insight. Unfortunately we never had the time to work on our project, so it ended up just being an idea.

Anyway, I googled around a bit. The biggest point of criticism for Spring Insight is apparently that it is tightly coupled to SpringSource's tc Server. Even though there a millions of Spring Framework powered applications, only a fraction of them is running in Spring tc Server. Doing a quick research, it seems like the main reason to use Spring tc Server for Spring Insight, is the usage of custom Container deployers to enable the functionality. The good news is that someone else has already started to work on a third party library, to enable Spring Insight functionality without actually forcing the user to go with Spring tc Server. The library is called spring4speedtracer. Even though it is not as powerful as the original Spring Insight - for instance JDBC query execution times cannot be measured - it is a promising project. When I get home, I will have a closer look at this library, maybe I can contribute to the project (and put my own project idea to rest for good).

Auto Secondary Indexes in Cassandra

Twenty minutes ago, Eric Evans talk about Cassandra ended at Disruptive Code. In the first 25 minutes or so, I was quite disappointed because it seemed to be exactly the same presentation, which I saw in June at the Berlin Buzzwords conference. Even the funny Bigtable-Dynamo lovechild slide was still there, though I believe the laughter was greater in Berlin than it was in Stockholm. Well I guess it's not so easy to get a Swede laughing.

Anyway, what I realised during Eric's presentation, was that he already added some stuff from the next Cassandra release 0.7. First of all, every time he was showing configuration, he had an excerpt from the a cassandra.yaml file. For instance this snippet from his timeseries example:

#conf/cassandra.yaml
keyspaces:
  -name: Sites
   column_families:
     -name Stats
      compare_with: LongType

new yaml configuration in Cassandra

Apparently as of version 0.7, the cassandra.yaml file is replacing the cassandra.xml file. I have not come in contact with yaml really, I believe it is common in the Ruby world. Another very cool feature is the addition of secondary indexes to Cassandra. In previous versions, Cassandra did not have indexes out of the box. To mimic the behavior of a secondary index, what you could have done is to create another Column Family (I believe it was called). This new Column Family would then be sorted differently and contain a key to the "original" entry. As a example, imagine having a Column Family to store addresses. To be able to search by the city, you could create another Column Family called "byCity" with two properties, "city" and "address key". Every time you insert or update an address, your code has to alter the byCity Column Family.

It looks like Cassandra will do this for you from version 0.7 on. There two new per-column settings called index_name and index_type. If I understood Eric correctly, adding this to your configuration will create you an inverted index, which can be used as a secondary access path. I think this is a very nice, yet very undocumented, feature. No clue when version 0.7 is going to be released but I hope it will be very soon, because we are only weeks away from starting a very big Cassandra project in my company.

Disruptive Code Party

Seriously, who needs Java One when you can go partying with Disruptive Code people at Gröna Lund? Okay, Weather might be a bit nicer in California :)

Day 1 was great I think. Great sessions, especially the ones about HTML5. Two things I did not like: Adam Skogmans "Designing For NoSQL" talk started earlier than it was printed on the badges or written on disruptivecode.com = missed it :( Also WIFI quality in the Big Hall was really bad. Nevertheless great stuff! Looking forward to day two.

Choosing wrong track

A problem that keeps following me on conferences, is picking the wrong talks. Often a session sounds nicer on the agenda, than it is in reality. For the first track I selected PayPal over the HTML5 session. I was hoping to get some insights into the PayPal API. How to use it? How to integrate it with some practical examples? The session however turned out to be not that detailed. It felt more like a sales talk. By the way, PayPal is one of the main sponsors of Disruptive Code. The fact, that the organisers put up all conference tweets during the talk, and it seemed to be really exciting over at the HTML5 track, made things worse.

The only good thing I could take away from this talk, are some ideas to possibly integrate payment into my projects. The first of the two PayPal speakers, gave some interesting project examples. ie. a person to persons send money application, a game where you can buy ammo or crowd-sourcing (paying users for uploading pictures, entering recipes etc.). There is also a portal called PayPal X, where developers can develop applications and tools on top of the PayPal API. Similar to the iPhone, these applications have to be approved by PayPal before they are available to everyone.

So it is likely I will embed PayPal in my applications in the future. This session just did not show me how to really. The only slide having code on it, wasn't helping there much either and it certainly was not PHP code like the speakers said.

HTML5 Web Workers and Geolocation

After having missed the first session about upcoming HTML5 features, I decided to go to Peter Lubbers talk "HTML5 Web Sockets, Web Workers and Geolocation Unleashed". Peter is the author of the recently published Apress book "Pro Html 5 Programming". He works for a company called Kaazing which I think is based in the Netherlands. Some time ago, my boss forwarded me a mail which was from Kaazing. It was about a Web Sockets presentation which they wanted to held at our office, since game clients are one of the primary use cases where Web Sockets can come in handy. I have to admit that back then I did not want to meet them. Primarily because I do not work with client products. In addition to that, I thought it was immature, but it was rather that I did not know so much about it back then and did not care.

Anyway, Peters talk covered three of the most interesting API's around HTML5 - Web Workers, Geolocation and Web Sockets. These API's have initially been part of the HTML5 spec but have now been removed and put into their own specification. The idea behind all three is to make life simpler for the developers. With the current generation of browsers and HTML4, developers have to come up with complicated hacks or they have to use plugins in order to mimic bi-directional communication. What HTML5 is aiming for, is to support this natively powered through the browser instead of building something similar based on a bad foundation.

Web Workers are a new feature which brings back UI responsiveness while long running, heavy Javascript is executed. It enables background processing of such script while the user can still use the browser. Peter had an excellent example of this and hopefully I can put this up here on my blog later on. In his example he had a webform with two buttons. Each of the buttons would fire a busy-keeping Javascript for 10 seconds. One button would do this without, the other one with using a Web Worker. Clicking the first button, it was impossible to use the dropdown element or even open a new tab in Firefox. All this worked with Web Workers. Currently this feature is available in Firefox, Chrome, Safari and Opera but not in Internet Explorer. Actually it would be fun to write a web application using Web Workers, where you print a message to the IE users like "if you cannot navigate right now, consider switching to another browser". Web Workers are an incentive that comes for free if you use anything else than Internet Explorer.

The second API Peter talked about was Geolocation. Again, Geolocation is only supported in some browsers right now. You want to check out caniuse.com to see if your browser supports it. Also html5test.com is a great site to check HTML5 compatibility. Geolocation is a name that stands for native support of user location inside a browser. There are really only two methods that the API supports, getCurrentPosition() and watchPosition(). The first one doing a one time call, the latter constantly receiving the location of the user. This of course makes much more sense on mobile devices than from withing a fixed network. What the two calls return is simply longitude, latitude and accuracy. It is up to the developer how he uses this information within the web application, ie. by displaying it using the Google Maps API. Along with the API calls, you can also request additional metadata but if your browser can not give it to you (like altitude, heading, speed) you might end up getting NULL instead. Looking under the hood, Geolocation is implemented by the browser vendors by using an external location service. The browser asks this service for the location and returns this to the user. I was thinking, the watchPosition() method from the API is probably a candidate where you want to use a Web Worker, unless it is already implemented on top of a Web Worker. Have to find this out.

Would like to blog more about Web Sockets but the next session about CSS3 has already started...