Evaluating Cassandra

In the last few months I had the chance to play with Hadoop for a bit. I did some prototyping and unit-testing to learn the framework and see, of it is something we can use in production in my current company. My team is re-designing an existing application which must be able to store 5 years of Poker related data. Five years does probably not sound too bad but given that our system deals about 500 poker hands at peak time, this can be challenging.

I did a presentation on Hadoop a couple of weeks ago but we decided, that we will not use it as data storage. There are a couple of reasons to it. First of all, the framework is going through a lot of changes. The API has changed a lot between version 0.18.3 and 0.20.2. Often the developer has to deal with code examples, documentation and libraries, that are not updated to the latest Hadoop API. Furthermore with Hadoop you have to tame the underlying HDFS file system and you have to tweak your Map Reduce jobs to make them run optimal. Also the Namenode is like Achilles heel in Hadoop. If the Namenode has a problem you have to have a great knowledge about how to fix and restore it, otherwise you are in trouble.

We ended up talking about Cassandra, which is also a Apache top-level project. On a very high level, it reminds me of a schema-less database, whereby Hadoop is more of a distributed file storage. I spent a day to familiarize myself with Cassandra. Here are my first thoughts.

Cassandra does not know any indexes. The data must be stored in an intelligent way, so that it can be retrieved with good performance using the primary key. What does that imply? It means you roughly need to know, how you will access the data in the future and design for it. Otherwise you might end up with uses cases that you cannot serve efficiently, ie. find all players having had Pocket Jacks pre-flop. The good thing about Cassandra is that the data is truly distributed over all nodes. You can read and write from each node at any given time. No single-point of failure like in Hadoop. On the other hand, Cassandra is not processing anything in parallel. If you want to access your data in a distributed, parallel fashion, you need to write a concurrent application talking to several nodes. Alternatively you could use the Hadoop-Cassandra integration, which was added in the latest release. It allows you to write Hadoop Map Reduce Jobs using Cassandra as their InputFormat. Fancy.

Another impression I also had, is that Cassandra hides a lot of the low level details that you come in contact with when using Hadoop. I also looked at HBase but it was rather complicated to get it running compared to Cassandra. Another feature I like is that it uses JSON a lot. It is easy to visualize your data model or even backing-up and restoring your entire database using JSON. I have not written any test code that actually uses Cassandra. We will see about that. I am looking forward to the Berlin Buzzwords conference in June, where they have a talk about Cassandra and a lot of sessions about Hadoop.

If you want to know more about Cassandra, I can recommend these great links:
http://arin.me/blog/wtf-is-a-supercolumn-cassandra-data-model
http://www.sodeso.nl/?p=80
http://blog.evanweaver.com/articles/2009/07/06/up-and-running-with-cassandra/

Tomcat and java.net.SocketException: Too many open files

God bless monitoring. An hour ago I received a SMS from one of my servers that one of my sites was not available anymore. It is a dedicated server that runs 4 Tomcats in parallel. To track down the problem I started looking at the Tomcat logfiles. Whoo whats that? The fourth Tomcat was spamming logfiles like crazy. For the past four days it had created me 4 logfiles having a combined size of 600 GB. I was hitting a java.net.SocketException for too many open files.

First I thought I had a leak somewhere, which prevented files and sockets from getting closed properly. Actually this was not the main problem. Since all Tomcats were running as the same user, and I had not touched the open file limit for this user, the default maximum of 1024 in Ubuntu 9.10 server was way too little. I checked how many files I had open for this user.

ps aux | grep tomcat

Then for every PID I ran

lsof -p PID | wc -l

I had cleaned the logs and rebooted already. The combined result was that I already was scratching the 1000 mark for all Tomcats after rebooting. Very thin ice. To make a long story short, here is how to change the maximum open file limit on Ubuntu 9.10 server.

First you edit /etc/security/limits.conf and add your new limit for the user running Tomcat. In my case the user was called virtual:

virtual         hard    nofile  5120
virtual         soft    nofile  4096

In addition to that, edit the file /etc/pam.d/common-session and add

session required        pam_limits.so

done! Reboot the machine, then verify the changes running

su virtual
ulimit -n

glftpd trouble after upgrading to Ubuntu 10.04

Today I upgraded three computers running Ubuntu 8.04 LTS to the latest version 10.04. On one of the computers I had glftpd running and it did not work anymore after the upgrade. When I checked for the open ports

netstat -anp --tcp --udp | grep LISTEN

the port glftpd was previously running on, was not in the list anymore and also ps aux | grep glftpd did not show the process. The error "service/protocol combination not in /etc/services: glftpd/tcp" could be seen when restarting xinetd

sudo /etc/init.d/xinetd restart

while tailing syslog

tail -100f /var/log/syslog

All that was missing was a entry in /etc/services at the end.

# Local services
glftpd          10087/tcp                       # glftpd

First I thought the 10.04 upgrade process was flawed but then I remembered that Ubuntu actually asked me to keep or overwrite some files which I had modified in 8.04. I almost always decided to overwrite the files, except for some local Apache modifications. Apparently /etc/services was one of the files that had changed.

Dependencies clashing with Maven Overlays

Last year I wrote about a Maven feature which I had discovered back then, that made it possible to "merge" two web-applications using overlays. Unfortunately I discovered a very annoying problem with overlays today. In my current project I am using the decode method in the Base64 class in commons-codec. The method was added in commons-codec 1.4. One of the overlay WAR files comes with an older version of commons-codec. What Maven does is that it just throws the dependencies from the overlay project together with your referring project. Make sure you have a look in the lib folder after running mvn package. When I looked into my lib directory in my WEB-INF folder, I realized that I had commons-codec-1.4.jar as well as commons-codec-1.3.jar (the one coming with the overlay) in there. When the web-application loads, version 1.3 is picked up causing a Runtime Exception because the decode method was missing.

This is pretty serious I think. First of all, I did not realize that the 1.3 dependency slipped in with the overlays. I ran mvn dependency:tree -Dverbose -Dincludes=commons-codec to see what happened but it did not show me any library using commons-codec 1.3. It took a while until I realized what the problem was and that the Maven dependency plugin would not help me here. I went into my local repository and ran a find . -name '*.pom' -exec grep -nH 'commons-codec' {} \; to be able to spot projects using commons-codec. I would like to see a Maven plugin where it can find you libraries using a certain version of a dependency. Anyway, I was lucky. The overlay projects could be upgraded to use codec 1.4 instead and the dependency clash went away.

Unfortunately one of the WAR overlays uses dom4j-1.6.1.jar which has a dependency to xml-apis-1.0.b2.jar. In the project where I refer to the overlay it depends on xms-apis-1.3.04.jar - so here I am having two jar's of the same type in the classpath and no easy way to fix this.

First Walk in the Clouds

During the week I tried the Hadoop framework for the first time. I wrote a proof of concept prototype for an application that we are likely going to develop. I managed to test my code using unit test (mrunit), local integration test starting embedded Hadoop and running it pseudo-distributed on my local Hadoop cluster. The final step was to test it in a real cluster in Amazon EC2.

I had never started any AMI in EC2 before, so everything was brand new for me. Access to your AWS account as well as you instances is well protected. Setting up proper access to my EC2 instances was very bumpy, especially since I made a mistake with one of the private key files. Unfortunately the error message I got, was not very helpful and I spent quite some time finding the problem.

If you want to use EC2 you need the following security credentials. Sign-in Credentials: this is basically a email address and a password protecting your AWS account. You need to keep this really safe.

Access Credentials: they consist of three different sub-groups. First there are the Access Keys. Each EC2 user can have up to two Access Keys. Each Access Key has a Access Key Id and a Secret Access Key. In your system environment variables system environment variables, you add them as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The next subgroup are the X.509 Certificates. Again, you can generate two X.509 Certificates at a time for each EC2 account. Create a new certificate in the AWS management console and download the public and private key. The public key will be in a file that starts with cert-xxxx, the private key will be in a file that starts with pk-xxxx. Copy these two files somewhere and add the full location to your system environment variables as EC2_PRIVATE_KEY and EC2_CERT. The last subgroup are the Key Pairs. A key pair is used when you install a AMI executing the ec2-run-instances command. This is a additional protection to restrict access to you running instance. A Key Pair also has a private and a public part. Amazon will keep the public key of the key pair and store it with your instance. To connect to the instance you need the private key part of the Key Pair.

Run the command: "ec2-add-keypair foo" to create a Key Pair named foo. This will return you the private key part which you will have to copy into a file. This is where I made a stupid mistake. I copied only the parts between BEGIN and END into the file but the file needs to contain the whole output. So it is much much better to run this instead: "ec2-add-keypair foo > ~/.ssh/foo.keypair.ssh". This will automatically sent the output to a new file in the .ssh directory. Finally give your new file the right permissions using "chmod 0700 ~/.ssh/foo.keypair.ssh". For further reading I recommend this page which I helped me fixing my problem. So if you try to ssh -i into your instance and it asks you for a passphrase, something is not correct with the private key part of your Key Pair. Another manifestation of the same problem if you do not use ssh to connect to your instance but the Cloudera scripts instead. If you are following the Cloudera guide for running Cloudera Distribution AMI for Hadoop, and you are on chapter 2.3 Running Jobs and execute: "hadoop fs -ls /" to get this:

WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively 
10/03/12 15:09:31 INFO ipc.Client: Retrying connect to server: ec2-184-73-44-221.compute-1.amazonaws.com/184.73.44.221:8020. Already tried 0 time(s). 

It was the same problem for me. Having the correct private key part of the Key pair fixed this for me.

The last bit of EC2 protection are the Account Identifiers. I think they are only relevant if you plan to share AWS resources with different accounts - not sure.

Product Import with Spring Batch - Part1

I have two websites, which on a regular basis import products belonging to affiliate programs. The websites were developed in 2005 and 2006. In the process of moving the applications from a Windows to a Linux server, I decided to rewrite and modularize a lot of code that was duplicated or poorly performing. For the product import, I selected Spring Batch as new framework. Spring Batch forces you to write little code chunks instead huge jobs, which will then make up your whole batch. Also, there is a ready-to-use tool for monitoring and I am familiar with the standard Spring Framework.

Since the full batch might be complex later, I decided to publish my experiences here while I implement. In the first part, I will use Spring Batch to download a product data file (csv) that belongs to an affiliate program. To build the application, I use Maven 2. In the version for this part, only 5 dependencies are needed. Spring Batch of course, commons-io and commons-lang to help me with some utility stuff, junit and log4j.

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    4.0.0
    com.kanzelbahn.utils
    product-import
    jar
    1.0-SNAPSHOT
    Product Import Library

    
        2.0.3.RELEASE
        2.5.6
        false
    

    

        
        
            commons-io
            commons-io
            1.4
        
        
            commons-lang
            commons-lang
            2.4
        

        
        
            org.springframework.batch
            spring-batch-core
            ${spring.batch.version}
        
        
            org.springframework.batch
            spring-batch-test
            ${spring.batch.version}
        

        
        
            junit
            junit
            4.4
            test
        

        
        
            log4j
            log4j
            1.2.9
        

    

    
        
            
                org.apache.maven.plugins
                maven-surefire-plugin
                
                    ${skipJunitTests}
                    -Xms128m -Xmx256m -XX:PermSize=128m -XX:MaxPermSize=256m
                    false
                
            

            
                org.apache.maven.plugins
                maven-compiler-plugin
                
                    1.5
                    1.5
                    false
                    false
                
            
        
    

pom.xml

One big disadvantage of Spring Batch 2.0, is that you cannot unit test it together with TestNG. The problem is rather in the Spring Framework than Spring Batch. When you write a Spring powered TestNG unit test, you need to extend AbstractTestNGSpringContextTests. There is no Runner to use in the @RunWith annotation, like SpringJunit4ClassRunner. This alone is not a problem, but since your Spring Batch test also need to extend from AbstractJobTests, and you cannot inherit twice, TestNG is out of the loop. This will be fixed in Spring Batch 2.1 because then you will not have to extend from AbstractJobTests anymore. I did not use version 2.1 because it was not release at the point of writing this post.

Let's have a look at the job configuration.

     xmlns:beans="http://www.springframework.org/schema/beans"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="
           http://www.springframework.org/schema/beans
           http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
           http://www.springframework.org/schema/batch
           http://www.springframework.org/schema/batch/spring-batch-2.0.xsd">

    
        
            
        

        
            
            
        

        
            
        
        
            
        
    

jobs.xinc

As you can see, it has only three (well three and a half) steps. Step 1 is implemented in the InitializingTasklet and all it really does is logging the start time. The next step is called csv_exists and is a decision step. If the csv-File for the current day exists, I move on to Step 3, otherwise Step 2 is invoked. Step 2 is implemented in the DownloaderTasklet. This Tasklet will download the csv-File of the current day. Step 3 is implemented in the FinishingTasklet and also performs basic logging. Let's look at the three different Tasklet's and the JobExecutionDecider.

The InitializingTasklet for Step 1 is pretty much self explaining. On a side note, see how I use FastDateFormat instead of SimpleDateFormatter because it is not thread-safe.

/**
 * Performs initialization tasks.
 *
 * @author reik.schatz Dec 11, 2009
 */
public class InitializingTasklet implements Tasklet {
    private static final Logger LOGGER = Logger.getLogger(InitializingTasklet.class);

    private static final FastDateFormat DATE_FORMAT = FastDateFormat.getInstance("yyyy-MM-dd HH:mm:ss");
    
    public RepeatStatus execute(final StepContribution stepContribution, final ChunkContext chunkContext) throws Exception {
        LOGGER.debug("Initializing at " + DATE_FORMAT.format(new Date()) + ".");
        return RepeatStatus.FINISHED;
    }
}

InitializingTasklet.java

The JobExecutionDecider for the csv_exists decision is implemented in a class called DoesCsvExistDecision. When constructing the bean, you need to specify a CsvFileFacade to handle the access to the csv-File. CsvFileFacade is an Interface and the my only implementation is the class CsvFileFacadeImpl. The implementation expects a ImportSettings instance upon creation. Everything is wired together by Spring. Using the ImportSettings instance, the CsvFileFacade knows about the root directory, to which the csv-Files shall be downloaded to, the affiliate program id and the location of the csv-File on the Internet. I used the German affiliate program Bakker as a sample implementation.

/**
 * A {@link CsvFileFacade} wraps the handling of csv files.
 *
 * @author reik.schatz Dec 11, 2009
 */
public interface CsvFileFacade {

    File getCsvFile();

    URL getDataFileURL();
}

CsvFileFacade.java

/**
 * A {@link CsvFileFacade} which retrieves informations about the csv file
 * location from a {@link ImportSettings} instance.
 *
 * @author reik.schatz Dec 11, 2009
 */
public class CsvFileFacadeImpl implements CsvFileFacade {
    private static final Logger LOGGER = Logger.getLogger(CsvFileFacadeImpl.class);

    private final ImportSettings _settings;

    public CsvFileFacadeImpl(final ImportSettings settings) {
        _settings = settings;
    }

    public File getCsvFile() {
        final String fileName = _settings.getImportable().getProgramId() + ".csv";
        final File dailyDirectory = _settings.getDirectory();
        return new File(dailyDirectory, fileName);
    }

    public URL getDataFileURL() {
        return _settings.getImportable().getDataFile();
    }
}

CsvFileFacadeImpl.java

/**
 * Wraps all settings for the current import run.
 *
 * @author reik.schatz Dec 13, 2009
 */
public interface ImportSettings {

    /**
     * Get's the directory to which the datafile shall be imported to.
     *
     * @return File
     */
    File getDirectory();

    /**
     * Returns the {@link Importable} which shall be used.
     *
     * @return Importable
     */
    Importable getImportable();
}

ImportSettings.java

/**
 * Encapsulates settings of for a single import run.
 *
 * @author reik.schatz Dec 13, 2009
 */
public class StandardSettings implements ImportSettings {

    private final File _rootDirectory;
    private final Importable _importable;

    public StandardSettings(final File importDirectory, final Importable importable) {
        _importable = importable;

        if (importDirectory == null || !importDirectory.exists()) {
            final String path = importDirectory == null ? "" : importDirectory.getPath();
            throw new IllegalArgumentException("Given importDirectory (" + path + ") does not exist.");
        }

        _rootDirectory = importDirectory;
    }

    /** @inheritDoc **/
    public File getDirectory() {
        final Date now = new Date();
        final FastDateFormat df = FastDateFormat.getInstance("yyyy-MM-dd");
        final String day = df.format(now);

        final File importableDataFileDirectory = new File(_rootDirectory, day);
        if (!importableDataFileDirectory.exists()) {
            try {
                FileUtils.forceMkdir(importableDataFileDirectory);
            } catch (IOException e) {
                throw new IllegalStateException("Unable to create daily import directory (" + day + ")", e);
            }
        }
        return importableDataFileDirectory;
    }

    /** @inheritDoc **/
    public Importable getImportable() {
        return _importable;
    }
}

StandardSettings.java

/**
 * Represents a importable program.
 *
 * @author reik.schatz Dec 13, 2009
 */
public interface Importable {
    
    public int getProgramId();

    public URL getDataFile();
}

Importable.java

/**
 * A {@link Importable} which wraps all parameters for the german affiliate program Bakker.
 *
 * @author reik.schatz Dec 13, 2009
 */
public class Bakker implements Importable, Serializable {

    private static final long serialVersionUID = 7526472295622776147L;

    private final int _programId;
    private final URL _dataFile;

    public Bakker(final int programId, final String dataFileLocation) {
        _programId = programId;
        try {
            _dataFile = new URL(dataFileLocation);
        } catch (MalformedURLException e) {
            throw new IllegalArgumentException("Given dataFileLocation " + dataFileLocation + " is not a valid URL");
        }
    }

    public int getProgramId() {
        return _programId;
    }

    public URL getDataFile() {
        return _dataFile;
    }

    @Override
    public boolean equals(final Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        final Bakker bakker = (Bakker) o;

        if (_programId != bakker._programId) return false;

        return true;
    }

    @Override
    public int hashCode() {
        return _programId;
    }

    @Override
    public String toString() {
        return "Bakker{" +
                ", _programId=" + _programId +
                ", _dataFile=" + _dataFile +
                '}';
    }
}

Bakker.java

Every JobExecutionDecider must implement the decide method. I get the csv-File from the CsvFileFacade, which will be a different file depending on the day you run the job and the affiliate program. If the csv-File exists, I return FlowExecutionStatus.COMPLETED which will invoke Step 3. Otherwise I return FlowExecutionStatus.FAILED which will invoke Step 2 – to download the file.

/**
 * Tests for the existence of the csv file in the specified
 * {@link CsvFileFacade}.
 *
 * @author reik.schatz Dec 11, 2009
 */
public class DoesCsvExistDecision implements JobExecutionDecider {

    private final CsvFileFacade _csvFileFacade;

    public DoesCsvExistDecision(final CsvFileFacade csvFileFacade) {
        _csvFileFacade = csvFileFacade;
    }

    public FlowExecutionStatus decide(final JobExecution jobExecution, final StepExecution stepExecution) {
        final File csvFile = _csvFileFacade.getCsvFile();
        if (csvFile.isFile()) {
            return FlowExecutionStatus.COMPLETED;
        } else {
            return FlowExecutionStatus.FAILED;
        }
    }
}

Bakker.java

The file download is wrapped in the DownloaderTasklet. The Tasklet again is injected with a reference to the CsvFileFacade. Using the Facade and FileUtils from commons-io, I download the csv-File and store it physical on disc.

/**
 * Downloads the csv file.
 *
 * @author reik.schatz Dec 11, 2009
 */
public class DownloaderTasklet implements Tasklet {

    private final CsvFileFacade _csvFileFacade;

    public DownloaderTasklet(final CsvFileFacade csvFileFacade) {
        _csvFileFacade = csvFileFacade;
    }

    public RepeatStatus execute(final StepContribution stepContribution, final ChunkContext chunkContext) throws Exception {
        final File csvFile = _csvFileFacade.getCsvFile();
        final URL location = _csvFileFacade.getDataFileURL();
        try {
            FileUtils.copyURLToFile(location, csvFile);
        } catch (IOException e) {
            throw new IllegalStateException("Unable to download csv file.", e);
        }
        
        return RepeatStatus.FINISHED;
    }
}

DownloaderTasklet.java

The job ends for now in Step 3, which simply invokes log4j one more time.

/**
 * Contains actions to be done when a Job is finishing.
 *
 * @author reik.schatz Dec 11, 2009
 */
public class FinishingTasklet implements Tasklet {
    private static final Logger LOGGER = Logger.getLogger(InitializingTasklet.class);

    private static final FastDateFormat DATE_FORMAT = FastDateFormat.getInstance("yyyy-MM-dd HH:mm:ss");

    public RepeatStatus execute(final StepContribution stepContribution, final ChunkContext chunkContext) throws Exception {
        LOGGER.debug("Finished at " + DATE_FORMAT.format(new Date()) + ".");

        return RepeatStatus.FINISHED;
    }
}

FinishingTasklet.java

Unit testing could not be easier.

/**
 * Tests the import job.
 *
 * @author reik.schatz Dec 11, 2009
 */
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/applicationContext.xml" })
public class ImportJobTest extends AbstractJobTests {

    @Autowired
    private CsvFileFacade _csvFileFacade;

    @Transactional
    @Test
    public void testChain() throws Exception {
        final JobExecution jobExecution = this.launchJob();
        assertEquals(jobExecution.getExitStatus(), ExitStatus.COMPLETED);
        assertTrue(_csvFileFacade.getCsvFile().exists());
    }
}

ImportJobTest.java

You can download the full source code from Google Groups. The zip-archive will also contain the remaining parts of the Spring configuration, which is needed to wire all beans together.

Whats new in Maven 3

Yesterday evening I went to a Java event in Stockholm called Java Forum. This was actually the first time I was there, even though the Java Forum meeting is held every month and at different locations in Stockholm. Yesterday's meeting was very interesting for me because it was all about Maven.

The first presentation was held by Jason van Zyl and was all about the new features in Maven 3. Jason is sort of the brain behind Maven. He is CTO at Sonatype, the Maven company.

The second speech was given by Dennis Lundberg, a guy who helps developing a lot of Maven plug-ins. His talk was about the Maven Site plug-in. Dennis spoke in Swedish, which was fine by me. However, since Jason was present, I think it would have been more polite, if he had talked English too - especially since he had an (unanswered) Maven 3 question which Jason maybe could have answered. Finally Jason took over again and talked about his idea of a next generation infrastructure build up from Maven, M2Eclipse, Nexus and Hudson. Unfortunately there was enough time left to talk about Hudson, so he could only cover M2Eclipse and Nexus.

M2Eclipse looked promising, as it will be the first working Maven integration plug-in for the Eclipse IDE. It wont be out until January 2010 however. Not that I care much, since I am using IntelliJ, but they had some nice features in M2Eclipse. One of them being some extra XML meta-data in the Maven POM, which is only picked up by M2Eclipse and ignored by command-line Maven. This meta-data will speed up the builds in Eclipse when using M2Eclipse. A build which took minutes before can be run in seconds that way. M2Eclipse will download all Sources automatically, this isn't something new, but it also with a single click create you a new project for any of your dependencies. This can be helpful if you project has a dependency and you need to patch something in that dependency very quick.

Anyway, for me the most interesting presentation was the one about Maven 3 - which will not be released until next year. So what are the big improvements?

Probably the biggest visible change is the polyglot POM support. You can write your POM files now in different languages. Jason has examples for Groovy, Yaml and Raven. I found some sneak previews here and here. It was a little bit funny when Jason had presented the polyglot feature of Maven 3 and then asked in the audience if anyone thought the original XML format was annoying. No one of the 80 people thought it was annoying. So maybe this feature will not be used very often.

Those of you who have worked in multi-module or multi-pom projects in Maven2 might have asked themselves, why do I have to specify the parent version in every sub module. Maven 3 will remove this redundancy and add version-less parent elements.

Another big problem in Maven2 is to find out for an effective POM, which dependency or POM supplied which artifact to the final outcome. Maven3 will address this and it will be easier to see who contributed which artifact. In connection with M2Eclipse it will then for the developer be possible to deselect a certain contribution and select another one instead. All this is only possible because Maven3 decouples execution plan and execution. You POM defines an execution plan which is then brought to execution. Users can make changes to the execution plan before execution. In general, will Maven 3 come with a lot of extensions points, which can be used to statically and dynamically alter the POM. This can for example be leveraged by companies, who have their dependencies and versions in different formats, that want to use parts of Maven 3.

Extension points seems to be the next big thing in Maven 3. This is actually something where Jason confirmed they stole from Eclipse. Instead of sub-classing a plug-in, like you would do in Maven 2, developers can hook up to different extension points to alter the plug-in behavior. For instance, you might have an extension point to alter the way the web.xml is processed by the WAR plug-in. You don't have to inherit anymore to get customized behavior.

Error messages will not be as cryptic anymore. Most of the 40+ something error messages will come with a link to the Maven 3 wiki where the error is explained in detail. A state of the art Maven 3 client is currently developed by the Jetty people. They have created their own asynchronous HTTP client library which will be used in the M3 client. Internally, the new Maven uses a micro OSGI container but only for classloading and bundle management. The Maven 3 source code uses Google Guice for dependency injection and a library called Peaberry which extends Guice with OSGI capabilities.

Finally, the whole dependency resolution is re-factored by Sonatype into a standalone product. The software will be called Mercury and Maven 3 will be a client who uses Mercury. Software companies and developers might as well use Mercury to integrate dependency resolution into their own solutions.

I got some impressions about Maven 3. It was not so much different on the surface, except for the polyglot stuff and the extension points. A lot of stuff is happening under the hood. Maven 3 is fully backwards compatible to Maven 2. It runs much faster though and the code base is 1/3 smaller.

I liked Jason's two presentations in Stockholm. Too bad that time was running out. I would have liked to see more Maven 3 in action from the command line or some extension point code examples in Eclipse.

Pimp ma JDBC ResultSet

Last week I had to work on an interesting problem. My team was working on some sort of reporting application which creates csv like reports based on JDBC ResultSet's returned from querying a database. Earlier this year, I refactored some code sections which created the report files, to make them testable by unit tests. Since the application was already using Spring 2.5, I decided to refactor the plain old JDBC code and use the Spring JdbcTemplate instead.

My unit tests passed and I was happy. The code never went live though, as other stuff got a higher priority. After a while the application was put on release schedule but (of course while I was away from work) system verification found a big problem. When creating the report the application crashed with a OutOfMemoryError. Another developer started looking at my refactored code. First of all, I was not aware that the reports could contain million of rows. The one report that caused the OutOfMemoryError had 16 million rows. It was pretty naive that I used the query method in the SimpleJdbcTemplate passing a ParameterizedRowMapper as argument. Obviously the returned list would contain 16 millions entries and never fit into memory.

Since I was not in the office, the developer who looked at my code wrote me a mail. I don't remember the exact words but he asked me, if I had a particular reason to use SimpleJdbcTemplate instead of the old code. I felt challenged. Of course it would be madness, not to use JdbcTemplate or SimpleJdbcTemplate in a Spring powered application. However he had one good argument - the old code worked! I started investigating how to archive the same performance using only Spring classes. I suggested to use the query method of the JdbcTemplate instead. When using this method, you have the opportunity to supply a RowCallbackHandler as argument. The processRow method of the RowCallbackHandler is then invoked for every row in the ResultSet and we could directly write a line in our report.

We changed the code once more. The unit test still ran. However, we quite soon discovered that we did not really fix the main issue. Even though it could handle more records now, it would still fail with an OutOfMemoryError. Instead of building up a huge List as before, it created a huge ResultSet in memory. Another big problem became apparent. Processing the rows was now very slow. Compared to before, a report with 16 Million rows which took 7 minutes to create before would now be created in 90 minutes. Now I felt really challenged! I did not want to go back to the old code and use good 'ol plain JDBC again.

I downloaded the Spring source code and compared our previous implementation with the way we run now. Soon I found out about the problem. The old code created something which I call a streaming ResultSet. This was done by specifying flags java.sql.ResultSet.TYPE_FORWARD_ONLY and java.sql.ResultSet.CONCUR_READ_ONLY in the createStatement method of the Connection and also specifying a fetch size of Integer.MIN_VALUE. I compared this with what was JdbcTemplate was doing. Spring also used the createStatement method of the Connection class but without specifying extra flags. This was fine, since TYPE_FORWARD_ONLY and CONCUR_READ_ONLY are used by default. The JdbcTemplate also had a setFetchSize method, cool. However, by looking at the source, I saw that it would completely ignore negative fetch sizes. This is pretty bad. I think it would be much nicer to throw an exception here, since the client calling setFetchSize with a negative value will now know that his fetch size is ignored. On the other hand it was easy enough to create a new subclass which allowed negative fetch sizes. I called it StreamingResultSetEnabledJdbcTemplate.

/**
 * A {@link JdbcTemplate} which will make it possible to mimic streaming Resultset's by allowing negative fetch sizes
 * to be set on the {@link Statement}.
 *
 * @author reik.schatz
 */
public class StreamingResultSetEnabledJdbcTemplate extends JdbcTemplate
{
    public StreamingResultSetEnabledJdbcTemplate(final DataSource dataSource)
    {
        super(dataSource);
    }

    public StreamingResultSetEnabledJdbcTemplate(final DataSource dataSource, final boolean lazyInit)
    {
        super(dataSource, lazyInit);
    }

    /**
  * Prepare the given JDBC Statement (or PreparedStatement or CallableStatement),
  * applying statement settings such as fetch size, max rows, and query timeout.
     * Unlike in {@link JdbcTemplate} you can also specify a negative fetch size.
     *
  * @param stmt the JDBC Statement to prepare
  * @throws java.sql.SQLException if thrown by JDBC API
  * @see #setFetchSize
  * @see #setMaxRows
  * @see #setQueryTimeout
  * @see org.springframework.jdbc.datasource.DataSourceUtils#applyTransactionTimeout
  */
    @Override
    protected void applyStatementSettings(final Statement stmt) throws SQLException
    {
  int fetchSize = getFetchSize();
        stmt.setFetchSize(fetchSize);

  int maxRows = getMaxRows();
  if (maxRows > 0) {
   stmt.setMaxRows(maxRows);
  }
  DataSourceUtils.applyTimeout(stmt, getDataSource(), getQueryTimeout());
    }
}

Using my new class killed all the issues we had. Memory was not a problem anymore and the speed was back. One drawback, if you use my solution, is that some methods like isLast or isFirst are not supported on the ResultSet anymore. If your code invokes them and the ResultSet was created using the described approach, an Exception is thrown.

To come up with some numbers, I created a simple test project using Maven. Feel free to download and test for yourself. You need a local MySQL database, a schema and a database user who can write to this schema. Download the zip file and extract to any directory. Go in src/main/resources and apply the correct database settings in applicationContext.xml. After that open a command prompt, go into the directory where you extracted the zip file to and run mvn test. Maven 2 must be installed of course.

This will run two TestNG unit tests. The first test is called JdbcTemplateTest. The test creates 1,5 million rows in the MySQL database and executes the same retrieval code first using a StreamingResultSetEnabledJdbcTemplate then using a JdbcTemplate. I could not write the unit test with more records as you will hit a OutOfMemoryError for JdbcTemplate otherwise. Here is the JdbcTemplateTest.

/**
 * Tests and measures the {@link JdbcTemplate} and {@link StreamingResultSetEnabledJdbcTemplate}.
 *
 * @author reik.schatz
 */
public class JdbcTemplateTest extends AbstractJdbcTemplateTest
{
    @Test(groups = "unit")
    public void testRun()
    {
        runTestUsingTemplate(getStreamingResultSetEnabledJdbcTemplate());
        runTestUsingTemplate(getJdbcTemplate());
    }

    private void runTestUsingTemplate(final JdbcTemplate jdbcTemplate)
    {
        final String selectStatement = getQuery();

        final AtomicLong count = new AtomicLong();

        final Date before = new Date();

        final String className = jdbcTemplate.getClass().getSimpleName();
        System.out.println("Testing " + className);

        jdbcTemplate.query(selectStatement, new RowCallbackHandler()
        {
            public void processRow(ResultSet resultSet) throws SQLException
            {
                final long i = count.incrementAndGet();
                if (i % 500000 == 0) System.out.println("Iterated " + i + " rows");
            }
        });

        final Date after = new Date();
        final long duration = after.getTime() - before.getTime();

        System.out.println(className + ".query method took " + duration + " ms.");

        assertEquals(count.get(), getNumberOfRecords());

        renderSeperator();
    }

    protected JdbcTemplate getJdbcTemplate()
    {
        final JdbcTemplate jdbcTemplate = new JdbcTemplate(getDataSource());
        jdbcTemplate.setFetchSize(Integer.MIN_VALUE);
        return jdbcTemplate;
    }

    protected JdbcTemplate getStreamingResultSetEnabledJdbcTemplate()
    {
        final JdbcTemplate jdbcTemplate = new StreamingResultSetEnabledJdbcTemplate(getDataSource());
        jdbcTemplate.setFetchSize(Integer.MIN_VALUE);
        return jdbcTemplate;
    }
}

The test data is created in the abstract base class AbstractJdbcTemplateTest.

/**
 * Inserts the test data.
 *
 * @author reik.schatz
 */
@ContextConfiguration(locations = "/applicationContext.xml")
public abstract class AbstractJdbcTemplateTest extends AbstractTestNGSpringContextTests
{
    @Autowired
    private DataSource m_dataSource;

    protected DataSource getDataSource()
    {
        return m_dataSource;
    }

    protected String getQuery()
    {
        return "SELECT * FROM rounds";
    }

    @BeforeClass
    protected void setUp()
    {
        System.out.println("\n\n " + getClass().getSimpleName() + ": \n");

        final JdbcTemplate jdbcTemplate = new JdbcTemplate(m_dataSource);
        
        renderSeperator();
        System.out.println("Dropping table");
        jdbcTemplate.update("DROP TABLE IF EXISTS rounds;");

        System.out.println("Creating table");
        jdbcTemplate.update("CREATE TABLE rounds (round_id INT, player_id INT DEFAULT 0, gaming_center INT DEFAULT 1, last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP);");
        jdbcTemplate.update("ALTER TABLE rounds DISABLE KEYS;");
        jdbcTemplate.update("LOCK TABLES rounds WRITE;");

        final Date now = new Date();

        final StringBuilder sb = new StringBuilder();
        final long records = getNumberOfRecords();

        for (int i = 0; i < records; i++)
        {
            if (i % 100000 == 0)
            {
                sb.append("INSERT INTO rounds(round_id) VALUES(" + i + ")");
            }
            else
            {
                 sb.append(",(" + i + ")");
            }


            if (i % 100000 == 99999 || i == (records - 1))
            {
                jdbcTemplate.update(sb.toString());
                sb.setLength(0);

                System.out.println("Inserted " + i + " rows");
            }

        }

        jdbcTemplate.update("UNLOCK TABLES;");
        jdbcTemplate.update("ALTER TABLE rounds ENABLE KEYS;");

        System.out.println("Insertion took " + (new Date().getTime() - now.getTime()) + " ms");

        renderSeperator();
    }

    protected long getNumberOfRecords()
    {
        return 1500000L;
    }

    protected void renderSeperator()
    {
        System.out.println("============================================================");
    }
}

Even though it is only iterating 1.500.000 million records, you already see a difference between JdbcTemplate and StreamingResultSetEnabledJdbcTemplate. Using the StreamingResultSetEnabledJdbcTemplate the iteration runs for 1526 ms. Using the JdbcTemplate the iteration runs for 5192 ms. Don't forget, you cannot even use the JdbcTemplate if you have 2, 3 or 4 million records in the ResultSet.

JdbcTemplateTest: 

Testing StreamingResultSetEnabledJdbcTemplate
Iterated 500000 rows
Iterated 1000000 rows
Iterated 1500000 rows
StreamingResultSetEnabledJdbcTemplate.query method took 1526 ms.

Testing JdbcTemplate
Iterated 500000 rows
Iterated 1000000 rows
Iterated 1500000 rows
JdbcTemplate.query method took 5192 ms.

Finally I wrote another unit test. The FastInsertionTest iterates 16 million rows using StreamingResultSetEnabledJdbcTemplate. The iteration runs for only 9507 ms. Not bad.

Summary: I did a very very small change with a gigantic effect. I recommend two things to the Spring development team. First, in the setFetchSize method an Exception should be thrown when someone sents in a negative fetch size. Second, in future Spring versions the JdbcTemplate should enable the use of negative fetch sizes, so that the StreamingResultSetEnabledJdbcTemplate becomes obsolete. Maybe something is coming with Spring 3.0.

On a side note, each of the two unit tests creates a lot of test data. In the first version, I wrote a for-loop that fired an Insert statement for every iteration. This was incredibly slow, about 20 seconds for just 100.000 Inserts. I checked a few good resources on the web, like the MySQL documentation or this blog post, and refactored my code.

The JdbcTemplateTest inserts the test data now using a MySQL feature called multiple value Insert. Instead of firing an Insert statement every iteration in the for-loop, I add a new value to multiple value Insert statement. Then every 100.000 iterations I fire the statement. So for 1.5 million rows, I fire only 15 Insert statements.

final StringBuilder sb = new StringBuilder();
final long records = getNumberOfRecords();

for (int i = 0; i < records; i++)
{
    if (i % 100000 == 0)
    {
        sb.append("INSERT INTO rounds(round_id) VALUES(" + i + ")");
    }
    else
    {
         sb.append(",(" + i + ")");
    }


    if (i % 100000 == 99999 || i == (records - 1))
    {
        jdbcTemplate.update(sb.toString());
        sb.setLength(0);

        System.out.println("Inserted " + i + " rows");
    }

}

This runs very fast as you can see in the test output. The 1.500.000 records are inserted in only 2601 ms.

Dropping table
Creating table
Inserted 99999 rows
Inserted 199999 rows
Inserted 299999 rows
Inserted 399999 rows
Inserted 499999 rows
Inserted 599999 rows
Inserted 699999 rows
Inserted 799999 rows
Inserted 899999 rows
Inserted 999999 rows
Inserted 1099999 rows
Inserted 1199999 rows
Inserted 1299999 rows
Inserted 1399999 rows
Inserted 1499999 rows
Insertion took 2601 ms

The FastInsertionTest uses another feature of MySQL called INFILE insertion. This time, the for-loop in my code builds up a gigantic text file which I then import into my MySQL database using the LOAD DATA INFLIE syntax. One drawback of using of this approach is that the number of columns in the file must match the columns in the table you are trying to insert to. In other words, you cannot use the DEFAULT feature of a column. In my table rounds, I have 4 columns but column 2, 3 and 4 have a DEFAULT value. It would be nice if my file would only contain the first column values, as the file would be much smaller in this case. This however is not possible. I have to add column values for column 2, 3 and 4 as well.

final File javaIoTmpDir = SystemUtils.getJavaIoTmpDir();
assertNotNull(javaIoTmpDir);
assertTrue(javaIoTmpDir.exists());

final File dumpFile = new File(javaIoTmpDir, "dump.txt");
if (dumpFile.exists())
{
    assertTrue(dumpFile.delete());
}

Writer out = null;
try
{
    out = new BufferedWriter(new FileWriter(dumpFile));
}
catch (IOException e)
{
    fail();
}
assertNotNull(out);

final long records = getNumberOfRecords();
try
{
    for (int i = 0; i < records; i++)
    {
        out.write("1");
        out.write('\t');
        out.write("1");
        out.write('\t');
        out.write("1");
        out.write('\t');
        out.write("0000-00-00 00:00:00");
        out.write('\n');
    }
}
catch (IOException e)
{
    fail();
}
finally
{
    out.close();
}

jdbcTemplate.update("LOAD DATA INFILE '" + dumpFile.getPath() + "' INTO TABLE rounds");

As you can see in the test output, the 16.000.000 rows are inserted into the MySQL database in only 24852 ms. Awesome.

FastInsertionTest: 
============================================================
Dropping table
Creating table
Inserting 16000000 rows took 24852 ms
============================================================

Parsing Tomcat Access Log for 404 Errors

Yesterday I set up a new dedicated server for a couple of domains. I have Apache with mod_proxy running in front of a Tomcat. It was pretty easy to set up. Since these were quite old domains, I have not really worked with them in a while. I was interested, if I get a lot of 404 errors for these websites. I came up with a nice looking Linux command. Something I remembered from my current job.

Given that you have logging enabled in your Tomcat server.xml configuration, probably like this:

<Valve className="org.apache.catalina.valves.FastCommonAccessLogValve" directory="/etc/tomcats/logs" prefix="access." suffix=".log" pattern="common" resolveHosts="false" />

Prefix and suffix could be different of course but this does not matter. This will create you a daily log file like access.2009-09-24.log. Now to get a nice overview and detect 404 errors fast, run this command:

cat access.2009-09-24.log | cut -f 7,8,9 -d \ | sort | uniq -c | sort -gr

Here are the details. First you display the file contents using cat. This is piped through the cut command. -f 7,8,9 -d \ specifies that you are interested in the fields 7,8 and 9 and the delimiter shall be a whitespace. The syntax for the whitespace delimiter only works that way because another pipe follows. The sort applies some alphabetical sorting. Next pipe is uniq -c which will eliminated duplicates but also adds a count for each unique row. Finally sort -gr will apply numerical sorting based on the result of uniq -c and in reverse order, having the highest number first. Here is some sample output:

6 /includes/css/schufafreie.css HTTP/1.1" 200
6 /images/spacer.gif HTTP/1.1" 200
6 /images/linksline.gif HTTP/1.1" 200
6 /images/banner_oben.jpg HTTP/1.1" 200
6 /favicon.ico HTTP/1.1" 404
5 /includes/js/schufafreie.js HTTP/1.1" 200
5 /images/pfeil_r_grau.gif HTTP/1.1" 200
5 / HTTP/1.1" 200

Loving Maven Webapp Overlays and Jetty Plugin

Alright maybe this is so basic for most of you that I should not blog about this, but yesterday I found out about a Maven feature which I really like. I guess most of you are familiar with the so called "multi-pom" or "multi-module" projects in Maven. This is a Maven project structure, where you have one root pom.xml file defined as parent and then a couple of sub-projects, each with it's own pom.xml. Each of the sub projects, can then be used to create their own build artifacts. This is a nice way to separate the logic of an application into small and reusable deployment artifacts.

Let's say you are writing a standard web-application. To maintain the website content, you have also added some jsp files and classes which function as a "semi-CMS". Additionally you have written a nice database access layer and some useful helper classes for Spring. The straightforward approach would be to create one single Maven project of archetype maven-archetype-webapp. This will produce a WAR file as deployment artifact and you are fine. However, it would be much nicer to have separate deployment artifacts for better maintainability and reuseability. This could be: one JAR file (A) containing all the Spring helper and database classes, one WAR file (B) containing the CMS part and one WAR file (C) containing the real web-application but also referencing the other two deploy artifacts. This setup would have been very easy and common, if A and B produced two JAR deploy artifacts. Fortunately it is also possible to reference one Maven project that produces a WAR file from another Maven project which also produces a WAR file. This is then called WAR overlay. For those interested in the source code, here is a very basic prototype.

Given the above scenario, all you have to do really is to add a dependency in your project C to project B like this:

<dependency>
    <groupId>javasplitter</groupId>
    <artifactId>webappB</artifactId>
    <version>1.0-SNAPSHOT</version>
    <type>war</type>
    <scope>runtime</scope>
</dependency>

This will merge the folder structure of your webappB into the webappC folder structure. The final WAR file, will then be a merge of all static resources (Images, JSP's, CSS, Javascript) of the two combined WAR files from webappB and C. It will also contain all classes of the two Maven projects in the WEB-INF/classes directory of the final WAR. However, in a real life project I had the problem, that if a class within webappC also uses a class from webappB, these classes were not found anymore. I fixed this by adding an additional dependency in webappC like this:

<dependency>
    <groupId>javasplitter</groupId>
    <artifactId>webappB</artifactId>
    <version>1.0-SNAPSHOT</version>
    <type>jar</type>
    <scope>provided</scope>
</dependency>

This might not be the preferred way to fix this but it worked for me. Now for the part, why I like the above setup so much and blog about it. The Maven Jetty Plugin gives you the opportunity to immediately test your web-application within a container. When you run mvn jetty:run it will start a Jetty that loads your webapplication. All the changes done to the static resources (JSP files, Images, CSS, Javascript etc.) are immediately visible when you reload the page in your browser. If your IDE project is set up to compile classes in target/classes (which will be the case if you use IntelliJ and Maven project type), the web application context reloads automatically when you recompile a single class. You can define how often you want the Maven Jetty Plugin to scan the classpath for changes before reloading. Given all this, you can do some real rapid development without long build-deploy cycles between every code change. In my previous set up I had used the Maven Cargo Plugin instead and had it deploy the WAR file into a running Tomcat somewhere else on my computer. This was a big problem, as every time I changed a single character in one of my JSP files, I had to rebuild and redeploy the WAR file. I lost a lot of time.

Here is how I have configured the Maven Jetty Plugin in webappC:

<build>
<plugins>
    <plugin>
        <groupId>org.mortbay.jetty</groupId>
        <artifactId>maven-jetty-plugin</artifactId>
        <version>6.1.12</version>
        <configuration>
            <scanIntervalSeconds>2</scanIntervalSeconds>
            <webAppConfig>
                <contextPath>/</contextPath>
                <baseResource implementation="org.mortbay.resource.ResourceCollection">
                    <resourcesAsCSV>src/main/webappC,../cms/src/main/webappB</resourcesAsCSV>
                </baseResource>
            </webAppConfig>
        </configuration>
    </plugin>
</plugins>
</build>

Some things worth mentioning. I set the classpath scan interval manually to 2 seconds. I use the root context to reach my webapplication in the browser. You have to pass in the two webapp directories containing the static resources as a Resource in the webAppConfig element. The order in which you do this might be important. I have a Servlet in my webappC which loaded on startup and read a file path out of the ServletConfig (ServletContext). If I had webappB before webappC in the above example, it would load my Servlet using the webappB ServletConfig which was a big problem because all the file paths were wrong that way. Finally note the resourcesAsCSV element. In the documentation of the Maven Jetty Plugin you are being told to use just a resources element but this will not work properly. You will end up with an error similar to Cannot assign configuration entry 'resources' to 'class [Lorg.mortbay.resource.Resource;' - so use resourcesAsCSV instead.

I would also like to add that developing a Grail webapplication using Maven gives you an even faster rapid development experience. I used the Maven Grails Plugin for one project, which also uses a Jetty (mvn grails:run-app) to test the deployment artifact. This Jetty however, was able to detect class changes automatically and much faster. I had not to manually compile in my IntelliJ IDEA anymore, just saving the modified source file in IntelliJ would immediately update my web-application context and the changes were visible. I have not checked how this behavior was implemented but obviously some very smart Grails people came up with a great idea.