First I thought I had a leak somewhere, which prevented files and sockets from getting closed properly. Actually this was not the main problem. Since all Tomcats were running as the same user, and I had not touched the open file limit for this user, the default maximum of 1024 in Ubuntu 9.10 server was way too little. I checked how many files I had open for this user.
ps aux | grep tomcat
Then for every PID I ran
lsof -p PID | wc -l
I had cleaned the logs and rebooted already. The combined result was that I already was scratching the 1000 mark for all Tomcats after rebooting. Very thin ice. To make a long story short, here is how to change the maximum open file limit on Ubuntu 9.10 server.
First you edit /etc/security/limits.conf and add your new limit for the user running Tomcat. In my case the user was called virtual:
virtual hard nofile 5120
virtual soft nofile 4096
In addition to that, edit the file /etc/pam.d/common-session and add
session required pam_limits.so
done! Reboot the machine, then verify the changes running