SBT and faster RPM packaging

We do a lot of Scala coding nowadays and I am trying to introduce SBT as build tool to all our new Scala projects. When we deploy these applications to Amazon EC2 nodes, we use Chef Solo and the Instance User Data feature to install an RPM file. We don’t use custom AMI’s. The RPM file is hosted in S3 and made available as package via this yum plugin. Each time we build our project via our continuous integration server (Bamboo), a new RPM package is created and uploaded to S3.

It became more and more of a problem that building that particular application in Bamboo took a long time. The build plan ran for more than 10 minutes. So yesterday I spent some time to make it build a bit faster.

First of all I have to say it is pretty lame that the SBT plugin is broken in Bamboo since version 4.4.3 and no one from Atlassian is interested in fixing it since August 2013! I tried to fix the Bamboo plugin myself but Atlassian has some non-public Maven repositories so I couldn’t even build it. Given that the top four Java/Scala build tools are Ant, Maven, Gradle and SBT you could also say that Bamboo is somewhat 25% broken currently. Anyway a workaround is to use the Script Task in a Job and run SBT, which is what we do currently.

When I looked at our build there were basically two steps which took a long time. First we were creating a big one-jar (also called uber-jar sometimes). This is a single jar file that contains all compiled classes from all dependencies as well as our own classes. To create the uber-jar we used the sbt-assembly plugin which can run for a bit if you have a lot of dependencies. But actually you don’t need to have a single big jar file as you can add an entire directory to the Java classpath when starting an application. So I switched to a plugin called sbt-pack which dumps the jar files of all managed dependencies into a folder under target along with your project jar. This folder is then used later when building the RPM. Not using the sbt-assembly plugin to create a single uber-jar already saved us about 2 minutes of buildtime.

The second change was addressing creation of the actual RPM package. Previously we were using SBT native packager to assemble the RPM file. Unfortunately it was also not running very fast. Another big issue in Bamboo was that the sbt-native-packager logs some stuff on Std Error. This failed the build because Bamboo is scanning the build log for errors. (Our hack around this issue was to write a SBT task that logs 250 lines of “Build Successful” into the Bamboo log - what a mess). Today the RPM is build using fpm. On your Bamboo server you need to install fpm which is a Ruby Gem (gem install fpm). Then install Python and the fabric library.

And here is how we use fabric and fpm. In the root of your Scala project create a folder called build. Inside this folder store the following file:

	#!/usr/bin/env python
	import os
	import os.path
	import shutil
	from fabric.api import *
	versionNumber = ""
	with open('build/version.txt', 'r') as f:
	versionNumber = f.readline()
	RPM_INCLUDE="projectname"
	INSTALL_DIR = "/opt"
	WORK_DIR = os.getcwd()
	BUILD_NUMBER = os.getenv('BUILD_NUMBER', '1')
	@task
	def build():
	""" Creates a build with an optional RPM package """
	print("Starting build %s... in %s" % (BUILD_NUMBER, WORK_DIR))
	# folder created by the sbt-pack plugin
	pack_dir = os.path.join(WORK_DIR, 'target/pack')
	# folder containing all jars
	lib_dir = os.path.join(pack_dir, 'lib')
	# project folder which be installed to /opt/projectname later
	opt_app_folder = os.path.join(pack_dir, 'projectname')
	if not os.path.exists(opt_app_folder):
	print("Creating %s" % opt_app_folder)
	os.makedirs(opt_app_folder)
	# move all jars into the project folder
	for file in os.listdir(lib_dir):
	print("Moving %s to %s" % (file, opt_app_folder))
	shutil.move(os.path.join(lib_dir, file), opt_app_folder)
	print("Creating RPM...")
	# delete old RPM files if any
	for file in os.listdir(WORK_DIR):
	if ".rpm" in file:
	os.remove(file)
	# creates an RPM using fpm, aborts if problems
	out = local('fpm -a all -s dir -t rpm -C %s --vendor "VENDOR NAME HERE" --prefix %s \
	--license "(C) 2011-2014, License Statement Bla Bla. All Rights Reserved." \
	--url "http://www.yourcompany.se/" --maintainer "<mail@address.com>" \
	--description "Project Name" \
	-n "packagename" -v %s --iteration %s %s' % (pack_dir, INSTALL_DIR, versionNumber, BUILD_NUMBER, RPM_INCLUDE))
	if out.failed:
	abort("RPM creation failed, check logs")

view raw fabfile.py hosted with ❤ by GitHub

You probably want to adapt projectname, packagename and the fpm settings to match your own project. To invoke the script during a build create a Script task in Bamboo that executes: fab -f build/fabfile.py build. When the Script is executed from Bamboo it is looking for a file called version.txt in the build folder. The file version.txt need to be created upfront via SBT to propagate the project version to the Python script. This is what the custom rpmPrepare task does.

	lazy val branchName = settingKey[String]("The name of the Github branch which will prefix the RPM")
	branchName := sys.props.getOrElse("branchName", sys.env.getOrElse("branchName", "master"))
	lazy val rpmPrepare = taskKey[Unit]("Executes steps which are needed before the RPM can be build")
	rpmPrepare := {
	val dir = Paths.get("build")
	val versionFile = dir.resolve("version.txt")
	Files.deleteIfExists(versionFile)
	Files.createFile(versionFile)
	Some(new PrintWriter(versionFile.toFile)).foreach{p => p.write("%s_%s".format(branchName.value, version.value)); p.close()}
	}

view raw build.sbt hosted with ❤ by GitHub

The rpmPrepare task reuses a SettingKey called branchName which contains the name of the branch in Github. The name of the RPM package will contain the branch name, so that you can build multiple branches of the same project in Bamboo in parallel without having to worry about version clashes. The branchName Setting in SBT is retrieved via either a system property or an environment variable called “branchName”. This variable is set from Bamboo. Each build plan in Bamboo is made of individual tasks and for a task you can set individual environment variables. So just add -DbranchName=${bamboo.repository.branch.name} and Bamboo will feed in the Github branch name into the task.

So after running the Python script you will have the RPM file in the WORK_DIR folder. For running Java command-line applications we use Supervisor. Here is an example how to invoke a Main class given that the RPM installs your project in /opt/projectname.

	[program:projectname]
	stdout_logfile=/var/log/projectname/projectname.log
	command=/usr/java/latest/bin/java -cp "/opt/projectname/*" me.MainClass
	directory=/opt/projectname
	autostart=true
	autorestart=true
	redirect_stderr=true

view raw project.ini hosted with ❤ by GitHub