Pant - Python powered Ant
I am not crazy about Ant, although I like the idea behind it: common, widely accepted, cross-platform building tool. In my humble opinion, writing and maintaining Ant build files is not an easy task as it should be.
What bothering me the most is that Ant build file is just a (XML) set of tasks, without power of anything that might look like 'building language or script'. I find unnecessary difficult to accomplish some specific project configuration - and, at end, what is lost is the visibility of whole building process. Due to its nature, Ant build files easily becomes huge, repeated, and, therefore, hard to maintain. For example, our Jodd project, that is not complex from building point of view, has total of more then 700 lines in its build.xml! Although size can be reduced with Ant modules and multiple smaller files, such modular structure (that I've made several times in the past) is not well understandable by IDEs. Moreover, having one build file makes life of (continuous) integration and deployment easier.
It seems that I am not the only one who thinks like that, since recently few alternative approaches have gain popularity. Although this may sound contradictory, I would still like to stick with Ant mainly because it is so well supported by all IDEs and integration tools and, at the end, I am familiar with it. So I start looking for solution in other direction: not to replace Ant, but to generate the build.xml using some solid scripting os-independent language, such as Python.
build.xml generator
Following requirements were followed during generator development: main build script must be small in size and very, very visible (i.e. readable). Generator should use string templates for encapsulating logical parts of build.xml (e.g. various tasks), but it still must be able to output to build.xml from any Python method, programmatically. And finally, generator must be able to use the full potential of Python language.
The current version of generator consists of following parts: pant.py (generator core), build templates and task methods (for various Ant tasks) and build.py (project build configuration). Explanation of each part follows.
Build templates
Build templates are simple XML snippets that represents logical parts of Ant build.xml. Build templates may be populated with parameters and arguments. Parameters are any Python expressions surrounded with $[]. Arguments are marked as %ARGn% (where n is ordinal number of an argument) and are simply replaced with their provided values. Of course, one template may contain any number of parameters and arguments
Here are few examples of build templates:
<path id="lib.$[libName]">
<fileset dir="${basedir}/lib/$[libName]" includes="**/*.jar"/>
</path>
pack.xml - defines packing target using arguments
<target name="pack-%ARG1%" depends="%ARG3%" description="packs distribution jars">
<delete file="${dist.dir}/%ARG2%-${prjVersion}.zip"/>
<zip destfile="${dist.dir}/%ARG2%-${prjVersion}.zip" basedir='.'
includes="%ARG4%" excludes="%ARG5%"/>
</target>
build.py
Basically, build.py is a Python program enhanced by template methods. Template method is a Python method with the same name as name of template file (without the xml extension, of course) and that may have any number of arguments. So, visually, there is no difference between template and regular Python method. Difference is that template method doesn't need to be defined - it is enough if there is a template file with the same name. Generator executes template methods by invoking rendering of template files with provided arguments. Such template methods are also called as implicit template methods, since they really do not exist in Python code.
Of course, although enhanced with parameters and arguments, template files are often not enough. There are cases where some global variables should be defined, or where input arguments should be parsed before passing to template, or where template has to be invoked several times for all input arguments and so on. In such cases, template method can be defined explicitly in Python, so generator will invoke the Python method instead of template file. Such methods are called explicit template methods.
Generator offers methods for outputting some content directly to resulting build file and to invoke rendering of any template file with some provided arguments. Explicit template method may use any technique to prepare and append content to the resulting build file.
Explicit template methods are stored in separate file, tasks.py, together with template methods, since they together represents chunks of logic for building build.xml.
Here is how build.py might look like (this is stripped version of actual Jodd's build.py):
# settings
prjName = 'Jodd'
prjDescription = 'Jodd - generic purpose open-source Java library and frameworks.'
prjVersion = '3.0.1'
# ant
project_header()
lib('asm')
lib('junit, emma')
lib('hsqldb, h2')
lib('mail')
lib('servlets')
module('jodd')
module_compile('production', 'jdk5', 'mail, servlets')
module_compile('test', 'jdk5', '#production, mail, servlets, junit, emma')
module_build('production, test')
module_javadoc('production')
module_test('jodd.TestJodd')
module_dist('jodd.Jodd')
module_findbugs()
module('jodd-wot')
module_compile('production', 'jdk5', '>jodd.production, servlets, asm')
module_compile('test', 'jdk5', '>jodd.production, #production, asm, hsqldb, h2, junit, emma')
....
project_task('build', 'jodd, jodd-wot, jodd-gfx')
project_task('javadoc', 'jodd, jodd-wot')
project_task('emma', 'jodd, jodd-wot')
project_task('dist', 'jodd, jodd-wot, jodd-gfx')
project_task('findbugs', 'jodd, jodd-wot')
project_clean()
project_target('release', 'clean, build, javadoc, emma, findbugs, dist', 'creates full release')
pack('dist', 'jodd', 'dist', '''
${jodd.jar}
${jodd-wot.jar}
file_id.diz
''', '')
project_footer()
This is the build script I am working with and that is for me much more visual then Ant's build.xml plus it comes with the power of Python language. Nothing much to explain, there are several sections: libraries definition, modules definitions, project tasks... You might notice some special notation used, such in line #28. These are just mine shortcuts for referencing external libraries and classpaths.
This is just one example how build.py may look like! All above methods are not hard-coded by generator. One may use different methods and templates in the behind! They can differ from project to project, as well.
pant.py
This is the actual generator - everything else is just related to the project. Generator is rather simple, containing just above hundred of lines, written in few hours. Here is only the interesting part:
...
# evaluates single string by replacing all properties $[] with appropriate values
# and all arguments with %ARG%
def evaltmpl(string, list = None):
# replace arguments
if list != None:
while True:
i = string.find('%ARG')
if i == -1:
break
j = string.find('%', i + 4)
arg = string[i + 4:j]
arg = str(list[int(arg)])
string = string[:i] + arg + string[j+1:]
# replace properties
while True:
i = string.find('$[')
if i == -1:
break
j = string.find(']', i)
data = eval(string[i+2:j])
string = string[:i] + data + string[j+1:]
return string
# invokes string by replacing all properties $[] with appropriate values
def out(str):
global result
result += evaltmpl(str)
# reads template file and outputs the content
def tmpl(*args):
args = list(args)
if len(args) < 1:
raise AttributeError("No template for filename")
filename = str(args[0])
tmpl = readFile(os.path.join(tmplRoot, filename + '.xml'))
global result
result += evaltmpl(tmpl, args)
# checks if template exist
def tmplExist(filename):
return os.path.isfile(os.path.join(tmplRoot, filename + '.xml'))
#-------------------------------------------------------------------------- main
result = ''
# run tasks.py
data = readFile(os.path.join(tmplRoot, 'tasks.py'))
exec(data)
# run build.py
build = readFile('build.py')
data = ''
f = open('build.py', 'r')
for line in f:
start = 0
for c in line: # count starting tabs
if c == '\t':
start += 1
else:
break;
n = line.find('(', start)
if n != -1:
method = line[start : n]
try:
eval(method) # is defined method?
except:
if tmplExist(method): # if not a method maybe it is a template
line = line[0 : start] + "tmpl('" + method + "', " + line[n + 1:] # template
data = data + line
f.close()
exec(data)
writeFile('build.xml', result)
Generator starts with running tasks.py - where all explicit template methods are defined. Then, it reads build.py and iterates it line by line. For each line, generator tries to figure if it is a method call (line #67). If it is not some defined method, maybe it is a implicit template method - if template file with the same name exists (line #69). If so then it will be converted to invocation of method for rendering templates (line #31). Rendering method just prepares arguments and then populates the template using method in line #4. There is also method out() for direct output to generated build.xml. And that is all.
Usage
I made a simple shell script that executes build.py, if such file exist, before the Ant call. That way build.xml is getting generated each time before the usage. You can check how it works in Jodd library, starting from version 3.0.1.
Contents
Read about...
