ASDF for the slightly confused

Well, you have a shiny new lisp compiler/interpreter/environment on your machine, and like an eager and smart newbie you want to do something with it. Right there and then many lisp newbies go SPLAT and end up as a bug on a windscreen, later to be scraped off and fed to Guido's pet snake. So here is ASDF for confused newbies (a very different proposition from a dummy, who should go back to reading bright yellow paperbacks and programming Java in the enterprise shop window).

ASDF is about 500 lines of lisp source, and a dot asd file is the lisp equivalent of the ubqutious make; you can't do anything in a serious modern lisp environment without it, so lets take you through it step by step. An asd file describes a system that contains modules and components. Components are usually individual files. Modules are collections of files, usually in subdirectories. Let's look at an example dot asd file: this one is for cowl - a simple opengl gui written by William Robinson who is working on Cityscape, a promising
game found at this address.

(defsystem #:cowl-ftgl
 :depends-on (#:cffi)
 :description "FTGL Common Lisp wrapper for Cowl."
 :components ((:module "src"
              :components 
             ((:file "cowl-ftgl")))))

The defsystem form then defines a system with one component which is a module called src which in turn contains one component in a file called cowl-ftgl, which will be a lisp file, naturally: since it's in a module called src, it will be loaded in the src directory below the directory that holds the dot asd file. Sometimes it is necessary to wrap the defsystem form in it's own package using defpackage and a throwaway package name. We do not do that here because we do not define any symbols that we would need to refer to externally, but we would need to if we were defining new component classes or methods to be used in the system, examples of which will follow later in the article.

Components, modules and systems are defined as CLOS objects in asdf.lisp, so of course they can be operated upon with generic functions, and there are quite a few that come with asdf. The key one is named with Arc-like gnomicness: "oos" - which I imagine is short for "Operate on System". It takes at least two parameters: the operation to perform and the system to perform it on.

The operation you are most likely to perfom is load-op: this loads and compiles a system, and the form is simply:

(asdf:oos 'asdf:load-op 'system-name)

Other operations are compile-op, and test-op: the latter will only have meaning if the author of your system has provided tests, as some packages do.

ASDF will search for systems using the list of functions referred to by *system-definition-search-functions* used to search for .asd files in the filesystem. By default this contains only one function, (sysdef-central-registry-search) which is perfectly adequate for file systems with symbolic links. The modus operandi on such system is to have a share/common-lisp/systems directory populated with symbolic links to dot asd files which may be downloaded and untarred in any part of the fs tree to which the user has access. Then the (sysdef-central-registry-search) scans the list of directories referred to by *central-registry* for dot asd files containing the system which it needs to operate on.

To put it another way, procedurally this translates to:

1. Download your asdf package


2. Unzip - say to ~/thinngy/cl-blab


3. Note that ~/thinngy/cl-blab/cl-blab.asd is the system.


4. Create a ~/share/common-lisp/systems directory for systems
   

5. cd ~/share/common-lisp/system
   

6. ln -s ~/thinggy/cl-blab/cl-blab.asd cl-blab.asd
   

7. now fire up your lisp and enter..
   

   
   ;; for systems that don't do (require 'asdf)
   (load #"/path/to/asdf.lisp")
   
   ;; the / on the end of the path is important, don't forget it
   (push #"/home/me/share/common-lisp/systems/" asdf:*central-registry*)
   
   (asdf:oos 'asdf:load-op 'cl-blab)
   

For Windows the situation is sligtly more complicated: either you write your own search function *or*, more practically apply a patch that lets you use shortcuts in place of symlinks.

You'd be right to point out that this is on the laborious side, but the purpose of this article is to demonstrate how asdf *works*. There are two packages for automating lisp package and asdf system installation: asdf-install and clbuild, which I encourage newbies to investigate, in the usual sadistic exercise for the reader.

Why go through all this palaver with asdf? Well the beauty of asdf is that systems, components and modules are in fact CLOS objects, so we can do tricks like this:

(cffi:load-foreign-library 
 (make-pathname 
  :name "cftgl" 
  :type "so" 
  :directory (directory-namestring (asdf:component-pathname 
                                    (asdf:find-system :cowl-ftgl)))))

..which loads "clftgl.so" which is found in the same directory as the dot asd file containing the system definition. Highly useful when distributing a C shared lib that wraps a C++ library so that it can be called into from Lisp.

The fact that source files are component, lets us write specialist methods for different kinds of source file such as c-source-file, java-source-file, html-file (all defined in asdf.lisp) allows asdf to work as a make replacement, operating on non-lisp source, thus:

(defmethod output-files ((op compile-op) (c c-source-file))
    (list (make-pathname :type "o" :defaults
                                                 (component-pathname c))))

(defmethod perform ((op compile-op) (c c-source-file))
    (unless
       (= 0 (run-shell-command "/usr/bin/gcc -fPIC -o ~S -c ~S"
                               (unix-name (car (output-files op c)))
                               (unix-name (component-pathname c))))
     (error 'operation-error :operation op :component c)))

(defmethod perform ((operation load-op) (c c-source-file))
    t)   

There is an extended example of this kind of thing where asdf is extended to work with a fortran to lisp converter here.

To conclude: like many things in Lisp, asdf is prickly for newbies and takes time to get to know and use well. However, when you do you can do things with it for which you'd normally need another build system, independent of your language. As Brucio would observe: Lisp does not have this limitation and Common Lispers laugh at things like ant.