How do I iterate through a directory in Common Lisp?

Question

I'm using OpenMCL on Darwin, and I'd like to do something like:

(loop for f in (directory "somedir")
  collect (some-per-file-processing f))

But I can't get directory to return anything other than NIL, and I can't seem to find any good explanation online (other than "its different for each system").

Any pointers?

Answer 1

Does your pathname specification contain a wildcard? Common Lisp's pathname stuff is somewhat hard to grasp at first - at least for me it was... As the CLHS states on the directory function:

If the pathspec is not wild, the resulting list will contain either zero or one elements.

In order to have your pathname include a wildcard, you might try the make-pathname function, like

(directory (make-pathname :directory '(:absolute "srv" "hunchentoot") :name :wild :type "lisp"))

Or even

(directory (make-pathname :directory '(:absolute "srv" "hunchentoot") :name :wild :type :wild))

I found the CL-FAD library a great help for dealing with pathnames and the file system. In particular, its list-directory function might be easier to use than the plain standard directory function.

Answer 2

There are basically two ways to specify pathnames:

• using strings

Strings are obviously depending on the platform: Unix syntax vs. Windows syntax for example.

"/Users/foo/bar.text"  is a valid pathname
"/Users/foo/*/foo.*"   is a valid pathname with two wildcards

You can create a pathname object from a string:

? (pathname "/Users/bar/foo.text")
#P"/Users/bar/foo.text"

The #p above assures that a pathname object (and not a string) is created, when you read it back.

? #P"/Users/bar/foo.text"
#P"/Users/bar/foo.text"

So, internally Common Lisp works with pathname objects, but it lets you use normal strings and makes pathname objects from them if needed.

When Common Lisp sees a pathname that has not all components specified (for example the directory is missing), then it fills in the components from the pathname object that is the value of the variabel *DEFAULT-PATHNAME-DEFAULTS* .

With the function DESCRIBE you can look at the components of a pathname (here Clozure CL):

? (describe (pathname "/Users/bar/*.text"))
#P"/Users/bar/*.text"
Type: PATHNAME
Class: #<BUILT-IN-CLASS PATHNAME>
TYPE: (PATHNAME . #<CCL::CLASS-WRAPPER PATHNAME #x3000401D03BD>)
%PATHNAME-DIRECTORY: (:ABSOLUTE "Users" "bar")
%PATHNAME-NAME: :WILD
%PATHNAME-TYPE: "text"
%PHYSICAL-PATHNAME-VERSION: :NEWEST
%PHYSICAL-PATHNAME-DEVICE: NIL

• using the Lisp functions creating pathname objects

MAKE-PATHNAME is the function and it takes a few keyword arguments to specify the components.

Sometimes it is also useful to create a new pathname based on an existing one:

(make-pathname :name "foo" :defaults (pathname "/Users/bar/baz.text"))

If you use DIRECTORY it is useful to use a pathname with wildcards. DIRECTORY then will return a list of matching pathnames. The name 'DIRECTORY' is slightly misleading, since DIRECTORY does not list the contents of a directory, but lists the matching pathnames for (usually) a pathname with wildcards. The wildcards can match a sequences of characters in components like /foo/s*c/list*.l*". There is also the wild card ** , which is used to match parts of a directory hierachy like /foo/**/test.lisp , which matches all files test.lisp under the directory foo and its subdirectories.

(directory "/Users/foo/Lisp/**/*.lisp")

Above should return a list of all 'lisp' files in '/Users/foo/Lisp/' and all its subdirectories.

To return the .c files in a single directory use:

(directory "/Users/foo/c/src/*.c")

Note that DIRECTORY returns a list of pathname objects (not a list of strings).

? (directory (make-pathname
               :name "md5"
               :type :wild
               :directory '(:absolute "Lisp" "cl-http" "cl-http-342" "server")))
(#P"/Lisp/cl-http/cl-http-342/server/md5.lisp"
 #P"/Lisp/cl-http/cl-http-342/server/md5.xfasl")

Above uses a pathname object that is created by MAKE-PATHNAME. It returns all the files that match /Lisp/cl-http/cl-http-342/server/md5.* .

This is the same as:

(directory "/Lisp/cl-http/cl-http-342/server/md5.*")

which is shorter, but depends on the Unix pathname syntax.

Answer 3

It doesn't work for me in Ubuntu 9.04. For instance: (directory "/usr/*.*")

(#P"/usr/lib/")

I don't understand why it only shows /usr/lib when there are several other directories.

(directory "/usr/local/*.*")

NIL

It doesn't show any of the directories in /usr/local

Answer 4

The modern Common Lisp library implementing directory listing is IOLIB.

It works like this:

CL-USER> (iolib.os:list-directory "/etc/apt")
(#/p/"trusted.gpg~" #/p/"secring.gpg" #/p/"trustdb.gpg" #/p/"sources.list"
 #/p/"sources.list~" #/p/"apt-file.conf" #/p/"apt.conf.d" #/p/"trusted.gpg"
 #/p/"sources.list.d")

Note that no trailing slash or wildcards are required. It is very robust and can even process file names with incorrectly encoded unicode characters.

Differences compared to CL-FAD:

• The objects you get are IOLIB file paths, a replacement for CL's pathnames which is closer what the underlying OS does.
• IOLIB implements its routines using CFFI, so it works the same on all Lisp implementations (provided IOLIB has a backend for the operating system), in contrast to CL-FAD, which tries to abstract over the implementation's DIRECTORY function with all its quirks.
• In contrast to CL-FAD, iolib deals correctly with symlinks (one major issue with CL-FAD that makes it virtually unusable on platforms other than Windows IMHO).