ASDF or other module system independent of home and site configurations

Question

I'm trying to find out the best practices for real-world application development. I'm having trouble understanding how to properly configure third-party libraries for deployment as a standalone package. It seems that ASDF-INSTALL and ASDF are intended to install libraries to either home or site, changing the state of the development platform. That's fine if I were developing a server side application and I wanted to administer these dependencies for the entire CL installation on the box.

But what if I wanted to create a standalone application, deployed via install scripts, and running alone in its own CL instance? I would want to avoid changing the CL instance or target system configuration for any other applications (i.e. overwrite other copies of libraries that other applications depend on). In Java, all I have to do is create an include directory that contains those libraries and set my CLASSPATH. How do I get the same level of isolation in CL, assuming that I don't install my own CL runtime, and instead use what's on the system?

It occurs to me that this is a common issue with all dynamically compiled/interpreted languages, since the runtime will have a lifetime longer than any particular application that the runtime runs (i.e. Ruby or Python), and applications will share the same library load state. I don't have any experiences with such languages, so the best practice is probably staring me in the face.

IDK if this would be implementation specific, but I am running Clozure CL.

EDIT:

Ramarren:

I'll check out Mudballs. Thank you.

Installing libraries is the opposite of what I want, since installing implies modifying the host system state. I think the key must be in your last paragraph. How do you create the startup script to set central-registry to an isolated directory? Is it possible to use ASDF-INSTALL to fetch stuff into said directory? And how do you bootstrap the whole thing if the base image of your CL implementation doesn't include ASDF (Clozure has ASDF by default, but how would CLISP do it)?

I'm thinking in terms of a dev team too. After I create a new CL project stub and do that initial commit to CVS or SVN, how do other devs check it out to their local environment and work with it? Even assuming that everyone has ASDF in their profile/site startup, other devs may have a different set of libs in their central-registry. We shouldn't have to sync up just to work on a project together. There's got to be a clean way to launch an project specific instance of the CL runtime from Emacs/SLIME and load exactly what's specified in the project, no more, no less.

If there's any best practice resources online, in CL or any other language, I'll be glad to roll my own solution and open source it.

Luis:

SAVE-APPLICATION is good for deployment, but not for the multiple-dev project stub I outlined.

EDIT 2:

vatine:

Version dependency is precisely why this is a problem. If I were developing a Perl or Ruby web app, then I can depend on the existence of a webadmin to manage these dependencies. Developing apps for retail or for small-to-mid-size businesses where Lisp is alien technology (and I cannot convince them to "skill up" their IT org to admin it), that approach is unacceptable.

I was able last night to kind of get what I want by creating a project-level .lisp file that loads its own project-specific ASDF instance, set a project-local central-registry, and manually downloading dependent libs (without ASDF-INSTALL, which was a painful cascade of dependencies for just CLSQL and Weblocks). It still isn't ideal from a dev tools standpoint, since I had to remove all customization from my home and site for both SLIME and Clozure itself. Also, shared dependencies aren't resolved either (CLSQL and Weblocks both use MD5).

Java has class loader isolation, which is how it solves the version dependency issue. Then there's the separate issue of how to get the libs you want into the project (a la Maven). The former is a core language issue; the latter has to do with tools. I'm going to hack together a SLIME extension that does what ASDF-INSTALL does to a project include directory (a la Maven), and modifies lib source code to intercept defpackage calls to somehow prepend a gensym string to enforce isolation. There's plenty of holes in this approach, I know, and I don't know enough about the package spec to know how deep I can bury this.

I don't know anything about Python, but I do know that retail-level apps exist; I use MusicBrainz Picard all the time. I'll look into how Python does it.

Answer 1

First, ASDF has nothing to do with installing libraries. It is simply a tool to compile and load a set of files in an order defined by dependencies between them. ASDF-INSTALL installs things, and it uses ASDF to determine dependencies (I think), but there are other mechanism for installing, like clbuild, extracting and symlinking by hand, or recent development Mudballs, which replaces ASDF as well.

Special variable asdf:*central-registry* holds a list of directories which are searched for asdf systems. It usually contains a patch to a directory with symlinks to actual system definition files, but this is not necessary, as it can just contain every directory with a system definition file.

This variable will be most of the time populated by startup scripts with system/site library registry, but you can change it to whatever you like, including isolated local directory. Regarding the one before last paragraph, note that it is not particularly common to run many applications on the same CL runtime. Normally some default bare core is launched and then libraries and application is loaded and run, and then shut down when it finishes.

EDIT:

ASDF is just a library and its variables are like any other lisp variable. In the case you describe you likely want to suppress the system/user startup script, which is implementation specific (in CCL it apparently is -n or --no-init command line argument), an then manually load asdf.lisp, (setf asdf:*central-registry* (list #p"pathtoprojectcentralregistry")). Or possibly create a function to recursively scan project directory and push all subdirectories with .asd files there.

Answer 2

You want CCL:SAVE-APPLICATION.

Answer 3

In general, I'd say that anything installed as a 3rd-party library SHOULD modify the host system state (as it were). Stuff developed locally can be handled by several methods, what I use is a symlink farm for myself (with an attendant shellscript to rebuild the farm when I need to). Lisp projects are checked out into their own directories. Inter-dependent projects are likewise checked out that way, reference each other in their ASDF system definitions and are picked up via the symlink farm.

The one possibel problem I see with this model is if you suddenly find that part of your support libraries depend on library BLAHONGA, checked out of CVS before 2008-12-01 and other parts of your support libraries also depend on BLAHONGA, but require a feature only available in CVS from 2008-12-29. I guess version dependencies may be a possibility too.

Answer 4

For your development environment, you can always check in the external libraries somewhere, and have each developer check that out into the asdf:*central-registry*.

Another option is to have a single dev box, and have each developer run a Lisp + Swank there.