Library dowloader and manager for Common Lisp.
Table of Contents
- 1 README
- 2 Documentation
- 3 Hacking
- 3.1 Separate library and shelf definition
- 3.2 Invent way of updating tarball libraries
- 3.3 Detect include loops
- 3.4 Automatic shelf template generation from .asd files
- 3.5 Update shelf files themselves, not only their content
- 3.6 Do update shelf contents
- 3.7 Convenient using and unusing shelved libraries
- 3.8 Check authenticity of downloaded libraries.
1 README
One problem I always had when hacking CL were libraries. And it was not the lack of them—contrary to popular belief, there is quite a lot of Lisp libraries. The real problem is how to comfortably find and manage them. For finding, CLiki and Common Lisp Directory work great. Managing, however, has to be done by hand. There was an attempt to solve this problem, ASDF-Install, but it relies on a very inconvenient (at least for me) assumption: that libraries are versioned .tar.gz files. In other languages it may be true, but Lisp is different (as usual). Many libraries are in a `permanently rolling mudball’ state, where the current version is accessible only by various version control systems, and tarballs appear rarely and quickly become outdated.
There are other library collections and collectors:
- Lisp in a Box and Peter Seibel’s variation Lispbox, which is complete development environment for newbies that includes Emacs and Lisp implementation;
- Edi Weitz’s Starter Pack, which is only for LispWorks on Windows;
- clbuild, which is just a shell script with hard-coded repos and tarball addresses.
That is what CL-Librarian aims to be: a manager for Common Lisp libraries, focused on libraries updated dynamically by version control systems and targeted at power users. It handles downloading such libraries, and keeping them up-to-date. This may seem like a trivial task, but after maintaining an UnCommon Web installation for over a year I am quite sure it’s not. Library management could use some automation.
CL-Librarian’s primary platform is SBCL on Linux. It should work with other Lisp implementations (for the only non-portable part: executing external commands, it uses the Trivial Shell library, which is portable). It should work on other Unix systems, including OS X. Porting it to Windows should not be very hard, but I am not a Windows expert, so most probably it won’t work out of the box.
CL-Librarian is © 2007, Maciej Pasternacki <maciekp@japhy.fnord.org> and available on BSD license without advertisement clause. See file =COPYING= for details.
1.1 Note of warning
As of now, this code is under heavy development. Both implementation and interface may change. I try to document fixed interfaces, but I cannot promise much at the moment. When you upgrade Librarian, please read the commit description. You have been warned.
1.2 Download
CL-Librarian lives in a darcs repository. To dowload the source, type:
$ darcs get http://www.pasternacki.net/repos/cl-librarian/
It depends on ASDF, SPLIT-SEQUENCE and
Trivial Shell.
Dependencies can be managed independently, or by Librarian itself.
To manage Librarian’s dependencies in Librarian, use included
bootstrap.sh script:
cl-librarian$ sh ./bootstrap.sh
It will download necessary libraries and create necessary symlinks.
Librarian can be also downloaded with ASDF-INSTALL.
1.3 Required binaries
CL-Librarian makes extensive use of system binaries. The only
thing that is Linux-specific on purpose (temporarily, I hope) is
explicit use of the ln -s command. Other required binaries
are:
-
wget(might be replaced by any other command that downloads file from a given URL) -
darcs -
cvs -
svn
ln, of course) are ported
to most of commonly used systems.
1.4 Contact
CL-Librarian is written and maintained by Maciek Pasternacki <maciekp@japhy.fnord.org>. Feel free to contact me by e-mail or jabber (JID is the same as e-mail). After sending me an e-mail from an address previously unknown to me, a confirmation message will be sent back; only after the sender replies to this message I will see the original mail.
There is no mailing list.
2 Documentation
CL-Librarian is just an ASDF library. Once it gets its
prerequisites, either with bootstrap.sh, or by loading the
necessary libraries by hand, it doesn’t need any further set up
other than being loaded:
(asdf:operate 'asdf:load-op :cl-librarian)
2.1 Library shelf model
CL-Librarian is designed around the concept of a library shelf. A shelf is a single unit consisting of several ASDF-defined libraries managed by the Librarian. The Librarian doesn’t bother moving around single libraries; only whole library shelves, which are self-contained sets of libraries, are downloaded or updated at a time.
A shelf specification is a single Lisp file, with the .shelf
extension, containing a DEFSHELF form. They resemble ASDF’s .asd
files with DEFSYSTEM forms, which is intentional. Shelves
managed by Librarian are located in the shelves subdirectory.
CL-Librarian is capable of downloading shelves from the network or
copying them from other files. The shelf contents are a directory,
located in shelves subdirectory of Librarian, whose name is the
basename of .shelf file. It contains one subdirectory per each
library it includes.
A shelf can include other shelves. By these I mean literal inclusions rather than inheritance—each shelf has its own copy of every library it directly or indirectly includes. This is to keep the shelf’s internal dependencies consistent and to make it possible to upgrade one shelf without touching others.
A sample Librarian’s description of a shelf, describing libraries used by the Librarian itself, looks like this:
(defshelf librarian-base () ((trivial-shell :darcs "[[http://common-lisp.net/project/trivial-shell/darcs/trivial-shell/"][http://common-lisp.net/project/trivial-shell/darcs/trivial-shell/"]]) (split-sequence :tarball "cclan:split-sequence.tar.gz"))
DEFSHELF form above defines a shelf, named LIBRARIAN-BASE,
including no other shelves, and consisting of the TRIVIAL-SHELL
package downloaded and updated by the darcs version control system,
and of SPLIT-SEQUENCE package, downloaded as .tar.gz file
(`tarball’) from CClan.
2.2 Finding and downloading shelves
Shelves are distinguished by names, which are symbols. As in
ASDF systems, symbols are distinguished only by their name,
regardless of their package. In order to find and load a shelve
definitions we use the FIND-SHELF function:
(defun find-shelf (name &key force-reload) "Find shelf `NAME', returning its shelf object. `NAME' can be a symbol or string. If it's a symbol or string not containing the colon and dot characters, it names the shelf that is already managed. If the shelf isn't already loaded, or `FORCE-RELOAD' is true, an appropriate .shelf file from `*SHELVES-DIR*' directory is loaded, and the shelf object is returned. If `NAME' is any other string, it is treated as an URL or full pathname to a .shelf file. The file is downloaded or copied into `*SHELVES-DIR*' and then loaded." )
For downloading contents of shelved libraries, Librarian exports
DOWNLOAD-SHELF function:
(defun download-shelf (shelf) "Download libraries included in `SHELF' into shelf's subdir of `*SHELVES-DIR*'. `SHELF' may be a SHELF instance, or a symbol, or string. If it's a symbol or string, `FIND-SHELF' is invoked to get the SHELF instance." )
2.3 Using shelved libraries
Currently, all .asd files from all downloaded shelves are linked
to the systems subdirectory of CL-Librarian. However, this will
probably change soon as it doesn’t fit well to the include model
(there may be many copies of the same library on different
shelves).
Most probably, .asd files will be linked to
shelves/shelf-name.systems directory and additional functions
USE-SHELF and UNUSE-SHELF will be introduced to push/delete
these directories to/from ASDF:*CENTRAL-REGISTRY*.
2.4 TODO Keeping shelves up to date
2.5 TODO Customizing Librarian
2.6 TODO Defining shelves
Example shelves are in shelves/
2.7 TODO Extending Librarian
3 Hacking
3.1 TODO Separate library and shelf definition
for easier using of the same library in different shelves
3.2 BUG Invent way of updating tarball libraries
3.3 BUG Detect include loops
3.4 TODO Automatic shelf template generation from .asd files
3.5 TODO Update shelf files themselves, not only their content
3.6 TODO Do update shelf contents
3.7 TODO Convenient using and unusing shelved libraries
3.8 BUG Check authenticity of downloaded libraries.
No authenticity checking is currently done on downloaded libraries. Care must be taken to make sure if a correct, trusted source is used and to check downloaded library’s integrity.
Date: 2007/10/30 00:31:05