• ← Epilogue
• [Appendix] All Else is Not Enough →

[Appendix] Why Common Lisp for This Project

No information here will be new to a Lisper. I discuss other languages on the next page.

Also see Common Lisp - Myths and Legends by LispWorks (just keep in mind it's from 2002).

In The Power of Structure I said how important it is to have an interactive programming environment for the true power-user experience. And if I didn't say so directly: it was implied.

Well, Common Lisp is simply one of the best languages, if not the best, for interactive programming.

Part of what makes it so is its superior error-handling capabilities. If an error occurs, the image will put the user into a debugger, at runtime. You may often continue evaluation right from the point of error, ignore an error, or try fixing, say, the problematic function in the source and then retry to try again.

The stack does not get unwound. You may inspect the current stack frame (and the ones leading to it). You can see all the local bindings right in the debugger and change them.

And then, Common Lisp allows you to update function definitions at runtime. (The new function definition will be used on the next call.)

But also, if you update a class definition, the objects of that class will update too.

Interactivity in Common Lisp is there by design. CL is geared for image-based, incremental development.

The common perception of development with REPL (read-eval-print-loop) is to use a command-line window. You can do that in CL, but that's not the most optimal approach to live coding.

What you would typically instead is write code in a file. Then you can evaluate that code in-place and the results of evaluation show up in another window or in some output field somewhere in your editor.

So, if you are developing a function like:

(in-package :some-package)

;; <some code>

(defun middle (x y)
  (/ (+ x y) 2.0))

(middle 5 10)

;; <rest of code in your file>

then what you will do is jump between the function and the test call invocations as you are modifying and re-evaluating them. To that end, you would usually use some keybinding to evaluate the top-level form at point.

And if you want to leave a test call there permanently but don't want it to evaluate when the project loads, you can prepend #+nil before the expression like this:

#+nil
(middle 5 10)

Typically, most such calls end up being test cases. So, by the time you are done developing a function, you will have likely gathered enough calls for a test suite for it. All you would have to do next is put all that code into some place where you can then organize it with the help of some testing framework.

So, the fact of CL being image-based is essential, because how else would you want to develop interactive applications?

For a power user, there's no real distinction between using a system and developing in it. And, so, there's no other choice other than to use an image-based language.

Interactivity is just as well the reason to choose Common Lisp among other Lisps: it's simply the most interactive of them all. And it's also the most practical one (barring Clojure, which some argue isn't a true Lisp).

—

CL has a rather mature ecosystem: currently there's ~2200 projects on quicklisp, and ~1500 on ultralisp. (There's probably a big overlap, though.) It's not a huge amount of packages, but most of them tend to be of rather high quality, and, importantly, they just keep working year after year.

And the variability and verve of some projects out there never ceases to surprise me. The flexibility of the language allows many linguistic experiments and projects.

And then, there are whole languages developed on top of CL: Coalton, Screamer, April. You can integrate those into your otherwise typical code.

Take a look at State of the Common Lisp ecosystem, 2020, These Years in Common Lisp: 2022 in review, and Awesome CL for further info.

Also, there's the The Common Lisp Cookbook where you can find book recommendations or to take a look at a few code examples for various use cases.

—

Very importantly, Common Lisp has a standard. A standard is, obviously, quite essential for stability. The code from 30 years ago will work just fine on today's compilers.

Stability, in the long run, is very important to Project Mage.

However, the existing of the standard which hasn't been updated since the 90s does not mean that there are no further ways of improvement.

For example, initially, there was no system for defining a project, and, so, ASDF was developed, just as were the package repositories (the previously mentioned quicklisp and ultralisp).

The CL community has also come up with a few defacto standards/libraries, such as bordeaux-threads for multithreading and the MetaObject Protocol (MOP) for extending the Common Lisp Object System (CLOS).

Also, thanks to macros, syntactic developments and improvements don't require rewriting the standard. These simply come packaged into libraries (such as iterate, which provides a more extensible alternative to the standard loop construct).

In short: if you are not satisfied with Common Lisp, you don't switch to another language, you extend Common Lisp to your liking. It's as much of a platform as it is a language.

—

I don't believe Common Lisp is perfect, though. It has a historical luggage and some rather unfortunate constructions like loop I just mentioned. Yes.

Common Lisp often gets flak for this and for having some non-orthogonal functions, sometimes it even gets tagged as bloated. This criticism usually comes from the Scheme camp, though, where a typical standard is around 50 pages (as opposed to CL's ~1000).

Perhaps there's a middle ground there somewhere, but where things like the condition system and some kind of an object system (not necessarily CLOS) are a part of the standard. This middle ground seems to be closer to CL, for all the practical matters.

(What I truly believe, though, is that the language should stress powerful building blocks, flexibility and meta-flexibility. I talk more about this here.)

Common Lisp adopted ideas from multiple lisps that existed at the time and the goal was to preserve compatibility with the existing code (to a sane degree).

There's, of course, also some historical luggage. It's hard to pinpoint exactly where it starts or ends, but it's there.

At that, I can't really call CL bloated. And it certainly makes a lot more sense than anything else with a standard of a comparable size. Most things in it are there for a reason. Yes, I personally don't like loop for its lack of extensibility and total disregard of s-expressions. But, by and large, this luggage is pretty irrelevant. It's not always pretty, but you can ignore it, at least in your own new projects, and just use the libraries that suit your taste better.

In other words, Common Lisp doesn't have any nagging problems that can't be solved by the tools of Common Lisp itself, via external libraries.

I think, most of its immediate problems are in the tooling department. For one, Emacs does not make for a good enough IDE experience. If the campaign is successful, though, then, in a few years time, this problem will be addressed more than adequately, I believe.

The one missing element for CL right now is, perhaps, first-class global environments. These would give the flexibility to load multiple versions of a library/program within a single image. This may be useful if you are developing separate projects that, say, use various library versions for project control. Basically these would allow you to have independent environments with shared memory within a single image as opposed to the alternative of having to start several images and send data between each one.

The lack of first-class environments doesn't present a pressing issue, though. However, they could prove to be a handy building block in the future, especially if is a lisp image is to be seen as an OS-like platform for running various applications. For example, imagine running two projects and each has a different requirement for some package version. It would be hard to run these two together in the same image.

Thankfully, the problem is acknowledged in the community and may have a working solution at some point in the future.¹ See Robert Strandh's paper First-class Global Environments in Common Lisp for an overview.

—

Claims that Common Lisp is too old are absurd. It's especially funny to hear so from people who use languages that force a compile-run cycle on you.

When people try to guess why Common Lisp isn't more popular if it's such a great language, they sometimes blame the macro system because it, apparently, puts too much responsibility on the programmer or because you can't just be making your language on the fly or because omg how can you understand someone else's macros or all that other nonsense.

No, truly, this is not the reason.

To deny flexibility is to welcome bloat. Consider the 420000+ Python packages and tell me how 90% of all functionality across all of them could possibly be not redundant.² OK. this statement is a tad loaded, but it feels right, doesn't it?

Macros simply give a compression mechanism for your code.

—

At last, people always assume CL is slow because it's a dynamic language. Thanks to Python for that, I guess.

But, really, CL is a very pliant language. It has a gradual typing system (you can give type hints to function arguments and to return values and even to expression). Some compilers (such as SBCL, the most popular one) produce native code directly. You can inspect that code and let it guide your optimization process.

You can also have structs as opposed to classes. These are much like structs in any other language.

In CL, you can also set various levels of optimization that you can apply per file, per function or even per form. Coupled with the optional type information, the compilers may often optimize or even inline certain calls.

CL is not interpreted, and it's not even slow.

Of course, it is still garbage collected, but even then, garbage collection isn't necessarily slower than stack allocation.³ I won't provide any evidence for this. There are many debates and papers out there. Although, it depends, of course. (I think, if the right hardware stuck around, it would have been quite a question by now, really. I am not an expert on the issue by any means, but it doesn't seem out of the ballpark that most software out there would never need manual memory management in the first place with the right compiler techniques and the hardware geared for running virtual machines.)

—

The groundwork for Project Mage includes building a platform for building specialized text editors.

And to some people it's pretty important that you can launch an editor rather quickly, and that it take as little memory as possible.

And that's fine. Because do this: start 20 instances of Vim and then open 20 Emacs buffers. See what takes more space.

Incrementally speaking, communicating with a running image is undeniably more space-efficient and faster than cold-starting an application every time you want to edit a file. For example, Emacs has an on-demand server that you can add to a startup of your OS. So, you never have to wait for Emacs to load if you use that image for opening files from a terminal.

Working with an image-based platform assumes this kind of approach. Treat is it as your second OS (the good one).

—

So, as I said, Common Lisp is not old. I don't even know how to interpret that statement for a language that is capable of a sufficient degree of evolution over time, which CL most certainly is.

At last, if you are unfamiliar with Lisp and feel scared of parenthesis: you might as well be scared of Santa or of being a real man.

Lisp simply has the most readable and sane representation of code known to humanity. Once you see it, any other language looks inadequate in comparison.

And if you wanted to have a worthy mini-language in your Lisp project, no problem: you can always use reader macros.

—

Lisps are often labeled as a functional family of languages. And, by extension, Common Lisp. But what it is is an expression-based language. That helps a great deal to write in the functional programming style.

But it doesn't force functional programming on you. You can pick any paradigm you like. And, I think, there's a great deal of flexibility in that.

—

Well, at the moment, I just can't see any downsides to using CL for a power-user machine.

As it happens, I was also lucky to find quite a few projects like Garnet and some others that were developed in this language. Those have really set up a productive context for thinking further, for making ever more ambitious, but compact designs. As did the quite friendly community with a lot of knowledgeable members, and a lot of great libraries developed by them.

But the combination of stability, power and maturity, coupled with the fact of being image-based, makes Common Lisp more-or-less the only candidate for Project Mage.

—

I discuss some other languages that come close on the next page, but they all seem to fall short on one metric or another.

• ← Epilogue
• [Appendix] All Else is Not Enough →

• Table of Contents#
• Top#
• All Essays