Okay, I've spent a lot of time writing Common Lisp and even more time reading about Common Lisp, and to be honest, I'm just a little tired of the cult around it. People who know it well gloat about how great Common Lisp is, which just so happens to make them look great too. And people who don't know Common Lisp all talk about the little Common Lisp they know, because they don't want to seem like they aren't in on it, and because they don't know better.
Let's talk about this fabled exchange between Norvig and McCarthy, where McCarthy asks if Python can gracefully manipulate code as data, and Norvig said no, and supposedly a thousand words were said in the ensuing silence.
Here's the thing; a thousand words weren't said. I don’t know what Tilton thinks was said there, but it certainly wasn’t a one sided silence in which only McCarthy’s side gets to smugly smirk as if it proves anything.
Let’s talk about code as data, shall we? What enables that? Of course, it’s the s-expressions and prefix notation. So let’s write a macro. But we won’t want to write a macro that simply sits at the beginning of an s-expression and takes in a bunch of arguments, because then we could just write a function. No, we want to do a transformation on the code that creates a domain-specific language, because that’s the power of macros, right?
So the very first thing you do with your macro powers, and pretty much the only useful thing you can do, is break s-expressions. Once the first macro is written you can no longer assume that inputs to macros will be s-expressions with the function at the beginning and arguments following. Every future macro must account for every previous macro. The more you use the capability to manipulate code as data gracefully, the less graceful it becomes.
And, in my opinion, far more damningly, macros make your program harder to reason about. Before macros, a reader could assume that the first thing in the s-expression was operating on everything else in the s-expression. But no longer. We can’t even assume that sub-s-expressions in the s-expression are evaluated first.
This isn’t a hypothetical problem. Common Lisp programmers spend a ton of time talking about how to write macros so that they’re not going to come back and bite you in the butt when they get used in an unexpected situation. And the reason is, nobody really knows how to do it.
So, with a great deal of respect for McCarthy based on his many other achievements, I have to say I don’t care about macros, and actually think we’re better off without them. First-class functions are a much more coherent, consumable way of using code-as-data.
Most of the features of Common Lisp I actually want are in other languages now. Garbage collection? REPLs? First-class functions? Higher-order functions? They’re all in other languages now. Python, for example has all those things.
And to be honest, other languages have done a lot better things with the functional programming aspects of Common Lisp. McCarthy gets credit for inventing a lot of stuff, but we don’t fly Wright-brothers style planes today and we shouldn’t use Common Lisp just because it was first to have those things. More sophisticated type systems make lambdas more powerful (and incidentally, a lot of the problems with macros can be seen as type problems).
People on this thread are claiming, “Learning Common Lisp turns you into a better programmer”. But I tend to think that learning functional programming is the part that people are referring to, and frankly, there are better languages in which to learn functional programming. Haskell, Standard ML, or OCaml would be a better choice.
And sure, there are other features that only Common Lisp has, but nobody is talking about those. Restarts? I’d love to see more people experimenting with those. Then again, Erlang has a way better threading model than anything else and much more sophisticated pattern matching. Scheme has call/cc. Standard ML has a powerful type system. Haskell has functional purity. Prolog has unification. A great many of these are more interesting than restarts.
I don’t hate Common Lisp; if nothing else, Common Lisp has a few very solid implementations out there and lots of existing libraries that make it a very useful tool. There are some programs which I wouldn’t consider writing in another language. I just don’t really think it’s the be-all and end-all of programming languages any more, and I’m kind of tired of the cult that has formed around it.
EDIT: s/Lisp/Common Lisp/g because not everyone takes “Lisp” to just mean “Common Lisp”.
> Let’s talk about code as data, shall we? What enables that? Of course, it’s the s-expressions and prefix notation.
s-expressions and prefix notation are orthogonal concepts. Nothing in the definition of s-expressions says it uses prefix notation. S-expressions are just a data format and its external notation.
> But we won’t want to write a macro that simply sits at the beginning of an s-expression and takes in a bunch of arguments, because then we could just write a function.
Functions and prefix notation are also unrelated concepts.
Lisp syntax uses prefix notation, but not everything is a function call. There are also Lisp variants, which don't use prefix notations, but still have s-expressions and even macros,
Take a lambda expression: (lambda (a b) (* a b (+ a b)))
It has a lambda symbol in prefix position, but it is not a function call.
> So the very first thing you do with your macro powers, and pretty much the only useful thing you can do, is break s-expressions.
Which is nonsense. Macros don't break s-expressions.
> Every future macro must account for every previous macro
Which is also nonsense. The typical macro expansion mechanism takes care of that most of the time.
> Once the first macro is written you can no longer assume that inputs to macros will be s-expressions with the function at the beginning and arguments following.
Well, now we not only have special forms (!), functions, lambda expressions, etc., but also macros. The main difficulty now is: more syntax and even user-defined syntax.
> First-class functions are a much more coherent, consumable way of using code-as-data.
Which are unrelated concepts.
> Most of the features of Common Lisp I actually want are in other languages now. Garbage collection? REPLs? First-class functions? Higher-order functions? They’re all in other languages now. Python, for example has all those things.
There are cars which have wings, can swim, etc. But that does not make them especially useful, say, to bring the kids to school every morning. The raw assembly of features is not the point, it's their integration for certain use cases.
> Wright-brothers style planes today and we shouldn’t use Common Lisp just because it was first to have those things.
Common Lisp wasn't first. CL was defined 26 years (in 1984) after Lisp (1958) was invented and standardized after ten years more work (1994).
> But I tend to think that learning functional programming is the part that people are referring to,
Not really. One can learn functional programming with much simpler languages. Legions of students used simple Lisp dialects/subsets to learn some FP concepts. See SICP (and many other books/courses) - which doesn't use macros, btw.
> and frankly, there are better languages in which to learn functional programming. Haskell, Standard ML, or OCaml would be a better choice.
Since Common Lisp was never designed to enforce or advance statically-typed Functional Programming, it's only logical that it is not particular good at it.
> So the very first thing you do with your macro powers, and pretty much the only useful thing you can do, is break s-expressions. Once the first macro is written you can no longer assume that inputs to macros will be s-expressions with the function at the beginning and arguments following. Every future macro must account for every previous macro. The more you use the capability to manipulate code as data gracefully, the less graceful it becomes.
You should check out Racket's macro system. It's a lot more sophisticated than Common Lisp's. Common Lisp macros are to C (e.g., gensym is a macro-level malloc, you need to manually destructure S-expressions, etc.) as Racket macros are to ML and Haskell (syntax objects are aware of which variables are in scope, so automatic fresh name generation is possible; user-defined syntax classes and patterns let you process arbitrarily complicated structures in a sane way, etc.). If you like the idea of metaprogramming, but `defmacro` left you with a bad taste in the mouth, Racket is totally the language for you.
> First-class functions are a much more coherent, consumable way of using code-as-data.
First-class functions are easier to use than macros, but they are not “code as data”. Furthermore, “code as data” itself is only true with a caveat: the full version is ”code in an object language is data in the metalanguage”, which is obvious to anyone who has written a compiler. Of course, macros make it easy to use Lisp as its own metalanguage, but there's still a phase distinction between macro-expansion time and when the generated code is actually used.
> And to be honest, other languages have done a lot better things with the functional programming aspects of Common Lisp.
Common Lisp is a ridiculously powerful language, but it isn't a functional language. It fails to meet the zeroth nonnegotiable requirement in a practical functional language, namely, a notion of compound value: https://news.ycombinator.com/item?id=12199981
>You should check out Racket's macro system. It's a lot more sophisticated than Common Lisp's. Common Lisp macros are to C (e.g., gensym is a macro-level malloc, you need to manually destructure S-expressions, etc.) as Racket macros are to ML and Haskell (syntax objects are aware of which variables are in scope, so automatic fresh name generation is possible; user-defined syntax classes and patterns let you process arbitrarily complicated structures in a sane way, etc.). If you like the idea of metaprogramming, but `defmacro` left you with a bad taste in the mouth, Racket is totally the language for you.
Better yet, check out some other schemes. ir, er, and sc macros has the raw procedurual power of defmacro, but with the hygene and safety of syntax-case/syntax-rules, without the declarative syntax of syntax-rules, and the disadvantages of syntax-case (stupidly complex, breaking the standard macro abstraction with syntax/datum distinctions, etc.).
Given, syntax-case has some advantages, but I don't think it carries its own weight from a programmer's perspective.
> You should check out Racket's macro system. It's a lot more sophisticated than Common Lisp's. Common Lisp macros are to C (e.g., gensym is a macro-level malloc, you need to manually destructure S-expressions, etc.) as Racket macros are to ML and Haskell (syntax objects are aware of which variables are in scope, so automatic fresh name generation is possible; user-defined syntax classes and patterns let you process arbitrarily complicated structures in a sane way, etc.).
Agreed. I did play around with this part of Racket quite a bit and I'm convinced that it's the best system if I wanted to create a domain-specific language. But it still runs into the problem where you're defining a new language with new syntax, which forces you to define even more new language with more syntax in order to make that language useful. It's the best way to build a DSL
But given we already have a pretty good multi-purpose language with a lot of work put into it (Racket) the number of situations where it's worthwhile to create an equally-well-thought-out DSL is pretty low. Racket makes it easier, but it's still not easy. Add to this the fact that other people are going to write half-assed DSLs in my code, the net tradeoff is still usually negative, even with Rackets clearly superior macro system.
> Furthermore, “code as data” itself is only true with a caveat: the full version is ”code in an object language is data in the metalanguage”, which is obvious to anyone who has written a compiler.
Uh, I've written a compiler and that's not obvious.
If you want to disagree with me on what "code as data" means, you're welcome to do so. As long as you understood what I said I don't care which words got me there.
> Common Lisp is a ridiculously powerful language, but it isn't a functional language. It fails to meet the zeroth nonnegotiable requirement in a practical functional language, namely, a notion of compound value: https://news.ycombinator.com/item?id=12199981
How about we assume when I said "functional programming" I'm using the Wikipedia definition[1].
> Racket makes it easier, but it's still not easy. Add to this the fact that other people are going to write half-assed DSLs in my code, the net tradeoff is still usually negative, even with Rackets clearly superior macro system.
You have a point there. In any case, metaprogramming shouldn't be an everyday activity.
> If you want to disagree with me on what "code as data" means, you're welcome to do so.
What I mean by “code in an object language is data in the metalanguage” is that, in the (meta)language in which you're writing a compiler or interpreter, the (object language) program that you're processing is represented as a data structure (say, as a syntax tree). It's just a triviality, I'm not saying anything really deep.
From this point of view, it should be clear that a macro system is essentially a language-integrated facility for writing compiler plugins.
> How about we assume when I said "functional programming" I'm using the Wikipedia definition[1].
My definition of functional programming is “using (procedures that compute) mathematical functions whenever possible”. A mathematical function is a mapping from values to values, so if a language doesn't have a good notion of (possibly compound) value, then you're going to run into trouble writing procedures that compute mathematical functions.
EDIT: The Wikipedia definition essentially agrees with me.
“In computer science, functional programming is a programming paradigm—a style of building the structure and elements of computer programs—that treats computation as the evaluation of mathematical functions [emphasis mine] and avoids changing-state and mutable data.”
And a mathematical function is a mapping from values (in a domain) to values (in a codomain). Again, quoting Wikipedia[0]:
“A function can be defined by any mathematical condition relating each argument (input value) to the corresponding output value.”
> What I mean by “code in an object language is data in the metalanguage” is that, in the metalanguage in which you're writing a compiler or interpreter, the object language program that you're processing is represented as a data structure (say, as a syntax tree). It's just a triviality, I'm not saying anything really deep.
I understood what you were saying, and didn't ask for an explanation. I just don't see why you felt the need to correct me on my calling first=class functions "code as data" and substitute your own definition that had nothing to do with what I was saying.
> My definition of functional programming is “using (procedures that compute) mathematical functions whenever possible”. A mathematical function is a mapping from values to values, so if a language doesn't have a good notion of (possibly compound) value, then you're going to run into trouble writing procedures that compute mathematical functions.
Again, I didn't ask what your definition was, because it wasn't relevant to the conversation.
You're basically interrupting me to tell me I'm not using the same definitions of words as you are, and it's not particularly endearing. If you don't understand what I'm saying, I'll be happy to explain. If you do, however, understand what I'm saying well enough to correct me on my usage of the English language, then my usage of the language has been clear enough for my goals, so I wouldn't be interested in your corrections even if they represented HN's common usage (which they don't).
> I just don't see why you felt the need to correct me on my calling first=class functions "code as data"
Because first-class functions aren't “code as data”. When you have a first-class function, the only thing you can do with it is call it. If it were a data structure, you could analyze its constituent parts.
> You're basically interrupting me to tell me I'm not using the same definitions of words as you are,
The Wikipedia article you linked says:
“In computer science, functional programming is a programming paradigm—a style of building the structure and elements of computer programs—that treats computation as the evaluation of mathematical functions [emphasis mine] and avoids changing-state and mutable data.”
The emphasized part is just what I said. That a mathematical function is a mapping from values to values is unquestionable - it's not something I'm saying, it's a mathematical fact. So if you can't express compound values, you're limited to a world where mathematical functions can only manipulate primitive values.
> Because first-class functions aren't “code as data”. When you have a first-class function, the only thing you can do with it is call it. If it were a data structure, you could analyze its constituent parts.
There are lots of ways in which you can treat a first class function as data, even if it can't be treated 100% equivalently to data in every single situation. You can, for example, pass it to other functions like data. Sure, most languages don't let you inspect the internals, but that's only one way of treating code as data.
> The emphasized part is just what I said. That a mathematical function is a mapping from values to values is unquestionable - it's not something I'm saying, it's a mathematical fact. So if you can't express compound values, you're limited to a world where mathematical functions can only manipulate primitive values.
I said, "And to be honest, other languages have done a lot better things with the functional programming aspects of Common Lisp." Common Lisp isn't a purely functional language, but it does have functional aspects. Note in the article I quoted, that Common Lisp is the first language listed in the "prominent programming languages which support functional programming such as" section.
You're literally not even disagreeing with me, you're just defining "functional programming language" as "purely functional programming language", when I literally never even called Common Lisp a functional programming language.
Insisting on your definitions instead of trying to understand what I said doesn't make me want to engage with you further.
> There are lots of ways in which you can treat a first class function as data, even if it can't be treated 100% equivalently to data in every single situation. You can, for example, pass it to other functions like data.
Strictly speaking, you can't pass functions as data. You can only pass thunks that, when forced, yield functions. A thunk is data, but a function is a computation. Computations are “too active” to be stored or passed around unthunked. The technical details are here: http://www.cs.bham.ac.uk/~pbl/cbpv.html, http://www.cs.bham.ac.uk/~pbl/papers/. (I am not the owner of the website, just in case.)
> Sure, most languages don't let you inspect the internals, but that's only one way of treating code as data.
What are the others?
> You're literally not even disagreeing with me, you're just defining "functional programming language" as "purely functional programming language",
How did you conclude that? I never said anything of the sort. What I said is “functional programming is programming with procedures that compute mathematical functions whenever possible”. Pure functional programming imposes further requirements, like effect segregation (as in Haskell) or even the total absence of effects (obviously unsuitable for a general-purpose language). FWIW, I'd count ML, Racket, Clojure and Erlang as functional languages.
> when I literally never even called Common Lisp a functional programming language.
You said Common Lisp has “functional aspects”. Well, closures make a language higher-order, but so do Java-style objects! For a language to be called “functional”, however, it has to make functional programming actually pleasant. I showed one fundamental limitation of Common Lisp in this regard: you can't define functions that take or return compound values, because Common Lisp doesn't have compound values in the first place.
> So the very first thing you do with your macro powers, and pretty much the only useful thing you can do, is break s-expressions. Once the first macro is written you can no longer assume that inputs to macros will be s-expressions with the function at the beginning and arguments following.
That's just completely wrong, and makes me wonder if you have actually spent much time writing Common Lisp at all. Macros are orthogonal to S-expressions: they manipulate code (which is just lists) in-memory, after it's been read in from S-expressions by the reader. They don't break S-expressions at all (and indeed, the output of a macro is necessarily PRINTable as an S-expression, e.g. (macroexpand '(with-open-file (foo "/tmp/"))) expands to:
When Lisp programmers talk of implementing a DSL with its own syntax, we almost always mean an S-expression-based DSL, where the fundamental syntax still uses S-expressions and the DSL is built atop them. An example would be CL-WHO[1], where HTML tags are represented e.g. as (:p "Some text " (:em "emphasised")).
Maybe you're thinking of read macros, which really can be used to implement other syntaxes, e.g. CLSQL[2], which uses custom reader syntax to implement inline SQL expressions? In that case, you're not wrong: implementing your own reader syntax does, indeed, break S-expressions. But it doesn't matter: due to the specification of the reader algorithm, neither elements containing nor elements within custom syntax care. The reader algorithm is quite elegant that way.
> I have to say I don’t care about macros, and actually think we’re better off without them. First-class functions are a much more coherent, consumable way of using code-as-data.
At the cost of being much more verbose. I've taken a look at trying to implement restarts in Go, and the sheer number of times I'd have to type 'func' is insane.
> But I tend to think that learning functional programming is the part that people are referring to
Once again, I disagree. IMHO writing Lisp makes one a better programmer because one starts to think of code as clay, which can be sculpted and crafted, moved around, deleted, generated &c. The worst good programmers I know of simply aren't comfortable with that concept; the best programmers I know are.
Compared to that, functional programming is merely interesting IMHO.
>Let's talk about this fabled exchange between Norvig and McCarthy, where McCarthy asks if Python can gracefully manipulate code as data, and Norvig said no, and supposedly a thousand words were said in the ensuing silence.
The difference is that when you can do it gracefully, you'll end up doing it and using this possibility to build your project. If you can't do it gracefully, you'll dread using it and will not consider it a possible solution to your problem unless all other options are exhausted.
Well, the key part is gracefully. Lots of languages can manipulate code as data, but so for for me only in homoiconic languages does it end up feeling graceful for a while (although as I noted, it stops feeling graceful pretty quickly).
Me too, but you seem to not be entirely clued in to how macros work. So there's that.
>So the very first thing you do with your macro powers, and pretty much the only useful thing you can do, is break s-expressions. Once the first macro is written you can no longer assume that inputs to macros will be s-expressions with the function at the beginning and arguments following. Every future macro must account for every previous macro. The more you use the capability to manipulate code as data gracefully, the less graceful it becomes.
While perhaps more true in Common Lisp than in scheme, that's still fairly untrue. IIRC, macro expansion is innermost-first, so macros don't have to worry about tripping over each other, but even if that's not true, you're still wrong. Macros are sexprs, just like anything else: they break the semantic rules of lisp, NOT the syntactic rules of lisp. Since macros operate primarily on a syntactic level (their job is to sugar over Lisp), and when they operate on a semantic level, they're being used to write DSLs, which aren't lisp, the problem you describe is rare to nonexistant.
>macros make your program harder to reason about.
Yes, but not for the reasons you think. Macros make your program harder to reason about for the same reason functions do: they're an abstraction. The higher level the abstraction you're using is, the harder it is to reason about your code, because by definition, you can't see what it's doing.
>Common Lisp programmers spend a ton of time talking about how to write macros so that they’re not going to come back and bite you in the butt when they get used in an unexpected situation. And the reason is, nobody really knows how to do it.
However, us schemers (NOT racketeers, Racket using syntax-case instead, which while similar seeming, is an entirely different ball game, although it also fixes this) have had hygenic defmacro-style lowlevel macros for years now, and that fixes the worst of it.
>Python, for example has all those things.
...Not actually true. Python's functions aren't first-class. They try very hard to be, but they're not. Which is why idiomatic Python doesn't use a lot of higher-order functions. Ruby does better, but of the trinity of popular high-level scripting languages, only JS has true first-class functions. In fact, for a while, JS had no distinct syntax for function declaration.
>People on this thread are claiming, “Learning Common Lisp turns you into a better programmer”. But I tend to think that learning functional programming is the part that people are referring to
It's not what I'd refer to. FP makes you a better programmer to, but learn than in Haskell or ML, or even Scheme, though it's not as good for that purpose. Learning lisp exposes you to code-as-data in a very viceral way, and can help you understand a variety of programming concepts, but more importantly, it exposes you to new, different ways of programming, which always makes you a better programmer.
>And sure, there are other features that only Common Lisp has, but nobody is talking about those. Restarts? I’d love to see more people experimenting with those. Then again, Erlang has a way better threading model than anything else and much more sophisticated pattern matching. Scheme has call/cc. Standard ML has a powerful type system. Haskell has functional purity. Prolog has unification. A great many of these are more interesting than restarts.
I agree, but restarts are the one I want most. It might seem like we schemers got a better deal with call/cc, and I love call/cc, but sometimes I look at restarts, and wish we had gone the other way (call/cc and functioning restarts are effectively mututally exclusive). Trust me, call/cc looks cool, but so does self-rewriting asm. Both come in handy from time to time, but you rarely want either in production (and especially not if your scheme doesn't have cheney on the mta compilation, because then call/cc is criminally slow)
>I just don’t really think it’s the be-all and end-all of programming languages any more, and I’m kind of tired of the cult that has formed around it.
That's true. No language is, or ever can be the be-all, end-all. Rust and FP taught me that: some paradigms and ideas require resrictions, others require freedom. There's always something new to learn. And the cult of common lisp is not one you need to join. In fact, if you're a good lisper, you can't really be part of it.
Let's talk about this fabled exchange between Norvig and McCarthy, where McCarthy asks if Python can gracefully manipulate code as data, and Norvig said no, and supposedly a thousand words were said in the ensuing silence.
Here's the thing; a thousand words weren't said. I don’t know what Tilton thinks was said there, but it certainly wasn’t a one sided silence in which only McCarthy’s side gets to smugly smirk as if it proves anything.
Let’s talk about code as data, shall we? What enables that? Of course, it’s the s-expressions and prefix notation. So let’s write a macro. But we won’t want to write a macro that simply sits at the beginning of an s-expression and takes in a bunch of arguments, because then we could just write a function. No, we want to do a transformation on the code that creates a domain-specific language, because that’s the power of macros, right?
So the very first thing you do with your macro powers, and pretty much the only useful thing you can do, is break s-expressions. Once the first macro is written you can no longer assume that inputs to macros will be s-expressions with the function at the beginning and arguments following. Every future macro must account for every previous macro. The more you use the capability to manipulate code as data gracefully, the less graceful it becomes.
And, in my opinion, far more damningly, macros make your program harder to reason about. Before macros, a reader could assume that the first thing in the s-expression was operating on everything else in the s-expression. But no longer. We can’t even assume that sub-s-expressions in the s-expression are evaluated first.
This isn’t a hypothetical problem. Common Lisp programmers spend a ton of time talking about how to write macros so that they’re not going to come back and bite you in the butt when they get used in an unexpected situation. And the reason is, nobody really knows how to do it.
So, with a great deal of respect for McCarthy based on his many other achievements, I have to say I don’t care about macros, and actually think we’re better off without them. First-class functions are a much more coherent, consumable way of using code-as-data.
Most of the features of Common Lisp I actually want are in other languages now. Garbage collection? REPLs? First-class functions? Higher-order functions? They’re all in other languages now. Python, for example has all those things.
And to be honest, other languages have done a lot better things with the functional programming aspects of Common Lisp. McCarthy gets credit for inventing a lot of stuff, but we don’t fly Wright-brothers style planes today and we shouldn’t use Common Lisp just because it was first to have those things. More sophisticated type systems make lambdas more powerful (and incidentally, a lot of the problems with macros can be seen as type problems).
People on this thread are claiming, “Learning Common Lisp turns you into a better programmer”. But I tend to think that learning functional programming is the part that people are referring to, and frankly, there are better languages in which to learn functional programming. Haskell, Standard ML, or OCaml would be a better choice.
And sure, there are other features that only Common Lisp has, but nobody is talking about those. Restarts? I’d love to see more people experimenting with those. Then again, Erlang has a way better threading model than anything else and much more sophisticated pattern matching. Scheme has call/cc. Standard ML has a powerful type system. Haskell has functional purity. Prolog has unification. A great many of these are more interesting than restarts.
I don’t hate Common Lisp; if nothing else, Common Lisp has a few very solid implementations out there and lots of existing libraries that make it a very useful tool. There are some programs which I wouldn’t consider writing in another language. I just don’t really think it’s the be-all and end-all of programming languages any more, and I’m kind of tired of the cult that has formed around it.
EDIT: s/Lisp/Common Lisp/g because not everyone takes “Lisp” to just mean “Common Lisp”.