I really dislike the tiny lisp implementation mentioned in the article. Part of the beauty of a real "lisp in lisp" interpreter is that it is meta-circular -- that is, the interpreter is able to interpret itself.
Using pmatch and some other tricks makes the interpreter shorter but prevents meta-circular evaluation, in which case you could have just defined the lisp interpreter as just being eval.
Yup, very familiar with SBCL. But the truth is, to get SBCL to the point where it's anywhere near the realm of C for performance there's a ton of work that the compiler has to do that GCC simply doesn't.
Escape analysis to work out when heap allocations can be avoided, type inference to work out when dynamic type checks can be omitted / fixnum math can be used instead of generic math, when functions can be linked directly instead of dynamically dispatched, when functions are pure to allow for more flexible optimizations with evaluation order, etc.
Scheme adds even more complexity despite being a far smaller language than common lisp. Reified continuations massively complicate any optimzation and so does automatic lisp-1 to lisp-2 conversion, which has a bunch of caveats making it much trickier than it seems. Add on the fact that scheme lets you overwrite core primitive operations like car/cdr and there just isn't really any ground to stand on to prove anything about the code at all.
I have no idea how guile scheme runs as fast as it does.
GCC on the other hand is coming from a much better point of view: C is statically typed with full type signatures on every function, argument and variable, so you can safely eliminate all runtime type-checking and arity-checking. There's no garbage collection, so you don't need that. functions can't be redefined, and all must be declared before being used so you don't need a mechanism for dynamic dispatch, there's no closures so you don't need to perform any lambda lifting or closure conversion, you don't need the compiler or the interpreter available at runtime to handle calls to eval, loops are clearly marked and easy to optimize (vs lisp using tag-body and scheme using widespread recursion with TCO), fast iteration of lists in lisp requires the compiler to prove that it can safely eliminate a null check and C just doesn't do bounds checks, and so on.
There's an endless of list of reasons why it's easier to generate near-optimal assembly from C and why it's very hard to do so from lisp. A simple C compiler will produce far, far faster code than a simple lisp compiler.
Emacs isn't really a follower of the unix philosophy since it's more a hold-over from the world of lisp machines.
Emacs also eschews the "everything is a file" philosophy of unix (programs in emacs are just lisp functions) and the idea of storing / transmitting data as plain text, instead using lisp data structures to do that.
Lisp itself is kind of antithetical to the unix philosophy too when compared with C. Lisp is presents an extremely simple, totally consistent core that is extremely difficult to implement efficient compilers for. C on the other hand was designed from the outset with the goal of being easy to write compilers for. Requirements for having function prototypes ahead of uses, having all variables declared at the top of a function, etc were done to make it easier to implement. Almost a perfect example of worse is better, really.
Check out the unix hater's handbook if you haven't, that sums up pretty nicely the world that emacs hails from.
Using pmatch and some other tricks makes the interpreter shorter but prevents meta-circular evaluation, in which case you could have just defined the lisp interpreter as just being eval.