Pixie: A small fast, native lisp(github.com)
github.com
Pixie: A small fast, native lisp
https://github.com/pixie-lang/pixie
57 comments
Please be sure to improve on the state of clojure error messages. It would help a lot.
This is an amazing effort! Now to clojure compatibility. You can only gain by making it clojure compatible. What would be the possible reasons to diverge? Another one: what are your plans for namespaces and code packaging and distribution? What are your thoughts about a build system? I think the world doesn't need another build and package distribution system. Which one would you reuse?
I love clojure, let me make this clear. But there are certain aspects of the language that don't apply if you are writing a VM from scratch in RPython. If I could make multimethods dispatching on types as fast as protocols, what is the point of protocols?
A great example of this is the (fn [& args]) bit. In Clojure, variadic parameters are passed in as a ISeq. In Pixie they are a vector (or an immutable array actually). This allowed me to tune the JIT quite a bit to allow it to remove the allocation of "args" completely. As well as allow for things like unrolling a reduce over the args. This stuff would have been much harder if the arguments were passed in as an ISeq.
This is the whole reason why Pixie exists, instead of something like "clojure-on-rpython". Compatibility means constraining the feature set. And that's not something I want to do yet.
A great example of this is the (fn [& args]) bit. In Clojure, variadic parameters are passed in as a ISeq. In Pixie they are a vector (or an immutable array actually). This allowed me to tune the JIT quite a bit to allow it to remove the allocation of "args" completely. As well as allow for things like unrolling a reduce over the args. This stuff would have been much harder if the arguments were passed in as an ISeq.
This is the whole reason why Pixie exists, instead of something like "clojure-on-rpython". Compatibility means constraining the feature set. And that's not something I want to do yet.
It might be interesting down the line to see if I couldn't target pixie using something like cljx (or actual feature expressions if they ever land in a mainline clojure release). This wouldn't necessarily require any effort on the part of my target, but still allow portability when it makes sense.
One advantage of having protocols in addition to multimethods is that you need to implement all methods in a protocol. This eases documentation, and is useful for methods that are fundamentally tied together.
It looks cool enough, but I wish more of these modern Lisp implementations would ask to implement the Common Lisp standard rate than their own, informally-specified version of Lisp. Yeah, Common Lisp is not perfect, but it does have a lot of excellent ideas, and some of the bits which don't look excellent actually are (e.g. pathnames and logical pathnames).
Common Lisp is too big for a typical application (e.g., embedded scripting engines, mataprogramming hosts, etc.). And it's a Lisp-2, which is quite a deterrent for many.
> And it's a Lisp-2, which is quite a deterrent for many.
I used to be on the fence about multiple namespaces in programming languages. After a decade of programming in Common Lisp and in other languages that pretend to have a single namespace, I see multiple namespaces as a clearly superior approach. I don't think it is a deterrent either - a reason to post complaints on the Internet, yes.
I used to be on the fence about multiple namespaces in programming languages. After a decade of programming in Common Lisp and in other languages that pretend to have a single namespace, I see multiple namespaces as a clearly superior approach. I don't think it is a deterrent either - a reason to post complaints on the Internet, yes.
For me it's clearly a deterrent - it does not fit at all into my model of building hierarchies of DSLs.
> it does not fit at all into my model of building hierarchies of DSLs
Links to source code or GTFO. The only way people manage to combine DSLs in a single-namespace language is by having stupid naming rules and restrictions (see for example Ruby on Rails). Compare this to something like https://github.com/vsedach/cliki2 where I could just throw arbitrary CL libraries that define their own DSLs together and not worry that my function name is going to clobber some reference in the template system.
Links to source code or GTFO. The only way people manage to combine DSLs in a single-namespace language is by having stupid naming rules and restrictions (see for example Ruby on Rails). Compare this to something like https://github.com/vsedach/cliki2 where I could just throw arbitrary CL libraries that define their own DSLs together and not worry that my function name is going to clobber some reference in the template system.
See https://github.com/combinatorylogic/mbase
Problem with two namespaces is, well, need to maintain two namespaces, in all your DSLs. Which in many cases may double the effort.
Of course, macros must have their own namespace, but that's an obvious thing, you cannot mix compile-time and run-time namespaces anyway.
Problem with two namespaces is, well, need to maintain two namespaces, in all your DSLs. Which in many cases may double the effort.
Of course, macros must have their own namespace, but that's an obvious thing, you cannot mix compile-time and run-time namespaces anyway.
> Problem with two namespaces is, well, need to maintain two namespaces, in all your DSLs. Which in many cases may double the effort.
How so? And why only two namespaces? Common Lisp probably has a dozen: package name, lexical variable, dynamic variable, function (stores either function or macro function), documentation string, property list, type name, class name, slot name, etc.
As long as you have first-class identifiers, you can define arbitrary namespaces without having to worry about conflicts between the namespaces.
> Of course, macros must have their own namespace
That doesn't have to be true, and is not true in Common Lisp: http://www.lispworks.com/documentation/HyperSpec/Body/03_bba...
Having an identifier denote a function and a macro at the same time like that enables adding partial evaluation and other compile-time optimizations to DSLs without having to dig into the compiler.
> but that's an obvious thing, you cannot mix compile-time and run-time namespaces anyway.
? You have to mix them if you want a compiler in your runtime.
How so? And why only two namespaces? Common Lisp probably has a dozen: package name, lexical variable, dynamic variable, function (stores either function or macro function), documentation string, property list, type name, class name, slot name, etc.
As long as you have first-class identifiers, you can define arbitrary namespaces without having to worry about conflicts between the namespaces.
> Of course, macros must have their own namespace
That doesn't have to be true, and is not true in Common Lisp: http://www.lispworks.com/documentation/HyperSpec/Body/03_bba...
Having an identifier denote a function and a macro at the same time like that enables adding partial evaluation and other compile-time optimizations to DSLs without having to dig into the compiler.
> but that's an obvious thing, you cannot mix compile-time and run-time namespaces anyway.
? You have to mix them if you want a compiler in your runtime.
Depends on what you mean by 'big'. Physically it is not too big for typical applications. 'mentally' it may be too big for some developers. After a while then greenspun's tenth rule kicks in...
'Common Lisp' also does not mean it needs the full language. Some applications use subsets.
'Common Lisp' also does not mean it needs the full language. Some applications use subsets.
Big is over 1000 pages of a spec.
I know folks used to say that Common Lisp was too big, comparing it to C or Pascal. But is it really big compared to, e.g., Python, JavaScript or Ruby? If anything, a common complaint now is that too little is standardised (which is true, but good portable libraries exist for most such things).
And, as lispm notes, one can implement a subset of Common Lisp—all that's required is that one define the subset.
And, as lispm notes, one can implement a subset of Common Lisp—all that's required is that one define the subset.
Interestingly, so far people preferred to implement dialects of Lisp-1 mostly. Not that many Common Lisp subsets or even Common Lisp-inspired languages around. I can only name Shen as one obvious exception from this rule.
This is a lisp engine written in python - RPython, to be precise.
It literally compiles the lisp code into python bytecode, which is pretty neat - sort of like clojure is for the JVM.
https://github.com/pixie-lang/pixie/blob/master/pixie/vm/com...
Perhaps not magical, but I like the effort (though lisp-as-shell has to contend with emacs first).
It literally compiles the lisp code into python bytecode, which is pretty neat - sort of like clojure is for the JVM.
https://github.com/pixie-lang/pixie/blob/master/pixie/vm/com...
Perhaps not magical, but I like the effort (though lisp-as-shell has to contend with emacs first).
You have misunderstood. Rpython compiles to C and then to a binary. There is no python bytecode there. Only pixie bytecode being interpreted by the compiled binary.
It also has a Jit engine inside it.
It also has a Jit engine inside it.
> This is a lisp engine written in python - RPython, to be precise.
So there is nothing "native" about it, and until we see this on http://benchmarksgame.alioth.debian.org/ no basis for calling it "fast."
So there is nothing "native" about it, and until we see this on http://benchmarksgame.alioth.debian.org/ no basis for calling it "fast."
You don't understand what rpython is. RPython is a statically typed subset of python which compiles to C. It also has code in it for generating a Jit.
You don't understand what the word "native" means. Transpiling to C is not "native." The RPython JIT 'compiler' is a trace optimizer for the bytecode interpreter. There is absolutely nothing "native" about that. If you don't have an assembler, it's not "native."
The trace JIT has an assembler, and the C code is compiled to native assembly.
https://bitbucket.org/pypy/pypy/src/f7bc5ed1602fb522625a3d02...
Assembler.py is the jit assembler.
https://bitbucket.org/pypy/pypy/src/f7bc5ed1602fb522625a3d02...
Assembler.py is the jit assembler.
It compiles to bytecode but it doesn't compile to python bytecode. The bytecode used by pixie is custom and also interpreted by pixie itself[1].
[1]: https://github.com/pixie-lang/pixie/blob/master/pixie/vm/int...
[1]: https://github.com/pixie-lang/pixie/blob/master/pixie/vm/int...
Ever since RPython had been announced I have wanted generator support in RPython. TCO'ed coroutines would be even better. PyPy devs did not see much use of it for writing their Python interprator so they left it out, which is a pity.
How so ever shiny and neat dynamic typing might be, I have had very little use of it in my own Python code. A situation where I absolutely must store values of different types in the same variable (that or bullet to the head) has not presented itself that often if ever. For me RPython would have been a fine enough replacement, except that I do like generators a lot.
Given the prominence of inversion of control (via coroutines) as a feature, I assume it would have helped pixie implementation too.
How so ever shiny and neat dynamic typing might be, I have had very little use of it in my own Python code. A situation where I absolutely must store values of different types in the same variable (that or bullet to the head) has not presented itself that often if ever. For me RPython would have been a fine enough replacement, except that I do like generators a lot.
Given the prominence of inversion of control (via coroutines) as a feature, I assume it would have helped pixie implementation too.
RPython is not a language for normal usage and mostly because e.g. error messages are obscure and hard to read. There is some basic support for generators these days, however, unless you're writing another language, we strongly encourage you don't use it.
I love little languages like this one! Just look at the source — it's still so readable and easy to understand, and the size — I could read through it in an evening!
It reminds me a bit of diy-lisp where you implemented Lisp in Python yourself! https://github.com/daGrevis/diy-lisp
It reminds me a bit of diy-lisp where you implemented Lisp in Python yourself! https://github.com/daGrevis/diy-lisp
I'm glad there will be something like clojure, only with semi sane start up times. (pypy based interpreters are still slow to start.)
I'm happy to see that some new 'modern lisp on x' projects are taking design cues from Clojure (See Rhine and now Pixie).
Clojure's success, I believe, is due to retaining Java's semantics (preserving library compatibility) while overhauling its syntax. Fighting the semantics of your host language/VM just makes work and causes trouble. Therefore, if you want a "Clojure on Python" that preserves the benefits of the original, you need to give it Python semantics, not Java/Clojure's. Hylang is that project - it uses Clojure syntax where sensible, and is fully compatible with Python.
https://groups.google.com/forum/#!topic/clojure-py-dev/HbeNE...
http://hy.readthedocs.org/en/latest/
https://groups.google.com/forum/#!topic/clojure-py-dev/HbeNE...
http://hy.readthedocs.org/en/latest/
Only hylang has a lot of stupid limitations and problems like scope leakage.
I take it you refer to your complaint here: https://github.com/hylang/hy/issues/543
Your problem is with Python, not Hy. Hy inherits the "stupid limitations" of Python, and not even all of those - for example, it does away with the statement/expression distinction, which means multiline lambdas (at last!).
It's just Python with s-expressions. If you don't like it, you either don't like Python or don't like s-expressions.
Your problem is with Python, not Hy. Hy inherits the "stupid limitations" of Python, and not even all of those - for example, it does away with the statement/expression distinction, which means multiline lambdas (at last!).
It's just Python with s-expressions. If you don't like it, you either don't like Python or don't like s-expressions.
I'm completely unfamiliar with Hylang - could you please elaborate on its limitations?
>Math system is fully polymorphic
Does it have fully polymorphic other basic functions, like setq, eql (or however they are named)? I guess not? Do you plan on introducing some parts of CLOS?
Lack of OO integration in CL is one of its biggest drawback for me... for me CLOS was the most powerful thing in lisp, not macros (mostly because they are not dynamic).
Does it have fully polymorphic other basic functions, like setq, eql (or however they are named)? I guess not? Do you plan on introducing some parts of CLOS?
Lack of OO integration in CL is one of its biggest drawback for me... for me CLOS was the most powerful thing in lisp, not macros (mostly because they are not dynamic).
CLOS has been on my list to research for some time now, and I haven't made a decision on it yet. The goal would be to get the JIT to remove the method lookup completely, and that might get a bit tricky depending on how complex the lookup code is, but it is the most open dynamic OO system I've seen, so it might just be worth it.
Neat project, although I am struggling with getting it working. Looking forward to installation instructions.
What's magical about it?
Well judging from the description, "native" actually means writing it in RPython (a "restricted" Python with types for use in the PyPy Project for static analysis and speeding things up[0]) and then compiling it down using PyPy (I am guessing on the second part). That could be neat, the HyLang people (Lisp written in vanilla CPython Python; I think no C extensions AFAIK).
[0] http://pypy.readthedocs.org/en/latest/coding-guide.html#id1
[0] http://pypy.readthedocs.org/en/latest/coding-guide.html#id1
Hy isn't a full Lisp interpreter in Python, it's an s-expression to Python AST parser, which means it does everything Python does (including C extensions) at essentially native speed anyway - you can even write RPython in it.
If you ask me, full Python interop makes Hy the most practical s-expression language today.
If you ask me, full Python interop makes Hy the most practical s-expression language today.
One problem with Hy for me is that it keeps Python scoping, either global or function; that doesn't feel natural to me in a Lisp, and it gets in my way. If it weren't for that and a few similar matters, I'd agree with you on Hy's practicality; as it is, I'd give my nod to Clojure.
I'm wondering the same thing. I dug through the entire repository, and I didn't see any references to "magical" properties.
Did you look through the rpython codebase too? Thats where the real magic is.
Added a few comments in the README about that.
Pixie implements its own virtual machine.
Isn't that how all the tiny lisps do it? :)
You know, maybe 'magic' simply sets the bar higher than the project is prepared to support. Consider 'clever'.
Isn't that how all the tiny lisps do it? :)
You know, maybe 'magic' simply sets the bar higher than the project is prepared to support. Consider 'clever'.
Umm, I don't think you understand fully what it is. it is using rpython which is a meta tracing jit. pretty magic if you read about it.
drtse4(1)
Have fun.
Would love some examples that showoff the benefits
I added a few comments to the README on that subject.
The author is the creator of clojure-py [1]. Pixie looks like a follow-up effort with strong influence from PyPy. The following post of his from over 2 years ago might provide some insight: https://groups.google.com/forum/#!topic/clojure-py-dev/Lmhd0...
[1] https://news.ycombinator.com/item?id=3649883
[1] https://news.ycombinator.com/item?id=3649883
That being said I'll try to answer any questions you may have. Thanks!