Looking for new LuaJIT maintainers(freelists.org)
freelists.org
Looking for new LuaJIT maintainers
http://www.freelists.org/post/luajit/Looking-for-new-LuaJIT-maintainers
39 comments
> And maybe we will see it catch up with Lua 5.3
For those that think that LuaJIT's 5.1 compatibility isn't too far behind 5.3, be aware that Lua changes a lot in its minor versions.
To give you an idea, 5.2 introduced changed how scope works (both some stuff with lexical/global scope, and by redoing environments), introduced the `goto` statement, finalisers, and changed how `break` works.
5.3 introduced integers (Lua was previously like Javascript with only one Number type which was backed by a `double`), and utf8 support.
This means that upgrading a Lua version can be a big deal and break all of the programs written in it. For instance, redis and WoW both still embed 5.1.
For those that think that LuaJIT's 5.1 compatibility isn't too far behind 5.3, be aware that Lua changes a lot in its minor versions.
To give you an idea, 5.2 introduced changed how scope works (both some stuff with lexical/global scope, and by redoing environments), introduced the `goto` statement, finalisers, and changed how `break` works.
5.3 introduced integers (Lua was previously like Javascript with only one Number type which was backed by a `double`), and utf8 support.
This means that upgrading a Lua version can be a big deal and break all of the programs written in it. For instance, redis and WoW both still embed 5.1.
Yes, 5.3 is not a "minor version" increment, it's a major version increment on 5.2. That's how Lua version numbering works.
Adding support for 5.3's integer subtype is probably going to be a very large project, I would guess that annoyed Mike.
People say that LuaJIT's source is difficult to understand, but I've seen source for several VMs with JIT compilers and this one certainly isn't the gnarliest; it's decently well commented. It only took me 30 min to add a new builtin function, with almost no understanding of the codebase beyond skimming a couple header files. As long as you don't want the JIT compiler to output asm for your language extensions, it's almost friendly.
Adding support for 5.3's integer subtype is probably going to be a very large project, I would guess that annoyed Mike.
People say that LuaJIT's source is difficult to understand, but I've seen source for several VMs with JIT compilers and this one certainly isn't the gnarliest; it's decently well commented. It only took me 30 min to add a new builtin function, with almost no understanding of the codebase beyond skimming a couple header files. As long as you don't want the JIT compiler to output asm for your language extensions, it's almost friendly.
Parrotting Mike Pall: having 64 bits integers require 128 bits for representing Lua values. This would be bad performance-wise because it would clutter the CPU cache (as it is the case for Lua 5.3 vs 5.2).
Currently, LuaJIT relies on NAN tagging which allows it to represent values in 64 bits.
Currently, LuaJIT relies on NAN tagging which allows it to represent values in 64 bits.
I noticed that LuaJIT has the LJ_DUALNUM define, which actually internally adds an int32 subtype which causes numbers which can fit in an int32 to be stored as such. It does not change the semantics, it's to avoid floating point ops on (mainly ARM) processors with slow or emulated FPUs.
However, I think it should be possible to extend it without a huge amount of work to implement Lua 5.3 semantics with a 32 bit (rather than the default 64 bit) integer subtype. I think it would still be useful to a lot of people, personally I would love to have it.
Increasing TValues to 128 bits would be very undesirable, but since luajit already supports 64 bit ints through the ffi library I don't see why they couldn't be stored as boxed udatas/cdatas in the same way.
Increasing TValues to 128 bits would be very undesirable, but since luajit already supports 64 bit ints through the ffi library I don't see why they couldn't be stored as boxed udatas/cdatas in the same way.
This is a good initiative because LuaJIT development has been mostly halted since May 2014, when the following note was added to the LuaJIT Open Sponsorships page:
"The main LuaJIT author (Mike Pall) is working on unrelated projects and cannot accept new sponsorships at this time..." [1]
Since then, Mike has only made bugfix commits to the project.
I really hope that the features mentioned on that Open Sponsorships page start happening after these news, and CloudFlare seems like a good candidate to me: even without knowing how much resources they're ready to offer, I'm willing to bet it will be an improvement over the current situation.
Everyone was surprised when LuaJIT seized the position of "fastest scripting language in the west", and it would be awesome if they could manage to keep that status - or has it been taken over by Javascript already? JS implementations have been improving dramatically thanks to the competition between browsers, and V8/SpiderMonkey are implementing features from LuaJIT's TODO list right now (I'm thinking about vectorized SIMD instructions). A lot of people use benchmarks as a major factor for choosing a scripting language, and I'd rather see more projects using Lua/LuaJIT rather than JS as a scripting language. It will be interesting to see how this turns out. Go LuaJIT!
[1]: http://wiki.luajit.org/Open-Sponsorships
"The main LuaJIT author (Mike Pall) is working on unrelated projects and cannot accept new sponsorships at this time..." [1]
Since then, Mike has only made bugfix commits to the project.
I really hope that the features mentioned on that Open Sponsorships page start happening after these news, and CloudFlare seems like a good candidate to me: even without knowing how much resources they're ready to offer, I'm willing to bet it will be an improvement over the current situation.
Everyone was surprised when LuaJIT seized the position of "fastest scripting language in the west", and it would be awesome if they could manage to keep that status - or has it been taken over by Javascript already? JS implementations have been improving dramatically thanks to the competition between browsers, and V8/SpiderMonkey are implementing features from LuaJIT's TODO list right now (I'm thinking about vectorized SIMD instructions). A lot of people use benchmarks as a major factor for choosing a scripting language, and I'd rather see more projects using Lua/LuaJIT rather than JS as a scripting language. It will be interesting to see how this turns out. Go LuaJIT!
[1]: http://wiki.luajit.org/Open-Sponsorships
"fastest scripting language in the west"
"fastest scripting language in the east"
But then again - Lua originated from Brazil and heavily used in game companies in the U.S. Not sure how much they use LuaJIT though.
This is a reference to the trope of quickdraw duels and "Fastest gun in the west" referring to the western United States in the "Wild West". Not an east/west of the world like you're thinking.
Hmm...wonder what this means about the status of the Quad-color bitmapped arena mark-n-sweep collector Mike was planning for LuaJIT 3.0 (http://wiki.luajit.org/New-Garbage-Collector)?
I remember reading what he had researched so far a couple years back and being really excited, but it seems like the work was never really continued. I'd be interested to know if there was some fatal flaw that would've kept the collector from living up to its potential? or was it just a matter of the sizable effort required to write a new collector, more-or-less from scratch?
I remember reading what he had researched so far a couple years back and being really excited, but it seems like the work was never really continued. I'd be interested to know if there was some fatal flaw that would've kept the collector from living up to its potential? or was it just a matter of the sizable effort required to write a new collector, more-or-less from scratch?
AFAIK, he never moved beyond planning it, unless the GC64 work is also a prereq for it. E.g. almost no significant updates to the new GC wiki page or mention of GC in git commits since 2012.
BTW, previous HN discussion of the new GC: https://news.ycombinator.com/item?id=4053969
BTW, previous HN discussion of the new GC: https://news.ycombinator.com/item?id=4053969
There is this recent thread[1] where people are looking for resources to help them understand the project. There are interesting pointers[2] as well as a part of a PhD thesis[3] that covers it.
[1] http://www.freelists.org/post/luajit/how-to-read-the-source-... [2] http://www.freelists.org/post/luajit/how-to-read-the-source-... [3] http://www.freelists.org/post/luajit/how-to-read-the-source-...
[1] http://www.freelists.org/post/luajit/how-to-read-the-source-... [2] http://www.freelists.org/post/luajit/how-to-read-the-source-... [3] http://www.freelists.org/post/luajit/how-to-read-the-source-...
Mike Pall is an international treasure whose work on LuaJIT has been without parallel in the embedded/gaming world. I hope this move is born of desire and moving on to great new things rather than any type of necessity.
I've never heard any actual stories of LuaJIT being used in games as opposed to stock Lua, but I know it was ported to power processors. Have any big games claimed to use LuaJIT? It would certainly make sense if the memory and gc pauses were no worse than Lua.
No big games that I know of, but it's certainly used in games; the LOVE engine embeds LuaJIT.
Can anyone actually fill Mike Pall's shoes? I always got the impression that what he was doing came from a very rare combination of JIT and low level optimization knowledge.
I'm a big fan of Luajit, I think it's ease of use of FFI makes it fantastic to integrate into bigger projects; all that 'glue' that I haven't had to write !!
Hope someone pick it up; I can understand he wants to move to other things, and sometime a projects becomes a bit too 'big' and you can feel a bit shackled to it after a while...
Wonder what Mike Pall will be up to next...
Maybe he'll find some other language to make an order of magnitude faster.
Hoping he picks python :)
I think he has written about why Lua was so good for JIT compilation and in doing so talked about python. I don't remember specifically what he said, but he had analyzed other languages and there were hurdles that kept them from being able to achieve the same speed as LuaJIT.
Facebook have said in the past that PHPs weirdness made getting HHVM off the ground a lot of hard work, and Javascript JITs have struggled for generations to get bang-for-the-buck in the browser
Are there any available writeups or talks which discuss these challenges for HHVM? This sounds like it would be very interesting.
here: http://hhvm.com/blog/2027/faster-and-cheaper-the-evolution-o...
http://hhvm.com/blog/431/on-garbage-collection
and some other stuff from page 6 forward: http://hhvm.com/blog/page/6
http://hhvm.com/blog/431/on-garbage-collection
and some other stuff from page 6 forward: http://hhvm.com/blog/page/6
Yes. Lua is a very small and orthogonal language, so even tiny improvements in one part of it help most programs.
For example, if you've figured out a small increase in reading a value from a table, you have sped up all other data structures, OO code, etc., because the table is used for everything.
For example, if you've figured out a small increase in reading a value from a table, you have sped up all other data structures, OO code, etc., because the table is used for everything.
You can start from here http://lambda-the-ultimate.org/node/3851#comment-57805 or read the whole thread.
I'd like to see cPython without a GIL first. It would be really nice to take advantage of multi core CPUs without having to use subprocesses.
What's the disadvantage of subprocesses?
There are plenty, but here's an easy example: sockets. File descriptors are shared across process forks. So let's take a typical web application that does:
1. Boot, setting up system wide resources like DB/memcached connections
2. Start doing work
Then any process forked after step #1 is sharing those sockets and can accidentally corrupt the work that the parent process is doing. With threads, you can share the connections through a pool that can be simply mediated within the process.
Yeah, you can design around it. But with threads, you don't have to.
Here's another: serialisation. If you want to parallelise any processing that builds up any complex data structures, you have to serialise them between your parent/child processes which can make it hard to keep the "richness" of the objects like interdependencies between processing runs. In addition, now your processing bottleneck becomes how fast your single-threaded parent process can deserialise the responses (which in Python in particular can really be quite slow).
And not everything can be serialised. It's not trivial to share sockets with other processes that have already been started. You can't easily accept a connection in one process and pass it to another to do the work and send the response (it's possible, just not as easy as passing an integer around like you can do between threads).
1. Boot, setting up system wide resources like DB/memcached connections
2. Start doing work
Then any process forked after step #1 is sharing those sockets and can accidentally corrupt the work that the parent process is doing. With threads, you can share the connections through a pool that can be simply mediated within the process.
Yeah, you can design around it. But with threads, you don't have to.
Here's another: serialisation. If you want to parallelise any processing that builds up any complex data structures, you have to serialise them between your parent/child processes which can make it hard to keep the "richness" of the objects like interdependencies between processing runs. In addition, now your processing bottleneck becomes how fast your single-threaded parent process can deserialise the responses (which in Python in particular can really be quite slow).
And not everything can be serialised. It's not trivial to share sockets with other processes that have already been started. You can't easily accept a connection in one process and pass it to another to do the work and send the response (it's possible, just not as easy as passing an integer around like you can do between threads).
You might find PyParallel interesting: http://pyparallel.org.
Are you talking about Python limitations?
Because for 1) if you can put your sockets 'close on exec' no?
For 2) how about using shared memory?
> Are you talking about Python limitations?
Yes, but:
> for 1) if you can put your sockets 'close on exec' no?
Yes, if you're the one that opened the socket. If you're using a library that opens that socket for you (e.g. a memcached or postgres library in my example), you don't necessarily have access to the socket itself. And now instead of children corrupting the parent's socket, you just have children with closed sockets. So you still have to solve the inherited-resource-sharing problem, FD_CLOEXEC isn't magic fairy dust.
> 2) how about using shared memory?
Possible but has its own downsides (you can't lock with regular posix mutexes, you have to make sure that the memory you're sharing is all in the shared block so you can't just use malloc, which means that you can't share objects that you don't control the allocation of, like, say, libraries' objects).
Yes, these are solveable problems. There are lots of other ones, these are just easy examples. Like I said:
> Yeah, you can design around it. But with threads, you don't have to.
Yes, but:
> for 1) if you can put your sockets 'close on exec' no?
Yes, if you're the one that opened the socket. If you're using a library that opens that socket for you (e.g. a memcached or postgres library in my example), you don't necessarily have access to the socket itself. And now instead of children corrupting the parent's socket, you just have children with closed sockets. So you still have to solve the inherited-resource-sharing problem, FD_CLOEXEC isn't magic fairy dust.
> 2) how about using shared memory?
Possible but has its own downsides (you can't lock with regular posix mutexes, you have to make sure that the memory you're sharing is all in the shared block so you can't just use malloc, which means that you can't share objects that you don't control the allocation of, like, say, libraries' objects).
Yes, these are solveable problems. There are lots of other ones, these are just easy examples. Like I said:
> Yeah, you can design around it. But with threads, you don't have to.
[deleted]
They cant share data as efficiently as threads because they don't share the same address space.
I got that working with PyParallel: http://pyparallel.org.
nah, ruby you folks have numpy and scipy already.
[deleted]
[deleted]
On the negative side, Mike Pall is an impressive individual who has contributed a lot to the field of JIT-compiled languages, so losing him is obviously bad for the project. LuaJIT is also a hard codebase to get into, I myself tried to get into it and gave up (I blogged about it last month [1]). There are probably still parts of it that only Mike understands, I guess we will see how much in the next months.
On the positive side, this will forcefully increase the bus factor of the project and take it out of this situation. And maybe we will see it catch up with Lua 5.3 :)
[1] http://blog.separateconcerns.com/2015-06-18-love-hate-luajit...