HackerTrans
TopNewTrendsCommentsPastAskShowJobs

Znafon

no profile record

Submissions

A Per-Interpreter GIL

lwn.net
2 points·by Znafon·3 tahun yang lalu·0 comments

Lolwut: A piece of art inside a database command

antirez.com
1 points·by Znafon·3 tahun yang lalu·0 comments

comments

Znafon
·3 tahun yang lalu·discuss
"Nothing came of it, but I took the code and shoved it into my back pocket for a rainy day.

My idea was to take this code and spruce it up for Uber’s use case."

"My first reaction was to publish the code on Github."

I’m very surprised by this, isn’t the code property of Box, or Uber? The author does not mention their authorisation before releasing it under MIT license.
Znafon
·3 tahun yang lalu·discuss
The AGPL says "including scripts to control those activities" but if I understand what you are saying properly it means that the text of the AGPL is overreaching and some parts of it wouldn't be applicable?

Would the details depend on the country? Like it might be one thing under US law, and another in another country?
Znafon
·3 tahun yang lalu·discuss
The AGPL defines "Corresponding Source" as "all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities".

Then in section 13 it says that "your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge".

Isn't "scripts to control those activities quite vague? Under those terms wouldn't AWS Proton, IBM Schematics, Env0, Digger, Spacelift, etc. be required to open-source a very large part of their product as well?
Znafon
·3 tahun yang lalu·discuss
It's not just big tech that hates AGPL. If Terraform was under AGPL a lot of companies might have to release their source code as AGPL too because of its virality. Where it stops is not exactly clear from the license.
Znafon
·3 tahun yang lalu·discuss
You could modify it easily, for example the user could register an additional function to the mapping. That would not be as easy to do with a module and getattr().

You can also list the registered function with math_exprs.keys() and that might not be as easy with a module either.
Znafon
·3 tahun yang lalu·discuss
Oh yeah, that’s right! Thanks for the correction
Znafon
·3 tahun yang lalu·discuss
I'm not sure

    print(firstname, lastname)
for example is more readable than

    print((firstname, lastname))
especially since I would then have to write

    print((surname,))
to just print a single string.

Variadic functions are rather classic, I think Go, Rust, C and JavaScript also have them.
Znafon
·3 tahun yang lalu·discuss
I think it is consistent, it works a bit like filtering an element from a mathematical set.

Given a set of sheeps, let x be the five-legged sheep is inconsistent because we know neither the existence or uniqueness of shuch sheep, so it raises an exception.

Given a set of sheeps, let x be the subset of five legged sheeps is the empty set because there is no such sheep.

but this may also just be because I internalised Python's behavior.

Some language have a specific value to denote the first thing, for example:

   ["a", "b", "c"][4]
gives `undefined` in JavaScript but it differs from `null` which would be the equivalent to `None` in Python (and I don't think Python has such concept).
Znafon
·3 tahun yang lalu·discuss
It is used when the number of argument can vary, like:

    def sum(*args: int) -> int:
        if len(args) == 0:
            return 0
        return args[0] + sum(*args[1:])
Znafon
·3 tahun yang lalu·discuss
spacectl, vcs-agent and terraform-provider-spacelift are only useful with your proprietary product. prometheus-exporter is a very small exporter to monitor your proprietary product and has no utility beyond that.

I'm not sure what spcontext does because it has no documentation but I'm sure I could read the 10 commits to find out. The only none trivial project in the list is celplate which actually looks nice, but even there most of its complexity is in cel-spec.

> Whenever there's an opportunity to sponsor a project on which we heavily rely, we do that.

Come on, on Spacelift GitHub profile there is one project sponsored, and only since August 24!

> we don't maintain any major open source projects as a company

So you never had to handle project management of large open-source supports, community support, doing code reviews every day and handling feature requests from the community?

Yes, HashiCorp has been slow to respond to PRs and did not always commmunicate clearly, but so does many large open-source project like PostgreSQL, Python, Linux, etc. Managing a large open-source project takes more time than clicking on the merge button on GitHub. Users don't magically come fix all the bugs in the software, and develop complex new features.

HashiCorp did contribute a lot, and there is still a lot to learn from their projects, there is Raft, the autopilot, various library that are used a lot in the Go ecosystem, including go-plugin, a programming language with its specific type system and plenty more. After all, the change of license is making noises because we their tools defines part fo the DevOps world and we all use them a lot.

I wish HashiCorp would have kept using the MIT license, but using the BUSL is still miles ahead better than companies that are completely closed source.

From the outside perspective it looks like you support Free Software as in Free Beer more than in Free Speech. You are supportive when others are actually maintaining the projects for you, and you just want them to merge your contributions quickly. It is a very efficient way to externalize some of your costs, but you only step forward to actually to help manage them when your bottom line is threatened.

That's ok, it looks like you love money more than open-source ideas. That's perfectly fine but don't pretend otherwise.

Some others companies that signed the pledge have actually been maintaining open-source projects and have a leg to stand on. Spacelift loves having the high moral ground and the extra publicity.

Release the core of your product as open-source, like HashiCorp did for 9 years, and it will change my mind.
Znafon
·3 tahun yang lalu·discuss
So the software from your company is fully closed-source, isn't it?

The employes can do open-source on Friday but no part of Spacelift is actually open-source?
Znafon
·3 tahun yang lalu·discuss
This is a very non-charitable read of the history here.

Also, none of the companies that pledge FTE support to OpenTF are open-source.
Znafon
·3 tahun yang lalu·discuss
The PEP goes into details about the motivations and why the single thread model has limitations: https://peps.python.org/pep-0703/#motivation.

> Yes, some applications will remain hard / impossible, but I'd argue Python is not a good fit for apps requiring high performance / high scalability anyway

People are already trying to use Python for parallel tasks. Forcing people to use another language does not help them much.
Znafon
·3 tahun yang lalu·discuss
They were always usable using the C-API but not exposed to the Python library (and until 3.12 you were blocked by the shared GIL so there was little point anyway).

Now that the work from Eric Snow has been merged, you can use https://pypi.org/project/interpreters-3-12/ to create one from Python code.
Znafon
·3 tahun yang lalu·discuss
> I would be fine with have it never be removed, and I think we should try sub-interpreter first.

There is a lot of work going on with sub-interpreters and the per-interpreter GIL is shipped in Python 3.12.

The results are very impressive: https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/, as good as one could have hoped I think.

It seems to me like the work on sub-interpreters will continue in parallel ;) to the work on free-threading.

Sub-interpreters and no-GIL have different use-cases though.
Znafon
·3 tahun yang lalu·discuss
I think you can make a non thread-safe list by just making the same object without the lock described at https://peps.python.org/pep-0703/#container-thread-safety if you really want the maximum performance of single threads.

Maybe it could be part of a non_threadsafe_containers module on Pypi.
Znafon
·3 tahun yang lalu·discuss
Yes this is a thing was trivially protected by the GIL. There is the same thing with mutating the same map concurrently in Go that will panic for example.

PEP 703 goes over this in the "Container Thread-Safety" (I think container here refers to the fact that the object has references to other objects, this is the things that already are special-cased in CPython to be managed specifically by the garbage collector):

> This PEP proposes using per-object locks to provide many of the same protections that the GIL provides. For example, every list, dictionary, and set will have an associated lightweight lock. All operations that modify the object must hold the object’s lock. Most operations that read from the object should acquire the object’s lock as well; the few read operations that can proceed without holding a lock are described below.

More information at https://peps.python.org/pep-0703/#container-thread-safety
Znafon
·3 tahun yang lalu·discuss
Would the work on sub-interpreters be interested for that then (https://lwn.net/SubscriberLink/941090/8bcb029dbf548f26/) ?
Znafon
·3 tahun yang lalu·discuss
> If you need concurrency at the moment, you have already switched to using multiprocessing, so having a no-GIL multithreading is useless.

> The only issue with Python/multiprocessing, is that sometimes you don't want queues, but shared mutable state. And as you said, placing Python objects in shared memory at the moment is convoluted, restrictive, and suboptimal.

The PEP goes into the motivation behind this work, and using multiple process does not magically solves all the issues:

> Multiprocessing, with communication via shared memory or UNIX sockets, adds much complexity and in effect rules out interacting with CUDA from different workers, severely restricting the design space.

> I reimplemented parts of HMMER, a standard method for multiple-sequence alignment. I chose this method because it stresses both single-thread performance (scoring) and multi-threaded performance (searching a database of sequences). The GIL became the bottleneck when using only eight threads. This is a method where the current popular implementations rely on 64 or even 128 threads per process. I tried moving to subprocesses but was blocked by the prohibitive IPC costs.

> NumPy does release the GIL in its inner loops (which do the heavy lifting), but that is not nearly enough. NumPy doesn’t offer a solution to utilize all CPU cores of a single machine well, and instead leaves that to Dask and other multiprocessing solutions. Those aren’t very efficient and are also more clumsy to use. That clumsiness comes mainly in the extra abstractions and layers the users need to concern themselves with when using, e.g., dask.array which wraps numpy.ndarray. It also shows up in oversubscription issues that the user must explicitly be aware of and manage via either environment variables or a third package, threadpoolctl. The main reason is that NumPy calls into BLAS for linear algebra - and those calls it has no control over, they do use all cores by default via either pthreads or OpenMP.

and it discusses the alternatives at https://peps.python.org/pep-0703/#alternatives.
Znafon
·3 tahun yang lalu·discuss
The performance patches have already been merged and there is ongoing working in the Faster CPython project.