Concurrency with Python: Threads and Locks(bytes.yingw787.com)
bytes.yingw787.com
Concurrency with Python: Threads and Locks
https://bytes.yingw787.com/posts/2019/01/12/concurrency_with_python_threads_and_locks/
21 comments
Do people do that in practice? I would think since async/await does not reference any particular thread that it is always off of main thread.
The Counter example has a bug -- it doesn't join the threads before printing Counter.count. this could partially explain the low answer.
Global lock much?
You can use threads, locks, and concurrency usefully even if you have a GIL.
Sadly, that's not practically possible.
Yes, you can use multiple processes to mitigate some of the problems, but you will create other problems (for example: you now have to serialize all the objects through a message pipeline, which is expensive, or you have to use shared memory, which also requires copy operations, unless you want to write your entire code in a low-level kind of way but in that case, you might as well use a different language like C). In an existing system, moving everything to a different process can be a cumbersome experience.
What you really want is multicore support with a garbage collector that is concurrent and a runtime that doesn't have a global lock. Sadly, few environments support this. Erlang comes to mind, but also doesn't support structural sharing. Ocaml is working on multicore support [1]. GoLang, Haskell and Java seem your best choice for GC+multicore.
[1] https://github.com/ocamllabs/ocaml-multicore
Yes, you can use multiple processes to mitigate some of the problems, but you will create other problems (for example: you now have to serialize all the objects through a message pipeline, which is expensive, or you have to use shared memory, which also requires copy operations, unless you want to write your entire code in a low-level kind of way but in that case, you might as well use a different language like C). In an existing system, moving everything to a different process can be a cumbersome experience.
What you really want is multicore support with a garbage collector that is concurrent and a runtime that doesn't have a global lock. Sadly, few environments support this. Erlang comes to mind, but also doesn't support structural sharing. Ocaml is working on multicore support [1]. GoLang, Haskell and Java seem your best choice for GC+multicore.
[1] https://github.com/ocamllabs/ocaml-multicore
> Sadly, that's not practically possible.
Why? With cpython there are plenty of cases where parallelism is possible with threads because the GIL is released (for instance most IO operations and many C based number crunching operations), making threads and locks useful.
Why? With cpython there are plenty of cases where parallelism is possible with threads because the GIL is released (for instance most IO operations and many C based number crunching operations), making threads and locks useful.
Yes, but it's very easy to hit a number crunching operation that locks the GIL.
So technically if you wrote most of your number-crunching code as a C module for python, then you could use threads...
Can't you see a problem here?
Can't you see a problem here?
If you don't want to use C (I don't blame you if you don't) you could use Cython or Numba. And certainly in my number crunching code the amount of code that actually crunches numbers, and thus could really benefit from being written in a GIL releasing way, is really quite small.
No you can still use pure Python, and use multiple threads, and get concurrency just fine. The global lock doesn't stop you doing that! C extensions being able to release the lock is just a bonus on top.
Lots of problems are merely I/O related, which can be solved just fine with Python threads. As for number crunching (let's assume that means CPU intensive tasks), you can always still resort to multiprocessing (which can also be combined with multithreading, I fail to see why such paradigms cannot be combined).
Well, io-related problems can be solved even better by using gevent.
The io-related problems I had with paramiko, on the other hand - last time I checked could neither be solved with threads or gevent.
So that’s a weak point to be made anyway.
The io-related problems I had with paramiko, on the other hand - last time I checked could neither be solved with threads or gevent.
So that’s a weak point to be made anyway.
You don’t need to run multiple processes - Python already has shared memory threads (that’s what the article is about) - and these work just fine concurrently even with the global lock. The big lock stops you doing some things but not many.
> What you really want is multicore support with a garbage collector that is concurrent and a runtime that doesn't have a global lock. Sadly, few environments support this.
I believe .NET also fits the bill.
I believe .NET also fits the bill.
Shallow drive by meme posts are frowned upon on HN
What's your point? Please elaborate!
Too bad, because you want to isolate things sometime.