Writing a chat application in Django 4.2 using async StreamingHttpResponse(valberg.dk)
valberg.dk
Writing a chat application in Django 4.2 using async StreamingHttpResponse
https://valberg.dk/django-sse-postgresql-listen-notify.html
35 comments
Can second the Django/Gevent/Gunicorn stack - we use it in production and IMO it's much easier to reason about and code in than asyncio. Just write synchronous Python, including long-running `requests` calls, and the system will yield control whenever you're blocked on network or disk I/O, no matter how deep that is in your stack. The entire ecosystem of Python libraries just works. I also wish more people knew about gevent - it's truly magical, especially if you need to work with APIs with unpredictable latency!
Where can I learn more about this? I've been thinking of trying to integrate Supabase Realtime (https://github.com/supabase/realtime) into my Django app (without the rest of Supabase), but I'd also like to keep things even simpler if possible.
Also, what was the reason not to go with Gevent?
Also, what was the reason not to go with Gevent?
>Also, what was the reason not to go with Gevent?
The biggest downside of Gevent IMO is that it enables the magic of turning sync code into async code via monkey patching things like the socket lib. This lack of explicitness can make things a bit difficult to reason about, without a good mental model of what Gevent is doing underneath the hood.
The biggest downside of Gevent IMO is that it enables the magic of turning sync code into async code via monkey patching things like the socket lib. This lack of explicitness can make things a bit difficult to reason about, without a good mental model of what Gevent is doing underneath the hood.
I'd strongly recommend django-channels if you want to change very little. It feels like Django and it integrates without a lot of headache (at least in my experience).
Hi, thank you for your comment. I have been looking for correct ways to integrate websockets on my django app. Currently I'm using gevent gunicorn setup. Can you elaborate a bit more on how to make websockets work with an existing app? Thanks.
I did this. If you're on a modern version of Django, it's actually pretty easy: swap out gunicorn for an async alternative and install django-channels. I even got it working with GraphQL subscriptions (using graphene) without much hassle.
Just a heads up that currently Django is not cleaning up open PostgreSQL connections when ran in ASGI mode, leading to too many open connections error: https://code.djangoproject.com/ticket/33497
I'll have zero trust running async Django until all sync_to_async and async_to_sync calls are removed from the code base.
Same here, but without these weird utils it doesn't get any better.
I have 7 YoE with Django. Its great at so many things. You see some code, like middlewares, and immediately understand what's going on.
Now, we also have Starlette. The base of all new, fancy asgi libraries. Here's the base middleware class.
https://github.com/encode/starlette/blob/8d7a1cacfb3e1a30cbb...
In the last couple of years I heard 'we're running fastapi on production. Wanna join us?' so many times... but the reality is that it's still not suitable for prod. Who wants to work with a code like that if you have a readable, stable Django? I'm clueless.
I have 7 YoE with Django. Its great at so many things. You see some code, like middlewares, and immediately understand what's going on.
Now, we also have Starlette. The base of all new, fancy asgi libraries. Here's the base middleware class.
https://github.com/encode/starlette/blob/8d7a1cacfb3e1a30cbb...
In the last couple of years I heard 'we're running fastapi on production. Wanna join us?' so many times... but the reality is that it's still not suitable for prod. Who wants to work with a code like that if you have a readable, stable Django? I'm clueless.
[deleted]
Care to elaborate?
Look at the intended semantics [1], and then read the implementation [2]. Can you figure out if the implementation is correct? Can you infer the possible limitations of the approach at glance? Can your async library actually handle being called with multiple event loops installed?
I have zero trust in this code and I have been bitten by fixes to this library that introduced deadlocks in my own code.
[1] https://github.com/django/asgiref#synchronous-code--threads.
[2] https://github.com/django/asgiref/blob/main/asgiref/sync.py#...
I have zero trust in this code and I have been bitten by fixes to this library that introduced deadlocks in my own code.
[1] https://github.com/django/asgiref#synchronous-code--threads.
[2] https://github.com/django/asgiref/blob/main/asgiref/sync.py#...
Doesn't look like something I'd want to have to debug in a pinch, that's for sure.
Is there anything else in Python-land you would go with for the use case you have in mind? Or does pretty much all the async stuff turn on you like this, at some point or another?
Is there anything else in Python-land you would go with for the use case you have in mind? Or does pretty much all the async stuff turn on you like this, at some point or another?
If you need to buy into the async ecosystem for some special reason my recomendation is to:
- pick a mature ASGI server
- pick a framework and libraries that have been designed from the start with async in mind. Be suspicious of libraries that support sync and async APIs without clear guard rails between the too implementations
- avoid setups where you have multiple event loops in a single process.
- pick a mature ASGI server
- pick a framework and libraries that have been designed from the start with async in mind. Be suspicious of libraries that support sync and async APIs without clear guard rails between the too implementations
- avoid setups where you have multiple event loops in a single process.
Thanks - so is there anything in the Python space that ticks those boxes, in your view?
Yeah, I've definitely seen odd things in Django ASGI. Had at least one instance where it looked like it leaked data between requests but couldn't get a clear enough reproduce. Switching back to WSGI worked fine though
that’s kind of a dealbreaker
could this be addressed by something like pgbouncer or another connection pooler?
Going off the pooling mode feature map[0], it might be possible to mitigate that issue using transaction pooling, but not session pooling — which means a restricted feature set and cooperation from the application (which would just not call restricted features up).
[0] http://www.pgbouncer.org/features.html#fn:2Perhaps yes, would be worthwhile to benchmark an internal (to your runtime) conn pooler versus external only.
I'm using internal pools + pgbouncer but as I write this out I am beginning to wonder if that is even necessary.
I'm using internal pools + pgbouncer but as I write this out I am beginning to wonder if that is even necessary.
This implementation is probably a little different but we were using long poll in the late 90s and early 2000s. The problem was that you were committing one thread (or worse, one process) to each client. That obviously doesn't scale unless threads are extremely light on RAM and either the OS or the runtime support a large number of them. I remember that a way out was using continuations. Jetty was a Java application server that supported them (random link [1]) One thread -> many connections. I didn't investigate how Django is implementing this now but CPUs and RAM are still CPUs and RAM.
[1] https://stackoverflow.com/questions/10587660/how-does-jetty-...
[1] https://stackoverflow.com/questions/10587660/how-does-jetty-...
> The problem was that you were committing one thread (or worse, one process) to each client.
This is mostly true today as well, although we do have beefier machines and more efficient thread pooling.
I did some personal research for infrequent real-time notifications. My conclusion was that many stateful connections are poorly memory-optimized by language runtimes and reverse proxies. Even with lightweight tech like Golang, gRPC, nginx, etc, it’s hard to push anything less than 30 kB for an idle conn, mostly from thread/goroutine stacks, user space buffers and (easy to overlook) a bunch of HTTP headers and TLS handshake state that often remain after establishing the conn. That’s without any middleware or business logic wants their share of the cake.
The only mature project I found that really took this stuff seriously is uWebSockets. It’s extremely lightweight, around an OOM better than most alternatives. Highly recommend – they also have a socket library so you can implement other protocols if needed.
Anyway, it’s important to be aware that massive amounts of long running connections is not just about adding a websocket/SSE library. Chances are you’ll need a mostly-separate serving stack for that purpose.
This is mostly true today as well, although we do have beefier machines and more efficient thread pooling.
I did some personal research for infrequent real-time notifications. My conclusion was that many stateful connections are poorly memory-optimized by language runtimes and reverse proxies. Even with lightweight tech like Golang, gRPC, nginx, etc, it’s hard to push anything less than 30 kB for an idle conn, mostly from thread/goroutine stacks, user space buffers and (easy to overlook) a bunch of HTTP headers and TLS handshake state that often remain after establishing the conn. That’s without any middleware or business logic wants their share of the cake.
The only mature project I found that really took this stuff seriously is uWebSockets. It’s extremely lightweight, around an OOM better than most alternatives. Highly recommend – they also have a socket library so you can implement other protocols if needed.
Anyway, it’s important to be aware that massive amounts of long running connections is not just about adding a websocket/SSE library. Chances are you’ll need a mostly-separate serving stack for that purpose.
“ The idea is that the client "subscribes" to an HTTP endpoint, and the server can then issue data to the client as long as the connection is open.”
To those who have been around longer than me: isn’t this just long polling, that predates websocket?
To those who have been around longer than me: isn’t this just long polling, that predates websocket?
https://github.com/sysid/sse-starlette makes token streaming so much easier in python
[deleted]
I’m using this for an internal ChatGPT UI clone and it’s working great.
The actual biggest pain for me has been the front end handling of it with the fetch api. But that’s likely just due my inexperience with it.
The actual biggest pain for me has been the front end handling of it with the fetch api. But that’s likely just due my inexperience with it.
Do modern chat actually use websocket and the like? Discord / Slack on the web ( browser ) what do they use?
The slack_bolt library has support for web sockets.
https://api.slack.com/apis/connections/socket
https://api.slack.com/apis/connections/socket
Discord uses websockets on all clients/platforms.
yes they use websockets
One thing to note, it's been possible to build these sort of things with Django by using Gevent and Gunicorn for over a decade, and it works well.
In many ways I wish Gevent had been adopted by Python rather than AsyncIO.