Real-Time Django: Get Started with Django Channels(blog.heroku.com)
blog.heroku.com
Real-Time Django: Get Started with Django Channels
https://blog.heroku.com/archives/2016/3/17/in_deep_with_django_channels_the_future_of_real_time_apps_in_django
120 comments
It's great to see Django getting first class support for real-time. I remember building a hacked-together real-time chat 10 years ago using Python using ajax, but I haven't done much since. I'll have to check it out.
This sounds amazing! Very excited to give it a try. I am working on a side project that I wanted to add real-time capabilities to and I'm not very happy about my Go + Django hybrid solution.
I assume this also means support for Rails 5's ActionCable.
Does this replace Celery+RabbitMQ/Queue service?
From the article:
"Remember: Channels is a general-purpose utility for running background tasks. Thus, many features that used to require Celery or Python-RQ could be done using Channels instead. Channels can’t replace dedicated task queues entirely: it has some important limitations, including at-most-once delivery, that don’t make it suitable for all use cases. Still, Channels can make common background tasks much simpler. For example, you could easily use Channels to perform image thumbnailing, send out emails, tweets, or SMSes, run expensive data calculations, and more."
"Remember: Channels is a general-purpose utility for running background tasks. Thus, many features that used to require Celery or Python-RQ could be done using Channels instead. Channels can’t replace dedicated task queues entirely: it has some important limitations, including at-most-once delivery, that don’t make it suitable for all use cases. Still, Channels can make common background tasks much simpler. For example, you could easily use Channels to perform image thumbnailing, send out emails, tweets, or SMSes, run expensive data calculations, and more."
I'd like to see an example of that vs. using celery.
Nice to see Django moving forward but is still backwards friendly. Love that.
"Channels is to become a built-in feature for Django 1.10, but in order to aid adoption and to encourage community usage and support, it will also be backported to Django 1.8 and 1.9 via a third-party app."
ASGI seems to be a very important piece of this.
"Despite the name of the proposal, ASGI does not specify or design to any specific in-process async solution, such as asyncio, twisted, or gevent. Instead, the receive_many function can be switched between nonblocking or synchronous. This approach allows applications to choose what’s best for their current runtime environment; further improvements may provide extensions where cooperative versions of receive_many are provided."
"Channels is to become a built-in feature for Django 1.10, but in order to aid adoption and to encourage community usage and support, it will also be backported to Django 1.8 and 1.9 via a third-party app."
ASGI seems to be a very important piece of this.
"Despite the name of the proposal, ASGI does not specify or design to any specific in-process async solution, such as asyncio, twisted, or gevent. Instead, the receive_many function can be switched between nonblocking or synchronous. This approach allows applications to choose what’s best for their current runtime environment; further improvements may provide extensions where cooperative versions of receive_many are provided."
This is great news. Django is a great web dev tool and having websockets support is icing on the cake. I guess this adds more weight to the thought I've been having: That Web applications are moving away from MVC and more into messaging layers. Front end JS frameworks have pushed things to be this way. Which brings the Web into a pseudo native environment of sorts. Anyhow, will be happily tinkering with this from today onwards.
The good news is that it's "real-time Django". The bad news is that it's Django.
Why?
Python is a language that gives us a lot of liberty in how we solve problems, but Django is a set of cookie cutters for doing things a certain way. Django's cookie cutter patterns and extensions don't lend themselves well to doing things outside the way that Django intended them to. It's why a lot of us prefer Flask or Pyramid as framework of choice.
http://nando.oui.com.br/2014/04/04/why_i_sort_of_dislike_dja...
http://nando.oui.com.br/2014/04/04/why_i_sort_of_dislike_dja...
I think the objection is referring to Django unequivocally as "bad news", rather than saying that your personal preference is for Flask.
(I work in a shop which has used both, and generally try to avoid claiming one or the other is objectively better; they're not quite at the apples-and-oranges level, but pretty close)
(I work in a shop which has used both, and generally try to avoid claiming one or the other is objectively better; they're not quite at the apples-and-oranges level, but pretty close)
I just get tired of seeing the same thing play out time and time again. Developer A picks Django because business described a project that can easily be handled by CRUD and thinks the Admin will be perfect for product owners and users to manage their data. 6 months go by, Product owners go off the rails of the original requirements, request changes to the admin and/or changes to the data source and/or changes that require a fundamentally different data model. Developer A bails and Developer B, C, and D get stuck with this mess that should be been developed with a more flexible solution. The more people use Django without recognizing this, the more I want to not even look at Python shops.
I've been working with Django for years and I can't think of any way in which it is not flexible.
I wrote a CMS for a large newspaper where I deployed new code on almost a daily basis using Django(uWSGI)-redis-PostgreSQL-Celery. 4 developers were active at the time. Requirements were not set in stone, we tested tons of stuff out and iterated every aspect of the site both on the development and operations side. Django was never the problem during those 18 months.
Django is like any other framework/language. Bad programmers can make a hash of it with ill-conceived ideas.
Django is like any other framework/language. Bad programmers can make a hash of it with ill-conceived ideas.
Flask and Pyramid don't help in that situation either...In fact, I switched from Flask (after years of being an avid Flask proponent) because I ran into too many projects that were a mess due to the lack of constraints / standard ways of doing things. After 5 years of solid Django use, I have yet to run into a situation where Django is too inflexible for a client's needs. No, the Admin isn't for everything (and that's even outlined in the docs); but it's not difficult to roll a custom dashboard (just like you would when using Flask or Pyramid).
The flask community really needs to rally around a project layout/boilerplate set-up; it took me a lot of trial and error to really nail scalable, intelligent flask projects and that is the #1 issue people run into with it.
Over the years I've used Flask for smaller projects with very specific requirements and simple web GUI requirements. Its my de-facto tool to prototype and/or deploy JSON APIs. It works perfectly for that because it does not impose anything on me. For more complex projects, or projects that use a web GUI (even if they are rendered with client side frameworks) Django proves to be more effective. It has a standard toolset that takes care of a lot of the edgecases one would face. For example, Flask lacks a default CSRF implementation whereas Django makes explicit use of one. Logging is also very simple with Django, something that gives some people grief. In Flask you have to setup logging from the bottom up. Both are quite different but good for what they were meant for. In fact, I tend to look for the Django or Flask of other languages to try and see how a specific type of application with look in that language. Not long ago someone posted a project that mimics flask in Rust. It was a very nice codebase and it made Rust very approachable because I could map what I know about Flask over to Rust and make a connection between language constructs and patterns.
If "product owners go off the rails" you have a management and process problem, not a Django problem.
That's not to say there are projects where another framework/language would have been better but no solution is going to save you from poor project management.
That's not to say there are projects where another framework/language would have been better but no solution is going to save you from poor project management.
This isn't a problem with Django. This is simply how software business works. No matter how well architected your solution and no matter what platform you will always have the request to do something it wasn't designed to do. If you're lucky you don't have a manager who says "can't we just put an if statement in."
Yeah, right..
There is no boilerplate or constraint how those 'flexible' solutions should be created, thus every developer writes however he wants and if those developers have not made constraints for themselves you will have a total mess, written in so many styles as there were developers. Jeez, I've even seen messed up Django apps, where you usually must follow some patterns.
Patterns, constraints, models, architecture are for a reason and if a framework/language/library makes you use some specific styles, it's much easier to maintain such projects when there is more than 1 person working on them.
There is no boilerplate or constraint how those 'flexible' solutions should be created, thus every developer writes however he wants and if those developers have not made constraints for themselves you will have a total mess, written in so many styles as there were developers. Jeez, I've even seen messed up Django apps, where you usually must follow some patterns.
Patterns, constraints, models, architecture are for a reason and if a framework/language/library makes you use some specific styles, it's much easier to maintain such projects when there is more than 1 person working on them.
This isn't a Django problem. Just don't customize the admin. Ever. If the admin can't do what you want, pull out React and Django REST Framework and build something that does what you want.
[deleted]
I work with various languages and frameworks. Django is great for general web CRUD systems. It being opinionated and somewhat inflexible is a blessing and a curse. No framework is perfect.
Flask provides the basics to build Web apps. It's low on opinions and flexible. I mostly use flask for systems that are very specific about their functinality and use case.
Flask provides the basics to build Web apps. It's low on opinions and flexible. I mostly use flask for systems that are very specific about their functinality and use case.
You can use as little of Django as you need. I like flask also, but to get it at all functional for larger apps you have to bring in dependencies.
That's one of the reasons I recently made a pared-down Django project, to highlight that you don't need to use all of the tools Django offers: https://github.com/rnevius/minimal-django
Nice example; thanks for posting a link. I didn't have anything handy when I posted my comment.
>Python is a language that gives us a lot of liberty in how we solve problems, but Django is a set of cookie cutters for doing things a certain way.
Actually a core tenet of Python is: "There should be one-- and preferably only one --obvious way to do it".
So Django couldn't be more Pythonic in this regard.
Actually a core tenet of Python is: "There should be one-- and preferably only one --obvious way to do it".
So Django couldn't be more Pythonic in this regard.
Then there is only 1 Python framework? Dumb comment. I was talking about architectures not syntax.
"Dumb comment" isn't the kind of language HN prefers.
>Then there is only 1 Python framework?
No, there are several. But all of them should strive to be Pythonic, including in the sense of offering "one, and preferably only one, obvious way to do something".
>Dumb comment.
Oh, the irony.
>I was talking about architectures not syntax.
And the Zen of Python is about architecture and APIs too, not just the syntax of the language. They are generic guidelines about how to write Pythonic programs.
No, there are several. But all of them should strive to be Pythonic, including in the sense of offering "one, and preferably only one, obvious way to do something".
>Dumb comment.
Oh, the irony.
>I was talking about architectures not syntax.
And the Zen of Python is about architecture and APIs too, not just the syntax of the language. They are generic guidelines about how to write Pythonic programs.
I've never understood this criticism of Django.
It's a batteries included framework, but at the end of the day it's just Python. You can integrate with whatever library is best for the job.
I've worked on Django apps that use SQLAlchemy instead of the Django ORM, Django apps that use Mako instead of the Django template language, and I'm currently exploring using Marshmallow instead of Django forms for some cases.
Is there anything in particular that you feel you can use with Flask or Pyramid but you couldn't use with Django?
It's a batteries included framework, but at the end of the day it's just Python. You can integrate with whatever library is best for the job.
I've worked on Django apps that use SQLAlchemy instead of the Django ORM, Django apps that use Mako instead of the Django template language, and I'm currently exploring using Marshmallow instead of Django forms for some cases.
Is there anything in particular that you feel you can use with Flask or Pyramid but you couldn't use with Django?
I view it from the other direction. If you aren't using the ORM, or the templating engine, or the admin interface, why would you choose Django? If your answer is for a plug-n-play RESTful api or authentication, you might want to check out PostgREST with PostgreSQL 9.5's row level security features, it's just plain better.
Because at that point, Django is a superset of Flask. It can do everything Flask can do, but it also has additional access to the Django ecosystem of tools and libraries.
Haha.
It's nice to dabble around Pyramid and/or Flask, but everytime I use, or someone I know uses them for not completely trivial microservices, they are always reinventing those 'cookie cutters' from scratch on their own (or choose a set of 'cookie cutters' made by someone else).
Django can work as a 'microframework', i.e. 'hello world' in one file and in ~10-15 LOC.
Not to mention Django's 'stability', i.e. stable releases at known time, with known ahead roadmap and forthcoming changes, great documentation, etc., whereas Flask has not been released for 2years and 9months (!!!), master is 900 commits ahead (2400 commits in total) of latest version, documentation is not the best (especially for current `master`)... Pyramid is made by true rockstar programmers, though it did not stand against hip image of flask and it suffers from the fact that I must reinvent all the 'cookie cutters'. IMHO this 'microframework' approach is good only for trivial microservices and/or completely non-trivial, crazy custom apps, where overriding Django would be just too much.
It's nice to dabble around Pyramid and/or Flask, but everytime I use, or someone I know uses them for not completely trivial microservices, they are always reinventing those 'cookie cutters' from scratch on their own (or choose a set of 'cookie cutters' made by someone else).
Django can work as a 'microframework', i.e. 'hello world' in one file and in ~10-15 LOC.
Not to mention Django's 'stability', i.e. stable releases at known time, with known ahead roadmap and forthcoming changes, great documentation, etc., whereas Flask has not been released for 2years and 9months (!!!), master is 900 commits ahead (2400 commits in total) of latest version, documentation is not the best (especially for current `master`)... Pyramid is made by true rockstar programmers, though it did not stand against hip image of flask and it suffers from the fact that I must reinvent all the 'cookie cutters'. IMHO this 'microframework' approach is good only for trivial microservices and/or completely non-trivial, crazy custom apps, where overriding Django would be just too much.
I really like Django, but you're right that if you for whatever reason needs to make Django do something that's it's not exactly designed for, you're going to have a bad time.
What I found is that if you can manage stay within the confines of Django, you will be rewarded and can avoid writing a ton of code. There really isn't to many situation where you can't just to things "The Django Way". If you find one, then maybe rethink your solution. I had a ton of code where I tried to fight the Django Rest Framework, when I stopped and just accepted that Django and the Rest Framework is opinionated I was able to reduce complexity and line count.
Of cause there are problems where Django is simply the wrong solution.
What I found is that if you can manage stay within the confines of Django, you will be rewarded and can avoid writing a ton of code. There really isn't to many situation where you can't just to things "The Django Way". If you find one, then maybe rethink your solution. I had a ton of code where I tried to fight the Django Rest Framework, when I stopped and just accepted that Django and the Rest Framework is opinionated I was able to reduce complexity and line count.
Of cause there are problems where Django is simply the wrong solution.
So "bad news" is that they added a feature to a framework you don't use?
For building clustered backend services, messaging is great. And Django Channels looks awesome.
However, in the context of the web (the client/server boundary), I think we need to be careful about how messaging is used. The web is made up of resources with URLs. HTTP is typically used for data transfer, and for good reason.
I sometimes see webapps communicating entirely over WebSockets. I don't mean just the realtime push part, I mean everything, request/response traffic and all. I understand that HTTP is suboptimal in certain contexts, but let's not throw the baby out with the bathwater.
However, in the context of the web (the client/server boundary), I think we need to be careful about how messaging is used. The web is made up of resources with URLs. HTTP is typically used for data transfer, and for good reason.
I sometimes see webapps communicating entirely over WebSockets. I don't mean just the realtime push part, I mean everything, request/response traffic and all. I understand that HTTP is suboptimal in certain contexts, but let's not throw the baby out with the bathwater.
yea, of course. It's a new option to build good systems not a tool to build shitty ones
Same as with Single Page Apps - websockets have their place but shouldn't be abused.
I can imagine a large Django app that uses mostly plain request/response but needs, say, a small customer service bot or occasional real time notifications. This new feature is perfect for that.
I can imagine a large Django app that uses mostly plain request/response but needs, say, a small customer service bot or occasional real time notifications. This new feature is perfect for that.
> However, in the context of the web (the client/server boundary), I think we need to be careful about how messaging is used. The web is made up of resources with URLs. HTTP is typically used for data transfer, and for good reason.
"Careful" == do not break urls, use pushState and make sure equivalent document is returned for GET.
"Careful" == do not break urls, use pushState and make sure equivalent document is returned for GET.
Sounds good to me. :)
Could you elaborate on specific issues you see with the use of web sockets over http/s for request response? I see this argument from time to time and I have a hard time with it when you take resource hogging out of the equation, which isn't much of a factor in many situations.
The issues with WebSockets are more political than technical. WebSockets are the superior technical solution with wide availability. However, at present, HTTP has better tooling (e.g. debug tools, devops tools) and developers are more familiar with it. HTTP is the de-facto standard for APIs as well.
If you're the sole developer of a Single Page App with no API, then this difference may not be important. But if you're working with a team, or across teams, then building a WebSocket-only app will bring pain. As another commenter mentioned, as long as equivalent actions are possible via HTTP then some of the issues are alleviated.
If you're the sole developer of a Single Page App with no API, then this difference may not be important. But if you're working with a team, or across teams, then building a WebSocket-only app will bring pain. As another commenter mentioned, as long as equivalent actions are possible via HTTP then some of the issues are alleviated.
upvoted for being a nuanced response, thanks!
I'm not sure we're moving that far out quite yet. For starters, most of the very big sites that do stuff (PayPal, Ebay, Airbnb, etc) are still mostly request/response, which works pretty well.
And even the newer stuff isn't so much Websockets as it is serving JSON to single-page apps.
I've wanted to like Django (because Python is my preferred language) but it still feels so crufty from its origins as a newspaper server.
And even the newer stuff isn't so much Websockets as it is serving JSON to single-page apps.
I've wanted to like Django (because Python is my preferred language) but it still feels so crufty from its origins as a newspaper server.
> but it still feels so crufty from its origins as a newspaper server.
Care to elaborate?
Care to elaborate?
"Django was born in the fall of 2003, when the web programmers at the Lawrence Journal-World newspaper, Adrian Holovaty and Simon Willison, began using Python to build applications. It was released publicly under a BSD license in July 2005. The framework was named after guitarist Django Reinhardt."
Source: https://en.wikipedia.org/wiki/Django_(web_framework)#History
Source: https://en.wikipedia.org/wiki/Django_(web_framework)#History
Right, but any examples of it feeling crufty?
Personally I think the templating system is less and less useful because I'm serving single page apps that I don't really benefit from serving from Django any more (or are Angular and the variable syntaxes conflict with each other)
Personally I think the templating system is less and less useful because I'm serving single page apps that I don't really benefit from serving from Django any more (or are Angular and the variable syntaxes conflict with each other)
Django has outgrown its humble beginnings. I am building a fairly sophisticated single-page business application using Django along with Django REST Framework, and have yet to run into a situation that made Django feel "crufty."
Django REST Framework is excellent.
And quite slow
I can't even begin to imagine what your doing with it that makes you think it's slow.
Links or references to documented slowness?
Links or references to documented slowness?
Well, obviously an SPA would not run into any of the cruftiness since you're just using Django as a "dumb pipe" to the DB.
My point is Django is not limited by the fact that it was initially conceived as a framework for serving static web pages.
Not all SPAs are based on the DB+"dumb pipe"+SPA stack, sometimes you need to process things on the server, based on your business requirements and/or performance considerations. Sometimes the processing done in that layer can be quite significant. If that is your need, both Django itself (using JsonResponse and serialization objects) and Django REST Framework allow you to do so much more than dumb piping.
So as a Scala Play dev... Trying to get the user stuff to work with email as username and still work on the admin page? Super crufty.
Overall very good docs though :)
Overall very good docs though :)
That's not been an issue for several years (since Django 1.5(?) introduced custom user models). Drop in a custom admin form and you're done.
That is not true with current (1.9).
Drop in a custom user model and
a) Better be a brand new project at all
b) You need to muck with the user manager also
c) How do you fix up the admin module? I have yet to see a fully fixed up admin module for the custom user.
I am not a Django pro so happy to be proved wrong. But take a default app, and record all steps to make the user model work (including password reset, login, logout, admin page). If that doesn't feel crufty to you I am not sure what cruft is.
Drop in a custom user model and
a) Better be a brand new project at all
b) You need to muck with the user manager also
c) How do you fix up the admin module? I have yet to see a fully fixed up admin module for the custom user.
I am not a Django pro so happy to be proved wrong. But take a default app, and record all steps to make the user model work (including password reset, login, logout, admin page). If that doesn't feel crufty to you I am not sure what cruft is.
Not my experience either. And in 2016, email-as-username should be default and work flawlessly out of the box.
Fair enough. Here's some code I wrote for a side project a couple of years ago - it was a side project, not meant for real world production, and it's a couple Django versions behind, so no guarantees as to quality, but should give you an idea:
https://github.com/danjac/ownblock/blob/master/ownblock/ownb...
https://github.com/danjac/ownblock/blob/master/ownblock/ownb...
[deleted]
One small but big example: still, or until recently, it was hard to use email addresses as usernames. Usernames are useful for sites with discussion but none other.
all successful software is crufty. django os not perfect but it's a very solid option. the fact that it's taken websockets seriously means that building real time apps became easier.
I did some Django years ago and if I remember correctly Django wasn't a "server" ie long running process but more like CGI. So how does it work exactly ? is there some python code running as a demon polling socket events and interacting with the actual Django app ?
Django is agnostic about its actual connection to the webserver. It provides a basic wsgi adapter (wsgi is the Python spec for delivering HTTP requests to an app and sending responses back to the client), which can be plugged into any wsgi container. When HTTP requests arrive at the wsgi container, it forwards them on to Django's entry point, which runs the normal routes/views/middleware/etc.
When I first got into Django in 2007 the preferred container was flup, using an scgi connection to the webserver. Nowadays, it seems like uwsgi over HTTP is preferred. CGI was an option for deployment since inception, but I don't know anyone that actually ran their Django apps over CGI, since it's many times slower than using uwsgi.
Django Channels extends wsgi into asgi (ASynchronous Gateway Interface), which provides both send() and receive_many() callables to the application server. In other words, Django can now send messages, unsolicited, back to the webserver, which presumably has a websocket connection back to the browser.
When I first got into Django in 2007 the preferred container was flup, using an scgi connection to the webserver. Nowadays, it seems like uwsgi over HTTP is preferred. CGI was an option for deployment since inception, but I don't know anyone that actually ran their Django apps over CGI, since it's many times slower than using uwsgi.
Django Channels extends wsgi into asgi (ASynchronous Gateway Interface), which provides both send() and receive_many() callables to the application server. In other words, Django can now send messages, unsolicited, back to the webserver, which presumably has a websocket connection back to the browser.
Django generally relies on the WSGI standard to separate the 'server' from the (Django) 'application' - this allows Django to run on a number of different server implementations.
https://en.wikipedia.org/wiki/Web_Server_Gateway_Interface
https://en.wikipedia.org/wiki/Web_Server_Gateway_Interface
[deleted]
The underlying model for all Python Web apps -- WSGI -- is CGI-like. But the actual deployment of WSGI almost always involves a long-running process or processes, which keep the application code in memory and then expose the appropriate WSGI API to whatever Web server you're using. Switching that from "expose WSGI API" to "expose ASGI API" gets you Channels :)
Here is a good interview with WSGI/Django devs talking about the rationale and starting work on the Django Channels project.
http://pythonpodcast.com/wsgi-2.html
http://pythonpodcast.com/wsgi-2.html
The big question is if other frameworks will adopt this ASGI/Channels model also, or will it remain a Django-specific thing.
I surmise that hendrix, based on Twisted, will be the first true ASGI / Django Channels backend.
The problem with Twisted is that everything needs to be rewritten for it, like a special snowflake. Hendrix is an example of that. The advantage of Django is you can use any python library without issues.
Really? I have exactly the opposite impression: that Twisted is happy to consume any streams of data you provide, while Django is only happy if you comply with the dictates of the LazySettings object.
In any case, I think you'll find hendrix unbelievably easy to use - here's a talk that gives some detail: https://www.youtube.com/watch?v=92VeMkjM1TQ
In any case, I think you'll find hendrix unbelievably easy to use - here's a talk that gives some detail: https://www.youtube.com/watch?v=92VeMkjM1TQ
I think you may be talking past each other.
If 100% of integration with the system consists of sending network requests to it and receiving network responses from it, then sure, it's not going to affect you much.
But when your integration involves taking existing applications/libraries and having them share memory space and call into and be called by functions in Twisted, then you're looking at a different story. Asynchronous code is infectious -- unless everything in your codebase is written to be or play nice with async, you get no benefits from the async parts and in fact can often get a net harm as the async and non-async bits clash with each other.
If 100% of integration with the system consists of sending network requests to it and receiving network responses from it, then sure, it's not going to affect you much.
But when your integration involves taking existing applications/libraries and having them share memory space and call into and be called by functions in Twisted, then you're looking at a different story. Asynchronous code is infectious -- unless everything in your codebase is written to be or play nice with async, you get no benefits from the async parts and in fact can often get a net harm as the async and non-async bits clash with each other.
This is not about async in general. This is about twisted's approach to async. For instance, there are numerous other languages that handle async elegantly and all the libraries work and there are no special snowflakes. Go comes to mind. Asyncio in python is a better answer for Python. But twisted is not asyncio. Twisted remains it's own separate thing.
> Asyncio in python is a better answer for Python. But twisted is not asyncio.
To my way of thinking about these things, this is plainly a non-sequitur. Twisted and asyncio don't do, or even try to do, the same things.
The only reason that they are comparable is because, for years, Python lacked adequate primitives for expressing concurrency. Thus, Twisted, and its event loop (the reactor), handle this.
And it's true that the reactor may eventually be replaced by async IO. However, the core of what Twisted does (handle network (and other I/O) traffic in an elegant, adaptive, concurrent way) is not something that asyncio is designed to replicate.
To my way of thinking about these things, this is plainly a non-sequitur. Twisted and asyncio don't do, or even try to do, the same things.
The only reason that they are comparable is because, for years, Python lacked adequate primitives for expressing concurrency. Thus, Twisted, and its event loop (the reactor), handle this.
And it's true that the reactor may eventually be replaced by async IO. However, the core of what Twisted does (handle network (and other I/O) traffic in an elegant, adaptive, concurrent way) is not something that asyncio is designed to replicate.
Yes, really. What if the library you are using uses blocking IO? What if it reads a file using a plain 'with open("file") as fh:'? Now you've blocked every connection in your twisted server. No warnings or errors, just strange intermittent pauses on every connection leaving you scratching your head.
This isn't weird esoteric packages either. Ever used requests? SQLAlchemy? Django's ORM? You can't use them with twisted.
Twisted proponents don't openly say this up front as it's not a great story, and their response is usually "just" port the library.
Or maybe, put it like this. If Django and Twisted work well together, why does Hendrix exist at all? Just use Django. Or why does klein exist? Just use werkzeug. You'll see lots of this rewriting snowflakes.
Just a quick search gives me txamqp, txmongo, txmandrill, txflask, txcelery, txdocker, txldap, txgithub, txrequests, txstripe, txzookeeper and on and on and on. Have a look: https://pypi.python.org/pypi?%3Aaction=search&term=twisted&s...
And then there's multiple ones of each to choose from. Look at amqp or reddis. There's something like four implementations of each. Even something like requests, theres treq, rxrequests, requests-twisted. Which one should I use?
And then there's Python 3. Only half of twisted works on Python 3. Which half? I guess one finds out when they give it a go.
This isn't weird esoteric packages either. Ever used requests? SQLAlchemy? Django's ORM? You can't use them with twisted.
Twisted proponents don't openly say this up front as it's not a great story, and their response is usually "just" port the library.
Or maybe, put it like this. If Django and Twisted work well together, why does Hendrix exist at all? Just use Django. Or why does klein exist? Just use werkzeug. You'll see lots of this rewriting snowflakes.
Just a quick search gives me txamqp, txmongo, txmandrill, txflask, txcelery, txdocker, txldap, txgithub, txrequests, txstripe, txzookeeper and on and on and on. Have a look: https://pypi.python.org/pypi?%3Aaction=search&term=twisted&s...
And then there's multiple ones of each to choose from. Look at amqp or reddis. There's something like four implementations of each. Even something like requests, theres treq, rxrequests, requests-twisted. Which one should I use?
And then there's Python 3. Only half of twisted works on Python 3. Which half? I guess one finds out when they give it a go.
This is a great summary. I considered Twisted prior to running into some of the issues you've described, and the complexity of the framework. I ended up using gevent and have been happy its monkey-patching of the socket and related libraries.
> This isn't weird esoteric packages either. Ever used requests? SQLAlchemy? Django's ORM? You can't use them with twisted.
I'm left completely scratching my head here. I use all three of these packages routinely with Twisted.
Are you saying that you are trying to share the reactor thread while blocking with, for example, requests.get()? Obviously you can't do that - why would you even try? Just run requests.get() on another thread.
On the other hand, if you, for example, use deferToThreadPool with a bunch of Django ORM calls, you can happily go on serving requests from another ThreadPool. This works like gravy.
These various adapters that you've described: in every case, their existence is not a testament to an inability to run their respective libraries' blocking I/O with Twisted; on the contrary, they provide support for actually using them with a proper Twisted protocol.
The fact that so many of them exist doesn't mean that Twisted is doing anything wrong, it means that people really like Twisted and want these various packages to be able to talk directly to a Protocol or Producer without having to maintain that code in their project.
If you want to run Django in Twisted without hendrix, go ahead: use the WSGIServer built in Twisted. However, you'll probably find that you are building lots of tooling around the awesome things you want to do with Twisted (ThreadPool tuning, websockets, concurrent functions running in views, and much more). So, instead of having to write all that yourself, hendrix ships with it.
For my practice, the Twisted ecosystem is the finest part of Python.
I'm left completely scratching my head here. I use all three of these packages routinely with Twisted.
Are you saying that you are trying to share the reactor thread while blocking with, for example, requests.get()? Obviously you can't do that - why would you even try? Just run requests.get() on another thread.
On the other hand, if you, for example, use deferToThreadPool with a bunch of Django ORM calls, you can happily go on serving requests from another ThreadPool. This works like gravy.
These various adapters that you've described: in every case, their existence is not a testament to an inability to run their respective libraries' blocking I/O with Twisted; on the contrary, they provide support for actually using them with a proper Twisted protocol.
The fact that so many of them exist doesn't mean that Twisted is doing anything wrong, it means that people really like Twisted and want these various packages to be able to talk directly to a Protocol or Producer without having to maintain that code in their project.
If you want to run Django in Twisted without hendrix, go ahead: use the WSGIServer built in Twisted. However, you'll probably find that you are building lots of tooling around the awesome things you want to do with Twisted (ThreadPool tuning, websockets, concurrent functions running in views, and much more). So, instead of having to write all that yourself, hendrix ships with it.
For my practice, the Twisted ecosystem is the finest part of Python.
The "twisted problem" is often summarised as "I know Twisted is great for asynchronous Python... If only it was the kind of Python I want to write"
It's not that Twisted is bad. It's just that many Python developers are not "fluent in the Twisted dialect" like how some people find a thick Scottish accent hard to understand and other more familiar with it can understand every word.
It's not that Twisted is bad. It's just that many Python developers are not "fluent in the Twisted dialect" like how some people find a thick Scottish accent hard to understand and other more familiar with it can understand every word.
To get more insight into the design of channels, see (my coworker!) Andrew Godwin's article "Laying the Channels Groundwork" at http://www.aeracode.org/2016/1/3/laying-channels-groundwork/
Can we come up with a new word for this? Real-Time has a definition w/r/t computing, and it's certainly not "Python code, running in a C runtime, on top of a non-realtime OS."
That ship sailed almost 10 years ago, the complaints now are getting bit tired. Words occasionally take on new meanings, live with it.
Yes, but the ship was guaranteed to sail no later than 8 years ago. Meanwhile, I'm still waiting for the blog post to load.
I'd refer to it as just "asynchronous". I'm not sure why they felt the need to redefine.
AJAX was already async (that's where the first A comes from..), so it makes sense to have new name for new generation
Does Nginx support ASGI? What's the recommended production ASGI web server?
The recommended ASGI server seems to be Daphne[0]. Nginx doesn't need to speak ASGI natively (as it doesn't speak WSGI natively), more likely Daphne would replace the WSGI server (gunicorn, etc) running behind nginx.
[0] https://github.com/andrewgodwin/daphne/
[0] https://github.com/andrewgodwin/daphne/
I am consistently impressed with the thoughtful API abstractions django develops. They were a little late to the game on this — you had to use other libraries to do WebSockets for a while. But they did this right.
maybe it is possible to be more specific and say "I am consistently impressed with the thoughtful API abstractions Andrew Godwin develops". There are other great Django developers of course, but he is just insanely good.
I think part of what he's insanely good at is carefully soliciting feedback from other people, particularly those with a lot of experience with similar but distinct problems. The final spec goes back close to a year (https://www.aeracode.org/2015/6/17/beyond-request-response/ https://groups.google.com/d/topic/django-developers/mt1mTNvc...) and the polish really shows.
I think of that as a “senior developer” skill which is easy to underestimate since it doesn't directly translate into tons of commits but is so critical to the long-term success of a project.
I think of that as a “senior developer” skill which is easy to underestimate since it doesn't directly translate into tons of commits but is so critical to the long-term success of a project.
Way to go DJANGO
Channels sound pretty similar to Observables/Streams/Signals from the reactive world. Weird that they didn't mention it.
From what I understand, Channels' provenance is from Go (scroll down to the What is a Channel? section): http://channels.readthedocs.org/en/latest/concepts.html
"Go channels are reasonably similar to Django ones"
Really? Only at a surface glance. Having used channels in both Go and in Clojure, I would say they are nothing like Django channels. The only thing they share is they are like a fifo pipe.
In Go and Clojure channels are an in-process async construct. They have nothing to do with the network at all. (There may be libraries that expose their network connections in the language as channels, but the channel itself is completely in process).
In these languages there is no 'routing' that needs to be set up. There is no 'linearisation' problem. They are first class and can be passed in and out of functions.
Also most importantly, pulling from or pushing to a channel appear to the programmer as a blocking operation, while under the hood the language inverts control to create a non-blocking situation. Send and receive on the Django channels look to be non-blocking and appear to the programmer to be non-blocking (correct me if I'm wrong, please).
Saying all this I'm not opposed to calling Django channels, channels. Channels is a generic term, and what is being done here in Django can be very much described as a channel. Just remember that they are actually very different to the "channels" as used in Go.
Really? Only at a surface glance. Having used channels in both Go and in Clojure, I would say they are nothing like Django channels. The only thing they share is they are like a fifo pipe.
In Go and Clojure channels are an in-process async construct. They have nothing to do with the network at all. (There may be libraries that expose their network connections in the language as channels, but the channel itself is completely in process).
In these languages there is no 'routing' that needs to be set up. There is no 'linearisation' problem. They are first class and can be passed in and out of functions.
Also most importantly, pulling from or pushing to a channel appear to the programmer as a blocking operation, while under the hood the language inverts control to create a non-blocking situation. Send and receive on the Django channels look to be non-blocking and appear to the programmer to be non-blocking (correct me if I'm wrong, please).
Saying all this I'm not opposed to calling Django channels, channels. Channels is a generic term, and what is being done here in Django can be very much described as a channel. Just remember that they are actually very different to the "channels" as used in Go.
[deleted]
This is great news. For those with flask apps, check out flask-socket[0] for websocket support.
[0]: https://github.com/kennethreitz/flask-sockets
[0]: https://github.com/kennethreitz/flask-sockets
I'm someone who moved to django from meteor (for work) and at first I was shocked at how similar they were, but after that I was shocked at how out-dated django seemed.
Websockets was just one of those issues so it's good to see it addressed. However, with meteor's DDP there's still no comparison. Looking ahead to Reactive GraphQL that meteor is working on, django seems to be permanently one step behind for the foreseeable future.
These are just my opinions, but overall definitely great news for django developers.
Websockets was just one of those issues so it's good to see it addressed. However, with meteor's DDP there's still no comparison. Looking ahead to Reactive GraphQL that meteor is working on, django seems to be permanently one step behind for the foreseeable future.
These are just my opinions, but overall definitely great news for django developers.
So does meteor support SQL backends yet (not talking unsupported plugins or whatever MDG is promising to do Real Soon Now).
If it's still tightly coupled to MongoDB then there is zero chance I can see it being used for anything more serious than an MVP or hackathon project by a company with non-trivial data requirements.
If it's still tightly coupled to MongoDB then there is zero chance I can see it being used for anything more serious than an MVP or hackathon project by a company with non-trivial data requirements.
Meteor's DDP only can have it's annoyances as well. It's ok till you want to do something like upload a photo which it's not great for.
Meteor is pretty unfit for prime time app development. They might one day hoist it over the fence, but for right now it remains a nice idea, I think.
For one thing, there were some regrettable architectural decisions made from the start: MongoDB just don't cut it. If my company tried to sell apps built with Meteor and/or MongoDB to corporate clients, we'd be waiting tables and working construction for a living.
But there are other issues there, perhaps even philosophical ones. I've become disheartened at what I see happening in the JavaScript arena. Change and never-ending-newness are distorting the reality field there. It's all shiny churn, frothy and irrational.
For one thing, there were some regrettable architectural decisions made from the start: MongoDB just don't cut it. If my company tried to sell apps built with Meteor and/or MongoDB to corporate clients, we'd be waiting tables and working construction for a living.
But there are other issues there, perhaps even philosophical ones. I've become disheartened at what I see happening in the JavaScript arena. Change and never-ending-newness are distorting the reality field there. It's all shiny churn, frothy and irrational.
Meteor knowns all of this hhh
Django is just getting into the new tech :P
Well Django is outdated since it has just been catching up with websocket :P
Meteor has that like 3 years ago :D
Meteor has that like 3 years ago :D
The best thing about Django is its documentation. It is one of the best (if not _the_ best) documented projects I have ever used. Example: Let's say you are reading about Django authentication related stuff and somebody links for further clarification to https://docs.djangoproject.com/en/1.7/topics/auth/default/.
Bam! You immediately know it's outdated (emphasized by the headline) and switch to https://docs.djangoproject.com/en/1.9/topics/auth/default/. This might seem minor, but I highly appreciate such attention to detail. And I really like the official page's layout, look and feel which concentrates on information, rather than stupid, flashy animations. Ha, looks like I'm getting old.
Bam! You immediately know it's outdated (emphasized by the headline) and switch to https://docs.djangoproject.com/en/1.9/topics/auth/default/. This might seem minor, but I highly appreciate such attention to detail. And I really like the official page's layout, look and feel which concentrates on information, rather than stupid, flashy animations. Ha, looks like I'm getting old.
Absolutely in agreement. I am 2 years into my app dev career, all of it spent pushing .NET around in the enterprise.
But I need to expand my skill set, I need to continue working remote, and I'm interested in OSS in general. Honestly, Python and Django seem like heaven compared to the other language/framework ecosystems I investigated. Not gonna name any names there...
But the documentation for Python and Django is A++. It's just really, really good. I can't praise it enough. The whole setup is elegant, intelligent and accessible. I really appreciate that.
But I need to expand my skill set, I need to continue working remote, and I'm interested in OSS in general. Honestly, Python and Django seem like heaven compared to the other language/framework ecosystems I investigated. Not gonna name any names there...
But the documentation for Python and Django is A++. It's just really, really good. I can't praise it enough. The whole setup is elegant, intelligent and accessible. I really appreciate that.
I'm in a bit of the same boat lately, deciding between updating my skills to the treadmill of JS frameworks, or Rails 5, or Django. Have you found any good leads on Django remote work?
I did a cursory survey, and saw that remote Django jobs existed, but I didn't do any due diligence. I figured I'd learn the stack first. ;)
My rough impression is that Rails remote jobs are relatively thick on the ground, but that any OSS dev stack will be better represented in remote job postings than any enterprise stack. So since I prefer Python to Ruby, and Django to Rails, I figured I'd take a leap of faith.
JavaScript is simply not going to be my main dish, not for the time being. Things are just too crazy there. I don't have time to play whack-a-mole with my career skill set.
My rough impression is that Rails remote jobs are relatively thick on the ground, but that any OSS dev stack will be better represented in remote job postings than any enterprise stack. So since I prefer Python to Ruby, and Django to Rails, I figured I'd take a leap of faith.
JavaScript is simply not going to be my main dish, not for the time being. Things are just too crazy there. I don't have time to play whack-a-mole with my career skill set.
Couldn't agree more.
Django's documentation has ruined me. I took the high quality for granted. Now when I use any of these tiny NodeJS/frontend modules, I'm surprised at how barebones the documentation can be.
Such an excellent project, community and set of docs. By far my favorite framework.
Django's documentation has ruined me. I took the high quality for granted. Now when I use any of these tiny NodeJS/frontend modules, I'm surprised at how barebones the documentation can be.
Such an excellent project, community and set of docs. By far my favorite framework.
Absolutely agree. And when you're really pushing the edges of the documentation (hard to do, but happens on occasion – like reasoning about authentication plugins, or model loading behavior) I've found the source code to be incredibly high quality. By that I mean: well commented, consistently styled, variables have descriptive names, etc. Definitely some of it can look a little crazy but if you clone the git repo and look at the revision history the intent of each commit is well documented and/or linked to a specific bug in their tracker. An absolutely incredible project.
One anecdote: I found something in the documentation that was outdated, I submitted a ticket and within hours two contributors were not only fixing it but discussing how to improve the underlying issue.
Whenever Django documentation quality comes up I like to recommend the related series of posts by Jacob Kaplan-Moss. It details much of the underlying reasoning that made Django docs so good, but it’s written as general documentation-writing advice and can be applied to any software project.[0]
I was surprised, though, by removal of the documentation for all versions preceding Django 1.7 from under docs.djangoproject.com recently. (It’s available now on Read the Docs.) The argument was that spending DSF funds on supporting docs for legacy versions can’t be justified, if I recall correctly from when in came up in the core dev mailing list. It seems to me maintaining documentation is becoming less of a priority now.
[0] jacobian.org/writing/great-documentation/
I was surprised, though, by removal of the documentation for all versions preceding Django 1.7 from under docs.djangoproject.com recently. (It’s available now on Read the Docs.) The argument was that spending DSF funds on supporting docs for legacy versions can’t be justified, if I recall correctly from when in came up in the core dev mailing list. It seems to me maintaining documentation is becoming less of a priority now.
[0] jacobian.org/writing/great-documentation/
It's great that Dijago Channels will allow a developer to use same framework and ORM to build the WebSocket based applications. However I think from the deployment and scaling plan, it's better to keep HTTP / REST servers separate from the Web Socket servers. Both models scale differently. So keeping them on different infrastructure would allow to scale those 2 systems independent of each other.
As far as I can tell, they are. In production you'll run your WSGI servers for the HTTP/REST side of things, your "ASGI" servers (with daphne) for the websockets side of things, and your workers, all separately. That should mean that each can be scaled pretty much independently.