Python Patterns (2013)(taoofmac.com)
taoofmac.com
Python Patterns (2013)
https://taoofmac.com/space/blog/2013/08/11/2300
48 comments
> My ground rule is that anything I easy_install will go into env, and env will quite often be (re)built on a separate machine that matches my target environment. So I treat it as a disposable folder, and it’s never committed to any repos.
Was wondering, do you pip freeze > requirements.txt? I don't see the file in there, but I assume that's what you mean since you mention that none of the env folders make it into version control.
On a semi-related note, I used to have tons of projects like this a year ago, and having moved most of our python code to AWS lambda, our repos have shrank to maybe 25% of their original size. Granted it's not fabulous if you need to run a JS app on top of your Bottle framework, but we used to have a lot of small services here and there, the provisioning and deployment was so much work.
Was wondering, do you pip freeze > requirements.txt? I don't see the file in there, but I assume that's what you mean since you mention that none of the env folders make it into version control.
On a semi-related note, I used to have tons of projects like this a year ago, and having moved most of our python code to AWS lambda, our repos have shrank to maybe 25% of their original size. Granted it's not fabulous if you need to run a JS app on top of your Bottle framework, but we used to have a lot of small services here and there, the provisioning and deployment was so much work.
Can I just state how awful this two-column layout is to read?
I found the "toggle columns" option (top right, under the search box) and it made me happy.
While it did away with the horrible 2-column layout, it looks like the article was writen with those columns in mind, so it uses short paragraphs that look good on 2-column, but feel a bit weird on a wall of text.
My (barely existant) OCD is going haywire over this article as I read it...
My (barely existant) OCD is going haywire over this article as I read it...
OT: Anyone know why all browser vendors implemented this CSS columns aberration so hurridly..? I have never seen a good use for them, so painful to read. It's always some designer trying to show off their "epic" CSS skillz.
Seems like it might be useful for a print style sheet. Haven't seen it used to good effect on a screen though.
I'm glad I'm not the only one. Decided to skim it instead of reading it all the way through because it was annoying.
Also, the picture with the directory tree got cropped out for me, and I couldn't figure out how to scroll it horizontally. Good thing the columns can be toggled.
Way better than the single column one on my 1024x768 lcd.
It goes away if you decrease the width of the viewport
[deleted]
[deleted]
Interesting as it's quite different from how I generally structure my own projects. A couple questions:
1. Why store dependencies locally? Seems a bit wasteful and unnecessary with tools like pip/virtualenv. (OP does mention virtualenv, but also "Include ALL the dependencies locally" so I'm not sure what to make of that)
2. You've chosen to separate your code into MVC-inspired directories instead of by function. Flask/Django have ways (apps, blueprints) of making application subcomponents modular. It seems like a single controllers directory and a single templates directory, etc. could get cluttered pretty quickly. What will you do as projects grow?
1. Why store dependencies locally? Seems a bit wasteful and unnecessary with tools like pip/virtualenv. (OP does mention virtualenv, but also "Include ALL the dependencies locally" so I'm not sure what to make of that)
2. You've chosen to separate your code into MVC-inspired directories instead of by function. Flask/Django have ways (apps, blueprints) of making application subcomponents modular. It seems like a single controllers directory and a single templates directory, etc. could get cluttered pretty quickly. What will you do as projects grow?
Not the OP, but w.r.t. [1] vendored dependencies are especially helpful if you happen to deploy the software on premise where you might not have full system access and in some cases, no internet or also strict policies on what can be installed.
What you can do is get wheel packages for all your dependencies and install them from a local "wheelhouse":
You can learn a bit more about wheel in this very good article: http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
pip install -r requirements.txt --no-index --find-links=/path/to/wheelhouse
You can learn a bit more about wheel in this very good article: http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
You can vendor your dependencies inside a proper OS package as well. Write an rpmspec that creates a virtualenv, runs your setup.py with the virtualenv python interpreter and boom - your dependencies are bundled without needing to go through acrobatics with manually vendoring them.
If your application isn't able to be installed at the system level I would really recommend this for handling deployment, regardless of whether you have full access or not - deploying tarballs with configuration management tools like puppet are hackish at best, RPM's/DEB's really are the best way to roll your software out.
EDIT: As an alternative to virtualenv which is actually a PITA since --relocatable is always broken, buildout works great.
If your application isn't able to be installed at the system level I would really recommend this for handling deployment, regardless of whether you have full access or not - deploying tarballs with configuration management tools like puppet are hackish at best, RPM's/DEB's really are the best way to roll your software out.
EDIT: As an alternative to virtualenv which is actually a PITA since --relocatable is always broken, buildout works great.
I can't honestly imagine any circumstances when this would be better than using tools that can manage my Python packages as part of deploying or just using solid Python environment management practices. Why would I want to add the encumbrance of needing to work with system package toolchains to create packages when I want to avoid platform specific mechanisms to minimise my exposure to unexpected issues and retain visibility into failure causes since system package tools can be extremely opaque as to the cause of install failure causes.
For example I use PyEnv instead of virtalenv/venv because PyEnv is written in bash and has a much better level of isolation than virtalenv or venv. It's simple bash scripting and the only system dependencies it has are based on features you choose to use. If you want to build Python from sources you'll need compilers and Libs, etc, but other than that sort of thing, it's zero dependencies.
Edit: PyEnv also has the ability for me to compress an entire Python environment and reuse it somewhere else provided I'm using the same OS and system libraries, so I can pre-build compilation steps and cut down on system package dependencies in production environments.
For example I use PyEnv instead of virtalenv/venv because PyEnv is written in bash and has a much better level of isolation than virtalenv or venv. It's simple bash scripting and the only system dependencies it has are based on features you choose to use. If you want to build Python from sources you'll need compilers and Libs, etc, but other than that sort of thing, it's zero dependencies.
Edit: PyEnv also has the ability for me to compress an entire Python environment and reuse it somewhere else provided I'm using the same OS and system libraries, so I can pre-build compilation steps and cut down on system package dependencies in production environments.
I don't understand how system package managers are considered an "encumberance" by so many developers. It takes all of 10 minutes to write an rpmspec file unless you have a super-hairy build/install process (in which case you should look at fixing that) - I have one in the git repository for all of my python projects along with a Makefile with a sources target that just calls `setup.py sdist` to create a source tarball for rpmbuild.
Pushing out a new release is as simple as running `koji build f23 git+ssh://my.git.server/my/project.git` and within 15 minutes it's been published to my internal yum repository and puppet is installing the newest version on all my servers. How is managing pyenv and dealing with fabric or whatever other tool of choosing any easier than this?
Pushing out a new release is as simple as running `koji build f23 git+ssh://my.git.server/my/project.git` and within 15 minutes it's been published to my internal yum repository and puppet is installing the newest version on all my servers. How is managing pyenv and dealing with fabric or whatever other tool of choosing any easier than this?
> How is managing pyenv and dealing with fabric or whatever other tool of choosing any easier than this?
Because its all in one language, python. And when stuff errors out in python, you have a traceback and know (mostly) exactly what went wrong.
Because its all in one language, python. And when stuff errors out in python, you have a traceback and know (mostly) exactly what went wrong.
OP here. Yes, some of the stuff we deployed was packaged as .deb files - after vendoring all the dependencies.
We usually stick to the official Debian python-* packages (if they exist upstream). When they don't we use python-virtualenv[1][2] to pull specific versions and dependencies from pip into a virtual env.
Now when we build our Debian package (debian/rules):
Now when we build our Debian package (debian/rules):
override_dh_virtualenv:
dh_virtualenv -p myproj --use-system-packages
Great example of this - my current client, who requires on-premise development, blocks Fastly. Guess what runs on Fastly? NPM.
So no development if you need to use npm. I can download e.g. Grunt from GitHub directly, but installing the local package immediately tries to download dependencies from npm.
So no development if you need to use npm. I can download e.g. Grunt from GitHub directly, but installing the local package immediately tries to download dependencies from npm.
> 1. Why store dependencies locally? Seems a bit wasteful and unnecessary with tools like pip/virtualenv. (OP does mention virtualenv, but also "Include ALL the dependencies locally" so I'm not sure what to make of that)
This is a good question, and I think it depends on your toolchain and also the nature of the code you are working with. Not every project you work on will be OpenSource and might be inaccessible from the public containers/vms where you deploy. So its much easier to bundle the whole application into one package and push that into the destination. e.g if you want to deploy your app to heroku/cloudfoundry/bluemix.
This is a good question, and I think it depends on your toolchain and also the nature of the code you are working with. Not every project you work on will be OpenSource and might be inaccessible from the public containers/vms where you deploy. So its much easier to bundle the whole application into one package and push that into the destination. e.g if you want to deploy your app to heroku/cloudfoundry/bluemix.
OP here.
I use dokku/CloudFoundry these days and just toss in dependencies to requirements.txt, but at the time I often had to deploy stuff on boxes with _zero_ outside access.
I use dokku/CloudFoundry these days and just toss in dependencies to requirements.txt, but at the time I often had to deploy stuff on boxes with _zero_ outside access.
1. Its useful if you don't want people to have to run stuff to install it, just check it in side by side and they can run it directly from checkout.
> Well, because JSON is about the only format I can read on all the languages I use without ambiguity or extra dependencies
Alas, there is a downside: The JSON spec does not allow comments, which are often important for configuration files. (Though given how much that simplifies parsing, it may have been the right decision.)
So you either omit comemnts, or end up trying to shove placeholder key-values into the closest place you can to whatever you want to comment about.
Alas, there is a downside: The JSON spec does not allow comments, which are often important for configuration files. (Though given how much that simplifies parsing, it may have been the right decision.)
So you either omit comemnts, or end up trying to shove placeholder key-values into the closest place you can to whatever you want to comment about.
For configuration files I just put in comments anyway and strip them before feeding the file to a JSON parser. (Like here: https://github.com/creshal/yspave/blob/master/yspave/pave.py... )
Hackish, but beats having to bother with YAML parsers and their regular RCE vulnerabilities.
Hackish, but beats having to bother with YAML parsers and their regular RCE vulnerabilities.
Bottle, peewee, gevent workers, eh? Interesting choices.
SimpleTemplate looks interesting, I hadn't seen it before.
Would be nice to hear some discussion there, instead of the column layout.
Would be nice to hear some discussion there, instead of the column layout.
I'm the author of Peewee and I've been running my blog with Flask, Peewee, gevent and sqlite. It's a very low-impact combination, lightweight and fun to use, although I'm a bit biased.
Thank you for your work on Peewee. The code is nicely written and is easily read start to finish.
If Raymond Hettinger says that, it's probably true. :-) I agree -- I've been using peewee for a side project, and it's great. I also love the fact that the source is one file, as that makes it so easy to find stuff in and navigate. I've been working offline, so didn't have access to the docs, and just referred to the source to figure out how to do things.
It's my pleasure! And thank you for the wonderful libraries, talks, and who could forget those snippets.
Ok, now I'm going to go study Peewee's code...
Ackkkk I'm actually rewriting it currently to try and ameliorate the cruft that's crept in since the 2.0 rewrite. I want to say "But wait! If you read the code in a couple weeks it'll be so much better!"
Trust me, I'm sure I can improve my Python skills by reading the current version. I don't get a chance to use Python enough to shed the baggage of other languages to write Pythonic code, if that makes sense.
I haven't used Peewee in a project yet, but I've poked around with it. It's really nice, so thank you.
I haven't used Peewee in a project yet, but I've poked around with it. It's really nice, so thank you.
I'm using peewee for the first time on a project starting yesterday.
Not using the ORM (I like writing SQL for some reason) but the connection pooling on Postgres is nice and I'm really enjoying working with it. Thanks for your work.
Not using the ORM (I like writing SQL for some reason) but the connection pooling on Postgres is nice and I'm really enjoying working with it. Thanks for your work.
I've learned a lot about Python from reading Peewee's source, thanks.
here's a link to the utils mentioned in the article: https://github.com/rcarmo/python-utils
I have to say the using Pika and RabbitMQ directly has been a lot easier than trying to use Celery. Just my experience.
Just out of interest, what were the main challenges you faced when using Celery?
Celery is a task queue. Pika could definitely be used to build a task queue (similarly to how Celery is built on top of Kombu), but I can't think of many good reasons why that would pay off. Unless your use case was not a task queue...
Celery is a task queue. Pika could definitely be used to build a task queue (similarly to how Celery is built on top of Kombu), but I can't think of many good reasons why that would pay off. Unless your use case was not a task queue...
Flask application context problems, hiding exceptions within tasks, Json Serialisation sometimes does weird things and hard to configure (the default, pickle, seems to be warned against as a security vulnerability), weirdness with Gevent/Socket.io flask stuff, difficulty (impossibility??) to emit socket events from within a worker, finally the canvas stuff seems great on the surface but I found it mind blowingly difficult to debug weirdness with it.
My Pika/RabbitMQ stuff was much easier to build, unit test and debug (i.e. it actually works). There is every chance the I am just a bit retarded when it comes to using celery, but the fact I've built the Pika version maybe these were just real issues? Dunno, it might all work flawlessly for you in which case it's a great tool.
My Pika/RabbitMQ stuff was much easier to build, unit test and debug (i.e. it actually works). There is every chance the I am just a bit retarded when it comes to using celery, but the fact I've built the Pika version maybe these were just real issues? Dunno, it might all work flawlessly for you in which case it's a great tool.
Brief clarification: some of those patterns were a consequence of us having to run Python 2.5 and 2.6 on older boxes - it's a survival technique, if you will - and having to vendor some dependencies to make it easier to deploy stuff under some other interesting constraints the article won't get into.
On the whole, these patterns make it easier for other people to not break your code when they want to add, say, another set of REST endpoints (which was my main occupation at the time).
You might also want to read my short piece on SSE: http://taoofmac.com/space/blog/2014/11/16/1940
And yeah, I've been meaning to change the blog layout. Hardly any time for it these days, honestly.