Guido van Rossum: patterns for fast Python(plus.google.com)
plus.google.com
Guido van Rossum: patterns for fast Python
https://plus.google.com/115212051037621986145/posts/HajXHPGN752
5 comments
Discussion from 2 weeks ago: http://news.ycombinator.com/item?id=4648275
I'm coming from the scientific computing side of python, so beyond the BDFL's suggestions, I use the following protocol to make python fast. Note, that I'm doing numerical analysis and manipulations, so this isn't a general workflow for all python applications.
1. Write algorithms using numpy, attempting to not use too many crazy indexing/array tricks. You can generally avoid costly for loops using broadcasting.
2. Profile code using cProfile and Robert Kern's line_profiler
3. Judiciously optimize by re-writing computationally intensive methods in Cython. Numba also looks like a contender in this arena as well, but I haven't done a benchmarked comparison.
1. Write algorithms using numpy, attempting to not use too many crazy indexing/array tricks. You can generally avoid costly for loops using broadcasting.
2. Profile code using cProfile and Robert Kern's line_profiler
3. Judiciously optimize by re-writing computationally intensive methods in Cython. Numba also looks like a contender in this arena as well, but I haven't done a benchmarked comparison.
Can you suggest any resources for learning about profiling in python? I'm just getting into scientific computing with numpy,scipy,etc. and I'd be interested to learn more.
I used timeit (http://docs.python.org/2/library/timeit.html), cProfile (http://docs.python.org/2/library/profile.html) and pycallgraph. There is a nice Q&A on this subject: http://stackoverflow.com/q/582336/343834.
Ian Ozsvald has a nice tutorial on high performance python, from profiling through cython and some multiprocessing and parallel techniques. You can find video and other materials here: http://ianozsvald.com/2012/03/18/high-performance-python-1-f...