How (Not?) to Use Python’s List Comprehensions(moderndescartes.com)
moderndescartes.com
How (Not?) to Use Python’s List Comprehensions
http://moderndescartes.com/essays/python_list_comprehensions
76 comments
I think this looks a lot nicer with Haskell-style do notation:
I was about to observation that the python code as posted in the blog missed an important optimization, reordering some of the tests to:
for b in range(1, 27)
for n in range(1, 27)
if b**3 + z**3 + n**3 + o**9 == 1997
for k in (3,4)
for e in range(1, 27)
for k in (3,4)
for b in range(1, 27)
for e in range(1, 27)
for n in range(1, 27)
if b**3 + z**3 + n**3 + o**9 == 1997
It amazes me that Haskell takes my Python hack and actually allows it to be expressed quite naturally! I'm curious what the semantics are for the interleaving of do and guard - would you say that Haskell will compile that code to the equivalent nested for loops, with the same short-circuiting as in the Python version? Or is there something fancier going on?
> It amazes me that Haskell takes my Python hack and actually allows it to be expressed quite naturally!
This isn't an accident. Python's list comprehensions were modeled on Haskell's list comprehensions, which were in turn modeled on the much more general concepts of monads. Haskell's do notation provides a nice syntactic sugar for monadic code. Since lists form a monad, such code is quite naturally expressed using do notation.
Haskell's list comprehensions can be directly translated to do notation
This isn't an accident. Python's list comprehensions were modeled on Haskell's list comprehensions, which were in turn modeled on the much more general concepts of monads. Haskell's do notation provides a nice syntactic sugar for monadic code. Since lists form a monad, such code is quite naturally expressed using do notation.
Haskell's list comprehensions can be directly translated to do notation
[x+y | x <- [1..10], y <- [1..10], x+y<5]
do x <- [1..10]
y <- [1..10]
guard (x+y<5)
return (x+y)
[1..10] >>= (\x ->
[1..10] >>= (\y ->
guard (x+y<5) >>
return (x+y)))
> Python's list comprehensions were modeled on Haskell's list comprehensions, which were in turn modeled on the much more general concepts of monads.
It's true that Python borrowed the concept from Haskell.
But Haskell in turn took the concept from set theory, not category theory.
List comprehensions are modeled after set comprehensions. Here's an example that uses set-builder notation [0] to define the set of Pythagorean triples with sides bounded by 10:
For a motivation behind the creation of Haskell was to use it to as a lightweight calculator / computer algebra system (sans heavy algebra) to teach set theory to students.
List comprehensions have been around since the dawn of Haskell. It was around even before Haskell, viz. in Miranda circa mid-80s.
Moggi's work on PL monads was only published in 1991.
I think you've mixed up monad comprehensions--which came much later--which is the observation that this notation and concept can be generalized to monads other than lists. Monad comprehensions segue into topos theory, understood as a monadic generalization of set theory, quite nicely.
Hopefully, the next 700 monad tutorials will give more visibility to topos theory.
[0] https://en.wikipedia.org/wiki/Set-builder_notation
It's true that Python borrowed the concept from Haskell.
But Haskell in turn took the concept from set theory, not category theory.
List comprehensions are modeled after set comprehensions. Here's an example that uses set-builder notation [0] to define the set of Pythagorean triples with sides bounded by 10:
{ (x,y,z) | x,y,z \in {1,2,...,10}, x^2 + y^2 = z^2 }
let u=[1..10] in [ (x,y,z) | x<-u,y<-u,z<-u, x^2 + y^2 = z^2 ]
For a motivation behind the creation of Haskell was to use it to as a lightweight calculator / computer algebra system (sans heavy algebra) to teach set theory to students.
List comprehensions have been around since the dawn of Haskell. It was around even before Haskell, viz. in Miranda circa mid-80s.
Moggi's work on PL monads was only published in 1991.
I think you've mixed up monad comprehensions--which came much later--which is the observation that this notation and concept can be generalized to monads other than lists. Monad comprehensions segue into topos theory, understood as a monadic generalization of set theory, quite nicely.
Hopefully, the next 700 monad tutorials will give more visibility to topos theory.
[0] https://en.wikipedia.org/wiki/Set-builder_notation
There isn't any magic going on, but remember that Haskell's lists are lazy: to write a truly equivalent version in Python would be a little tricky because you'd have to deal with generators. But the conversion from do+guard to list comprehensions is really simple (see the definition in source at https://hackage.haskell.org/package/base-4.9.0.0/docs/src/GH... and use the way the do-notation desugars into applications of (>>=) and (>>)), the definition of guard itself is similarly trivial: https://hackage.haskell.org/package/base-4.9.0.0/docs/src/Co...).
It's so much easier to see all this by just working out a small example for yourself with pen and paper.
So these two are equivalent, and the list comprehension version is very close to what nested loops (read it from left to right) would give you, except for the laziness.
It's so much easier to see all this by just working out a small example for yourself with pen and paper.
So these two are equivalent, and the list comprehension version is very close to what nested loops (read it from left to right) would give you, except for the laziness.
do x <- [1, 2, 3]; guard (f x); y <- [1, 2, 3]; guard (g x y); z <- [1, 2, 3]; guard (h x y z); return (x, y, z)
[(x, y, z) | x <- [1, 2, 3], f x, y <- [1, 2, 3], g x y, z <- [1, 2, 3], h x y z]
for x in [1, 2, 3] { if (f(x)) { for y in [1, 2, 3] { if (g(x,y)) { for z in [1, 2, 3] { if (h(x,y,z)) { yield((x, y, z)) }}}}}} -- "pseudocode"
guard True = [()]
guard False = []
Just for fun, this part
(et, i) <- [(xi - 7, xi - 11), (xi - 11, xi - 7),
(xi + 7, xi + 11), (xi + 11, xi + 7)]
(et,i) <- do (a,b) <- let tuple = (7,11) in [tuple, swap tuple]
sign <- [id,negate]
[(xi+sign a,xi+sign b)]Just for fun,
guard $ (a % k)^2 + (b % n)^2 + (d % xi)^2 +
(e % pi)^2 + (et % mu)^2 + (i % s)^2 == 6
guard $ (6 ==) $
sum $ map (^2) $
zipWith (%)
[a ,b ,d ,e ,et,i ]
[k ,n ,xi,pi,mu,s ]Ah! But should it be written like that?
I don't know. To me,
sum $ map (^2)
guard $ (6 ==) $
sum $ map (^2)
[(a%k),...]I think the original is still a lot cleaner, though.
Have an upvote for the fun-factor of your comment, but know that I find the aesthetics of the two to be about the same.
Thanks for that code. I don't know anything about Haskell so I attempt a UX test:
results is an array of Int, but do the double [[ ]] brackets mean something special? I hope they do, otherwise why planting doubts in people's mind.
<- is assignment or pattern matching, probably the latter.
the $ after guard is... what? Maybe a noop like Python's :
Maybe
u is probably the variable that's iterated over in where u = [1..26]
Not very readable to put that after every use of u. This is easier to read
Don't take these as criticisms of Haskell, only as a UX test. Python's list comprehensions have never been very readable to me. This Haskell loop is somewhat better.
I'm sure there are good reason for designing Haskell's syntax like that. Programming languages belong to lineages and tend to perpetrate the syntax of their parents. It makes them easier to adopt to developers well versed in their parent languages but that could also make them difficult to grasp for developers of other language families. Maybe luring imperative developers into the functional world is not one of the goals of Haskell and Haskell has achieved its goals as it is.
Still I believe that the first functional language that looks like an imperative language will move many developers from OOP to functional. Elixir attempted that, consciously or not, but it's not enough. The barrier there are the OTP patterns, which are not easy to use properly. There are many different choices for writing a process (what in OOP would be an object with its own CPU) and its hard to chose the proper one. GenServer, Agent, Task? Nobody is asking these kind of questions in Ruby or Python https://groups.google.com/forum/#!topic/elixir-lang-talk/DCT...
There is also the code to add them to the supervision tree (the cool systemd-inside-the-language thing of Erlang and Elixir). I can't help thinking that Elixir has to be streamlined.
results is an array of Int, but do the double [[ ]] brackets mean something special? I hope they do, otherwise why planting doubts in people's mind.
<- is assignment or pattern matching, probably the latter.
the $ after guard is... what? Maybe a noop like Python's :
Maybe
guard $ expression
something
if expression
something
u is probably the variable that's iterated over in where u = [1..26]
Not very readable to put that after every use of u. This is easier to read
where u = [1..26] do
for u in [1..26] do
Don't take these as criticisms of Haskell, only as a UX test. Python's list comprehensions have never been very readable to me. This Haskell loop is somewhat better.
I'm sure there are good reason for designing Haskell's syntax like that. Programming languages belong to lineages and tend to perpetrate the syntax of their parents. It makes them easier to adopt to developers well versed in their parent languages but that could also make them difficult to grasp for developers of other language families. Maybe luring imperative developers into the functional world is not one of the goals of Haskell and Haskell has achieved its goals as it is.
Still I believe that the first functional language that looks like an imperative language will move many developers from OOP to functional. Elixir attempted that, consciously or not, but it's not enough. The barrier there are the OTP patterns, which are not easy to use properly. There are many different choices for writing a process (what in OOP would be an object with its own CPU) and its hard to chose the proper one. GenServer, Agent, Task? Nobody is asking these kind of questions in Ruby or Python https://groups.google.com/forum/#!topic/elixir-lang-talk/DCT...
There is also the code to add them to the supervision tree (the cool systemd-inside-the-language thing of Erlang and Elixir). I can't help thinking that Elixir has to be streamlined.
> <- is assignment or pattern matching, probably the latter.
Neither really, though I guess it's kind of like assignment, but it's usually read "draw from". Do-notation is sugar for working with monadic values, which are kind of like containers that hold a value. It's a bit hard to explain, but basically it means to pull the enclosed value out of the container on the right hand side and reference it according to the name on the left hand side.
> the $ after guard is... what?
$ is function application. It basically means to evaluate the right hand side before passing the result to the left hand side. It's used to avoid parentheses, `f $ g h` is like `f (g h)` (Haskell doesn't use parenthesis for function evaluation).
Haskell syntax is lovely when you're used to it, but it's also subtle. IMO it can be easy for people who aren't familiar to get the general idea of what's going on but the specifics are often not what you think.
Neither really, though I guess it's kind of like assignment, but it's usually read "draw from". Do-notation is sugar for working with monadic values, which are kind of like containers that hold a value. It's a bit hard to explain, but basically it means to pull the enclosed value out of the container on the right hand side and reference it according to the name on the left hand side.
> the $ after guard is... what?
$ is function application. It basically means to evaluate the right hand side before passing the result to the left hand side. It's used to avoid parentheses, `f $ g h` is like `f (g h)` (Haskell doesn't use parenthesis for function evaluation).
Haskell syntax is lovely when you're used to it, but it's also subtle. IMO it can be easy for people who aren't familiar to get the general idea of what's going on but the specifics are often not what you think.
In the same boat of not knowing Haskell, this is how I interpreted it - the [[]] denotes a nested array of ints, where each array represents a set of variable values on 0..26 that satisfy all guards.
Then, once all guard statements have been put into effect, the full set of combinations of values that pass each guard are placed in the array.
Critically, it is not "for u in [0..26]" but "u=[0..26]", which gives me the idea that this isn't a loop but a filter.
Guard $ foo Then, once all guard statements have been put into effect, the full set of combinations of values that pass each guard are placed in the array.
Critically, it is not "for u in [0..26]" but "u=[0..26]", which gives me the idea that this isn't a loop but a filter.
>> Still I believe that the first functional language that looks like an imperative language will move many developers from OOP to functional.
That's supposed to be the deal with javascript (but please don't ask me for a source; I was first introduced to the idea that js is functional in a book in an old workplace's library, where I was waiting for my boss to chew me up. Needless to say, I haven't used javascript much).
That's supposed to be the deal with javascript (but please don't ask me for a source; I was first introduced to the idea that js is functional in a book in an old workplace's library, where I was waiting for my boss to chew me up. Needless to say, I haven't used javascript much).
>Not very readable to put that after every use of u. This is easier to read
> where u = [1..26] do
This isn't valid Haskell, but this is
Instead of writing
> where u = [1..26] do
This isn't valid Haskell, but this is
let u = [1..26] in
do ...
Instead of writing
f (1+2*3)
f $ 1+2*3
Is [[Int]] a nested array of ints?
This is because there could be more than one set of solutions to the simultaneous eqn
This is because there could be more than one set of solutions to the simultaneous eqn
[deleted]
There are very few reasons to nest comprehensions. When the need to go lazy arise and you end up with a long code, use a generator. "yield" is much more readable.
Author here. If I may, could I ask you to write the equivalent code as you have suggested? I can see one way to implement with generators but it's not particularly concise. The benefit of the list comprehension approach is that its mechanics are laid bare and there's almost nothing to debug.
The normal benefit of generators is that they don't store the intermediate result in memory, but that isn't particularly necessary in this case
The normal benefit of generators is that they don't store the intermediate result in memory, but that isn't particularly necessary in this case
I actually tried, and my generator version ended up less readable than yours. I was not expecting this, but for such an algo the way you write it gives this impression the data flow from one end to the other, which reads better than splitting it in several functions including nested blocks.
The only drawbacks I can see:
- if you want to debug that with PDB, you are out of luck.
- anybody not wanting to spend the time understanding what it does will freak out.
Unrelated, but you can do:
def get_ranges(i): return product(*(range(1, 27) for _ in range(i)))
To replaced stuff like:
I still have no idea what's this code is for though :)
The only drawbacks I can see:
- if you want to debug that with PDB, you are out of luck.
- anybody not wanting to spend the time understanding what it does will freak out.
Unrelated, but you can do:
def get_ranges(i): return product(*(range(1, 27) for _ in range(i)))
To replaced stuff like:
for th in range(1, 27)
for o in range(1, 27)
for r in range(1, 27)
for th, o, r in get_ranges(3)
I still have no idea what's this code is for though :)
Here's a generator version. It is not as terse, but seems to run faster. [Edit:] Or not faster in entirety, but finds solution more quickly during the iteration.
from fractions import Fraction as F
def thor():
for th in range(1, 27):
for o in range(1, 27):
for r in range(1, 27):
if 2*(th**2) + o**2 + r**2 == 1000:
yield th, o, r
def sz(thors):
for th, o, r in thors:
for s in range(1, 27):
for z in range(1, 27):
if (th**2 * z**2 * (s - z) + s**2) == 7225:
yield th, o, r, s, z
def gdxi(szs):
for th, o, r, s, z in szs:
for g in range(1, 27):
for d in range(1, 27):
for xi in range(1, 27):
if (g-1)**2 + d**2 + xi**2 - xi == 600:
yield th, o, r, s, z, g, d, xi
def eti(gdxis):
for th, o, r, s, z, g, d, xi in gdxis:
for (et, i) in (
(xi-7, xi-11), (xi-11,xi-7), (xi+7, xi+11), (xi+11, xi+7)):
if 0 < et <= 26 and 0 < i <= 26:
yield th, o, r, s, z, g, d, xi, et, i
def amupi(etis):
for th, o, r, s, z, g, d, xi, et, i in etis:
for (a, mu) in ((4, 1), (2, 2), (1, 4)):
for pi in range(1, 27):
if a*(a+pi) == 4 * g:
yield th, o, r, s, z, g, d, xi, et, i, a, mu, pi
def kben(amupis):
for th, o, r, s, z, g, d, xi, et, i, a, mu, pi in amupis:
for k in (3,4):
for b in range(1, 27):
for e in range(1, 27):
for n in range(1, 27):
if b**3 + z**3 + n**3 + o**9 == 1997:
if (F(a, k)**2 + F(b, n)**2 + F(d, xi)**2 +
F(e, pi)**2 + F(et, mu)**2 +
F(i, s)**2 == 6):
yield (a, b, g, d, e, z, et, th, i,
k, th, mu, n, xi, o, pi, r, s)
thors = thor()
szs = sz(thors)
gdxis = gdxi(szs)
etis = eti(gdxis)
amupis = amupi(etis)
results = kben(amupis)
print('\t'.join(('a', 'b', 'g', 'd', 'e', 'z', 'et', 'th',
'i', 'k', 'l', 'mu', 'n', 'xi', 'o', 'pi', 'r', 's')))
for r in results:
print('\t'.join(map(str, r)))Python also supports dict and set comprehensions btw:
{ a : 2*a for a in range(1, 10)} #produces {1:2, 2:4, ...}
{ 2*a for a in range(1, 10)} #produces set([2, 4, ...])Wish more people knew about dict comprehensions, it's maddening the contortions people go to because they don't know about it.
Lots of people saying "nobody does this is real life". I have.
My PhD code (which was too slow and RAM-hungry, having been written in Python...) was a monster, and was a lot less of a monster for order-2 and order-3 nested list/dict comps.
It started out as a 200 line proof of concept (which worked perfectly on an ideal of the system I was studying), and it evolved into a 6,000 line behemoth (~2,000 lines for OpenGL rendering error cases) which would have been 10,000 lines without nested list and dict comps.
By the way, never use networkx on anything you expect to have a large number of vertices or edges. It uses a lot of RAM (esp in undirected graphs where you have backrefs for each edge.) Wrap a C++ graph lib like boost::graphs or something, and code against that interface from the beginning.
My PhD code (which was too slow and RAM-hungry, having been written in Python...) was a monster, and was a lot less of a monster for order-2 and order-3 nested list/dict comps.
It started out as a 200 line proof of concept (which worked perfectly on an ideal of the system I was studying), and it evolved into a 6,000 line behemoth (~2,000 lines for OpenGL rendering error cases) which would have been 10,000 lines without nested list and dict comps.
By the way, never use networkx on anything you expect to have a large number of vertices or edges. It uses a lot of RAM (esp in undirected graphs where you have backrefs for each edge.) Wrap a C++ graph lib like boost::graphs or something, and code against that interface from the beginning.
I have not used networkx on anything serious, but this article lead me to believe that it is to be avoided, since the library missed obvious Python optimizations:
https://mathieularose.com/choosing-the-right-data-structures...
https://mathieularose.com/choosing-the-right-data-structures...
Well, at least the title was correct.
This is definitely not a good way to use list comprehensions.
I appreciate the OP's effort to break the language. Well done.
But no one does this in practical reality.
This is definitely not a good way to use list comprehensions.
I appreciate the OP's effort to break the language. Well done.
But no one does this in practical reality.
From the article:
Except for the simplest comprehensions, I tend to write mine similar to above, with indents to highlight what part of the comprehension is what.
The utility of a comprehension is not so much concise notation. Instead it's that a comprehension is an expression (that has an associated value), rather than the for loop which is a statement.
Similar to the trinary operator: its advantage is not merely that it replaces an if statement visually, it's that it's a directly usable expression (that can be composed in a (preferably short) chain of trinaries).
This highlights the four things that fizzbuzz requires:
result = [x
for sublist in nested_list
for x in sublist
if x % 3 == 0
]
Except for the simplest comprehensions, I tend to write mine similar to above, with indents to highlight what part of the comprehension is what.
The utility of a comprehension is not so much concise notation. Instead it's that a comprehension is an expression (that has an associated value), rather than the for loop which is a statement.
Similar to the trinary operator: its advantage is not merely that it replaces an if statement visually, it's that it's a directly usable expression (that can be composed in a (preferably short) chain of trinaries).
This highlights the four things that fizzbuzz requires:
print '\n'.join([
'fizzbuzz' if x%15 == 0 else
'fizz' if x%3 == 0 else
'buzz' if x%5 == 0 else
str(x) for x in xrange(1, 101)])
print '\n'.join([
'fizzbuzz' if x%15 == 0 else
'fizz' if x%3 == 0 else
'buzz' if x%5 == 0 else
str(x) for x in xrange(1, 101)
if x%2 == 0 or x%2 != 0])
subrange = {x for x in range(1, 101)}
print '\n'.join([
'fizzbuzz' if x%15 == 0 else
'fizz' if x%3 == 0 else
'buzz' if x%5 == 0 else
str(x) for x in subrange
if x%2 == 0 or x%2 != 0])
subrange = {x for x in range(1, 101)}
print '\n'.join([
'fizzbuzz' if x%15 == 0 else
'fizz' if x%3 == 0 else
'buzz' if x%5 == 0 else
str(x)
for x in subrange
if x%2 == 0 or x%2 != 0])
subrange = (x for x in range(1, 101))
FYI: if you use list comprehension within a function or method that immediately consumes the resulting list, you can usually omit it altogether:
subrange = {x for x in range(1, 101)}
print '\n'.join(
'fizzbuzz' if x % 15 == 0 else
'fizz' if x % 3 == 0 else
'buzz' if x % 5 == 0 else
str(x)
for x in subrange
if x % 2 == 0 or x % 2 != 0)
Huh. Never seen that.
But without the [] it appears to produce a generator expression (which is immediately consumed) rather than a list (which is immediately consumed).
EDIT: Also:
But without the [] it appears to produce a generator expression (which is immediately consumed) rather than a list (which is immediately consumed).
In [9]: print ' '.join(str(x) for x in range(3))
0 1 2
In [10]: type(str(x) for x in range(3))
Out[10]: generator
In [11]: type([str(x) for x in range(3)])
Out[11]: list
EDIT: Also:
In [3]: %timeit ' '.join(str(x) for x in range(3))
100000 loops, best of 3: 3.11 µs per loop
In [4]: %timeit ' '.join([str(x) for x in range(3)])
1000000 loops, best of 3: 1.47 µs per loop
I'll refer you to the original source where I picked up this trick: http://chimera.labs.oreilly.com/books/1230000000393/ch01.htm...
"Transforming and Reducing Data at the Same Time"
I highly recommend David Beazley's excellent book, it made me a much better Python developer, helping me produce both idiomatic and efficient solutions over the last year. That book is programming poetry.
"Transforming and Reducing Data at the Same Time"
I highly recommend David Beazley's excellent book, it made me a much better Python developer, helping me produce both idiomatic and efficient solutions over the last year. That book is programming poetry.
Thanks for that. Looks good so far.
Considering Python's goal of legibility, that syntax is hideous.
Python goes to extensive lengths to work around its mediocre lambdas. I'll take a language with good lambda syntax and higher-order-functions like map and filter (or select and where in C#).
Python goes to extensive lengths to work around its mediocre lambdas. I'll take a language with good lambda syntax and higher-order-functions like map and filter (or select and where in C#).
Guido van Rossum has vocally[1] expressed a distaste for functional programming idioms. His loss, I think.
List comprehensions have always struck me as a very ugly mechanism which is convenient in simple instances and which composes together poorly. This kind of special-casing is all over the place in Python's design. Consider, for example, all the unintuitive syntactic and semantic contortions introduced to support interval comparisons[2]. The overall result is a language which is apparently simple but in practice surprisingly complex.
[1] http://www.artima.com/weblogs/viewpost.jsp?thread=98196
[2] https://docs.python.org/2/reference/expressions.html#not-in
List comprehensions have always struck me as a very ugly mechanism which is convenient in simple instances and which composes together poorly. This kind of special-casing is all over the place in Python's design. Consider, for example, all the unintuitive syntactic and semantic contortions introduced to support interval comparisons[2]. The overall result is a language which is apparently simple but in practice surprisingly complex.
[1] http://www.artima.com/weblogs/viewpost.jsp?thread=98196
[2] https://docs.python.org/2/reference/expressions.html#not-in
>> List comprehensions have always struck me as a very ugly mechanism
I've always thought they are basically an implementation of set builder notation, e.g.:
[1] https://en.wikipedia.org/wiki/Set-builder_notation
I've always thought they are basically an implementation of set builder notation, e.g.:
X = {a | a ∈ Y}
X = [a for a in Y]
[1] https://en.wikipedia.org/wiki/Set-builder_notation
While normally following math notation is good, the lack of composability shown in the article is a real drawback of that syntax.
I think that's more of an implementation issue, rather than a problem with the idea of comprehensions. In set builder notation you can order elements as you like without changing the semantics. So you can denote the following "nested membership" statement in either order:
Consequently, you can't just throw any set of comprehension elements together and hope that you'll get something that makes sense. This messes up the elegance of the notation considerably, and it also makes it very, very fiddly to write some things that you might conceivably want to write, so much so that it ends up being much more readable to write a traditional for-loop, than use a comprehension.
There might even be orderings that are not allowed by the Python compiler, even though they might make sense in a declarative formalism. I can't think of any right now, but I'm pretty sure there are plenty- and lambdas are not going to be a get-out-of-jail-free card either.
X = {a | A ∈ Y, a ∈ A }
X = {a | a ∈ A, A ∈ Y}
>>> Y = (('a','b','c'), ('d','e','f'))
>>> X = [a for A in Y for a in A]
>>> X
['a', 'b', 'c', 'd', 'e', 'f']
>>> X = [a for a in A for A in Y]
>>> X
[('a', 'b', 'c'), ('a', 'b', 'c'), ('d', 'e', 'f'), ('d', 'e', 'f')]
Consequently, you can't just throw any set of comprehension elements together and hope that you'll get something that makes sense. This messes up the elegance of the notation considerably, and it also makes it very, very fiddly to write some things that you might conceivably want to write, so much so that it ends up being much more readable to write a traditional for-loop, than use a comprehension.
There might even be orderings that are not allowed by the Python compiler, even though they might make sense in a declarative formalism. I can't think of any right now, but I'm pretty sure there are plenty- and lambdas are not going to be a get-out-of-jail-free card either.
Here's a Python example in "map/filter" functional programming style, adapted from the generator version I posted elsewhere.
from fractions import Fraction as F
def thor():
for th in range(1, 27):
for o in range(1, 27):
for r in range(1, 27):
if 2*(th**2) + o**2 + r**2 == 1000:
yield th, o, r
def sz(thors):
th, o, r = thors
for s in range(1, 27):
for z in range(1, 27):
if (th**2 * z**2 * (s - z) + s**2) == 7225:
yield th, o, r, s, z
def gdxi(szs):
th, o, r, s, z = szs
for g in range(1, 27):
for d in range(1, 27):
for xi in range(1, 27):
if (g-1)**2 + d**2 + xi**2 - xi == 600:
yield th, o, r, s, z, g, d, xi
def eti(gdxis):
th, o, r, s, z, g, d, xi = gdxis
for (et, i) in (
(xi-7, xi-11), (xi-11,xi-7), (xi+7, xi+11), (xi+11, xi+7)):
if 0 < et <= 26 and 0 < i <= 26:
yield th, o, r, s, z, g, d, xi, et, i
def amupi(etis):
th, o, r, s, z, g, d, xi, et, i = etis
for (a, mu) in ((4, 1), (2, 2), (1, 4)):
for pi in range(1, 27):
if a*(a+pi) == 4 * g:
yield th, o, r, s, z, g, d, xi, et, i, a, mu, pi
def kben(amupis):
th, o, r, s, z, g, d, xi, et, i, a, mu, pi = amupis
for k in (3,4):
for b in range(1, 27):
for e in range(1, 27):
for n in range(1, 27):
if b**3 + z**3 + n**3 + o**9 == 1997:
if (F(a, k)**2 + F(b, n)**2 + F(d, xi)**2 +
F(e, pi)**2 + F(et, mu)**2 +
F(i, s)**2 == 6):
yield (a, b, g, d, e, z, et, th, i,
k, th, mu, n, xi, o, pi, r, s)
flatten = lambda l: [item for sublist in l for item in sublist]
thors = thor()
szs = flatten(filter(None, map(lambda x: list(sz(x)), thors)))
gdxis = flatten(filter(None, map(lambda x: list(gdxi(x)), szs)))
etis = flatten(filter(None, map(lambda x: list(eti(x)), gdxis)))
amupis = flatten(filter(None, map(lambda x: list(amupi(x)), etis)))
results = flatten(filter(None, map(lambda x: list(kben(x)), amupis)))
print('\t'.join(('a', 'b', 'g', 'd', 'e', 'z', 'et', 'th',
'i', 'k', 'l', 'mu', 'n', 'xi', 'o', 'pi', 'r', 's')))
for r in results:
print('\t'.join(map(str, r)))The last part can be simplified to this:
thors = thor()
szs = flatten(map(list, map(sz, thors)))
gdxis = flatten(map(list, map(gdxi, szs)))
etis = flatten(map(list, map(eti, gdxis)))
amupis = flatten(map(list, map(amupi, etis)))
results = flatten(map(list, map(kben, amupis)))
# print resultsI get that this may have been a reasonable attitude back in the '90s when most languages doing higher order functions were a bit cryptic-looking about it, but in 2017 there's enough variations on the theme that he should be able to come up with something clean and pythonic.
If your ideal is Lisp, it's not surprising if you find every other language lacking.
Lisp represents the ultimate sacrifice of legibility, type-safety, and familiarity in exchange for expressiveness. But it's reasonable to say "I like other languages than Lisp for other reasons, but I wish they had Lisp feature X in a form that fits the philosophy of the language".
I mean, Python could offer an alternate sexpr syntax that makes it a lisp, and that would provide nice higher-order-function support for map and filter and foldr, but that wouldn't be a "pythonic" solution to the problem.
I want a "pythonic" answer for map and filter and reduce. List Comprehensions are supposed to provide that, but imho they're a failed attempt. They're ugly and restrictive.
I mean, Python could offer an alternate sexpr syntax that makes it a lisp, and that would provide nice higher-order-function support for map and filter and foldr, but that wouldn't be a "pythonic" solution to the problem.
I want a "pythonic" answer for map and filter and reduce. List Comprehensions are supposed to provide that, but imho they're a failed attempt. They're ugly and restrictive.
Do you know about the Common Lisp loop macro? You can write the same list comprehension as in the article with almost exactly the same syntax using CL loop for, if and collect clauses. It is hard to take criticism like "<Language X> represents the ultimate sacrifice of legibility, type-safety, and familiarity in exchange for expressiveness" seriously if you do not even have basic proficiency with <Language X>.
That's because there's an argument to be made here that the blog poster here is misusing the comprehensions syntax.
While what he does works, it can just as easily be written as
You can do the same thing in the bigger case, and you end up nesting your structures such that you get something like
While what he does works, it can just as easily be written as
list(itertools.chain(*[[x for x in sublist if x%3 == 0] for sublist in nested_list]))
You can do the same thing in the bigger case, and you end up nesting your structures such that you get something like
results = [[[[[[[[[[[[[[[(a, b, g, d, e, z, et, th, i, k, th, mu, n, xi, o, pi, r, s)
for th in range(1, 27) if (2*(th**2) + o**2 + r**2 == 1000) and ((th**2 * z**2 * (s - z) + s**2) == 7225)]
for o in range(1, 27) if (b**3 + z**3 + n**3 + o**9 == 1997)]
for r in range(1, 27)]
for z in range(1, 27)]
for n in range(1, 27) if F(a, k)**2 + F(b, n)**2 + F(d, xi)**2 + F(e, pi)**2 + F(et, mu)**2 + F(i, s)**2 == 6]
for b in range(1, 27)]
for e in range(1, 27)]
for s in range(1, 27)]
for (a, mu) in ((4, 1), (2, 2), (1, 4)) if a*(a+pi) == 4 * g]
for pi in range(1, 27)]
for g in range(1, 27) if (g-1)**2 + d**2 + xi**2 - xi == 600]
for d in range(1, 27)]
for (et, i) in ((xi-7, xi-11), (xi-11,xi-7), (xi+7, xi+11), (xi+11, xi+7)) if 0 < et <= 26 and 0 < i <= 26]
for xi in range(1, 27)]
for k in (3,4)]
Nested list comprehensions in Common Lisp:
https://gist.github.com/lispm/7c009676fe6b86cff851bc8a1ba82d...
https://gist.github.com/lispm/7c009676fe6b86cff851bc8a1ba82d...
CL-USER 20 > [x (y <- '((0 1) (2 3) (4 5) (6 7) (8 9)))
(x <- y)
(zerop (mod x 3))]
(0 3 6 9)You might enjoy Peter Norvig teaching you this technique which he uses to solve the Zebra Puzzle in lesson 2 of CS212:
https://classroom.udacity.com/courses/cs212/lessons/48632874...
https://classroom.udacity.com/courses/cs212/lessons/48632874...
From the article:
However I would advise against any further nesting than this, you will not want to debug that if anything goes wrong with one of the inputs here, especially if you also do any conditional logic here in the comprehension.
result = [x for sublist in nested_list for x in sublist if x % 3 == 0]
result = [a for b in c for a in b if a % 3 == 0]
However I would advise against any further nesting than this, you will not want to debug that if anything goes wrong with one of the inputs here, especially if you also do any conditional logic here in the comprehension.
I'm a little confused by
for A in range(1, 27):
for B in range(1, 27):
if A == 2 * B:
for A in range(1, 27):
if A % 2 == 0:
B = A / 2
for A in range(2, 27, 2):
B = A / 2
If I were to start it in Common Lisp, I'd probably do something like this. Next steps might be to make normal variable declarations automatic by scanning guard expressions (which would prevent unwittingly mentioning a variable too early); and depending on the intended usage, maybe permit infix math via the infix-math package.
The b, n, k, e definitions are reordered better as in @eutectic's haskell example.
(defpackage solver (:use :cl :iterate) (:shadow pi))
(in-package :solver)
(defmacro defsolver (name default constraints result)
(flet ((normal-var (var) `(iter (for ,var ,@(rest default))))
(custom-var (var spec) `(iter (for ,var in ,spec)))
(guard (expr) `(when ,expr))
(solved () `(return-from ,name ,result)))
(macrolet ((yield (form) `(append ,form (list (constraint (rest cs))))))
(labels ((constraint (cs &aux (c (first cs)))
(cond ((null cs) nil)
((eq c :solved) (solved))
((symbolp c) (yield (normal-var c)))
((eq (first c) 'for) (yield (custom-var (second c) (third c))))
(t (yield (guard c))))))
`(defun ,name () ,(constraint `(,@constraints :solved)))))))
(defsolver result
(:default from 1 to 26)
(th
o
r
(= (+ (* 2 (expt th 2)) (expt o 2) (expt r 2)) 1000)
s
z
(= (+ (* (expt th 2) (expt z 2) (- s z)) (expt s 2)) 7225)
g
d
xi
(= (+ (expt (1- g) 2) (expt d 2) (expt xi 2) (- xi)) 600)
(for (et i) `((,(- xi 7) ,(- xi 11)) (,(- xi 11) ,(- xi 7))
(,(+ xi 7) ,(+ xi 11)) (,(+ xi 11) ,(+ xi 7))))
(and (< 0 et) (<= et 26) (< 0 i) (<= i 26))
(for (a mu) '((4 1) (2 2) (1 4)))
pi
(= (* 4 g) (* a (+ a pi)))
b
n
(= (+ (expt b 3) (expt z 3) (expt n 3) (expt o 9)) 1997)
(for k '(3 4))
e
(= (+ (expt (/ a k) 2) (expt (/ b n) 2) (expt (/ d xi) 2)
(expt (/ e pi) 2) (expt (/ et mu) 2) (expt (/ i s) 2)) 6))
(list a b g d e z et th i k th mu n xi o pi r s))
SOLVER> (time (result))
Evaluation took:
0.010 seconds of real time
0.012000 seconds of total run time (0.012000 user, 0.000000 system)
120.00% CPU
23,160,723 processor cycles
851,968 bytes consed
(4 9 19 12 15 3 1 20 5 4 20 1 9 12 2 15 14 5)
The b, n, k, e definitions are reordered better as in @eutectic's haskell example.
I don't see why yield has to be a macrolet rather than a labels.
You're right, it could, if you prefer. In the same labels form with an extra parameter cs, which the yield's all pass in; or in a labels/flet form inside constraint.
(defmacro defsolver (name default constraints result)
(flet ((normal-var (var) `(iter (for ,var ,@(rest default))))
(custom-var (var spec) `(iter (for ,var in ,spec)))
(guard (expr) `(when ,expr))
(solved () `(return-from ,name ,result)))
(labels ((constraint (cs &aux (c (first cs)))
(flet ((yield (form) (append form (list (constraint (rest cs))))))
(cond ((null cs) nil)
((eq c :solved) (solved))
((symbolp c) (yield (normal-var c)))
((eq (first c) 'for) (yield (custom-var (second c) (third c))))
(t (yield (guard c)))))))
`(defun ,name () ,(constraint `(,@constraints :solved))))))All your functions can be in a single labels block now. It's okay for a non-recursive function to be a labels. flet is really for those situations in which labels cannot be used, like defining an inner foo which shadows and calls (i.e wraps) an outer foo.
Instead of
for a in range(1,27) for b in range(1,27) if a == 2*b ...
for b in range(1,27) for a in [2*b] if a < 27 ...This is true for the toy example. In the actual problem, the loop guards are quadratic equations or even higher order, and his first method generalizes better (and for small ranges is hardly slower).
What about making an interactive version of theaeticle with klipse?
More like "why not to use Python"
better title: Why software engineers clearly need no math skills whatsoever - solving a 12x18 matrix in as little as 15 loops
ever heard of gauss algorithm?
ever heard of gauss algorithm?
Those are non-linear equations.
These are all polynomial equations, so we can likely solve them by computing a Grobner basis over the set and using the Elimination Theorem.
I don't know what any of those are but I know how to write 15 nested for loops and let it run for a few minutes. Hacking code seems fine if that's what someone knows...
That's the failure of maths education. Plenty of names for formal things and ideas, yet mostly inaccessible to the lay person.
So I know 6 programming languages, maybe 600 libraries, devops, dozens of tools.
So I could go back to math class and learn some more formal proofs. What part of my programming knowledge am I going to drop to learn that? Will it actually help my career?
I have found little to no use for advanced mathematics in 10 years of programming. I am not saying it had no use, but I have not needed it.
Most academic types majorly overestimate the usefulness of math in programming. I can code circled around people who rock out at math.
So I could go back to math class and learn some more formal proofs. What part of my programming knowledge am I going to drop to learn that? Will it actually help my career?
I have found little to no use for advanced mathematics in 10 years of programming. I am not saying it had no use, but I have not needed it.
Most academic types majorly overestimate the usefulness of math in programming. I can code circled around people who rock out at math.
I don't mean to demean your programming experience or toolkit decisions in anyway. Egg on my face--it appears that's how it was received! Let me try again.
> I have found little to no use for advanced mathematics in 10 years of programming.
I reckon you use or stumble upon the algorithms often, perhaps all the time. Be it with Boolean logic, or network design, or optimal stopping, or fuzzy matching, or statistics in logging, or what have you. These principles show up often. My point wasn't you can't or shouldn't be able to code circles around people who rock out at math, but that maths education often fails to emphasize the logical methods of figuring things out from first principles over the already solved proofs, methods, or equations.
> Most academic types majorly overestimate the usefulness of math in programming.
To some degree, I agree here. If there are well developed libraries and mature paradigms needed (RDBMS, web dev, etc.) then there is little need for mathematical forte. Nothing wrong with that!
> I have found little to no use for advanced mathematics in 10 years of programming.
I reckon you use or stumble upon the algorithms often, perhaps all the time. Be it with Boolean logic, or network design, or optimal stopping, or fuzzy matching, or statistics in logging, or what have you. These principles show up often. My point wasn't you can't or shouldn't be able to code circles around people who rock out at math, but that maths education often fails to emphasize the logical methods of figuring things out from first principles over the already solved proofs, methods, or equations.
> Most academic types majorly overestimate the usefulness of math in programming.
To some degree, I agree here. If there are well developed libraries and mature paradigms needed (RDBMS, web dev, etc.) then there is little need for mathematical forte. Nothing wrong with that!
[deleted]
Math often provides a much easier and elegant solution to things:
http://www.evanmiller.org/mathematical-hacker.html
http://www.evanmiller.org/mathematical-hacker.html
There are only so many hours in the day to learn things.
OK now you've got a variety over Q, good luck finding the integral points in the domain that you care about.
According to OP these are "12 equations in 18 variables, where the variables were restricted to integers in the range [1, 26]". Can you solve them using the elimination theorem?
Good grief.
Must..write..in..functional..style..to..be..l33t....
From the article: "Don't do this at work, adults!"
From the article: "Don't do this at work, adults!"