Numpy is absurd(gist.github.com)
gist.github.com
Numpy is absurd
https://gist.github.com/Moelf/59d6312c51c250ba251125e54bea7282
51 comments
Well yeah you're right OP could have explained this better; but this argues their point. From the definition that `a in b` is `(a == b).any()` it's trivial to see things like `np.array([1, 0, 0]) in np.array([1, 2, 3])` will return `True` too, which is absurd, as the title says. I find it hard to believe when numpy team was designing the API they missed this.
Even that took me a while to understand it and, I guess, is another example of Numpy’s absurdity.
My initial thought was “None of the examples have arguments that are equal to each other, so that should evaluate to false.any()”, but that would mean all should return the same value, so that can’t be true, so I figured out that Numpy overloads/changes a == b to do item-by-item comparison on arrays, and return an array of booleans.
But then, the statement “This really only makes sense when `a` is a scalar” doesn’t make sense to me. That seems to indicate Numpy also overloads a == b as a comparison of each element of a with a single value b and vice versa, but that doesn’t make sense, given that arrays can contain arrays, making it impossible to discriminate between [1,2] as a two-item sequence and [1,2] as a single-item sequence where the item is a two-item sequence.
Reading https://stackoverflow.com/questions/10580676/comparing-two-n... further strengthens my impression that one should avoid using == in Numpy.
My initial thought was “None of the examples have arguments that are equal to each other, so that should evaluate to false.any()”, but that would mean all should return the same value, so that can’t be true, so I figured out that Numpy overloads/changes a == b to do item-by-item comparison on arrays, and return an array of booleans.
But then, the statement “This really only makes sense when `a` is a scalar” doesn’t make sense to me. That seems to indicate Numpy also overloads a == b as a comparison of each element of a with a single value b and vice versa, but that doesn’t make sense, given that arrays can contain arrays, making it impossible to discriminate between [1,2] as a two-item sequence and [1,2] as a single-item sequence where the item is a two-item sequence.
Reading https://stackoverflow.com/questions/10580676/comparing-two-n... further strengthens my impression that one should avoid using == in Numpy.
I've worked with numpy for years and I don't think I ever used the 'in' operator with arrays. What would be the use case for that? For me a larger gotcha was around object multiplication (i.e. element-wise vs. matrix multiplication) and how it is different based on whether you have a "matrix" or an "array" type (or a mix of both). That caused some pretty big headache in the beginning.
But that just seems like a foot gun for the poor unfortunate soul that eventually does.
Meh. Footguns are underrated. Let’s return to the Wild West of the 90s.
I’m only partially joking. But this was one case where they probably thought it was a good idea, turned out to be a bad idea, then couldn’t turn it off without breaking a bunch of code.
I’m only partially joking. But this was one case where they probably thought it was a good idea, turned out to be a bad idea, then couldn’t turn it off without breaking a bunch of code.
All these talk about code safety just want me to stop wearing my bike helmet.
For most of my programs safety is of zero concern. I'll trade it for convenience any day. Just let me implicit cast signed to unsigned thank you very much.
For most of my programs safety is of zero concern. I'll trade it for convenience any day. Just let me implicit cast signed to unsigned thank you very much.
I don't think people who use python are concerned with footguns. Numpy is very much not at fault here.
Why would you think that?
Because the typing system used in Python is a big footgun. If the basis of the language you are using encourages footguns then it's not surprising that a large library in that ecosystem doesn't care about that either.
iirc matrix is deprecated
Interpreting “in” as “is an element of”, all of these are type errors. It only makes sense to ask if an X is in an Array<X>. In a language that lets you ask nonsense questions, expect nonsense answers.
Numpy has the highly useful operation of `a < b` where a and b are arrays and the answer is an array of booleans - i.e broadcasted elementwise compare.
They changed the language to change what question `<` asks.
They changed the language to change what question `<` asks.
Sure, and languages like C++ let you do this kind of thing very broadly, but it has a lot of pitfalls. One of my biggest gripes with python is that function interfaces and "type signatures" are hard to discover even when documentation is good.
FWIW I think it's very hard to discover this from C++ code as well. It has to be documented.
This gets ugly with !=. Instead of 'a != b' you usually want 'not (a == b).all()'.
It really doesn't if you think about it. a==b cannot by itself be used as an if condition. You would always need to use (a==b).all(). So it would be very difficult to make the mistake you are suggesting
https://stackoverflow.com/questions/10062954/valueerror-the-...
https://stackoverflow.com/questions/10062954/valueerror-the-...
I think the general point is that it was a bad idea to implement the operators as elementwise because it breaks all of the properties they are supposed to hold.
Really there should be separate elementwise operators like `.>` and so on but you can only do that in some languages.
Really there should be separate elementwise operators like `.>` and so on but you can only do that in some languages.
> I always gripe about Python not having built-in array type
> Numpy doesn't distinguish "vector of vector" from "matrix"
from array import array
> Numpy doesn't distinguish "vector of vector" from "matrix"
numpy.matrix
when built-in array type is not performant nor serves as an interface (abstract class), it's as if it doesn't exist. I guess the point is Numpy predates Python's array by a long shot.
Compare and contrast with languages with built-in N-dimensional array for scientific computing.
Compare and contrast with languages with built-in N-dimensional array for scientific computing.
> Numpy predates Python's array by a long shot
I don't think so. The array module was included even in Python 1.0.
Or did you mean to use another verb?
I don't think so. The array module was included even in Python 1.0.
Or did you mean to use another verb?
To be more specific, "git log arraymodule.c" reports:
commit 778983b48165da25ee11fb97f6855af7c67f4ff2
Author: Guido van Rossum <[email protected]>
Date: Fri Feb 19 15:55:02 1993 +0000
Added new module "array" (for now optional) defining array objects.like matlab, that lies?
What do you mean?
in matlab:
Numpy can:
>> size(4)
ans =
1 1
Numpy can:
>> np.shape(4)
()I'm confused by what you mean. In Matlab, everything is a matrix. A naked integer is a 1x1 matrix, hence why the size is reported as [1 1]. Those are the semantics of the language, not a lie.
a naked integer is not a 1x1 matrix, it’s a 0x0. Matlab silently promotes it to one, though, which is a deliberately misleading thing to do because it’s convenient for matlab’s data model.
i think it’s fair to call that a (white) lie
i think it’s fair to call that a (white) lie
> a naked integer is not a 1x1 matrix, it’s a 0x0
Do you have a link to where this is asserted in the documentation? I'm confused as to the basis for this.
Do you have a link to where this is asserted in the documentation? I'm confused as to the basis for this.
https://mathworld.wolfram.com/TensorRank.html
it should be clear that i don’t care very much about what matlab has to represent things as. i don’t mean “matlab integers”, i mean the mathematical objects; matlab’s representation of them is poor.
it should be clear that i don’t care very much about what matlab has to represent things as. i don’t mean “matlab integers”, i mean the mathematical objects; matlab’s representation of them is poor.
Okay, well that's not Matlab lying, that's you not understanding how the tool works. It would be like saying that 0.1 + 0.2 == 0.3 evaluating to false is a lie, because floating point numbers don't accurately represent real numbers as a concept. But floating point numbers aren't actually real numbers, and integers in Matlab (they're actually stored as floating point doubles, is that a lie too?) aren't actually mathematical objects.
You have to work within the constraints of the tool. Matlab makes it very clear upfront that the semantics of the language mean that naked numbers are in fact matrices, because everything in the language is a matrix. Maybe you feel that's not a good representation, but it's not a lie, and in fact it's very consistent and useful in the context of the language itself.
You have to work within the constraints of the tool. Matlab makes it very clear upfront that the semantics of the language mean that naked numbers are in fact matrices, because everything in the language is a matrix. Maybe you feel that's not a good representation, but it's not a lie, and in fact it's very consistent and useful in the context of the language itself.
Julia
Op here, I'd also like to say I blame this on Python's lack of broadcasting syntax, (dot in Matlab and Julia)
Which forces Numpy to be designed in a way to automatically broadcast, guessing what user wants is hard in nested container
Which forces Numpy to be designed in a way to automatically broadcast, guessing what user wants is hard in nested container
Julia's broadcasting syntax depends on having Julia-style compilation to be efficient. It's not clear to me Python could ever provide the same behavior.
import numpy as np
print(np.__version__)
a = np.array([1,2,3])
b = np.array([1])
print(a in b)
print(b in a)
print(np.isin(a, b))
print(np.isin(b, a))
print(np.isin(a.astype(float), b))
print(np.isin(a, b.astype(float))))
1.21.6
True
True
[ True False False]
[ True]
[ True False False]
[ True False False]Core numpy is really weird, and has really sloppy logic around higher-dimensional objects. I've a module to work around lots of the warts. If somebody has smarter versions of intersect() or in() or whatever, contribute!
https://github.com/dkogan/numpysane
https://github.com/dkogan/numpysane
Apologies if this was explained in the talk, but why not upstream your changes to numpy?
No real good reason, honestly. I wasn't feeling positively about numpy devs at the moment, and didn't want to negotiate getting the code in. The library is solid though, and should be merged. I need somebody to pester me about it, to actualy work on that today.
For a language with "explicit is better than implicit" as part of its mantra[1], Python is chock full of surprising behavior like this where the programmer really has no idea what's happening under the hood.
I'm happy I moved on to Go, which actually adopts this as a core design tenet.
[1]: Even though PEP-20 is largely a meme.
I'm happy I moved on to Go, which actually adopts this as a core design tenet.
[1]: Even though PEP-20 is largely a meme.
To be fair this person's gripes are with the behavior of Numpy arrays, not anything that's native to Python's core language. Numpy arrays are tricky beasts because they try to balance fast operations on huge amounts of data, with expressive Python operator syntax.
Although, another way of looking at what you just said is that Numpy arrays exist because core Python fails to strike a balance between being both fast and expressive.
That does not necessarily follow. Python may be striking a balance between the needs of numerical and non-numerical computing, while NumPy arrays do not need to consider non-numerical computing.
Case in point, historically NumPy has been designed for long-running programs, like hour-long calculations or notebook processing. They've chosen to import "the world" so users don't need explicit subpackage imports, allowing:
Case in point, historically NumPy has been designed for long-running programs, like hour-long calculations or notebook processing. They've chosen to import "the world" so users don't need explicit subpackage imports, allowing:
>>> import numpy
>>> numpy.polynomial.hermite.hermfromroots([2,2,3,4])
array([ 7.075e+01, -4.625e+01, 1.175e+01, -1.375e+00, 6.250e-02])
[deleted]
> I always gripe about Python not having built-in array type
Python does have an array type. It’s existed from at least v2.6 (released 2008). https://docs.python.org/3.10/library/array.html
Python does have an array type. It’s existed from at least v2.6 (released 2008). https://docs.python.org/3.10/library/array.html
Oh no! Undefined behaviour! What ever shall we do?
Finally! It had to be said. Im a huge Python fan and a huge Pandas fan and have written hundreds of scripts. But numpy is so "R" esque and technical and stiff and unintuitive for sure.
Answer: `a in b` is the same as `(a == b).any()`. This really only makes sense when `a` is a scalar and any behavior for other types of arrays are more "accidental" than designed.