Beartype: Fast runtime type checking in Python(github.com)
github.com
Beartype: Fast runtime type checking in Python
https://github.com/beartype/beartype
61 comments
Yes, though as eg QuickCheck shows, randomised testing is unreasonably effective in practice.
assuming you're sampling the same distribution. every call to a function can have different arguments. this seems to make the assumption that the types and shape don't change so you can amortize the cost of type checking over multiple function calls... but they can, and do, sometimes with low probability and those are the hard bugs to solve.
especially when you think types are being checked.
especially when you think types are being checked.
I suspect you are reading different things into my comment than I wanted to convey.
Your argument is perfectly valid, if you try to steelman my argument into something like a mathematical argument. But it wasn't meant that way.
I meant exactly what I said: empirically, in practice, randomised checking is unreasonably effective. You illustrated the 'unreasonable' part.
Your argument is perfectly valid, if you try to steelman my argument into something like a mathematical argument. But it wasn't meant that way.
I meant exactly what I said: empirically, in practice, randomised checking is unreasonably effective. You illustrated the 'unreasonable' part.
i'd never seen this quickcheck thing before... it's basically fuzzing built into unit tests, neat...
it is, however, quite different from a type checker though. where a fuzz tester generates random inputs to try and break a function... a type checker enforces assertions on data that is actually flowing through the system.
i think my big beef with this library is that it does so probabilistically, while marketing itself as something that does complete checks and has been highly optimized. it's not a fast assertion checker, it is one that employs random sampling... that's fine, but it should be extremely upfront about that as this misunderstanding has the potential to cause a lot of pain on top of already painful situations.
it is, however, quite different from a type checker though. where a fuzz tester generates random inputs to try and break a function... a type checker enforces assertions on data that is actually flowing through the system.
i think my big beef with this library is that it does so probabilistically, while marketing itself as something that does complete checks and has been highly optimized. it's not a fast assertion checker, it is one that employs random sampling... that's fine, but it should be extremely upfront about that as this misunderstanding has the potential to cause a lot of pain on top of already painful situations.
Yes, calling this a typechecker is perhaps a bit euphemistic.
Btw, if you want to try something like QuickCheck, Python's Hypothesis is perhaps the most accessible implementation of the idea.
Btw, if you want to try something like QuickCheck, Python's Hypothesis is perhaps the most accessible implementation of the idea.
QuickCheck generates its own test cases and possible shapes. More importantly, it tries all possible constructors of a type (or a reasonable number). So you're reasonably confident that you tried a lot of "shape-cases".
> it tries all possible constructors of a type
It tries a 'reasonable' number. If you want to try everything, have a look at SmallCheck.
I would imagine you would want to combine Beartype with tests, too. Perhaps written manually, or with test data generated via Hypothesis.
It tries a 'reasonable' number. If you want to try everything, have a look at SmallCheck.
I would imagine you would want to combine Beartype with tests, too. Perhaps written manually, or with test data generated via Hypothesis.
> If you want to try everything, have a look at SmallCheck.
FYI my favourite approach is LazySmallCheck, which runs tests with an unevaluated input, and only generates data if the input gets forced; at which point it runs the test for each constructor. The original version lacked some useful generators, like functions; the "LazySmallCheck2012" fork added these (e.g. generating functions in terms of an input/output lookup table), but overall these libraries seem to be unmaintained :(
IIRC there are also similar alternatives based around Prolog-like search.
FYI my favourite approach is LazySmallCheck, which runs tests with an unevaluated input, and only generates data if the input gets forced; at which point it runs the test for each constructor. The original version lacked some useful generators, like functions; the "LazySmallCheck2012" fork added these (e.g. generating functions in terms of an input/output lookup table), but overall these libraries seem to be unmaintained :(
IIRC there are also similar alternatives based around Prolog-like search.
but not all cases are generated..
for example, if I have an external api response that I'd like to validate, beartype fails to do so.
for example, if I have an external api response that I'd like to validate, beartype fails to do so.
I don't understand the section on the coupon collector's problem and its relevance to beartype. The coupon collector problem is talking about the expected number of random beartype-style checks it would take to fully check every element in a collection (basically how long on average it would take to do the same thing typeguard does).
But I fail to see how that's relevant to beartype. Beartype's constant-time runtime has nothing to do with the coupon collector problem. It has to do with the simple fact that no matter how large the container beartype will only sample one random element from that container.
Likewise, when the README mentions
> Since h ≥ 1, beartype samples at least as many items each call as assumed in the usual coupon collector's problem and thus paradoxically takes a fewer number of calls on average to check all items of a container containing arbitrarily many subcontainers as it does to check all items of a container containing no subcontainers.
that's just because beartype happens to check one element from each level of nesting of a container so a deeply nested container just gets more checks in a single call than a non-nested container right? If I compare a list with 50 elements and a list with 50 levels of nesting in beartype but a single element underneath all that nesting, clearly the latter only requires a single call to fully type-check while the former takes 225 calls to fully check on average. But that's just because beartype is doing a different amount of work in each case, since in the latter case beartype samples all 50 levels all at once.
Unless I'm missing something the entire discussion on the coupon collector problem seems irrelevant.
But I fail to see how that's relevant to beartype. Beartype's constant-time runtime has nothing to do with the coupon collector problem. It has to do with the simple fact that no matter how large the container beartype will only sample one random element from that container.
Likewise, when the README mentions
> Since h ≥ 1, beartype samples at least as many items each call as assumed in the usual coupon collector's problem and thus paradoxically takes a fewer number of calls on average to check all items of a container containing arbitrarily many subcontainers as it does to check all items of a container containing no subcontainers.
that's just because beartype happens to check one element from each level of nesting of a container so a deeply nested container just gets more checks in a single call than a non-nested container right? If I compare a list with 50 elements and a list with 50 levels of nesting in beartype but a single element underneath all that nesting, clearly the latter only requires a single call to fully type-check while the former takes 225 calls to fully check on average. But that's just because beartype is doing a different amount of work in each case, since in the latter case beartype samples all 50 levels all at once.
Unless I'm missing something the entire discussion on the coupon collector problem seems irrelevant.
Oof, if it really picked elements at random to type check, I'd actually say that'd be "unbearable"
That's what it does.
> Basically, beartype type-checks randomly sampled data.
So, it doesn't really type check and the chance that
So, it doesn't really type check and the chance that
l: list[str] = ["good", 1]
Pretty much, yes; however this would flag up as a flaky integration test, prompted you to fix if you have a hard rule about no flakyness in tests
What always annoys me to no end is when people complain that python type hints are not runtime enforced, when this is quite normal for most statically typed languages: C++, go, rust.
Python already had runtime type checking mechanisms and the use of these does inform the behavior of static type checkers for python like mypy.
All that being said, Beartype is really nifty and seems quite useful.
Python already had runtime type checking mechanisms and the use of these does inform the behavior of static type checkers for python like mypy.
All that being said, Beartype is really nifty and seems quite useful.
The spirit of a type-system is that invalid code never runs.
The compilation step in static languages type-checks such that the compiled-instructions wont run if the source-code is invalid.
Since this isnt how python works, one would reasonably expect the spirit of a type system to be preserved -- ie., to enforce runtime checks.
The compilation step in static languages type-checks such that the compiled-instructions wont run if the source-code is invalid.
Since this isnt how python works, one would reasonably expect the spirit of a type system to be preserved -- ie., to enforce runtime checks.
> The spirit of a type-system is that invalid code never runs.
> The compilation step in static languages type-checks such that the compiled-instructions wont run if the source-code is invalid.
Nothing is forcing anyone to run code that does not pass type checking with `mypy --strict`.
> Since this isnt how python works, one would reasonably expect the spirit of a type system to be preserved -- ie., to enforce runtime checks.
Python already had mechanisms for this, which again, as I said, already informs the static type checking mechanisms.
Also, no, this is not what I expect from the type hints, this is not what I felt was missing from python before type hints.
What I want from type hints and in python in general is static type checking, I want to know, without executing one single line of code, if I have type errors, that is the key selling point of mypy and type hints to me and the key deficiency that python had without type hints and mypy.
I already had every ability to do runtime type checking in python before type hints and mypy, and they were woefully inadequate in comparison to static type checking.
> The compilation step in static languages type-checks such that the compiled-instructions wont run if the source-code is invalid.
Nothing is forcing anyone to run code that does not pass type checking with `mypy --strict`.
> Since this isnt how python works, one would reasonably expect the spirit of a type system to be preserved -- ie., to enforce runtime checks.
Python already had mechanisms for this, which again, as I said, already informs the static type checking mechanisms.
Also, no, this is not what I expect from the type hints, this is not what I felt was missing from python before type hints.
What I want from type hints and in python in general is static type checking, I want to know, without executing one single line of code, if I have type errors, that is the key selling point of mypy and type hints to me and the key deficiency that python had without type hints and mypy.
I already had every ability to do runtime type checking in python before type hints and mypy, and they were woefully inadequate in comparison to static type checking.
Is the standard library type checked?
Are there still mistakes in the typeshed like 2**-1 is int?
Are there still mistakes in the typeshed like 2**-1 is int?
> Is the standard library type checked?
There are type hints for the standard library and using `--strict` works just fine if that is all you are using. This does not preclude bugs from the standard library or errors in types but bugs in a statically typed language's compiler also does not defeat the purpose of static typing either and having a static typed language does not eliminate the possibility of bugs in the compiler or standard library either.
It seems people forget that reinterpret_cast, void * (c++), Object (java), (interface{}) (golang) exists. If all you are using is the standard library then you will be able to write much safer code in python than in golang, given it passes static type validation. At least python's type system has generics.
> Are there still mistakes in the typeshed like 2*-1 is int?
I'm sure there are still mistakes, just like I'm sure there are still bugs in most compilers.
There are type hints for the standard library and using `--strict` works just fine if that is all you are using. This does not preclude bugs from the standard library or errors in types but bugs in a statically typed language's compiler also does not defeat the purpose of static typing either and having a static typed language does not eliminate the possibility of bugs in the compiler or standard library either.
It seems people forget that reinterpret_cast, void * (c++), Object (java), (interface{}) (golang) exists. If all you are using is the standard library then you will be able to write much safer code in python than in golang, given it passes static type validation. At least python's type system has generics.
> Are there still mistakes in the typeshed like 2*-1 is int?
I'm sure there are still mistakes, just like I'm sure there are still bugs in most compilers.
C++ type system allows invalid code (that uses freed or uninitialized memory) to run and just do weird incorrect things.
Sort-of. That code is invalid in some informal sense, but not invalid as far as the formal typesystem is concerned.
There's also something about 'soundness' one could say.
There's also something about 'soundness' one could say.
So does every Turing-complete language with non-termination.
Type systems are decidable and thus cannot, by definition, rule out all errors. Instead, they look at a subset of possible errors. This subset is provably eliminated from valid code. "Dynamically typed" languages reduce that subset to the empty set.
Type systems are decidable and thus cannot, by definition, rule out all errors. Instead, they look at a subset of possible errors. This subset is provably eliminated from valid code. "Dynamically typed" languages reduce that subset to the empty set.
This comment is in the top #10 of the stupidest things I've read on this site. Congratulations!
As personal advice, I suggest not being annoyed about a thing you're totally wrong about.
As personal advice, I suggest not being annoyed about a thing you're totally wrong about.
Am I unreasonable in hoping for optional type hints to be taken advantage of by a compiler to produce correct, optimized code, similar to what (sufficiently smart) Common Lisp compilers do?
Can I use this on my full codebase all at once, without inserting "from beartype import beartype" into each file and decorating every single function and method?
I guess you could, near the beginning of your program's entry point, decorate the importlib's functions to decorate the top-level functions of imported modules with "@beartype" decorator.
Or there is always find+grep+sed, if you prefer a different style of madness.
Or there is always find+grep+sed, if you prefer a different style of madness.
> emphasizing efficiency, portability, and thrilling puns.
Results: Runs 1000x slower than not using it, but 200x faster than the competition!
So only use this in correctness-critical paths, or use a language that supports strong typing at compile time.
Results: Runs 1000x slower than not using it, but 200x faster than the competition!
So only use this in correctness-critical paths, or use a language that supports strong typing at compile time.
Whatever your opinion of the library or runtime type checking in general, you've got to admit that's a helluva README they've put together.
absolutely, I don't like type checking at all, and especially not runtime type checking, but the README has made me want to try it.
Could someone explain the benefit of runtime type checking? I thought the whole point of type checking was to catch errors before runtime. If it's already running, wouldn't it be better to just let it keep running and see what happens?
Is the idea that you would run this during your tests and it would give you more insight than a static type checker?
EDIT: The answer is in the FAQ https://github.com/beartype/beartype#frequently-asked-questi...
It seems like it would never cause an error, and it's really for type checking dynamic code. I'm still trying to wrap my head around it, but it seems promising so far.
Is the idea that you would run this during your tests and it would give you more insight than a static type checker?
EDIT: The answer is in the FAQ https://github.com/beartype/beartype#frequently-asked-questi...
It seems like it would never cause an error, and it's really for type checking dynamic code. I'm still trying to wrap my head around it, but it seems promising so far.
Type hints aren't actually enforced in Python. So for example a third party can import your code and call functions with the wrong type. Because of duck typing, sometimes this won't result in an error but could still cause a bug. You can explicitly check types with isinstance(), this library automates that.
Type hints aren't actually enforced in Python. So for example a third party can import your code and call functions with the wrong type. Because of duck typing, sometimes this won't result in an error but could still cause a bug.
Sure, but it also might work, if you've already made it through testing and deployment, surely you don't want to cause a new production bug because there might be another bug.
What I would really want at runtime is something that automatically casts and validates. As in, I don't care if it's a string, I care that if I wrap it in str it matches this regex.
Sure, but it also might work, if you've already made it through testing and deployment, surely you don't want to cause a new production bug because there might be another bug.
What I would really want at runtime is something that automatically casts and validates. As in, I don't care if it's a string, I care that if I wrap it in str it matches this regex.
> something that automatically casts and validates
This is what Pydantic does.
This is what Pydantic does.
I've been using Pydantic to validate inputs for one of my projects and really like it.
Finding bugs as early as possible, rather than soldiering on, is most often what you want. The reason is that the farther the code executes, the details and error messages of what went wrong in the past can get fuzzier and harder to diagnose.
If you're doing ducktyping... it should be specified explicitly, believe Python typing calls it a "protocol."
If you're doing ducktyping... it should be specified explicitly, believe Python typing calls it a "protocol."
From my guesses, this is mainly designed for data engineering, where you do run the same operation on a lot of entries. N is very very big, so it will negate the probabilistic nature of the checker.
Other than that, this is simply useless. Note that, if your function arguments are primitives (int, str, etc), you can design your function to fail early when the type is incorrect. Things can be trickier with objects, but still doable. So, no, this won't be useful for anything w/ deep logic.
Also, half serious half joke, but you should reconsider your language choice if you need stuffs like this, because Python isn't good at both performance and type safety.
Other than that, this is simply useless. Note that, if your function arguments are primitives (int, str, etc), you can design your function to fail early when the type is incorrect. Things can be trickier with objects, but still doable. So, no, this won't be useful for anything w/ deep logic.
Also, half serious half joke, but you should reconsider your language choice if you need stuffs like this, because Python isn't good at both performance and type safety.
Runtime type checking.. What a dreadful idea
Most places I see runtime type checking, is when checking the type of incoming data, aka, Json, or if the function is a library and you are helping the callers use the correct types.
Go/java, etc all do this by default. You read an int from a in memory untyped object, and you get an error if it's not actually an int.
Much worse than static type checking.
But infinitely better than no type checking.
So does it truly take the level of resources of Microsoft to make a good type system? It’s strange to see TypeScript infect the whole JS world while Python remains fractured at best with regards to type systems
?
This isn't a fork of the type system. Python's type annotations are formally part of the language, a program correctly typing on mypy will also typecheck on Pylance etc. The OP is a different implementation of type checking, runtime as opposed to static.
Python's problem is that there is no equivalent of @types/*, which is in fact a notable issue but also entirely orthogonal.
This isn't a fork of the type system. Python's type annotations are formally part of the language, a program correctly typing on mypy will also typecheck on Pylance etc. The OP is a different implementation of type checking, runtime as opposed to static.
Python's problem is that there is no equivalent of @types/*, which is in fact a notable issue but also entirely orthogonal.
> Python's type annotations are formally part of the language
> Python's problem is that there is no equivalent of @types/*
Yes, this is exactly why I asked:
> Microsoft to make a good type system
Operative word "good". A type system that loses types for libraries has, to say the least, extremely limited value. More to the point, whatever typing Python has is wildly unpopular amongst Python engineers, and when I talk to my Python-using colleagues they seem to have no clue they can even use types. Perhaps it's also a community problem in this way.
> Python's problem is that there is no equivalent of @types/*
Yes, this is exactly why I asked:
> Microsoft to make a good type system
Operative word "good". A type system that loses types for libraries has, to say the least, extremely limited value. More to the point, whatever typing Python has is wildly unpopular amongst Python engineers, and when I talk to my Python-using colleagues they seem to have no clue they can even use types. Perhaps it's also a community problem in this way.
Are there any code editors/editor plugins that make effective use of python type hints? The python plugins for visual studio code didn't make much use of the type hints, last time that I checked. i was hoping that type hints would be used for code completion, but that doesn't seem to be the case.
> The python plugins for visual studio code didn't make much use of the type hints, last time that I checked.
When was that? The Python Vscode extension with Pylance is pretty much perfect, Intellisense works as well as it does in, say, Java.
When was that? The Python Vscode extension with Pylance is pretty much perfect, Intellisense works as well as it does in, say, Java.
about two or three months ago, I added type annotations to one of my libraries, but the intellisense didn't make any use of that.
PyCharm has its own engine and while not perfect, it's fairly good.
I’m not a huge fan of pycharm’s inference and autocomplete. It’s not that it’s bad, I was really just that I was hoping for more.
There is so much that could be done eg IDEs could enable proper exhaustive signature checks when running code in debuggers. Pypy could probably do some really good cheap checks in debug mode too. And if not checking while debugging, they could at least highlight failed hints in the call stack when the debugger stops.
A long long time ago before MyPy I made a runtime checker called Obiwan. I always preferred its syntax to MyPy, eg [int] instead of list[int]. I then got into playing with SSA within functions and that really helped type checking.
So there’s loads more I think could be done to do what typescript did for JavaScript but to python.
There is so much that could be done eg IDEs could enable proper exhaustive signature checks when running code in debuggers. Pypy could probably do some really good cheap checks in debug mode too. And if not checking while debugging, they could at least highlight failed hints in the call stack when the debugger stops.
A long long time ago before MyPy I made a runtime checker called Obiwan. I always preferred its syntax to MyPy, eg [int] instead of list[int]. I then got into playing with SSA within functions and that really helped type checking.
So there’s loads more I think could be done to do what typescript did for JavaScript but to python.
And when it fails to infer the type of something you can add `assert isinstance(foo, T)`, and PyCharm will make use of that.
This will throw an error in runtime for bad types, which might or might not be your intention (unless of course you build the package, which only makes it all the more harder to debug if this is where your program fails at this step). So, just don’t use assert outside of tests?
For sure, adding that line of code will affect the runtime as well as type hinting in the IDE. Certainly don't use it unless you're sure that the asserted condition is actually always true (or not sure and want to know if it isn't!).
are you using the free community edition of pycharm, or do you need the paid one for that?
I'm using CE, don't think you need the paid one for that
VS Code with Pylance (which uses the Pyright type-checker) is really good. It even works well with Numpy and Pandas, which AFAIK mypy doesn't.
> When decorated by beartype, every call to the same function checks only a single random integer contained in a single random nested list contained in a single random nested list contained in that parent list. This is what we mean by the quaint phrase "one-way random walk over the expected data structure."
so it's not actually faster, it just doesn't actually check the types or even the complete shape of the data structure. seems like it would catch obvious problems, but make more subtle ones way harder to find as it creates a sense that "oh, the types are checked" when they aren't really.