Box: Python dictionaries with recursive dot notation access(github.com)
github.com
Box: Python dictionaries with recursive dot notation access
https://github.com/cdgriffith/Box
123 comments
There are quite a few nay-votes here, so I'll give a yay vote.
In the past I have had to deal with very complex configuration systems; several thousands of lines of XML which were supposed to direct many applications.
And writing:
The other case where I have found this applicable is when I have complicated JSON structures in the form of user input. Typically I augment this style by using a JSONSchema that ensures attributes do exist and have defaults (or None).
In the past I have had to deal with very complex configuration systems; several thousands of lines of XML which were supposed to direct many applications.
And writing:
value['this']['that']['the_other'] value.this.that.the_other
The other case where I have found this applicable is when I have complicated JSON structures in the form of user input. Typically I augment this style by using a JSONSchema that ensures attributes do exist and have defaults (or None).
One option would be:
value['this.that.the_other']
I see the value in the latter for code legibility. My gripe with it is, when I'm new to a project that uses a dict implementation like that: how do I know what happens if for example `that` is missing? Does it raise AttributeError, KeyError or just return None?
Personally I'd prefer a helper functions like
EDIT: To clarify a bit. An experienced Python developer should immediately recognize a call like `deep_get(value,'this.that.the_other')` as something project/framework specific and not built-in, while `value.this.that.the_other` is ambiguous.
Personally I'd prefer a helper functions like
deep_get(dict_, dotted_path[, default]) -> value
EDIT: To clarify a bit. An experienced Python developer should immediately recognize a call like `deep_get(value,'this.that.the_other')` as something project/framework specific and not built-in, while `value.this.that.the_other` is ambiguous.
You can get part of the way there with operator.attrgetter:
NT = collections.namedtuple('NT', ['x', 'y'])
nt1 = NT(1, 2)
nt2 = NT(nt1, 'asdf')
assert nt2.x.y == 2
get_two = operator.attrgetter('x.y')
assert get_two(nt2) == 2
If you wrap the operator.attrgetter in a try/catch, you can force a default too: https://pastebin.com/bFxGr22E
NT = collections.namedtuple('NT', ['x', 'y'])
nt1 = NT(1, 2)
nt2 = NT(nt1, 'asdf')
assert nt2.x.y == 2
get_two = operator.attrgetter('x.y')
assert get_two(nt2) == 2
If you wrap the operator.attrgetter in a try/catch, you can force a default too: https://pastebin.com/bFxGr22E
I never thought much about best implementation because I only used it i think once in ten years. But afaik I used something like this, that does the job and is easy to read
_empty = object()
def deep_get(dct, dotted_path, default=_empty):
for key in dotted_path.split('.'):
try:
dct = dct[key]
except KeyError:
if default is _empty:
raise
return default
return dctYou can use `get` from the `toolz` package:
toolz.get(value, ['this', 'that', 'the_other']
Personally I try to implement it as an AttributeError, it seems the most intuitive. By the way, I do understand your point and it's pretty valid in a lot of cases.
What do you typically use for enforcing a schema? I've been using Good[0], however since it is meant for general data structures, users don't get line/column printouts for invalid data.
[0] https://pypi.python.org/pypi/good/0.0.7-0
[0] https://pypi.python.org/pypi/good/0.0.7-0
I like jsonschema [1][2], primarily because it's a cross-language standard. Though, I have never used `Good` - it seems to be pretty nice if you're Python focused.
[1] https://pypi.python.org/pypi/jsonschema
[2] http://json-schema.org/
[1] https://pypi.python.org/pypi/jsonschema
[2] http://json-schema.org/
Give `destructure` a try https://github.com/selik/destructure
Better would be to code depth-first-search and breadth-first-search functions and use those on plain nested dictionaries. That's what XML did via XPATH.
If you use something like this, future maintainers are going to hate you.
Agreed. This is not the sort of code you want to be dealing with in a few years time.
Also, I seldom find that string based dicts are what I need. One of the best things about Python is being able to use virtually anything as a key.
Also, I seldom find that string based dicts are what I need. One of the best things about Python is being able to use virtually anything as a key.
> One of the best things about Python is being able to use virtually anything as a key.
I find that to be the thing one of the worst thing ever made into a programming language.
You are facing a program/API/library and you can have absolutely no clue of what it will give or expect without extensive reverse engineering.
I find that to be the thing one of the worst thing ever made into a programming language.
You are facing a program/API/library and you can have absolutely no clue of what it will give or expect without extensive reverse engineering.
> You are facing a program/API/library and you can have absolutely no clue of what it will give or expect without extensive reverse engineering.
Or reading the docs.
Or reading the docs.
I find that's more a problem of bad design and bad names, rather than flexible types.
Wait, you choose what to put in the dict. I like being able to map from, say, db objects to internal representations. Why wouldn't you want that?!
The concern is facing something that expects a map, and not knowing how to correctly populate it.
Time has taught me to prefer lists / flat structures as arguments to functions (and in most other storage and transfer contexts). Besides, either you're passing in entities that conform to the api, or you're not. Nothing about arbitrary keys in dicts changes that.
Yep. Or keys with dots in their names, or keys actually named something like "__eq__" (strange but not impossible).
The worst thing about abstractions is when they break. I can always figure out how to fix or extend ugly but straightforward code. I may not even know where the manual is when a fancy wrapper dumps a trace.
The worst thing about abstractions is when they break. I can always figure out how to fix or extend ugly but straightforward code. I may not even know where the manual is when a fancy wrapper dumps a trace.
>I may not even know where the manual is when a fancy wrapper dumps a trace.
I read somewhere that if it is written in self-documenting code, the problem and its solution will be obvious.
I read somewhere that if it is written in self-documenting code, the problem and its solution will be obvious.
I ended up making a custom data structure and query language around this kind of notation. Not in Python(am working in Haxe) but easily generalized. The structure is either array or string map or ordered string map with a dynamic data field and optional string tags. So it maps very well onto JSON or XML style data, but comes with the benefit of a very unambiguous way to do queries, where I can mix together addressing styles(take this numbered index OR take the named key) and pull elements into an array without much difficulty. The lookup syntax is "@<key>#<index>..." and each key or index entry is parsed into an ADT. I can pass along partially constructed queries as the relevant data to be processed. It's not quite as flexible as SQL since the data has to come in tree form, but it creates a major design affordance: the same query can be passed to a custom algorithm or custom data structure, with less ambiguity than a simple string lookup.
To some extent this retreads ground covered by e.g. XPath, but the specific structure puts it closer to my day-to-day needs.
To some extent this retreads ground covered by e.g. XPath, but the specific structure puts it closer to my day-to-day needs.
Yep. Last time I tried one of these I had a dict that had keys that conflicted with the dictionary keywords, like "items" or "keys" or "values", so I wound up functions back instead of data.
Author here: I totally agree with the fact it has shortcomings and definitely isn't aimed to replace everyone's dict everywhere.
I actually created the first iteration of this around 3 years ago, aimed primarily for sysadmins and others who use the console more than actual developing. As I find it extremely convenient to tab complete keys and a lot less typing.
From there it has obviously evolved to try and fit more peoples needs, and I use it regularly in my projects.
I actually created the first iteration of this around 3 years ago, aimed primarily for sysadmins and others who use the console more than actual developing. As I find it extremely convenient to tab complete keys and a lot less typing.
From there it has obviously evolved to try and fit more peoples needs, and I use it regularly in my projects.
Also dot means attribute means less speed.
If you have to worry about that, you've taken many wrong turns in the past.
That's a huge micro-optimization. For a trivial example of a class having the attributes "a", "b" and "c", and a dict of the same,
In [4]: %timeit d['b'] # dict
10000000 loops, best of 3: 42.7 ns per loop
In [6]: %timeit f.b # class using __slots__
The slowest run took 30.10 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 44 ns per loop
In [9]: %timeit b.b # class using __dict__
10000000 loops, best of 3: 48.1 ns per loop
[deleted]
It used to irritate me having to use alternative grammar for accessing values in a mapping vs accessing object attributes. However this is an area where I learned to appreciate Python's hardline on consistency. Guido in particular has been responsible for keeping the language predictable and concise by disallowing patterns like this in the language itself. So on one hand it's beautiful that it's so easy to implement things like this (which I am guilty of doing too), but on the other it is a perversion of the intentional distinction between attributes and mappings.
I've since learned to love the distinction between object attributes and items in a mapping and happily implement ['this style'] accessors without complaint now.
That being said, if the library just did difflib.get_close_matches() on the key lookup that would be neat for some use cases where you want fuzzy keep lookup.
I've since learned to love the distinction between object attributes and items in a mapping and happily implement ['this style'] accessors without complaint now.
That being said, if the library just did difflib.get_close_matches() on the key lookup that would be neat for some use cases where you want fuzzy keep lookup.
Honestly Python's hardline on consistency is very nearly my favorite thing about the language.
This feels a lot like Bunch, and a lot of private reimplementations I've seen elsewhere.
I dislike these structures. Yes, Python lets you pull this trick. Everything, yes, can be represented as a dict, a list, or one of the primitive str/int/etc. types. But I find it's a lot clearer in the long run (and even in the short run), if you have a collection of heterogenous attributes, to define a class for them. Leave dicts (and the subscript notation) for homogenous collections of k/v pairs.
A class gives you the benefit of a type: you get a name, so you can recognize this bag-of-attributes from other, different bags-of-attributes, because it's been given a name. A class also — usually — gets you a list of attributes, and hopefully documentation about what those attribute's types and expected values are.
If you don't like typing (on a keyboard), the attrs package makes things easier[1]; it has the advantage, however, that you get a real class at the end.
The only place I've seen something like Box or Bunch work well is in config files, and even then, only at the uppermost layers of the config (some of the leaves, esp. when you start having a "list of X" in a config — X needs a type).
Python lets you do magic, but with great power and all. IMO, but this is one of those times: Explicit is better than implicit. Simple is better than complex.
[1]: https://pypi.python.org/pypi/attrs/16.0.0
I dislike these structures. Yes, Python lets you pull this trick. Everything, yes, can be represented as a dict, a list, or one of the primitive str/int/etc. types. But I find it's a lot clearer in the long run (and even in the short run), if you have a collection of heterogenous attributes, to define a class for them. Leave dicts (and the subscript notation) for homogenous collections of k/v pairs.
A class gives you the benefit of a type: you get a name, so you can recognize this bag-of-attributes from other, different bags-of-attributes, because it's been given a name. A class also — usually — gets you a list of attributes, and hopefully documentation about what those attribute's types and expected values are.
If you don't like typing (on a keyboard), the attrs package makes things easier[1]; it has the advantage, however, that you get a real class at the end.
The only place I've seen something like Box or Bunch work well is in config files, and even then, only at the uppermost layers of the config (some of the leaves, esp. when you start having a "list of X" in a config — X needs a type).
Python lets you do magic, but with great power and all. IMO, but this is one of those times: Explicit is better than implicit. Simple is better than complex.
[1]: https://pypi.python.org/pypi/attrs/16.0.0
thanks for the rec on attrs-- cool stuff!
I've worked on codebases that use a similar thing, and I have yet to see one where this wasn't a mistake.
Most of the time, this probably gives you exactly what you want, but then there are times where you discover bugs in production because data you assumed is your custom class is a plain old dict, and now you're raising AttributeError all over the place. Another wart is if you are unfortunate enough to have keys that match the name of one of dict's methods, then you have to resort to instance['items'], which defeats the purpose of using this in the first place.
This is a fun trick, but if someone one my team tried to introduce this, it won't make it through code review.
Most of the time, this probably gives you exactly what you want, but then there are times where you discover bugs in production because data you assumed is your custom class is a plain old dict, and now you're raising AttributeError all over the place. Another wart is if you are unfortunate enough to have keys that match the name of one of dict's methods, then you have to resort to instance['items'], which defeats the purpose of using this in the first place.
This is a fun trick, but if someone one my team tried to introduce this, it won't make it through code review.
I agree, but when I saw this, my first assumption was that it was for personal projects primarily. I very rarely see libraries that change syntax like this in large scale or long term codebases.
I agree, I had a legacy code base with this as well. It was a giant pain both in prod as you say but also for green developers. Really slowed down development.
For those who just want a simple a.b instead of a['b'], use this:
class Obj():
def __init__(self, d):
self.__dict__ = d
d = Obj({
'a': 1,
'b': 2,
})
print(d.a)These days (Python >= 3.3) SimpleNamespace is a more concise way of doing this:
from types import SimpleNamespace
d = SimpleNamespace(a=1, b=2)
print(d.a)Haha neat, that means this works very cleanly as a conversion mechanism:
my_dict = {
'a': 1,
'b': 2
}
d = SimpleNamespace(**my_dict)
print(d.a)I am guilty of writing classes which are really simple, but only writing a class for the sake of consistency across my codebase.
Although there is this PyCon talk from 2012 advocating not to write classes if there is only one or two methods. [1]
[1]: https://news.ycombinator.com/item?id=3717715
Although there is this PyCon talk from 2012 advocating not to write classes if there is only one or two methods. [1]
[1]: https://news.ycombinator.com/item?id=3717715
Seems like a lot of work and magic when compared to, say:
class Box(UserDict):
def __getattr__(self, key):
return self.__getitem__(key)
def __setattr__(self, key, value):
return self.__setitem__(key, value)
> Box(UserDict):
There's no reason to use UserDict, just extend `dict` directly, or implement `MutableMapping` instead. UserDict hasn't been useful since the types/class unification back in… Python 2.3 I think?
There's no reason to use UserDict, just extend `dict` directly, or implement `MutableMapping` instead. UserDict hasn't been useful since the types/class unification back in… Python 2.3 I think?
In my experience, this always seems like a cool idea until you use it with keys that an end-user can define. Then you get to watch your code blow up when they override some other function you've defined on the class.
My thoughts exactly. Wait until someone adds a '__call__', '__iter__', '__del__' key...
So Django Templates also use dot notation lookups for dict, lists, and objects[0]
EDIT: This came up because our JSON commonly uses `items` as a key for a list of items, which I expect to be at `a_dict['items']`, and it has nothing to do with python's `a_dict.items`.
Dictionary lookup, attribute lookup and list-index lookups are implemented with a dot notation:
{{ my_dict.key }}
{{ my_object.attribute }}
{{ my_list.0 }}
If a variable resolves to a callable, the
template system will call it with no
arguments and use its result instead of the callable.
In [17]: a = {"a": 1, "items": {"b": {"c": {}}}}
...: a_box = Box(a)
...: a_box
...:
Out[17]: <Box: {'a': 1, 'items': {'b': {'c': {}}}}>
In [18]: a_box
Out[18]: <Box: {'a': 1, 'items': {'b': {'c': {}}}}>
In [19]: a_box.items
Out[19]: <function items>
In [20]: a_box.a
Out[20]: 1
EDIT: This came up because our JSON commonly uses `items` as a key for a list of items, which I expect to be at `a_dict['items']`, and it has nothing to do with python's `a_dict.items`.
[deleted]
I like using NamedTuple for dot notation access, it's a good middle ground between dicts and custom classes.
Pandas must use something similar under the hood to provide dot notation access to columns. I wish h5py did the same for hdf5 objects. In py3, I find myself needing to type list(X.items()) and then list(X['Y'].items()) and so on when I'm exploring a new dataset... fairly awkward for interactive use.
Pandas must use something similar under the hood to provide dot notation access to columns. I wish h5py did the same for hdf5 objects. In py3, I find myself needing to type list(X.items()) and then list(X['Y'].items()) and so on when I'm exploring a new dataset... fairly awkward for interactive use.
Thumbs up for namedtuple. It is a good middle ground.
This kind of thing has been implemented a lot of times by a lot of people. It starts with someone who just thinks that addressing deeply nested dicts is ugly, and ends with people trying to tack on all kinds of features to justify their modification of the basic grammar.
Probably not a huge issue if it's an isolated use-case. Not something I would want to see everywhere in my codebase.
Probably not a huge issue if it's an isolated use-case. Not something I would want to see everywhere in my codebase.
What happens if the dictionary has the keys 'John Doe' and 'John_Doe'?
Can you use 'self' as a key?
Ive used a class to provide this kind of dot notation fererencing of hierarchical data. If you could enclose the keys in quotes I might feel better about it but that's probably not possible.
Can you use 'self' as a key?
Ive used a class to provide this kind of dot notation fererencing of hierarchical data. If you could enclose the keys in quotes I might feel better about it but that's probably not possible.
>Can you use 'self' as a key?
Without looking at the code: yes. `self` in python is just convention.
Without looking at the code: yes. `self` in python is just convention.
How is this different from addict?
https://github.com/mewwts/addict
https://github.com/mewwts/addict
I wonder if addict convert keys like "Funky Foobar Key!" into Funky_Foobar_Key? The README seems to use keys that cleanly convert to python dict keys, so it's not clear.
I got excited by the heading thinking that there was a change to the language and we could refer to the dictionary itself in dictionary comprehension.
Just yesterday I commented that if generators could refer to themselves in generator comprehension (recursion) I could express some algorithm as a single expression.
http://stackoverflow.com/a/41617394/180464
... anyway, this just turns out to be what looks like an "addict" clone.
Just yesterday I commented that if generators could refer to themselves in generator comprehension (recursion) I could express some algorithm as a single expression.
http://stackoverflow.com/a/41617394/180464
... anyway, this just turns out to be what looks like an "addict" clone.
Something like that would be handy. Just to clarify the addict mention.
It works similarity to addict, but does have important distinctions and IMO seniority (was in my 'reusables' pypi project named 'Namespace' before addict existed. Just finally spun it into it's own project).
Biggest differences:
* Box will convert items added to the object after creation, addict does not
* addict only acts as a defaultdict, Box can act as either regular or defaultdict
* Box updates it’s __dir__ so that attributes (keys) are tab completed in stuff like IPython
* Box repr clearly shows it is an object
It works similarity to addict, but does have important distinctions and IMO seniority (was in my 'reusables' pypi project named 'Namespace' before addict existed. Just finally spun it into it's own project).
Biggest differences:
* Box will convert items added to the object after creation, addict does not
* addict only acts as a defaultdict, Box can act as either regular or defaultdict
* Box updates it’s __dir__ so that attributes (keys) are tab completed in stuff like IPython
* Box repr clearly shows it is an object
A single class that:
* De/serializes JSON and YAML
* De/mangles, de/encodes keys
* Provides automatic, expensive hashcode
* Blacklists/transforms a bunch of likely keys because they conflict with reserved words
* Overlays attrs (__box_heritage)
* All in pretty complex code that obfuscates what you're really doing (especially to a maintainer)
For the ability to avoid importing json/PyYAML and use clear key lookups? The author must really like JavaScript syntax, or something, because I'm not seeing the point in this layer at all. It's the sort of library that you discover your inherited codebase is using, and just say, "fuckfuckfuck...", because it implies that the author cares more about pushing round pegs into square holes than writing the freaking application code; it reeks of inexperience.
* De/serializes JSON and YAML
* De/mangles, de/encodes keys
* Provides automatic, expensive hashcode
* Blacklists/transforms a bunch of likely keys because they conflict with reserved words
* Overlays attrs (__box_heritage)
* All in pretty complex code that obfuscates what you're really doing (especially to a maintainer)
For the ability to avoid importing json/PyYAML and use clear key lookups? The author must really like JavaScript syntax, or something, because I'm not seeing the point in this layer at all. It's the sort of library that you discover your inherited codebase is using, and just say, "fuckfuckfuck...", because it implies that the author cares more about pushing round pegs into square holes than writing the freaking application code; it reeks of inexperience.
I get the itch its scratching. Working with REST services in particular can become very tedious, real fast, without some sort of de/serialization beyond json -> dict.
Many python REST API client libraries end up rolling their own one-off dict -> "dot dict" transformation helpers, for better or worse.
This sort of thing can be great for interactive use-cases, where terseness really helps (notebooks, REPL's etc). I'd definitely use something like this, there. But I would share the same reservations as you in using this library in a serious project.
Many python REST API client libraries end up rolling their own one-off dict -> "dot dict" transformation helpers, for better or worse.
This sort of thing can be great for interactive use-cases, where terseness really helps (notebooks, REPL's etc). I'd definitely use something like this, there. But I would share the same reservations as you in using this library in a serious project.
Why so negative here?
I like Python a lot, and I don't write much Javascript, but one thing I wish I could do in Python is the dot notation from a dictionary. I sometimes used namedtuple as a cheap (but "immutable") class, so I can simply use dot notation when I am passing my object around my functions, instead of always stuffing the data into a dictionary.
I like Python a lot, and I don't write much Javascript, but one thing I wish I could do in Python is the dot notation from a dictionary. I sometimes used namedtuple as a cheap (but "immutable") class, so I can simply use dot notation when I am passing my object around my functions, instead of always stuffing the data into a dictionary.
[deleted]
"I like Python a lot, and I don't write much Javascript, but one thing I wish I could do in Python is the dot notation from a dictionary"
Why though?
Why though?
because of duck typing.
a['b'] = 'c' # means i must use a dict a.b = 'c' # means must be an obj
if they were interchangeable then I could have more reusable code.
a['b'] = 'c' # means i must use a dict a.b = 'c' # means must be an obj
if they were interchangeable then I could have more reusable code.
Because it saves me a lot of chars.
foo['name']['attr1']['attr2']['morefuckingattr'] vs foo.name.atr1.attr2.morefuckingattr
More of a personal preference.
foo['name']['attr1']['attr2']['morefuckingattr'] vs foo.name.atr1.attr2.morefuckingattr
More of a personal preference.
That's a tough preference with Python, because of the semantics of what [] and . mean and how ingrained those semantics are in the language. Put another way, the following should hold:
One way I've approached your problem before is a recursive helper where I borrowed some concepts from jq:
x.y == x.__getattr__("y")
x["y"] == x.__getitem__("y")
assignment == set{attr,item}
del x.y, del x["y"], etc
len() and slices of items, not attrs
One way I've approached your problem before is a recursive helper where I borrowed some concepts from jq:
val = fetch(result_dict, "foo.bar.baz.plonk[0]")
Saving chars is often a poor reason to design a language feature.
Nope. It's called developer ergonomics.
The annoyance here is a feature not a bug. Deeply nested dictionaries, in my experience, are a source of infinite KeyErrors. The problem with using a dictionary as a cheap object is that it's very easy for it to be an object you weren't expecting. Don't nest dictionaries in Python!
So then what are you supposed to deserialize nested data structures like JSON into? Dictionaries are the obvious "right" way, Python's annoying tooling aside.
I'd say that dictionaries are only the obvious "right" way if you're not planning on accessing any of their fields. Otherwise you have some set of expected fields, in which case you can define a namedtuple and avoid a vast array of nasty surprises. It's really easy to initialize a namedtuple from a dict (use ), as long as the dict has the right fields. If it doesn't, and you're planning on using its fields, the fact that this raises an exception is a good thing.
What do you think of this library: https://github.com/selik/destructure
It'll enforce schema and destructure the nested dict/list that results from JSON decoding.
It'll enforce schema and destructure the nested dict/list that results from JSON decoding.
That's like asking, "What data structure should I use for this CSV file?"
Depends on the usage. Sometimes you leave them as nested dictionaries and lists. Sometimes you transform them into tabular data. Sometimes you aggregate, etc.
Depends on the usage. Sometimes you leave them as nested dictionaries and lists. Sometimes you transform them into tabular data. Sometimes you aggregate, etc.
Developer ergonomics does not (always) mean "preventing effort".
If that were true, there would be no type systems.
If that were true, there would be no type systems.
Part of ergonomics is simplicity about what the thing you're looking at actually is.
No, it's called being terrible at your job. If you need to add complexity to your project just to save you some typing, you have no concept of what you're doing. It's like an accountant not understanding math.
Complexity can hide in unexpected places and sometimes a dash of complexity in one place can make something in another place much simpler.
I think this library has it's uses. I wouldn't use it uncritically but I wouldn't dismiss it either.
I think this library has it's uses. I wouldn't use it uncritically but I wouldn't dismiss it either.
True, but it saves a relatively small number of chars (~12) and does so at the same time as:
A) Giving off a signal that you're dealing with an object rather than a dict.
B) Making it cumbersome to swap out some of those selectors with variables.
C) Making it difficult to deal with the attribute not being there (with a dict you can say .get("attr2", {}) and it returns a default.
A) Giving off a signal that you're dealing with an object rather than a dict.
B) Making it cumbersome to swap out some of those selectors with variables.
C) Making it difficult to deal with the attribute not being there (with a dict you can say .get("attr2", {}) and it returns a default.
I understand. I have another motivation which is getting JSON and wanting a OOP object to mainpluate with. Take AWS' boto3 response. Well documented but the nested response structure gives me a chill and I wish there's a direct object out of the JSON. I instead had to write my own class for the conversion. If I could mainpluate dict like mainpluatig attribute in a class then I could just write my function's contract "input is a response object from X APi" instead of "dict of this form."
Quite annoying and I know some people considered what I wish is a bad practice.
Quite annoying and I know some people considered what I wish is a bad practice.
[deleted]
Personally, I only use hard-coded strings like that for shallow nesting. If I need to index 4 layers deep, something down there is going to be turned into an object, otherwise enumerating all the possible keys to ensure the code is correct becomes a nightmare. And once you do that, you can use dot notation anyway.
You're using named tuple correctly, but why would you want dot notation on a dictionary? There's a reason square bracket notation is used to access dict items. Because it's an item of a dictionary, not an attribute.
Nested namedtuples are way, way better than the proposed approach. I've found that nested dictionaries in Python are fragile in two dimensions: not only do you have no guarantee that the values in your dictionaries are appropriately typed, you have no guarantee that the keys even exist! The whole thing just collapses in a messy heap if you look at it sideways. Furthermore, it's very hard to check to see whether anything has been changed deep within your crazy nested dictionary structure. With nested namedtuples, immutability lets you use "is" on the root to see whether anything has changed deep inside. Sure, you can't say foo.bar.baz.quux = 5, but did you really want to do that, knowing how hard it would be to use that information?
Check out attrs. You can use frozen=True to get the immutable behavior as well, but there are many other benefits over namedtuple.
https://attrs.readthedocs.io/en/stable/index.html
https://attrs.readthedocs.io/en/stable/index.html
>It's the sort of library that you discover your inherited codebase is using, and just say, "fuckfuckfuck...", because it implies that the author cares more about pushing round pegs into square holes than writing the freaking application code
Ugh, this really hits home.
I just started a new day-job where I'm essentially being asked to un-fuck a python codebase. Your statement neatly captures my feelings when I discovered the presence of hacked-together singleton classes littered throughout the thing.
Good luck unit-testing that...
Ugh, this really hits home.
I just started a new day-job where I'm essentially being asked to un-fuck a python codebase. Your statement neatly captures my feelings when I discovered the presence of hacked-together singleton classes littered throughout the thing.
Good luck unit-testing that...
I think the python implementation of jsonpath is more appropriate for doing this sort of untyped access. If clean dots are really preferred, make classes! :)
I think if 'Box' was broken down into smaller 'ZipLockBags', it would be better; it is doing too much things.
Agreed, I have no idea how this is easier. If anything it makes it impossible to write dynamic code. This is a perfect example of a library that (theoretically) helps the programmer at the expensive of more inefficient code.
> For the ability to avoid importing json/PyYAML and use clear key lookups?
I think the advantage over json/PyYAML is not having to supply conversion functions for Decimal, datetime, or user defined datatypes.
I think the advantage over json/PyYAML is not having to supply conversion functions for Decimal, datetime, or user defined datatypes.
I agree that that's useful, but it should be a separate library, and I'd be surprised if there weren't a few already (I'm guilty of writing one-off serializers for datetimes/dates).
Though many reasonable rebuttals have been mentioned, the one case in which I find this pattern indispensable is when creating mock-instances for a variety of classes that need to be interleaved amongst genuine instances (say, ORM objects, if you're unlucky enough to have those in your life) so that code operating on collections of the latter doesn't get littered with special cases for the former.
This looks really nice. I might be reading the DefaultBox docs wrong, but does it support this:
I find the bulk of my dict code is checking for keys before access, or using .get('', default) recursively. Gets really hairy for deeply nested dicts.
box.might_exist.might_exist.might_exist.desired_key
I find the bulk of my dict code is checking for keys before access, or using .get('', default) recursively. Gets really hairy for deeply nested dicts.
I also found myself doing that pretty often, so I wrote this:
def nget(d, *ks, **kwargs):
for k in ks:
d = d.get(k)
if d is None:
return kwargs.get('default')
return d
>>> d = {'a': {'b': {'c': 12 }}}
>>> nget(d, 'a', 'd', 'c', default='Not Found!')
'Not Found!'
>>> nget(d, 'a', 'b', 'c')
12Your implementation suffer from a bug, if the final item is None it will return the default instead of None.
A better implementation would be:
A better implementation would be:
def rget(d, *ks, **kwargs):
for k in ks:
if k not in d:
return kwargs.get('default')
d = d[k]
return d> Gets really hairy for deeply nested dicts.
Write a function. `def my_deep_get(dict, keys)`
Write a function. `def my_deep_get(dict, keys)`
How is the performance on this for very large dictionaries vs. standard python dictionary lookups?
Just finished my article that includes speed comparisons. http://www.codecalamity.com/python-box-3-the-reddit-release/...
Basically it's faster on standard soft creation as it converts on lookup. (And most use cases don't call for referencing every single key)
Basically it's faster on standard soft creation as it converts on lookup. (And most use cases don't call for referencing every single key)
A really good question. I haven't looked much at the code but I'd imagine still O(1) because it's just converting the keys into a different format.
But that says nothing about whether there is a constant penalty being applied.
certainly
Another similar implementation I've used is: http://www.stat.washington.edu/~hoytak/code/treedict/
If you want readonly access using dot notation instead of keys then I have a simple gist [1] I use that works well for tracking nested dicts - even nested inside lists. It's a good way to pass a dict into a function expecting an object in some scenarios.
[1] https://gist.github.com/mmerickel/ff4c6faf867d72c1f19c
[1] https://gist.github.com/mmerickel/ff4c6faf867d72c1f19c
Neat. I've bem using a simpler approach for a few years now:
https://github.com/rcarmo/python-utils/blob/master/core.py#L...
...but this takes that notion much further.
https://github.com/rcarmo/python-utils/blob/master/core.py#L...
...but this takes that notion much further.
Checkout addict
https://github.com/mewwts/addict
https://github.com/mewwts/addict
I wrote a similar library that is more restricted and "obvious" in what it does:
https://github.com/skorokithakis/jsane/
https://github.com/skorokithakis/jsane/
How is that different from easydict?
- https://pypi.python.org/pypi/easydict/
First things off top of head: Box does recursion through lists, new dicts (and lists of dicts) added will also be dot notation without manually converting, Box can be a recursive default dict, Box will allow keys with spaces or other issues to be accessible as attributes, and can be frozen, and has YAML and JSON functions built-in.
If all you want is the dot and indexing syntactic sugar:
class Box(dict):
def __init__(self, **kwargs)
super(Box, self).__init__(kwargs)
self.__dict__.update(kwargs)Cute, though the use of x to prepend to integer keys is potentially confusing with the convention of x as a hexadecimal indicator.
That is a very good point, never thought of that. I will have to add a feature to make that configurable. Thanks for the heads up!
I never understood why Python differentiates between key access and attribute access. Dig down one layer, and they're the same thing anyway: a.b is just a.__dict__['b'].
For all its faults, JavaScript has some pretty sensible defaults. Accessing a missing key returns what is basically its equivalent of nil. Extra parameters to function calls are simply ignored. Unsupplied parameters default to nil.
For all its faults, JavaScript has some pretty sensible defaults. Accessing a missing key returns what is basically its equivalent of nil. Extra parameters to function calls are simply ignored. Unsupplied parameters default to nil.
I can think of a couple of reasons:
safety, attr access by .<thing> means you knew what you wanted at write-time, so it's reasonable to assume that you're gonna want to know if it doesn't exist, and not just want to roll the dice on passing a null along.
language concision, it seems counterintuitive but attrs vs items gives you two options for one class: . access and [] access, which it's very reasonable give different results, so your default mode of writing classes has a "I think this should be here/nondynamic" mode of access and a 'wonder if this is here'/dynamic mode of access. I think that there are benefits to that. If you don't eventually do some dynamic attribute access I think you throw away an advantage of dynamic languages, but it's nice to me that there's this by default soft line between less and more dynamic code
Useful. My only concern would be performance.
no thanks.
[deleted]
This thread reminds me why I don't use Python anymore
I think the real reason not to do this is one that as of this writing still hasn't been mentioned yet, which is that when using dot notation, your key names have to fit the Python grammar for identifiers, so you can't have a key "the thing" with a space, or anything else that isn't a Python identifier. So you can't just say "I can access this dictionary with dots", it has to be "I can access some keys of this dictionary with dots, but others I have to access another way". This greatly, greatly reduces its utility. There's a variety of ways of trying to address this; this package appears to try to normalize key names, which is very prone to surprising behaviors and makes it difficult to reason about what will go into what bucket, and also seriously mitigates the virtue of this entire approach because you, the programmer, must also run the normalization algorithm yourself in order to use the dot notation, which rapidly eats away the gains of typing a dot instead of two brackets and two quote characters. Rewriting keys like that is really icky.
The other problem is that the "dot" namespace, as it were, is as others have noted here used for method and property resolution, so you also end up with "dictionaries with random values they can't really contain because they're being used as method names" or "dictionaries that if you put the wrong key in them override a method" or something else like that. Also, consider the ability to subclass these things, making many of the things you might think to do to hack around this break in subclasses.
It's very superficially tempting but it has a looooot of issues that become evident over time. This isn't even a complete list.