So, my spouse was a CPU designer at AMD for many years and now does secure computing work for, well, the US government. I showed her your comment. She laughed. A lot.
To be clear, our experience using gcsfuse and friends to do basically the same things was extremely painful and a performance nightmare. The HDF format was designed for a world where seeks are free which makes cloud access very high latency and very low throughput.
Ah, thanks for these! But I see nothing has changed.
* pyfive is interesting but immature and doesn't seem to have any cloud bucket support
* h5s3 is an abandoned experiment that hasn't been touched in two years
* h5py is fine but again, no cloud support
* kita is a commercial offering from the HDF Group and -- I cannot stress this enough -- these people are shockingly incompetent; plus when I last looked at their system architecture diagram I thought it was a joke (well, I thought it was an intentional joke)
Efficient access to scientific datasets hosted on S3/GCP is a full blown crisis in the scientific computing community. People aren't switching to zarr for the fun of it, but because zarr is here, today, and isn't a joke, and is actually open.
No, you can certainly write in parallel, despite the GIL. The GIL makes this inefficient if your work is CPU-bound, but for IO-bound workloads it can be fine.
Otherwise, HDF5 offers every single advantage that zarray has and is much more mature, stable, better documented, and has better support.
Absolutely not. HDF5 is an awful format with terrible implementations. For example, try writing a python program with multiple threads where each thread writes to a different HDF5 file. This should just work -- there's no concurrent access. And yet it doesn't because HDF5 implementations are piles of ancient C code that use lots of global state. There's no technical reason for this; one could easily store all the state needed in a per-file object. But back in the day, software eng standards were lower (especially for scientists) and HDF5 changes at a glacial place.
I've been bitten by this particular bug, but you really have to wonder: given how poorly it speaks to the software engineering behind HDF5 implementations, what else is broken in the code or specifications?
If you're working in a situation where it makes sense to have things on disk or some sort of NFS share, use HDF5. If you're working with objects in a cloud bucket, you'll incur additional overhead with HDF5, as you'll have to read its table of indices, then make range requests to each chunk. Zarr is optimized for the cloud use case.
When last I looked, there were no open source HDF5 implementations that were smart enough to do range requests to cloud hosted hdf files. Has this changed?
What's the plan for when grandma develops dementia? The paranoia will make her unwilling to part with her beloved firearm...do we just wait until she murders a family member or home health aid?
That seems like a much more important question than bickering over precisely which firearm grandma should have as she slides into dementia.
The idea that children should spend their days staring at screens and never interacting with other human beings seems really sick. But oh so on brand for the Valley.
I worked at FB briefly so maybe I can explain. FB has a corporate culture that really discourages critique. When things are broken, especially internal things, people look at you funny if you speak up about it. A big part of that is that quarterly bonuses are given for "making an impact" and your group's status (and part of your bonus) is based on delivering a consistent set of "impacts" over time. So it is better for your comp to do things badly really fast since you get (1) did something super fast! and (2) get to record a big impact a few months later when you fix the obvious brokenness.
Pretty quickly, people learn to keep their mouth shut.
Also, many, many FB engineers are early-career folk who are fresh out of school. More senior folk are few and far between and are even more strongly incentivized to keep their mouth shut, because their bonuses are bigger.
The way that white supremacy works is that black people are penalized even in the small enclaves where they constitute a local majority. Note that the laws that punish crack (used by lower income black people) 100 times more harshly than powder cocaine [0] are federal in nature.
> When it comes to exceptions, it's generally true on x64 that you don't pay for what you don't use (there's no performance penalty to exception handling if you don't throw) although that hasn't always been true for all platforms and implementations. It's also generally true that you couldn't implement that kind of non local flow control more efficiently yourself, although the value of that guarantee is a little questionable with exception handling.
I'm inclined to agree with you but just about everyone at Google says the opposite and most C++ shops I've seen agree with them. I've made this argument and lost repeatedly. So, it seems like the community can't even agree on which abstractions are zero cost (or maybe whether some zero cost abstractions are actually zero cost?). To the extent that the community itself has no consensus about these things, maybe they're not a marketing slogan that's helpful to use.
(2) Even reducing it to runtime costs, it seems a bit nonsensical. Are C++ exceptions a zero cost abstraction? All the googlers I argued with about them would insist that they have unacceptably high runtime costs.
OK, but templates are surely zero (runtime) cost abstractions, right? Unless you start to worry about duplicate code blowing out your instruction cache but if that's a problem, no profiler in the world will ever be able to tell you, so I guess you'll never know just how costly the abstraction is, so you might as well continue believing it is zero...?
Zero-cost abstractions only exist in a world where you don't highly value language simplicity and comprehensibility.
Simplicity and comprehensibility were things the committee had to give up in order to pretend they had "zero-cost" abstractions. Nothing in life comes free: everything, including all abstractions, comes at some cost.
To clarify: of course they like simplicity when it costs nothing. But they consistently value other goods over simplicity.
For example: maintaining backwards compatibility. The community believes that it is more important that 20 year old C++ code run unmodified than that the language should be simplified. There's lots of stuff you could do to simplify the language but options dry up in a world where 20 year old code must be able to run unmodified.
So sure, the committee talks a lot about simplicity, but it isn't willing to sacrifice much.
Don't get me wrong: I'm glad that finally, in 2020, C++ will be almost but not quite as good as Common Lisp was at metaprogramming back in 1982. But it remains the case that eval-when and defmacro are both more powerful and dramatically simpler than anything the C++ committee has ever considered.
I think that's dependent on language culture. Because the C++ community doesn't take language simplicity or comprehensibility seriously, there are lots of C++ developers who can't use or reason about surprisingly large parts of the language. So the community has rallied around the notion that "library" developers need to understand everything and that most developers will just glue together bits that the library devs made.
I mean, how many C++ developers actually write serious template code? How many of them could reliably explain what the keywords in post do?
The idea that every developer is a library author (or the lisp extension that every developer is a language author) is common in many other language communities but it relies on the community working hard to make mastery of the language feasible for lots of people. The C++ community never bought into that notion; they inherited a very stratified class structure from Bell Labs.
That's (maybe) great for folks in China, but what about folks outside? Is the rest of the world really better off if Google can easily be bullied and manipulated by the Chinese government?
Once they start taking orders from Beijing about what results to show in China, why shouldn't they take orders about what to show in the US? After all, the Chinese government controls their access to an enormous market worth lots of money. Will we be able to find articles about the concentration camps for Muslims in China after they're operating in China?
Japan is intensely racist. My spouse lived there for a year. When she went about her business outside, small children who saw her would literally burst into tears screaming to their parents about gaijin.
I saw this up close in Boston (really Cambridge MA). In most American municipalities, neighborhood groups have enormous power to shape development. All development has to get approved by the city, and the city listens to neighborhood groups. Those groups can also petition a landmark commission to landmark a building making redevelopment much more difficult. And they can sue. Even when these tactics don't work, they're extremely time consuming. I've seen one court case drag on for years even though the people filing literally submitted briefs written in crayon. So it is often better to just settle with neighborhood groups, but like patent trolls, that only encourages them.
One issue is that in the US, we treat housing as the primary source of wealth for middle class people. So everyone is expected to be a homeowner. A world where the price of your primary wealth asset is rising is good! And a world in which its value is flat or falling is disastrous. And so local laws get shaped so as to keep housing prices high; in practice that means choking the supply of new more dense housing. After all, any change might harm the value of your home!
Another good resource from Google is [1] which focuses more on operational impacts after you deploy a system that relies on ML (I'm a coauthor). [2], which was written by my boss, is also great.
This is all completely wrong.