HackerTrans
TopNewTrendsCommentsPastAskShowJobs

lidavidm

no profile record

Submissions

Data Wants to Be Free: Fast Data Exchange with Apache Arrow

arrow.apache.org
5 points·by lidavidm·पिछला वर्ष·0 comments

comments

lidavidm
·29 दिन पहले·discuss
AIUI, Kobo devices have a more advanced rendering engine if you name the file with .kepub.epub. (I think it's based on ePub 3?) Not sure if it would fix the problem here. But I personally run ePubs through kepubify (https://pgaskin.net/kepubify/try/) before transferring them to my Kobo.
lidavidm
·4 माह पहले·discuss
[Not the overall point of the poem, but] yet for all that, it turns out chemical weapons aren't even that useful: https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch...
lidavidm
·6 माह पहले·discuss
You may like this book called The Origins of Efficiency: https://press.stripe.com/origins-of-efficiency

Or the author's newsletter: https://www.construction-physics.com/
lidavidm
·7 माह पहले·discuss
Olympus and other cameras can do this with "pixel shift": it uses the stabilization mechanism to quickly move the sensor by 1 pixel.

https://en.wikipedia.org/wiki/Pixel_shift

EDIT: Sigma also has "Foveon" sensors that do not have the filter and instead stacks multiple sensors (for different wavelengths) at each pixel.

https://en.wikipedia.org/wiki/Foveon_X3_sensor
lidavidm
·11 माह पहले·discuss
Oh that is awesome! I would really appreciate it!

Could you also get d2 into GitHub and Notion while you have it here :)
lidavidm
·11 माह पहले·discuss
I always liked D2 more than mermaid, except IMO, this makes grid layouts essentially useless: https://github.com/terrastruct/d2/issues/1164

Having to figure out the exact pixel widths defeats the point of these tools, at least for me.
lidavidm
·पिछला वर्ष·discuss
Someone on the issue they made explained why Clone is "broken": https://github.com/JelteF/derive_more/issues/490#issuecommen...

Which links to this blog post explaining the choice in more detail: https://smallcultfollowing.com/babysteps/blog/2022/04/12/imp...
lidavidm
·पिछला वर्ष·discuss
The graphics would be called chibis, IMO (or デフォルメ if you wanna be fancy) and IDK about developers, but perhaps weaboos/weebs would be the general term
lidavidm
·पिछला वर्ष·discuss
Iceberg-go is working on it! (edit: it being write support)
lidavidm
·पिछला वर्ष·discuss
It might be funny to use this as a software versioning scheme. ("What do you mean the next version after v3 is 20A?")
lidavidm
·पिछला वर्ष·discuss
At least they appear to be partnering with Kobo "later this year" [1]. I've been a big fan of Kobo's devices so this is a nice plus. (I just wish they could figure out some way to get Kindle exclusives, but well that's a contradiction in terms, so...)

[1]: https://bookshop.org/info/ebooks ("Can I read my ebooks on my Kindle, Kobo, Nook, etc.?")
lidavidm
·पिछला वर्ष·discuss
https://www.ralfj.de/blog/2019/07/14/uninit.html perhaps (the OP also talks about this when linking to a talk about jemalloc)
lidavidm
·पिछला वर्ष·discuss
To get a really precise answer you'd have to profile or benchmark. I'd say it's also hard to do an apples to apples comparison (if you only replace the data format in the wire protocol, the database probably still has to transpose the data to ingest it). And it's hard to do a benchmark in the first place since probably your database's wire protocol is not really exposed for you to do a benchmark.

You can sort of see what benefits you might get from a post like this, though: https://willayd.com/leveraging-the-adbc-driver-in-analytics-...

While we're not using Arrow on the wire here, the ADBC driver uses Postgres's binary format (which is still row oriented) + COPY and can get significant speedups compared to other Postgres drivers.

The other thing might be to consider whether you can just dump to Parquet files or something like that and bypass the database entirely (maybe using Iceberg as well).
lidavidm
·पिछला वर्ष·discuss
Arrow has several other related projects in this space:

Arrow Flight SQL defines defines a full protocol designed to support JDBC/ODBC-like APIs but using columnar, Arrow data transfer for performance (why take your data and transpose it twice?)

https://arrow.apache.org/blog/2022/02/16/introducing-arrow-f...

There's an Apache-licensed JDBC driver that talks the Flight SQL protocol (i.e. it's a driver for _any_ server that implements the protocol): https://arrow.apache.org/blog/2022/11/01/arrow-flight-sql-jd...

(There's also an ODBC driver, but at the moment it's GPL - the developers are working on upstreaming it and rewriting the GPL bits. And yes, this means that you're still transposing your data, but it turns out that transferring your data in columnar format can still be faster - see https://www.vldb.org/pvldb/vol10/p1022-muehleisen.pdf)

There's an experiment to put Flight SQL in front of PostgreSQL: https://arrow.apache.org/blog/2023/09/13/flight-sql-postgres...

There's also ADBC; where Flight SQL is a generic protocol (akin to TDS or how many projects implement the PostgreSQL wire protocol), ADBC is a generic API (akin to JDBC/ODBC in that it abstracts the protocol layer/database, but it again uses Arrow data): https://arrow.apache.org/blog/2023/01/05/introducing-arrow-a...