HackerTrans
TopNewTrendsCommentsPastAskShowJobs

lidavidm

no profile record

Submissions

Data Wants to Be Free: Fast Data Exchange with Apache Arrow

arrow.apache.org
5 points·by lidavidm·el año pasado·0 comments

comments

lidavidm
·hace 30 días·discuss
AIUI, Kobo devices have a more advanced rendering engine if you name the file with .kepub.epub. (I think it's based on ePub 3?) Not sure if it would fix the problem here. But I personally run ePubs through kepubify (https://pgaskin.net/kepubify/try/) before transferring them to my Kobo.
lidavidm
·hace 4 meses·discuss
[Not the overall point of the poem, but] yet for all that, it turns out chemical weapons aren't even that useful: https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch...
lidavidm
·hace 6 meses·discuss
You may like this book called The Origins of Efficiency: https://press.stripe.com/origins-of-efficiency

Or the author's newsletter: https://www.construction-physics.com/
lidavidm
·hace 7 meses·discuss
Olympus and other cameras can do this with "pixel shift": it uses the stabilization mechanism to quickly move the sensor by 1 pixel.

https://en.wikipedia.org/wiki/Pixel_shift

EDIT: Sigma also has "Foveon" sensors that do not have the filter and instead stacks multiple sensors (for different wavelengths) at each pixel.

https://en.wikipedia.org/wiki/Foveon_X3_sensor
lidavidm
·hace 11 meses·discuss
Oh that is awesome! I would really appreciate it!

Could you also get d2 into GitHub and Notion while you have it here :)
lidavidm
·hace 11 meses·discuss
I always liked D2 more than mermaid, except IMO, this makes grid layouts essentially useless: https://github.com/terrastruct/d2/issues/1164

Having to figure out the exact pixel widths defeats the point of these tools, at least for me.
lidavidm
·el año pasado·discuss
Someone on the issue they made explained why Clone is "broken": https://github.com/JelteF/derive_more/issues/490#issuecommen...

Which links to this blog post explaining the choice in more detail: https://smallcultfollowing.com/babysteps/blog/2022/04/12/imp...
lidavidm
·el año pasado·discuss
The graphics would be called chibis, IMO (or デフォルメ if you wanna be fancy) and IDK about developers, but perhaps weaboos/weebs would be the general term
lidavidm
·el año pasado·discuss
Iceberg-go is working on it! (edit: it being write support)
lidavidm
·el año pasado·discuss
It might be funny to use this as a software versioning scheme. ("What do you mean the next version after v3 is 20A?")
lidavidm
·el año pasado·discuss
At least they appear to be partnering with Kobo "later this year" [1]. I've been a big fan of Kobo's devices so this is a nice plus. (I just wish they could figure out some way to get Kindle exclusives, but well that's a contradiction in terms, so...)

[1]: https://bookshop.org/info/ebooks ("Can I read my ebooks on my Kindle, Kobo, Nook, etc.?")
lidavidm
·el año pasado·discuss
https://www.ralfj.de/blog/2019/07/14/uninit.html perhaps (the OP also talks about this when linking to a talk about jemalloc)
lidavidm
·el año pasado·discuss
To get a really precise answer you'd have to profile or benchmark. I'd say it's also hard to do an apples to apples comparison (if you only replace the data format in the wire protocol, the database probably still has to transpose the data to ingest it). And it's hard to do a benchmark in the first place since probably your database's wire protocol is not really exposed for you to do a benchmark.

You can sort of see what benefits you might get from a post like this, though: https://willayd.com/leveraging-the-adbc-driver-in-analytics-...

While we're not using Arrow on the wire here, the ADBC driver uses Postgres's binary format (which is still row oriented) + COPY and can get significant speedups compared to other Postgres drivers.

The other thing might be to consider whether you can just dump to Parquet files or something like that and bypass the database entirely (maybe using Iceberg as well).
lidavidm
·el año pasado·discuss
Arrow has several other related projects in this space:

Arrow Flight SQL defines defines a full protocol designed to support JDBC/ODBC-like APIs but using columnar, Arrow data transfer for performance (why take your data and transpose it twice?)

https://arrow.apache.org/blog/2022/02/16/introducing-arrow-f...

There's an Apache-licensed JDBC driver that talks the Flight SQL protocol (i.e. it's a driver for _any_ server that implements the protocol): https://arrow.apache.org/blog/2022/11/01/arrow-flight-sql-jd...

(There's also an ODBC driver, but at the moment it's GPL - the developers are working on upstreaming it and rewriting the GPL bits. And yes, this means that you're still transposing your data, but it turns out that transferring your data in columnar format can still be faster - see https://www.vldb.org/pvldb/vol10/p1022-muehleisen.pdf)

There's an experiment to put Flight SQL in front of PostgreSQL: https://arrow.apache.org/blog/2023/09/13/flight-sql-postgres...

There's also ADBC; where Flight SQL is a generic protocol (akin to TDS or how many projects implement the PostgreSQL wire protocol), ADBC is a generic API (akin to JDBC/ODBC in that it abstracts the protocol layer/database, but it again uses Arrow data): https://arrow.apache.org/blog/2023/01/05/introducing-arrow-a...
lidavidm
·el año pasado·discuss
It used to use DataFusion. https://www.paradedb.com/blog/iceberg_lakehouse
lidavidm
·el año pasado·discuss
Can't you use DataFusion single node/without any Apache ecosystem stuff? They have a Python library and DataFusion is "just" a query engine. (If anything, I'd call Pandas the batteries included option...)

I think the difference is more that DataFusion is built as a library so you can plug it into the product you're building (e.g. Comet, which plugs it into Spark, or pg_lakehouse, which plugs it into Postgres). Polars could be used that way, but it's also a functional package you can pip install and use as a Pandas alternative right now.
lidavidm
·el año pasado·discuss
I'm also curious why it's a fork (I'm sure there's a good reason?)

Coincidentally I was looking into C++ documentation generators again.

In terms of integration, what I've settled on for apache/arrow-adbc is using Sphinx as the toplevel site generator, then writing a script that generates fake Intersphinx indices for a Doxygen site. That way you can link to Doxygen items from within Sphinx without having to hardcode URLs, instead by referencing a class name or similar, and Sphinx will warn if you reference something nonexistent, without having to use something like breathe that tries to render the Doxygen output within Sphinx. (Same approach with Javadoc -> Sphinx, too.)
lidavidm
·hace 2 años·discuss
I suppose this feature should have been termed closed captioning and not subtitling. It seems you're not going to get much sympathy for human translation here.
lidavidm
·hace 2 años·discuss
Thanks! Frankly so long as I can get a browser and install some reader apps (Kobo, Manga-one, etc.) that would fit my needs fine, and as long as they support older versions of Android for enough years (or I can avoid upgrading the app version) then things should be fine. The 10.3" Boox is 80k JPY which is a bit pricey, though, but I'll consider it vs the Kobo device next time I upgrade e-readers.
lidavidm
·hace 2 años·discuss
Which tablet do you use? I've been considering a Boox but the licensing issues and apparent outdated Android give me pause...