Show HN: DemandMap – Memory Mapping S3 into Polars on macOS Without FUSE(github.com)

2 pointsby sonthonaxقبل 5 أيام1 comments

Show HN: DemandMap – Memory Mapping S3 into Polars on macOS Without FUSE

https://github.com/sonthonaxrk/demandmap

Basically this exists to solve one of my biggest pain-points with dataframe libraries and big data. It's basically impossible to lazily download data on S3 with polars unless you're processing it strictly sequentially, and even then, it won't work with arrow and numpy data.

This allows you to "download" a multi-gigabyte file on S3 into a Polars DataFrame in <100ms.

# Demo

    alloc = demandmap.S3Alloc(
        "./cache.bin",
        # number of blocks
        capacity=512,
        # one megabyte block (per request chunk size)
        block_size=1048576
    )

    buf1 = alloc.get(S3_PATH)
    # Big file
    assert buf1.nbytes > 400000000
    # But this takes ~100ms
    df = pl.DataFrame([buf1])

It's one of those problems that manages to be both simple and difficult. I'd wager <5% of devs know what a memory map is (higher on HN) and I'd wager 1% of devs who know what mmap is know you can catch page faults in user-space, and yet another 1% of those devs know that it's possible to do userfaultfd in macOS with truly obscure mach_send_msg calls.

I'd really like to build a cross platform user faulting library covering Linux and Windows too, because nearly everyone who's touched a dataframe has had this exact problem.

2 comments

goatofallقبل 3 أيام

Someone do something of all the AI slops in Hackernews!

[deleted]قبل 3 أيام