HackerTrans
TopNewTrendsCommentsPastAskShowJobs

ylow

no profile record

Submissions

Improving Parquet Dedupe on Hugging Face Hub

huggingface.co
47 points·by ylow·2 anni fa·17 comments

Benchmarking Versioning Tools: S3, DVC, Git LFS, and XetHub

about.xethub.com
13 points·by ylow·2 anni fa·4 comments

You Do Not Need a Vector Database (For RAG): Benchmarking IR Methods

9 points·by ylow·3 anni fa·0 comments

NFS > FUSE: Why We Built Our Own NFS Server in Rust

about.xethub.com
141 points·by ylow·3 anni fa·91 comments

Git xet mount: Rust FS mount implementation for large datasets (using NFSv3)

about.xethub.com
14 points·by ylow·3 anni fa·2 comments

comments

ylow
·5 mesi fa·discuss
Just for you I will switch my FTP server to run on Port 23.
ylow
·5 mesi fa·discuss
Most tech startups will be Delaware. And a lot comes down to the definition of "own" which is ambiguous especially as a C-Corp. One may not be CEO, but can be a H1B Co-founder with a non-trivial (for some amount of non-trivial) number of shares. The O-1 far as I can tell allows for startups, and there may be other visa types which I am unaware of.

Really, I am just saying that the statement "you need a GC to own a business" is far too broad a claim to be true.
ylow
·5 mesi fa·discuss
There is no obligation to provide the public with his life story. Even if provided, few really understand the US immigration process to really comprehend what it means. And finally, does it matter? Even if deportation is fully legally and ethically justified, do the ends justify the means?
ylow
·5 mesi fa·discuss
Whether he did really have valid work permits, or not, I have no idea. You seem knowledgeable. But I am just generally mildly frustrated by people online jumping to conclusions assuming malice or criminal intent, while knowing nothing about the US immigration process. It is not surprising that people don't know how US immigration works. Why would one need to unless it is something you have to work with? There are so many misunderstands about H1B, GC, etc.

I do agree that really that the core issue is not with this one particular case, but broadly a pattern of how people are treated, and a failure of due process. People make mistakes. Governments are made up of people who also make mistakes. Process is how you catch mistakes and minimize its occurrence. A failure of due process reduces trust that even fully legal aboveboard immigrants will be treated reasonably and fairly. And that is reducing my confidence that I will be staying in this country long term.
ylow
·5 mesi fa·discuss
This is not true. You do not need to be a US resident to register a company, and anyone own shares in a company. There are a variety of visa options, and ways to navigate the process that will work.
ylow
·5 mesi fa·discuss
Its not so complicated. Not everyone wants a green card. It triggers international taxation, exit taxes if you give it up, etc. If you can maintain a work permit for 20 years, why not? Until life circumstances change sufficiently that it makes sense to have a green card, the balance of pros and cons may not lean towards it.
ylow
·12 mesi fa·discuss
The application, method and algorithm needs to be separated. The application is movie recommendation. One of the methods which works pretty well for this is low rank matrix completion. There are several algorithms for this method, one of which is quantum.
ylow
·12 mesi fa·discuss
The actual problem that is being solved here is well defined mathematically and is matrix completion via low rank matrix factorization. And using a sampling approach for it. (I have not read the paper in its entirety, just skimmed the intro a bit). It is called "recommendation system" largely due to some history around some of its common appplications (Netflix challenge). But this is not addressing the subjective recommendation problem, but a very particular instantiation of it.
ylow
·12 mesi fa·discuss
This is using statistics to tell a preconceived story. Underlying this a notion that foreign workers are simply “imported” like they are dug out of the ground or something. How do these STEM OPT people find jobs? Guess what. They interview like everyone else does.

1: Every big tech interview I have been in the visa status is not even a question in the interview process. There is just a simple gate that “can you legally work in the US”? The hiring committee is not even thinking about visa (that’s a HR problem)

2: Are there confounders in that foreign workers are less likely to negotiate? Absolutely.

3: are there confounders in that people who come to US for study are likely already a self selected bunch who are striving to succeed? What are the typical grade distributions between foreign STEM students and US STEM students? Is grade a confounding variable? What happens if we control for GPA?

And finally does H1B abuse happen? Absolutely.

There is a lot of nuance that are not captured by surface level statistics. But nuance does not make outrage.
ylow
·2 anni fa·discuss
Even just audio transcription can hallucinate in bizarre ways. https://arstechnica.com/ai/2024/10/hospitals-adopt-error-pro...
ylow
·2 anni fa·discuss
I think both are necessary. The cdc technique is file format independent. The row group method makes Parquet robust to it.
ylow
·2 anni fa·discuss
We are here to help lower that :-) . As we can push dedupe to the edge we can save on bandwidth as well. And hopefully make everyone upload and download faster.
ylow
·2 anni fa·discuss
Great question! Rsync also uses a rolling hash/content defined chunking approach to deduplicate and reduce communication. So it will behave very similarly.
ylow
·2 anni fa·discuss
I believe Parquet predates Arrow. That's probably why.
ylow
·2 anni fa·discuss
Can you elaborate? As I understand Delta Lake provides transactions on top of existing data and effectively stores "diffs" because it knows what the transaction did. But when you have regular snapshots, its much harder to figure out the effective diff and that is where deduplication comes in. (Quite like how git actually stores snapshots of every file version, but very aggressively compressed).
ylow
·2 anni fa·discuss
Hi all! Yucheng (CEO XetHub) here, happy to answer any technical questions anyone might have. Our current tech is a significant enhancement over the original Git Is For Data paper we published last year https://www.cidrdb.org/cidr2023/papers/p43-low.pdf . Hope to write more about it soon! (Maybe with follow up paper or at minimum, a blog post)
ylow
·2 anni fa·discuss
The dedupe is optimistic and is designed to scale to 1-10 PB range. There is a more complicated architecture blog post we are working on. We can dedupe across repositories but we do not right now largely for privacy reasons so that blocks are not shared across different people as that can cause information leakage.
ylow
·2 anni fa·discuss
(Yes 71 + 27 != 100, but that LLMs can't count is a whole other issue)
ylow
·2 anni fa·discuss
Indeed this is unsurprising given how LLMs work. I mean if you ask a human to generate a random number, and then reset the universe and all state of the human and ask again, you will get the same number.

But instead if I ask it to generate 100 samples, it actually works pretty well.

"You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Generate 100 samples of either "left" or "right". Do not say anything else. "

I got 71 left, and 27 right.

And if I ask for 50%, 50%. I get 56 lefts and 44 rights.
ylow
·2 anni fa·discuss
I think this is because a graph is not a data-structure nor a data-type. It is really an abstraction.

Fundamentally, all I need to define a graph is a set of vertices v \in V and function Neighbors(v). And that really is all is needed for the most foundational set of graph algorithms.

Everything else are case-by-case constraints. Does A->B imply B->A? is the node set partitionable with certain constraints? Are there colors? labels?

To make things even more general I can go up one level and consider the hypergraph. In which case I just have a set of vertices, and a set of sets of vertices. This can be represented in a myriad of different ways depending on what you are interested in. Of which (non-hyper) graph is simply a special case.

An alternative way to think about it perhaps from the database perspective, is that its a query optimization and indexing problem. Depending on what questions you want to ask about the graph, there will be different ways to represent the graph to answer the question better. Just like there is not one way to represent the abstraction called "Table", there is not one way to do "Graph" either. It really depends on the questions you care about.