HackerTrans
TopNewTrendsCommentsPastAskShowJobs

grantjpowell

no profile record

Submissions

Era brings last mile analytics to any data warehouse via DuckDB

cotera.co
2 points·by grantjpowell·قبل سنتين·0 comments

comments

grantjpowell
·قبل سنتين·discuss
This looks neat. I'm the author of a similar project in typescript we use at Cotera called Era [0]. Y'all might be implement something similar to our caching layer [1] which we think is super useful. Once you have a decent cross warehouse representation it's pretty easy to "split" queries across the real warehouse and something like duckdb. The other thing that we find useful in Era that y'all might like are "Invariants"[2]. Invariants work by compiling lazily evaluated invalid casts into the query that only trigger under failing conditions. We use "invariants" to fail a query at _runtime_, which eliminates TOUTOC problems that come from a DBT tests style solution.

    [0] https://newera.dev/
    [1] https://cotera.co/blog/how-era-brings-last-mile-analytics-to-any-data-warehouse-via-duckdb
    [2] https://newera.dev/docs/invariants
grantjpowell
·قبل 3 سنوات·discuss
Love the article.

In my mind I see the problem of dynamic linking in rust to have a bunch of overlap with the "I want this rust library to be exposed in my higher level GC'd language with minimal safety issues/tedious handmaintained bindings" problem.

My hunch is that the lack of expressiveness of the C ABI is holding back both. the thing I'd love to see some sort of "higher level than the C ABI" come out. And something like `wasm-bindgen`[0] to exist for more languages.

Here's a link to the rust "interopable_api" proposal! I don't understand all the implications, but it seems to be in the right direction https://github.com/rust-lang/rust/pull/105586

[0]https://rustwasm.github.io/docs/wasm-bindgen/
grantjpowell
·قبل 4 سنوات·discuss
I enjoy existential comics, which like this site also aims to be a more approachable introduction to philosophy https://www.existentialcomics.com/
grantjpowell
·قبل 4 سنوات·discuss
> now you need a Monad

"need a Monad" sounds scary but in practice it looks like this

    impl<T> Secret<T> {
        pub fn map<U>(&self, func: impl FnOnce(&T) -> U) -> Secret<U> {
            Secret(func(&self.0))
        }

        pub fn flat_map<U>(&self, func: impl FnOnce(&T) -> Secret<U>) -> Secret<U> {
            func(&self.0)
        }
    }
If you need an escape hatch for something more complicated, you could provide an api to that

    impl<T> Secret<T> {
        pub unsafe fn reveal(&self) -> &T {
            &self.0
        }
    }
grantjpowell
·قبل 4 سنوات·discuss
Great callout, I haven't had my coffee yet. Here is a version that better shows what I intended

    pub struct Secret<T>(T);

    impl<T> Secret<T> {
        pub fn map<U>(&self, func: impl FnOnce(&T) -> U) -> Secret<U> {
            Secret(func(&self.0))
        }
    }
    
    impl<T: AsRef<[u8]>> PartialEq<&[u8]> for Secret<T> {
        fn eq(&self, other: &&[u8]) -> bool {
            constant_time_eq(self.0.as_ref(), other)
        }
    }

    /* Some other file */

    use secret::Secret;
    
    // Translated from the example
    fn check_mac<T: AsRef<[u8]>>(mac_secret: Secret<T>, message: &[u8], mac: &[u8]) -> bool {
        // This returns a new Secret<[u8; 32]>
        let computed_mac = mac_secret.map(|secret| hmac_sha_256(secret.as_ref(), message));

        // This uses the `constant_time_eq` impl from above
        computed_mac == mac
    }


I think the interesting part of the example is what you _can't_ do in the other file. It's pretty hard to misuse because the return type of `Secret::map` is a new `Secret`, the only way to do `==` on a `Secret<T>` uses a constant time compare.

I guess my main point is that when you have a instead of having to add new things at the language _level_, if I have something as powerful as the rust type system I can implement the same functionality in not much of code.
grantjpowell
·قبل 4 سنوات·discuss
~I love when I see programming languages who's first advertised features are implementable in 8 lines of rust~

Edit: ^ the above had the wrong tone. Thanks to dang for pointing it out. What I meant to express was that it's possible to accomplish a similar safety/ergonomics at the library level in rust in not too many SLOC. My personal preference is towards Rust's approach because the type system gives really powerful composable primitives which makes it possible to have the compiler check a wide range of invariants, instead of just the ones that are common/special enough to go into the language itself

(Example edited after comments from mumblemumble)

    // The struct is public, but the contents are private, meaning you can't directly access the secret once it's inside the struct
    pub struct Secret<T>(T);

    impl<T> Secret<T> {
        // The only public way to access the secret, returns a new secret
        pub fn map<U>(&self, func: impl FnOnce(&T) -> U) -> Secret<U> {
            Secret(func(&self.0))
        }
    }
    
    impl<T: AsRef<[u8]>> PartialEq<&[u8]> for Secret<T> {
        // == does the correct thing (and only works for types that would make sense (`AsRef<[u8]>`)
        fn eq(&self, other: &&[u8]) -> bool {
            constant_time_eq(self.0.as_ref(), other)
        }
    }

    /* Some other file */

    use secret::Secret;
    
    // Translated from the example
    fn check_mac<T: AsRef<[u8]>>(mac_secret: Secret<T>, message: &[u8], mac: &[u8]) -> bool {
        // This returns a new Secret<[u8; 32]>
        let computed_mac = mac_secret.map(|secret| hmac_sha_256(secret.as_ref(), message));

        // This uses the `constant_time_eq` impl from above
        computed_mac == mac
    }

Edit: It looks like you can implment SOA as a macro too https://github.com/lumol-org/soa-derive

Edit: mumblemumble helpfully points out I demonstrated this poorly, so I tried to better demostrate what I was going for in this comment https://news.ycombinator.com/item?id=33764037
grantjpowell
·قبل 4 سنوات·discuss
If you use the vim fugitive plugin[1], The `:Gbrowse` command [2] will open your browser to github on the correct file/commit. It also works on visually selected ranges, automatically linking to the range in github

[1] https://github.com/tpope/vim-fugitive

[2] https://github.com/tpope/vim-fugitive/blob/master/doc/fugiti...
grantjpowell
·قبل 4 سنوات·discuss
Both of my tech jobs specifically have had non-solicitation clauses to prevent me from doing this type of thing. They both have 1 year lock outs on convincing my friends to quit their jobs
grantjpowell
·قبل 4 سنوات·discuss
Wow, the stack in the article feels like a ton of innovation tokens[1].

I hate to be an armchair expert, but I'll do my best to give the _counter_ opinion to "this is a model of a good startup stack".

If you're looking to build a web app for a business on a small team, some guiding values I've found to be successful (that feel counter to the type of values that lead to the stack in the article)

1.) Write as much "Plain Ol'" $LANGUAGE as possible[2]. Where you do have to integrate with the web framework, Understand your seams well enough that it's hard for your app _not_ to work when it receives a well formed ${HTTP/GQL/Carry Pigeon/Whatever} request

2.) Learn the existing "boring" tools for $LANGUAGE, and idioms that made _small_ shops in similar tech stacks successful.

3.) Learn $LANGUAGE's unit test/integration test framework like the back of your hand. Learn how to write code that can be tested, focus on making the _easy_ way to write code in your codebase to be to write tests _then_ implement the functionality

4.) Have a strong aversion from adding new technologies to the stack. Read this [1], then read it again. Always be asking "how can I solve this with my existing tools". Try to have so few dependencies that it would be hard to "mess up" the difference between local and prod (you can go a LONG way with just Node and PostgreSQL).

Some heuristics to tell if you're doing a good job at the above points,

1.) You don't have to prescribe Dev tool choices (shells, editors, graphical apps, git flows, etc)

2.) You can recreate much of your app on any random dev machine, and feel confident it acts similar to production.

3.) Changing lines of code in your code base at random will generate compiler errors/unit test failures/etc

Most every real world software project I've worked on in the SaaS world ended up with "complexity" as the limiting factor towards meeting the business's goals. When cpu/network/disk etc was the culprit, usually the hard part of fixing it was trying to _understand_ what was going on.

Plain may be very successful in their flow, but I'd say most everything in this article runs counter from the ideas that I've seen be successful in the past.

[1] https://boringtechnology.club/

[2] At our shop we'd say "You're a ruby programming, not a rails programmer, your business logic is likely well factored/designed if it could be straight-forwardly reworked into a rails free command line app"
grantjpowell
·قبل 4 سنوات·discuss
I constantly recommend this at work. The specific content isn't super helpful to Saas day to day development, but for me it built an intuition about postgres that has been invaluable. I think once I understood the "heart and soul" of postgres, the heap and the mvcc, many other properties about the database just "clicked" in my head.
grantjpowell
·قبل 5 سنوات·discuss
I learned basic command line skills from the "Bandit" game[0]. Huge fan of the genre and it's one of the things I always recommend new developers.

[0] https://overthewire.org/wargames/bandit/
grantjpowell
·قبل 5 سنوات·discuss
I've worked in Erlang/Elixir for the past few years and I haven't had the opportunity to work "in anger"[0] with languages with traditional threads/mutexes/sempahores. This game was a blast and made me appreciate the Erlang's approach to concurrency. The best beginner resouce I've read on Erlang's concurrency model is here https://learnyousomeerlang.com/the-hitchhikers-guide-to-conc...

[0] https://erlang-in-anger.com/
grantjpowell
·قبل 5 سنوات·discuss
Yeah! I answer how I got my trail name and a few of my other favorite trail names below
grantjpowell
·قبل 5 سنوات·discuss
Yep! I went by my real name "Grant" for a long time because I knew I would have to tell the story of how I got my trail name 1000 times and I wanted it to be a good story. Then one night I told people that's why I was waiting to take a trail name and a woman started calling me "Bear Bait" and there was no story...

Edit:

Couple of my other favorite trail names I heard

- Pissbag -- on his first night on trail "ol' Pissbag" peed in a ziplock bag while on the second floor of a A.T. Shelter[0], and then "bragged" about his "brilliant" idea to not wake people up in the middle of the night the next morning and _instantly_ earned the name "Pissbag". I heard stories about "ol' Pissbag" for the first 700 hundred miles of the trail and I was super excited to meet (what I assumed to be) the degenerate who was _proudly_ going by that name. I was shocked when I finally ran into him and he turned out to be a super put together former SF enterprise software salesman. He ended up getting a pretty serious case of lyme disease and had to take several weeks off from the Trail.

- Ballsack -- A former ballerina who carried a leather bag with two massage balls on her pack. Her favorite was when she would meet 70 year old women from church groups who came out to feed hikers and when they would ask her her trail name and she got to see "the light drain from their eyes" when she proudly said 'ballsack'. The woman who went by "D*ck Nipples" told a similar story about how she enjoyed seeing the reaction of little old ladies when she told them

- Chingona -- A woman who didn't speak Spanish got called "Chingona" by a man who did speak spanish, who told her that it meant "Badass Woman". She was insanely proud of it for 500 miles until someone told her that word had some vulgar connotations in spanish and she switched to "Chin"

- High Five -- When someone would ask his trail name, he'd raise his hand up and say "High Five". It usually took people several minutes before they figured it out...

-- 5 & 6 -- A man who's this was his 5th attempt to finish and a woman who would sneak up "on people's 6". Who got their trail names totally separately and ended up hiking the last 800 miles together. Everyone assumed they were a couple because of the names...

-- Pack Professor -- Former traveling musician who was very serious about gear (especially his pack). When we would start drinking he'd turn into "Party Professor".

-- China -- She'd tell people her name was "China" and people would instantly try to guess how she got that trail name. Then she'd let them know that "China" was her real name and she went by it because it was funny to watch people guess and "her parents already gave her a trail name"

-- Schrodinger -- I asked him if he was called "Schrodinger" was because he was really into physics and he replied "No, it's because I'm like schrodinger's cat but with whether I've ** my pants". Later we started calling him "Tool of war" because we found out he had that phrase tattooed on his genitals and he was honestly kinda a tool

[0] https://appalachiantrail.org/explore/hike-the-a-t/thru-hikin...
grantjpowell
·قبل 5 سنوات·discuss
I finished the A.T. in October after 174 days, by far the hardest thing I've ever done in my life. I can't imagine trying to do a whole triple crown in a single year. Hats off to these two!

- Bear Bait
grantjpowell
·قبل 5 سنوات·discuss
As silly as it sounds, I use the technique of using of using _really_ informal language to avoid spookiness

"Wanna see somthing cool?" "Lets shoot the shit after this" "Yo, you wanna hang for a minute at 4"