mkaic

5개월 전discuss

Just cancelled mine after reading this comment, I only really cared about the bigger file uploads and the HD screen-sharing anyways and I can live without those.

Now that I think of it, I bet I could host a decent instance of some open-source alternative in a public cloud for around the same cost as what I paid for Nitro ($100 a year)...

mkaic

7개월 전discuss

I suppose both of us watched the same Youtube video by Metta Beshay (i think that is his name?)

mkaic

7개월 전discuss

perhaps I shouldn't share my workaround, but I've found that Mullvad's Norway nodes consistently get past Reddit's IP-blocking :)

mkaic

7개월 전discuss

Heavy and chronic usage of cannabis is associated with some of these things. I think "plenty of evidence that marijuana can cause {effects}" is somewhat overstating the consensus on the topic.

Like many things in life, cannabis can be enjoyed responsibly or irresponsibly. Irresponsible use is inadvisable and can absolutely ruin your life and the lives of others around you. I see no issue with responsible use, though. All things in moderation. Alcohol, social media, and caffeine all come to mind as examples of other drugs that can be largely safe and enjoyable if used responsibly, but which become dangerous/harmful when moderation is abandoned.

Signed,

a responsible/occasional cannabis user :)

mkaic

7개월 전discuss

"It's... a regional dialect."

"What region?"

"Er, upstate New York."

"Really. Well, I'm from Utica and I've never heard anyone use the phrase '100M' to mean '100 thousand'"

"Oh, no, not in Utica. It's an Albany expression."

mkaic

8개월 전discuss

I think some of the coolest changes in this release are on the nodes side of things — they added Closures (kinda like lambdas!), Bundles (tuples/structs, I guess?), and Repeat (loops!) (already was in Geo Nodes, now it's in Shader Nodes too).

Blender nodes have come a long way over the past decade and it's incredibly satisfying to see the care with which they have been developed. Blender's node editor is my personal favorite node editor I've ever used in any software, and I often find myself wishing other software adopted some of their UI and UX conventions.

Been a happy user since, oh, v2.75? And looking forward to being a user for many more releases to come.

Donate to Blender! [0]

[0] https://fund.blender.org

mkaic

8개월 전discuss

IIIII HHHAAAAAVE THE POWERRRRR

mkaic

10개월 전discuss

Location: Boulder, Colorado

Remote: Open to Remote, Hybrid, or In-Person

Willing to relocate: Only for an exceptional offer.

Technologies: AI / Machine Learning, Docker, Python, PyTorch, Numpy, Pandas, Linux, TrueNAS, Pandas, SQL, Blender, Git / Github

Résumé/CV: https://docs.google.com/document/d/1io7PASX56d7eJTCXQnPxjqy3...

Email: matthew.kai.christensen (at) gmail.com

Hello, my name is Kai and I'm a machine learning researcher with an interest in computer vision and novel neural network architecture design. I have 4 years of experience working on computer vision models for medical imagery, including publishing an echocardiography-focused finetune of CLIP in Nature Medicine last year. I'm self taught and enjoy work where I can be creative and learn new skills. I would love to be a part of the team that invents the Transformer-killer. I'm also open to more data-sciencey jobs if the work is interesting.

I also have 3D modeling, filmmaking/editing, and music-making skills and experience if they're relevant. I've been making movies and doing VFX ever since I was little.

mkaic

2년 전discuss

Counterpoint: the hidden state at the beginning of ([the][cat][is][black]) x 3 is (probably) initialized to all zeros, but after seeing those first 4 tokens, it will not be all zeros. Thus, going into the second repetition of the sentence, the model has a different initial hidden state, and should exhibit different behavior. I think this makes it possible for the model to learn to recognize repeated sequences and avoid your proposed pitfall.

mkaic

2년 전discuss

> there would be a massive amount of redundant information stored in hidden activations

Is there a way to prove this? One potential caveat that comes to mind for me is that perhaps the action of lerping between the old state and the new could be used by the model to perform semantically meaningful transformations on the old state. I guess in my mind it just doesn't seem obvious that the hidden state is necessarily a collection of "redundant information" — perhaps the information is culled/distilled the further along in the sequence you go? There will always be some redundancy, sure, but I don't think that such redundancy necessarily means we have to use superlinear methods like attention.

mkaic

2년 전discuss

The counterargument here is that you can just scale the size of the hidden state sufficiently such that it can hold compressed representations of whatever-length sequence you like. Ultimately, what I care about is whether RNNs could compete with transformers if FLOPs are held constant—something TFA doesn't really investigate.

mkaic

2년 전discuss

I strongly enjoy the simplicity of their "minGRU" architecture. It's basically just:

  class MinGRU(nn.Module):
    def __init__(self, token_size, hidden_state_size):
      self.token_to_proposal = nn.Linear(token_size, hidden_size)
      self.token_to_mix_factors = nn.Linear(token_size, hidden_size)

    def forward(self, previous_hidden_state, current_token):
      proposed_hidden_state = self.token_to_proposal(current_token)
      mix_factors = torch.sigmoid(self.token_to_mix_factors(current_token))
      return torch.lerp(proposed_hidden_state, previous_hidden_state, mix_factors)

And since the proposed hidden states and mix factors for each layer are both only dependent on the current token, you can compute all of them in parallel if you know the whole sequence ahead of time (like during training), and then combine them in linear time using parallel scan.

The fact that this is competitive with transformers and state-space models in their small-scale experiments is gratifying to the "best PRs are the ones that delete code" side of me. That said, we won't know for sure if this is a capital-B Breakthrough until someone tries scaling it up to parameter and data counts comparable to SOTA models.

One detail I found really interesting is that they seem to do all their calculations in log-space, according to the Appendix. They say it's for numerical stability, which is curious to me—I'm not sure I have a good intuition for why running everything in log-space makes the model more stable. Is it because they removed the tanh from the output, making it possible for values to explode if calculations are done in linear space?

EDIT: Another thought—it's kind of fascinating that this sort of sequence modeling works at all. It's like if I gave you all the pages of a book individually torn out and in a random order, and asked you to try to make a vector representation for each page as well as instructions for how to mix that vector with the vector representing all previous pages — except you have zero knowledge of those previous pages. Then, I take all your page vectors, sequentially mix them together in-order, and grade you based on how good of a whole-book summary the final vector represents. Wild stuff.

FURTHER EDIT: Yet another thought—right now, they're just using two dense linear layers to transform the token into the proposed hidden state and the lerp mix factors. I'm curious what would happen if you made those transforms MLPs instead of singular linear layers.

mkaic

4년 전discuss

Fellow Stripe Atlas-er here. I have no experience with corporate anything and started my C Corp because I won the Pioneer Tournament and they said I had to to give them a 1% SAFE to claim my winnings, even though I'd made it clear I did not intend to try to monetize my project at all. The company has literally done zero things since inception except cost me money for my registered agent, Delaware corporation tax, and having to pay someone to file taxes for $0 of income because I am too terrified of the tax legal system to do it myself.

If you don't mind me asking, what was your process for dissolving the corporation? I'd love to shut down mine but last time I looked it up I got bogged down in all the legalese about Notices of Dissolution and Board Approval and all that jazz.

mkaic

4년 전discuss

And there you have it. As an aspiring filmmaker and an AI researcher, I'm going to relish the next decade or so where my talents are still relevant. We're entering the golden age of art, where the AIs are just good enough to be used as tools to create more and more creative things, but not good enough yet to fully replace the artist. I'm excited for the golden age, and uncertain about what comes after it's over, but regardless of what the future holds I'm gonna focus on making great art here and now, because that's what makes me happy!

mkaic

4년 전discuss

I assume you meant just "parameters" since "hyperparameters" has a specific alternate meaning? Sorry for the pedantry lol.

Submissions

Mojito – Autocomplete :emoji: everywhere

comments