>I found this part to be the most interesting. If this current proclamation is a prelude to the overhaul of H1B system in a way that would make it work like described above, then it is somewhat exciting for a couple of reasons.
I think most people who've dealt with the immigration system would think that that is naive. This administration has proven time and time again with regards to immigration that they will pay lip service to making improvements while almost always simply making life harder for immigrants and people on visas.
See: how they suspended H-1B premium processing for a while some time back, also claiming that it was in service of "overhauling" the H-1B system.
I seem to find myself in the minority, but I don't think distill.pub is a particularly ideal model for publicizing research.
distill.pub heavily favors fancy and interactive visualization over actually meaningful research content. This is not to say that the research publicized on distill.pub is not meaningful, but that it is biased to research that can have fancy visualizations. So you end up seeing a lot of tweakable plots, image-augmentations, and attention weights visualizations. It is also further biased towards research groups that have the resources to create a range of D3 plots with sliders, carved out of actual research time.
For instance, I don't think BERT could ever make it into a distill.pub post. Despite completely upending the NLP field over the last 2 years, it has no fancy plots, multi-headed self-attention is too messy to visualize, and its setup is dead simple. You could maybe have one gif explaining how masked language modeling works. The best presentation of the significance of BERT is "here is a table of results showing BERT handily beating every other hand-tweaked implementation for every non-generation NLP task we could find with a dead-simple fine-tuning regime, and all it had was masked language modeling."
To give another exmaple: I think it's one of the reasons why a lot of junior researchers spend time trying to extract structure from attention and self-attention mechanisms. As someone who's spent some time looking into this topic, you'll find a ton of one-off analysis papers, and next to no insights that actually inform the field (other than super-trivial observations like "tokens tend to attend to themselves and adjacent tokens).
One is a language generation model, the other is a fill-in-the-blank model. It sounds like they might be similar, but in practice they are different enough objectives (and in particular the "bi-directional" aspect of BERT-type models) that the models learn different things.
Those are question-answering and language-understanding benchmarks respectively, neither of which has been suitable for language generation mode evaluation since GPT-1 was roundly beating by BERT. GPT-2 didn't evaluate on them either.
Is that essentially repeating the position embedding? I'm surprised that works, since the model should have no way to distinguish between the (e.g.) 1st and 513th token. (If I'm understanding this correctly.)
1. GPT hasn't really been about model/architectural experimentation, just scale. GPT-2 and GPT were architecturally very similar. Scale, especially at the scale of GPT-*, is one avenue that TensorFlow does have an edge over PyTorch
2. Work on GPT-3 probably started quite a while ago.
Especially with the caveat of "with a programming background", it is far easier to reason and debug through PyTorch with just Python knowledge, compared to TensorFlow/Keras, which sooner or later requires you to learn a condensed history of TensorFlow/Keras development to understand why things are the way they are.
is NOT a good example of a beginner friendly library. It's a thin wrapper facade that hides all of the actual complexity behind "Train ImageNet in 3 lines of code!"
You keep insisting that they're the same when they're not, and then you try to subtly expand your original claim of "using less memory by hashing" to "to be more memory and compute efficient" (emphasis mine), just to force them into the same bucket.
Yes, obviously locality sensitive hashing is a form of hashing. The fact that it's locality sensitive is important for this application, but you'd rather ignore that and insist on labeling them as the same thing just because they're both hashing.
How is it like linear regression, considering linear regression has a closed-form solution, and even if you're using an iterative solver, is a convex problem.
The quick answer to your broader question is that there are multiple ways of slicing how GDP is computed, and yes, economists are aware of them and have thought through the edge-cases as well as being aware of where the measures fall short.
> This and other discussions on HN make me think on the fact that so much of your voice has been recorded by companies when you call their support lines. Are these troves of voice recordings stored safely with appropriate levels of access control?"
"When we said 'quality and training purposes', we were referring to training a neural network."
I think most people who've dealt with the immigration system would think that that is naive. This administration has proven time and time again with regards to immigration that they will pay lip service to making improvements while almost always simply making life harder for immigrants and people on visas.
See: how they suspended H-1B premium processing for a while some time back, also claiming that it was in service of "overhauling" the H-1B system.