I was curious about this since it kind of makes sense, but I offer a few reasons why I think this isn't the case:
- In the 10% noise case at least, the second descent eventually finds a minima that's better than the original local minima which suggests to me the model really is finding a better fit rather than just reducing itself to a similar smaller model
- If it were the case, I think we might also expect the error for larger models to converge to the performance of smaller models? But instead they converge lower and better
- I checked the logged gradient histograms I had for a the runs. While I'm still learning how to interpret the results, I didn't see signs of vanishing gradients where dead neurons later in the model prevented earlier layers from learning. Gradients do get smaller over time but that seems expected and we don't have big waves of neurons dying which is what I'd expect to have the larger network converge on the size of the smaller one.
I built a personal finance app (https://tender.run) in the style of mailbox (swiping, keyboard shortcut, inbox-based workflow for reviewing transactions).
It's built on the automerge CRDT and sqlite running in the browser, which has been really fun to work with. I'd like to keep going, though honestly I've struggled with the marketing side (growth has been slow) and it's a pretty competitive space.
I am actually currently working on something like this - inbox zero style (don't have swipe gestures built yet) finances tool where you check/set the category. What's old is new I guess.
Is basil still around? Would be interested in taking a look.
Sorry, to be fully clear - i had an internship offer from apple that i declined, and i think i also declined to do the interview process at sonos before getting to the offer stage.
A bit light on the technical details perhaps, but I recall getting stuck on getting the right airplay parameters, learning how byte endianness works... happy to try to answer any other questions as best I can remember.
Consuming a lot of literature on how different systems work helped me develop intuitions around how you might take something apart. Then it's a matter of trying things and banging your head against the wall a lot, e.g. at some point I was interested in how compilers worked so I tried hacking typescript syntax support into babel (circa 2017 maybe) - I got pretty far! and got a lot better sense of how compilers work.
A long time ago sonos didn't support apple airplay.
I did some protocol reversing and wrote a small program that pretended to be an airplay speaker to pipe audio to a sonos speaker (archive: https://github.com/stephen/airsonos)
I ended up getting recruiting messages from both the airplay team at apple and some folks from sonos. I didn't end up taking either offer, but it was also an interesting talking point when interviewing for the job I did take.
I was curious about this since it kind of makes sense, but I offer a few reasons why I think this isn't the case:
- In the 10% noise case at least, the second descent eventually finds a minima that's better than the original local minima which suggests to me the model really is finding a better fit rather than just reducing itself to a similar smaller model
- If it were the case, I think we might also expect the error for larger models to converge to the performance of smaller models? But instead they converge lower and better
- I checked the logged gradient histograms I had for a the runs. While I'm still learning how to interpret the results, I didn't see signs of vanishing gradients where dead neurons later in the model prevented earlier layers from learning. Gradients do get smaller over time but that seems expected and we don't have big waves of neurons dying which is what I'd expect to have the larger network converge on the size of the smaller one.