If you have to advertise - shove your product in people's faces - to keep sales, your product is not supplying enough real value, does not have staying power, and you should lose.
"Just being a bit better or a little cheaper isn't necessarily going to win over a lot of people if they never bother trying it due to existing brand loyalties"
This is a feature, not a bug. Brand loyalties are built when products are reliable and good. Your product should be enough of an improvement to make people move of their own accord.
If your new product solves frustrations present in an incumbent, on a long enough timescale, your product will come out on top.
If both products are presented equally in a marketplace, the better one will win. If your company does not survive because you can't shove it in people's faces, this is a good thing.
I've been messing around with GA recently, esp indirect encoding methods. This paper seems in support of perspectives I've read while researching. In particular, that you can decompose weight matrices into spectral patterns - similar to JPEG compression and search in compressed space.
Something I've been interested in recently is - I wonder if it'd be possible to encode a known-good model - some massive pretrained thing - and use that as a starting point for further mutations.
Like some other comments in this thread have suggested, it would mean we can distill the weight patterns of things like attention, convolution, etc. and not have to discover them by mutation - so - making use of the many phd-hours it took to develop those patterns, and using them as a springboard. If papers like this are to be believed, more advanced mechanisms may be able to be discovered.
Matta (https://matta.ai) | Industrial Computer Vision/AI for Manufacturing | UK – London (Old Street) | ONSITE | Full-time | Salary + Equity
We’re building industrial AI for factories – deploying into some of the messiest, noisiest, most fascinating real-world environments you can imagine. We build our own hardware, write our own software and train our own models. Our systems watch over manufacturing processes and flags issues real-time - missing parts, out-of-specification dimensions and nearly imperceptible visual defects. We work across manufacturing sectors - aerospace, defence, automotive, food + beverage, electronics, textiles, polymers and all the rest!
interesting! Super cool idea to augment software built with traditional DBs
I had some thoughts [1] around a concept similar to this a while ago, although it was much less refined. My thinking was around whether or not we could have a neural net remember a relational database schema, and be able to be queried for facts it knows, and facts it might predict.
This seems like a much more sensical (and actualised) stab at this kinda concept.
I wonder if, for a given dialect (and even DDL), you could use that token masking technique similar to how that Structured Outputs [1] thing went:
Quote:
"While sampling, after every token, our inference engine will determine which tokens are valid to be produced next based on the previously generated tokens and the rules within the grammar that indicate which tokens are valid next. We then use this list of tokens to mask the next sampling step, which effectively lowers the probability of invalid tokens to 0. Because we have preprocessed the schema, we can use a cached data structure to do this efficiently, with minimal latency overhead."
I.e. mask any tokens that would produce something that isn't valid SQL in the given dialect, or further, a valid query for the given schema. I assume some structured outputs capability is latent to most assistants nowadays, so they probably already have explored this
I'm quite enthusiastic about reading this. Since watching the progress by the larger LLM labs, I've noted that they're not making material changes in model configuration that I think to be necessary to proceed toward more refined and capable intelligence. They're adding tools and widgets to things we know don't think like a biological brain. These are really useful things from a commercial perspective, but I think LLMs won't be an enduring paradigm, at least wrt genuine stabs at artificial intelligence. I've been surprised that there hasn't been more effort to transformative work like in the linked article.
The two things that hang me up on current progress in intelligence is that:
- there don't seem to be models which possess continuous thought. Models are alive during a forward pass on their way to produce a token and brain-dead any other time
- there don't seem to be many models that have neural memory
- there doesn't seem to be any form of continuous learning. To be fair, the whole online training thing is pretty uncommon as I understand it.
Reasoning in token space is handy for evals, but is lossy - you throw away all the rest of the info when you sample. I think Meta had a paper on continuous thought in latent space, but I don't think effort in that has continued to anything commercialised.
Somehow, our biological brains are capable of super efficiently doing very intelligent stuff. We have a known-good example, but research toward mimicking that example is weirdly lacking?
All the magic happens in the neural net, right? But we keep wrapping nets with tools we've designed with our own inductive biases, rather than expanding the horizon of what a net can do and empowering it to do that.
Recently I've been looking into SNNs, which feel like a bit of a tech demo, as well as neuromorphic computing, which I think holds some promise for this sort of thing, but doesn't get much press (or, presumably, budget?)
is "becoming truth seeking" not some sort of religion - like the sports team - and the bay area is your tribe? Perhaps you were already suggesting this in your article and I've missed this - if so I apologise.
you seem to suggest that truth-seeking > tribalism, and we should pity the poor fools who are about tribalism. In this way, you're being tribalist against tribalism, no?
If ignorant tribalism brings people community and happiness, isn't that just as valid and commendable as truth-seeking?
Truth-seeking might provide a level of understanding of the world which is of value to your operating in life. It is not necessarily a sublime good of it's own right. Too much of it will alienate you from your mates.
I'd wager types like you might find on HN, Bay Area, could do with a little less seeking, in fact.
The Underground Man comes to mind, and presents the extreme of this spectrum. But then maybe he'd find mates in an area filled with other Underground Men?
the inferior methods were slower but more flexible - could handle any and all edge cases. Currently we have a UX that really efficiently realises 80% of cases.
To relate to the article - google flights is the Keyboard and Mouse - covering 80% of cases very quickly. Conversational is better for when you're juggling more contextual info than what can be represented in a price/departure time/flight duration table. For example, "i'm bringing a small child with me and have an appointment the day before and I really hate the rain".
Rushed comment because I'm working, but I hope you get the gist.
Current flight planning UX is overfit on the 80% and will never cater to the 20% because cost/benefit of the development work isn't good
A mate of mine built a works scheduler using RL + MCTS. It was interesting seeing the scheduler get smarter as they added in reward for real life constraints. For example, certain types of work couldn't happen on a tuesday - they add that in to the reward calculation, retrain, it now avoids Tuesdays. Build up that reward calculation based on available data, and it got to be super capable at making a workable schedule. Also orders of magnitude faster than linear solvers (albeit without guarantee of "optimality").
I had this thought the other day that the whole chain of thought reasoning pattern contributing to improved performance in LLM-based systems seems to sit parallel to Kahneman's two-system model of the mind that he covers in 'Thinking, Fast and Slow'.
Haven't read it in a few years, but I recall the book suggests that we use one 'System 1' in our brains primarily for low-effort, low computation thinking - like 1+1=? or "the sky is ____".
It then suggests that we use a 'System 2' for deliberate, conscious, high-cognitive tasks. Dense multiplication, reasoning problems, working with tools - generally just decision-making. Anything that requires focus or brain power. Our brain escalates tasks from S1 to S2 if they feel complex or dangerous.
Maybe I'm being too cute, but it feels like critique that "LLMs aren't intelligent because they are stochastic parrots" is an observation that they are only equipped to use their 'System 1'.
When we prompt an LLM to think step-by-step, we allow it a workspace to write down it's thoughts which it can then consider in it's next token prediction, a rudimentary System 2, like a deliberation sandbox.
We do a similar thing when we engage our System 2 - we hold a diorama of the world in the front of our mind, where we simulate what the environment will do if we proceed with a given action - what our friend might respond to what we say, how the sheet steel might bend to a force, how the code might break, how the tyres might grip. And we use that simulation to explore a tree of possibilities and decide an action that rewards us the most.
I'm no expert, but this paper seems to recognise a similar framework to the above. Perhaps a recurrent deliberation/simulation mechanism will make it's way into models in the future, especially the action models we are seeing in robotics.
What if in the stores, botvacs and irobots were presented right next to each other with the same amount of real estate?