Well, like most engineering in the real world, people will just test the crap out of it
Having a model that understands physics helps us certify safety. But how much physics is enough? There's a lot to knowing about gravity. You probably don't need orbital dynamics, but you do need to not jump out a second story window.
There's an exceptional amount of work to get a transformer to learn such a thing from scratch even if you had data from all the different cameras and basements and lighting conditions out there. Mentioning opencv is just silly
The entire point is that robotics lives in the full complexity of the real world and therefore cannot afford to do all that for every little corner of every factory, engine room, warehouse, aircraft, etc.
Robots are finally using ML because ML is finally useful!
The gauge is an excellent example partly because of its mundanity. There's a huge number of these little tasks everywhere in real environments. It does an incredible job at adapting to the specifics of its camera resolution, lighting, and the gauge labels to do the task.
Architecturally this is great. Introducing more infra in an ML stack is a huge pain in the short and long run and I really love that this doesn't do that (or at least, the new infra are things I understand).
In the past I've always opted for a feature store as a library that operates over an existing database/data warehouse/data lake in the offline case, and computing features on the fly in the online case. The internal feature cache for scaling an online service is nicely implemented here using Redis. Bravo, that's probably how I would do it too.
My one bit of feedback is the API. The code just doesn't look nice, out the gate there's a bunch of objects and methods I don't immediately understand the need for. I'm sure they're useful, but for starting out I'd expect a lot more from that interface. I'd suggest something higher level that looks pretty and is easy to understand. That would be my one hesitation.
this catches my eye because it highlights an industry mindset shift to come:
"[...] the concept of integrating power ML methods into the real-time experimental data processing loop to accelerate scientific discovery"
data scientists study the science of what the business does (laundry delivery, manufacturing TVs, tracking patient health), and the point of science is insight and understanding from data to build a theory of how it all works
what this article highlights is that ML can be an exceptional tool for discovery. this is in stark contrast to how ML is usually deployed, which is some big analytics or product effort. the obvious big reason for that is the infra is expensive, the know-how is lacking, and the data sucks. well, that all is quickly changing and we're gonna see folks weaving in ML to bolster their workflows in a much bigger way
great to see academic scientists leading the charge here too. they stand to gain a lot from that perspective
Beautiful. So many annotation tools focus on "text classification" which assumes you've already got segmented samples. In the real world of documents that's a whole challenge in itself.
Another challenge is that sometimes you're working with PDFs and that means not only ingesting but also displaying. The difficulty is in keeping track of annotations and predictions across the PDF<->text string boundary, both ways.
There are understandably even fewer solutions to that problem because it's a harder UI to build.
LOL. let the car drive itself, but give up music control? over my dead body!
it's funny to me because you'd expect music to be lower stakes and such, but it just highlights that actually driving a car is a much more well defined problem than picking the music i like
i think i believe that. what is the easiest thought abstraction that can be captured by our sensors? well the abstraction is largely defined by the UI. i like to think of it like language. UI components (words) come together to enable complex actions (sentences or thoughts). it evokes questions, like what language does the brain speak in certain contexts for certain outputs? that's gonna be interesting to follow. what if we all think super differently and that makes it hard? i can't imagine why, but i don't have a background in real brains
it may be that our current "AI" tools might be helpful--they're really good at composing "languages" for tying together different types of data. seems that tying noisy brain sensors data to our English alphabet might be an example of that.
i think focusing entirely on one type of evidence is a little unimaginative. if these guys have data on doctors performing some task with and without their tool, they're in a good place to measure the difference. they can take that all the way to the bank, and to me that would contribute to what id call evidence.
It helps to be able to deliver value at a low bar. This stuff takes a lot of time and energy to improve upon, and it won't "just work". Search is usually a good place to start. Just make one little process a tiny bit better. Make sure you're tracking data very well, because if you're not already at a place to be making data-driven decisions you can't possibly take advantage of machine learning. After all that, just iterate until you've found a better problem to tackle with your newfound capabilities :)
And even if you could, you're getting more value by getting that thing sooner. Or an even better argument: in the end, maybe it took just enough resources at the quality you got to even be born in the current market.
Yeah most things are frivolous... you can survive without a better camera in your phone for another year. But that ham-fisted implicit cultural aspect is what also brings people medical devices 5-10 years sooner than it could have. We all know these gains compound over time. So maybe it's ok if our programs are a little slow. We buy the truly mission critical technology faster with that frivolity.
But I'll also say, I still get mad at my phone and throw it pretty often. Just the way it is.
There's an important thing missing from the HPC community, and that is a focus on software usability.
Maybe things have changed since I worked as a scientist in training at a government lab with one of these crazy machines. But I doubt it.
Trying to use such a computer efficiently was one of the things that took me to my current career as an engineer (and the scientific background landed me in ML), and that was a great thing for me. But for a scientist who is focused on science and not computer architecture, distributed algorithms, programming quirks of working with accelerators (thank you deep learning, because writing CUDA code was painful), they are extremely unproductive when they don't get the help they need.
So I'm fairly skeptical that we're really getting the best return on our investment with Cray and IBM at the helm of supercomputing.
But I have to confess I don't know what I would do to fix it. One thing comes to mind: hire a software tools team at these labs to build a nice software layer to make the rest of the scientific teams more productive. But this is very much what I would do as a company executive, not director of a government science lab. I'm not sure what constraints exist on the finance or politics side. But a change is definitely going to require some leadership.
Having a model that understands physics helps us certify safety. But how much physics is enough? There's a lot to knowing about gravity. You probably don't need orbital dynamics, but you do need to not jump out a second story window.
(Video generation models are an interesting case study: https://www.alphaxiv.org/overview/2501.09038)