Sophont Inc. | Founding Research Engineer / Scientist | Remote | Full-time
Sophont is a public-benefit corporation building open multimodal foundation models for medicine. Joining Sophont means you are paid to lead and contribute to high-impact medical AI research ultimately aimed to transform healthcare and life sciences.
We are looking for exceptional, high-agency ML research engineers who are passionate about using AI to build the future of healthcare. No previous background in medical AI is needed!
If you’re interested in working on highly impactful AI projects that could have the potential to transform and save many lives, Sophont is the company to work at.
Because you can take advantage of pretrained CNNs and perform transfer learning, which is significantly more data-efficient than training from scratch, which is what you'd likely have to do with raw digital signals. This paper is not unique in this approach and many papers have obtained SOTA results by processing digital signals as images.
Our model generates CLIP image embeddings from fMRI signals and those image embeddings can be used for retrieval (using cosine similarity for example) or passed into a pretrained diffusion model that takes in CLIP image embeddings and generates an image (it's a bit more complicated than that but that's the gist, read the blog post for more info).
So we are doing both reconstruction and retrieval.
The reconstruction achieves SOTA results. The retrieval demonstrates that the image embeddings contain fine-grained information, not just saying it's just a picture of a teddy bear and then the diffusion model just generates a random teddy bear picture.
I think the zebra example really highlights that. The image embedding generated matches the exact zebra image that was seen by the person. If the model only could say it's just a zebra picture, it wouldn't be able to do that. But the model is picking up on fine-grained info present in the fMRI signal.
The blog post has more information and the paper itself has even more information so please check it out! :)
Yes we've been looking into ControlNet as well, and I think there is one recent fMRI-to-image paper that also has tried ControlNet. Maybe we'll use ControlNet in MindEye v2 :)
We think it could be useful for clinical research and maybe even diagnostics. For example, you could imagine a person with depression(or other neurological disorders) may have a different perception of the same image than a healthy person. Now with the much higher fidelity that both more powerful MRI machines and better generative AI tools can provide, this may now be a very promising direction for future research.
I use GitHub Copilot very frequently for any programming I do.
I also use BingChat very often, I consider it to be a very underrated tool that people haven't fully explored. Of course, it's great for when you want your LLM queries augmented with search results. But BingChat can also view the current webpage, so for example you can pull up papers and ask BingChat questions about it. I've used it to help write abstracts for my papers. When I do this, BingChat literally searches up "how to write an abstract" which I found absolutely hilarious , it's like it's learning skills on the go as well. I didn't use the outputs directly, but it served as useful inspiration. I think BingChat uses a mix of models including GPT-4, all for free! Apparently more features are coming soon, including multimodal features.
If you're interested in taking this course, you can check the forums and/or Discord where people will organize study groups going through the course. Some of the study groups organized are in-person, some are virtual. These are usually great opportunities to study with some real people at the same time and get that camaraderie.
I have worked on the course with @jph00 and it's been an incredible experience! This is one of the most cutting edge courses about deep learning and diffusion models. Highly recommend folks check it out!
I have worked on the course with @jph00 and it's been an incredible experience! This is one of the most cutting edge courses about deep learning and diffusion models. Highly recommend folks check it out!
the actual platform is called "HuggingFace Hub". The company itself is called "HuggingFace" or "Hugging Face" (I have seen it referred to in both ways, I am unsure which is officially correct). There is no namespace collision.
Sophont is a public-benefit corporation building open multimodal foundation models for medicine. Joining Sophont means you are paid to lead and contribute to high-impact medical AI research ultimately aimed to transform healthcare and life sciences.
We are looking for exceptional, high-agency ML research engineers who are passionate about using AI to build the future of healthcare. No previous background in medical AI is needed!
If you’re interested in working on highly impactful AI projects that could have the potential to transform and save many lives, Sophont is the company to work at.
## Roles
Founding Research Engineer, LLM Team - https://sophont.med/job_postings/llm Founding Research Scientist, Vision Team - https://sophont.med/job_postings/vision Founding Research Scientist, Miscellaneous - https://sophont.med/job_postings/misc
How to apply:
Email resume to [email protected]