The ship has sailed on this term of art, but as a native English speaker, I agree with you that it's ungrammatical. I would use a hyphen, though: "physics-based." This answer agrees with you (and me): https://english.stackexchange.com/a/428679.
Here's one way to think about _why_ it sounds wrong. "Physically based" would mean that "physically" is the answer to the question "How is it based?" To a native English speaker, that question sounds odd. A more natural question is "What is it based on?" To answer that question, you need a noun: "It's based on physics." Hence the construct "physics-based," like "evidence-based medicine" or "merit-based scholarships."