This isn't a problem at all for modern type systems. OCaml, Haskell, and even C++ have Turing complete type systems and this is one of the last concerns for developers.
The reinforcement learning framework is perfect for representing cause and effect. An agent could learn that in a state of no fire, taking an action of rubbing sticks together would transition into a state of having fire. This concept is formalized as learning the dynamics function.
Neural networks do exactly what you are describing as "symbolic reasoning". It seems to be a common thing recently to dismiss modern ML techniques as curve fitting, but these fundamental models are extremely powerful.
Neural networks are capable of approximating any system to arbitrary precision.
The "self-model" you're talking about is the agent in the Reinforcement Learning framework. It moves between states in an environment and learns from reward it earns from each action.