First, control the speed of a DC Motor. Here you can learn about transfer functions and system dynamics, PID Controllers, pole allocation, Phase-lead/phase-lag and etc.
Then try to control the angle position of a DC Motor using a Cascade loop.
Then you can try to control an inverted pendulum on a cart using State Feedback. Here you can use pole allocation, or other methods to find your controller such as Optimal Control, or even MPC. I have a toy program in C to simulate and control an inverted pendulum using C. It is a Literate program and you can use it as a guide to implement in other languages: https://github.com/Accacio/pendulum.