msackmann · HackerTrans

During an internship almost ten years ago, I was working on a component of the ESA Exomars 2020 mission. After leaving the company, I occasionally checked the progress of the project. First, there was a delay for the launch of two years. Later, Russia invaded Ucraine. As ESA was cooperating with the Russian space agency, the project was stopped. It has since been rescheduled to 2028. It was a great learning experience, but I’m still not sure whether the stuff that I worked on at that time will ever be used.

msackmann

há 3 anosdiscuss

Interesting paper, thanks for bringing this up! I have been working on methods for trajectory optimization using both, analytic gradient computations and black box stochastic gradient approximations (proximal policy optimization).

I was always wondering about a question that is touched in the paper: despite the analytic gradient computation being intuitively more efficient and mathematically correct, it is much harder to learn a policy with it than with the “brute force trial-and-error” black box methods.

This paper brings many new perspectives on why.

msackmann

há 3 anosdiscuss

My guess: the cart pole is an inverted pendulum, and requires multiple left-right-swing-up movements to bring it from the “hanging” position to the “standing” position. Finding this action sequence using gradients of “where is the tip” vs “where should it be” is very hard, as swinging the pendulum to the left and right goes against the gradient.

Instead, using stochastic gradient approximations (policy gradient method such as proximal policy optimization) might be better suited to solving these kinds of problems. Effectively, they do not compute the exact gradient locally, but rather kind of a global approximation by trying out random sequences of actions and determining which of them are closest to the desired outcome.

Hence, stochastic gradient approximations might be considered some kind of hybrid between greedy local optimization (such as following the exact gradient) and global optimization.

comments