Show HN: SymForce – Fast symbolic computation, code generation, and optimization(github.com)
github.com
Show HN: SymForce – Fast symbolic computation, code generation, and optimization
https://github.com/symforce-org/symforce
18 コメント
I think the approach makes a lot of sense. In the past I’ve been using Matlab and Mathematica to solve robotics / trajectories problems symbolically and then generate code from the solutions, as big fat inlined functions.
Having a toolbox based on modern open source and widely adopted tools like Sympy to do this automatically is super powerful.
Having a toolbox based on modern open source and widely adopted tools like Sympy to do this automatically is super powerful.
Super cool! How does this compare with Casadi? I didn't see it mentioned in the related AD frameworks in the paper, but I think that targets a similar niche? Thanks for releasing the library!
I think Casadi is solid. Primarily, I'd say Casadi is more specialized towards optimal control. To my knowledge it's much easier in SymForce to formulate complex objectives (ex: involving 3D poses, camera projections, Lie group operations), and has a more flexible code generation system that can generate structured APIs in multiple languages.
How does it compare with TensorFlow (graph-based, i.e. symbolic) or Theano? Or maybe also JAX, which you could also see as symbolic.
In the paper it is briefly mentioned that TensorFlow/JAX are slower due to more overhead? I doubt this is true. Also, TensorFlow could be compiled to TF-lite or XLA and then C++ or whatever you like. Same for JAX.
It is also mentioned that TensorFlow/JAX perform poorly on second-order optimization. But this is not true.
Further, it is said that TensorFlow/JAX have poor performance for sparse matrices. While the performance is not great, I am not sure that other frameworks would perform faster.
Some fair benchmarks would be nice.
But then, despite just benchmarks, also a more direct comparison on a conceptual level would be nice. Because from the description, I don't directly understand how it is much different.
In the paper it is briefly mentioned that TensorFlow/JAX are slower due to more overhead? I doubt this is true. Also, TensorFlow could be compiled to TF-lite or XLA and then C++ or whatever you like. Same for JAX.
It is also mentioned that TensorFlow/JAX perform poorly on second-order optimization. But this is not true.
Further, it is said that TensorFlow/JAX have poor performance for sparse matrices. While the performance is not great, I am not sure that other frameworks would perform faster.
Some fair benchmarks would be nice.
But then, despite just benchmarks, also a more direct comparison on a conceptual level would be nice. Because from the description, I don't directly understand how it is much different.
Part of the motivation for SymForce was in fact serious attempts (multiple months) by us to do similar things with TensorFlow XLA and being stymied by some of these effects. Many of the problems we deal with, and many problems in robotics in general, deal with small "tensor sizes", where dimensions of individual variables are often <10, and batch sizes aren't huge either. SymForce-generated C++ code is typically one machine instruction per scalar per "op" - compiling TF / JAX etc using methods like you mention definitely helps a ton, but it doesn't reach that level of efficiency for small problems. And computing Hessians for second-order optimization tends to blow up op counts even more.
Similarly, SymForce examines sparsity of individual matrices and jacobians at code generation time, so that operations with sparse matrices are similarly one instruction per op - at runtime there's no looping or indirection through sparse data structures or allocations of sparse matrices in the optimization loop. There's definitely a cross-over point where dimensions become large enough that TF / JAX / etc will be the tool to use - we'd love to provide some more benchmarks and comparisons here. For the first two experiments in the paper (see Tables II and III), we do compare to JAX and amortizing across a batch size of 1000, SymForce is still way ahead. We tried to be careful, but the code is available so please come beat up our numbers.
Similarly, SymForce examines sparsity of individual matrices and jacobians at code generation time, so that operations with sparse matrices are similarly one instruction per op - at runtime there's no looping or indirection through sparse data structures or allocations of sparse matrices in the optimization loop. There's definitely a cross-over point where dimensions become large enough that TF / JAX / etc will be the tool to use - we'd love to provide some more benchmarks and comparisons here. For the first two experiments in the paper (see Tables II and III), we do compare to JAX and amortizing across a batch size of 1000, SymForce is still way ahead. We tried to be careful, but the code is available so please come beat up our numbers.
> amortizing across a batch size of 1000
This put the problem outside of L2 and thus is not a fair comparison, especially if your problem is small enough to fit in L1.
In addition, XLA does a poor job in optimizing for CPU cache reuse, so I'm not surprised that directly transcoded C++ code would be much faster than XLA.
However, TensorFlow code can directly run on GPU. SymForce cannot.
This put the problem outside of L2 and thus is not a fair comparison, especially if your problem is small enough to fit in L1.
In addition, XLA does a poor job in optimizing for CPU cache reuse, so I'm not surprised that directly transcoded C++ code would be much faster than XLA.
However, TensorFlow code can directly run on GPU. SymForce cannot.
To be clear, we tested a large set of batch sizes from 1 up to ~1e8. The smaller the batch size, the less competitive JAX was. The crossover point for JAX to be faster was amortizing across a batch size in the tens of thousands I believe. The ideal batch size for some of our applications is one to five. Please see symforce/benchmarks/notebooks for some of our timing.
SymForce is not trying to compute on large data-parallelism, but it could generate an inner kernel for that. In that way it complements TensorFlow nicely.
SymForce is not trying to compute on large data-parallelism, but it could generate an inner kernel for that. In that way it complements TensorFlow nicely.
> TensorFlow code can directly run on GPU. SymForce cannot.
I think this is just a matter of tweaking the code generator, and not a hard limitation of Symforce.
I think this is just a matter of tweaking the code generator, and not a hard limitation of Symforce.
Congratulations, this looks intriguing!
I am somewhat familiar with Ceres and GTSAM. Can you explain how it differs from these? What are the key features of SymForce that make it stand out?
I am somewhat familiar with Ceres and GTSAM. Can you explain how it differs from these? What are the key features of SymForce that make it stand out?
Thank you! At my company we've used GTSAM extensively in production (and the creator of GTSAM was our chief scientist). Much of the math and APIs of SymForce were inspired by GTSAM. Ceres less so but we had someone who has used Ceres professionally implement the experiment in the paper.
The main comparison is that in SymForce one writes symbolic expressions in Python, and generates runtime code. In GTSAM and Ceres you write C++ code. This is really nice for exploration and code maintainability. SymForce also has a powerful system of automatically competing tangent space Jacobians that is more elegant and powerful than what the alternatives offer. Finally, SymForce has the benefit of a fresh implementation using the other libraries as a reference, and I think we were able to get better performance by greatly improving memory access patterns and designing functional APIs with solid abstractions (in the C++ optimization library). You can see this with the benchmarks described in detail in the paper.
The main comparison is that in SymForce one writes symbolic expressions in Python, and generates runtime code. In GTSAM and Ceres you write C++ code. This is really nice for exploration and code maintainability. SymForce also has a powerful system of automatically competing tangent space Jacobians that is more elegant and powerful than what the alternatives offer. Finally, SymForce has the benefit of a fresh implementation using the other libraries as a reference, and I think we were able to get better performance by greatly improving memory access patterns and designing functional APIs with solid abstractions (in the C++ optimization library). You can see this with the benchmarks described in detail in the paper.
How was your experience using this compared with autodiff directly in C++, eg. Jet from Ceres, and depending on the compiler to optimize?
The paper talks a lot about this, and compares directly in the second and third experiments (code included). Overall we made SymForce largely to improve on autodiff workflows like in Ceres. The combination of symbolic code in Python, automatic computation of tangent space Jacobians, and code generation can simultaneously make it much faster to iterate, much faster at runtime (there is no runtime chain rule as with autodiff libraries), and less code to maintain. We've seen this very directly in production on our drones.
Fascinating and incredibly exciting -- can't wait to see how the world uses this library
I've been cobbling together crappy things to do similar things for robot kinematics a lot, this looks awesome, thank you!
This is super awesome! Can't wait to see what it enables!
This is awesome
Well done, thanks a lot
[deleted]
SymForce is a library that makes it easy to code a problem once in Python with an augmented SymPy API (backed by C++), experiment with it symbolically, generate optimized code in C++ or any backend language, and then run highly efficient nonlinear optimization problems based on the original problem definition. This workflow elegantly solves a wide variety of tasks in robotics and related domains, and can speed up common tasks by an order of magnitude while requiring less handwritten code and reducing the surface area for bugs. See our paper at https://arxiv.org/abs/2204.07889 for experiments (accepted to RSS 2022).
We developed it at Skydio for real-time robotics algorithms like SLAM, calibration, bundle adjustment, MPC, and system identification on our drones. It’s a key pillar of our autonomy stack that has accelerated our iteration cycle from prototypes to production systems. We are releasing it to benefit the open-source community, and think its components are useful to anyone writing algorithmic code, like students, research teams, and tech companies.
You can pip install it, play around with a formulation in a notebook, and deploy production code in a couple of hours. Try it at https://github.com/symforce-org/symforce