There isn’t really much benefit to having compute on orbit unless you’re working on VERY specific applications that have such tight latency requirements where you need to process the data immediately as it comes out of the sensor. In which case you just implement the algorithms in ASICs or FPGAs anyways.
There have been NVIDIA Jetsons or better on orbit since at least 2021 and that had no meaningful impact on any actual meaningful compute workloads beyond proof of concept demos.
Generally people really don't want collisions due to cascading effects, so they take the worst-case probability of collision found with bounding assumptions. Additionally, while often all these vehicles have active attitude (orientation) control, sometimes they go into safe mode and are spinning (often spin stabilized to point at the sun), so it will clear the entire potential radius while rotating.
Also how do you define the probabilistic average area for a space object that you don't know how it's control system works or what it's been commanded to do / point at. Yes we can make some pretty good assumptions for things like Starlink, but even those do take safemodes occasionally.
So It's an engineering judgement call on how to model it. It's hard to get a probabilistic average for attitude that you can confidently test and say is "right", it's a lot easier and conservative to take the worst-case upper-bound. That's at least not-wrong.
It’s a conservative definition in the field. It’s generally defined as the hard body radius: take the smallest sphere centered at the center of mass that would entirely enclose the object, then use the maximum cross section of that sphere to define the potential “area” of the colliding object.
Maybe put more simply, it’s the worst case area size / orientation you could be looking at. So yes. Solar arrays have a narrow cross section from the side but looking at them head-on (which is the angle used for Pc calculations) they’ll be very large.
Need to do a full read in more depth but it looks like they used a collision cross section of A=300 m^2, which is a little conservative but not insane given that the current Starlink v2 mini has about 90-120 m^2 of total surface area on its solar arrays. [1] The solar arrays are the largest part of these spacecraft by far and what defines the “collideable” area. A combined hard-body radius of 2 x 120 = 240 is in the ballpark for starlink-on-starlink collisions.
However most of collisions of concern are going to be starlink-on-debris, which is back down at the 120 m^2 level. Starlink already self screens for collisions and uplinks the conjunction data messages over the optical intersatellite link backbone or over their global ground station network.
If they aren’t able to talk to their satellites regularly from somewhere, you’re right we have MUCH bigger things to worry about on the ground.
So other ways of combining greater numbers of microbatch gradients in an effective/consistent manner for performing an update is one area of potential future work. I think your idea is an interesting way to approach it. Though there are a bunch of potentially effective ways of doing it.
I think the idea of averaging the k-models afterwards though is at odds with the core concept of gradient agreement filtering though because you're back at combining two distinct directions of improvement without a guarantee that the combination is better (even though it does seem to be in practice). The the core idea is that you philosophically only want to learn the patterns that agree across multiple specific examples and build some some algorithmic protections to ensure that is happening. Just averaging, while it might work and even yield improvement, but it doesn't necessarily lead to proper generalized learning.
> 1. Do you think this is a form of variance reduction or more a form of curriculum (focus first on the bulk, then on remaining errors)?
I'd say generally more of a cirriculum (using your terminology). Broadly speaking the idea is to restrict stepping to "high-quality" directions where there is agreement/consistency in the direction of update.
> 2. Did you observe any overfitting/additional adversarial risk?
No, actually one of the coolest findings of the work is that when training with GAF we have found that it prevents overfitting. It might slow/down stop training improvement, but it also prevents overfitting. Essentially what happens in late training when overfitting would occur the gradient directions become orthogonal, when that occurs GAF instead means you just don't take a step. Training ends up plateauing as it becomes harder to find two minibatches that have agreement so you end up having more no-op epochs, but you don't overfit. I think we still have one training run going (after months) on CIFAR-100N-Fine that has yet to overfit. It's still slowly improving, last time we check train and val were both around ~60%.
Adversarial risk is an interesting question, but this should help with that as well provided that the adversarial examples are a minority of the training data and that the adversarial attack comes from overfitting / memorizing the adversarial part of those examples.
> 3. Did you try this on just single-node minibatches as well? How did that perform?
The number of nodes is more of a performance implementation detail in terms of to what extent/scale you parallelsize. For the technique to work you just need to 2+ macrobatches that you can compare to determine to take your step. CIFAR-100-N is small enough that you can run multiple minibatches on a single GPU (node) and it all fits into VRAM. Even it it didn't fit into VRAM you could theoretically save off the gradient to disk before taking a step and the technique would still apply/help/work, it would just be slower.
Great question! There are a few different aspects to this.
With gradient agreement filtering having a greater number of batches (generally) increases the likelihood of finding another microbatch that agrees with the gradient simply by virtue of having more gradient "samples" to compare. So having more batches increases the chance of success there. The algorithm as laid out in the paper is a simple approach to combining groups of batches where larger numbers batches doesn't necessarily improve you chances of success if the batch you're comparing against is an outlier itself. There are almost certainly better ways of combining greater numbers of batches to get a successful update. This is one of the exciting areas of future work.
Increasing the batch size generally can be though of as "averaging" out the noise in your samples to find a consistent update. This has an interesting affect though where that as you increase the batch size, when using gradient agreement filtering you want to lower the filter threshold as your batch size increases to become "stricter" in terms of the level of agreement to look for to accept a gradient. This is Figure 9 of the paper. This is also consistent with what other researchers have found that simply increasing batch size isn't always better. There is a trade here with diminishing returns of increasing batch size as well (https://arxiv.org/abs/1812.06162). One interesting finding from the work was that smaller batch sizes actually improved training accuracy for CIFAR-100N, very roughly speaking this can be explained by having more "signal" in each batch with smaller batch sizes, at the cost of potentially throwing out batches/gradients if they disagree.
There have been NVIDIA Jetsons or better on orbit since at least 2021 and that had no meaningful impact on any actual meaningful compute workloads beyond proof of concept demos.