Kelvins Satellite Pose Estimation

Overview

The ESA Kelvins Satellite Pose Estimation Challenge ran on the SPEED dataset. The task is to take a single 2D image of a satellite and predict its full 6-DoF pose — a quaternion for orientation plus a position vector. This project was my entry, and the repo is the research codebase I built around it.

My Walkthrough

This was one of the earliest smaller AI/ML competitions I entered. I only heard about it at the halfway point, from a LinkedIn post my old Aerospace professor put up. That got me curious and it turned into many a late night over 30 days, reading papers and trying different ideas out each evening.

I had already played with TensorFlow in a previous competition, so I opted for the same, something familiar instead of PyTorch.

I needed to understand the problem, and thankfully they had a starter script using Keras to help with the initial math and entry submission logistics. As with most CNNs, augmentation is a must have. The challenge was augmenting them in a way that didn’t degrade performance. Warping, distorting, and cutting out whole parts of the image I expected to hurt, so I opted to keep things simple with just rotation, translation, and various illumination patterns and intensities to mimic different lighting from the sun’s albedo. I explored using z-translation too, with the intent of boosting quality of distant image regression, but quickly figured that augmentation did more harm than good.

Back then we didn’t have ChatGPT, it was all Stack Overflow. I wanted a way to quickly inspect the qualitative performance of my regressions. I designed a small visualization, plotting the true vs predicted orientations over a sphere. Simple, but I could then see where the blind spots were.

There were some good papers around at the time, and following one Kaggle grandmaster who said they just read as many papers as they could on the subject, I tried to do the same. One stood out: Langlois et al., “3D Orientation Estimation of Industrial Parts from 2D Images using Neural Networks” (ICPRAM 2018). They swapped the usual L2 loss on quaternions for a geodesic distance loss, which makes sense — L2 can’t tell q from −q, and it forces the network towards one particular quaternion representation. I tried it, it helped, I kept it. Later I also wired in the uncertainty-weighted multi-task loss from Kendall et al. (CVPR 2018), where each head gets its own learned weight — that’s where the σx/σq stats in my training runs comes from.

I recall EfficientNets got released around the same time as this competition, so I explored those for some time. Alongside the typical newbie approach of transfer learning and fine-tuning again and again for different sized models and image resolutions, that didn’t really help much, although the transfer learning did speed experiments up.

I also explored various network topologies, pyramidal, hourglass (encoder-decoder) types, but improvements were negligible. Tried getting a Siamese model training well with modified triplet geodesic losses, didn’t work, I gave up on that idea.

How It Turned Out

The official final table put my entry at 7th of 48 teams on the synthetic test set, with every team using deep learning somewhere in their pipeline. The write-up that covers the whole competition is Kisantal et al., “Satellite Pose Estimation Challenge: Dataset, Competition Design and Results” (IEEE TAES 2020) — that’s the authoritative version of the standings. Sadly the forum discussions from the challenge were wiped during a platform migration, so there’s no record of the conversations had, only the papers and repos that came out of it.

Only the top three entries beat the SLAB baseline network:

• 1st — UniAdelaide: a keypoint pipeline. They recovered the 3D coordinates of 11 landmarks on the Tango satellite via multi-view triangulation, used an object-detection CNN to crop the satellite, trained an HRNet landmark regressor to predict 2D keypoint heatmaps, and finished with a robust nonlinear PnP solve.
• 2nd — EPFL_cvlab: A segmentation-driven CNN predicts 8 corner keypoints of the satellite body plus confidences, and a RANSAC-based PnP solver turns those into a pose.
• 3rd — pedro_fairspace (UrsoNet, Proença & Gao): the one that made me go “of course”. They regressed position with a ResNet but framed orientation as soft classification — the ground truth is encoded as a mixture of Gaussians over a quantized rotation space, so the encoding variance absorbs the attitude ambiguity, instead of point-regressing the quaternion. Their paper is arXiv:1907.04298, and it also introduced URSO, their Unreal Engine 4 simulator that renders labelled spacecraft images with different geometry, albedo, and Earth backgrounds. Comparatively, I was just point-regressing a four-component quaternion plus a position vector, they were letting the loss represent uncertainty over the orientation a far more effective strategy that I’ve carried over to my now professional product responsibilities.

There’s also a great lesson in here about the simulation-to-real gap. The SPEED training set was 12,000 synthetic images and only 5 labelled real images, with the 300 real test images held back unlabelled, so you’re training almost entirely on clean renders and being scored on real optics. The real testbed images had exposure, contrast and noise characteristics the renders didn’t, and the mock-up’s surface didn’t quite match the real Tango. The 3rd place entrant attacked it head-on: they took those 5 real images and bent the synthetic pipeline toward the real domain by varying exposure and contrast, adding AWG noise, blurring the images, and dropping out patches, on top of small camera-orientation warps. They credit that sim-to-real augmentation with letting a URSO-trained model transfer to real footage, and it earned them 2nd on the real test set using just those 5 real images. It’s the same gap ADAS and robotics still wrestle with today, despite how fast the models keep improving.

Architectures Explored

Given a single 2D image of a satellite, a model predicts the full 7-DoF pose: a quaternion for orientation plus a position vector. The work is organized as a research codebase with reusable modules under src/ and self-contained training scripts under experiments/, so architectures and techniques can be compared on equal footing.

• ResNet (34, 50, 101) — baseline CNNs with residual connections
• Inception v3/v4 and Inception-ResNet-v2 — the most heavily experimented family, and the best performer
• DenseNet121 — with and without regularization, including uncertainty prediction and LSTM heads. (LSTM’s obviously did not work.)
• EfficientNet (B3, B4) — later experiments
• Hourglass — stacked hourglass networks for pose estimation
• ResNeXt — aggregated residual transformations
• CBAM — Convolutional Block Attention Module stacked on Inception v3 (I had some good results with this, not quite as good as the best performer, but still top 5.)
• Siamese — The twin branch, shared weight, triplet loss network architecture
• Faster R-CNN — object detection + pose-regression hybrid

Key Techniques Used for my Final Entry

• Loss functions — MSE, geodesic quaternion loss, quaternion norm penalty, uncertainty-weighted multi-task loss (Kendall et al.), auxiliary regression losses
• Augmentation — rotation-only, rotation+translation, and illumination variants
• Regularization — L2 weight decay, batch normalization, dropout
• Optimization — Adam with gradient accumulation, cyclical learning rates, ReduceLROnPlateau
• Output heads — dual-head (quaternion + position) with learned uncertainty