Calibration Sensitivity Study

This experiment quantifies the sensitivity of the depth-fusion accuracy to the affine-alignment calibration set. The motivating question:

If the affine parameters (α, β) in the fusion equation d_metric = α · d_pred + β are estimated from one set of frames and applied to a disjoint test set, how does accuracy degrade compared to the optimistic case in which calibration and evaluation share the same frame?

The standard evaluation procedure in this work (run_costmap_ablation.py) uses per-frame median scaling: for each frame, compute \(s = \text{median}(d_{tof}) / \text{median}(d_{DA3})\) on valid pixels, then scale the DA3 prediction by \(s\). This protocol places calibration and evaluation on the same frame, which is the most optimistic configuration measurable.

The experiment below characterizes the degradation that occurs when calibration is performed on a separate set of frames, including the cross-scene case where the calibration and test sets are drawn from distinct environments.

Experiment design

Script: calibration_sensitivity.py

Protocol:

1. Take the 459-frame corridor evaluation set.
2. Split into a calibration set (N frames, randomly sampled) and a test set (remaining 459 - N frames).
3. From the calibration set, compute fixed \(\alpha, \beta\) via least-squares: \(d_{metric} = \alpha \cdot d_{DA3} + \beta\).
4. Apply the fixed \(\alpha, \beta\) to every frame in the test set.
5. Compute AbsRel and RMSE on the test set.
6. Sweep N = {1, 5, 10, 25, 50, 100} frames.
7. Repeat each split 10 times (random seed) to get confidence intervals.

Cross-scene test: Calibrate on corridor → test on glass corridor, and vice versa.

Results

Within-scene: calibration set size

Measured on the 459-frame corridor evaluation set, 10 random seeds per N. The “oracle” row is per-frame alignment — α, β fit independently on every frame’s own pixels. All other rows fit α, β once on a held-out calibration set of N frames and apply the fixed pair to the disjoint test set.

Counter-intuitive result: fixed-set calibration improves over per-frame alignment. Pooling α, β across multiple frames acts as a regulariser — per-frame fits over-adapt to each frame’s outlier pixels (specular floor patches, single-pixel mismatches in glass regions), inflating RMSE on the validation pixels. A fixed-set fit averages these outliers out.

The improvement is robust across N. Even N = 1 (a single calibration frame) reduces RMSE by 16 % relative to the per-frame oracle. The improvement saturates around N = 25 frames at a 19 % reduction.

Cross-scene transfer

Calibration on one scene, test on a disjoint scene. The “Δ” column reports test RMSE relative to that test scene’s own per-frame oracle.

Bold diagonal entries are within-scene calibration (the floor of achievable error). Bold off-diagonals on the corridor column are catastrophic failures: calibrating on glass or bag-213831 and testing on the corridor produces RMSE in the hundreds of meters. The cause is sign inversion of the α parameter — those scenes have a depth-vs-prediction relationship that, when extrapolated to the corridor’s near-field distribution, produces negative or implausibly large depths.

Calibrating on the corridor and testing elsewhere works robustly: the corridor’s depth distribution covers most of the indoor range, so its α, β values do not extrapolate as aggressively to the glass corridor’s narrower distribution or bag 213831’s mid-field-dominant distribution.

α, β stability across frames

For each scene, α and β computed per-frame:

The median-scale factor is stable within each scene (coefficient of variation below 3 %), supporting the runtime substitution policy that uses confidence-gated substitution rather than re-computing α, β per frame. The full α, β series for each scene is recorded in calibration_sensitivity_results.json.

Findings

1. Fixed-set calibration improves over per-frame alignment by 16–19 % on the corridor evaluation set. Pooling regularises the affine fit and removes per-frame outlier sensitivity. This finding contradicts the implicit assumption (used by Eigen et al. 2014 and standard monocular depth evaluation) that per-frame alignment is the most favorable condition. For this distillation pipeline, fixed-set is both cheaper to deploy and more accurate.

2. The improvement saturates around N = 25 frames at a 19 % RMSE reduction. For deployment, collecting approximately 25 calibration frames (~ 8 seconds at 3 FPS, or a single short pass through the environment) is sufficient. Beyond N ≈ 50, additional calibration frames produce no measurable improvement.

3. Cross-scene calibration is asymmetric. Calibrating on the corridor (the most distributionally diverse scene) transfers to other scenes with small improvements. Calibrating on glass corridor or bag 213831 and testing on the corridor produces catastrophic failures (RMSE of hundreds of meters) due to α sign inversion. A robot deploying to a new environment should calibrate in that environment; in particular, calibrating on a narrow distribution and testing on a broader one is not safe.

4. The α, β parameters are stable within each scene (coefficient of variation below 3 %), which supports the deployment realization’s choice to omit explicit per-frame median computation and use confidence-gated substitution directly — see Specification and Deployment Realization.

Reproducing

python calibration_sensitivity.py

# Or via Docker
docker compose run calibration

Output: results/calibration_sensitivity.json with per-N-per-seed breakdown.