V6: Multi-Domain Pretraining Followed by NYU Fine-Tuning

V6 introduces a two-stage training schedule: multi-domain pretraining on SUN RGB-D and DIODE Indoor, followed by NYU Depth V2 fine-tuning under the V5 augmentation pipeline. The configuration achieves 0.519 m NYU validation RMSE (the lowest NYU result in the lineage) and serves as the fine-tuning base for the corridor specialist V9.

Configuration

Method

The configuration applies a two-stage training schedule designed to broaden the encoder’s prior over indoor depth structure before specialization on the primary training corpus.

Diagram source: assets/diagrams/models/v6-pretrain-finetune-timeline.mmd.

Stage 1: Multi-domain pretraining

SUN RGB-D and DIODE Indoor are folded into the training mixture alongside NYU Depth V2. Both auxiliary datasets provide metric depth ground truth that does not require teacher inference. The objective is to expose the encoder to indoor texture and lighting variance not represented in NYU’s apartment-scale scenes.

The auxiliary datasets do not provide segmentation supervision; pixel labels are uniformly set to the ignore index. The segmentation head receives no gradient on auxiliary-dataset batches, requiring the cross-entropy loss to handle all-ignore conditions safely (see Loss-Function Edge Case below).

Stage 2: NYU fine-tuning

The auxiliary datasets are removed. Training continues on NYU Depth V2 alone with the V5 augmentation pipeline (horizontal flip + ColorJitter + random crop / resize). The pretrain establishes a richer encoder prior; the fine-tune adapts the encoder to NYU’s scene statistics and re-engages the segmentation head.

Quantitative Results

The NYU val RMSE improvement of 9.3 % at fixed architecture is attributable to the multi-domain pretraining stage. The improvement is smaller than V4 → V5 (26 %) but is the second-largest improvement on the NYU validation metric in the lineage. The corridor RMSE is statistically unchanged from V5: the multi-domain pretrain does not improve corridor performance directly, since the auxiliary datasets share the apartment-scale geometry of NYU rather than the corridor-scale geometry of the deployment environment.

Operational History

The V6 result of 0.519 m NYU val RMSE was produced by a training program that included two scheduler-terminated runs, one walltime termination, and one configuration error. The full sequence:

The final V6 checkpoint represents two terminated runs producing usable artifacts. Subsequent training programs include explicit dataloader profiling at job submission time and partition-aware walltime budgeting to reduce reliance on terminated-run snapshots.

Loss-Function Edge Case

Job 3084108 was terminated by NaN loss propagation. The cause was a corner case in PyTorch’s nn.CrossEntropyLoss(ignore_index=255): when every pixel in a batch is set to the ignore index, the reduction divides by zero and returns NaN. The NaN propagates through the multi-task loss and corrupts the optimizer state within a small number of iterations.

The condition occurs whenever a batch consists entirely of frames from datasets without segmentation supervision (SUN RGB-D, DIODE), which becomes likely under randomized batch sampling. The fix:

# Within MultiTaskLoss.forward
has_valid_seg = (seg_target != 255).any()
if has_valid_seg:
    L_seg = self.ce(pred_seg, seg_target)
else:
    L_seg = pred_seg.sum() * 0.0   # Returns zero with gradient connectivity preserved

The guard is now applied across all multi-domain training configurations.

Mixed-Supervision Effects

The 15.2 pp regression in NYU val mIoU (V6: 48.5 % vs V5: 63.7 %) is attributable to mixed-supervision effects during Stage 1 pretraining. The segmentation head receives effective gradient on only the NYU subset of pretrain batches (~6 % of frames). By the start of Stage 2 fine-tuning, the segmentation decoder has drifted toward zero-output predictions on the auxiliary-dataset distribution and must re-learn the 6-class structure on NYU. The 154-epoch fine-tune (truncated by walltime) does not fully recover the V5 mIoU.

This is a documented tradeoff: the depth head benefits from the pretrain (richer encoder representations transfer); the segmentation head pays for the auxiliary-dataset supervision gap. The configuration accepts the segmentation regression because depth is the load-bearing output for the costmap fusion pipeline. For deployments where segmentation accuracy is primary, V5 is the preferred checkpoint.

Comparative Performance as Fine-Tuning Base

V6’s primary value lies in its role as a fine-tuning base for domain-specialized configurations. The V6 vs V5 comparison as initialization for corridor specialization:

Initializing from V6 rather than V5 produces a 14 % relative improvement in LILocBench corridor RMSE at fixed fine-tuning protocol. The richer encoder prior established by the V6 pretrain transfers to corridor specialization more effectively than V5’s NYU-only training.

Demonstration

Six-panel video sequence over the 459-frame corridor_eval set. Top row (raw inputs and reference): RGB input · raw Femto Bolt ToF depth · zero-shot DA3-Small reference depth (median-scale aligned). Bottom row (V6-specific predictions and fusions): V6 student raw inference · confidence-gated fusion of ToF and DA3 · confidence-gated fusion of ToF and V6. V6’s prediction structure on the corridor distribution is comparable to V5 (both general-purpose models with similar Femto Bolt corridor RMSE: 2.158 m and 2.186 m, respectively). V6 (Cornerstone) is the recommended fine-tuning base for additional corridor specialists rather than the recommended deployment checkpoint for corridor environments — see V9 for the production corridor model.

Findings

V6 supports two findings on training-program design:

1. Multi-domain pretraining transfers to downstream specialization quality. The 14 % LILocBench corridor RMSE improvement between V7 (V5-initialized) and V9 (V6-initialized) under fixed fine-tuning protocol isolates the effect of the pretraining stage on the specialization endpoint.

2. Mixed-dataset training requires explicit handling of supervision gaps. Datasets without complete annotation coverage produce loss-function edge cases (NaN propagation under ignore-index reductions) that surface only at runtime. Training programs combining sources with heterogeneous annotation must include explicit guards against zero-supervision conditions per loss term.

→ V7 reports the corridor specialization initialized from V5, establishing the baseline against which V9 is compared. → V9 reports the corridor specialization initialized from V6, the recommended deployment checkpoint for corridor environments.