DA3 Underwater: LoRA Depth Estimation for AUV

Depth Anything 3 is one of the stronger monocular depth models on terrestrial benchmarks. Underwater it breaks completely: AbsRel of 0.774 and δ<1.25 accuracy of 0.9%, because the model has never seen wavelength-dependent light attenuation, backscatter, or the colour cast that comes from water absorbing red before green before blue. This project fine-tuned it for underwater metric depth and wired the result into a full AUV target localisation pipeline for SAUVC, RoboSub, and RoboTX competitions.

Why LoRA

DA3 has around 300M parameters. Full fine-tuning on a single A100 would take days and require storing full gradients. Rank-8 LoRA on all DINOv2-L attention projections plus a fully trainable DPT prediction head brings the trainable parameter count to about 9.24% of the total model, roughly 3M LoRA parameters against ~25M in the DPT head.

The LoRA adapter is a custom LoRALinear layer written from scratch in train/lora.py. Each targeted nn.Linear gains a parallel low-rank side path:

output = W_frozen(x) + lora_B(lora_A(x)) * (alpha / rank)

lora_A is initialised with Kaiming uniform; lora_B is zeroed. The delta is zero at initialisation; the model starts as the original DA3 pretrained weights and deviates from them gradually during training. inject_lora() walks every Attention block in the DINOv2 backbone and replaces both the fused qkv projection and the output proj with LoRALinear wrappers. Everything else, the DPT prediction head and layer norms, is fully trainable or frozen by freeze_backbone().

For deployment, merge_lora_weights() bakes the delta (lora_B.weight @ lora_A.weight * scaling) back into the base weight matrix, producing a plain nn.Linear that can be exported to TensorRT without any LoRA scaffolding.

Dataset: MIMIR-UW

MIMIR-UW is a synthetic underwater dataset rendered in Unreal Engine 4. It provides synchronised RGB + metric depth pairs across four environments with physics-accurate light attenuation, backscatter, and caustics. Training used the SeaFloor Algae subset: 9,987 frames with dynamic algae occlusions and high texture density, chosen because those surface properties provide the strongest depth cues without the confounding factors of extreme darkness (OceanFloor) or featureless sand (SandPipe).

Split: 7,990 train / 1,997 val (80/20, seed 42). Inputs are resized to 518×518 (= 37 × 14, a multiple of the ViT patch stride) before feeding to the model.

Depth ground truth is stored as float32 inverse-depth (1/metres). The dataset loader inverts each value at load time (depth = 1.0 / stored where stored > 0); background pixels with near-zero stored values map to depths above 10,000 m and are eliminated by the 10 m max-depth cutoff.

Physics-aware preprocessing

Rather than hoping the model learns to invert the physics of light attenuation from data alone, the preprocessing pipeline handles the worst of it at load time:

• Gray World white balance corrects for the blue-green colour cast from red wavelength absorption, scaling each channel so the per-channel mean equals the global mean
• Percentile histogram stretching (2nd to 98th percentile, per channel) compensates for the low-contrast, haze-heavy appearance of underwater scenes

Both operations run identically at train and inference time, making the preprocessing a fixed part of the model’s input contract rather than a training-time augmentation.

Loss function

Standard depth losses diverge easily on underwater data because the scene scale is ambiguous; the model can minimise reconstruction error by predicting a constant depth plane. The training objective combines three terms:

SILog (Eigen et al., 2014) supervises depth shape while tolerating global scale offset:

$L_silog = (1) / (N)\sum g_i^2 - 0.85 · \left((1) / (N)\sum g_i\right)^2 + 0.1 · \left|(1) / (N)\sum g_i\right|$

where g_i = log\hat{d}_i - log d_i. The first two terms are the standard scale-invariant formulation. The third, a scale anchor weighted at 0.1, prevents a specific failure mode observed during early training: the variance-focus term cancels the global scale penalty, the model learns an arbitrary depth offset, and AbsRel diverges while SILog stays low.

The Sobel gradient loss adds L1 supervision on depth edges to maintain sharp boundaries at object surfaces:

$L_grad = (1) / (N)\sum \left(|\nabla_x \hat{d}_i - \nabla_x d_i| + |\nabla_y \hat{d}_i - \nabla_y d_i|\right)$

Combined loss: L = L_silog + 0.5 · L_grad, applied over valid (positive, finite) depth pixels only.

Training

Trained on NSCC ASPIRE, single A100 40 GB, 30 epochs (~8.5 hours). Inference uses bfloat16 autocast (native on A100, no loss scaler needed); gradients are clipped to max_norm=1.0 to prevent single-batch explosions. Best checkpoint was saved at epoch 27.

Results on held-out SeaFloor benchmark (14,828 frames across 3 tracks × 2 cameras)

AbsRel: 0.774 → 0.320 (59% reduction). SqRel: 5.966 → 2.790. RMSE: 8.356 → 6.402 m. δ<1.25: 0.9% → 56.8%. δ<1.25²: 2.2% → 69.6%. δ<1.25³: 3.8% → 75.6%. In-distribution on SeaFloor Algae validation: AbsRel 0.099, RMSE 0.739 m, δ<1.25 91.0%.

These results are cross-environment: trained on SeaFloor Algae, benchmarked on the held-out SeaFloor environment, which the model has never seen.

AUV localisation pipeline

The fine-tuned checkpoint is the depth backend in a ROS 2 Jazzy pipeline built for competition (SAUVC / RoboSub / RoboTX). The two inference branches run in parallel on each frame:

Camera feed (monocular RGB)
    ├─── YOLOv11-seg (TensorRT) ──► instance masks (per-object binary)
    └─── DA3 Mono Metric Large  ──► per-pixel depth map (metres)
              ▼
    Mask × Depth Fusion
    (median of valid depth pixels inside each mask)
              ▼
    "Gate at 2.3 m": depth threshold ROS 2 topic
              ▼
    SQ-UKF Tracker (Hungarian association between frames)
              ▼
    TF frames → AUV autonomous navigation stack

The SQ-UKF (Square-Root Unscented Kalman Filter) maintains persistent track identities across frames using Hungarian matching on mask-to-track overlap, handling occlusion and target re-entry without losing identity. Tracks are published as TF frames that the AUV’s navigation stack consumes directly.

Validated in real pool tests with live AUV footage. The full pipeline code for the ROS 2 integration is withheld; it is the active competition stack of the NTU Mecatron team.

Stack

• Depth Anything 3 (DINOv2-L backbone, DPT head)
• Custom rank-8 LoRA (train/lora.py): no PEFT dependency
• YOLOv11-seg with NVIDIA TensorRT
• ROS 2 Jazzy
• Square-Root Unscented Kalman Filter (SQ-UKF) with Hungarian association
• NSCC ASPIRE A100 40 GB for training
• MIMIR-UW SeaFloor Algae (9,987 frames) for training; SeaFloor (14,828 frames) for benchmarking
• Checkpoint published on HuggingFace: Frieddeli/da3-underwater