MSN-008 · 2026 · lora, computer-vision, ros2, depth-estimation
LoRA fine-tuning of Depth Anything 3 for underwater metric depth estimation, integrated into a ROS 2 AUV localisation pipeline. 59% AbsRel reduction, validated in live pool tests with the NTU Mecatron AUV team.
Depth Anything 3 is one of the stronger monocular depth models on terrestrial benchmarks. Underwater it breaks completely — AbsRel of 0.774 and δ<1.25 accuracy of 0.9%, because the model has never seen wavelength-dependent light attenuation, backscatter, or the colour cast that comes from water absorbing red before green before blue. This project fine-tuned it for underwater metric depth and wired the result into a full AUV target localisation pipeline for SAUVC, RoboSub, and RoboTX competitions.
DA3 has around 300M parameters. Full fine-tuning on a single A100 would take days and require storing full gradients. Rank-8 LoRA on all DINOv2-L attention projections plus a fully trainable DPT prediction head brings the trainable parameter count to about 9.24% of the total model — roughly 3M LoRA parameters against ~25M in the DPT head.
The LoRA adapter is a custom `LoRALinear` layer written from scratch in `train/lora.py`. Each targeted `nn.Linear` gains a parallel low-rank side path:
output = W_frozen(x) + lora_B(lora_A(x)) * (alpha / rank)
`lora_A` is initialised with Kaiming uniform; `lora_B` is zeroed. The delta is zero at initialisation — the model starts as the original DA3 pretrained weights and deviates from them gradually during training. `inject_lora()` walks every `Attention` block in the DINOv2 backbone and replaces both the fused `qkv` projection and the output `proj` with `LoRALinear` wrappers. Everything else — the DPT prediction head and layer norms — is fully trainable or frozen by `freeze_backbone()`.
For deployment, `merge_lora_weights()` bakes the delta (`lora_B.weight @ lora_A.weight * scaling`) back into the base weight matrix, producing a plain `nn.Linear` that can be exported to TensorRT without any LoRA scaffolding.
MIMIR-UW is a synthetic underwater dataset rendered in Unreal Engine 4. It provides synchronised RGB + metric depth pairs across four environments with physics-accurate light attenuation, backscatter, and caustics. Training used the SeaFloor Algae subset — 9,987 frames with dynamic algae occlusions and high texture density, chosen because those surface properties provide the strongest depth cues without the confounding factors of extreme darkness (OceanFloor) or featureless sand (SandPipe).
Split: 7,990 train / 1,997 val (80/20, seed 42). Inputs are resized to 518×518 (= 37 × 14, a multiple of the ViT patch stride) before feeding to the model.
Depth ground truth is stored as float32 inverse-depth (1/metres). The dataset loader inverts each value at load time (`depth = 1.0 / stored` where `stored > 0`); background pixels with near-zero stored values map to depths above 10,000 m and are eliminated by the 10 m max-depth cutoff.
Rather than hoping the model learns to invert the physics of light attenuation from data alone, the preprocessing pipeline handles the worst of it at load time:
- Gray World white balance corrects for the blue-green colour cast from red wavelength absorption, scaling each channel so the per-channel mean equals the global mean - Percentile histogram stretching (2nd to 98th percentile, per channel) compensates for the low-contrast, haze-heavy appearance of underwater scenes
Both operations run identically at train and inference time, making the preprocessing a fixed part of the model's input contract rather than a training-time augmentation.
Standard depth losses diverge easily on underwater data because the scene scale is ambiguous — the model can minimise reconstruction error by predicting a constant depth plane. The training objective combines three terms:
SILog (Eigen et al., 2014) supervises depth shape while tolerating global scale offset:
$L_\text{silog} = \frac{1}{N}\sum g_i^2 - 0.85 \cdot \left(\frac{1}{N}\sum g_i\right)^2 + 0.1 \cdot \left|\frac{1}{N}\sum g_i\right|$
where $g_i = \log\hat{d}_i - \log d_i$. The first two terms are the standard scale-invariant formulation. The third — a scale anchor weighted at 0.1 — prevents a specific failure mode observed during early training: the variance-focus term cancels the global scale penalty, the model learns an arbitrary depth offset, and AbsRel diverges while SILog stays low.
The Sobel gradient loss adds L1 supervision on depth edges to maintain sharp boundaries at object surfaces:
$L_\text{grad} = \frac{1}{N}\sum \left(|\nabla_x \hat{d}_i - \nabla_x d_i| + |\nabla_y \hat{d}_i - \nabla_y d_i|\right)$
Combined loss: $L = L_\text{silog} + 0.5 \cdot L_\text{grad}$, applied over valid (positive, finite) depth pixels only.
Trained on NSCC ASPIRE, single A100 40 GB, 30 epochs (~8.5 hours). Inference uses bfloat16 autocast (native on A100, no loss scaler needed); gradients are clipped to `max_norm=1.0` to prevent single-batch explosions. Best checkpoint was saved at epoch 27.
AbsRel (orange, left axis) and δ<1.25 accuracy (green dashed, right axis) over 30 epochs — both metrics converge by epoch 20
AbsRel: 0.774 → 0.320 (59% reduction). SqRel: 5.966 → 2.790. RMSE: 8.356 → 6.402 m. δ<1.25: 0.9% → 56.8%. δ<1.25²: 2.2% → 69.6%. δ<1.25³: 3.8% → 75.6%. In-distribution on SeaFloor Algae validation: AbsRel 0.099, RMSE 0.739 m, δ<1.25 91.0%.
These results are cross-environment — trained on SeaFloor Algae, benchmarked on the held-out SeaFloor environment, which the model has never seen.
Baseline — DA3 predicts a near-constant depth plane on this seafloor scene. AbsRel 0.650, δ<1.25 0.000
Fine-tuned — same scene. Structure recovers, AbsRel drops to 0.202, δ<1.25 rises to 0.858
Baseline — second scene. AbsRel 0.656, δ<1.25 0.000. Depth is flat across the entire frame.
Fine-tuned — same second scene. AbsRel 0.149, δ<1.25 0.908. Terrain relief and algae foreground structure both resolve.
The fine-tuned checkpoint is the depth backend in a ROS 2 Jazzy pipeline built for competition (SAUVC / RoboSub / RoboTX). The two inference branches run in parallel on each frame:
Camera feed (monocular RGB)
├─── YOLOv11-seg (TensorRT) ──► instance masks (per-object binary)
└─── DA3 Mono Metric Large ──► per-pixel depth map (metres)
▼
Mask × Depth Fusion
(median of valid depth pixels inside each mask)
▼
"Gate at 2.3 m" — depth threshold ROS 2 topic
▼
SQ-UKF Tracker (Hungarian association between frames)
▼
TF frames → AUV autonomous navigation stack
The SQ-UKF (Square-Root Unscented Kalman Filter) maintains persistent track identities across frames using Hungarian matching on mask-to-track overlap, handling occlusion and target re-entry without losing identity. Tracks are published as TF frames that the AUV's navigation stack consumes directly.
Validated in real pool tests with live AUV footage. The full pipeline code for the ROS 2 integration is withheld — it is the active competition stack of the NTU Mecatron team.
ROS 2 Jazzy running in the NTU Mecatron Unity simulation. Left: YOLOv11-seg detections with per-object metric depth overlaid. Right: Foxglove 3D panel showing SQ-UKF tracks (T0–T5) alongside ground-truth TF frames for gate, flares, and buckets.
- Depth Anything 3 (DINOv2-L backbone, DPT head) - Custom rank-8 LoRA (`train/lora.py`) — no PEFT dependency - YOLOv11-seg with NVIDIA TensorRT - ROS 2 Jazzy - Square-Root Unscented Kalman Filter (SQ-UKF) with Hungarian association - NSCC ASPIRE A100 40 GB for training - MIMIR-UW SeaFloor Algae (9,987 frames) for training; SeaFloor (14,828 frames) for benchmarking - Checkpoint published on HuggingFace: `Frieddeli/da3-underwater`