FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

Xin Zhang; Robby T. Tan

FuseAdapt:
Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

NeurIPS 2026 · Poster

Xin Zhang, Robby T. Tan

National University of Singapore

Two DELIVER semantic segmentation examples comparing input images, ground truth, CMNeXt, StitchFusion, and FuseAdapt. The top example uses RGB only; the bottom uses RGB and LiDAR. — Semantic segmentation with RGB only (top) and RGB + LiDAR (bottom) on DELIVER.

FuseAdapt combines modality-private adaptation and cross-modal residual fusion to segment scenes with missing sensors.

Abstract

Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time. Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops. We propose FuseAdapt, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation. Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs adaptation-space fusion by keeping Q/V low-rank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone. The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths. To handle missing modalities, we further introduce Predictive Modality Modeling (PMM), a direct representation-level objective for cross-modal redundancy. During training, PMM masks a target modality and predicts its embedding from the visible ones, supervised by a teacher target computed under full-modality input. PMM is used only during training and adds no inference-time overhead. Across three multi-modal segmentation benchmarks (MCubeS, DELIVER, MUSES), FuseAdapt consistently improves both modality-complete and modality-incomplete segmentation with a small trainable budget, yielding up to +3.01 mIoU under full modalities and +12.56 mIoU under missing modalities over prior state-of-the-art.

Method

FuseAdapt architecture: modalities share frozen backbone weights, retain private attention LoRA branches, and exchange FFN low-rank residual activations. PMM predicts a masked modality representation using visible context and a full-context EMA teacher. — Figure 1. FuseAdapt and Predictive Modality Modeling. Frozen backbone weights are shared; sensor streams retain their own representations.

FuseAdapt adapts a shared frozen vision backbone using modality-specific low-rank adaptation (LoRA). Query and value updates stay private to each sensor, while token-wise routing exchanges feedforward low-rank residuals across available modalities. Missing sources are masked and routing weights are renormalized.

Predictive Modality Modeling (PMM) trains the model to predict a missing sensor’s representation from visible inputs, using a full-context exponential moving average teacher. The teacher and predictor are removed at inference. Routing and averaging operate on a common projected token grid.

Results

We evaluate modality-complete and missing-modality semantic segmentation on MCubeS, DELIVER, and MUSES. The controlled comparison below isolates the fusion design with a matched encoder and training recipe.

Fusion across backbones

MUSES · all modalities · mean mIoU ± sample standard deviation over three seeds

MUSES full-modality mIoU, seeds 3407, 3408, 3409
Backbone	Shared	Separate	FuseAdapt	Gain vs. Shared
MiT-B0	54.13 ± 0.59	55.86 ± 1.38	57.60 ± 0.15	+3.47
MiT-B2	65.80 ± 0.48	67.40 ± 0.77	70.62 ± 0.05	+4.82

Same frozen encoder, SegFormer decoder, training recipe, and checkpoint-selection rule within each backbone. FuseAdapt exceeds Shared in every seed. Three seeds characterize variation; they do not establish formal statistical significance.

Additional ablations and inference cost

What should cross sensor streams?

All-modality mIoU · seed 3407

Information path	MCubeS DINOv2-B	MUSES MiT-B0
Dense-feature routing	55.08	55.94
All-path routed LoRA	55.74	56.17
FuseAdapt	57.62	57.54

Internal alternatives test complete FFN feature routing or routing Q/V and FFN residuals, versus private Q/V and FFN-only residual exchange.

Why predict a missing modality?

MCubeS · missing-modality segmentation · mean mIoU

Training objective	Mean
Modality dropout only	43.67
Output knowledge distillation	49.84
Global-feature distillation	50.57
Full PMM	52.89

Mean over all 15 nonempty input subsets, including all four modalities. PMM predicts a designated missing sensor’s latent representation, rather than generic outputs or fused features.

Parameter efficiency, with measured deployment cost

At 1024 × 1024, batch 1 and FP32 on an H100, three-modality MiT-B0 FuseAdapt takes 21.05 ms, versus 19.63 ms for CMNeXt and 26.69 ms for StitchFusion. These timings use 20 warm-ups and 100 measured runs. Parameter efficiency does not imply the lowest latency or memory use.

BibTeX

The OpenReview record may retain the original title, “FuseAdapt: Adaptation-Space Fusion for Multi-Modal Semantic Segmentation with Missing Modalities.” Both titles refer to the same work.

@inproceedings{zhang2026fuseadapt,
  title     = {FuseAdapt: Parameter-Efficient Multimodal Fusion for
               Semantic Segmentation with Missing Modalities},
  author    = {Zhang, Xin and Tan, Robby T.},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://openreview.net/forum?id=SWGx6oxOoy}
}

Download .bib

The paper will be linked here once it is available on arXiv.