Tuesday, November 17, 2026
HomeData ScienceMulti-Task Learning – Hard vs Soft Parameter Sharing, Gradient Surgery and Loss...

Multi-Task Learning – Hard vs Soft Parameter Sharing, Gradient Surgery and Loss Weighting

Table of Content

📋 KEY INSIGHTS

  • Multi-task learning (MTL) trains a single neural network to perform multiple tasks simultaneously, sharing representations across tasks to improve generalisation, reduce training time, and lower inference cost compared to training separate models per task. The key inductive bias: tasks that are semantically related share useful intermediate representations, and sharing a feature extractor across tasks acts as a regulariser — each task provides auxiliary training signal that prevents the shared layers from overfitting to any single task’s training distribution. MTL consistently outperforms single-task learning when tasks are related and training data per task is limited.
  • Hard parameter sharing is the most common MTL architecture: a shared backbone (Transformer encoder, ResNet, or multi-layer MLP) feeds into separate task-specific prediction heads. The shared backbone is updated by the combined gradient from all task losses; each head is updated only by its own task loss. The fraction of shared parameters is a hyperparameter — deeper sharing means stronger regularisation and more efficient computation but higher risk of task interference when tasks require conflicting features. Soft parameter sharing maintains separate networks per task but adds regularisation to keep corresponding parameters across tasks close (L2 penalty on weight differences), or uses cross-stitch networks (Misra et al., 2016) that learn a linear combination of activations from different task networks at each layer — the cross-stitch weights are learned and indicate how much each task should borrow representations from each other task.
  • Task interference — when optimising one task hurts performance on another — is the central failure mode of naive MTL with hard sharing. It occurs when task gradients conflict: tasks with conflicting gradients cannot both be satisfied by a single gradient step in weight space. The PCGrad algorithm (Yu et al., 2020) detects and resolves gradient conflicts by projecting each task’s gradient onto the plane orthogonal to any conflicting task’s gradient before the shared update: if g_i · g_j < 0 (negative cosine similarity = conflict), project g_i to remove its component in the g_j direction. Gradient surgery (the informal name for PCGrad and related methods) consistently reduces task interference and improves worst-task performance without any additional hyperparameters.
  • Multi-task loss weighting is critical: simply summing task losses (L = ΣL_k) is suboptimal because tasks with larger loss magnitudes or gradients dominate the update. Uncertainty weighting (Kendall et al., 2018) learns the loss weights from data by modelling each task-specific loss as a Gaussian likelihood with a learnable task-dependent noise parameter σ_k: L = Σ_k (1/(2σ_k^2))·L_k + log σ_k. The log σ_k regularisation term prevents the model from setting all σ_k → ∞ (which would zero out all losses). In practice, uncertainty weighting automatically down-weights tasks that are harder (higher σ_k) or at a different scale, and is one of the most robust loss weighting strategies for MTL across domains. GradNorm (Chen et al., 2018) normalises gradient magnitudes across tasks by learning loss weights that equalise gradient norms at the last shared layer, keeping all tasks training at similar speed.
  • MTL in NLP is especially powerful because language understanding tasks (NER, POS tagging, dependency parsing, semantic role labelling, coreference resolution, sentiment) all benefit from shared contextualised representations. The MT-DNN model (Liu et al., 2019) multi-task fine-tunes BERT across all GLUE benchmark tasks simultaneously, achieving state-of-the-art on most tasks over single-task fine-tuning — the multi-task signal from simpler tasks (NLI, sentence similarity) regularises the shared encoder and improves generalisation on harder tasks (reading comprehension, NER). The connection to our Transformers guide is direct: Transformer encoders are the dominant shared backbone in NLP MTL, and their deep contextualised representations provide feature sharing across all language understanding tasks. Modern instruction-tuned LLMs (GPT-4, Claude, Gemini) are extreme multi-task learners: they are fine-tuned on millions of diverse instruction-following examples across thousands of tasks simultaneously, with the base LLM serving as the universal shared backbone.
  • MTL in computer vision: multi-task networks jointly predict depth estimation, surface normals, semantic segmentation, and instance segmentation from the same input image using a shared encoder (ResNet or ViT) with task-specific decoder heads. TaskPrompter, PAD-Net, and MTI-Net are leading architectures for dense prediction MTL that use feature propagation across task-specific decoders, allowing intermediate features from one task (depth) to inform the prediction of another task (surface normals — geometrically related via the relationship between depth gradients and normals). The connection to our transfer learning guide is direct: the shared encoder in multi-task CV models is typically initialised from ImageNet pre-training, and MTL fine-tuning can be viewed as multi-task transfer learning where the pre-trained backbone is adapted to multiple downstream tasks simultaneously.

Deploying a production ML system often requires solving not one prediction task but many related tasks simultaneously: a fraud detection system must simultaneously classify transaction type, assess account age, predict merchant risk, and generate a fraud probability; a clinical NLP system must jointly extract medications, dosages, diagnoses, and procedures from the same clinical note. Training separate models for each of these tasks is expensive in both training compute and inference latency — each model must run separately at inference time. Multi-task learning solves this by sharing representations across related tasks: a single forward pass through a shared backbone computes features that all task-specific heads use, reducing inference latency to approximately that of a single-task model while training on the richer supervision signal from all tasks simultaneously. This guide covers the full MTL toolkit: hard and soft parameter sharing architectures, task interference and gradient surgery solutions, uncertainty weighting and GradNorm for loss balancing, MTL in NLP (MT-DNN, instruction-tuned LLMs), MTL in computer vision (dense prediction networks), and practical implementation guidance with PyTorch multi-task training loops.

Hard vs Soft Sharing and Gradient Surgery for Task Interference

The architecture choice between hard and soft parameter sharing determines the tradeoff between computational efficiency and task independence. Hard sharing (one backbone, N heads) achieves maximum computational efficiency — a single backbone forward pass serves all N tasks — but requires all tasks to be solvable from the same shared representation. When tasks have conflicting requirements (a sentiment task needing coarse-grained emotional features, a POS tagging task needing fine-grained syntactic features), hard sharing compresses both into the same representation space, potentially hurting both tasks. Soft sharing (N backbones, cross-task communication) allows task-specific representations to develop while enabling selective knowledge transfer via cross-stitch units or feature propagation modules. The tradeoff: soft sharing is N× more parameter-intensive and N× slower at inference, but can represent tasks with incompatible feature requirements. Cross-stitch networks learn the optimal sharing pattern end-to-end: the cross-stitch weights X_{ij} at each layer encode how much task i should use the activations of task j’s corresponding layer. When X_{ij} → 0 for i ≠ j, the network degenerates to N independent single-task networks; when X_{ij} → 1/N, the network degenerates to fully hard-shared. The learned cross-stitch weights are interpretable: inspecting them reveals the actual task relatedness structure as learned from data, often recovering the intuitive groupings (syntactic tasks sharing more with each other than with semantic tasks). Gradient conflict detection: given per-task gradients g_1, …, g_N at the shared parameters, task i and task j conflict if cos(g_i, g_j) < 0. The fraction of conflicting task pairs is a useful diagnostic: high conflict indicates the tasks need more architectural separation; near-zero conflict indicates the shared representation is adequate. The connection to our calculus and backpropagation guide is direct: PCGrad and GradNorm both operate on the gradient vectors of the shared network parameters — understanding multi-task gradient dynamics requires a solid foundation in gradient geometry and the relationship between loss landscape curvature and gradient direction.

ApproachSharingParams vs N tasksInference CostBest For
Hard sharing (shared backbone)Full sharing~1× single task1× forward passRelated tasks; latency-critical
Cross-stitch networksLearned soft sharingN× + cross-stitchN× forward passesMixed relatedness tasks
MMoE (Multi-gate MoE)Shared experts + task gatesK experts + N gates~1× (sparse)Large-scale MTL; recommendation
PCGrad (gradient surgery)Hard (gradient-corrected)~1× single task1× forward passConflicting task gradients
Uncertainty weightingAny architectureN learnable σ paramsNegligible overheadTasks with different loss scales

MTL at Scale — LLM Instruction Tuning, Dense Prediction and Practical Tips

The most successful large-scale application of MTL is LLM instruction fine-tuning: models like GPT-4, Claude, and Gemini are fine-tuned on millions of diverse instruction-following examples spanning thousands of tasks — text summarisation, code generation, question answering, translation, math reasoning, creative writing, tool use — simultaneously, with the LLM backbone acting as the universal hard-shared encoder. The instruction format (system prompt + task description + input → output) is a clever single-head solution: rather than N separate task heads, all tasks are cast as conditional text generation, and the model learns to route between task-specific behaviours via the instruction prompt. The connection to our LLM fine-tuning guide is direct: instruction fine-tuning with LoRA is the dominant practical approach for MTL with large pre-trained models — LoRA adapts the shared backbone efficiently while the instruction format handles task routing, avoiding both the N-head architecture overhead and catastrophic forgetting of earlier tasks. For computer vision dense prediction (simultaneous depth, normals, segmentation from one image), the dominant architecture is an encoder-decoder with task-specific decoder heads sharing the encoder. TaskPrompter (2023) uses visual task prompts — learnable task-specific tokens prepended to the encoder input — to condition the shared encoder on the target task without separate decoder heads, achieving hard sharing with task-specific conditioning analogous to LLM instruction prompting. The connection to our multimodal AI guide is direct: multi-task vision-language models (like GPT-4V, LLaVA) are extreme MTL systems that jointly solve visual question answering, image captioning, visual reasoning, OCR, and chart interpretation from a single shared vision-language backbone. Multi-gate Mixture of Experts (MMoE, Ma et al., KDD 2018) replaces the single shared layer with K expert networks and N task-specific gating networks: each task combines the K expert outputs via a learned softmax gate, enabling different tasks to use different expert combinations. MMoE achieves the efficiency of hard sharing (the K experts are reused across all N tasks) while the gating mechanism provides task-specific expert selection — the gating pattern is interpretable and reveals which experts specialise in which task-relevant features. The connection to our Mixture of Experts guide is direct: MMoE is the MTL application of the broader sparse MoE paradigm used in LLM scaling, bringing the same efficiency-specialisation tradeoff to MTL architectures. Practical MTL training tips: (1) start with uniform task sampling and switch to uncertainty-weighted loss after a few epochs; (2) use task-specific learning rate scaling (tasks with smaller gradients may need higher learning rates); (3) freeze the shared backbone for a few steps per task before allowing joint updates to prevent early-stage catastrophic forgetting of the backbone initialisation; (4) monitor per-task validation metrics separately and check for negative transfer (any task where MTL underperforms the single-task baseline).

✦ SUMMARIZE THIS ARTICLE WITH AI

Transformers (the dominant shared backbone for NLP MTL) are in our Transformers guide. LLM fine-tuning with LoRA (the practical approach to MTL with LLMs) is in our LLM Fine-Tuning guide. Mixture of Experts (MMoE as the MTL extension of MoE) is in our MoE guide. Transfer learning (the relationship between pre-training, transfer, and MTL) is in our Transfer Learning guide. Continual learning (the temporal analogue of MTL — learning tasks sequentially rather than simultaneously) is in our Continual Learning guide. Calculus for ML (gradient geometry underlying PCGrad and GradNorm) is in our Calculus for ML guide. Multimodal AI (MTL at the vision-language intersection) is in our Multimodal AI guide.

Leave feedback about this

  • Rating

Durgesh Kekare
Durgesh Kekarehttps://www.dataexpertise.in
Durgesh Kekare is a data science educator and founder of DataExpertise.in. With expertise in Python, machine learning, and analytics, he helps 10,000+ learners break into data careers.

Latest Posts

List of Categories