The Detour Advantage: Discovering Better Objectives Through Indirect Transfer

We call this "The Detour Advantage": under specific conditions, models trained on one objective and then transferred to another outperform models trained directly on the target. Our hypothesis is that in non-convex optimization landscapes, training temporarily on a related objective can guide the model past local minima that would trap direct gradient descent. To test this, we propose a greedy algorithm that explores objective sequences and demonstrate the phenomenon across function fitting, image reconstruction, sequence prediction, and attribute classification.

Contributions: (1) We show that indirect training paths can outperform direct optimization across multiple domains; (2) We characterize when detours help—underparametrized models, non-convex landscapes, structurally similar tasks—and provide preliminary analysis of why, including weight-space visualizations showing cross-training navigating around local minima.

2. Detour Discovery Algorithm

To determine when a detour helps, we formalize a method that explores combinations of objective sequences and greedily selects the best performing candidates after each round of training. We call this algorithm Cross-Training with Model Selection. We also define a baseline that trains a single model on each objective independently.

Each objective f_i has an associated training set D_i^train. For cross-training, we additionally use a validation set D_i^val for model selection.

2.1. Cross-Training with Model Selection

In the cross-training approach, each objective can adopt models trained for other objectives at each round. This is a greedy search algorithm: at each round, each objective selects whichever model currently performs best on its validation set. A model originally trained on objective f_j might perform better on objective f_i's validation set (after being trained on f_j) than the model trained directly on f_i from the beginning. This creates a dynamic transfer learning environment where models traverse through different objectives over time.

Note: This greedy algorithm is not guaranteed to outperform direct training—a model selected for high early-stage validation performance may lead to poor results after further training. This $O(n^2)$ approach approximates the exhaustive $O(2^{nR})$ search over all possible training paths, which would guarantee finding the optimal trajectory.

2.2. Baseline: Independent Training

The baseline trains each model independently on its own objective—no cross-objective transfer.

2.3. Single-Objective Transfer

To test whether benefits come from evaluating multiple initializations rather than cross-objective transfer, we run Algorithm 1 with all objectives set to the target (f_i = f_j for all i, j). Comparing these three approaches isolates when indirect paths outperform direct training.

3. Synthetic Experiments

To build intuition, we start with three synthetic experiments: function learning, pixel-wise face reconstruction, and sequence transform tasks. These simplified experiments clearly demonstrate the detour advantage.

3.1. Function Learning

We train neural networks to approximate different mathematical functions. Each network learns to predict one of six related functions: f(x), f'(x), f''(x), f'''(x), sign(f''(x)) (indicating pointwise convexity), and f(f(x)) (composition).

Given a fixed base function f(x), these six objectives share underlying structure. Figure 2 shows results across several choices of f(x), each with different complexity characteristics. For each objective, we plot: ground truth (gray), the trajectory that ultimately produces the best model (green, "Final Path"), the current round's best model (blue, "Round Choice"), and baseline direct training (red dashed).

Figure 2 shows that cross-training achieves lower final loss than baseline on nearly all tested functions. The trajectories reveal qualitative differences: baseline training smooths over sharp transitions and fine details, while cross-training preserves the precise shape of the target function. Training temporarily on a related objective allows the model to "snap" to the ground truth structure more quickly—the network adapts faster and captures intricate features that direct training struggles to learn. Crucially, cross-training also outperforms single-objective transfer in every case—the gain is not merely from selecting the best of multiple runs, but from the diversity of objectives themselves.

Notably, derivative objectives (f''(x), f'''(x)) show the largest gains. Derivatives amplify high-frequency components of the base function, making them harder to learn directly. A model trained on f(x) already encodes the function's structure; fine-tuning for derivatives can then leverage this representation rather than learning the amplified signal from scratch.

This advantage is most pronounced on complex functions with rich structure. For simpler functions with smoother, more regular structure, baseline models can learn them directly without requiring cross-training's inductive biases—see Appendix A for these results.

3.2. Pixel-wise Face Reconstruction

To test a higher-complexity task, we train models to reconstruct faces pixel by pixel—a regression task mapping (x, y) coordinates to RGB color values. Using 20 different celebrity faces from CelebA as objectives gives us 20 related but distinct tasks, making this domain ideal for testing cross-training.

Figure 3 shows the complete training progression across 10 rounds with all 20 faces (Figure 1 shows 5 rounds with a subset of faces). At each round, each objective can either continue training its current model or adopt another objective's model that performs better on its validation set.

Training temporarily on another face allows the model to learn intricate facial features more precisely and converge faster. Compare the Detour row to the Baseline row—cross-training captures fine details like texture and edges that baseline training blurs or misses entirely.

Why does training on face Y help reconstruct face X? All faces share common structure: eyes, noses, mouths appear in predictable locations with similar local geometry. Training on any face teaches the network useful primitives—edge detectors, color gradients, symmetry patterns—that transfer across individuals. The detour through face Y provides a curriculum of general facial features before specializing to face X.

3.3. Sequence Transform Tasks

We train small transformer models on 20 different sequence transformation tasks. Each task transforms the same input sequence (16 tokens from a vocabulary of 8) into a different output pattern. For example, one task might reverse the sequence, another might sort it, and another might compute cumulative sums.

The model architecture is intentionally constrained (a 1-layer, 1-head transformer with only 2,300 parameters). This capacity bottleneck makes the experiment particularly revealing: when models lack the capacity to independently learn 20 different tasks, does cross-training help by sharing learned representations?

The 20 tasks span three categories: position-based tasks (Copy, Reverse, various shifts and swaps), value-based tasks (Sort, cumulative operations, differences), and combination tasks that merge multiple operations. We compare baseline training (20 independent specialists) against cross-training where tasks can adopt each other's weights based on validation performance.

Cross-training improves average accuracy from 49.0% to 51.3% (+4.6%). The transfer patterns reveal natural task families: sorting tasks (Sort, SortDesc, SortRev) constantly exchange weights; position tasks (Reverse, ShiftR1, Swap) share positional indexing; inverse operations (CumSum/Diff) transfer bidirectionally. Position-based tasks show dramatic gains (ShiftR1: +36%, Swap: +28%) by discovering shared reasoning circuits.

Critically, these relationships emerge automatically through validation-based selection—we don't manually specify which tasks should help each other. Not all transfer is positive: tasks with unique structure (Diff, Dedupe: -6.8%) perform worse, confirming that the detour advantage requires structural similarity.

4. High-Dimensional Experiment

To test whether the detour advantage scales to higher-dimensional settings, we apply cross-training to multi-attribute classification on the CelebA dataset.

CelebA Multi-Attribute Classification

The CelebA dataset contains over 200,000 celebrity face images, each annotated with 40 binary attributes (e.g., Smiling, Male, Eyeglasses, Blond_Hair). We use 30 of these attributes as training objectives for cross-training, holding out 10 attributes to test generalization to entirely new tasks. This generalization task is meant to measure how good the cross-trained models are at transfering to a new task from the same task distribution, but never seen during training. Good performance on generalization suggests that the cross-trained models have learn task-agnostic features that are useful for all tasks in the same distribution.

We compare three approaches: (1) Specialists, one model trained independently per attribute; (2) Generalist, a single multi-task model predicting all attributes simultaneously; and (3) Cross-Training, our approach that maintains a pool of models and selects the best performer for each attribute based on validation accuracy.

The results reveal a compelling pattern: while cross-training performs comparably to specialists on the 30 training attributes (87.4% vs 87.2%), it significantly outperforms on the 10 held-out attributes (86.1% vs 84.4%). This +1.8% improvement on unseen tasks suggests that cross-training builds more generalizable representations. The diverse training signal from multiple attributes creates features that transfer better to new tasks.

This generalization result is particularly significant. The held-out attributes were never used during training, so cross-training had no opportunity to optimize for them directly or indirectly. Yet cross-trained models consistently outperform both specialists and the generalist on these novel tasks.

We hypothesize this occurs because cross-training acts as implicit representation learning: maintaining a diverse pool of models trained on different objectives naturally preserves features that capture broadly useful patterns. Specialists, by contrast, may overfit to task-specific patterns that don't transfer. The selection pressure (keeping models that perform well across validation sets) implicitly favors representations with good inductive biases for the domain. In practical terms, a cross-trained model pool may serve as a better starting point for any related task, even ones not anticipated during training.

Importantly, the cross-training approach also outperforms the generalist model on the generalization task, which implies that SGD on its own is not able to sufficiently regularize the models to capture cross-task general features.

The per-attribute breakdown reveals that cross-training particularly excels on the held-out attributes (highlighted in green background). Notably, Smiling shows a +5.8% improvement—the largest gain among held-out attributes. This likely reflects that smiling correlates with other trained attributes like High_Cheekbones and Mouth_Slightly_Open, so cross-training builds representations that capture expression-related features even without direct supervision on Smiling.

The generalist model, despite seeing all training attributes simultaneously, fails to match cross-training's performance. This suggests that explicit multi-task learning can actually harm transfer compared to selective weight sharing—perhaps because gradient interference between competing objectives prevents the network from fully exploiting shared structure.

5. Analysis and Discussion

Why do detours help? We analyze the learning dynamics through weight space visualizations and trajectory statistics.

5.1. Weight Space Visualization

To visualize what happens during training, we project the high-dimensional weight space into 2D. We need two weight vectors to define a plane—interpolating between them produces a loss landscape we can visualize. Using the two training trajectories (cross-training and direct training) as our coordinate axes provides an intuitive choice: it aligns the visualization with the actual optimization paths taken by both methods.

The visualizations below reveal qualitative differences in how the two approaches navigate the loss landscape. Direct training (red dashed lines) often follows a convex-like path toward nearby local minima. Cross-training (green solid lines) may temporarily increase error as it trains on a different objective, but eventually navigates toward better solutions. Because these are 2D projections of a high-dimensional space, the detour paths don't always perfectly reach the global minimum shown in the visualization—the true optimum may lie outside our projected plane. See Appendix D for full interactive exploration across all functions and objectives.

Cross-training trajectories curve around regions where direct training gets stuck. The intermediate objective effectively "pushes" the model past local minima that would trap gradient descent. This is especially visible for f'''(x), where direct training converges to a suboptimal plateau while cross-training escapes to a lower-loss region.

5.2. Trajectory Statistics

We analyze optimization trajectories by computing metrics from weight snapshots during training. The table below (Figure 10) shows quantitative comparisons across representative functions and objectives, while the detailed metrics reveal fundamental differences in how cross-training and direct training navigate the loss landscape.

Cross-training exhibits fundamentally different optimization dynamics than direct training. While achieving similar efficiency (loss improvement per unit distance), cross-training takes much longer, more tortuous paths (2.3× length, 3.3× tortuosity). The high exploration metrics—frequent backtracking (25× more) and momentum misalignment (85× more)—show that cross-training actively explores the loss landscape rather than descending directly. The dramatic increases in acceleration (9.5×) and jerk (11.2×) indicate that cross-training frequently changes direction during optimization. These metrics quantify the intuition from Figure 9: cross-training "wanders" through weight space, temporarily moving away from the goal to discover better paths, while direct training follows the most direct route to nearby minima. This is expected—optimizing toward different (even if related) objectives naturally produces more varied trajectories through weight space.

5.3. What Didn't Work?

Not all detours help. We identified three failure modes: (1) Random transfers rarely help—without considering task similarity, the advantage disappears; (2) Too many transfers hurt—longer chains (A→B→C→D→target) show diminishing returns from catastrophic forgetting; (3) Dissimilar tasks don't transfer—preliminary CIFAR-100 experiments between different superclasses showed minimal gains. The detour advantage requires structural similarity between objectives.

Limitations. Our search algorithm is $O(n^2)$ per round, making it impractical for large-scale applications. The phenomenon is demonstrated primarily in underparametrized settings; whether it persists in heavily overparametrized modern architectures remains open. Finally, while we characterize when detours help, we do not yet have a reliable method to predict beneficial detours a priori.

5.4. When Should Detours Help?

Our results raise a fundamental question: when should we expect detours to help? Theory suggests an answer. In the overparametrized regime, loss landscapes become increasingly convex [23], [25]—and in convex optimization, direct gradient descent is provably optimal [27]. No detour can improve upon the straight path. This predicts that as models become more overparametrized, detour benefits should diminish.

By contrast, our experiments use small networks (3-layer MLPs with 64-128 hidden units) in the underparametrized regime, where loss landscapes exhibit sharp non-convexities and numerous local minima [11]. Here, training on a related objective can guide optimization around obstacles that trap direct descent. This aligns with our findings: detours help most on complex objectives where direct training struggles, and may explain why large overparametrized models succeed with direct supervised learning—their scale smooths the terrain where detours provide advantage.

These observations suggest testable predictions: detours should help most when (1) models are underparametrized, (2) landscapes are non-convex, and (3) tasks share structure. Future work should systematically vary these factors to delineate the boundaries of the detour advantage.

6. Related Work

Transfer and Multi-Task Learning. Classical transfer learning assumes fixed source-to-target directions [2], [3]; we explore all paths to find which indirect routes outperform direct optimization. Unlike multi-task learning [5], [1] which trains on objectives simultaneously, our sequential approach shows that temporal ordering matters—training on A then B can outperform training on both together. Taskonomy [8] maps transfer dependencies for data efficiency; we show indirect paths help even with unlimited data, due to optimization geometry.

Curriculum and Continual Learning. Curriculum learning orders data by difficulty [6], [7]; we order objectives, and beneficial detours need not be "easier." Continual learning preserves performance on previous tasks [13]; we explicitly allow forgetting, using intermediate tasks as disposable stepping stones.

Evolutionary Stepping Stones. Quality Diversity algorithms [19] and Novelty Search [20] discover that indirect paths through diverse solutions enable better final outcomes in deceptive landscapes. Our work reveals this same principle in gradient-based training: intermediate objectives provide stepping stones around poor local minima. Lexicase selection [21] and multi-objective decomposition [22] offer related perspectives.

Loss Landscapes. Mode connectivity [9] shows optima are connected by low-loss paths; flat minima generalize better [10]. Our visualizations suggest cross-training navigates around poor local minima, consistent with work on strategic initialization [11]. Greedy layer-wise pretraining [17] used sequential training to escape local minima; we apply this principle in objective space.

7. Conclusion

The Detour Advantage reveals a counterintuitive principle: the shortest path to an objective is not always through the objective itself. By taking strategic detours through related objectives, we achieve better final performance than direct optimization.

This finding has implications for understanding optimization and transfer learning. Just as GPS sometimes recommends indirect routes that arrive faster by avoiding traffic, neural network training may benefit from strategic detours through objective space that avoid poor local minima.

More broadly, our results suggest that the standard paradigm of "train on what you want to optimize" may be leaving performance on the table. In multi-objective settings, the choice of training sequence matters—and the optimal sequence is not always the obvious one. Future work should explore practical algorithms for efficiently discovering beneficial detours without exhaustive search, potentially using task similarity metrics or meta-learning to predict which detours will help.

References

Appendix

A. Cross-Training on Simpler Functions

While Figure 2 in the main text shows cross-training advantages on complex functions, we also tested simpler functions with more regular structure. These functions show smaller performance gains, as their smoother, more regular structure allows baseline models to learn them directly without requiring the inductive biases provided by cross-training.

B. Top Cross-Task Transfers

C. Per-Task Accuracy Comparison

D. Interactive Weight Space Exploration

This interactive visualization allows you to explore weight space trajectories across all tested functions and objectives. Use the controls to select different functions, objectives, and projection methods (trajectory-aligned only).