Number 10 of 12 · November 15, 2026

Reasoning Supervision and Capability Retention in LoRA Domain Adaptation

In fine-tuning Qwen3-8B on financial tasks, training on labels alone beat adding teacher reasoning traces for in-domain results, while general abilities stayed close to the original model.

What they found

The authors ran many ablations (tests that change one thing at a time) on Qwen3-8B using LoRA, a light method that trains a small add-on instead of the whole model. They compared three setups: labels only, one adapter trained on labels plus teacher reasoning traces, and two separate adapters, one for traces and one for labels. Labels-only LoRA improved in-domain performance and left general abilities near those of the frozen model. The biggest in-domain gains were in calibration and argmax accuracy. Adding reasoning traces left in-domain metrics behind the labels-only setup. MMLU-Pro could rise slightly, and the other retention benchmarks stayed within ordinary variation. The dual-adapter setup gave no further gain. Summarizing the traces to make them shorter moved calibration toward the labels-only result, without reaching it.

Why it matters

Many teams adapt general SI models to fields like finance, and they worry about losing general skills along the way. This paper suggests that simple label-only training can work well, and that adding teacher reasoning does not help in-domain results in this setup. The abstract does not give numbers, so the size of the gaps is unclear. It also covers one model and one domain.

How it changes the race

This is an incremental, practical result. If it holds, domain adaptation could stay cheaper and simpler, since label-only LoRA avoids the cost of collecting teacher reasoning traces.

The abstract, by the authors

Fine-tuning foundation models on financial data has a wide range of applications, including identifying event types from news, monitoring risk and compliance language, and answering questions over filings and earnings calls. In this setting, the main challenge is to improve task performance without degrading the generalization capabilities of the model. In this paper, we present an extensive set of ablations for fine-tuning Qwen3-8B with low-rank adaptation (LoRA). We compare three training regimes: (1) response-only LoRA, where the supervised response is the label alone; (2) single-adapter reasoning, where reasoning traces from a teacher model are included alongside the labels in the training loss; and (3) dual-adapter composite training, where separate LoRA adapters are trained on traces and labels. Response-only LoRA improves in-domain performance and leaves general capabilities near those of the frozen model. The in-domain change is largest for calibration and argmax accuracy. Under single-adapter reasoning, the in-domain metrics remain behind those of the response-only regime, while MMLU-Pro can rise slightly and the other retention benchmarks stay within ordinary variation. Dual-adapter composite training remains on the same side of this comparison and yields no further gain. Finally, we compress the teacher traces by summarizing them before training, and show that shorter summaries move calibration toward the response-only regime without reaching it.

Quoted from arXiv:2611.17072. The research and its words belong to its authors.

The authors

Saleh Ashkboos, Dhruv Desai, Lavinia Ghita

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.