---
title: "Reasoning Supervision and Capability... | SI Paper Desk"
url: https://thesimind.si/papers/2611.17072/
description: "In fine-tuning Qwen3-8B on financial tasks, training on labels alone beat adding teacher reasoning traces for in-domain results, while general abilities..."
---

Number 10 of 12 · November 15, 2026

# Reasoning Supervision and Capability Retention in LoRA Domain Adaptation

In fine-tuning Qwen3-8B on financial tasks, training on labels alone beat adding teacher reasoning traces for in-domain results, while general abilities stayed close to the original model.

Saleh Ashkboos, Dhruv Desai and Lavinia Ghita · Submitted November 12, 2026 · arXiv 2611.17072v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.17072) · [The PDF](https://arxiv.org/pdf/2611.17072) · [The day's 12](https://thesimind.si/papers/day/2026-11-15/)

## What they found

The authors ran many ablations (tests that change one thing at a time) on Qwen3-8B using LoRA, a light method that trains a small add-on instead of the whole model. They compared three setups: labels only, one adapter trained on labels plus teacher reasoning traces, and two separate adapters, one for traces and one for labels. Labels-only LoRA improved in-domain performance and left general abilities near those of the frozen model. The biggest in-domain gains were in calibration and argmax accuracy. Adding reasoning traces left in-domain metrics behind the labels-only setup. MMLU-Pro could rise slightly, and the other retention benchmarks stayed within ordinary variation. The dual-adapter setup gave no further gain. Summarizing the traces to make them shorter moved calibration toward the labels-only result, without reaching it.

## Why it matters

Many teams adapt general SI models to fields like finance, and they worry about losing general skills along the way. This paper suggests that simple label-only training can work well, and that adding teacher reasoning does not help in-domain results in this setup. The abstract does not give numbers, so the size of the gaps is unclear. It also covers one model and one domain.

## How it changes the race

This is an incremental, practical result. If it holds, domain adaptation could stay cheaper and simpler, since label-only LoRA avoids the cost of collecting teacher reasoning traces.

### The abstract, by the authors

Fine-tuning foundation models on financial data has a wide range of applications, including identifying event types from news, monitoring risk and compliance language, and answering questions over filings and earnings calls. In this setting, the main challenge is to improve task performance without degrading the generalization capabilities of the model. In this paper, we present an extensive set of ablations for fine-tuning Qwen3-8B with low-rank adaptation (LoRA). We compare three training regimes: (1) response-only LoRA, where the supervised response is the label alone; (2) single-adapter reasoning, where reasoning traces from a teacher model are included alongside the labels in the training loss; and (3) dual-adapter composite training, where separate LoRA adapters are trained on traces and labels. Response-only LoRA improves in-domain performance and leaves general capabilities near those of the frozen model. The in-domain change is largest for calibration and argmax accuracy. Under single-adapter reasoning, the in-domain metrics remain behind those of the response-only regime, while MMLU-Pro can rise slightly and the other retention benchmarks stay within ordinary variation. Dual-adapter composite training remains on the same side of this comparison and yields no further gain. Finally, we compress the teacher traces by summarizing them before training, and show that shorter summaries move calibration toward the response-only regime without reaching it.

Quoted from arXiv:2611.17072. The research and its words belong to its authors.

Desk score **66** out of 100

Why it was picked

- Reasoning (in the title)
- Evals
- Studies frontier models (Qwen3)
- Cross-listed in 3 fields
- Language models

Topics

- [Training](https://thesimind.si/papers/?tag=training)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Efficiency](https://thesimind.si/papers/?tag=efficiency)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)

**arXiv ID:** 2611.17072v1
**Submitted:** November 12, 2026
**Fields:** cs.AI (Artificial Intelligence), cs.CL (Computation and Language), cs.LG (Machine Learning)
**Level:** Intermediate

Fingerprint, SHA-256

`fd147af5342412a1a211579232b53d3e34c6cc5159e35b8ac818873ebad25cae`

On Solana: [Proof of Mind seal, Szprsf...FGjr](https://solscan.io/tx/SzprsfAMG4vXvYD7hrurYFce76smMcrTbLnANy8xS96bh14DoynCdjEEMZf7RPtRUgYeejS8HGnxjqtD1XuFGjr)

### The exact text fingerprinted

{"arxiv":"2611.17072","version":1,"title":"Reasoning Supervision and Capability Retention in LoRA Domain Adaptation","authors":["Saleh Ashkboos","Dhruv Desai","Lavinia Ghita"],"day":"2026-11-15","rank":10,"score":66,"status":"written","summary":{"takeaway":"In fine-tuning Qwen3-8B on financial tasks, training on labels alone beat adding teacher reasoning traces for in-domain results, while general abilities stayed close to the original model.","found":"The authors ran many ablations (tests that change one thing at a time) on Qwen3-8B using LoRA, a light method that trains a small add-on instead of the whole model. They compared three setups: labels only, one adapter trained on labels plus teacher reasoning traces, and two separate adapters, one for traces and one for labels. Labels-only LoRA improved in-domain performance and left general abilities near those of the frozen model. The biggest in-domain gains were in calibration and argmax accuracy. Adding reasoning traces left in-domain metrics behind the labels-only setup. MMLU-Pro could rise slightly, and the other retention benchmarks stayed within ordinary variation. The dual-adapter setup gave no further gain. Summarizing the traces to make them shorter moved calibration toward the labels-only result, without reaching it.","matters":"Many teams adapt general SI models to fields like finance, and they worry about losing general skills along the way. This paper suggests that simple label-only training can work well, and that adding teacher reasoning does not help in-domain results in this setup. The abstract does not give numbers, so the size of the gaps is unclear. It also covers one model and one domain.","race":"This is an incremental, practical result. If it holds, domain adaptation could stay cheaper and simpler, since label-only LoRA avoids the cost of collecting teacher reasoning traces.","safety":"","level":"intermediate","tags":["training","language-models","evals","efficiency","reasoning","benchmarks"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Saleh Ashkboos, Dhruv Desai, Lavinia Ghita

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 14, 2026 · Number 5Parameter-Efficient Conciseness Adaptation: Differentiable Length Control for Concise Clinical Reasoning in Smaller Language ModelsThe authors add one simple term to ordinary fine-tuning so a small medical SI model gives much shorter answers, cutting length by 72.8% while accuracy rose from 40.8% to 44.0%.EfficiencyHealthLanguage models](https://thesimind.si/papers/2611.17128/) · [Nov 15, 2026 · Number 11ArchSpace: Recording, Reviewing, and Referencing LLM Architecture ResearchArchSpace is a public platform where SI model architecture ideas are recorded with their experiments, reviewed by the community, and reused, using scaling laws to judge ideas before costly large training runs.ScalingLanguage modelsEvals](https://thesimind.si/papers/2611.17332/) · [Nov 18, 2026 · Number 2Nemotron 3.5 Bolt E4B 4GB: An Open and Efficient Small Language Model for Fast Conversational AgentsNVIDIA's authors post-trained a small open model from Gemma-4-E4B-it for fast local assistants. They report the top average accuracy on 21 benchmarks and faster decoding with a drafter model.EfficiencyLanguage modelsAgents](https://thesimind.si/papers/2611.20771/)

[Number 9](https://thesimind.si/papers/2611.16190/) · [All 12 for Nov 15, 2026](https://thesimind.si/papers/day/2026-11-15/) · [Number 11](https://thesimind.si/papers/2611.17332/)
