Consensus-Aligned Neuron Efficient Fine-Tuning Large Language Models for Multi-Domain Machine Translation

Recent research on domain-specific machine translation with LLMs has explored both inference-time adaptation and fine-tuning. Aycock and Bawden (2024) propose a topic-guided demonstration retrieval method to enhance translation performance of ICL without fine-tuning. Li et al. (2025) compare retrieval- and generation-based domain prompting, showing retrieval provides better grounding while generation enables flexibility. Hu et al. (2024) build a MDMT benchmark and enhance cross-domain performance via domain-aware chain-of-thought fine-tuning. Zheng et al. (2024b) introduce dictionary- and retrieval-augmented fine-tuning to bridge terminology gaps in domain translation.

Recent studies have revealed that neurons in LLMs exhibit modular behavior, with certain neurons responsible for specific languages (Tan et al. 2024; Tang et al. 2024; Zhu et al. 2024), tasks (Song et al. 2024; Leng and Xiong 2025), or knowledge (Dai et al. 2022; Chen et al. 2024; Niu et al. 2024; Mao et al. 2025). This intrinsic structure has motivated a growing works on neuron-level analysis (Voita et al. 2024) and selective fine-tuning (Xu et al. 2025). Tan et al. (2024) show that neuron activations correlate with language typological proximity, while Tang et al. (2024) introduce LAPE to identify language-specific neurons and control output via targeted activation. Zhu et al. (2024) achieve strong multilingual translation performance by routing updates through language-specific and general neurons. For tasks, Song et al. (2024) and Leng and Xiong (2025) identify task-specific neurons using activation patterns and gradient attribution, respectively. In terms of factual knowledge, Dai et al. (2022) define ”knowledge neurons” that activate for specific facts. Chen et al. (2024) shedding light on the mechanisms of cross-lingual factual knowledge storage in LLM neurons.

Unlike previous work, we identify consensus-aligned neurons for MDMT by measuring mutual information with domain context, and propose a neuron-efficient fine-tuning framework that enhances cross-domain translation while mitigating PEFT limitations.

Neurons that consistently exhibit strong gradient and activation responses to task-specific inputs are likely to encode MDMT-relevant features. To identify such neurons, we compute gradient-activation importance scores for each neuron in the FFN layers of an LLM $f_{\theta}$ during MDMT inference.

For each domain $d$, we utilize parallel data $D_{d}=(\mathcal{T}^{d},\mathbf{x}^{d},\mathbf{y}^{d})$, where $\mathbf{x}^{d}=\{x^{d}_{1},...,x^{d}_{K}\}$ is the source sequence and $\mathbf{y}^{d}=\{y^{d}_{1},...,y^{d}_{T}\}$ is the target sequence. $\mathcal{T}^{d}$ is the domain-specific instruction designed to inform the LLM of the domain context. For example, a De$\Rightarrow$En IT domain instruction is: ”You are a translation specialist who specializes in translating texts from German to English in the IT domain. Translate the following content into English and only reply to the translated sentence without line breaks or special symbols.”

Inspired by studies on importance-based neuron selection for multilingual machine translation (Xie et al. 2021), we assess a neuron’s importance by multiplying its activation during the forward pass by the loss gradient with respect to that activation during the backward pass. Neurons with high importance scores are identified as strongly associated with the MT task for a given domain $d$.

Specifically, let $A_{l,j}^{(d)}$ denote the activation of the $j$-th neuron in the $l$-th FFN layer when processing an input $\mathbf{x}^{d}$ from domain $d$, and $G_{l,j}^{(d)}$ be the gradient of the loss with respect to that activation:

$$G_{l,j}^{(d)}=\frac{\partial\mathcal{L}^{(d)}(\mathbf{x}^{d},\mathbf{y}^{d})}{\partial h_{l,j}^{(d)}},$$

(1)

where $h_{l,j}^{(d)}$ is the output of the $j$-th neuron in the $l$-th FFN layer for domain $d$. And the token-level cross-entropy loss $\mathcal{L}^{(d)}$ between the model’s prediction and the reference translation $\mathbf{y}^{d}$ is:

$$\mathcal{L}^{(d)}(\mathbf{x}^{d},\mathbf{y}^{d})=-\sum_{t=1}^{T}\log p_{\theta}(y_{t}^{d}\mid y_{<t}^{d},\mathbf{x}^{d},\mathcal{T}^{d}).$$

(2)

The neuron importance score for $d$ is then computed as:

$$I_{l,j}^{(d)}=\mathbb{E}{(\mathbf{x}^{d},\mathbf{y}^{d})\sim D_{d}}\left[\left|A_{l,j}^{(d)}\cdot G_{l,j}^{(d)}\right|\right].$$

(3)

To further prove the proposed gradient-based activation importance metric, we approximate a neuron’s contribution by evaluating the change in loss when the neuron is removed. Thus, we apply a first-order Taylor Expansion (Molchanov et al. 2017) to estimate the impact of ablating individual neurons. Let $H^{(d)}$ represent neurons in layer $l$ excluding the $j$-th neuron. Assuming the output of each neuron contributes independently to the loss, the change in loss due to removing the $j$-th neuron can be expressed as:

	$\displaystyle\left|\Delta\mathcal{L}^{(d)}(h_{l,j}^{(d)})\right|$	$\displaystyle=\left|\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)}=0)\right.$
		$\displaystyle\quad\left.-\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)})\right|,$		(4)

where $\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)}=0)$ is the loss on domain $d$ when the $j$-th neuron is removed, and $\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)})$ is the loss when it is retained. Then, applying a first-order Taylor approximation, this change in the loss for domain $d$ can be estimated as:

	$\displaystyle\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)})$	$\displaystyle=\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)}=0)$		(5)
		$\displaystyle+\frac{\partial\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)})}{\partial{h_{l,j}^{(d)}}}+R_{1}(h_{l,j}^{(d)}),$

where $R_{1}(h_{l,j})$ is the Lagrange remainder term associated with the approximation for domain $d$:

$$R_{1}(h_{l,j}^{(d)})=\frac{\partial^{2}\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)})}{\partial^{2}\delta{h_{l,j}^{(d)}}}(h_{l,j}^{(d)})^{2},$$

(6)

where $\delta\in(0,1)$. The first derivative of the loss function with respect to the neuron’s output for domain $d$ tends to become constant. Consequently, during the final stages of training for domain $d$, the second-order term approaches zero. Therefore, by neglecting the remainder term, the importance evaluation function for a neuron with respect to domain $d$ can be approximated as:

$$|\Delta\mathcal{L}^{(d)}(h_{l,j}^{(d)})|\approx\left|\frac{\partial\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)})}{\partial h_{l,j}^{(d)}}\cdot h_{l,j}^{(d)}\right|.$$

(7)

Thus, the importance score $I_{l,j}^{(d)}$ serves as an approximation to the contribution of neuron $j$ in layer $l$ to the MDMT loss in domain $d$:

	$\displaystyle I_{l,j}^{(d)}$	$\displaystyle=\mathbb{E}{(\mathbf{x}^{d},\mathbf{y}^{d})\sim D_{d}}\left[\left|A_{l,j}^{(d)}\cdot G_{l,j}^{(d)}\right|\right]$		(8)
		$\displaystyle\approx\left|\frac{\partial\mathcal{L}^{(d)}(H^{(d)},h_{l,j}^{(d)})}{\partial h_{l,j}^{(d)}}\cdot h_{l,j}^{(d)}\right|,$

which serves as the foundation for multi-domain consensus-aligned neuron selection, as detailed in the next subsection.

After computing MDMT task-relevant importance scores $I^{(d)}_{l,j}$, we identify consensus-aligned neurons that consistently exhibit high relevance to translation across all domains. These neurons capture domain-invariant translation mechanisms while retaining sensitivity to domain-specific nuances, thereby enabling generalization to unseen domains and efficient fine-tuning for seen ones.

To quantify the relationship between a neuron’s importance and the domain identity, we employ MI measurement. MI can precisely quantify the statistical dependencies between variables, which enables the assessment of whether neurons are multi-domain consensus-aligned. Specifically, we first discretize the continuous importance scores into a set of fixed bins indexed by $i$ to facilitate probability estimation over domain feature. For the $j$-th neuron in the $l$-th FFN layer, we then estimate its MI with the domain label $d$ as follows:

		$\displaystyle\text{MI}_{l,j}=\sum_{i}\sum_{d\in D}\!H(I^{(d)}_{l,j})\!+\!H(d)\!-\!H(I^{(d)}_{l,j},d)$		(9)
		$\displaystyle=\sum_{i}\sum_{d\in D}p(I_{l,j}^{(d)}\!=\!i,d)\log\Bigl(\frac{p(I_{l,j}^{(d)}\!=\!i,d)}{p(I_{l,j}^{(d)}\!=\!i)p(d)}\Bigr),$

where term $p(I_{l,j}^{(d)}=i,d)$ denotes the joint probability that the neuron’s importance score $I_{l,j}^{(d)}$ falls into importance bin $i$ for a data sample belonging to domain $d$. This captures the joint behavior between neuron importance and domain identity, reflecting how frequently a neuron exhibits a particular level of importance within a specific domain. The marginal probability $p(I_{l,j}^{(d)}=i)$ represents the overall distribution of the neuron’s importance score across all domains, indicating how frequently its score falls into bin $i$ regardless of the domain. The term $p(d)$ is the marginal probability of encountering a data sample from domain $d$.

To select our multi-domain consensus-aligned neurons $\mathcal{N}_{\text{MDCA}}$, we avoid selecting neurons with high MI for individual domains, as this can introduce domain-specific noise and lead to overfitting. Instead, we identify neurons that consistently exhibit high MI across all domains:

$$\mathcal{N}_{\text{MDCA}}=\left\{(l,j)\mid\min\text{MI}_{l,j}\geq\gamma\right\},$$

(10)

where $\gamma$ is a threshold, and ensures that a neuron is selected only if its MI with the domain label is at least $\gamma$ in every domain $d\in D$. This allows us to identify neurons that are robustly aligned with multi-domain characteristics across the entire set of domains $D$.

We propose a Neuron-Efficient Fine-Tuning strategy that only updates the multi-domain consensus-aligned neurons.

Formally, let $\mathcal{M}_{\theta}$ denotes a LLM with parameters $\theta$, we freeze all parameters except those directly associated with neurons in $\mathcal{N}_{\text{MDCA}}$. The FFN layer contains three modules, up projection, down projection and gate projection. For each module $m$, let $W_{m}\in\mathbb{R}^{in\times out}$ denotes the weight matrix, where $in$ is the input dimension to the module and $out$ is the output dimension. The number of neurons is identical to $out$. To enable selective gradient updates, we construct a binary mask $M\in\mathbb{R}^{in\times out}$, such that:

$$M_{:i}=\begin{cases}1,&\quad\text{if }i\in\mathcal{N}_{\text{MDCA}}\\ 0,&\quad\text{otherwise}.\end{cases}$$

(11)

This mask is applied during the backward pass to suppress gradients for all non-selected neurons. Specifically, the gradient update for each weight matrix $W_{m}$ is modified as:

$$\nabla{W_{m}}\leftarrow\nabla{W_{m}}\odot M,$$

(12)

where $\odot$ denotes element-wise multiplication. Consequently, only the parameters corresponding to the multi-domain consensus-aligned neurons in $\mathcal{N}_{\text{MDCA}}$ are updated.

Dataset We conducted experiments in 2 translation directions: German-English (De$\Rightarrow$En) and Chinese-English (Zh$\Rightarrow$En). For De$\Rightarrow$En, we used a multi-domain dataset from Aharoni and Goldberg (2020). To validate the generalization capability of the proposed method, we set 4 seen domains (IT, Law, Medical and Subtitles) and 1 unseen domain (Koran). Similarly, For Zh$\Rightarrow$En, we use UM-Corpus (Tian et al. 2014), and also set 3 seen domains (Education, Spoken and Thesis) and 2 unseen domains (Science and Microblog). For neuron-based methods, we randomly sampled 10k data for neuron selection and further sampled 2k data from this set for neuron-efficient fine-tuning. And other baseline methods were fine-tuned on the full 10k data to ensure a fair comparison in terms of data volume.

Backbone Models We take LLaMA2-7B-Chat²²2https://huggingface.co/meta-llama/Llama-2-7b-chat, LLaMA3.1-8B-Instruct³³3https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct, and Qwen2.5-7B-Instruct⁴⁴4https://huggingface.co/Qwen/Qwen2.5-7B-Instruct as the backbone models for training.

Implementation Details For neuron-based methods, we select 1% neurons for fine-tuning. And for our CANEFT, the threshold $\gamma$ (as defined in Eq. 10) is dynamically determined by selecting the top $1\%$ of neurons with the highest MI scores across all domains as consensus-aligned neurons. For LoRA-based baselines, including LoRA, LoRA-MLP, DoRA, we set rank to 8. For LLaMA Pro, we add two adapter layers after layers 16 and 32. All experiments are executed on 8 NVIDIA A40 GPUs.

Baselines We compared our approach with several representative baselines. Base Inference performs zero-shot inference using the original LLMs without any fine-tuning or domain adaptation. Full Fine-tuning fine tunes all parameters of LLMs. LoRA and LoRA-MLP (L-MLP) (Hu et al. 2022) apply low-rank adaptation to enable PEFT. While LoRA adds low-rank matrices to all modules and layers, LoRA-MLP restricts this to only the FFN modules, further reducing the number of trainable parameters. DoRA (Liu et al. 2024) decouples model parameters into magnitude and direction components, updating only the directional component to achieve efficient domain adaptation. LLaMA Pro (Pro) (Wu et al. 2024) extends LLM depth by inserting identity-initialized transformer blocks, and fine-tuning only these additional layers using domain-specific data. For neuron-based method, we adopt LAPE (Tang et al. 2024) as a representative baseline. This method leverages neuron activation probability entropy statistics to detect and fine-tune domain-specific neurons. Random Chosen Neurons (RCN) fine-tunes a randomly selected subset of neurons, comparable in number to those selected by our CANEFT.

Table 1 presents the performance of our method against several PEFT baselines across both seen and unseen domains, evaluated using SacreBLEU⁵⁵5BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a (B) and COMET⁶⁶6https://huggingface.co/Unbabel/wmt22-cometkiwi-da (C).

Overall Performance Our CANEFT consistently delivers the best or second-best performance across all models and domains, underscoring its effectiveness in multi-domain translation. With Qwen2.5, it achieves 45.5 BLEU and 86.6 COMET on De$\Rightarrow$En and 29.6 BLEU and 83.1 COMET on Zh$\Rightarrow$En, surpassing the best-performing baseline by +1.2 BLEU and +0.7 COMET. Comparable gains are observed on LLaMA2 and LLaMA3.1, where CANEFT achieves an average improves by +1.4-1.6 BLEU and +1-1.7 COMET on De$\Rightarrow$En, and +0.7-2 BLEU and +0.2-0.6 COMET on Zh$\Rightarrow$En over the strongest competing baselines. Importantly, while full fine-tuning is the strongest baseline on average and achieves results comparable to CANEFT across most domains, CANEFT further surpasses it while updating only 1% of the parameters.

CANEFT also demonstrated substantial gains on unseen domains. For example, in Zh$\Rightarrow$En, CANEFT showed enhanced robustness when handling linguistically diverse domains like Science (formal, technical) and Microblog (informal, colloquial). These domains represent significant textual distribution divergences from the training data, underscoring the enhanced robustness of our method in handling linguistic style variations and cross-domain adaptation.

Notably, LLaMA Pro slightly outperforms CANEFT in the Law domain (e.g., +0.8 BLEU on Qwen2.5), likely due to its 4 times additional parameters capturing legal-specific patterns. However, this comes at the expense of generalization, as LLaMA Pro suffers substantial drops in IT (-8.6 BLEU) and Subtitles (-9 BLEU) compared to Base Inference. Similarly, most PEFT baselines show uneven domain gains, improving in some domains while degrading in others. This reflects parameter interference during multi-domain fine-tuning and underscores the risk of catastrophic forgetting. In contrast, our method maintains robust performance across all domains, highlighting its advantage in balancing domain specialization with generalization.

Table 1 also shows that CANEFT introduces no additional parameters and updates only a minimal set of critical neurons. Although PEFT methods like LoRA and DoRA involve fewer trainable parameters, their performance across domains is significantly inferior, with some even underperforming base inference. In general, CANEFT strikes a superior balance between efficiency and translation quality. Moreover, CANEFT significantly outperforms the RCN and LAPE across all domains, demonstrating that the selected neurons capture meaningful multi-domain consensus-aligned information rather than arbitrary features. Furthermore, the consistent gains across 3 distinct backbones further demonstrate that our method is model-agnostic and applicable to various LLM-based translation frameworks.

w/o MDMTN: This variant computes MI between neurons and domain labels without identifying MDMT task-relevant neurons. As a result, MI is estimated over a larger and noisier neurons, making it harder to isolate meaningful consensus-aligned signals. As shown in Table 2, this leads to a noticeable performance drop across domains. These results underscore the importance of task-specific filtering prior to MI computation. In the absence of this filtering, the selected neurons include features not relevant to the task, which degrades the quality of adaptation.

w/o MDCAN: This variant omits the multi-domain consensus-aligned neuron selection and directly fine-tunes the top 1% of neurons based solely on importance scores, resulting in a clear performance drop. While MDMTN ensures task relevance, not all identified neurons could reach a consensus alignment across domains. Without MDCAN, fine-tuning a less-refined neuron set increasing risks of parameter interference across domains and dilutes domain-invariant signals. The MI-based MDCAN step is thus essential for identifying a harmonized neuron subset that enables robust and generalizable adaptation across domains.

We investigate how varying the proportion of selected multi-domain consensus-aligned neurons (from 0.25% to 1.5%) influences performance using LLaMA3.1-8B-Instruct.

As shown in Figure 2, increasing the ratio of selected neurons consistently improves translation quality across both BLEU and COMET metrics. Notably, the IT and Law domains exhibit the most substantial improvements, with BLEU gains of 3–6 points, suggesting that incorporating more consensus-aligned neurons facilitates more effective cross-domain knowledge transfer. COMET scores follow a similar upward trend, reflecting enhanced semantic adequacy and fluency. However, performance plateaus beyond 0.75% and even degrades when over 1.25% in most domains, suggesting that most informative neurons have already been leveraged, and additional parameters contribute marginal returns. These results confirm that fine-tuning only 0.5%–1% of well-chosen neurons is sufficient to achieve strong multi-domain translation performance while avoiding unnecessary parameter updates.

To assess the relevance of the selected consensus-aligned neurons in MDMT, we analyze their behavior from a gradient perspective. We compare the mean absolute parameter changes in LLaMA3.1-8B’s FFN components (gate_proj, up_proj, down_proj) under two neuron selection strategies: random selection and multi-domain consensus-aligned neuron selection. Model layers are grouped into three ranges: lower (layers 0–10), middle (layers 11–20), and higher (layers 21–32). As shown in Figure 4, consensus-aligned neurons consistently exhibit larger gradient updates across all components and layers.

These substantial gradient differentials suggest that consensus-aligned neurons possess higher optimization potential for MDMT. Our selection method effectively identifies parameters most responsive to multi-domain adaptation, enabling the LLM to refine these neurons for improved translation performance. Such neurons undergo more meaningful, task-specific transformations, whereas randomly selected neurons display weaker and less focused changes. Moreover, the pronounced shifts in up_proj and down_proj highlight their pivotal role in domain adaptation, underscoring their importance in facilitating deeper integration of domain knowledge into the model’s representations.

To visually demonstrate the distribution of multi-domain consensus-aligned neurons across different layers in LLaMA3.1-8B-Instruct. Figure 3 provides heatmaps of the selected neurons across the 32 layers for each of the three FFN projections. The selected neurons are unevenly distributed, with higher density in lower and middle layers particularly in gate_proj and up_proj. In contrast, down_proj neurons are selected sparsely, indicating a relatively limited role in cross-domain generalization. This pattern aligns with the understanding that lower layers capture general syntactic and semantic features beneficial for transfer, while higher layers encode more domain-specific information thus contributing fewer consensus-aligned neurons.

The salience of gate_proj and up_proj points to their critical role in regulating information flow and enriching representations for multi-domain processing. gate_proj modulates the activation gate, while up_proj expands hidden representations. The high density of selected neurons in these components suggests that consensus knowledge are more effectively encoded and propagated during the gating and expansion phases, rather than in the dimensionality reduction stage (down_proj). This is further supported by Figure 4, where these components show stronger fine-tuning signals, indicating their structural and functional significance in encoding transferable knowledge.