PACE: Defying the Scaling Hypothesis of Exploration
in Iterative Alignment for Mathematical Reasoning

In real-world scenarios, the verifier $\mathcal{V}$ is rarely a perfect oracle. It may contain noise (e.g., labeling errors) or structural biases (e.g., reward hacking spots). We demonstrate that increasing $N$ turns the sampling process into an adversarial miner against the verifier. Consider a challenging problem $x$. We partition the solution space into the true solution set $\mathcal{T}=\{y\mid R(y)=1\}$ and the hallucinated set $\mathcal{H}=\{y\mid R(y)=0,\mathcal{V}(y)=1\}$. The generation process is modeled as follows: the model produces a valid reasoning path with probability $\alpha=P(y\in\mathcal{T}\mid x)$. If the reasoning fails, the model may still trigger a verifier flaw with a conditional intrinsic defect rate $\epsilon=P(y\in\mathcal{H}\mid y\notin\mathcal{T})$. Consequently, the verifier pass probability $\rho=P(\mathcal{V}(y)=1)$ is $\rho=\alpha+(1-\alpha)\epsilon$. Since the verifier cannot distinguish between true and spurious solutions, the probability that a selected positive sample is a false positive (FP) is given by:

From a prior perspective, the FP probability (2) is formally independent of $N$. However, since DPO training necessitates the construction of positive-negative pairs, increasing $N$ effectively induces a non-random drift in the distribution of training samples: as multiple attempts raise the probability of passing the verifier, it introduces challenging tasks that would rarely yield a correct solution under smaller $N$. These newly incorporated samples typically exhibit an extremely low empirical pass rate $\hat{\rho}$. For instance, when the sample budget increases from $N-1$ to $N$, the marginal observed pass rate for a newly activated sample is merely $1/N$. Based on this observation, we employ Maximum Likelihood Estimation (MLE) to perform a posterior correction on (2). Specifically, the MLE of the model’s intrinsic reasoning capability $\hat{\alpha}$ is given by (see Appendix A for details):

where $\Pi_{[0,1]}$ denotes the projection onto the interval $[0,1]$. Combining the monotonicity of (3) and (2), it follows that the FP probability decreases monotonically with the $\hat{\rho}$. Each time the sample budget increases by 1, the empirical pass rate $\hat{\rho}$ of newly added samples successively decreases ($\sim 1/N$). Consequently, while increasing $N$ expands the scale of the training set, it also systematically introduces marginal samples with a heightened risk of being false positives, thereby exposing the model to significant reward pitfalls. In contrast, PACE’s constrained exploration minimizes the “surface area” exposed to verifier noise, yielding a robust training signal even under imperfect supervision.

We previously discussed how increasing $N$ introduces more false positives samples. However, even with a perfect validator $\mathcal{V}$ (i.e., $\epsilon=0$), a large $N$ still causes deviation from the trust region. The core premise of stable RL methods, such as PPO, is to maintain the trust region, ensuring that the target distribution does not excessively diverge from the current policy. We further demonstrate that the BoN method with a large $N$ implicitly violates this principle.

To improve model performance, we define a target distribution $\pi_{\text{new}}(y|x)$ such that the probability of producing the correct answer for a given sample is at least $\eta>\alpha$. During the fine-tuning or alignment process, the objective is for the trained model distribution to converge toward $\pi_{\text{new}}$. However, achieving this target distribution inevitably entails a deviation from the reference distribution. If there is a significant gap between the original success probability $\alpha$ and the target probability $\eta$, the KL divergence between the two distributions must possess a non-trivial lower bound. According to the Data Processing Inequality, the minimum cost of this distributional shift can be expressed as:

As previously discussed, the introduction of new samples as $N$ increases typically corresponds to a lower observed pass rate, which tends to result in a lower model accuracy $\alpha$, thereby leading to greater distributional deviation. Under the assumption of a perfect verifier ($\epsilon=0$), according to (3), the posterior estimate of the model accuracy is equivalent to the sample pass rate $\hat{\rho}$. For an incremental budget increase from $N-1$ to $N$, considering a newly introduced sample (where $\hat{\alpha}=1/N$), the lower bound (LB) in (4) evolves as:

where $H(\eta)$ denotes the binary entropy function. The detailed derivation is provided in Appendix B.

When training on datasets constructed via $N=16$ (or $64$), we force the model to update towards outlier trajectories. This large distribution shift ($\Delta\pi$) often exceeds the stable learning rate capacity of the optimizer, leading to policy collapse or overfitting to specific, narrow reasoning paths, as observed empirically in  Gao et al. (2023) and  Lambert and Calandra (2024). By constraining $N=2$, PACE implicitly acts as a regularizer, keeping the target distribution within the proximal trust region of the current policy.

Finally, we consider the quality of the learning signal. The gradient of the DPO loss is scaled by $\sigma(\hat{r}_{\theta}(y_{l})-\hat{r}_{\theta}(y_{w}))$, where $\hat{r}_{\theta}$ is the implicit reward. In DPO-R1 ($N=16$), $y_{w}$ is the “best” of many, while $y_{l}$ is often a random “loser”. Standard DPO-R1 constructs Easy Pairs where $y_{w}$ and $y_{l}$ are semantically distant, leading to optimization shortcuts. In contrast, PACE constructs Proximal Pairs that lie close to the decision boundary. Theoretically, optimizing against these Hard Negatives maximizes the information gain per update, forcing the model to refine specific reasoning steps rather than memorizing surface-level heuristics.

To avoid the $\Omega(\log N)$ distribution shift inherent in high-$N$ sampling (Section 3.2), PACE restricts the sampling budget to the minimal sufficient set required to estimate local epistemic uncertainty. For a given prompt $x$, we generate exactly two trajectories $\mathcal{Y}=\{y_{1},y_{2}\}$ from the current policy $\pi_{\theta}$. Generated via stochastic sampling with temperature $T=1.0$. This probes the local variance of the policy.

Theoretical Justification: This $N=2$ strategy serves as a Proximal Probe. Unlike $N=16$, which searches for outliers far from the mean, $N=2$ assesses whether the mass of the current distribution $\pi_{\theta}(\,\cdot\,|x)$ covers the correct solution. If both samples fail, it indicates a lack of capability (or high uncertainty) within the trust region, signaling the need for correction rather than further random searching.

Trace-Aware Correction. Unlike standard self-correction which relies on generic prompts, we utilize a Trace-Aware strategy. We provide the model with its own error trace $y_{err}$ and the ground truth label $y^{*}$. The instruction requires the model to identifying the logical divergence between $y_{err}$ and $y^{*}$ and generate a corrected path $y_{fix}$.

The Quality Gate (Self-Verification). A known risk of label-conditioned generation is “rationalization”, where the model hallucinates a flawed reasoning chain that coincidently ends in the target number. To filter these toxic samples, we apply a strict Consistency Filter:

1. 1.

   We strip the final answer from $y_{fix}$, retaining only the reasoning steps $r_{fix}$.

2. 2.

   We verify if the reasoning strictly entails the answer: $\mathbb{I}(\text{Greedy}(x,r_{fix})==y^{*})$.

Trajectories that fail this check are discarded, ensuring only grounded, logically valid corrections enter the training pool.

The core innovation of PACE lies in how it constructs the preference dataset $\mathcal{D}$ for DPO. We categorize the outcomes of Phase I into two scenarios, each yielding a distinct type of gradient signal. The final $y_{fix}$ has been obtained.

Scenario A: The Natural Pair (Exploration Success) If one sample is correct ($y_{w}$) and the other incorrect ($y_{l}$) during the initial $N=2$ rollout, we construct a standard preference pair. If one sample is correct ($y_{w}$) and the other incorrect ($y_{l}$) during the initial $N=2$ rollout, we construct a standard preference pair. This captures data within the model’s current competence, reinforcing known capabilities.

Scenario B: The Synthetic Pair (Exploration Failure) If both initial samples are incorrect, we utilize the refined trajectory $y_{fix}$ from Phase II. We select the original incorrect sample, denoted as $y_{hard\_neg}$, to serve as the loser.

Why this matters (Gradient Efficiency): In standard BoN, the winner and loser are often semantically distant (a random lucky guess vs. a random failure), leading to high variance updates. In PACE, $y_{fix}$ is a Proximal Counterfactual to $y_{greedy}$, they share the same prompt and potentially similar initial steps, diverging only at the critical error. Optimizing against such Hard Negatives provides a high-density gradient signal, effectively suppressing the specific error mode without disrupting the rest of the policy.

Finally, we use $\mathcal{L}_{\text{DPO}}$ optimizes the relative log-ratios of the pairs constructed in Phase III:

Standard Best-of-N strategies suffer from static computational rigidity: the budget is fixed at $N$ regardless of prompt difficulty. In contrast, PACE employs a dynamic allocation strategy, expending additional compute only when epistemic uncertainty is high (i.e., exploration failure). Given that the verification time is relatively negligible, we will disregard this portion of the time in our discussion.

Let $C_{gen}$ be the unit cost of generating one trajectory. The expected computational cost per prompt for PACE is:

where $P(\text{fail}|x,\pi_{\theta})$ is the probability that the initial proximal exploration ($N=2$) fails to yield a correct solution, and $K$ is the refinement budget (typically $K=1$). This formulation yields three critical efficiency properties: 1) Amortized Efficiency: For easy instances where $P(\text{fail})\to 0$, the cost converges to the lower bound $2C_{gen}$. Standard BoN ($N=16$) remains statically inefficient at $16C_{gen}$. 2) Bounded Worst-Case: Even for hard failures ($P(\text{fail})\to 1$), the cost is bounded by $(2+K)C_{gen}$. With $K=1$, PACE achieves a $5.3\times$ speedup over the $N=16$ baseline in the worst-case scenario. 3) Self-Optimizing Dynamics: Crucially, as the policy $\pi_{\theta}$ improves during DPO training, the failure rate $P(\text{fail})$ naturally diminishes. Consequently, PACE becomes progressively cheaper throughout the training lifecycle, whereas BoN costs remain constant.

Benchmarks: We follow  Yang et al. (2024) using common English math benchmarks, Math (Hendrycks et al., 2021b), and using different sets of math tests at different stages to test stronger math skills, such as Minerva Math (Lewkowycz et al., 2022), Gaokao 2023 En (Liao et al., 2024), Olympiad Bench (He et al., 2024) , and AMC 23 (AoPS, 2023). We report greedy performance on all benchmarks in the zero-shot setting, except for the Amc23. Considering the limited size of Amc23, we sample 8 times (avg@8) for each question to mitigate randomness. During the answer generation process for these two challenge datasets, we use a temperature of 0.1 and a top_p of 0.95.

Baselines and Models: To rigorously isolate the contribution of our proposed method, we compare PACE against three distinct tiers of baselines: DPO-R1 ($N=8$ or 16): The current industry standard for iterative alignment (Dubey et al., 2024). For each prompt, we sample $N=$8 or 16 trajectories at temperature $T=1.0$. The trajectory maximizing the verifier score is selected as $y_{w}$, and a random failing trajectory is selected as $y_{l}$. This baseline represents the high-resource upper bound. DPO-R1 ($N=2,4$) (The Compute-Matched Baseline): A naive implementation of DPO restricted to the same sampling budget as PACE ($N=2,4$). It selects the best of $N$ samples without any corrective mechanism. This baseline is critical for ablation: comparisons against it demonstrate that PACE’s gains stem from the quality of the corrective pairs (Hindsight Refinement). We evaluate our method on three state-of-the-art RL-tuned models (Llama-3.1-8B-Instruct and Qwen3-4B/8B-Instruct), representing the strongest reasoning capabilities under the 8B parameter class. Given that the base models have already been saturated with complex reasoning capabilities through intensive RL (GRPO/PPO), training on extended Think trajectories via DPO is computationally redundant and resource-intensive for tasks that do not require long-form thought. Consequently, we adopt a “No-Think” mode to ensure efficient logical alignment while avoiding the unnecessary overhead of over-optimizing intermediate reasoning tokens on targeted datasets.

We present the evaluation results across five mathematical reasoning benchmarks in Table 1. The results reveal a critical limitation in standard iterative alignment strategies and highlight the efficiency of our proposed method.

Efficiency We compared the average seconds required for each method to process a single sample, with 40,000 samples processed per iteration. In the traditional DPO-R1 baseline, the processing time per sample increases linearly with the sampling budget $N$. For example, on the Qwen3-4B model, DPO-R1 (High, $N=16$) takes up to 5.2 seconds, while PACE completes in just 1.0 seconds, achieving approximately 5.2$\times$ acceleration.

Effectiveness Experimental results demonstrate that PACE achieves significantly higher sample utilization efficiency than baseline models while maintaining extremely low computational costs ($2<N<3$). Taking Llama-3.1-8B-Instruct as an example, PACE achieved an average score of 22.7 by generating only 6,197 training pairs. This significantly outperforms DPO-R1 (High, $N=16$), which consumed 16,241 pairs but only achieved 21.1 points. Increasing the sampling budget $N$ grows the dataset size, but it triggers an inverse scaling law where more data actually yields lower performance.

The Luck Trap happens because brute-force sampling at high $N$ selects lucky guesses, solutions with correct answers but broken logic, from the distribution’s tail. In contrast, PACE employs Hindsight Refinement to transform discarded failed attempts into high-quality proximal refinement pairs. On Qwen3-4B, while PACE’s average score (57.9) is nearly on par with DPO-R1 (High)’s 58.1, its core advantage lies in its exceptionally high computational efficiency. PACE requires only a minimal sampling time (1.0) to generate as many as 10,717 high-quality training pairs. In contrast, DPO-R1 (High) incurs a substantial sampling cost (5.2) of $N=16$ to generate 8,964 pairs. This shows that PACE achieves performance comparable to high-budget benchmarks with approximately 5 $\times$ acceleration, maintaining a significant advantage in output efficiency per sample and the use of computational resources.

To dissect the contribution of each component in PACE, we conducted an ablation study on the Llama-3-8B model (Table 2). The results reveal that Trace-Aware Refinement is the most critical factor; replacing it with Label-Only guidance leads to the most significant performance degradation (-4.7%). This confirms that conditioning on the specific error trace is essential to force comparative debugging and prevent the model from merely rationalizing the target label. Furthermore, substituting Proximal Pairs with random negatives results in a -2.9% drop, validating our hypothesis that optimization against hard, on-policy negatives yields higher-fidelity gradient signals than random baselines. Finally, removing the Consistency Filter causes a -1.7% decline, underscoring the necessity of quality gating to prevent hallucinated reasoning from polluting the training dynamics.

Standard rejection sampling strategies implicitly assume a high-fidelity verifier. However, in real-world scenarios, reward signals are often noisy or exploitable. To empirically validate our theoretical hypothesis (Section 3.1), we conduct a stress test designed to induce Reward Hacking. To further investigate the resilience of PACE, we track the iterative alignment of Llama-3.1-8B under 20% label noise (Table 3, e.g., changing the ground truth to 0), a setting we term the Small-Digit Trap. We report the average results of the dataset in the main table. Our analysis reveals a fundamental divergence in how methods handle corrupted feedback.

While $N=16$ maintains a higher absolute accuracy in later iterations, its training buffer is heavily compromised by a 28.1% Noise Hit Rate (NHR). However, with the final iteration, the effectiveness declined sharply. This suggests that high-$N$ exploration acts as an adversarial miner, effectively synthesizing hallucinations to match erroneous ’0’ labels. Although the sheer volume of correct samples initially buffers this toxicity, the high NHR poses a risk of long-term policy degradation. Conversely, $N=2$ exhibits low NHR (4.3%) but suffers from severe signal starvation (Yield Drop 11.8%), causing it to stall at 12.7% accuracy.

PACE resolves this trade-off by achieving a low NHR (4.9%) comparable to $N=2$, while recovering 42.8% more valid signals through hindsight refinement. The 13.7% yield drop in PACE is a deliberate logical purge, the consistency filter rejects synthesized paths that cannot be logically reconciled with the erroneous labels. This results in the highest iterative performance, proving that synthesized purity is a more robust scaling axis than mined volume for alignment.

Generalization We conducted a series of out-of-domain (OOD) benchmarks on the Llama-3.1-8B model, including symbolic reasoning (BBH, Suzgun et al. 2023), programming (HumanEval, Chen et al. 2021, pass@1), general knowledge (MMLU, Hendrycks et al. 2021a), and factuality (TruthfulQA, Lin et al. 2022, MC1). As shown in Table 4, traditional sampling strategies exhibit significant performance degradation across multiple OOD dimensions when increasing the sampling budget $N$ (from 2 to 8), a classic manifestation of adversarial mining causing models to overfit to validator rewards and thereby compromise general capabilities. In contrast, PACE leverages its Proximal Refinement mechanism to substantially enhance mathematical reasoning capabilities while nearly perfectly preserving the model’s original distribution. Its average OOD score (55.2) not only significantly outperforms the computationally more expensive BoN (8) baseline (54.0), but also demonstrates positive transfer learning on programming tasks (1.4% improvement over BoN(2)). This demonstrates that PACE achieves efficient logical alignment without sacrificing foundational model intelligence, providing a robust scientific paradigm for resolving the performance-generalization paradox in iterative alignment.

Iterative Dynamics Figure 2 reveals a Quality-Quantity Trade-off in iterative alignment. The DPO-R1 with high sampling budget ($N=16$) exhibits a classic overfitting trajectory: it starts with strong performance (21.7% in Iter 1) by maximizing coverage of the solution space. However, this advantage is transient; accuracy degrades to 21.1% by Iteration 4. We attribute this collapse to the unsupervised nature of rejection sampling, as the model improves, $N=16$ retrieves an increasing number of spurious positives (hallucinated reasoning with correct answers) from the distribution tail, which poison the reward signal. In contrast, PACE (red line) demonstrates a robust scaling law. Although it begins with lower accuracy (19.9%), a direct result of our Consistency Filter rejecting low-confidence lucky guesses, it is the only method that maintains monotonic improvement, reaching 22.7% in Iteration 4. This confirms that for iterative learning, the purity of the training signal (ensured by refinement and filtering) is a more critical determinant of long-term performance than the sheer volume of exploration.