ViperEkura
/
AstrAI-paper
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix duplicate paragraph and misleading table caption

This commit is contained in:
ViperEkura 2026-07-04 22:39:16 +08:00
parent b40fcc86c7
commit 9534316bac
1 changed files with 29 additions and 24 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

53
main.tex
View File

@ -381,10 +381,10 @@ for the base model.
\begin{table}[H] \begin{table}[H]
\centering \centering
\caption{Representative IFD samples (base model sorted by descending IFD).} \caption{Representative IFD samples covering four patterns.}
\label{tab:ifd_examples} \label{tab:ifd_examples}
\small \small
\begin{tabular}{@{}c c c c c c p{4.5cm}@{}} \begin{tabular}{@{}c c c c c c c p{4.5cm}@{}}
\toprule \toprule
\textbf{Idx} & \textbf{Idx} &
\textbf{Base IFD} & \textbf{Base IFD} &
@ -392,18 +392,16 @@ for the base model.
\textbf{$L_{\text{cond}}^{\text{base}}$} & \textbf{$L_{\text{cond}}^{\text{base}}$} &
\textbf{$L_{\text{uncond}}^{\text{base}}$} & \textbf{$L_{\text{uncond}}^{\text{base}}$} &
\textbf{$L_{\text{cond}}^{\text{ckpt}}$} & \textbf{$L_{\text{cond}}^{\text{ckpt}}$} &
\textbf{Instruction (truncated)} \\ \textbf{$L_{\text{uncond}}^{\text{ckpt}}$} &
\textbf{Instruction} \\
\midrule \midrule
0 & 4.605 & 1.525 & 12.38 & 2.69 & 3.77 & Complete the following analogy \dots \\ 0 & 4.605 & 1.525 & 12.38 & 2.69 & 3.77 & 2.47 & Complete analogy: loud is to quiet as day is to \\
1 & 4.331 & 0.645 & 11.44 & 2.64 & 1.66 & Classify the following text \dots \\ 1 & 3.741 & 0.702 & 11.75 & 3.14 & 2.17 & 3.09 & Label news article as ``Political'' or ``Entertainment'' \\
2 & 3.741 & 0.702 & 11.75 & 3.14 & 2.17 & Label the following news article \dots \\ 2 & 1.044 & 0.089 & 3.50 & 3.35 & 0.28 & 3.10 & Find the capital of Spain \\
3 & 0.977 & 0.904 & 2.57 & 2.63 & 2.24 & Describe the role of a project manager \\ 3 & 1.056 & 0.147 & 4.09 & 3.88 & 0.60 & 4.07 & Edit sentence for correct grammar: ``I were just going to'' \\
4 & 0.977 & 0.915 & 2.19 & 2.25 & 1.98 & Select a historical figure \dots \\ 4 & 0.977 & 0.904 & 2.57 & 2.63 & 2.24 & 2.48 & Describe the role of a project manager \\
5 & 0.977 & 0.949 & 2.57 & 2.63 & 2.26 & Write the lyrics for an upbeat song \dots \\ 5 & 0.370 & 0.249 & 1.37 & 3.70 & 0.85 & 3.42 & Convert the given paragraph to a list \\
6 & 0.977 & 0.925 & 2.94 & 3.00 & 2.62 & Explain how neural networks \dots \\ 6 & 0.307 & 0.062 & 0.70 & 2.29 & 0.15 & 2.43 & Remove third-person words from sentence \\
7 & 0.370 & 0.249 & 1.37 & 3.70 & 0.85 & Convert the given paragraph to a list \\
8 & 0.338 & 0.197 & 0.98 & 2.91 & 0.55 & Insert a suitable greeting \dots \\
9 & 0.307 & 0.062 & 0.70 & 2.29 & 0.15 & Remove third-person words \dots \\
\bottomrule \bottomrule
\end{tabular} \end{tabular}
\end{table} \end{table}
@ -431,28 +429,35 @@ Over $N=3000$ SFT samples:
\subsection{Observed Patterns} \subsection{Observed Patterns}
\paragraph{High-IFD samples (base IFD $> 3$, e.g.,~rows~0--2).} \paragraph{High-IFD samples (base IFD $> 3$, e.g.,~rows~0,~1).}
These are tasks requiring task-intent comprehension: analogy completion, These are tasks requiring task-intent comprehension: analogy completion
text classification, article labeling. In the base model (15B pretraining), conditional and article labeling. In the base model (15B pretraining), conditional
loss is extremely high ($L_{\text{cond}} \approx 12$), meaning the loss is extremely high ($L_{\text{cond}} \approx 12$), meaning the
instruction still acts as noise. After 1K SFT steps, IFD drops instruction still acts as noise. After 1K SFT steps, IFD drops
sharply (e.g., $4.605 \rightarrow 1.525$), demonstrating sharply (e.g., $4.605 \rightarrow 1.525$), demonstrating
that SFT teaches the model to interpret and follow abstract task that SFT teaches the model to interpret and follow abstract task
descriptions. descriptions.
\paragraph{Low-IFD samples (base IFD $< 0.4$, e.g.,~rows~7--9).} \paragraph{Low-IFD samples (base IFD $< 0.4$, e.g.,~rows~5,~6).}
These are formatting or extraction tasks: ``Convert paragraph to list,'' These are formatting or extraction tasks: ``Convert paragraph to list,''
``Remove third-person words,'' ``Insert a greeting.'' Unconditional ``Remove third-person words.'' Unconditional
loss is much higher than conditional loss even in the base model, loss is much higher than conditional loss even in the base model,
because the instruction naturally constrains the output space. The because the instruction naturally constrains the output space. The
pattern persists after SFT but with lower absolute values. pattern persists after SFT but with lower absolute values.
\paragraph{Mid-range samples (base IFD $\approx 0.98$, e.g.,~rows~3--6).} \paragraph{Mid-range with large drop (e.g.,~rows~2,~3).}
These cover factual Q\&A and generation tasks: ``Describe the role of These are factual QA or grammar correction tasks. Base IFD is
a project manager,'' ``Write lyrics for a song,'' ``Explain how neural $\approx 1.05$ (instruction has little effect), but after SFT
networks work.'' In the base model IFD $\approx 1$ (instruction has IFD drops to $\approx 0.1$ as the model learns the precise answer
little effect); after SFT IFD drops to $\approx 0.9$, driven by (e.g., ``Madrid'' for ``capital of Spain''), making conditional loss
a clear reduction in conditional loss. near-zero while unconditional loss remains high.
\paragraph{Mid-range with small drop (e.g.,~row~4).}
These are open-ended generation tasks (``Describe the role of a
project manager''). Base IFD $\approx 0.98$; after SFT it drops
only modestly to $\approx 0.9$, since both conditional and
unconditional losses decrease proportionally without a memorized
target.
\paragraph{Cross-model correlation.} \paragraph{Cross-model correlation.}
The moderate Pearson correlation ($r = 0.38$) suggests that while The moderate Pearson correlation ($r = 0.38$) suggests that while