diff --git a/main.tex b/main.tex index 2063933..69c1d3d 100644 --- a/main.tex +++ b/main.tex @@ -91,6 +91,44 @@ Two storage backends serve the DataLoader: A resumable distributed sampler provides seed-based shuffle with epoch/iteration resume. +\subsection{SFT Data Cleaning} + +For supervised fine-tuning (SFT), raw data requires additional curation +beyond pretraining tokenization. {\sc Alembic}~\cite{alembic} is a companion +pipeline that handles SFT data generation, cleaning, and quality scoring: +three-generation strategies (topic-driven, seed-driven, self-instruct), +built-in cleaning (HTML/URL/markdown removal, char/word repetition filters), +and a MinHash-based near-duplicate detection system~\cite{broder1997syntactic}. +Given a set of $P$ hash functions ($P=128$) and a text $T$, the MinHash +pipeline proceeds as follows: + +\begin{enumerate}[nosep,leftmargin=*] + \item \textbf{Tokenization}: $T$ is split into character $n$-grams + ($n=3$): + \begin{equation} + \Gamma(T) = \{\,c_i c_{i+1} c_{i+2} \mid i = 1,\dots,|T|-2 \,\}. + \end{equation} + \item \textbf{Signature}: For each hash function $h_k$, the minimum hash + value over all $n$-grams forms the $k$-th element of the fingerprint: + \begin{equation} + s_k = \min_{t \in \Gamma(T)} h_k(t), \qquad + h_k(t) = \operatorname{SHA256}(42 : k : t)_{[0:63]}. + \end{equation} + The full fingerprint is $\mathbf{s} = (s_1,\dots,s_P)$. + \item \textbf{Similarity}: The Jaccard similarity between two sets is + estimated by the fraction of agreeing fingerprint positions: + \begin{equation} + \widehat{J}(\mathbf{s}^{(a)},\mathbf{s}^{(b)}) = + \frac{|\{\,k \mid s^{(a)}_k = s^{(b)}_k \,\}|}{P}. + \end{equation} + \item \textbf{Filtering}: Samples are processed sequentially; a sample + is dropped if $\widehat{J}(\mathbf{s}, \mathbf{s}') \ge 0.7$ for any + previously kept sample $\mathbf{s}'$. +\end{enumerate} + +An optional LLM-as-Judge scoring module provides multi-dimensional +quality scores that can be used to filter low-quality samples. + % ====================================================================== \section{Model Architecture} % ====================================================================== @@ -295,14 +333,23 @@ J.~Ainslie, J.~Lee-Thorp, M.~de Jong, Y.~Zemlyanskiy, F.~Lebr\'on, S.~Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. \textit{EMNLP}, 2023. +\bibitem{alembic} +Alembic Contributors. \textit{Alembic: A lightweight LLM-driven SFT data +generation, cleaning, and scoring pipeline.} +\url{https://github.com/ViperEkura/Alembic}, 2026. + \bibitem{astrai} AstrAI Contributors. \textit{AstrAI: An open-source training and inference framework for Transformer language models.} \url{https://github.com/ViperEkura/AstrAI}, 2026. +\bibitem{broder1997syntactic} +A.~Z.~Broder. On the resemblance and containment of documents. +\textit{SEQUENCES '97}, 1997. + \bibitem{ieee754} IEEE Computer Society. \textit{IEEE Standard for Floating-Point Arithmetic}, -IEEE Std 754-2008, 2008. +IEEE Std 754-2019, 2019. \bibitem{loshchilov2019adamw} I.~Loshchilov, F.~Hutter. Decoupled weight decay regularization.