Similarly to SpeechTokenizer (Zhang et al, 2024b), we distill semantic information from a self-supervised model (WavLM8 (Chen et al, 2022) in our case) into the first level of the RVQ. WavLM projects a 16kHz waveform into 1024-dimensional embeddings sampled at 50Hz, while Mimi projects a 24kHz waveform into 512-dimensional at 12.5Hz. During training, we thus produce targets for distillation by downsampling the input waveform to 16kHz before computing WavLM embeddings followed by average pooling with a stride of 4 and a kernel size of 8, to reach 12.5 Hz. Interestingly, we observed that it was critical for performance to perform this average pooling in a non-causal way, which is compatible with streaming inference as these embeddings are only used during training. We apply a linear projection with an output dimension of 1024 to the output of the first RVQ level, parallel to the actual embedding going into the decoder. We then compute a cosine distance between the output of the first quantizer and the transformed WavLM embeddings, to perform distillation. Table 3 shows that this distillation loss conflicts with reconstruction and adversarial losses targeting quality. Indeed, while distillation significantly improves the phonetic discriminability of the first quantizer (as measured by ABX (Schatz et al, 2013)), it also affects audio quality negatively. We hypothesize that this is due to distilling semantic information into the first level of a single RVQ: As higher-order quantizers operate on the residual of the first one, the latter needs to trade audio quality for phonetic discriminability. We address this issue by proposing a split RVQ. Rather than a single RVQ with 8 levels, we distill semantic information into a plain VQ and apply an RVQ with 7 levels in parallel. We sum their outputs, such that while both can be used for reconstruction, we remove the constraint that acoustic information should be conserved in the residual of the semantic quantizer. <PIC> illustrates this architecture and Table 3 shows that this solution provides a better semantic-acoustic trade-off overall. Moshi is a multi-stream speech-to-speech Transformer model, which allows for full-duplex spoken dialogue with a user thanks to an innovative architecture summarized in <PIC>. Moshi is built on top of Helium, a text LLM which we build from scratch (Section 3.2), relying on high-quality text data to provide strong reasoning abilities to the model. We also propose Inner Monologue (Section 3.4.4), a training and inference procedure in which we jointly model text and audio tokens. This allows the model to fully exploit the knowledge imparted from the text modality, while remaining a speech-to-speech system. To enable real-time dialogue, we also design Moshi as a multi-stream architecture from the get-go (Section 3.4.3): The model is able to both speak and listen to the user at the same time, and does not need to explicitly model speaker turns. In addition, to capture the input user audio and output Moshi’s voice with high quality and in an efficient manner, we propose Mimi (Section 3.3), a neural audio codec combining semantic and acoustic information into a single tokenizer by using residual vector quantization and knowledge distillation. To jointly model the audio streams from Moshi and the user, as well as Moshi’s text tokens, we rely on a Depth Transformer compatible with streaming inference (Sections 3.4.1, 3.4.2).


Evaluation Protocol
We now discuss how we evaluate the retrieval performance. The most widely adopted retrieval paradigm is to rank passages in the corpus based on the relevance scores predicted by the retrieval model and retrieve the Top-K candidates to form a ranked list. The performance of the retrieval model is then assessed based on the ranked list using well-defined ranking metrics such as NDCG@K and MAP@K. However, such metrics are not continuous due to their discrete nature and reliance on a cutoff parameter, K. Because the ranking metrics of a ranked list would not change unless the sequence of the passages changes, these ranking metrics are not sensitive to the changes of model outputs in many cases. Also, with the cutoff in ranking metric, a positive passage only contributes to the metric when ranked within the top K results. If it falls beyond K, whether at K+1 or further, it has no impact on the metric score. The characteristics of these existing ranking metrics make them unsuitable for the investigation of scaling laws in dense retrieval. To solve these problems, we propose to utilize a continuous metric that sensitively reflects the overall retrieval capability of the models. Inspired by the analysis of scaling laws in large language models, which utilize the perplexity of token generations as evaluation metrics, we propose to use the contrastive entropy as our evaluation metric. Formally, for each query-passage pair in the test set, we randomly select a fixed number (256 in this paper) of negative passages and define the contrastive entropy as: \begin{align} -\log{\exp\left(s(q_i, p_{i}^+;\theta)\right) \over {\exp\left(s(q_i, p_{i}^+;\theta)\right) + \sum_{j} \exp\left(s(q_i, p_{j}^-;\theta)\right)}} \end{align} We investiagte the correlation between the contrastive entropy and existing ranking metrics. We train multiple dense retrieval models. To efficiently evaluate their retrieval performance, we sample a subset corpus that contains 100,000 passages during evaluation. Figure~\<PIC> shows the contrastive entropy and ranking metrics, including MAP@10, NDCG@10, and Recall@1000. We can see that the correlation between the contrastive entropy and existing ranking metrics is strong and positive. It is close to a linear correlation. Therefore, we believe that using contrastive entropy is an effective measure to assess the overall retrieval ability of models in our study.
Model Size Scaling
We finetune models of various sizes using the human-annotated training pairs. The finetuning is performed on the entire training sets. We do not utilize early stopping and instead report the best test set loss throughout the training process. This is mainly to mitigate the influence of suboptimal early stopping, which could lead to models being underfitted or overfitted.
Figure~\<PIC> illustrates the contrastive entropy on the test set with respect to model sizes. As shown in the figure, the retrieval performance improves (indicated by a lower test loss) as the model size increases. On the left side of the diagram, red stars represent the official checkpoints of variously sized BERT models, while blue points denote other official variants released concurrently. These variants differ in aspects such as the number of attention heads or feed-forward dimensions. The right diagram, in contrast, only features red stars, as the different shape variants of ERNIE are not publicly available.
Based on the observation, we propose to fit the scaling law in terms of model sizes as follows:
\begin{equation}
\label{eq:model_size_scaling_law}
L(N) = \left( {A \over N} \right)^{\alpha} + \delta_N
\end{equation}
where $N$ represents the number of non-embedding parameters of the model, and $L(N)$ denotes the model's contrastive entropy on the test set. Parameters $A$, $\alpha$ and $\delta_N$ are the coefficients.
Note that we introduce a parameter $\delta_N$, which represents a irreducible loss term. It means that a sufficiently large model (setting $N$ to infinity) can only reduces the loss to $\delta_N$ rather than zero. This irreducible loss is reasonable given the incomplete annotations and subjective understanding of relevance. On one hand, some relevant passages may not be annotated because they are not sucessfully recalled and are outside the annotation pool. On the other hand, relevance may be subjective to different annotators, which results in even imperfect agreement among different human annotators. Consequently, it is hard for models to perfectly agree with human annotations. Therefore, we believe there should be a irreducible term in the scaling law.
We employ least squares method to fit the linear curve. The coefficients are detailed in Table~\ref{tab:model_size_param}. The coefficient of determination (R²) suggests a good fit. Based on these results, we validate that the contrastive entropy follows a power-law scaling in relation to the size of non-embedding parameters.
Such discoveries offer new perspectives for future research experiments. For example, given this scaling law, we can initially train smaller models, fit the corresponding scaling curves, and then extrapolate them to predict the performance of larger models. This significantly reduces the cost of conducting experiments directly on larger models and instead offers the opportunity to experiment with different training strategies on smaller models to validate the effectiveness of new approaches.


5. FollowRAG Benchmark
To bridge the gap in automatic instruction-following evaluation for RAG systems, we introduce FollowRAG in this section. We provide a detailed introduction from two aspects: Data Construction'' and Evaluation and Statistics''.
5.1. Dataset Construction
\textbf{Instruction Collection \& Extraction.} FollowRAG aims to assess the model's ability to follow user instructions in complex multi-document contexts. Drawing from general IF datasets like IFEval~\citep{ifeval} and FollowBench~\citep{followbench}, we collect and verify definitions and examples of atomic instructions using rules (e.g., code), excluding those irrelevant to RAG scenarios. Ultimately, we identify 22 types of instruction constraints, encompassing language, length, structure, and keywords.
\textbf{Instruction Reforming.} We use widely-used question-answering (QA) datasets, such as Natural Questions ~\citep{NQ}, as the foundation for constructing FollowRAG samples. For a query sampled from the QA datasets, we need to generate a complex instruction containing $n$ atomic instruction constraints (with $n$ ranging from 1 to 4). To enhance the diversity of atomic instruction representations, we employ GPT-4o as the instruction generator. Specifically, given a query, we first sample $n$ instructions from the atomic instruction set and perform conflict detection. Subsequently, with examples as demonstrations, we prompt the LLM to generate a new varied instruction text for each type of atomic instruction, along with parameters for instruction-following evaluation.
\textbf{Combination.} Finally, we integrate the retrieved passages, query and atomic instructions to construct the sample input for FollowRAG. To avoid mechanically concatenating the query and instructions in a template-based manner, we prompt supervised model to naturally blend the multiple atomic instructions and the query into a coherent instruction-query paragraph. We then add the top-$K$ document passages retrieved based on the query to the instruction-query paragraph, forming the complete sample input.
% Finally, we integrate the retrieved passages, query, and atomic instructions to construct the sample input for FollowRAG. To avoid simply concatenating the query and instructions, we prompt the supervised model to blend them into a coherent instruction-query paragraph. We then add the top-$K$ document passages retrieved based on the query to this paragraph to form the complete sample input.
5.2. Evaluation and Statistics
After obtaining the model's output, we evaluate it from two perspectives: instruction following and question answering (QA) under the RAG paradigm:
\begin{itemize}
\item \textit{\textbf{Instruction Following:}} Utilizing the verifiable nature of our atomic instructions and following the IFEval approach, we automate the verification of the model’s adherence to each instruction through code validation. We then calculate the average pass rate for each atomic instruction across all samples to determine the instruction-following score in FollowRAG.
\item \textit{\textbf{RAG:}} Under new instruction constraints, the model's target output differs from  the gold answers in the original QA dataset, rendering traditional metrics like Exact-Match ineffective. To address this, we use the original gold answers as a reference and utilize GPT-4o to evaluate whether the model's outputs correctly address the questions. The scoring criteria are as follows: Completely correct (1 point), Partially correct (0.5 points), Completely incorrect (0 points). The average score of all samples is taken as the RAG score for FollowRAG.
\end{itemize}
% \subsection{Statistics}
For detailed statisticsin in Figure~\<PIC>, FollowRAG is the first instruction-following evaluation dataset under  RAG scenario comprising 2.8K samples, covering 22 fine-grained atomic instructions across 6 categories. The queries in FollowRAG are sourced from 4 QA datasets across 3 types: 1) Open-Domain QA: \textbf{Natural Questions (NQ)}~\citep{NQ} and \textbf{TriviaQA (TQA)}~\citep{TriviaQA}; 2) Multi-Hop QA: \textbf{HotpotQA (HQA)}~\citep{HotpotQA}; and 3) Knowledge Base QA: \textbf{WebQuestionsSP (WebQSP)}~\citep{webqsp}. To further construct varying levels of instruction-following difficulty, FollowRAG includes 0.9K samples of single and dual atomic instructions, as well as 0.5K complex multi-instruction samples containing 3 and 4 atomic instructions, respectively.
6.4. Quantitative Analysis
\textbf{Ablation Study.} To examine the effects of various components in VIF-RAG, we conduct an ablation study in Table \ref{tab:ablation}. The term "w/o" indicates versions where specific components are removed. Our key observations are:
\begin{itemize}
\item Removing any component from VIF-RAG results in decreased performance, indicating that all components, such as the complex instruction composition strategy and quality verification design, are crucial to its effectiveness.
\item The largest performance decline in FollowRAG is observed when executor verification is removed. This underscores the critical role of automated instruction-response validation in improving synthetic data quality and confirms the advantage of using LLMs to oversee instruction-following abilities through other core skills like coding.
\item Surprisingly, the consistency verification proves beneficial in preserving RAG capabilities. It effectively filters out samples with high-level semantic conflicts between instructions and queries, reducing noise in IF tasks and maintaining RAG performance integrity.
\end{itemize}
\textbf{Scaling Analysis.} To explore the impact of retrieved document quantity on instruction-following performance in RAG scenarios, we refer to Table \<PIC>. For the baseline models (SFT versions), instruction-following capability declines as the number of passages increases. Specifically, performance drops sharply by over 6\% when the document quantity in FollowRAG increases from 0 to 1. Further increasing the number to 10 leads to a significant performance decline, with Qwen-14B-SFT experiencing a drop of over 10\%. This indicates that integrating knowledge through retrieval-augmented techniques challenges the instruction-following abilities of existing models.
In contrast, VIF-RAG shows a minor performance drop ($<$3\%) when encountering the first document. As the number of documents increases to 10, VIF-RAG’s performance remains relatively stable, demonstrating its robustness.
\textbf{Instruction Difficulty Analysis.} To explore the effect of different instruction quantities (i.e., instruction-following difficulty) on model performance in RAG scenarios, we evaluate VIF-RAG and various baseline models on the FollowRAG benchmark, using test sets with 1, 2, and 3 instructions. As shown in Figure \<PIC>, as the number of instructions increases, all models generally show a decline in instruction-following capability, but VIF-RAG consistently outperforms the rest. Notably, even with 3 instructions present simultaneously, VIF-RAG still demonstrates over a 5\% IF prompt (strict acc.), further validating its superior capability in handling complex instruction-following tasks in RAG scenarios.


Methodology
In this section, we first recap the typical RAG pipeline (Sec.~\ref{sec:method:preliminary}), then present our VisRAG framework (Sec.~\ref{sec:method:ours}) and the construction of our training and evaluation data (Sec.~\ref{sec:method:data}).
Preliminary: Retrieval-augmented Generation
\label{sec:method:preliminary}
A typical retrieval-augmented generation (RAG) pipeline consists of a retriever and a generator, both built on large language models (LLMs)\footnote{In many cases, the retriever uses language models smaller than 1B parameters, which may not be considered ``large'', but we use the term LLM for simplicity.}.
This pipeline operates on a knowledge corpus $\mathcal{D}$, which is processed into units for retrieval and generation, denoted as $\mathcal{D} = \{d_1, \ldots, d_n\}$, where $n$ is the number of retrieval units.
Given a text query $q$ and the retrieval corpus $\mathcal{D}$, the retriever functions as $\mathcal{R}:(q, \mathcal{D}) \rightarrow \mathcal{D}{\mathcal{R}}$, taking $q$ and $\mathcal{D}$ as inputs and producing a candidate set $\mathcal{D}{\mathcal{R}} \subset \mathcal{D}$.
To enable efficient search, the units in the knowledge corpus $\mathcal{D}$ are pre-encoded into embeddings.
During RAG pipeline inference, approximate nearest neighbor (ANN) search is applied to retrieve $\mathcal{D}{\mathcal{R}}$, which serves as the knowledge source for generation.
The generation process can be defined as a function $\mathcal{G}:(q, \mathcal{D}{\mathcal{R}}) \rightarrow a$, where $a$ represents the answer and $\mathcal{G}$ denotes the LLM generator.
This is achieved by prompting the LLM with the query and the retrieved units $\mathcal{D}{\mathcal{R}}$ to generate an answer.
As shown in Figure~\<PIC> (left), traditional RAG frameworks (TextRAG) typically utilize text-based units for retrieval and generation.
However, in real-world scenarios, data often appear in complex, multi-modal documents, requiring an additional parsing step to obtain text.
In this paper, we propose to use the \textit{page} as the fundamental unit for retrieval and generation, which is directly processed by vision language models (VLMs) as an image without further processing during retrieval and generation.
In subsequent sections, we use the terms page'' and document'' interchangeably.
VisRAG: Vision-based Retrieval-augmented Generation
\label{sec:method:ours}
In this section, we present \textbf{Vis}ion-based \textbf{R}etrieval-\textbf{a}ugmented \textbf{G}eneration (VisRAG), as shown in Figure~\<PIC> (right).
In contrast to traditional RAG frameworks which use text segments for both retrieval and generation, VisRAG leverages the image of the document to preserve all information.
\subsubsection{Retrieval}
The first stage of VisRAG, VisRAG-Ret, aims to retrieve a set of pages from the corpus $\mathcal{D}$ given $q$.
We follow the dual-encoder paradigm in text-based dense retrieval models but employ a VLM rather than an LLM to encode the query and page.
Specifically, the query and page are encoded separately as text and image in the VLM, producing in a sequence of hidden states.
To derive the final embedding, and given that we use generative VLMs with causual attention, we adopt the position-weighted mean pooling over the last-layer VLM hidden states, giving higher weights to later tokens:
\begin{equation}
\small
\mathbf{v}=\sum{i=1}^S w_i \mathbf{h}i,
% \quad \text { where } \quad w_i=\frac{i}{\sum{i=1}^S i},
\end{equation}
where $\mathbf{h}i$ is the $i$-th hidden state, $S$ is the sequence length, $w_i=\frac{i}{\sum{j=1}^S j}$ is the $i$-th weight, and $\mathbf{v}$ is the query or page embedding.
The similarity score is calculated by the cosine similarity of the query and page embedding.
VisRAG-Ret is optimized using the InfoNCE loss:
\begin{equation}
\label{eq:loss}
\small
l(q,d^+,D^-) = -\log\frac{\exp(s(q,d^+)/\tau)}{\exp(s(q,d^+)/\tau)+\sum_{d^- \in D^-}\exp(s(q,d^-)/\tau)},
\end{equation}
where $d^+$, $D^-$ are positive document and the negative document set of $q$, respectively, $s(q,d)$ is the similarity score between $q$ and $d$, and $\tau$ is the temperature.
Generation
The second stage of VisRAG, VisRAG-Gen, focuses on generating the answer according to the user query and retrieved pages using a VLM.
We propose the following mechanisms to enable VisRAG-Gen to handle multiple retrieved pages in $\mathcal{D}\mathcal{R}$ for generation.
The prompts used for generation is presented in Appendix~\ref{appendix:generation_prompt}.
\paragraph{Page Concatenation.}
A straightforward approach is to concatenate all pages in $\mathcal{D}\mathcal{R}$ into a single image to accommodate most VLMs that are trained to accept a single image.
Formally,
\begin{equation}
\small
a \xleftarrow{} \text{VLM-Single}(q,\text{Concat}(\{d|d \in \mathcal{D}\mathcal{R}\})),
\end{equation}
where VLM-Single is a VLM that accepts a single image with text prompt and Concat is the image concatenation operation.
In this paper, we experiment with horizontal concatenation.
\paragraph{Weighted Selection.}
Another approach is to ask the VLM to generate an answer for every page from top-$k$, and select a final one with the highest confidence.
The final confidence is defined as the weighted generation probability of the answer:
\begin{equation}
\small
P(a|q,\mathcal{D}\mathcal{R}) = P(a|q,d) \cdot \lambda(q,d),
\end{equation}
where $P(a|d,q)$ is calculated as the reciprocal of the perplexity of generating the answer $a$ conditioned on the single document $d$, and $\lambda(d,q)$ is the normalized retrieval score:
\begin{equation}
\small
\lambda(q,d)=\frac{e^{s(q,d)}}{\sum{d' \in \mathcal{D}\mathcal{R}} e^{s(q,d')}}.
\end{equation}
\paragraph{VLMs Accepting Multiple Images.}
Some recent VLMs like MiniCPM-V 2.6 and Qwen-VL 2 are designed and trained to accept multiple images as input to perform cross-image reasoning.
This capability may be useful for the generation as the required information could be located on a single page from the retrieved document set $\mathcal{D}\mathcal{R}$ for single-hop questions or spread across multiple pages for multi-hop questions.
Formally, we have
\begin{equation}
\small
a \xleftarrow{} \text{VLM-Multi}(q,\{d|d \in \mathcal{D}\mathcal{R}\}),
\end{equation}
where VLM-Multi is the VLM that accepts multiple images with text prompt.
Training Data Efficiency
\label{sec:result:data_efficiency}
As retrieval acts as the bottleneck in an RAG pipeline, it is crucial to have an effective retrieval component to maintain optimal performance.
In this experiment, we study the training data efficiency of VisRAG-Ret by evaluating the performance of VisRAG-Ret trained under different amounts of synthetic training data, i.e. in the out-of-domain setting.
As shown in Figure~\<PIC>, when only trained on 20k q-d pairs, VisRAG can surpass bge-large (OCR).
After training on 150k pairs, it can further surpass NV-Embed-v2 (OCR), the SOTA 8B-sized text embedding model trained on millions of curated text pairs.
This highlights VisRAG-Ret's high training data efficiency and strong generalization capability, as all models are evaluated out-of-domain.
When compared with MiniCPM (OCR), which uses extracted text for training, VisRAG-Ret consistently achieves a performance gain of about 17\% and exhibits a more stable training process.
The results show VisRAG-Ret's potential for further performance improvements by scaling up the training data.
Performance on Different Data Subsets
\label{sec:result:performance_grouped}
In this experiment, we assess the retrieval and generation performance of VisRAG and TextRAG defined in Figure~\<PIC>, as well as VisRAG (SigLIP), which replaces the retriever in VisRAG with SigLIP.
We report their performance across different data subsets by categorizing queries based on the lengths of their positive documents, measured by the number of tokens of the extracted text.
Documents with a higher volume of extracted text may prioritize textual information over visual content.
As illustrated in Figure~\<PIC>, queries in ArxivQA and InfographicsVQA are divided into equal-sized bins according to the lengths of their relevant documents.
For each bin, we calculate and plot the average performance differences between VisRAG and TextRAG, as well as between VisRAG (SigLIP) and TextRAG, to compare how each model performs relative to TextRAG.
We observe that, in general, the relative performance of VisRAG and VisRAG (SigLIP) improves as the length of the relevant document decreases.
This suggests that models with vision encoders can better understand documents that emphasize visual information.
However, VisRAG (SigLIP) consistently underperforms VisRAG across all data subsets and, in some cases, even performs worse than TextRAG.
In contrast, VisRAG consistently outperforms TextRAG, indicating that the underlying language model in VisRAG is crucial for better understanding the semantics conveyed through visual cues.


Introduction
\textbf{Background.}
With the boom of digital information, billions of user requests are produced daily, so recommender systems (RSs) have become an integral part of Internet platforms.
To capture users' interests more accurately, RSs have gone through several milestones, such as the logistic regression with hand-crafted features (e.g., FM), the neural networks (e.g., WideDeep, YoutubeNet), the sequential signal (e.g., DIN, SIM), and the multi-hop graph signal (e.g., PinSage, DGRec).
In retrospect, these effective methods are based on the collaborative filtering (CF) idea and extend the boundaries of RSs. However, the CF framework also limits them under the case of cold-start and data sparsity problems.
The reason is that the CF idea aims to mine the user/item pattern intelligence from data, to discover and recommend high-click candidate items for a user while it is hard to understand the users' fine-grained and multi-aspect interests.
In fact, instead of mining user preferences from massive user-item interaction logs, the user always leaves some reviews/comments to explain further his/her feelings about this interaction, which provides an explicit way to understand the users' complex interests in language semantic space.
And the recent effort MoRec claimed that multi-modal information yield better results in sequential recommendation.
\textbf{Related work.}
To extract valuable content semantic information, the pioneering works are formed as a rating prediction task: for a user-item pair in test set, give the historical user/item contents in training set (e.g., reviews), then predict their possible interaction rating.
In early years, the DeepCoNN employed two convolutional neural networks (CNNs) towers to aggregate user/item content tokens individually to measure their dot-similarity, and the D-Attn further extended DeepCoNN by introducing local and global attention mechanism to replace CNNs to aggregate tokens.
Following the D-Attn, the ALFM and ANR focused on extracting the fine-grained multi-aspect semantic information and assigning different weights for aspects.
The recent progress is RGCL, which used BERT to generate the user-item content scores and then leveraged them as user-item graph edge weights to conduct a multi-hop graph neural network to make prediction.
\textbf{Motivation.}
Although these methods raised model ability with content information, they ignore the following problems:
\begin{itemize}[leftmargin=,align=left]
\item \textit{Transfer semantic knowledge across domains}: Actually, RS needs to serve several domains simultaneously, such as electronics, clothing, books, etc.
Since different domains always express different aspects of users' interests (e.g., food is delicious, price is appropriate, etc.), the previous methods need to re-tune their semantic component for different domains, which is time-consuming.
\item \textit{Enhancing the correlation between content and ID}:
The user-item content and ID-interaction information can be seen as two different modalities connected by users.
Nevertheless, previous methods focus on utilizing separate components to model the two corresponding content/ID spaces while ignoring how to exploit the correlation and align them in a unified space.
\end{itemize}
\textbf{Our Work.}
To alleviate the above problems, we propose \textbf{LoID}, a LLM-based model for transferring semantic knowledge across domains based on LoRA, and aligning content/ID information with contrastive objectives.
It mainly includes two steps:
\begin{enumerate}[leftmargin=,align=left]
\item \textit{Pretraining semantic plugin'}: On the one hand, the ideal semantic information should act as a universal role to support all domains. On the other hand, different domains have their main aspects of semantic information that are related to recommendation.  We borrow the LoRA strategy idea to train a small set of parameters for each domain, which could serve as plugins.  Then we fused them by DARE without further re-training.  \\item \\textit{Aligning the content/ID feature spaces}: As discussed before, the content and ID information can be seen as two modalities of user-item interaction.  To minimize their gap, we introduce the contrastive idea of maximizing content/ID features' mutual information to align their feature spaces. \\end{enumerate} Finally, to validate our LoID effectiveness, we extensively test LoID under 11 different domain datasets to show its superior ability. \\textbf{Contributions.} Our contributions are summarized as follows: \\begin{itemize}[leftmargin=*,align=left]  \\item We give a plugin' idea to transfer semantic knowledge, which sheds light on building a new paradigm for recommendation\footnote{Codes at \url{https://github.com/cjx96/LoID}.}.
\item We devise a novel content/ID feature alignment objective.
\item We conducted detailed analyses with SOTA methods and LLMs.
\end{itemize}
Methodology
Overview
The architecture of our method LoID is illustrated in Figure~\<PIC>.
In part (a), we train the LoRA parameters of source domains as "plug-ins" to enhance target domain prediction without further re-training.
In part (b), we extract historical contents of user/item to obtain user/item semantic representation, and then align the ID and semantic to make target domain rating prediction, \textbf{note that the source domain LoRA plugin is an optional choice}.


Fact-aware Capability Control
The factual similarity threshold in Equation \ref{eq:radgraph_similarity} plays a critical role in controlling the fact-awareness of our multimodal retriever.
We examine the performance of FactMM-RAG under different thresholds, as shown in Figure \ref{fig:hyperparameter_search}.
Not only utilizing F1RadGraph thresholds, we also employ F1CheXbert to curate additional thresholds from the report's diagnostic labels to mine report pairs.\\
\newline
Under the same F1CheXbert threshold for mining report pairs, we observe that an increase in the F1RadGraph threshold correlates with an improvement in factual performance.
However, adopting stricter thresholds for identifying report pairs does not yield further improvements and reaches saturation.
After calculating the average number of report pairs per query, we find that high thresholds can exclude many relevant report pairs, as shown in Figure \<PIC>. This exclusion results in the potential loss of factually useful pairs, thereby hindering the training of our multimodal retriever driven by additional factual medical knowledge.\\
\newline
Rather than relying on diagnostic labels from CheXbert to identify high-quality report pairs, Figure \<PIC> demonstrates that the F1RadGraph threshold alone can also effectively mine factual report pairs for training our multimodal retriever.
As the F1RadGraph threshold increases, FactMM-RAG even matches the performance under high threshold settings in Figure \<PIC>.
This signifies that employing our training strategy with curated factual query-report pairs still imposes useful supervision signals without relying on explicit diagnostic label guidance.


Slide-LoRA: Enhancing Image and Text Generation Consistency
To harmonize multi-modal generation tasks in a single model, the parameter space would be optimized for inconsistent or conflict training objectives, as stated before. We propose Slide-LoRA (which is shaped like a ``slide" between different LoRA experts as shown in Figure \ref{algorithm}), a novel module that can be conveniently inserted into Transformer layers as Low-Rank Adaptation (LoRA) and introduces limited parameter increase. As such, Slide-LoRA spontaneously processes text generation and image generation in separate parameter spaces, thus relieving the inconsistent training problem.
As shown in Figure \<PIC>, Slide-LoRA is composed of a gating network $\mathcal{G}$ and three LoRA modules $(\boldsymbol{R}^{T}, \boldsymbol{R}^{I}, \boldsymbol{R}^{S})$. $\boldsymbol{R}^{T}$ and $\boldsymbol{R}^{I}$ serve as the separate parameter space for text and image generation, respectively, while $\boldsymbol{R}^{S}$ aims to learn the knowledge shared by both text and image generation. Specifically, Given the input token sequence $\boldsymbol{x} \in \mathbb{R}^{L \times D}$, the gating network $\mathcal{G}$ (concatenation of two linear layers in specific) determines whether the processing of the input token sequence requires knowledge of text generation or image generation, and produces a scalar $\gamma = \mathcal{G}(\boldsymbol{x}) \in [0, 1]$. The output of the Slide-LoRA layer can be formulated as
\begin{equation}
\boldsymbol{O} = \frac{1}{2}\cdot\{[\gamma \geq 0.5] \cdot \boldsymbol{R}^{T}(\boldsymbol{x}) + [\gamma < 0.5] \cdot \boldsymbol{R}^{I}(\boldsymbol{x}) + \boldsymbol{R}^{S}(\boldsymbol{x})\},
\end{equation}
where $[\cdot]$ equals $1$ if the condition inside is true and $0$ otherwise. Slide-LoRA incorporates task-specific and task-shared knowledge from input tokens, thus separating the inconsistent training objective and learning the shared knowledge of text and image generation.
\subsection{Multi-Modal Pre-Training and Comprehensive Fine-Tuning}
TextHarmony training process consists of two stages. In the multi-modal pre-training stage, TextHarmony is trained on text-rich image-text corpus and learns to produce multi-modal outputs. In the comprehensive fine-tuning stage, we concurrently cultivate the text and image generation capabilities of TextHarmony by training on a series of text-centric tasks.
Stage 1: Multi-Modal Pre-Training
TextHarmony is pre-trained based on the pre-training weight of MM-Interleaved , with extra text-rich datasets including MARIO-LAION and DocStruct4M. MARIO-LAION contains 9M web images with brief captions and the according OCR results. DocStruct4M consists of 2M documents and 1M natural images with text-oriented structure annotations. We use MARIO-LAION for both text and image generation ({\it, i.e.}, either predict the caption of the image or generate the image based on the caption), and we use DocStruct4M for text generation only. In this stage, we freeze the vision encoder and the LLM, training only the Q-Former and the image decoder to obtain basic image understanding and generation capabilities.
Stage 2: Comprehensive Fine-Tuning
We integrate various text-centric datasets and employ uniform instructions for all tasks. In this stage, the vision encoder, Q-former, image decoder, and the proposed Slide-LoRA are trained to enhance the multi-modal generation and human-instruction-following capabilities of TextHarmony.
\textbf{Visual Text Generation.} In this task, TextHarmony generates images according to the text description and is required to render accurate and coherent text. Although MARIO-LAION contains captions of text-rich images, the description is oversimplified and lacks concentration on the textual elements within the image. To this end, we sample 100K images from MARIO-LAION and generate detailed captions about them, termed DetailedTextCaps-100K. The captions focus on both visual and textual elements in the images. This is achieved by prompting Gemini Pro, a pioneer multi-modal large language model, to generate detailed descriptions based on the sampled image and the OCR results. As shown in Figure \<PIC>, the image description from DetailedTextCaps-100K is more comprehensive compared with MARIO-LAION and can better depict the textual elements in the image.
\textbf{Visual Text Editing.} In this task, TextHarmony substitutes or renders text in the given location of the image and keeps the background consistent. We randomly mask the image with the help of MARIO-LAION's OCR results and fine-tune TextHarmony in a self-supervised manner.
\textbf{Visual Text Comprehension.} We employ the training set collected by Monkey for the text-centric VQA fine-tuning. The training set involves 1.4M QA pairs and covers various text-rich scenarios.
\textbf{Visual Text Perception.} For the basic OCR capabilities, we randomly sample 1M images from MARIO-LAION and leverage the OCR annotations.


Overreliance on Unimodal Priors
Overreliance on unimodal priors is a key factor contributing to hallucinations in LMMs.
This issue arises when the model over-relies on the knowledge learned from one modality during training, rather than integrating knowledge of all available modalities.
In such cases, the model defaults to strong unimodal priors learned during training, leading to outputs that follow familiar unimodal patterns even when those patterns are not supported by the multimodal input.
Following the general issue of overreliance on unimodal priors, we categorize this into three distinct types: \textbf{Language Dominance}, \textbf{Visual Dominance}, and \textbf{Audio Dominance}. Each form of dominance presents unique challenges for LMM performance and contributes to hallucinations in different ways.
\noindent\textbf{Language Dominance}, also know as language biases~\citep{rohrbach2018object,leng2024mitigating,guan2024hallusionbench,wang2024mitigating}, arises when a model excessively depends on pre-trained large language models (LLMs), generating responses that adhere to linguistic patterns or prior knowledge from large language corpora, even when visual or audio inputs provide contradictory information. This issue is particularly prevalent in LMMs that integrate LLMs as their decoder base. These LLMs~\cite{chiang2023vicuna,jiang2023mistral,yang2024qwen2}, due to their proficiency in capturing linguistic structures and semantic relationships, often dominate the decision-making process, overshadowing contributions from visual or audio modalities.
As illustrated in Fig.~\<PIC>, a video depicts finger skateboarding with shoes on fingers. When asked by the language-biased question “Did you see shoes worn on feet?”—reflecting commonsense event that follows linguistic priors—LMMs respond “yes,” contradicting the actual content and inducing hallucination. This demonstrates LMMs’ tendency to rely on language priors over factual multimodal inputs.
\noindent\textbf{Visual Dominance} occurs when a model over-relies on visual information, underutilizing or disregarding linguistic and auditory cues. In such cases, the model-generated outputs are heavily influenced by visual context, often neglecting important information from the other modalities.
As illustrated in Fig.~\<PIC>, a video depicts a person planning a woodworking project with a hammer in sight, while the audio track contains only the person speaking and bird chirping. Despite this, advanced LMMs may over-rely on the visual presence of the “hammer” and incorrectly infer a “hammer hitting” sound, ignoring the actual audio content where no such sound is present.
\noindent\textbf{Audio Dominance} arises when a model excessively relies on auditory input, disregarding visual or linguistic information.
As illustrated in Fig.~\<PIC>, a video captures a person recording a village view through a window, showing dark clouds. The audio track contains evident thunderstorm sounds, but no lightning is visible. Despite this, LMMs may over-rely on the audio cues, hallucinating that lightning is visible in the scene, thereby disregarding the actual visual content.


Literature database construction \label{sec:liter-data-constrct}
Just like human researchers, reading other literature and drawing inspirations from them is an important process for LLMs to generate valuable ideas.
However, online literature reading is a very time-consuming process, so we collect a literature database in advance for the following literature retrieval and idea proposal process.
To be specific, we collect papers published in ICLR, NeurIPS, ICML, ACL, NAACL, and EMNLP in past ten years, yielding a database with 48,895 papers. For each paper, we parse the PDF file and extract its title, abstract, introduction, method, and references sections. Then, as shown in Figure~\<PIC> , given an LLM $f$, we prompt it to read and summarize the paper as:
\begin{equation}
\begin{aligned}
% (T_b, T_i, T_m) = f(T_p, T_r),
\mathbb{E}^{(p)} &= f(\tau_1, T_a^{(p)}), \\
(T_b^{(p)}, T_s^{(p)}, T_i^{(p)}) &= f(\tau_2, T_t^{(p)}, T_a^{(p)}, T_n^{(p)}), \\
T_d^{(p)} &= f(\tau_3, T_m^{(p)}, T_i^{(p)}),
\end{aligned}
\end{equation}
where $T_t^{(p)}, T_a^{(p)}, T_n^{(p)}, T_m^{(p)}$ are the paper $p$'s title, abstract, introduction, and method sections. $\mathbb{E}^{(p)}, T_b^{(p)}, T_s^{(p)}, T_i^{(p)}, T_d^{(p)}, T_r^{(p)}$ are extracted entities, background, summary, main ideas, detailed ideas, and core references, as shwon in Figure~\<PIC>.
$\tau_i, i \in \{1, 2, 3\}$ represent our designed prompt templates, and specific prompts are shown in the Appendix~\ref{apd:pfee}. In practice, we use GLM-4\footnote{We use the GLM-4 released in May 20th, 2024 (glm4-20240520).} as $f$.
Besides, ``Core References'' in Figure~\<PIC> means extracting papers referenced in introduction and method sections, because we believe these references have the greatest impact on paper $p$ among all references.
Additionally, the background, summary, and main ideas are also encoded with Sentence-BERT for their embeddings $e_b^{(p)}, e_s^{(p)}$ and $e_i^{(p)}$, respectively. All extracted information are recorded into our literature database.
To retrieve literature faster, we also construct a paper-entity graph in the database.
we also store all occurrence relationships of papers and entities in the database. As shown in Figure~\<PIC>, if an entity $T_{e1}$ appears in the paper $p1$, there will be an edge between the two paper nodes. Idea proposal  \label{sec:idea-proposal}
Upon completion of the literature retrieval, we propose three approaches for generating research paper ideas. In essence, the idea generation process can leverage two types of information: the first is derived from the content of the retrieved papers, which inspires the LLM to generate ideas; the second involves the LLM freely brainstorming to produce new ideas. Based on this principle, we delineate three methods of idea generation that vary in their application of brainstorming.
As illustrated in Figure~\<PIC>(a), the direct proposal method (SciPIP-A), does not use brainstorm. While the first dual-path proposal method (SciPIP-B), as Figure~\<PIC>(b) shows, utilizes the user-provided background into two branches. The first branch employs this background for literature retrieval, problem summarization, and idea generation based on the retrieved literature, while the second branch engages in brainstorming solutions directly from the user-provided background. Following the independent generation of ideas in both branches, the outputs are merged and subsequently filtered and refined to yield the final ideas. Similarly, as shown in Figure~\<PIC>(c), the second dual-path proposal method (SciPIP-C) follows a process analogous to SciPIP-B, with the key distinction being that the content generated through the LLM's brainstorming is utilized not only for idea generation but also integrated with the user-provided background for entity extraction and other literature retrieval processes. We will provide a detailed exposition of these three methods of idea proposal in the following sections. We use GPT-4o\footnote{We use the GPT-4o released in May 13th, 2024 (gpt-4o-2024-05-13),  which has an October 2023 knowledge cutoff.} by default in this section.
Direct idea proposal method
As depicted in Figure~\<PIC>(a), in the direct proposal method, we first retrieve papers following the pipeline described in Section~\ref{sec:liter-retrive}. Then, the user-provided background along with the retrieved papers are utilized to prompt the LLM to summarize the core problem we aim to address and provide justifications. The specific prompts can be found in the Appendix~\ref{apd:pfee}.
With the summarized problem and justifications, the LLM is prompted to generate around 10 initial ideas. In the prompt, both the problem, the justification and the retrieved papers are provided. The LLM is encouraged to generate clear, innovative, valid, and comprehensive ideas. The specific prompts for this step can be also found in the Appendix~\ref{apd:pfee}.
Though the prompt has declared, the initially generated ideas may still have shortcomings in terms of novelty or relevance to the problem. To address this, we filter the initial ideas using prompt engineering (prompts are illustrated in the Appendix~\ref{apd:pfee}), with the primary criterion being that the ideas are generated in response to the given problem. Additionally, the ideas must exhibit a high degree of novelty and feasibility. During this process, each generated idea is evaluated independently, and about half of them will be filtered.
Then, the LLM is encouraged to further improve the filtered ideas by considering their inter-relationships. That is, the LLM is tasked with considering the compatibility of the ideas, ensuring that it does not generate conflicting or repetitive ideas. Moreover, the LLM is required to generate formulas or algorithms to better elaborate the ideas if needed. The prompt is shown in the Appendix~\ref{apd:pfee}. Finally, about 3 to 4 refined ideas will be proposed.
Dual-path idea proposal methods
We find that the directly generated ideas often rely heavily on the retrieved literature, sometimes closely resembling the methods presented in those papers. They frequently involve transferring approaches from other fields or making minor improvements to existing methods within the same field, resulting in relatively ordinary novelty and rarely yielding breakthrough thinking.
Therefore, we further propose idea proposers that incorporates brainstorming, encouraging the LLM to produce more novel thoughts. Specifically, brainstorming can play a role in both processes of idea generation. As shown in Figure~\<PIC>(b), the SciPIP-B has two paths, where one path follows the direct proposal approach, while the other path uses the LLM to brainstorm possible solutions based on the user-input background, outputting these as ideas. Ultimately, these ideas will be merged with those generated based on the retrieved papers, filtered and refined to produce the final ideas. In this model, the results of brainstorming are independent of the generation based on retrieved papers.
In another approach, as shown in Figure~\<PIC>(c), brainstorming generates ideas independently while also being utilized in literature retrieval. Specifically, we extract entities from the brainstorming results and incorporate them as part of the entity set in the literature retrieval process. With this method, some keywords arising from the brainstorming will also help enhance the effectiveness of literature retrieval. The ideas generated through brainstorming will also be merged with those produced after literature retrieval.


Knights and Knaves Logical Reasoning Benchmark
\label{sec:proposal_dataset}
\emph{Knights and Knaves (\kk)} is a type of logical puzzle where some characters can only answer questions truthfully, and others only falsely. The goal is to infer each character’s truthfulness.
An example of the puzzle and its corresponding answer is provided in~\<PIC>.
Based on the \kk puzzle, we design a \emph{dynamic} benchmark that supports generating new puzzles and perturbing existing puzzles. Our library automatically solves the \kk puzzles and generates detailed reasoning steps and solutions
for evaluation and training. Moreover, to support measuring memorization, we also provide a procedure to perturb a given puzzle and recompute the new solution after the perturbation.
Specifically, our benchmark consists of two modules:
\textbf{The Abstract Module} %
has four components that can generate and manipulate \kk puzzles in an abstract form (see \cref{app:kk-abstract-repr}). (1) The \textbf{Generator} creates a random \kk puzzle given a specification $(N,D,W)$ that determines the difficulty level. Specifically, it generates a puzzle with $N$ people, and for each person, a statement that consists of a random tree of maximum width $W$ and depth $D$, where each node is sampled uniformly at random from the following candidate sets.
The \textit{leaf} node can be a claim that a specific person is lying (i.e., knave) or truth-telling (i.e., knight), and the \textit{branching} node can be \emph{and}, \emph{or}, \emph{not}, \emph{imply}, and \emph{equivalence}. (2) The \textbf{Solver} finds the solution to a given puzzle. In our study, we ignore puzzles with no or more than one solution. So we implement the solver by converting the puzzle to a Boolean satisfiability problem and enumerate all Boolean assignments so that we can easily obtain a list of all valid solutions to filter out unwanted puzzles.
(3) The \textbf{Reasoner} generates a step-by-step reasoning procedure that leads to the solution. We design it to mimic the reasoning steps used by humans and some LLMs: instead of enumerating all Boolean assignments, it examines each person sequentially, makes an assumption (knight/knave) and checks if it leads to a contradiction with statements from people with specific identities assumed. If not, it continues to examine the next person; otherwise it will try an alternative assumption or backtrack to a previously examined person (details in \Cref{app:kk-solver-reasoner}). (4) The \textbf{Perturber}, given a puzzle, generates a locally perturbed version that is (superficially) similar to the original puzzle, and solvable with the same underlying principle and at a similar difficulty level. The Perturber replaces either an entire statement or a leaf node in a statement with a newly sampled one. The process is rerun until the perturbed puzzle has a unique solution different from the original puzzle, or until a maximum  number of attempts is reached. This rarely happens for $N\geq 3$ people puzzles. When it happens we skip the puzzle in our perturbation analysis.
See \cref{tab:example_2ppl_puzzle} for concrete examples.
\textbf{The Natural Language Module} has three components that operate in natural language space. (1) The \textbf{NL-Generator} takes an abstract \kk puzzle and formats it in natural language. It is template-based, with randomly sampled person names and a random template for making claims, and it uses a few heuristics to convert tree-structured logical statements to natural language. (2) The \textbf{NL-Reasoner} converts the reasoning steps computed by the abstract Reasoner to the natural language format in a similar manner. See \c<PIC> for dataset length distributions. (3) The \textbf{NL-Perturber} generates perturbed puzzles by keeping the abstract puzzle intact and manipulating the language-level descriptions as follows (See \Cref{app:kk-nl-perturber}): (i) replace character names with uncommon names; (ii) replace knight/knave with other similar pairs of role names, e.g., saint/sinner; (iii) reorder the statements from the characters; (iv) flip the role name from knight/knave to knave/knight. Note that the flipped role perturbation is somewhat adversarial as it goes against the common intuition that a good character would tell the truth and a bad one would lie, so we include it mostly for reference purposes.
Fine-tuning with 10k 8-people puzzles.
Given the significant performance improvement from fine-tuning, a natural question arises: can brute-force fine-tuning on a very large number of puzzles eventually solve the \kk puzzles, by observing/memorizing a variety of combinations of persons' claims and their corresponding answers?
We \ft \gptfouromini on $1k$/$10k$ of the most challenging 8-people puzzles for 5 epochs. ~\<PIC>
shows that \textbf{(1)} $10k$-FT significantly outperforms $1k$-FT across all tasks, reaching $\sim90\%$ test accuracy on moderately difficult 4/5-people puzzles. \textbf{(2)} \cotft is generally more effective than \ft with $10k$ samples, likely due to the guidance provided by reasoning steps. \textbf{(3)} An exception is the 2-people task, where the training and testing distribution gap causes the {\cotft}ed model to occasionally get stuck in a loop of listing assumptions and contradictions, resulting in long, repetitive responses without reaching a conclusion\footnote{We observe similar accuracy drop on 2-people task for \llamathree  (see \c<PIC>) when it is {\ft}ed for overly long epochs.
We provide more examples and discussions in \cref{appsubsec:4omini_eval_reason}.}.
\textbf{(4)} \ft with $10k$ puzzles achieves surprisingly high test accuracy on all tasks, e.g., 52\% on 8-people tasks, where the un-FTed model scores near 0. Notably, the models do not see reasoning steps during training and rely solely on memorizing answers. %
We also observe high transferability for $10k$ {\ft}ed \llamathree in~\<PIC>, e.g., $87\%$ test accuracy on 3-people puzzles.


Introduction
Although AI has brought transformative changes to various aspects of life, its impact on researchers unfolds in a nuanced manner. On the one hand, AI assists in various research disciplines, such as Social Science \citep{DBLPNeumanCY23}, Finance \citep{ai4finance}, Medicine \citep{rakhimov2022artificial}, GeoScience \citep{praskievicz2018river}, etc., significantly expediting academic processes. However, many of these applications are superficial, often limited to data-driven clustering or classification. On the flip side, the AI era poses challenges for researchers. Despite its ability to streamline some activities, researchers still face demanding, cognitively intensive tasks such as staying current through extensive paper reading, rapidly generating ideas in response to fast-paced advancements, conducting rigorous experiments to substantiate claims, and managing an increasing volume of peer reviews. Then a question looms: \emph{How effectively can AI assist researchers in tasks that are domain-specific, expertise-demanding, and reasoning-intensive?}
Existing works proved the promising potential for using LLMs in assisting AI research. \citet{si2024can} conducted a large-scale human study and found that LLMs can generate creative research ideas. \citet{lu2024aiscientist} proposed an autonomous agent to handle complicated research workflow and write a whole research paper.
However, most of these works focus on addressing highly subjective problems that require a high degree of expertise, making evaluation laborious and hard to reproduce. This underscores the need for a comprehensive benchmark that rigorously assesses LLMs' capabilities in expertise-intensive research activities.
To this end, in this work, we introduce \textsc{AAAR-1.0}, a novel benchmark that aims to comprehensively assess the LLMs' capacity on expert-level research tasks. As illustrated in Figure~\<PIC>, \textsc{AAAR-1.0}~decomposes four distinct expert-level AI research tasks from the researcher's daily activities, including i) \textsc{EquationInference}, investigating whether the LLMs can infer the equation correctness based on the paper context; ii) \textsc{ExperimentDesign}, validating LLMs' ability on designing reliable experiments for a research idea; iii) \textsc{PaperWeakness}, testing the quality of weaknesses discovered by LLMs from paper drafts; and iv) \textsc{ReviewCritique}, investigating whether LLMs can identify and explain the deficient/unreliable human-written paper reviews. To ensure data quality, senior AI researchers with extensive domain expertise perform data annotation for \textsc{AAAR-1.0}, followed by rigorous multi-round data examination and filtering.
All four tasks require models to possess strong domain knowledge covering various cutting-edge research findings, as well as expert-level research experience, to the extent that even humans need substantial research accumulation to tackle the tasks we designed.
Benefiting from the proposed automatic metrics, we conduct extensive experiments across numerous mainstream LLMs, where we find that:
\vspace{-0.5em}
\begin{itemize}[leftmargin=2em]
\itemsep0em
\item With a random guess baseline of 25\%, the performance of most LLMs on \textsc{EqInfer}~hovers just slightly above chance, with the top models reaching around 60\%. This highlights the difficulty of the task, despite its reliance only on local context reasoning.
\item In \textsc{ExpDesign}, LLM-designed experiments are innovative and more diverse than those by humans; however, many are trivial, lack feasibility, and stray from the original research objectives.
\item In \textsc{PaperWeakness}, LLM-identified weaknesses often lack depth and specificity, making them broadly applicable and less useful for providing feedback on paper drafts.
\item In \textsc{ReviewCritique}, LLMs struggle to effectively identify deficient human reviews, indicating limited usefulness in assisting meta-reviewers in evaluating the quality of individual paper reviews.
\end{itemize}
\paragraph{Settings.}
As different LLMs have distinct context windows, to ensure a fair comparison, we fix the maximum input length for all models. According to the data statistics of  Table~\ref{tab:equations_statistics}, we empirically use 1,000 words for both contexts before and after equations, i.e., 2,000 surrounded words.
\paragraph{Main results.}
Table~\ref{tab:equation_main_tab} shows the main results. Firstly, the open-source LLMs, especially the Falcon and Gemma, perform unexpectedly disappointing (even worse than random guesses). These screwed scores are mainly due to the poor long-context instruction following ability, where we find some open-source LLMs are confused with the massive input and often copy the LaTeX code from the input. In contrast, closed-source LLMs generally achieve superior accuracy, probably owing to the richer scientific knowledge from the larger model parameters. However, considering the conventional multi-choice QA formulation of~\textsc{EqInfer}, the recently-released GPT-4o solely gets 43.18, implying the unique challenge of \textsc{EqInfer}~compared with other scientific QA benchmarks~\citep{song2023nlpbench}. Notably, with the help of internal CoT, o1 gains stronger performances than GPT-4/GPT-4o, indicating the potential benefits of adopting reasoning for this task.
\paragraph{$\mathcal{Q}$: Do more contexts boost performance?} Table~\ref{tab:equation_main_tab} unifies the input context lengths to 1,000 tokens for various LLMs. To answer this question, we experiment with long-context LLMs to investigate the impact of the input context lengths. Particularly, we scale the input length (per side) from 100 to 1,500 words. As shown in Figure~\<PIC>, for the open-source LLMs (Llama and Qwen), after 300 words length, increasing the input context doesn't help the performance and even significantly drops Qwen's scores. While for the closed-source GPT-4-Turbo and GPT-4o, scaling up input length gradually boosts the performances at the first 1,000 words, but stabilizes afterwards. This is in line with human intuition, i.e., surrounding context is required for the equation inference, as the adjacent context usually provides important information, such as the target algorithm description or the notation definition. However, after exceeding a specific threshold, more context information is not beneficial anymore and even confuses those LLMs with poor long-context handling capacity~\citep{wang2024leave,liu2024lost}.


We first give an overview of the proposed Code-as-Monitor (\fname)~(\cref{subsec: overview}).
Then, we elaborate on the constraint element in Constraint Painter, especially constraint-aware segmentation~(\cref{subsec: constraint element proposal}).
Finally, we present Constraint Monitor for real-time detection~(\cref{subsec: real-time monitor module}).
 we give the necessary implementation details in \cref{{subsec: implementation details}}.
Overview}
{subsec: overview}
The proposed {\fname} comprises three key modules: the Constraint Generator, Painter, and Monitor.
We focus on long-horizon manipulation task instructions $\mathcal{L}_{\mathrm{global}}$ (\eg, \textit{``Move the pan with the bread to the stove, and be careful not to let the bread fall out''}), using RGB-D observations $\mathcal{O}$ from two camera views~(front and top).
As shown in \c<PIC>, the RGB images~$\mathcal{O}$, along with instructions~$\mathcal{L}_{\mathrm{global}}$, previous subgoal~${l}_{\mathrm{pre}}$, and failure feedback from the Constraint Monitor~$f_{\mathrm{pre}}$~(\eg, subgoal success or failure reason), are fed into the Constraint Generator~$\mathcal{F}_{\mathrm{VLM}}$~(\ie, GPT-4o~\cite{achiam2023gpt}) to generate the next subgoal~${l}_{\mathrm{next}}$ and associated textual constraints~$\mathcal{C}$.
This process can be formulated as follows:
These outputs are integrated and refined through manual inspection to produce the multi-granularity dataset, which is combined with LISA's training data to fine-tune our model. More details are in Supp.~\ref{supsec: implementation details}.
 We use this dataset and LISA's training data to co-fintune our model. Further details are in Supp.~A.
 Our dataset utilizes images from BridgeData V2~\cite{walke2023bridgedata}.
To determine per-frame subgoals and constraints, we require annotations at the frame level; however, BridgeData V2 provides instructions only at the trajectory level.
herefore, we sample pick-and-place data and process it using external references such as gripper open/close states, resulting in $10,181$ trajectories comprising $219,356$ images.
o generate ground-truth annotations, we employ GPT-4o~\cite{achiam2023gpt} to decompose trajectory instructions into per-stage subgoals, corresponding textual constraints, and associated objects and parts.
e then employ Grounded SAM~\cite{ren2024grounded} for instance-level segmentation and Semantic SAM~\cite{li2023semantic} for part-level segmentation.
By integrating these outputs, we achieve multi-granularity annotations, which are further refined through manual inspection.
 We use this dataset and LISA's training data to co-fintune our model.
For more details, please check Supp.~A.We use BridgeData V2~\cite{walke2023bridgedata} as the image source for our dataset. We need to determine the subgoal of each frame in a trajectory to ascertain the constraint of the current frame. Then, fine-grained part-level masks \( \mathcal{M}_{p} \) and corresponding element type descriptions \( l_{\mathrm{e}} \) are produced, as shown in \c<PIC>. However, BridgeData V2 provides instructions only for entire trajectories. Therefore, we sampled pick-and-place data from BridgeData V2 and processed the dataset using external reference information such as the gripper's open/close states.
Ultimately, we sampled $10,181$ trajectories to construct our dataset, which contains a total of $219,356$ images.
After sampling the image data, we used a combination of off-the-shelf foundation models to obtain the ground truth.
Specifically, we input the entire trajectory instruction provided by BridgeData V2 into GPT-4o for decomposition.
This process yielded the subgoals of each stage, the corresponding textual constraints, and the object and part associated with each constraint.
 We then employed Grounded SAM~\cite{ren2024grounded} for instance-level segmentation and Semantic SAM~\cite{li2023semantic} for part-level segmentation.
 By combining these results, we obtained annotations at multiple granularities.
Finally, we conducted manual inspections to further improve annotation quality.


To characterize the performance of VPA in correcting the Q-estimations, we leverage Spearman’s rank correlation coefficient, which measures the strength and direction of a monotonic relationship between two ranked variables. The reason is that the relative ranking of Q-values are more important than the absolute values for policy update.
Specifically, given a dataset collected by rolling out the offline policy in the environment, we compare the ranking of learned reward/cost Q-values before and after VPA with that of estimated actual return by using Monte Carlo simulations. A large Spearman’s rank correlation coefficient implies that the distribution of learned Q-values is more aligned with the distribution of true Q-values.
As shown in Table \ref{table:Spearman}, it is evident that the coefficient increases significantly after VPA for both reward and cost Q-values, no matter if the offline policy rolls out from a seen state-action pair in the offline dataset or from a randomly selected OOD state-action pair. This clearly demonstrates the effectiveness of VPA in aligning the pretrained Q-functions.
subsection{Finetune Phase}
{Challenge II: Lagrange multiplier mismatch.~}
Conventional value-based online safe RL relies on updating Lagrange multipliers alongside the policy and Q-functions during training, so as to push the overall cost below the limit while striking a right balance between maximizing the reward and minimizing constraint violations. While the policy and Q-functions can benefit from offline pretraining for a warm start, offline safe RL algorithms like CPQ \citep{xu2022constraints} and BEAR-lag \citep{ray2019benchmarking} cannot accurately estimate Lagrange multipliers with regularizing strengths matching with the cost of the offline policy, e.g., a small Lagrange multiplier is not power enough to push down the policy cost, while a large multiplier prevents active exploration of high-reward state-action pairs.
For instance, in the BallCircle environment, the offline pretrained Lagrange multiplier value obtained using the BEAR-lag algorithm is approximately 1500, whereas during online finetuning, the SAC-lag requires a value of only about 0.65. The gap between these values clearly precludes the direct use of offline pretrained Lagrange multipliers.
Improper initialization can lead to extensive constraint violations  or training stagnation, an issue we term as the \textbf{Lagrange multiplier mismatch}.
On the other hand, as VPA promotes active exploration of high-reward state-actions by optimistically estimating rewards and pessimistically estimating costs, it inevitably increases the risk of exploring high cost state-actions, which in turn amplifies the need for appropriate Lagrange multipliers to quickly reduce the constraint violations.
Figure \<PIC> shows the online finetuning performance comparison after VPA in two environments between 1) empirically setting a good initial value for the Lagrange multiplier and 2) setting it to zero, where the traditional dual ascent method is used to update the multiplier. It is clear that a good initial multiplier can manage the cost very well,  while the policy with a very small initial multiplier value suffers from large constraint violations and takes a much longer time to reduce the cost below the limit. The results also imply that although VPA aligns the distributions of Q-values, it may introduce high costs for online finetuning, which can be addressed with an appropriate initial Lagrange multiplier.
ction{Experiments}
In this section, we conduct extensive experiments to verify the effectiveness of our approach, aiming to answer the following questions: 1) {\bf RQ1:} How does our method compare with naive finetuning and other SOTA baselines in both reward and cost? 2) {\bf RQ2:} How do different components in Marvel affect the performance? Due to the space limit, we delegate the experimental details and some additional results to \cref{sec:more_exp_} and cref{sec:Experimental_Details}.
ubsection{Evaluation Setup}
mph{Benchmarks.} We consider the DSRL benchmark \citep{liu2023datasets} and select xtbf{ten} environments from the { Bullet Safety Gym \citep{gronauer2022bullet} and Safety Gymnasium \citep{ji2023safety}}: BallRun, BallCircle, CarRun, CarCircle, HalfCheetah,  AntCircle, AntRun, DroneCircle,  Hopper, and Swimmer (results for the last four are in \cref{sec:more_exp}). The cost threshold is set to be 20 in these environments.
As mentioned earlier in \cref{preliminary}, we
choose CPQ and SAC-lag as base algorithms in our proposed framework Marvel for offline training and online finetuning, respectively, due to the effectiveness and representativeness of them.
As shown in \c<PIC>, Marvel demonstrates better or comparable performance compared to all baselines consistently across all environments, i.e., achieving the higher return while keeping the cost below the threshold.
In stark contrast, the naive warm start method proves largely ineffective, often causing performance drop or stagnation during training.
Without aligning the Q-estimations, both JSRL and PEX struggles a lot to improve during online learning and fails to control the cost. Besides, PEX also  suffers from poor training stability and high variance across different settings.
While SO2 mitigates the inaccuracies of Q-estimations related to O2O RL, it does so only to a limited extent and cannot maintain its performance consistently across different environments, although aPID has already been used to boost its performance. On the other hand, the fact that SO2 performs better than other baselines further indicates  1) the great potentials of enabling fast and safe online learning through policy finetuning (compared to using the pretrained policy only as a guide policy as in JSRL and PEX) and 2) the need of correcting pretrained Q-estimations before online finetuning.
Each experiment was conducted using five random seeds, and the results were averaged to generate the final learning curves. {We use a dataset that includes data provided by DSRL \citep{liu2024offlinesaferl} and random data generated by a random policy to control the quality of the offline dataset.}
mph{Baselines.} While Guided Online Distillation \citep{li2024guided} is the only work studying O2O safe RL, its usage of large pretrained model leads to an unfair comparison with standard RL frameworks using typically small-scale policy networks. In this work, we compare Marvel with \textbf{JSRL} \citep{uchendu2023jump},
as Guided Online Distillation mainly follows this approach except using DT as the pretrained policy. Besides, we further adapt some SOTA approaches in O2O RL to O2O safe RL, including \textbf{SO2} \citep{zhang2024perspective} and \textbf{PEX} \citep{zhang2023policy}, and a \textbf{Warm Start} approach as baselines.
SO2 improves Q-value estimation through Perturbed Value Updates, JSRL and PEX utilize offline pretrained policies for exploration, and Warm Start directly finetunes the policy and Q-networks from offline safe RL without modifications.


textbf{Online 3D Scene Perception:}
Accurate comprehension of 3D scenes is an indispensable capability for embodied agents.
Many tasks, such as 3D occupancy prediction\cite{monoscene, yu2024monocular} and object detection\cite{Qi_2019_ICCV, wang2022cagroup3d}, are direct manifestations of this capability.
Currently, most works on 3D scene perception\cite{qi2017pointnet, graham20183d, yi2019gspn} was conducted offline, taking pre-acquired and reconstructed 3D data to obtain a relatively lagging perception.
Based on this situation, Online3D\cite{xu2024online} introduced an adapter-based model that equips mainstream offline frameworks (in Figure~\<PIC>) with the competence to perform online scene perception, which means they can process a real-time RGB-D sequences.
However, this framework still fails to overcome the intrinsic limitation of conventional point cloud modality.
In a more general embodied scenario, real-time monocular visual input for scene perception can further advance the research on embodied agents.
textbf{3D Gaussian Splatting:}
3D Gaussian Splatting\cite{kerbl3Dgaussians} uses anisotropic 3D Gaussians to model a 3D scene, renowned for its fast speed and high quality in the field of radiance field rendering.
The explicit physical characteristics of 3D Gaussians and the splat-based rasterization employed during rendering have also motivated rapid advancements in research fields such as scene editing\cite{guédon2024gaussianfrostingeditablecomplex, palandra2024gseditefficienttextguidedediting, silva2024contrastive}, dynamic scenarios\cite{Lu_2024_CVPR, gao2024gaussianflowsplattinggaussiandynamics, xiao2024bridging}, and SLAM\cite{yugay2023gaussian, yan2024gs, deng2024compact, li2025sgs}.
GaussianFormer\cite{gaussianformer} pioneers the application of 3D Gaussians in outdoor 3D semantic occupancy prediction, updating Gaussians through comprehensive features extracted from multi-view images.
These Gaussians are ultimately converted into local 3D occupancy prediction through an elaborately designed Gaussian-to-voxel splatting module.
Compared to conventional voxel-based methods, using 3D Gaussian representation constitutes a more flexible approach.
In this paper, we will leverage this significant attribute to accomplish embodied occupancy prediction in indoor scenarios.
To this end, we introduce an additional tag \(\gamma \) for all the Gaussians in the memory.
When initializing a novel scene, tags of these Gaussians are set to \(0\).
Every time we put a number of updated Gaussians back into the memory, their tags are set to (1\).
For the Gaussians taken from the memory, we generate a set of confidence values \(\Theta \) based on their tags \(\Gamma\).
For those Gaussians within the frustum and marked as having been previously updated(\(\Gamma=\mathbf{1} \)), we set their confidence values to a certain value between (0\) and \(1\), and they are slightly updated in the current update.
For those Gaussians that have never been updated, we set their confidence values to \(0\), indicating that they will be the focus in the current update.
During the refinements, we have:
here we use \(\oplus\) to represent the composition of rotation quaternions and the add operation of other parts.
We use Figure~\<PIC> to illustrate how we maintain the Gaussian memory and refine Guassians according to their confidence values.
ction{EmbodiedOcc: An Embodied Framework}
We present the training framework of our EmbodiedOcc model for indoor embodied occupancy prediction.
During the whole prediction process, we use the current monocular input to update our Gaussian memory in real time, which can be easily converted into 3D occupancy prediction.
We first train our local occupancy prediction module using the focal loss \(\mathit{L}_{focal}\), the lovasz-softmax loss \(\mathit{L}_{lov}\), the scene-class affinity loss \(\mathit{L}_{scal}^{geo}\) and \(\mathit{L}_{scal}^{sem}\) following RetinaNet\cite{lin2017focal}, TPVFormer\cite{tpvformer} and Monoscene\cite{monoscene}. We use monocular occupancy within the frustum \(\mathbf{Y}_{mono}^{fov}\) and the corresponding ground truth \(\mathbf{Y}_{gt}^{fov}\) to compute the loss.


h{Teacher-student training}
he teacher-student approach to generalization in deep RL assumes access to $c_t \in \mathcal{C}_{\textnormal{ID}}$ at every timestep throughout training, and leverages this privileged context information to train a teacher policy that can adapt to different contexts. The teacher policy applies a context encoder $f: \mathcal{C} \rightarrow \mathcal{Z}$ that maps the context to a latent feature $z_t = f(c_t)$, which is then provided as an input to the policy $\pi: \mathcal{S} \times \mathcal{Z} \rightarrow P(\mathcal{A})$ and critic $V^{\pi}: \mathcal{S} \times \mathcal{Z} \rightarrow \mathbb{R}$. In this work, we consider the latent feature space $\mathcal{Z} = \mathbb{R}^d$. The context encoding $z_t = f(c_t)$, policy $\pi(a_t \mid s_t, z_t)$, and critic $V^\pi(s_t, z_t)$ are trained to minimize the average actor-critic RL loss over ID contexts given by
here $\mathcal{L}_{\pi}(c)$ and $\mathcal{L}_{V}(c)$ represent the policy loss and critic loss, respectively, of a given RL algorithm for the context $c \sim \mathcal{C}_{\textnormal{ID}}$. In our experiments, we apply Proximal Policy Optimization (PPO) \citep{schulman_2017} as the RL algorithm. See the Appendix for details.
ote that the teacher policy cannot be applied at deployment time because it requires privileged information about the context $c_t$ in order to compute the latent feature $z_t$. For this reason, RL training is followed by a supervised learning phase where a student policy is trained to imitate the teacher policy using only the recent history of states and actions from the last $H$ timesteps $h_t = \left(s_{t-H}, a_{t-H}, \ldots, s_t \right) \in \mathcal{H}$. In particular, an adaptation module $\phi: \mathcal{H} \rightarrow \mathcal{Z}$ that maps recent history to a latent feature $\hat{z}_t = \phi(h_t)$ is trained to minimize the loss
where expectation is taken with respect to trajectories sampled using the student policy in ID contexts $c \sim \mathcal{C}_{\textnormal{ID}}$ during training. This training represents a form of implicit system identification across ID contexts. Using the history encoding $\hat{z}_t = \phi(h_t)$, the policy $\pi(a_t \mid s_t, \hat{z}_t)$ can be applied at deployment time because it does not require privileged information as input.
paragraph{Robust adaptation module}
In the teacher-student architecture, the adaptation module $\phi$ is trained to identify ID contexts from history, resulting in strong performance across $c \in \mathcal{C}_{\textnormal{ID}}$. However, because it is only possible to train on ID contexts, the adaptation module $\phi$ and resulting policy $\pi(a_t \mid s_t, \hat{z}_t)$ may not generalize well to OOD contexts $c \in \mathcal{C}_{\textnormal{OOD}}$ that were not seen during training. Because $\phi$ is a learned module, its output $\hat{z}_t$ is only reliable for the distribution of history inputs $h_t$ that were observed during training. This suggests that $\hat{z}_t = \phi(h_t)$ may not be useful if the environment dynamics of OOD contexts result in trajectories that are different from the ID trajectories seen during training.
n order to quantify the level of uncertainty present in the
The robust latent feature $z_{\textnormal{rob}}$ defines an anchor point in latent feature space where we can apply robust training. Because we consider a single unified policy architecture, note that the RL loss $\mathcal{L}_{\textnormal{RL}}$ in \eqref{eq:rl_loss} incentivizes the privileged context encoding $z_t = f(c_t)$ to move away from $z_{\textnormal{rob}}$ if it will lead to better performance. Given this intuition, we propose the use of $z_{\textnormal{rob}} = \mathbf{0}_d$ in our implementation of GRAM to allow the outputs of a randomly initialized context encoder to start near $z_{\textnormal{rob}}$ at the beginning of training and move away from it as needed. Then, at deployment time we bias the latent feature estimate back towards the robust anchor point $z_{\textnormal{rob}}$ if the estimate is unreliable due to OOD environment dynamics. See the right side of ig<PIC> for an illustration.
By using a special robust latent feature $z_{\textnormal{rob}}$, we incorporate the failure mode of OOD dynamics directly into the existing adaptation framework as a single instance in latent feature space. This allows us to leverage the existing training procedure and architecture for adaptation in ID contexts, while applying tools from robust RL to encode robust behavior into $\pi(a_t \mid s_t, z_{\textnormal{rob}})$ for OOD generalization \emph{within the same architecture}. The use of a single architecture also allows for implicit regularization benefits of this robust training beyond $z_{\textnormal{rob}}$. In the next section, we provide details on the robust RL techniques that we use during training to achieve OOD generalization.
ction{Training for robust adaptation}\label{sec:ra_train}
Our robust adaptation module provides an intuitive structure for achieving both ID and OOD dynamics generalization within a single architecture. In order to accomplish this goal, we jointly train our policy $\pi(a_t \mid s_t, z_t)$ for adaptive performance in ID environments and robust performance in OOD environments (i.e., when $z_t = z_{\textnormal{rob}}$). For a given iteration of RL training, we assign each training environment to either \emph{ID training} or \emph{OOD training}. This assignment determines how the latent feature vector is calculated, as well as how data collection occurs in the environment. See g<PIC> for an overview.
Within each iteration, all data for a given training environment is collected according to either standard ID data collection or adversarial OOD data collection, as described in the following paragraphs. This provides temporal consistency when training the policy for adaptive or robust performance, respectively. However, we alternate these assignments between iterations, which allows full trajectories to contain a mixture of both forms of data. As we show in our experiments, this mixed data collection design provides additional robustness benefits compared to using the same training assignment for entire trajectories.


subsection{More About No-Skill Performances}
abel{sec:no-skill-performances}
mysection{How can we rank no-skill performances ex aequo?}
The ranking scores allow ranking all no-skill performances (\ie, those for which the groundtruth and predicted classes are independent) ex aequo when some constraints on the compared performances are added. The more common constraint is undoubtedly that the priors are fixed. In this case, the canonical ranking scores that put the no-skill performances on the same footing are located on the curve
eCurvePriors : \priorpos^2 \, a \, b = \priorneg^2 \, (1-a) \, (1-b)$.
Another interesting constraint is that the rates of predictions are fixed. By symmetry, the canonical ranking scores that put the no-skill performances on the same footing are located on the curve
leCurveRates : \ratepos^2 \, a \, (1-b) = \rateneg^2 \, (1-a) \, b$.
The $\tileCurvePriors$ and $\tileCurveRates$ curves are depicted in <PIC> and <PIC>.
We can see that $\scoreBalancedAccuracy$ is perfectly correlated with the ranking scores at the intersection between the curve tileCurvePriors$ and the rising diagonal, which is at $(\priorneg,\priorneg)$, and that $\scoreCohenKappa$ is perfectly correlated with the ranking scores at the intersection between the curve ileCurvePriors$ and the median horizontal, which is at frac{\priorneg^2}{\priorneg^2+\priorpos^2},\frac{1}{2})$.
lobal\long\def\randVarCanonicalImportanceBis{\randVarImportance_{a',b'}}
lobal\long\def\canonicalRankingScoreBis{\rankingScore[\randVarCanonicalImportanceBis]}
mysection{Correction for chance.}
The idea of correcting a score for what can be achieved by chance is common in the literature. Scott~\cite{Scott1955Reliability} and Fleiss~\cite{Fleiss1971Measuring} proposed a correction for accuracy. Cohen~\cite{Cohen1960ACoefficient} proposed another correction for it, that differs in what is considered to be achievable by chance. We noticed that Scott's $\scoreScottPi$ and Fleiss's $\scoreFleissKappa$ do not satisfy the axioms of ranking, even in the case of fixed priors. Cohen's correction, on the other hand, allows nothing more than what we can do with ranking scores, with fixed priors: correcting $\canonicalRankingScore$ in the same way as Cohen~\cite{Cohen1960ACoefficient} did with $\scoreAccuracy$, that is
        \canonicalRankingScore-\canonicalRankingScore\circ\opNoSkill
       1-\canonicalRankingScore\circ\opNoSkill
leads to a score that is perfectly rank-correlated with $\canonicalRankingScoreBis$, where $a'=\frac{\priorneg^2(1-b)}{\priorneg^2(1-b)+\priorpos^2b}$ and $b'=b$. %Taking a=b=\nicefrac12$, we recover the result that $\ordering_\scoreCohenKappa$ is located at (\aCohen,\bCohen)$ on the \tile.section{A new look at the balanced accuracy $\scoreBalancedAccuracy$ and Cohen's kappa <PIC> and <PIC> shows the rank correlations for both  (on the left) and $\scoreCohenKappa$ (on the right).
Geometrically, an entire horizontal line of the \tile is crushed into a single point (the intersection it has with the $\tileCurvePriors$ curve). There is thus an enormous loss of diversity
when applying Cohen's correction to our scores.


egin{abstract}
Large Language Models (LLMs) exhibit impressive performance across various tasks, but deploying them for inference poses challenges. Their high resource demands often necessitate complex, costly multi-GPU pipelines, or the use of smaller, less capable models. While quantization offers a promising solution utilizing lower precision for model storage, existing methods frequently experience significant performance drops at lower precision levels. Additionally, they typically provide only a limited set of solutions at specific bit levels, many of which are extensively manually tuned. To address these challenges, we propose a new method called \textbf{SKIM}: Scaled K-means clustering wIth Mixed precision. Our approach introduces two novel techniques: 1. A \textit{greedy algorithm} to solve approximately optimal bit allocation across weight channels, and 2. A \textit{trainable scaling vector} for non-differentiable K-means clustering. These techniques substantially improve performance and can be adapted to any given bit. Notably, in terms of model perplexity, our method narrows the gap between 3-bit quantized LLaMA models and their full precision counterparts by xtbf{16.3\%} on average.
d{abstract}
ction{Introduction}
arge Language Models (LLMs) including GPT \cite{radford2019language} and LLaMA \cite{touvron2023llama}, have achieved remarkable performance across a diverse range of tasks. These models not only excel in language processing \cite{achiam2023gpt, dubey2024llama, chowdhery2023palm, zhang2022opt} but also adapt effectively to multimodal applications \cite{wang2024visionllm, driess2023palm}, marking a crucial step toward artificial general intelligence \cite{bubeck2023sparks}. However, the computational and memory demands of LLMs pose significant challenges. For instance, when loading parameters in FP16, GPT requires 350GB of memory, while LLaMA-65B needs at least 130GB, both of which far exceeding the capabilities of A100-80G GPUs. Even when conducting inference with the smallest LLaMA model, which has 7 billion parameters, an Out-of-Memory exception can occur on a widely used 24GB GPU. These challenges significantly complicate the storage and practical deployment of such models.
One promising technique to address these issues is quantization, which involves transforming high-precision data into lower-precision formats, such as converting FP16 parameters to INT4. This method directly reduces the memory required to deploy and load LLMs and improves inference latency due to the phenomenon of the memory wall \cite{gholami2024ai}, which identifies the memory bandwidth as a key bottleneck in LLM inference. In addition, quantization has shown promising performance benefits. For example, previous studies have shown that both LLM weights and activations can be stored in 8 bits \cite{xiao2023smoothquant}, or only LLM weights can be stored in 4 bits \cite{kim2023squeezellm}, with little performance degradation. This encourages researchers to explore lower-precision solutions while maintaining reasonable performance levels.
However, standard quantization techniques in recent methods can suffer a significant drop in performance when using low bit widths. To mitigate this decline, these methods often introduce additional techniques that incur extra memory costs. For example, SqueezeLLM \cite{kim2023squeezellm} retains certain sensitive elements and outliers with full precision using a sparse tensor, while AWQ \cite{lin2024awq} divides the quantization group into smaller ones, requiring the storage of more quantization factors. Additionally, the extra memory needed to achieve a reasonable trade-off between memory usage and performance often requires manual tuning and selection, making the process cumbersome.
extbf{Contribution} In this paper, we address the above issues with our proposed method, Scaled K-means clustering wIth Mixed Precision (SKIM), which optimizes the bit allocation using a greedy algorithm and regularizes the column-wise difference with a scaling vector. Our method can easily adapt to any specified bit, including even non-integer values, and achieve better performance. Figure \<PIC> illustrates how our method breaks the fixed bit grid and delivers better results. Our main contributions can be summarized as follows: (1) We conduct a mathematical analysis of two optimization targets: layer-wise and sensitivity-based quantization, identifying a unified framework that highlights their core differences and allows us to evaluate their effectiveness. (2) We observe a significant disparity in data distribution across channels and propose a greedy algorithm for approximately optimal bit allocation in response to this disparity. Our mixed precision method adapts to any specified bit level and significantly improves performance. (3) For the non-differentiable K-means clustering operator, we incorporate a trainable scaling vector based on our novel iterative optimization strategy. This vector effectively regularizes the data across columns and serves as a valuable complement to the mixed precision method.
section{Methodology}
label{sec:method}
As previously mentioned, the quantization error of a weight matrix $W$ is the sum of that of its individual rows. However, each row, represent a channel, soften exhibits distinct data distributions and quantization errors, as illustrated in Figure \<PIC>. These observations indicate that applying the same bit-width for quantizing all rows results in an disproportionate allocation of resources. Motivated by these insights, we propose the adaptive channel-wise \textbf{Mixed Precision} technique. It begins by allocating different bits to each channel using a greedy algorithm, based on their quantization errors. Next, weighted K-means clustering is applied to each channel, generating centroids and labels with the allocated bits. Finally, we incorporate the scaling vector and train it through an iterative optimization strategy, keeping the labels fixed to enable gradient-based optimization. The dequantized weights can be recovered from the final labels and centroids, along with the scaling vector. Full algorithm of our SKIM method can be found in Appendix \ref{sec:full}. In the following subsections, we will explain our Mixed Precision and Scaling Vector techniques, as well as the principles behind our objective selection, in detail.
subsection{Objective Selection}
abel{sec:obj}
As discussed, our SKIM framework consists of three main steps: the greedy algorithm, weighted K-Means clustering, and the scaling vector. A key question arises regarding how optimization objectives should be selected for these steps. Specifically, should the same objective be used consistently across all steps, or is cross-objective optimization feasible? Moreover, under different computational scenarios, which objective is the most effective, and how should it be chosen?
These questions can be addressed through our previous reformulation, from which we can conclude that although the layer-wise and sensitivity-based objectives differ, they can ultimately be transformed into either the \textit{full} or \textit{diag} forms. In these forms, both objectives exhibit a similar structure, ensuring their synchronization towards the final goal and allowing for cross-objective optimization in our work. Related experimental results are detailed in Section \ref{sec:iter}. The only difference lies in whether $g_{y_i}$ is introduced to serve as a guide for the subsequent model architecture.
Therefore, we can intuitively assess the effectiveness of each objective in the following order, from best to worst: \textit{S-full}, \textit{L-full}, \textit{S-diag}, and extit{L-diag}. This is because the \textit{S} form incorporates gradient information as a guide, while the \textit{diag} form takes an aggressive diagonal approximation. The analysis aligns with our experimental results, which will be discussed in more detail in Section ef{sec:eff}.
However, due to the interdependence among the elements of the gradient and the input, we cannot compute the four objectives using directly recorded expectations of $g$ or $X$. This limitation prevents us from adopting the most effective \textit{S-full} objective, as recording the corresponding $E(g^\top g)$ for each row requires quadratic memory complexity which is impractical for LLMs.
Consequently, in any scenario requiring a complete error calculation, such as the proposed mixed precision and scaling vector methods, we adopt the \textit{L-full} form. In contrast, for scenarios involving element-wise sums, such as the weighted K-means clustering, we adopt the \textit{S-diag} form. This will be the main principle for our objective selection.
subsection{Mixed Precision}


subsection{On-device DNN inference}
In recent years, there has been an explosion of Artificial Intelligence (AI) applications and services thanks to significant advancements in DL. These applications span a wide range, from personal assistants and recommendation systems to autonomous vehicles and healthcare diagnostics. Furthermore, the widespread adoption of mobile computing and the Internet of Things (IoT) has led to billions of interconnected mobile and IoT devices, collectively generating an immense volume of data at the network edge \cite{ZhiEdgeIntelligence2019}. This has created the need to push the execution of DNN applications at the edge of the network leading to a substantial surge in the deployment of DL models on resource-constrained devices \cite{oodin2021smartcomp}. Although on-device training of DL models remains a challenging endeavor due to the limited computational resources and memory constraints of such devices, significant progress has been made in on-device inference. Several techniques have been proposed to enable efficient on-device inference, including:
textbf{Lightweight model design:} One approach to enable on-device inference is the design of lightweight DL models. Models like MobileNetV2 \cite{sandler2018mobilenetv2}, EfficientNet-Lite \cite{tan2019effnet} and NasNet-Mobile \cite{zoph2017LearningTA} have been specifically crafted to achieve high accuracy while minimizing computational and memory requirements.
Previous research on cascade architectures is predominantly centered around isolated scenarios, where a single device enjoys exclusive access to a dedicated server. Nevertheless, this assumption no longer aligns with the reality of AI-driven indoor environments like the one in Fig.~\<PIC>. The pervasive integration of AI technologies has given rise to an expanding array of AI-powered devices, resulting in a pressing demand for simultaneous support from a shared server.
xtbf{Model quantization:} Quantization techniques \cite{han2015deep} reduce the precision of model weights and activations, effectively decreasing memory and computational demands without substantial loss in accuracy.
extbf{Model pruning:} Pruning methods, such as channel pruning \cite{he2017channel}, aim to reduce the size of DNNs by removing unimportant neurons, thereby reducing computational overhead.
extbf{Knowledge distillation:} Knowledge distillation \cite{hinton2015distilling} involves training a smaller, more efficient model to mimic the predictions of a larger, complex model. This allows for the transfer of knowledge from larger models to smaller ones.
xtbf{Optimized scheduling:} Scheduling and runtime optimization techniques \cite{CoBiCh2022} help allocate computational resources efficiently, ensuring that DL inference tasks run smoothly on constrained devices.
espite these advancements, modern intelligent environments like smart homes and offices are often equipped with small-form-factor, resource-constrained devices (e.g., smart cameras, AI speakers). These devices lack the processing power to support high-accuracy, computationally-intensive models, driving the need for distributed collaborative inference approaches.
ubsection{Distributed collaborative inference}
Distributed collaborative inference systems leverage a central server to assist mobile and em
In Fig.~\<PIC>, we present the comprehensive system architecture of a multi-device cascade, specifically designed for executing DL inference tasks on IoT devices in a collaborative setting. Within this architecture, all IoT devices are engaged in performing a common task, such as object detection, albeit they may host different DL models tailored to their computational capabilities and requirements.
The main components of this system architecture include the following.
textbf{IoT devices:} These devices are the primary endpoints where the DL inference tasks are executed. Each IoT device is equipped with its own DL model designed to process incoming data efficiently. These devices generate predictions independently based on their respective models.
extbf{Device-hosted models:} DL models on the devices are trained on the same task, operating independently in all other aspects. Each model can be of different architecture, complexity and efficiency, depending on the computational resources and demands of the respective device. Thus, it is of paramount importance that the scheduler maintains high performance in scenarios of device heterogeneity.
textbf{Forwarding decision function:} The output predictions generated by each IoT device are first given to a forwarding decision function. This decision function assesses the confidence of the DL model's output on each device. If the model is sufficiently confident in its prediction, the result remains unchanged, and no further action is taken. However, if there is uncertainty or low confidence in the prediction, the sample is earmarked for further analysis.
extbf{Server:} Samples that require additional scrutiny are forwarded to a centralized server for in-depth processing. The server hosts a more accurate and computationally-intensive DL model capable of refining the predictions made by the IoT devices.
textbf{Request queue:} The forwarded samples from all IoT devices are temporarily stored in a request queue at the server. This queue serves as a staging area where samples awaiting processing are collected. The request queue ensures efficient and organized data flow from the IoT devices to the server.
textbf{Server-hosted model:} The server-side model processes the samples drawn from the request queue. This model is shared among all connected IoT devices, allowing for collaborative refinement of predictions by leveraging the advanced "knowledge" of the network.
textbf{Result distribution:} Finally, the results produced by the server-side model are distributed back to their corresponding IoT devices as soon as they become available. This seamless distribution of refined predictions ensures that each IoT device benefits from the improved accuracy achieved by the server.