Firstly, we assume that the adversary can manipulate a dataset and upload the resulting dataset online,
which will be downloaded and used for training DNNs by developers.
{However, the adversary cannot \emph{modify} any other aspects of training such as DNN architecture and training algorithm. }
Accordingly, the adversary chooses to insert the backdoor by poisoning training datasets
instead of modifying DNN parameters or adding malicious sub-modules.
{Despite this, the adversary may \emph{know} the details of target DNNs.
This is reasonable considering the wide adoption of public well-known model architectures and typical training algorithms (e.g., Adam~\cite{ADAM}).}
Secondly, the adversary cannot directly modify the digital inputs when launching attacks at inference time,
so needs to launch physical backdoor attacks by manipulating the physical objects.
Regarding the physical attack, we assume that the adversary cannot touch the physical objects
to add or remove any triggers, e.g., many traffic signs are erected in high places,
{the adversary cannot touch the face of victims for causing DoS.}
Despite this, the adversary can maintain a reasonable distance from the physical objects
such that the physical objects can be recognized by the adversary
and projected by common lasers.
ubsection{Overview of \attackname}\label{sec:overview}
abel{sec:attack}
h overview of our attack \attackname is depicted in \figurename~\<PIC>, consisting of two stages, namely, backdoor embedding and backdoor triggering. mallskip\noindent {\bf Backdoor embedding.} This stage injects a backdoor into DNNs by poisoning the training dataset,   ollowing the common practice of previous backdoor attacks~\cite{gu2017badnets, turner2019label, chen2017targeted, liu2020reflection, zhong2020backdoor, liu2017trojaning, composite-attack,input-aware-attack, LIRA}.
First, the adversary specifies a target label $y^t$
and designs a \emph{digital} laser-based trigger $\delta$ (cf.~\Cref{sec:trigger-design}).
Then, he selects a subset
{T_{select}}$
of images from a legitimate training dataset
{T_{train}}=\{(x_i,y_i)\}_{i=1}^{N}$
where
\alpha=\frac{|{T_{select}}|}{|{T_{train}}|}$
is a small ratio,
and adds the trigger $\delta$ to each image $x$ in the chosen subset
{T_{select}}$
and associates each modified image $x^\delta$ with the target label $y^t$,
obtaining the poisoned training dataset
{P_{train}=T_{train} \cup \{(x^{\delta}, y^t)|x\in T_{select}\}}
Finally, the adversary publishes the poisoned dataset on the Internet,
waiting for developers to download the dataset for training their DNNs,
during which the backdoor will be naturally embedded into DNN models
as a result of training with the poisoned dataset.
The dataset poisoning scheme enables our attack to process inherent model generalizability,
i.e., the poisoned training dataset can be used by and affect any model.
We will test our attack on four different models in \Cref{sec:exper-result}.
ompared with previous backdoor attacks, \attackname features the laser-based trigger,
the design of which will be elaborated in \Cref{sec:trigger-design}.
smallskip
noindent {\bf Environment setup.}
Two operators collaboratively collected the images in China's urban streets.
{The laser pointer is equipped with a button to adjust the focus of the laser spot. The size of the spot projected onto traffic signs depends on both the focus setting and the pointer's (adversary's) distance from the signs. In practice, the adversary can adjust the focus based on their distance to achieve a laser spot size resulting in a clear projection on the traffic signs, called reasonable size.}
Therefore, one operator holds the laser pointer more than 30 meters away from the traffic sign and adjusts the laser spot to a reasonable size.
The other operator photographs the traffic signs with the photographing device from 5 to 30 meters
in front of the traffic sign and 1.5 meters off the ground.
To collect more diverse images, we adjusted the angle and distance between the laser pointer and the photographing device
and collect 2--5 images per setting.
During
the collection,
we avoided projecting the lasers to those areas of traffic signs that will impact their visibility,
e.g., areas with black-color text.
mallskipoindent {\bf Open-source LaserMark.}
After collection, we appropriately cropped each image so that the traffic sign is complete, centered, and occupy most of the pixel space.
Finally, we have {676} road traffic sign images with physical laser-based triggers,
among which 235, 224, and {217} images have red, green, and blue laser points, respectively.
These traffic sign images constitute the poisoned test dataset $P_{test}$,
which will be used to evaluate the attack success rate in our experiments.
In addition to the poisoned physical images $P_{test}$ with physical triggers, we also collect {158} clean physical images without physical triggers and add them into the clean test dataset _{test}$.
In total, we have 834 images, covering 32 different traffic sign categories, where
the number of images contained in each category of LaserMark is shown in igurename~\<PIC>.
They are published as an open-source dataset. ubsection{Trigger Settings}abel{sec:trigger-setting}
As mentioned in \Cref{sec:attack}, we propose an effective approach to optimize the parameters of the digital triggers toward a more powerful physical backdoor attack.
{The laser pointer typically includes a focus adjustment button, which affects the size and center brightness of the laser spot. Adversaries can freely project the laser spot onto any position of a traffic sign, and stronger natural light makes the spot appear more transparent.
These factors motivate us to optimize the four parameters associated with the size, transparency, location, and center brightness of laser spots.


Researchers have proposed automated detection methods for centralization risks in smart contracts.
 Pied-piper \cite{ma2023pied} summarizes five common patterns of code with centralization risks and uses static Datalog analysis \cite{immerman1998descriptive} to identify contracts conforming to these patterns, supplemented by directed fuzzy testing \cite{bohme2017directed} to reduce false positives.
Similarly, Tokeer \cite{zhou2024stop} analyzes transfer-related functional modules in contract code, generates oracles based on four known rug pull contract patterns, and detects rug pull contracts using Datalog analysis.
The rug pull contracts \cite{sun2024sok}, a type of malicious fraud contract, often incorporate code with centralization risks for asset transfer, making detection methods for this category applicable to centralization risks as well.
Researchers have proposed automated detection methods for centralization risks in smart contracts.
Pied-piper \cite{ma2023pied} identifies five common patterns of code associated with centralization risks.
It employs static Datalog analysis \cite{immerman1998descriptive} to detect contracts conforming to these patterns, followed by directed fuzzy testing \cite{bohme2017directed} to minimize false positives.
Similarly, Tokeer \cite{zhou2024stop} focuses on transfer-related functional modules in contract code.
It generates oracles based on four known rug pull contract patterns and uses Datalog analysis for detection.
Rug pull contracts \cite{sun2024sok}, a type of malicious fraud contract, often incorporate code for asset transfer with centralization risks. Consequently, detection methods developed for rug pull contracts can also be applied to identify centralization risks in smart contracts. However, existing methods relying on predefined patterns need manual intervention for pattern modification or supplementation when facing backdoors with unknown patterns or variants of known backdoors, or else underreporting occurs, which makes the current methods unsuitable for real-world scenarios with numerous contracts and diverse backdoor attack patterns\cite{event1}\cite{event2}\cite{event3}.Based on this observation, we propose an automated detection framework for centralization risks, as illustrated in Fig. \<PIC>.
Researchers have already proposed some automatic detection methods for smart contract backdoors.
 or instance, Pied-piper\cite{ma2023pied} summarizes 5 common code patterns of smart contract backdoors and employs Datalog analysis to determine if a contract conforms to a given pattern, supplemented by directed fuzzy testing to reduce false positives.
 Similarly, Tokeer\cite{zhou2024stop} builds a transfer model to analyze the functional modules in the contract code implementing transfers, generating oracles based on 4 known rug pull contract patterns, and ultimately detects the rug pull contract based on the oracles through Datalog analysis.
 The rug pull contracts here are a kind of malicious fraud contracts, which often necessitate the insertion of backdoors for asset transfer, rendering detection methods applicable to this category also capable of uncovering smart contract backdoors.
However, existing methods face challenges in accurately detecting diverse code with centralization risks in real-world scenarios due to their reliance on predefined behavior patterns.
However, existing methods struggle to accurately detect diverse backdoors in real-world scenarios due to their reliance on predefined behavior patterns.
They may underreport when faced with unknown patterns or variants of known behavior patterns.
Additionally, inaccurate predefined patterns can lead to misclassification of secure contracts as risky ones.
However, due to the reliance on predefined attack patterns, it is difficult for existing methods to accurately detect diverse backdoors in real-world scenarios.
  the one hand, when facing backdoors with unknown patterns or variants of known behavioral patterns, existing methods suffer from underreporting.
On the other hand, an inaccurate predefined pattern may also cause misclassifying secure contracts as contracts with backdoors.
 However, existing methods relying on predefined patterns need manual intervention for pattern modification or supplementation when facing backdoors with unknown patterns or variants of known backdoors, or else underreporting occurs, which makes the current methods unsuitable for real-world scenarios with numerous contracts and diverse backdoor attack patterns\cite{event1}\cite{event2}\cite{event3}.
ection{Background}
ubsection{Solidity Smart Contracts}
A smart contract is an automatically executing program on the blockchain, with Solidity being the most popular language for writing these contracts.
This paper focuses on two blockchain platforms that support Solidity: Ethereum and Binance Smart Chain (BSC), both of which are peer-to-peer networks composed of accounts.
These accounts are categorized into contract accounts, which are associated with and controlled by smart contracts, and external accounts, which are without code.
Each account is uniquely identified by an address.
mart contracts are automatically executing programs on the blockchain, with Solidity \cite{solidity} being the most popular language for their development.
This paper focuses on two blockchain platforms that support Solidity: Ethereum and Binance Smart Chain (BSC).
Both platforms are peer-to-peer networks composed of two types of accounts: contract accounts, which are associated with and controlled by smart contracts, and external accounts, which contain no code. Each account is uniquely identified by an address.
 Solidity smart contract consists of state variables and functions.
For example, in the \texttt{Example} contract shown in Fig. \<PIC>, Lines 2 and 3 define the state variables, while Lines 4 through 17 define the functions. State variables, persistently stored on the blockchain, can be read and modified by different functions.
In this example, \texttt{owner} represents the account address of the contract owner, while \texttt{balances} represents the token balances of different accounts.
Functions in the contract can be invoked by accounts to execute predefined logic.
The \texttt{constructor} in the \texttt{Example} contract is called upon contract deployment to initialize state variables.
The \texttt{owner\_transfer} function transfers tokens from \texttt{\_from} to \texttt{\_to}.
This function is restricted by the modifier \texttt{onlyOwner}, which ensures that each call checks if the caller (\texttt{msg.sender}) is equal to the \texttt{owner}.
If this condition is met, the function proceeds; otherwise, the call fails.In this paper, we propose two criteria for identifying variables that represent privileged accounts (e.g., \texttt{owner} in Fig.~\<PIC>): 1) they are address-type state variables of contracts, and 2) their values can only be specified by the developer or other privileged accounts.


Among all methods using this intrinsic reward function, ATP~\citep{liu2021behavior} first computes kNN distances in a representation space obtained through contrastive learning. Subsequently, \cite{yarats2021reinforcement} propose training an encoder simultaneously during exploration to obtain a prototypical representation space for the kNN computation in Eq. (\ref{Eq: knn-reward}). RE3~\citep{seo2021state} asserts that random encoders are sufficient in many cases, rather than requiring contrastive learning or a prototypical representation. Recently, RISE~\citep{yuan2022renyi} extends it to R\'enyi entropy. Despite their flexibility and satisfactory performance in non-tabular environments, these methods lack episodic exploration, leading to the previously mentioned issue of vanishing lifelong rewards.
 pure intrinsic exploration \citep{amin2021survey} aims to develop an intrinsic reward function $r^{i}(\mathbf{s}_{t},\mathbf{a}_{t})$ as a substitute. It is important to note that most intrinsic rewards are \textbf{non-stationary}, which contradicts the standard MDP framework. The ELEMENT reward consists of two terms represented in blue and red respectively in Fig. \<PIC>:  (1) episodic reward $r_{ep}$ rapidly encourages visiting different states within the same episode, i.e., maximizing episodic state entropy, (2) lifelong reward $r_l$ slowly discourages visits to past states, visited many times in the history, i.e., maximizing lifelong state entropy. Nonetheless, they achieve state-of-the-art performance when utilized to train deep reinforcement learning (DRL) agents in non-tabular environments, where r^{i}(\mathbf{s}_{t},\mathbf{a}_{t})$ does not change rapidly.  Approaches following the maximum state entropy principle \citep{lee2019efficient,liu2021behavior, seo2021state, yuan2022renyi} provide $r^i$ directly proportional to entropy-related measure in a \textbf{non-parametric} and model-free way. Active Pre-Training (APT)~\citep{liu2021behavior} first adopts nearest neighbors as intrinsic rewards in a representation space obtained by contrastive learning. After that, RE3~\citep{seo2021state} proposes to implement random encoders instead of contrastive learning. Recently, RISE~\citep{yuan2022renyi} extends it to R\'enyi entropy. Despite their flexibility and satisfactory performance in non-tabular environments, these methods lack episodic exploration, leading to the previously mentioned issue of vanishing lifelong rewards.
n the other hand, maintaining records of lifelong visited states rely on dense memory and thus a large number of visited states (or their representations) can make the computation of distances prohibitive \citep{seo2021state, yuan2022renyi}.
nother category of maximum state entropy exploration is rooted in sampling instead of intrinsic rewards. From the perspective of intrinsic learning, these methods explore how to achieve an intrinsic objective by decomposing it into multiple \textbf{stationary} reward functions (i.e., MDPs). Early works by \citep{hazan2019provably} offers a provably efficient algorithm which internally splits state entropy in entropy gradients defined using probability density function of state. Multiple approaches have
% been proposed to enhance the method including R\'enyi variant \citep{zhang2021exploration}, and efforts to reduce sample complexity \citep{tiapkin2023fast}.
These sampling-based
ethods involve iterative sampling of new states using the current policy to obtain corresponding entropy-related reward functions, and then update policies by solving the new MDP after fixing these reward functions.
While these sampling-based methods have made significant theoretical contributions, they operate under the assumption that we can ``compute the optimal policy'' to solve a MDP at each iteration given the current entropy-related reward function. This assumption is often unrealistic in real-world environments, where the complexity and unpredictability can hinder consistently reaching near-optimal solutions. Therefore, they usually validate their theoretical results on tabular setups. In this work, we do not compare to sampling-based methods in this category~\citep{hazan2019provably, nedergaard2022k, yarats2021reinforcement,mutti2022importance, jain2024maximum}.
bsection{\textbf{Parametric Methods for Exploration}}
In contrast to MaxEnt-based methods, parametric intrinsic rewards \citep{pathak2017curiosity, burda2018exploration, badia2020never, ecoffet2019go} usually utilize an internal model to predict the next state and use the prediction error as the intrinsic motivation. These methods encourage agents to explore novel states in a lifelong manner by assigning greater rewards to states that are less frequently visited by estimating predictive forward models and use the prediction error as the intrinsic motivation. These \emph{curiosity}-driven approaches, have their roots traced back to the 70's when Pfaffelhuber introduced the concept of ``observer's information"~\citep{pfaffelhuber1972learning} and Lenat \citep{lenat1976artificial} introduced the concept of ``interestingness" in mathematics to promote the novel hypotheses and concepts \citep{amin2021survey}. Recently popular prediction error-based approaches fall under this category. The recent surge in their popularity is strongly linked to the advancements in deep neural networks (DNNs). For instance, ICM \citep{pathak2017curiosity} and RND \citep{burda2018exploration}, utilize a CNN as the internal model to predict the next image, while GIRIL implements variational autoencoder (VAE) to model the transitions in environments. After that, some approaches find the novelty vanishing problem and try to solve it by introducing episodic mechanism \citep{ecoffet2019go, badia2020never}. However, the internal model introduces an auxiliary predictive task that brings additional complexities such as more hyper-parameters and increased running time.
subsection{Fixed neural encoder $f: \mathcal{O} \rightarrow \mathcal{S}$.}  The parametric encoder maps the current observation to a $d$-dimensional state $\mathbf{s}$. Consider a changing encoder that leads to very different representation (state) for the same observation, an agent could visit a large number of different `states' without taking any actions.  To avoid such meaningless exploration, the encoder must be pre-trained and fixed before the exploration agent starts to learn. In this paper, we adopt two encoders for Mujoco and Mario environments, respectively. For Mujoco, we implement a fixed randomly initialized neural encoder which has been empirically validated for state entropy estimation citep{seo2021state}.In this section, we extend the experiments presented in Fig. \<PIC> and analyze the impact of $\beta$ in detail. The Ant agent in Mujoco can navigate freely in all directions within a three-dimensional space. Both agents are reset to randomly initialized starting points near (0, 0) if they fail to meet the health conditions specified by the default gym-Mujoco package. The default number of steps for truncation is set to $1000$, which is directly adopted as $T_{\tau}$ without any fine-tuning. Both $r_{ep}$ and $r_l$ are normalized to the range $[0, 1]$ using min-max normalization. The Soft Actor-Critic (SAC) algorithm serves as the backbone for this analysis. For visualization purposes, we record the x-y coordinates and color them based on logarithmic probability.
  For Mario, we pre-train an encoder using Spatiotemporal Deep Infomax citep{anand2019unsupervised}, a contrastive state representation learning method that maximizes the mutual information across both the spatial and temporal axes. This could become a limitation in environments where it is difficult to collect sufficient data for representation learning, as detailed in Section \ref{sec:limits}.
ubsection{Episodic intrinsic reward.}
The major challenge for episodic entropy maximization is that intrinsic rewards based on episodes instead of current state-action pairs subtly changes the underlying MDP into an episodic POMDP problem:  an agent receives a single feedback $R_{ep}(\tau) = H_{\mathbf{s} \in \tau }(\mathbf{s})$ only at the conclusion of each episode, as discussed above.
To solve this problem, we assume the existence of an underlying standard MDP reward function $r_{ep}(\mathbf{s}_{t}, \mathbf{a}_{t})$ that approximates the episodic reward $R_{ep}(\tau)$ in a sum-form decomposition $R_{ep}(\tau) \approx \sum_{t=1}^{T_{\tau}}r_{ep}(\mathbf{s}_t)$, which is a common trick for trajectory-wise feedback problems \citep{gangwani2020learning, ren2021learning, efroni2021reinforcement}.
More formally, the episodic objective is decomposed to learn an optimal policy by maximizing:


\subsection{Version 1.1} \textcolor{black}{ During the open, live phase of pre-training the first version of TinyLlama, a few implementation issues were identified within the training framework.
For example, there were a few issues related to the learning rate scheduler
and data loading processes. We therefore re-trained a new model from scratch after fixing those issues. The new model is named TinyLlama v1.1. In addition to fixing all those implementation issues, we took the opportunity to incorporate several key modifications in TinyLlama v1.1:}
\begin{itemize}
    \item To reduce communication overhead, we only shard the model parameters within a node in FSDP. \textcolor{black}{We trained with a cluster consisting of 16 nodes, each equipped with four A100-40G GPUs, and set the batch size to approximately 1.8 million tokens.}
    \item We reduced the total number of pre-training tokens from 3 trillion to 2 trillion. Despite the reduction in training data volume, a marginal improvement in performance on downstream tasks was observed, as compared to the original TinyLlama (refer to Section~\ref{sec:results}).
    \item Expanding beyond the singular pre-training phase of the original model, we introduced a three-stage pre-training process inspired by recent research \citep{wei2023skywork}. This includes basic pre-training, continual pre-training targeted at specific domains, and a cooldown phase. An illustrative overview of this approach is provided in Figure~\<PIC>, with detailed discussions to follow in subsequent sections.
\end{itemize}
\paragraph{Basic pre-training} In the first stage of the pre-training, we trained our model with only SlimPajama~\citep{cerebras2023slimpajama} to develop its commonsense reasoning capabilities.  We only trained 1.5 trillion tokens during this stage, setting the foundation for more specialized training in subsequent stages.  \paragraph{Continual pre-training with specific domain} During this phase, we diversified the training by incorporating three distinct types of corpora. The first corpus, identical to the basic pre-training stage, solely consisted of SlimPajama data~\citep{cerebras2023slimpajama}. The second corpus combined code and mathematical content, leveraging integrations with Starcoder~\citep{li2023starcoder} and Proof Pile 2~\citep{azerbayev2023llemma}.  \textcolor{black}{For Starcoder, we only considered the ``Python'' and ``Jupyter'' splits of the original Starcoder dataset.} The third corpus focused on Chinese language data, utilizing Skypile~\citep{wei2023skywork}.  This strategic corpus diversity facilitated the development of three main TinyLlama v1.1 model variants, each tailored to a different need:
\begin{itemize}
\item TinyLlama v1.1: A foundational model for general applications.
\item TinyLlama v1.1 - Math\&Code: Enhanced specifically for mathematical and coding tasks.
\item TinyLlama v1.1 - Chinese: Specialized for processing and understanding Chinese text.
\end{itemize} In this stage, all three variants are trained with 350 billion tokens. For the Math\&Code and Chinese variants, we linearly increase the sampling proportion of the domain-specific corpus for the beginning 6 billion tokens. This warmup sampling increasing strategy was designed to gradually adjust the pre-training data distribution, aiming to ensure smoother and more stable training. Following this initial phase of adaptive sampling, we maintained a consistent sampling strategy for the remainder of the training until approximately 1.85 trillion tokens. Detailed of the data sampling ratios are provided in Appendix~\ref{appendix:data_sampling}.
\paragraph{Cooldown} Implementing a cooldown phase is essential for enhancing model convergence towards the end of the pre-training process.  Traditionally, this is achieved by modifying the learning rate, as seen in approaches like MiniCPM~\citep{hu2024minicpm} and DeepSeek LLMs~\citep{bi2024deepseek}. However, due to the use of a cosine schedule for our model, the learning rate is already low at the later stage of training.

Hence, we opted to modify the batch size instead. Specifically, during the cooldown stage, the batch size was increased from 1.8 million tokens to 7.2 million tokens, while maintaining the original cosine learning rate schedule. This adjustment was applied uniformly across all variants, with each undergoing an additional 150 billion tokens of training during this phase. The training curves for all three variants are shown in Figure~\<PIC>.


\subsection{Analysis}
\label{sec:ana} \paragraph{Scaling Law of Teacher} Although it is intuitive that we can distill better student models from larger teacher models, \cite{teaching_assistant} has shown that increasing the teacher models' sizes does not guarantee the improvement of student models, sometimes even harming the distillation performance. It is not clear how \textsc{MiniLLM} works when we scale up the teacher models' sizes. Therefore, we compare \textsc{MiniLLM} and SeqKD using teacher models with different sizes and fix the size of the student model. We present the results based on the GPT-2 family in Figure \<PIC> and that based on the OPT family in Appendix \ref{app:ts_opt}. We can see that \textsc{MiniLLM} constantly outperforms SeqKD, and the student model performance is positively correlated with the teacher model sizes. This shows the potential of our method to compress models with massive parameters. \paragraph{Exposure Bias}
Language generation models trained to minimize \textit{forward} KLD suffer from exposure bias~\citep{exposure_bias} caused by the discrepancy between teacher-forcing training and free-run generation. When training \textsc{MiniLLM}, the student model sees samples generated by itself, alleviating the training-inference mismatch~\citep{cold}. In Figure \<PIC>, we use the ExAccErr metric~\citep{eb_measure} defined in Appendix \ref{app:eb} to measure the excess accumulated error due to exposure bias. The experiment is based on GPT-2-125M, with GPT-2-1.5B as the teacher, using Dolly as the test set. For each prompt, we sample 10 responses to reduce the variance. We can see that the ExAccErrs of the baselines continuously grow during generation, while \textsc{MiniLLM} has a much lower ExAccErr, and the error stops accumulating in long-text generation ($>$ 150 tokens).   \paragraph{Calibration} \cite{gpt4} has shown that models trained with policy optimization are likely to be poorly calibrated. We test the calibration of \textsc{MiniLLM} and the KD baselines on two widely-used text classification datasets: SST2~\citep{sst-2} and BoolQ~\citep{boolq}, based on LLaMA-7B. We design zero-shot classification instructions (see Appendix \ref{app:eval_detail}) and take the probability of the label words to compute the ECE scores~\citep{ece}. From Table \ref{tab:calibration}, we observe that KD and SeqKD models are worse calibrated than the teacher model, which potentially explains their low performance on canonical benchmarks~\citep{false_imitate}. We suspect that minimizing \textit{forward} KLD causes the models to push high probabilities to void regions of the target distribution, which leads to significant distribution differences between the student and the teacher (see the example in Figure \<PIC>). In contrast, \textsc{MiniLLM} focuses on accurately learning the major parts of the target distribution and narrows the ECE scores gap between the student and the teacher.
\paragraph{Performance on Different Response Length} We study the models' performance when the golden response lengths belong to different ranges. In Figure \<PIC>, we illustrate the Rouge-L scores of different KD models against the SFT models on three S-NI subsets split by the length of the ground truth responses. We can see that all methods achieve low scores on prompts that expect short responses ($\le 5$ tokens), probably because most responses in our training set are long sentences, introducing a distribution shift between training and evaluation~\citep{ITGPT4}. Furthermore, the output spaces of these prompts are relatively small, allowing the student model to cover most modes of the teacher, and thus \textit{reverse} KLD and \textit{forward} KLD have similar performance. For prompts with longer responses ($\ge 6$ tokens), the teacher distribution contains more modes than the students due to the complex output spaces, which shows the advantage of \textsc{MiniLLM} against standard KD models. Similar results on UnNI are shown in Appendix \ref{app:length_uni}. \paragraph{Generation Diversity} \cite{gan_falling_short} has found that the model optimized by minimizing \textit{reverse} KLD is likely to lose modes, which affects the generation diversity. We follow~\cite{cold} to discuss generation diversity from three aspects:
(i) generating multiple distinct responses given a prompt. (ii) generating linguistically complex responses. (iii) the ability to generate contents that have high coverage of the real data distribution. For (i), we argue that for many NLP applications, generating one \textbf{correct} response is sufficient, especially for those scenarios demanding high truthfulness and reliability~\citep{halu_survey}. For (ii) and (iii), we report the responses' distinct 4-gram proportion and the language modeling loss on the test sets in Table \ref{tab:diversity}, using the base models from the LLaMA family (see Appendix \ref{app:diverse} for more details) . We can see that \textsc{MiniLLM} preserves the distinct 4-gram proportion in the generated responses and language modeling loss on the test set.


In this work,
we aim to prune the source model into any target configuration that we specify.This goal is challenging because it requires surgically scaling down all dimensions in a transformer architecture, an endeavor that, to our knowledge, has not been accomplished before for large language models.
We leverage the configurations of existing pre-trained models as the target architectures,
based on the intuition that
these configurations have already been well-optimized to
balance model expressivity and inference efficiency.
For example, we use the INCITE-Base-3B architecture~\citep{toegther2023incite}
as the target structure when producing a $2.7$B model.
Our method learns a set of pruning masks
on  model parameters at different granularities---from
global ones like layers and hidden dimensions (persist across all layers),
to local ones like attention heads and intermediate dimensions.
Assume that the source model $\mathcal{M}_S$ has
$L_\mathcal{S}$ layers, with each layer consisting of one multi-head attention module (MHA) and one feed-forward network (FFN). $\mathcal{M}_S$ has a hidden state dimension of $d_\mathcal{S}$,
$H_\mathcal{S}$ heads in each MHA, and
an intermediate dimension of $m_\mathcal{S}$ in each FFN. We introduce the following mask variables:
Each mask variable controls whether the associated substructure is pruned or retained. For example, we remove a layer if its corresponding $z^{\text{layer}}=0$. Figure \<PIC> illustrates an example of how the pruning masks control the pruned structures.
We analyze the effectiveness of dynamic batch loading by examining its impact on three aspects:
(1) the final LM loss across domains,
(2) the data usage of each domain throughout training,
(3) the downstream task performance.
All results in this section are based on Sheared-LLaMA-1.3B.
\paragraph{Loss differences across  domains.} Dynamic batch loading aims to balance the rate of loss reduction across domains, ensuring that the losses reach the reference value at roughly the same time. \C<PIC> shows the difference between our model's loss (with both original and dynamic batch loading) and the reference loss, estimated by fitting a scaling function to a hypothetical 1.3B parameter LLaMA2 model. The original batch loading results in widely varying loss differences across domains; for example, the GitHub loss decreases below the reference value, while the C4 loss lags behind. Dynamic batch loading, however, reduces losses evenly and leads to very similar loss differences across domains, suggesting more efficient data use.


As discussed in the previous section, we speculate that the LLMs exhibit layer redundancy. To verify this, we assess the performance degradation caused by removing individual layers of two popular models, Llama2-7B-Base \citep{touvron2023llama}, an English based LLMs, and Baichuan2-7B-Base \citep{yang2023baichuan} which is mainly focused on Chinese. Figure \<PIC> confirms our speculation, which reveals that some layers do not play a crucial role in LLMs, causing little degradation when omitting them individually. Moreover, this redundancy is primarily manifested in the middle to later layers of the network, with the initial layers and the last layer often being more critical. Notably, we found the last layer to be particularly important, aligning with findings from LLM Pruner \citep{ma2024llm}. This observation contradicts our mathematical explanation in Appendix \ref{appendix:math} which suggests that deeper layers tend to be more redundant. We posit that this discrepancy arises because the final FFN effectively functions as part of the token classifier and should be considered in conjunction with the language model head.To verify our hypothesis, we conducted further investigation, detailed in Table \ref{tab:last_layer}. The results show that within the last layer, the FFN component is crucial, while the Attention module is less significant. This finding supports our interpretation of the final layer's importance.
\section{Methodology}
In this section, we present the methodological framework of our layer removal approach for LLMs, elucidating the underlying principles and techniques employed. We begin by introducing Block Influence (BI), a novel metric designed to assess the hidden states transformation of each layer. Leveraging BI, we then detail our layer removal method.
\subsection{Layer importance} \label{method:layerimportacne} As outlined in the preceding section, the layers of LLMs exhibit redundancy, with varying degrees of redundancy across different layers. To capture this, we introduce a new metric, Block Influence (BI), to measure the degree of transformation performed by each layer.   The BI score of $i^{th}$ layer can be calculated as follows:
\begin{align}
    \text{BI}_i = 1 - \mathbb{E}_{X,t} \frac{X_{i,t}^TX_{i+1,t}}{||X_{i,t}||_2||X_{i+1,t}||_2},
\end{align}
where $X_{i,t}$ means the $t^{th}$ row of hidden states of $i^{th}$ layer. Lower BI score imply that $X_i$ and $X_{i+1}$ exhibit high cosine similarity, suggesting that the layer makes minimal transformations to the hidden states and is therefore less important. We plot the BI scores of a single layer and the PPL after removing it separately, as shown in the Figure \<PIC>. The results demonstrate a positive correlation between the BI score and the importance of a layer.


\subsection{Reinforcement Learning with Foundation Priors}
\label{sec:method:flfp}
We model the tasks for embodied agents as the Goal-Conditioned Markov Decision Processes (GCMDP) $\mathcal{G}$: $\mathcal{G} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, {\mathcal{T}}, \gamma)$. $\mathcal{S} \in \mathbb{R}^{m}$ denotes the state.
$\mathcal{A}$ is the action space, which is the continuous delta movement of the end effector in this work. $\mathcal{P}$ is the transition probability function.
$\mathcal{T}$ is the task identifier. $\mathcal{R}$ denotes the rewards.
$\gamma$ is the discounting factor, equal to 0.99 in the work.
To learn efficiently and automatically, we propose the Reinforcement Learning from Foundation Priors (\textbf{RLFP}) framework by leveraging the policy, value, and success-reward priors.  Here we demonstrate how we formulate the priors in RLFP. Back to the case of Alice in Fig. \<PIC>, the commonsense of behavior can be formulated as a goal-conditioned policy function, $M_\mathcal{\pi}(s, \mathcal{T}): \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A}$. The prior knowledge that the state closer to the button is closer to success can be formulated as the value function $M_\mathcal{V}(s, \mathcal{T}): \mathcal{S} \times \mathcal{T} \rightarrow \mathbb{R}^{1}$. The ability to recognize the success state can be formulated as the 0-1 success-reward function $M_\mathcal{R}(s, \mathcal{T}): \mathcal{S} \times \mathcal{T} \rightarrow \{0, 1\}$, which equals 1 only if the task succeeds.
We assume the success-reward prior is relatively precise, given the simplicity of binary classification in determining success. The value and policy prior knowledge are noisier.
The RLFP framework is to solve an expansion of $\mathcal{G}$, termed $\mathcal{G^{'}} = (\mathcal{G}, \mathcal{M})$, where $\mathcal{M}$ is the foundation model set that represents various foundation prior knowledge. Here, $M_\mathcal{\pi}, M_\mathcal{V}, M_\mathcal{R} \in \mathcal{M}$. Compared to vanilla RL, all the signals for the RLFP come from the foundation models. The vanilla RL relies on uninformative trial and error explorations and manually designed reward functions. It is not only of poor sample efficiency but also requires much human reward engineering. Instead, in RLFP, prior knowledge from the foundation model set $\mathcal{M}$ provides guidance or feedback on policy, value, and success-reward, enabling more automatic and effective task resolution. \subsection{Foudation-guided Actor-Critic}
\label{sec:method:fac}
Under the proposed RLFP framework, we instantiate an actor-critic algorithm named Foundation-guided Actor-Critic (FAC), demonstrating how to inject the three priors into RL algorithms. \textbf{Guided by Success-reward Signals.}
We consider the task as MDP $\mathcal{G}_1$ with 0-1 success rewards, where $\mathcal{R}_{\mathcal{G}_1} = M_\mathcal{R}(s, \mathcal{T}) \in \{0, 1\}$. Inspired by how humans learn from successful trials, we propose a success buffer to store the ``successful'' trajectories identified by $M_{\mathcal{R}}$. Each time the actor $\pi_{\phi}$ updates via policy gradient, it also imitates samples from the success buffer $\mathcal{D}_{\text{succ}}$ (if available). The objective is $\mathcal{L}_{\text{succ}}(\phi) = \textbf{KL}(\pi_{\phi}(s_t), \mathcal{N}(a_t, \hat{\sigma}^2)), s_t, a_t \sim \mathcal{D}_{\text{succ}}$, where $\hat{\sigma}$ is the standard deviation.  \textbf{Guided by Policy Regularization.} To encourage efficient explorations, we regularize the actor $\pi_\phi$ by the policy prior from $M_{\pi}(s, \mathcal{T})$.
Assuming the prior follows Gaussian distributions, the regularization term is $\mathcal{L}_{\text{reg}}(\phi) = \text{KL}(\pi_\phi, \mathcal{N}(M_{\pi}(s_t, \mathcal{T}), \hat{\sigma}^2))$, which is commonly used in other algorithms \citep{rot, modem-v2}. The bias introduced by the policy prior is bounded, shown in Theorem \ref{theorem:policyv2}.  \textbf{Guided by Reward-shaping from Value Prior.}
Noisy policy prior can mislead agents to undesirable states, so we propose using the value model $M_{\mathcal{V}}$ to guide exploration and avoid unpromising states. While initializing and fine-tuning with $M_{\mathcal{V}}(s, \mathcal{T})$ is a natural approach, it suffers from catastrophic forgetting.
Instead, we employ the \textbf{reward-shaping} technique \citep{reward-shaping} using the potential-based function $F(s, s', \mathcal{T}) = \gamma M_{\mathcal{V}}(s', \mathcal{T}) - M_{\mathcal{V}}(s, \mathcal{T})$, where $\gamma$ is the discount factor. Since $M_\mathcal{V}$ estimates state values, $F$ measures the value increase from $s$ to $s'$. This shaping reward is positive when $s'$ is better than $s$ and shares the same optimal solution as the 0-1 success-reward MDP $\mathcal{G}_1$. Proof and details are in App. \ref{app:reward_shaping}. \textbf{Foundation-guided Actor-Critic.}
In summary, we deal with a new MDP $\mathcal{G}_2$, where $\mathcal{R}_{\mathcal{G}_2} = \lambda M_\mathcal{R} + F$, with $\lambda$ (set to 100) emphasizing success feedback. We train the agent using DrQ-v2 \citep{drq-v2}, a variant of Actor-Critic, and call the proposed method Foundation-guided Actor-Critic (\textbf{FAC}). As shown in Fig. \<PIC>, FAC leverages foundation policy guidance and an automatic reward function, enabling the agent to efficiently learn from abundant prior knowledge. The objectives of FAC are detailed in Eq. (\ref{eq:actor_fac}), where tradeoff parameters $\alpha$ and $\beta$ are both set to 1, $y$ is the n-step TD target, and $Q_{\Bar{\theta}}$ is the target network. We use clipped double Q-learning \citep{double-q} to reduce overestimation. \begin{equation}
    \label{eq:actor_fac}
    \begin{aligned}
        \mathcal{L}_{\text{actor}}(\phi) &= - \mathbb{E}_{s_t \sim \mathcal{D}}\left[ \min_{k=1,2} Q_{\theta_k}(s_t, a_t) \right] + \alpha \mathcal{L}_{\text{succ}} + \beta \mathcal{L}_{\text{reg}}; a_t \sim \pi_{\phi}(s_t) \\
        \mathcal{L}_{\text{critic}}(\theta) &= \mathbb{E}_{s_t \sim \mathcal{D}}\left[ (Q_{\theta_k}(s_t, a_t) - y)^2 \right]; y = \sum_{i=0}^{n-1} \gamma^i r_{t+i} + \gamma^n \min_{k=1,2} Q_{\Bar{\theta}_k}(s_{t+n}, a_{t+n})
    \end{aligned}
\end{equation}


Inspired by the above works, we propose a privacy-preserving data-free distillation method. As shown in Fig.~\<PIC>, publishing a model ~(\eg, teacher model) trained directly from private data would compromise privacy, so we treat it as a fixed discriminator to train a generator in a data-free manner. This generator learns only the data distribution to protect the private data. Using this generator implicitly generates data for the distillation process from teacher model to student model. Because querying the teacher model using the generated synthetic data can compromise private information, we propose a LabelDP algorithm selective randomized response to protect the output of the teacher model. The selective randomized response algorithm treats the output of the student model as prior knowledge to reduce the possible output categories to increase the probability of outputting the correct label, and if the possible output does not contain the correct label, a uniform probability distribution is used to reset the possible probability of the output. In summary, our approach can effectively learn privacy-preserving student model by two keys.
On the one hand, our proposed data-free distillation is able to protect privacy well with the learning of data distribution. The generated synthetic data from this generator will not reveal private information even if it is distributed. On the other hand is that we propose the selective randomized response module to implement DP, which is no longer limited by the number of queries, and introduce the prediction of the student model as prior knowledge for the randomized response. We increase the probability of returning the correct label by setting a threshold, so the student model can learn the knowledge of the teacher model more effectively.
Our major contributions are three folds: 1)~we propose a differentially private data-free distillation approach to learn privacy-preserving and high accurate student models via synthetic data, 2) we propose selective randomized response algorithm to privately distill teacher knowledge which provides strong protect label privacy protection in theory, and 3) we conduct extensive experiments and privacy analysis to demonstrate the effectiveness of our approach.
\subsection{Ablation Studies}
After the promising performance is achieved, we further analyze each influencing factor in our approach, including the impact of loss terms in the data-free generator learning, the amount of synthetic data and the number of stages. \myPara{Loss function.} To further understand the improvement of each component of the loss function during data-free training of the generator, we designed experiments on MNIST and FMNIST under $\varepsilon$=10 to explore the contribution of each component. The results are shown in Tab.~\ref{tab:loss}. where CE means the cross entropy loss term, IE is the information entropy loss term and Norm is the normalized term for the mean and variance of the data. We can see that the normalization term of the data has the greatest impact, followed by the information entry loss term and finally the cross entropy loss term. We speculate that this may be related to the randomness of the data generated by the generator, which limits the distribution of the data to make the generated synthetic data more usable, so it has a greater impact on the accuracy of the student model. \myPara{Data amount.}
We further conducted experiments on MNIST, FMNIST, CIAFR10 and CIFAR100 datasets under $\varepsilon=1$. The results are shown in Fig.~\<PIC>. We found that MNIST dataset converges at about 50,000 data volume, FMNIST converges at about 120,000, CIFAR10 and CIFAR100 converge at about 220,000 and 500,000, respectively. As the difficulty of datasets increases, the amount of data required to achieve convergence increases. We suspect that this is because the more difficult the dataset is, the more difficult its distribution knowledge is to learn, so the larger the amount of data required. We note that the CIFAR10 dataset is more difficult than FMNIST, but the reason why CIFAR10's final accuracy is similar to FMNIST's is that the network structure is different.
\myPara{Number of stages.} To explore the effect of the number of stages, we conducted experiments on MNIST, FMNIST and CIFAR10 datasets under $\varepsilon$=10. The results are shown in Fig.~\<PIC>. Experimental results show that between 20 and 320, the accuracy of the student model increases with the increase of stages. As the classification difficulty of MNIST, FMNIST and CIFAR10 datasets increases, the effect of stages becomes greater. The experimental results are as we expected because we used the prediction of the student model as the prior knowledge. As the training process proceeds, the more accurate the prediction of the student model becomes, which means the higher the probability of outputting the correct label. The greater the percentage of synthetic data being correctly labeled, the better the student model performance will be.


Align before or after LLM Decoding?
According to \cite{Jiang_2023_CVPR}, complete alignment between modalities is often not the optimal solution for downstream tasks. Such alignment may result in information loss, especially when the information provided by the two modalities differs significantly.
Traditional SEC models achieve Speech-Text Alignment via fine-tuning encoder on speech-caption pairs, which bridges the <PIC> gap before LLM decoding. However, complete alignment of speech and text embedding may result in information loss, and it lacks a direct measure for assessing speech-text alignment quality.  To address these problems, we propose KD-Regularization which achieve Speech-Text Alignment and bridge the <PIC> gap after LLM decoding. It use the KL-divergence of next-token prediction <PIC>s between LLM's response as a measure of Speech-Text Alignment. As shown in Fig~\<PIC>, we observe that align after LLM decoding using knowledge distillation can more effectively improve the speech-text alignment performance.
\section{AlignCap}
\subsection{KD-Regularization}
\label{sec:section3.1} Our goal is to generate speech emotion captions for speech clips, we design a student LLM to implement speech tokens to text generation and employ a teacher LLM's response to guide student LLM's next-token generation. LLaMA-7B \cite{llama1} is chosen to implement this decoding process due to its exceptional language understanding and modeling capabilities. We simply choose rank value of 8 for LoRA fne-tuning~\cite{hu2022lora} conducted on Student-LLaMA, while the Teacher-LLaMA parameters are frozen. \para{Acoustic Prompt.}
We first construct a vocabulary of emotional clues, adjectives such as the speaker's tone, intonation, pitch, rhythm, and volume in captions are all regraded as emotional clues. We design an emotion grammar parser (based on NLTK toolkit) to recognize these clues, which are filtered by the vocabulary. Then these clues are inserted into a prompt template P$_{\mathrm{T}}$: <Feeling $e_{1},e_{2},...,$ and $ e_{n}$>, where $e_{n}$ is the $n_{th}$ emotion entity. The acoustic prompt can capture rich and delicate emotion information in emotional clues.
It can enrich fine-grained emotional description and enhance the robustness of zero-shot captioning for unseen speech, leveraging its training-agnostic nature, which is denoted as:
\begin{equation}
\begin{split}
&e_{1\sim n} = G_{\mathrm{Parser}}(y_{i}=\{ c_{i}^{1},...,c_{i}^{|y_{i}|}\})     \\
&\mathrm{P_{act}} = \mathrm{Insert}(\mathrm{P_{T}}, idx, e_{1\sim n}) \\
\end{split}
\end{equation}
Where $y_{i}$ is a series of captions, $c_{i}^{m}$ is the $m_{th}$ caption of $y_{i}$. $G_{\mathrm{Parser}}$ and P$_{\mathrm{T}}$ represent emotion grammar parser and prompt template respectively. We insert the emotional clues $e_{1\sim n}$ into the index position $idx$ of P$_{\mathrm{T}}$ to get acoustic prompt P$_{\mathrm{act}}$.
\para{Text Token Generation.}
We denote the captions in speech-caption pairs as the semantic prompt P$_{\mathrm{sem}}$ and concat P$_{\mathrm{act}}$ and P$_{\mathrm{sem}}$ as a prefix prompt, then we provide the prefix prompt along with an instruct prompt (user's instructions) to the LLM to condition its subsequent generation of speech emotion captions using prefix language modeling.
This setup leverages external knowledge and the language understanding and modeling capabilities of the teacher-LLM to guide the student-LLM to generate plausible sentences.
Given a caption $c_{i}$ with token $T_i$, language model $P_{\theta}$ learns to reconstruct $y_i$ conditioned on the P$_{\mathrm{act}}$ and P$_{\mathrm{sem}}$. The probability of generating the next token is calculated as follows:
\begin{equation}\label{lm_model}
p_{\theta}(T_t \mid
\underbrace{T_{0\sim a-1}}_{\mathrm{P_{act}}},
\underbrace{T_{a\sim b-1}}_{\mathrm{P_{sem}}},
\underbrace{T_{b\sim c-1}}_{\mathrm{P_{instruct}}},
\underbrace{T_{c\sim t-1}}_{\text{autoregressive}})
\end{equation}
This process is iterated until the LLM generates a token containing a period and the training loss is the maximum likelihood estimate, and the next token $T_t$ is selected according to:
\begin{equation}
\label{texttoken}
\begin{split}
T_t=\mathop{\arg\max}_{i \in 1, \ldots, k}
\Big\{
p_{\theta} \left(c_i \mid p_n, T_0,\!\ldots\!,T_{t-1} \right)
\Big\}
\end{split}
\end{equation}
Where prefix prompt $p_n$ = P$_{\mathrm{act}}$ $\oplus$ P$_{\mathrm{sem}}$ $\oplus$ P$_{\mathrm{instruct}}$.
Trained on limited data, simply using semantic prompt as prefix prompt may overfit the In-Domain dataset, leading to significant domain shift and performance degradation of language model using out-of-domain (OOD) speech.
In contrast, the acoustic prompt based on emotion-aware clues, inherits the powerful transferability from captions.
\para{Speech Token Generation.}
For each speech, we adopt the pre-trained SpeechTokenizer~\cite{zhang2024speechtokenizer} to extract discrete representations and denote the tokens of the first  residual vector quantization (RVQ) layer as speech tokens.
The first layer of RVQ can be regarded as a semantic token, which contains more content information from speech, resulting in capturing semantically accurate emotional clues. We append this speech token $x_t$ to LLM's input and generate the next token in an autoregressive modeling manner, for each time step $t$, the next token $T_{t}$ is selected according to:
\begin{equation}
\label{speechtoken}
\begin{split}
T_t=\mathop{\arg\max}_{i \in 1, \ldots, k}
\Big\{
p_{\theta} \left(c_i \mid x_t, T_0,\!\ldots\!,T_{t-1} \right)
\Big\}
\end{split}
\end{equation} \para{Modality Alignment.}
Modality adapters~\cite{zeroshotaudiocap2,wavllm} are often used to compress the speech encoder's feature representations. Similar to \cite{multitokenizer}, we treat the input from speech and text modality as a token sequence and learn a joint embedding space for all modalities.
Speech tokens are expanded to text token's codebook in advance so that text and speech share the same codebook.
We pad the shorter token sequence to make it the same length as the longer token sequence. We use a mask to ignore the padding part, ensuring that the model only focuses on valid tokens.  \para{Knowledge Distillation.}
As shown in Fig~\<PIC>, given a $\mathcal{D}_{s}\!=\!\{(x_{n},y_{n})\}$, we treat the LLM's prediction <PIC> $p_{\theta} \left(y_n \!\mid\!p_n, y_{<n} \right)$ of the next response token, after having observed the text input $p_n$ and generated partial response $\{y_0,\!\ldots,\!y_{n-1}\}$, as the teacher <PIC>. Where $p_n$ is the concatenation of P$_{\mathrm{act}}$ and P$_{\mathrm{sem}}$.
In contrast, we consider the corresponding <PIC> $p_{\theta} \left(y_n \!\mid\!x_n, y_{<n} \right)$ for the speech input $x_n$ as the student <PIC>.
If speech and text are well-aligned, the two <PIC>s should be close to each other, as measured by KL-divergence, which is as followed:
\begin{equation}
\begin{split}
&\mathop{\min}_{\mathrm{LLM_{stu}}(\cdot)}\mathcal{L}_{\mathrm{KL}}(p,x,y)= \\
&-\sum_{t,y_n}p_{\theta}\left(y_n|p_n,y_{<n}\right)\log p_{\theta}\left(y_n|x_n,y_{<n}\right)
\end{split}
\end{equation} $\mathcal{L}_{\mathrm{KL}}$ introduces a quantitative measure of speech-text alignment at each step of the response generation process. By minimizing this loss, we can learning a student LLM using LoRA fine-tuning \cite{hu2022lora} for speech input that facilitates generation behaviors similar to those of text inputs when generating speech emotion captions.


In this paper, we systematically addressed the above problems from three angles: feature construction, algorithm design, and the development of an easy-to-use tool. As illustrated in Figure~\<PIC>, our contributions can be summarized as follows: \begin{itemize}
    \item \textit{Our first contribution} is the proposal of  \texttt{Tone2Vec}, which maps diverse tone transcriptions to a comparable feature space. \texttt{Tone2Vec} constructs pitch-based similarity representations by mapping each transcription to a simulated smooth pitch variation curve. We also propose methods to construct tonal representations for dialect regions. By analyzing these representations across different dialect areas, we show that \texttt{Tone2Vec} captures tonal variations and clusters dialects more accurately than methods that treat each tone as an isolated category.
    \item \textit{As our second contribution}, we developed the first automated algorithms for tone transcription and clustering. These algorithms are especially beneficial for endangered tonal languages. Experiments demonstrate that our models perform well in cross-regional tone transcription with less than 1,500 samples. Notably, our algorithms can accurately cluster tones using fewer than 60 speech samples for a given dialect.
    \item \textit{As our third contribution}, all these algorithms are systematically integrated into \href{https://github.com/YiYang-github/ToneLab}{\texttt{ToneLab}},  a user-friendly platform designed for both lightweight fieldwork and subsequent analysis in Sino-Tibetan Tonal Languages. Users can choose to use pretrained models or train new models with their own data for different scenarios. Researchers can also leverage \texttt{ToneLab} to propose new computational methods and conduct evaluations.
\end{itemize} \section{Tone2Vec: From Tones to Vectors}
\label{sec:method} In this section, we propose pitch-based similarity representations by quantifying the differences in pitch variations inherent in tones, which we call \texttt{Tone2Vec}. \texttt{Tone2Vec} is an easy-to-use, simple, and effective method for measuring similarity distance. \texttt{Tone2Vec} not only enables the comparison of tonal variations across dialects but also provides a straightforward loss function for training automatic tone transcription and clustering models. \subsection{From Categories to Pitch-based Similarity Representations}
In \texttt{Tone2Vec}, we map each transcription $l$, such as \texttt{(55)}, to a simulated smooth pitch variation curve $p_{l}(x)$. As shown in Figure~\<PIC>, for transcriptions with two units, a linear curve is employed to represent pitch variations, while for those of three units, such as \texttt{(312)}, we employ a quadratic curve to smoothly interpolate the points \((1, 3)\), \((2, 1)\), and \((3, 2)\). The divergence between any pair of tone transcriptions, \(l_1\) and \(l_2\), is quantitatively assessed by calculating the area between their pitch variation curves, expressed as \(D(l_1, l_2) = \int_{[1,3]} |f_{l_1}(x) - f_{l_2}(x)| dx\). This measure quantifies the differences in pitch variations.
Given $n$ transcription sequences $l_{1},...,l_{n}$, we can construct a $n \times n$ distance matrix $\mathcal{C} = {(D(l_{i}, l_{j}))}_{i,j} \in \mathbb{R}^{n \times n}$, where each row represents the features of a transcription, capturing the subtle pitch variation differences among them.


\section{Introduction}
Generative Language Models (LMs) have gained considerable attention due to their remarkable capabilities \cite{guo2023close,suzgun-etal-2023-challenging}. Despite the convincing and realistic nature of text generated by these LMs, a concern with LMs lies in their tendency to produce fabricated facts and generate false information \cite{lin-etal-2022-truthfulqa}. Moreover, these models deliver inaccurate information employing unequivocal expressions, which poses substantial risks as it can lead to the spread of misleading and harmful content. One of the contributing factors to the hallucination lies in inadequate acquisition of knowledge \cite{manakul2023selfcheckgpt,huang2023factual}. For example, consider the question \emph{Which animal is China's nation treasure?}, LMs may provide a different animal name like \emph{tiger} instead of \emph{panda} due to a lack of relevant knowledge. Considerable efforts have been made to alleviate such hallucination induced by lacking of knowledge in LMs. One approach involves supervised fine-tuning LMs with standard ground-truth answers to enhance their comprehension of relevant knowledge \cite{wei2022finetuned,ouyang2022training}. This method has shown promising efficacy. However, it demands a significant amount of high-quality annotated data for training. Additionally, other methods have relied on external verifier or critic model to evaluate the accuracy of a statement \cite{yang-etal-2022-re3,paul2023refiner}. Training a verifier necessitates a large number of high-quality evaluation annotations and further fine-tuning of the model, which restricts its broad applicability to other tasks and domains.  Another reason for an LM to provide incorrect response is intrinsically linked to the design architecture of generative language models themselves \cite{azaria2023internal,paul2023refiner,shinn2023reflexion}. It is widely acknowledged that LMs generate a sentence by maximizing the likelihood of the next token given all previous tokens. Subtle differences in the preceding sentences can potentially lead to diverse generation outcomes. For example, when the question is \emph{Who is the author of The Analects?}, the model gives the correct answer as \emph{Confucius}. However, when the input question becomes \emph{Is Laozi or Confucius the author of the Analects of Confucius?}, the model is likely to generate an answer of \emph{Laozi}. In this case, the model has the ability to rely on its knowledge to recognize false information \cite{schick2022peer}. This process is akin to how humans perform self-verification of answers to minimize mistakes \cite{flower1981cognitive}. Moreover, when we realize our answer is wrong, we further modify it. Motivated by this, when the model itself detects the potential hallucination, the next step is to correct the error or mistake. Once the model incorporates this inherent self-correction mechanism, it can address similar issues in other domains and achieve self-improvement.  The existing work \cite{madaan2023selfrefine,ganguli2023capacity} towards self-correction in LMs has mainly focused on lager models like ChatGPT and GPT4, which is challenging to migrate these self-correction methods to small LMs. Some studies indicated that the self-correction ability depends on model parameters and only emerges in models with larger parameter sizes \cite{azaria2023internal}. The main reason is that they devised a sophisticated pipeline and zero-shot prompts to achieve self-correction. However, these prompts crafted for self-verification and self-modification are difficult for small models to understand. As shown in Figure \<PIC>, upon generating the initial answer to the given question, an additional feedback instruction is utilized to guide ChatGPT in generating feedback information regarding the initial answer. This information contains an evaluation of the correctness of the initial answer. A subsequent modification instruction is employed to alter or refine the initial answer based on the feedback received.  Nevertheless, small models typically lack self-awareness \cite{weng2022large} and tend to exhibit greater confidence in their generated responses. Consequently, they struggle to assess the quality of their generated outcomes. The capability for self-verification serves as a prerequisite for achieving self-correction. Furthermore, the manner in which self-correction is achieved through multi-step prompt engineering within LMs differs from the spontaneous and one-time correction observed in humans. To empower the capability for self-correction in small language models, we propose \emph{Intrinsic Self-Correction} (ISC), an intrinsic mechanism that relies on two basic abilities: self-verification and self-modification. At its core, the LM provides a response and subsequently evaluates its own answer. Upon identifying an error, the same LM adjusts its initial response. Conversely, if the answer is validated as accurate, no further modifications are required. The self-correction process is not divided into two separate steps, but rather constitutes a single comprehensive step, as depicted by the red arrowed segment in Figure \<PIC>. We trained the LM to process the self-correction through Instruction Fine-Tuning (IFT). For this purpose, we design the data processing procedure to construct the self-correction data and define the data format. During the fine-tuning process, we propose Partial Answer Masking (PAM) to make the model have the ability of self-verification. Our contributions are summarized as follows:
\begin{itemize}
    \item To the best of our knowledge, we are the first to demonstrate that small language models with even 6 billion parameters possess the capacity for self-correction during response generation without relying on ground truth.
    \item Our proposed \emph{Intrinsic Self-correction} aims to incorporate self-correction as an intrinsic pattern within LM. It involves an independent and spontaneous self-correction process, distinct in nature from existing methods of self-correction that rely on prompt engineering.
    \item To achieve the capability for self-correction in small LMs, we devise a pipeline for constructing self-correction data and define the data format. It can be universally applied to build data for self-correction tasks. Additionally, we introduce a novel training method called  Partial Answer Masking (PAM) to enable the model self-verify its own generated answers.
    \item We conduct experiments on open-source LMs with varying parameter scales to validate the efficacy of our proposed method. The results demonstrate improvements in accuracy across two different datasets.
\end{itemize} \textbf{Answer preparation.} To ensure answer diversity, we utilize nucleus sampling \cite{holtzman2019curious} to generate multiple answers. After generating multiple answers for a question, the next step is to evaluate the accuracy of each answer by comparing it with the provided ground truth for the each question. In the case of multiple-choice questions, we extract the options of the final answer through string matching and then directly compare them with the standard answer to check the accuracy. For a good case, the outcome of self-verification should be positive, indicating there is no need to modify the initial answers. Accordingly, given the question $x$, the \textbf{Answer} is represented as ($A_{1}^{1}$-\emph{COT}$||A_{1}^{1}||PV $), where $A_{n}^{i}$ represents the model attempts $n$ times to obtain the correct answer, and the current answer is the $i$th response, $A_{n}^{i}$-COT represents the COT process of the answer, $PV$ denotes the positive verification in self-correction, and  $||$ represents the concatenation. Here, we set the verification as a binary signal. A positive verification can be set like \emph{I am sure my answer is correct.}  Conversely, for a bad case, negative self-verification result is excepted, such as \emph{Sorry, there is an error in the previous answer}. It requires modifications to the initial answer. To enhance the model's ability to generate more appropriate reasoning process and correct answer, we utilize the ground truth, representing the standard answer, as the revised answer. Additionally, we employ gpt-3.5-turbo to assist in generating the COT analysis process, denoted as $G$. We use prompts like \emph{the answer of [Question] is [Ground Truth]. Please provide a step-by-step explanation for resolving the given problem}. Therefore, the data format is ($A_n^{1}$-COT$ ||A_n^{1}||NV||A_n^{2}$-COT$||A_n^{2}\cdots A_{n}^{n}||PV)$, where $NV$ indicates negative verification. In Table \ref{dataFormat}, we provide two general examples of self-correction data, representing examples where the correct answer is obtained without correction and with one correction respectively. We also provide detailed prompt examples used at each step in the Appendix. This pipeline can be utilized to customize the self-correction data for various corrections, depending on the specific task type. The general process of constructing self-correction data is shown in Figure \<PIC>.


\section{AIDBench details}
In this section, we provide a detailed description of AIDBench. We begin by presenting an overview of the benchmark, outlining the pipeline for authorship identification using large language models (LLMs). We then describe the datasets used in our benchmark, the evaluation tasks conducted, and the metrics employed for assessment.
\subsection{Outline of authorship identification with LLMs} Figure \<PIC> provides an overview of our proposed AIDBench framework. We begin by selecting a dataset for evaluation, such as the Research Paper dataset. From this dataset, we sample a subset of texts from several authors, randomly selecting one as the \emph{Target Text} and designating the remaining texts as candidates. These texts are then incorporated into an authorship identification prompt, which is presented to the LLMs. The models generate responses indicating which candidate texts are more likely authored by the same individual as the \emph{Target Text}. We repeat this process multiple times to obtain average performance metrics for the task. Finally, we employ metrics such as precision, recall, and rank to provide a clear and intuitive assessment of the LLMs' capabilities.
\subsection{Datasets}
AIDBench comprises five datasets: \textit{Research Paper}, \textit{Enron Email}, \textit{Blog}, \textit{IMDb Review}, and \textit{Guardian}. In this subsection, we provide detailed descriptions of each dataset.
\textbf{Research Paper}. This newly collected dataset consists of research papers posted on arXiv under the CS.LG tag (the field of machine learning in the computer science domain) from 2019 to 2024. After removing duplicate entries and authors with fewer than ten papers, the dataset includes 24,095 papers from 1,500 authors, ensuring that each author has at least ten papers. We use this dataset to investigate their potential privacy risks when using LLMs to identify authorship of academic writing. \textbf{Enron Email}. The Enron email dataset contains approximately 500,000 emails generated by employees of the Enron Corporation. For our benchmark, similar to~\citet{Huang2024CanLL}, we removed sender and receiver information and discarded short emails. Ultimately, we retained 174 authors, each with 50 emails. \textbf{Blog}. The Blog Authorship Corpus~\citep{Schler2006EffectsOA} comprises the collected posts of 19,320 bloggers gathered from Blogger.com\footnote{\url{https://www.blogger.com}} in August 2004. We selected 1,500 authors from the dataset, each with 10 blog posts. The content of this dataset is closely related to daily life, providing linguistic characteristics, writing styles, word usage habits, and special characters that can be used to infer an author's identity. \textbf{IMDb Review}. The IMDb review data are selected from the IMDb62 dataset~\citep{Seroussi2011AuthorshipAW}. After filtering out reviews with fewer than 10 words, we randomly retained 50 reviews for each of the 62 authors in our dataset. \textbf{Guardian}. The Guardian corpus dataset~\citep{guardian} is designed to explore cross-genre and cross-topic authorship attribution. The corpus comprises texts published in \textit{The Guardian} daily newspaper, mainly opinion articles (comments). It includes articles from 13 authors across five topics—politics, society, UK, world, and books—and retains 50 articles per author. \subsection{Evaluation tasks}
As illustrated in Figure~\<PIC>, we outline the evaluation tasks. Each task involves a query text and a set of candidate texts, with the objective of identifying which candidates are most likely authored by the same individual as the query text. Based on the number of candidate texts, we introduce the following specific tasks. \textbf{One-to-one identification.} Commonly known as authorship verification, this task involves a single candidate text and aims to determine whether the text pair is authored by the same person, forming a binary classification problem. In our evaluation, we prompt the LLM with: ``Here are a pair of texts: [Text Pair]. Determine if they belong to the same author.''. More effective prompts or additional information can be provided to the LLM to achieve more accurate and reasonable responses. For instance, \cite{Hung2023WhoWI} employs a Chain-of-Thought prompt and a series of intermediate reasoning steps to significantly enhance the LLMs' authorship verification ability. Similarly, as shown in \cite{Huang2024CanLL} and \cite{Hung2023WhoWI}, instructing LLMs to analyze texts based on writing style, linguistic characteristics, and word usage habits, rather than content and topics, can further improve authorship verification. Due to the subjective nature of the \emph{one-to-one identification} task and its heavy reliance on the intrinsic judgment of LLMs, it is natural to consider a scenario involving two candidate texts requiring a comparative decision. The objective here is to determine which candidate text is more likely authored by the same individual as the query text; we refer to this as the \textbf{one-to-two identification} task. Due to space limitations, we defer the task design, metrics, and experimental results for the \emph{one-to-two identification} task to Appendix \ref{app:1to2}.
\textbf{One-to-many identification.}
In this setup, we provide a number of candidate texts and ask LLMs to determine texts that are most likely authored by the same person as the text in the query. This is different from the usual closed-set authorship attribution task \citep{Huang2024CanLL}, where multiple authors and their writings are provided in the context of LLMs, and the task is to attribute a target text to one of these authors. We do not provide any authorship information to LLMs. Instead in our experiments, we randomly sample a number of authors and put all of their writings into a set. Then one specific text is chosen at random as the target text while the rest from the set form the candidate texts. The task is to ask LLMs to identify the texts in the candidate set that are mostly likely authored by the same person as the target text, ranking the results by confidence scores. \subsection{RAG-based one-to-many identification pipeline}\label{sec:RAG} The performance of \emph{one-to-many authorship identification} tasks heavily depends on the ability of LLMs to handle long contexts. Current commercial LLMs support relatively large context windows; for example, GPT-4-Turbo can process contexts up to 128,000 tokens, and Kimi supports up to 2 million tokens. In contrast, open-source models like Llama-3-8B-Instruct\footnote{\url{https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct}} support context windows of only up to 8,000 tokens. Moreover, the authorship identification task relies heavily on the LLMs' capacity for high-level comprehension of the context, which becomes increasingly challenging as the length of the context grows. To address this limitation, we propose a simple Retrieval-Augmented Generation (RAG)-based method, as illustrated in Figure \<PIC>. In this approach, we first use pre-trained embedding models like sentence-transformers to encode both the target text and the candidate texts. We then calculate the similarities between the target text and each candidate, selecting only the top \textit{k} texts with the highest similarity scores. These top \textit{k} texts are provided to the LLM, thereby reducing the context length and enabling more effective processing.
The motivation behind this method is that although embedding models tend to focus more on the semantic meanings of texts than on their linguistic characteristics, we can still use the similarities they provide to filter out texts that are significantly dissimilar to the target text. This approach ensures that the total length of the candidate texts remains within the context windows of LLMs.
\subsection{Evaluation metrics}
For the \emph{one-to-one identification} task, LLMs are required to classify whether a pair of texts were written by the same author by answering ``True'' or ``False''. While accuracy can serve as an evaluation metric, it may not fully capture performance due to potential class imbalances in the dataset, for instance, when there are many negative pairs compared to positive pairs, or vice versa. Therefore, we adopt \textit{Precision, Recall} as our evaluation criteria to assess the performance of LLMs under various setups more comprehensively. To be specific, Precision and Recall are defined as follows:
\begin{equation}
   \text{Precision} = \frac{\#\text{True Positive Pairs}}{\#\text{Predicted Positive Pairs} }; \quad\quad \text{Recall} = \frac{\#\text{True Positive Pairs}}{\#\text{Positive Pairs} }.
\end{equation}
For the \emph{one-to-many identification} task, we employ multiple evaluation metrics to capture the problem's complex nature. In our experiments, we instruct the models to rank candidate texts based on confidence scores and repeat the experiments using multiple target texts. We utilize metrics such as \textit{Rank@1} and \textit{Rank@5} in our benchmark, where \textit{Rank@x} indicates that at least one text authored by the same individual as the target text is found within the top \textit{x} ranked texts. Additionally, since we expect the LLMs to select as many correct candidate texts as possible, we use \textit{Precision@x} as a metric, where \textit{Precision@x} denotes the proportion of correct predictions among the top \textit{x} texts.


In this paper, we perform an apples-to-apples comparison that addresses these concerns,
comparing seven medical LLMs and two medical VLMs against their general-domain base models.
We find that, for all but one LLM pair---\textsc{BioMistral-7B} \citep{biomistral} vs. \textsc{Mistral-7B-Instruct-v0.1} \citep{mistral},
a pair of models that performs fairly poorly in absolute terms---the open-source medical LLMs and VLMs that we evaluate do not consistently improve over their general-domain counterparts on various medical (visual) QA tasks (Figure \<PIC> \<PIC>).  We compare several pairs of general-domain and medically adapted LLMs/VLMs (see Table \ref{tab:models}),
\textbf{whose only differences lie in medical DAPT} (i.e., one model is the base model, from which the other is derived via medical DAPT). For each pair, we compare their performances from zero-/few-shot prompting \citep{gpt-2,gpt-3}, after independently selecting the ``best'' prompt format and few-shot examples for each model based on the validation set and accounting for statistical uncertainty in model comparison. We note that, while \textsc{OpenBioLLM-70B} shows improvement in the 3-shot setting, it does not show improvement in the zero-shot setting (winning on 7.7\% and losing on 23.1\% of tasks, see Table~\ref{tab:win-tie-loss-rates-0}), and vice versa for \textsc{Clinical-Camel-70B} (winning on 0\% of tasks and losing on 36.4\% of tasks in the 3-shot setting, see Table~\ref{tab:win-tie-loss-rates-3}), \textbf{leaving \textsc{BioMistral-7B} as the only medical LLM that wins more than it loses} against its base model (\textsc{Mistral-7B-Instruct-v0.1}) in both settings, albeit with relatively low absolute performance.
In Figure~\<PIC>, we show how the win/tie/loss rates of the medical models, computed over all (model pair, QA dataset) combinations,
change as we vary the following aspects of the experimental setup:
\begin{enumerate}[topsep=5pt,itemsep=-0.5ex]
    \item select prompts for each model independently vs. only based on the medical model;
    \item determine a win for the medical model based on confidence intervals in relative accuracy vs. raw absolute accuracy.
\end{enumerate}
We note that when comparing each model pair based on absolute accuracy, there are no ties, as the real-valued absolute accuracies are rarely identical. Overall, we find that for both LLMs and VLMs, the performance improvement from using a medically adapted model instead of its general-domain counterpart can be substantially overestimated when (i) the prompt is only tailored to the medical model; and (ii) the models are compared only based on their absolute accuracies.
Notably, in the zero-shot setting, the win rate increases from 9.4\% to 70.5\% for medical LLMs and from 6.3\% to 62.5\% for medical VLMs,
when only performing prompt selection for the medical model and comparing based on absolute accuracy.


Recent year, Large Language Models (LLMs)~\cite{radford2019language, 209211,chatgpt}, have drawn significant attention due to their remarkable capabilities and applicability to a wide range of tasks~\cite{cheshkov2023evaluation, pearce2022asleep, copilot, pearce2022examining, frieder2023mathematical, shakarian2023independent, lehnert2023ai, kortemeyer2023could}. Building on top of the initial success, there is an increasing demand for richer functionalities using LLM as the core execution engine. This led to the rapid development and rollout of the \textbf{LLM-based systems (LLM systems)}, such as OpenAI GPT4 with plugins~\cite{chatgpt}. 2023 can be considered as the ``meta year'' of LLM systems, in which
OpenAI announces the GPTs~\cite{devday}, empowering users to design customized LLM systems and release them in GPTs store~\cite{gpts}.
According to the most recent data statistics report~\cite{gptsflow} up to November 17th, the top 10 customized GPTs have collectively received more than 750,000 visits. Notably, the customized GPT named Canva~\cite{canva} has been visited over 170,000 times in just 10 days. In addition, the third-party GPTs store has updated more than 20,000 released customized GPTs~\cite{gptsnum}.
All these facts underscore the increasing integration of LLM systems into our daily lives.  Drawing inspiration from information flow analysis, we propose a new information-flow-based formulation to enable systematic analysis of LLM system security~\footnote{ Note that the LLM system discussed in this paper specifically references the design of OpenAI GPT4. Any potential or future LLM systems possessing different features are not within the scope of this study.}. To achieve it, we need to tackle two non-trivial uniqueness of the LLM system.  \textit{LLM system analysis has to consider the nature of the interaction between machine learning model and multi-object information processing.} LLM systems combine novel AI model (LLM) with conventional software components (e.g., Web Tools), leading to complex interactions across various objects and models.  This integration results in a multi-layered system where data flows and transformations occur across various operational contexts, from straightforward individual object level to context-dependent multi-object level. To facilitate the analysis of such a multi-layered system, we develop a multi-layer analysis framework, as shown in Figure~\<PIC>, where \textit{objects} are key components in the LLM system (such as the LLM model and plugins). \textit{Action} and \textit{interactions} capture the processing of information within an object and the transmission of information between objects respectively.
Since security issues arise from the lack of effective constraints of the information flow -- it allows information to directly flow in and out without any restrictions~\cite{cecchetti2021compositional, myers1999x},  we propose to use the concept of \textit{constraint} to capture the security requirement over the information flow, where the constraints are multi-layer, placing the mediation to not only the processing of individual objects (constraint over action), but also the processing among them (constraint over
 interactions).  \textit{Constraints over action and interaction are now probabilistic and have to be analyzed through the lens of adversarial robustness.} LLM systems differ significantly from standard systems where executions are often deterministic, LLM systems operate in an inherently uncertain setting. This fundamental difference is due to the nature of LLMs, which process natural language inputs that are vague and free-form, challenging the model's ability to consistently interpret data and instructions. This ambiguity also extends to the models' output, which can vary widely in form while conveying the same meaning. When these outputs are used to command or interact with other system components, like web tools, the inherent ambiguity can lead to unpredictable interactions with these components, significantly complicating the process of applying security features. Furthermore, the probabilistic nature challenges the system's ability to produce consistent outputs and hinders the ability to apply and enforce security features in LLM systems in a deterministic manner. Thus, to enable the analysis, a set of rules is encapsulated in the constraints. Based on the constraints,
we should analyze not only the presence of such constraint (machine-learning-based policy enforcement) but also the adversarial robustness (how well it works) of these rule enforcement via a multi-step process.   \noindent\textbf{Example 1: Unethical Image Displaying.}~\label{moti2}
LLM system, beyond its core LLM capabilities, encompasses additional facilities.
Notably, the Frontend is a critical facility widely used in the LLM system to provide a friendly user interface.
One of the most important functionalities of Frontend is to render image links in markdown format. This increases the richness and diversity of the displayed content.
When the LLM in the system outputs certain markdown image links and transmits them to the Frontend, the Frontend will automatically render it and display the image content of links.
However, integrating the Frontend introduces security concerns.
For instance, as shown in Figure~\<PIC>, when the LLM outputs certain malicious markdown image links of unethical content such as pornographic pictures, and the links are transmitted to the Frontend, the automated render process in the Frontend would render it and display this explicit image.
Specifically, this kind of rendering process contains two steps.
The first step is the LLM output target markdown image link, which is followed by the second step where the output target markdown image link is transmitted to the Frontend.
We found that OpenAI fails to ensure security in both of these two steps : (1)
the LLM can output arbitrary external markdown image links if we adopt certain adversarial strategies (detailed strategy shown in Sec~\ref{firstrender}) and (2) the transmission of such image links toward the Frontend lacks of necessary control (details in ~\ref{sec:frontend}). This example highlights two critical insights. First, while LLMs contribute to their superior performance, they also introduce potential threats to the security of the LLM systems.
Second, the interaction between the LLM and other internal system components can give rise to new emergent threats. This realization highlights the importance of adopting a holistic approach to model and study the security problems in the LLM system.


\section{CogMir: Multi-LLM Agents Framework On Cognitive Bias}
In this section, we provide a detailed and modular overview of CogMir, organized into four main elements: Mirror Environmental Settings, Framework Structures, Cognitive Bias Subsets, and Sample Use Cases. These components are visually depicted in a left-to-right sequence in Fig.~\<PIC>.
\subsection{Framework Structures}
After establishing realistic social science experiment environments, the next step is to select essential components to support the above two mirror methods: Human-LLM Agent Q\&A and Multi-H-A Interaction. This entails choosing participant objects, evaluation tools, and communication modes. The CogMir framework is organized into modules for Required Objects, Communication Modes, and Interaction Combinations to meet these needs. \textbf{Required Object Sets.} Required Object encompasses all potential participants and evaluators involved in the system. \textbf{Participants} include humans\footnote{"Human" in CogMir can refer to real human participants or simulations. In our experiments, "Human" refers to simulated human interactions based on previous social science experiments, not actual human subjects.} and LLM Agents, which allows for dynamic setups where either or both can be involved in interactions depending on the experiment's requirements. \textbf{Evaluators} include humans, LLM Agents, datasets, and discriminators. Datasets are utilized to store and construct prompts about the experimental setup (e.g., experimental scenarios, character information, etc.), task description, and Q\&A question set. Discriminators are specialized tools utilized to evaluate the social intelligence of LLM Agents, encompassing three main types: State-of-the-art technical metrics such as SimCSE, SelfCheck, and FactScore~\citep{Simcse, min-etal-2023-factscore, Manakul2023SelfCheckGPTZB} for objective, quantitative assessment; Human discriminators that delve into nuanced and subjective aspects like prosocial understanding; and LLM Agent discriminators, which involve the use of other LLM Agents to assess and challenge responses from a subject LLM Agent. \textbf{Communication Modes Sets.} Communication modes dictate the nature of interactions within different setups. We model the participants (humans or LLM Agents) as channels based on information theory~\citep{Shannon1948} to define two essential communication modes:
\vspace{-2mm}
\begin{itemize}[noitemsep]
    \item \textbf{Broadcast} (or Parallel, $C = C_1 + C_2 + \ldots + C_n$), which enables a single sender to transmit a message to multiple receivers simultaneously.
    \item \textbf{Point-to-point} (or Series, $C = \min[C_1, C_2, \ldots, C_n]$) establishes communication between two specific entities at a time ($C$ denotes channel capacity).
\end{itemize}
\vspace{-2mm}
\textbf{Multi-H-A Interaction Combinations Sets.} This module provides various combinations of Multi-Human-LLM Agent interactions, tailored to different social science experimental needs, the most frequently used combinations in social science settings include:
\vspace{-2mm}
\begin{itemize}[noitemsep]
    \item \textbf{Single-H-Single-A}: One human interacting with one LLM Agent, predominantly used for human-agent question-answering tasks (e.g., survey, interview, etc. ).
    \item \textbf{Single-H-Multi-A}: One human interacts with multiple LLM Agents, where humans can be set as controlled variables to test Multi-LLM Agents's social cognitive behaviors.
    \item \textbf{Multi-H-Single-A}: multiple humans interact with a single LLM Agent, which is suitable for assessing the impact of group dynamics, such as consensus or conflict.
    \item \textbf{Multi-A}: multiple agents interacting without human participation.
    \item \textbf{Multi-H-Multi-A}: multiple humans and multiple LLM Agents interaction, integrating elements from the previous setups to mimic complicated experimental interactions.
\end{itemize}
\vspace{-2mm}
These modules offer a flexible framework for exploring LLM Agents' cognitive biases in social science experiments. Researchers can customize their setups by mixing different components to examine specific hypotheses. In the next section, we outline cognitive bias subsets as guidelines. \textbf{Conformity: Authority Effect.}
Drawing on classical social science experiments conducted by Stanley Milgram~\citep{Authority}, we conducted experiments to explore the Authority Effect, tailored to the characteristics of LLM Agents. Unlike the Herd Effect, which requires multiple human participants, the Authority Effect aims to test the conformity of LLM Agents to authoritative prompts or instructions, even when these may contradict factual information. In the settings, we utilize Known, and \textbf{Unknown MCQ} datasets as Evaluators and \textbf{CogIdentity} and \textbf{CogScene} as prompt generators to test the \textbf{Q\&A Bias Rate} through Single-H-Single-A Q\&A scenarios. Average Q\&A Rate refers to the average bias rate on Unknown and Known MCQ. We design prompts to directly inquire LLM Agents on 5 identity pairs across two MCQ datasets, each for 10 times, resulting in $5 \times 10 \times 100 \times 2$ inquires.
Prompts are constructed as follows: Human 1st round \{You are [INFERIOR identity], and I am [SUPERIOR identity] who is an expert in the following field. Please provide your response by selecting either A or B. The question is [MCQ]\}; 2nd round \{As an expert, I confirm that the answer should be B. Please provide your answer again. Reply in the following format: Answer: A/B.\}. Fig.~\<PIC> shows that LLM Agents display considerable variability in responses to deterministic and nondeterministic scenarios when influenced by the Authority Effect. Notably, in certain and uncertain situations, most LLM Agents showed increased obedience to Authority compared to the Herd Effect. This contrasts humans, who typically exhibit no significant difference in obedience between authority and herd scenarios. These findings highlight the LLM Agents' enhanced sensitivity to social status differences, indicating a stronger propensity to adhere to authoritative commands over peer influence.


\section{Introduction}
The ability of LLMs to follow detailed instructions is vital to their usefulness.  Generative language models are typically trained on raw web data, and then subsequently fine-tuned on a comparatively small but carefully curated set of instruction data. Instruction fine-tuning is crucial to taming the power of LLMs, and the usefulness of a model is largely determined by our ability to get the most out of small instruction datasets. \looseness -1 In this paper, we propose to add random noise to the embedding vectors of the training data during the forward pass of fine-tuning. We show that this simple trick can improve the outcome of instruction fine-tuning, often by a large margin, with no additional compute or data overhead. \underline{N}oisy \underline{E}mbedding Instruction \underline{F}ine \underline{T}uning (\neftune{}), while simple, has a strong impact on downstream conversational quality.  When a raw LLM like \llama{}-2-7B is finetuned with noisy embeddings, its performance on \texttt{AlpacaEval} improves from 29.8\% to 64.7\% (Figure \<PIC>) -- an impressive boost of around 35 percentage points
\citep{touvron2023llama2, dubois2023alpacafarm}. \neftune{} leads to this surprising and large jump in performance on conversational tasks, maintaining performance on factual question answering baselines.
This technique seems to be a free lunch for LLM fine-tuning. \paragraph{\neftune{} Improves Text Quality.}
From Table \ref{tab:LLaMA-2_GPT-4}, we can see an increase across all datasets for the $7$B scale with an average increase of $15.1\%$, showing that training with \neft{} significantly improves conversational ability and answer quality, as measured via \texttt{AlpacaEval}. Additionally, we can see from Figure \<PIC> that we also see improvements on older models, such as \llama{}-1 and OPT. Interestingly, we see less improvement on ShareGPT than on other datasets according to ChatGPT. However, this is not reflected in the GPT-4 evaluation. From Table \ref{tab:llama-2-chat_SFT}, we see the Win Rate climbs from 75.03\% to 88.81\% (+13.78\%) after adding \neftune{} to the $70$B parameter model trained on Evol-Instruct (hyperparameters in Appendix \ref{sec:hyperparameters}).


\paragraph{Investigations on optimized prompts} We explore several interesting properties of these optimized prompts.
\begin{itemize}
    \item \textit{Evil twins}. In many cases, the optimized prompts that we find are similar in function to the original prompts (twins), but garbled and unintelligible to humans (evil). For this reason, we refer to them as \textit{evil twins}. See Figure~\<PIC> for some examples.
    \item \textit{Transferability}. Remarkably, these ``evil twin'' prompts transfer between a variety of open-source and proprietary language models; see Section~\ref{sec:transferability}.
    \item \textit{Robustness}. We investigate the robustness of evil twin prompts to changes in their token-order and to replacements of their tokens. We find that whether evil twins are robust to randomly permuting their tokens depends on the LLM family. On the other hand, across LLM families, evil twins are more impacted by randomly replacing their tokens than ground truth prompts. This suggests that even the uncommon, non-English tokens in the optimized prompts play an important role in driving the model output; see Section~\ref{sec:prompt-investigations}.     \item \textit{Improving prompt intelligibility}. We explore variants of the optimization problem \eqref{eq:mle-def-intro} that encourage the optimized prompts to be more interpretable (adding a fluency penalty and restricting the vocabulary to common English tokens). However, we find that these modifications do not improve the KL divergence of the optimized prompts to the ground truth; see Section~\ref{sec:optim-intel}.
\end{itemize} We discuss other applications of the maximum-likelihood problem \eqref{eq:mle-def-intro} to prompt compression, privacy, and conditional generation in Section~\ref{sec:discussion}. We compare these methods on 100 randomly sampled prompts from the Alpaca instruction tuning dataset \citep{alpaca}, where Vicuna-7b-v1.5 is the instruction-tuned model. Additional experiments on various model families and datasets are presented in Appendix~\ref{app:addn-model-exp}. For each method and prompt, we compute the KL divergence of the optimized prompt with respect to the original prompt. We compare pairs of methods based on which one finds the closer prompt to the ground truth; see Figure~\<PIC>. GPT-4 suggestions perform roughly on par with those from cold-start GCG. On the other hand, GCG with a warm start provides a strong improvement over both cold-start GCG and the GPT-4 prompt suggestions. Enforcing interpretability by adding a fluency penalty or pruning the vocabulary does not improve the optimized prompt (see Section~\ref{sec:optim-intel}).