We present DeSiRe-GS, a self-supervised gaussian splat-ting representation, enabling effective static-dynamic decom-position and high-fidelity surface reconstruction in complexdriving scenarios. Our approach employs a two-stage opti-mization pipeline ofdynamic street Gaussians. In the firststage, we extract 2D motion masks based on the observationthat 3D Gaussian Splatting inherently can reconstruct onlythe static regions in dynamic environments. These extracted2D motion priors are then mapped into the Gaussian space ina differentiable manner, leveraging an efficient formulationthe introduced geometric regularizations, our method areable to address the over-fitting issues caused by data sparsityin autonomous driving, reconstructing physically plausibleGaussians that align with object surfaces rather than float-ing in air. Furthermore, we introduce temporal cross-viewconsistency to ensure coherence across time and viewpoints,resulting in high-quality surface reconstruction. Comprehen-sive experiments demonstrate the efficiency and effectivenessof DeSiRe-GS, surpassing prior self-supervised arts andachieving accuracy comparable to methods relying on ex-ternal 3D bounding box annotations.
1. IntroductionModeling driving scenes [11, 28] is essential for au-tonomous driving applications, as it facilitates real-worldsimulation and supports scene understanding [46]. An ef-fective scene representation enables a system to efficientlyperceive and reconstruct dynamic driving environments. Re-cent 3D Gaussian Splatting (3DGS) [16] has emerged as aprominent 3D representation that can be optimized through2D supervision. It has gained popularity due to its explicitnature, high efficiency, and rendering speed.While 3D Gaussian Splatting (3DGS) has demonstratedstrong performance in static object-centric reconstructions,the original 3DGS struggles to handle dynamic objects in un-bounded street views, which are common in real-world sce-narios, particularly for autonomous driving applications. It isunable to effectively model dynamic regions, leading to blur-ring artifacts due to the Gaussian model’s time-independentparameterization. As a result, 4D-GS [35] is proposed, mod-eling the dynamics with a Hexplane encoder. The Hexplane[1] works well on object-level datasets, but struggles withdriving scenes because of the unbounded areas in outdoorenvironments. Instead, we choose to reformulate the originalstatic Gaussian model as time-dependent variables with mi-nor changes, ensuring the efficiency of handling large-scaledriving scenes.In this paper, we present DeSiRe-GS, a purely Gaus-sian Splatting-based representation, which facilitates self-supervised static-dynamic decomposition and high-qualitysurface reconstruction in driving scenarios. For static-dynamic decomposition, existing methods such as Driving-Gaussian [47] and Street Gaussians [38], rely on explicit3D bounding boxes, which significantly simplifies the de-composition problem, since dynamic Gaussians in a movingbounding box can be simply removed. Without the 3D anno-tations, some recent self-supervised methods like PVG [5]and S3Gaussian [15] have attempted to achieve decomposi-tion but fall short in performance, as they treat all Gaussiansas dynamic, relying on indirect supervision to learn motionpatterns. However, our proposed method can achieve ef-fective self-supervised decomposition, based on a simpleobservation that dynamic regions reconstructed from 3DGSare blurry—quite different from the ground truth images. De-spite the absence of 3D annotations, DeSiRe-GS producesresults comparable to, or better than, approaches that useexplicit bounding boxes for decomposition.Another challenge in applying 3D Gaussian Splatting(3DGS) to autonomous driving is the sparse nature of im-ages, which is more pronounced compared to object-centricreconstruction tasks. This sparsity often leads 3DGS to over-fit on the limited number of observations, resulting in inaccu-rate geometry learning. Inspired by 2D Gaussian Splatting(2DGS) [14], we aim to generate flatter, disk-shaped Gaus-sians to better align with the surfaces of objects like roadsand walls. We also couple the normal and scale of eachGaussian, which can be optimized jointly to improve surfacereconstruction quality.To further address the overfitting issue, we propose tempo-ral geometrical cross-view consistency, which significantlyenhances the model’s geometric awareness and accuracy byaggregating information from different views across time.These strategies allow us to achieve state-of-the-art recon-struction quality, surpassing other Gaussian splatting ap-proaches in the field of autonomous driving.Overall, DeSiRe-GS makes the following contributions:• We propose to extract motion information easily fromappearance differences based on a simple observation that3DGS cannot successfully model the dynamic regions.• We then distill the extracted 2D motion priors in localframes into global gaussian space, using time-varyingGaussians in a differentiable manner.• We introduce effective 3D regularizations and temporalcross-view consistency to generate physically reasonableGaussian ellipsoids, further enhancing high-quality de-composition and reconstruction.We demonstrate DeSiRe-GS’s capability of effectivestatic-dynamic decomposition and high-fidelity surface re-construction across various challenging datasets [11, 28].Yet, the separationremains incomplete and lacks thoroughness.Neural Surface Reconstruction. Traditional methodsfor Neural Surface Reconstruction focus more on real geom-etry structures. With the rise of neural radiance field (NeRF)technologies, neural implicit representations have shownpromise for high-fidelity surface reconstruction. Approacheslike [19, 33, 41, 43] train neural signed distance functions(SDF) to represent scenes. StreetSurf [13] proposes disen-tangling close and distant views for better implicit surfacereconstruction in urban settings, while [27] steps furtherusing sparse lidar to enhance depth details.3D GS has renewed interest in explicit geometric recon-struction, with recent works [2, 3, 9, 12, 14, 32, 42] focusingon geometric regularization techniques. SuGaR [12] alignsGaussian ellipsoids to object surfaces through introducingand additional regularization term, while 2DGS [14] directlyreplaces 3D ellipsoids with 2D discs and utilizes the trun-cated signed distance function (TSDF) to fuse depth maps,enabling noise-free surface reconstruction. PGSR [2] intro-duces single- and multi-view regularization for multi-viewconsistency. GSDF [42] and NeuSG [3] combine 3D Gaus-sians with neural implicit SDFs to enhance surface details.TrimGS [9] refines surface structures by trimming inaccu-rate geometry, maintaining compatibility with earlier meth-ods like 3DGS and 2DGS. While these approaches excel insmall-scale reconstruction, newer works like [4, 7, 10] aimto address large-scale urban scenes. [4] adopts a large-scenepartitioning strategy for reconstruction, while RoGS [10]proposes 2D Gaussian surfel representation which alignswith physical characteristics of road surfaces.
4. DeSiRe-GSAs shown in <PIC>, the training process is divided intotwo stages. We first extract 2D motion masks by calculat-ing the feature difference between the rendered image andGT image. In the second stage, we distill the 2D motioninformation into Gaussian space using PVG [5], enablingthe rectification of inaccurate attributes for each Gaussian ina differentiable manner.
4.1. Dynamic Mask Extraction (stage I)During the first stage, we observe that 3D Gaussian Splat-ting (3DGS) performs effectively in reconstructing staticelements, such as parked cars and buildings in a drivingscene. However, it struggles to accurately reconstruct dy-namic regions, as the original 3DGS does not incorporatetemporal information. This limitation results in artifactssuch as ghost-like floating points in the rendered images,as illustrated in <PIC> (stage 1). To address this issue, weleverage the significant differences between static and dy-namic regions to develop an efficient method for extractingsegmentation masks that encode motion information.Initially, a pretrained foundation model is employed toextract features from both the rendered image and the groundtruth (GT) image used for supervision. LetFˆdenote thefeatures extracted from the rendered image ˆI, and F repre-sent the features extracted from the GT image I. To distin-guish dynamic and static regions, we compute the per-pixeldissimilarity D between the corresponding features. Thedissimilarity metric D approaches 0 for similar features, in-dicating static regions, and nears 1 for dissimilar features,corresponding to dynamic regions.We observed that both3DGS and PVG could produce oversized Gaussian ellipsoidswithout additional regularization, particularly in unboundeddriving scenarios, as illustrated in <PIC> (a).Our primary objective is to fit appropriately scaled Gaus-sians that support accurate image rendering and surface re-construction. While oversized Gaussian ellipsoids with lowopacity may have minimal impact on the rendered image,they can significantly impair surface reconstruction. This isa limitation often overlooked in existing methods focusedsolely on 2D image rendering. To address this issue, weintroduce a penalty term for each Gaussian:We observed that both3DGS and PVG could produce oversized Gaussian ellipsoidswithout additional regularization, particularly in unboundeddriving scenarios, as illustrated in <PIC> (a).Our primary objective is to fit appropriately scaled Gaus-sians that support accurate image rendering and surface re-construction. While oversized Gaussian ellipsoids with lowopacity may have minimal impact on the rendered image,they can significantly impair surface reconstruction. This isa limitation often overlooked in existing methods focusedsolely on 2D image rendering. To address this issue, weintroduce a penalty term for each Gaussian:We compare the rendered depth of various methods in<PIC>. S3Gaussian [15] fails to predict accurate depth map,because they use only LiDAR point clouds for initialization,where there are no points at the upper part. Other than theLiDAR points, PVG [5] and DeSiRe-GS randomly samplepoints, enabling us to render much better depth map.


We find these directions using Sparse Autoencoders (SAEs) (Bricken et al., 2023; Cunningham et al.,
2023). SAEs are an interpretability tool for finding a sparse, interpretable decomposition of model
representations.
Dictionary learning (Olshausen & Field, 1997) offers a powerful approach for disentangling features
in superposition. Sparse Autoencoders (SAEs) have proven to be effective for this task (Sharkey
et al., 2022; Bricken et al., 2023).
We then categorize entities into 'known' or 'unknown'. Known entities are those where the model
gets at least two attributes correct, while unknown are where it gets them all wrong, we discard
any in-between. To measure correctness we use fuzzy string matching2.
We find that the separation scores of some of the SAE latents in the training set are high, i.e. they
fire almost exclusively on tokens of either known or unknown entities, as depicted in the scatter plot
in <PIC> for Gemma 2 2B and <PIC>, Appendix C for Gemma 2 9B. An interesting observa-
tion is that latent separation scores reveal a consistent pattern across all entity types, with scores
increasing throughout the model and reaching a peak around layer 9 before plateauing (<PIC>).
This indicates that latents better distinguishing between known and unknown entities are found in
the middle layers.
We also examine the level of generality of the latents by measuring their minimum separation score
across entity types (t): players, song, cities and movies. A high minimum separation score indicates
that a latent performs robustly across entity types, suggesting strong generalization capabilities.
Appendix D for Gemma 2 9B suggests that more generalized latents—those
that distinguish between known and unknown entities across various entity types—are concentrated
in these intermediate layers. This finding points to a hierarchical organization of entity represen-
tation within the model, with more specialized, worse quality, latents in earlier layers and more
generalized, higher quality entity-type-agnostic features emerging in the middle layers.
We hypothesize that entity recognition directions could be used by chat models to induce knowl-
edge refusal. To evaluate this, we use a test set sample of 100 questions about unknown entities,
and measure the number of times the model refuses by steering (as in Equation (4)) with the entity
recognition latents the last token of the entity and the following end-of-instruction-tokens.4 <PIC> illustrates the original model refusal rate (blue bar), showing some refusal across entity
types. We see that the entity recognition SAE latents found in the base model transfer to the chat
model and, by increasing the known entity latent activation we are able to reduce refusal in three
entity types. Conversely, increasing the unknown entity latent induces almost 100% refusal across
all entity types. We also include an Orthogonalized model baseline, which consists of doing weight
orthogonalization (Arditi et al., 2024) on every matrix writing to the residual stream. Weight orthog-
onalization modifies each row of a weight matrix to make it perpendicular to a specified direction
vector d.
By doing this operation on every output matrix in the model we ensure no component is able to
write into that direction. The resulting orthogonalized model exhibits a large reduction in refusal
responses, suggesting this direction plays a crucial role in the model's knowledge refusal behav-
ior. We also include the average refusal rate after steering with 10 differents random latents, using
the same configuration (layer and steering coefficient) that the known and unknown entity latents
respectively. Additional analysis of the Gemma 2 9B model, detailed in Section F, reveals similar
patterns, albeit with less pronounced effects compared to the 2B model.
<PIC> shows a refusal response for a well-known basketball player generated by steering
with the unknown entity latent. In <PIC> (right) we observe that when asked about a non-existent
player, Wilson Brown, the model without intervention refuses to answer. However, steering with the
known entity latent induces a hallucination.


Designing the Conversation Seed Prompt: The design of the conversation seed prompt is grounded in social identity perspective (Tajfel and Turner,2004), which posits that individuals form identi-ties through their association with various social groups, encompassing multiple simultaneous iden-tities, such as nationality, gender, and interests (Maet al., 2023). According to Abrams and Hogg(2010), contextual cues can render a social iden-tity more salient when compared to other social identities. Thus, to make the race/caste identities salient when generating conversations, our con-versation seed prompt includes the background context regarding the colleagues' group identi-ties (e.g. “White”, “Brahmin”) and the initial dia-logue, in which the applicant’s group identity (e.g.“Black”, “Dalit”) is discussed (e.g.“he appears tobe [group]”) (see <PIC>).
Colleagues & Applicant name selection: To intro-duce diversity in the name selection, we randomly selected names that are culturally indicative of dif-ferent races and castes. All models were provided with identical prompts, including the same names and groups, to ensure consistency across the exper-iments. We discuss additional details in §H.
Hiring occupation selection: We consider four occupations in our experiments: Software Devel-oper, Doctor, Nurse, and Teacher. These roles are chosen due to their varied societal perceptions and stereotypical associations along both race and caste dimensions, as highlighted in prior work on race(Ghosh and Caliskan, 2023; Veldanda et al., 2023) and caste (Pathania et al., 2023; Barua and Verma,2021; Kumbhar, 2021; George, 2015, 2019).
LLM model selection: For a comprehensive anal-ysis, we selected eight LLMs – two models from OpenAI and six widely used open-source models,as listed in <PIC>. We set the temperature to 0.7 for all models with a 512-token limit.
We introduce the Covert Harms and Social Threats(CHAST) metrics, a set of 7 metrics grounded in social science literature, such as the Social Iden-tity Threat Theory (Branscombe et al., 1999a; Maet al., 2023) and Intergroup Threat Theory (Stephan and Cookie, 2000). These frameworks offer a nu-anced understanding of the various forms of harm and threat to identity groups. The CHAST metrics are particularly relevant to our experimental setup,
wherein LLMs generate conversations involving two colleagues discussing an applicant from a dif-ferent identity group (e.g. “Black” or “Dalit”).
According to the Social Identity Threat Theory(SIT), social identity threat is evoked when peo-ple feel concerned about being negatively treated,devalued, or stereotyped based on group member-ship, often arising from intergroup communica-tion (Branscombe et al., 1999a; Ellemers et al.,2002). We selected three types of social iden-
tity threat: categorization threat , which is felt when people experience being reduced to a single category; morality threat and competence threat which arise when the group's morality or compe-tence are undermined.
We also included metrics from Intergroup Threat Theory (ITT), which argues that intergroup threat is experienced when an in-group member perceives that another group is in a position to cause them harm (Stephan and Cookie, 2000). ITT offered two metrics: Realistic threat , which occurs when an in-group member is concerned about their physical safety, political and economic prosperity, or re-sources due to an out-group and symbolic threat ,
which arises when an in-group member is con-cerned about the integrity of the in-group’s values, standards, morals, or attitudes due to an out-group member (Stephan and Cookie, 2000).
Lastly, we incorporated disparagement as part of our metrics, which encapsulates any behavior by a model that reinforces a notion that certain groups are less valuable than others (Dev et al., 2022). Fol-lowing prior research on harms in job hiring scenar-ios, we additionally included opportunity harm(Yam and Skorburg, 2021; Roberts, 2015), which arises due to withdrawal or reduced chance of an opportunity based on background or group iden-
tity. <PIC> presents the CHAST metrics, including their definitions and examples extracted from LLM-generated conversations in our study.


However, dividing long videos into short clips would inevitably introduce an accompanied chal-lenge, i.e., multi-granularity noisy correspondence (MNC). As shown in <PIC>, MNC refers to the misaligned video-text pairs at two different granularities: i) Coarse-grained misalignment (Clip-caption). Coarse-grained misalignment includes asynchronous and irrelevant misalignments ac-cording to whether a clip/caption is alignable with the captions/clips in the long video. To be specific,asynchronous misalignment refers to temporal misalignment between subtitles and visual clips, e.g.,t1 in <PIC>. It often occurs when people explain their actions before or after actually performing them, resulting in the mismatch between the order of statements and actions. On the other hand,irrelevant misalignment refers to irrelevant or meaningless captions that cannot be aligned with any available video clips (e.g., t2 and t6 in <PIC>), and vice versa for video clips. According to Han et al.(2022), only 30% of clip-caption pairs are visually aligned in HowTo100M (Miech et al., 2019), with even fewer 15% being naturally well-aligned; ii) Fine-grained misalignment (Frame-word). Within each video clip, the narration sentences may only partially correlate with the visual frames. As de-picted in <PIC>, “the sugar goes on top” in t5 is strongly correlated with visual content v5 while the action “watch the glaze take off” is uncorrelated. Irrelevant words or frames can distort the identi-fication of crucial ones and result in inaccurate similarity measurements, further contaminating the clip-caption alignment. Note that only a few methods (Han et al., 2022) consider the coarse-grained misalignment problem in temporal learning while none of them realize this fine-grained misalign-ment problem. Undoubtedly, MNC poses a significant obstacle to effective temporal modeling.
As long videos are typically composed of a sequence of short video clips, we propose to use the optimal transport distance between video clips and captions as the similarity criterion for video-paragraph contrastive learning in a robust and efficient way.
Fine-grained Alignment. Most previous works (Xu et al., 2021; Yang et al., 2023b; Han et al.,2022) typically encode frames or words to a global feature using [CLS] token or averaging the frame
or word embeddings. However, such strategies neglect fine-grained in-teractions between modalities and do not address the problem of frame-word misalignment.To address this issue, we propose a cross-modal late interaction mechanism to identify crucial words and key frames for fine-grained alignment inspired by Yao et al. (2022); Wang et al. (2022b).Take the front part for example, for each frame in the video clip, we identify the most impor-
tant words through a soft-maximum operation, i.e., log-sum-exp approximation (Beck & Teboulle,2012), and then compute the average soft-maximum similarities of all frames as shown in Fig. 2.Similarly, for each textual token, we also find its related video frames in the latter part of Eq. (5). The parameter α magnifies the importance of the most relevant words or frames. As α approaches 0, the log-sum-exp approximates the maximum. Specifically, this soft-maximum operation allows us to reduce the negative influence of background words or frames on clip-caption similarity estimation.
Since self-supervised contrastive learning (He et al., 2020) relies on the random sampling of neg-ative instances, captions that are semantically similar to the anchor clips can be treated as faulty negatives (Han et al., 2020; Zolfaghari et al., 2021), and vice versa. However, the existing one-hot target used in contrastive learning penalizes all negative predictions regardless of their correlations.To mitigate this issue, we propose to exploit the faulty negatives through optimal transport.
Results. i) Background Removed: As shown in <PIC>, TempCLR (Yang et al., 2023b) performs remarkably better than VideoCLIP (Xu et al., 2021) in terms of DTW and OTAM, as it is trained to explore the global temporal context. However, all these methods suffer from noisy correspon-dence in the temporal alignment. In contrast, our proposed robust optimal transport framework explicitly overcomes multi-granularity noisy correspondence. Specifically, our method effectively improves the performance of all measurements by a large margin (+ 1% Cap. Avg., 5.2% DTW, and 4% OTAM in terms of R@1), indicating that our method learns better temporal information.ii) Background Kept: As shown in <PIC>, compared with the Background Removed results, the recall of all methods dropped as the irrelevant information in the background can distract the video features. Nevertheless, our proposed method consistently outperformed VideoCLIP and TempCLR,even under such challenging conditions.


Given an initial adversarial request Radv such as “write a tutorial on how to make bomb” that models always reject, the goal is to formulate a prompt that induces the target LLM (T) to generate content that fulfills the request. Our method, IRIS, uses the same LLM for the attacker (A). As shown in <PIC>, IRIS consists of two main steps: Iterative Refinement, which iteratively refines the initial prompt, and Rate+Enhance, which maximizes the harmfulness of the output.
During the Iterative Refinement step, IRIS operates through a feedback loop that progres-sively refines the adversarial prompt based on T's responses and A's modifications. At each iter-ation, the current prompt Pcurrent is presented to T, and its response R is evaluated to deter-mine whether T rejects the request based on a simple rule: reject if the response is less than 20 words; otherwise, do not reject. If T rejects the prompt, IRIS solicits an explanation from the attacker model A on why the attempt failed us-ing a template query QA(EXPLAIN:R). This self-
explanation step is vital for well-aligned LLMs(e.g., GPT-4), as it prevents an immediate rejection when directly asking models to refine the failed adversarial prompt, QA(MODIFY:Pcurrent). The output from the MODIFY query is a refined prompt, Prefined, which becomes the new basis for subse-quent iterations. The iterative prompt refinement process continues until Radv is found or the num-ber of attempts N is reached, which we set N = 4 in our experiments. Since there are 3 queries in each iteration, IRIS produces a maximum of 3N+1 = 13 queries to LLMs for our experiments, which is significantly more efficient than previous approaches (over 20 queries). However, over 80% of the time, only one or two iterations are used. Ex-periment artifacts show refined prompts Prefined
produced by IRIS always request the same harmful behavior as the original prompts Padv.
In the Rate+Enhance step, IRIS further prompts the target model to rate the harmfulness of Radv from 1 to 5 and refine the response to maximize its harmfulness rating, as Radv could just be a long out-put that containing safe educational content rather than harmful output. We provide an algorithmic implementation of IRIS in Algorithm 1.
Jailbreaking Methods for Comparison. In ad-dition to IRIS, we consider two state-of-the-art methods that use LLMs to refine jailbreak prompts: PAIR (Chao et al., 2023) and TAP (Mehrotra et al., 2023). PAIR uses Vicuna-13B (Chiang et al., 2023) to iteratively refine the prompt, while TAP further improves the method by incorporating the tree-of-thought reasoning (Yao et al., 2024). There is an- other method, PAP (Zeng et al., 2024), that fine-tunes GPT-3.5 to generate prompts but requires 400 queries when jailbreaking GPT-4. We exclude it from our comparisons.
<PIC> shows the main results that compare IRIS with TAP and PAIR, whose results were reported in Mehrotra et al. (2023). IRIS-2x represents an ensemble of two independent IRIS trials on each ad-versarial prompt, where the jailbreak is considered successful if at least one of the trials succeeds. The average number of queries for IRIS-2x is the sum of the queries in the two trials. We find that IRIS achieves higher jailbreak success rates with signif-icantly fewer queries than TAP and PAIR. IRIS has success rates of 98% and 92% for GPT-4 and GPT-4 Turbo, respectively, using under 7 queries on average. With two independent trials (IRIS-2x), these rates rise to 100% and 98% with under 13 queries on average, which is approximately 55% fewer queries than other methods while increasing the jailbreak success rate by at least 22%.


<PIC> shows the overall architectural diagram of our proposed regeneration-based approach. The input to the framework is the paper and its review which we aim to determine whether they are written by AI or Human.
[<PIC>]
The idea behind this approach is that if a simi-lar prompt is given repeatedly to a large language model (LLM), the LLM is likely to generate re-views or responses that exhibit a consistent style,tone, and content, as outlined in the provided con-text. This consistency occurs because a large lan-guage model generally applies the patterns it has learned during training to the new content it gen-erates based on the given prompt. The study in (Hackl et al., 2023) found that GPT-4 demonstrated high inter-rater reliability, with ICC scores ranging from 0.94 to 0.99, in rating responses across mul-tiple iterations and time periods (both short-term and long-term). This indicates consistent perfor-mance when given the same prompt. Furthermore,the results showed that different types of feedbacks (content or style) did not affect the consistency of GPT-4's ratings, further supporting the model's ability to maintain a consistent approach based on the prompt.
We employ GPT to regenerate a review Rreg using the prompt Preg. We create two distinct embed-dings ER for Rreg and EF for R (review which we have to determine if the review is AI-generated or not). The idea is that if the reviewR is generated by an AI, we hypothesize that its embedding EF will exhibit a closer similarity to ER, the embedding of a known AI-generated review Rreg.Then, we quantify the similarity between the embeddings using the cosine similarity metric.
Here, · represents the dot product, and ∥R∥ and ∥F∥ represent the Euclidean norms of the embed-dings. This formula calculates the cosine of the an-gle between the two embeddings ER and EF, pro-viding a measure of similarity where values closer to 1 indicate higher similarity and thus a greater likelihood that both reviews are AI-generated.
Next, we utilize the computed similarity score as input to train a neural network aimed at detect-ing AI-generated reviews. The training process involves optimizing the network's parameters via backpropagation. This optimization is directed by the cross-entropy loss function.
We propose an attack method to reduce the prob-ability of reviews being classified as AI-generated described in Algorithm-1 where we target the most frequent tokens in AI-generated reviews and re-place them with their synonyms, which are less frequent in the AI-generated content.
Here, we focus exclusively on adjectives, refer-ring to this approach as the "adjective attack." We chose adjectives because substituting nouns and adverbs with their synonyms often leads to nonsen-sical statements or drastically alters the meaning of the review. We discuss this in detail in Appendix C.
In the adjective attack, we substitute the top 100 highest probability adjective tokens (e.g., "novel," "comprehensive") with their synonyms.
To obtain synonyms for the selected tokens, we utilize the NLTK WordNet database7. To preserve the original meaning of tokens as much as possible, we ensure that any synonym used to replace a token is also present in the AI corpus. If a suitable syn-onym is not found in the corpus, we do not replace the token.
In order to determine which tokens from the review should be replaced with their synonyms, we performed PoS tagging on the review. For example, if we are conducting an adjective attack, we replace only the adjective tokens in the review with their synonyms.
We also illustrate this with an example of an adjective attack, as shown in <PIC>. In this ex-ample, the adjective tokens 'better' and 'various' from a review are among the top 100 AI token list. We replace them with their synonyms, 'improved' and 'numerous,' respectively.


We first analyze how each pre- or inprocessing fairness method compares with each other, without the
effects of postprocessing. <PIC> shows the Pareto frontiers achieved by each method when using
GBM base models (see also <PIC> of the appendix). Overall, preprocessing methods (LFR and
CR) achieved lacklustre fairness-accuracy trade-offs across all datasets, while the EG and FairGBM
inprocessing methods performed best (highest area above Pareto frontiers). Specifically, LFR is
indeed able to achieve high fairness fulfillment, but at a steep accuracy cost. To clarify: the plotted
colored Pareto frontiers correspond to multiple (up to 50) different underlying base models, while the
black dashed line corresponds to the postprocessings of the single GBM-based model with highest
accuracy. The following subsection contains a more detailed analysis of postprocessing.
Interestingly, some fairness methods were able to achieve higher test accuracy than unconstrained
GBM models, suggesting improved generalization performance. <PIC> shows one potential reason:
fairness methods can take notoriously high compute resources to train, potentially giving them
a compute advantage with respect to their unconstrained counterparts. Recent related work has
put forth other explanations for why fairness-constraining can improve learning. Wei et al. (2023)
find that fairness can improve overall performance under label noise by improving learning on tail
sub-populations. On the other hand, Creager et al. (2021) establish ties between common fairness
constraints and goals from the robustness literature, suggesting that fairness-aware learning can
improve generalization under distribution shifts.
Finally, <PIC> shows results for a similar experiment where fairness constraints were learned only
on the two largest sub-groups (White and Black). This leads to an arguably easier problem to solve,
which is reflected on the general compression of models on the vertical axis (reduced constraint
violation for all models). While previously the maximum unprocessed accuracy on ACSIncome was
achieved at 0.38 constraint violation, on this binary-group setting it is achieved at 0.16 constraint
violation. Nonetheless, the same trend is visible on all studied datasets. Unconstrained models —
either trained in an unconstrained manner (circles) or made unconstrained via unprocessing (m∗) —
occupy regions of high accuracy and low fairness (high constraint violation). However, the Pareto
frontier that results from postprocessing m∗ (the best-performing unconstrained model) to different
levels of fairness relaxation again dominates or matches the remaining fairness methods.


Frequency spectrum analysis of the diffusion process. The natural images with different reso-
lutions can be viewed as the result of visual signals sampled at varying frequencies. To compare
the frequency features of a 64 × 64 image and a 256 × 256 image, we can upsample the 64 × 64
one to 256 × 256, perform DCT and compare them in the 256-point DCT spectrum. The second
row of <PIC> shows the signal noise ratio (SNR) at different frequencies and diffusion steps.
In <PIC>, we clearly find that the same noise level on a higher resolution results in a higher
SNR in the (low-frequency part of) the frequency domain. Detailed frequency spectrum analysis are
included in Appendix C.
At a certain diffusion step, a higher SNR means that during training the neural network presumes
the input image more accurate, but the early steps may not be able to generate such accurate images
after the increase in SNR. This training-inference mismatch will accumulate over step by step during
sampling, leading to the degradation of performance.
Why does the cascaded models alleviate this issue? Experiments in previous works (Nichol &
Dhariwal, 2021; Dhariwal & Nichol, 2021) have already shown that cascaded models perform better
than end-to-end models under a fair setting. These models usually use the same noise schedule in all
stages, so why are the cascaded models not affected by the increase of SNR? The reason is that in
the super-resolution stages, the low-resolution condition greatly ease the difficulty of the early steps,
so that even the higher SNR requires a more accurate input, the accuracy is within the capability of
the model.
A natural idea is that since the low-frequency information in the high-resolution stage has already
been determined by the low-resolution condition, we can continue generating directly from the up-
sampled result to reduce both the training and sampling steps. However, the generation of low-
resolution images is not perfect, and thus the solution of the distribution mismatch between ground-
truth and generated low-resolution images is a priority to “continue” the diffusion process.
The Effectiveness of block noise. We compare the performance of RDM with and without adding
block noise in <PIC> and <PIC>. With a sufficient phase of training, RDM with block noise outperforms
the model without block noise by a remarkable margin on both ImageNet and CelebA-HQ. This
demonstrates the effectiveness of the block noise. The addition of block noise introduces higher
modeling complexity of the noise pattern, which contributes to a slower convergence of training in
the initial stage, as illustrated by <PIC>. We assume that training on a significantly smaller
scale of samples leads to a fast convergence of the model, which obliterates such a feature, therefore
a similar phenomenon cannot be observed in the training of CelebA-HQ.


There are various characteristics of this environment that make it open-ended (<PIC>): Imperfect
information, where players only have access to information discovered by their own units and
cities. Stochastic dynamics with random events and crises that can disrupt plans. Multiple victory
paths are possible (e.g., conquer, science, or highest score), requiring a balance between economic
expansion, military development, diplomacy, culture, and technology. A dynamic game space with
continuous changes in state and action space for a single agent. Multi-agent interactions with built-
in AI players or other models, providing the potential for self-play. General-sum game that allows
alliance formation during gameplay, which changes the game structure and makes the victories of
different players non-exclusive. Changes in the number of players during a game due to revolts
or conquers, leading to significant alterations in the joint state and action space. Communication
between players through diplomatic actions and natural language chat, allowing agents to use their
natural language capabilities. In summary, CivRealm presents unique challenges and complexities,
making it an open-ended testbed for decision-making agents. Please see § A.1 for more details.
Agent-architecture-agnostic framework. CivRealm empowers each agent to act as a player in
the open-source turn-based strategy game Freeciv [74]. CivRealm employs a server-proxy-client
framework and implements proxy APIs so that a server hosts the game and the proxy establishes
the connection between agents (i.e., clients) and the server. The proxy distributes the game states
received from the server to each agent and submits the actions returned by agents to the server. By
this design, agents with various architectures can seamlessly engage in Freeciv by interpreting the
observations provided by the proxy and generating actions that adhere to CivRealm's specifications..
LLM-based-agent-friendly. Freeciv is a turn-based game that operates without the need for real-
time reactions. This affords players ample time for thoughtful deliberation. This pace aligns well
with the operation of LLM agents, which typically demand substantial time for inference.
Evaluation platform for generalization ability. CivRealm offers multiple convenient methods to
create novel scenarios, such as generating random maps with diverse landscapes and varying player
and unit numbers, or modifying the rule sets that define the fundamental game rules. These elements
result in new configurations, demanding agents to reason the underlying game mechanics rather than
relying solely on memorized experiences and public knowledge. Therefore, CivRealm serves as an
effective platform for assessing the generalization capabilities of decision-making agents.
Support for a variety of tasks. CivRealm offers a wide range of learning and reasoning tasks. These
tasks include not only the comprehensive full game of Freeciv, but also smaller-scale mini-games
designed using Lua scripts. In § 3.2, we will provide detailed descriptions of these tasks.
In CivRealm, players take the role of civilization leaders with the objective of guiding their civi-
lization from its humble beginnings to greatness, where full games can last from several hours to
several days. Civilizations evolve through eras, with an explosion in the number of controllable
objects as the game progresses, resulting in vast state spaces and joint actions (<PIC>). Decisions
in the game have multifaceted impacts, encompassing both long-term strategic consequences and
short-term tactical outcomes. This complexity necessitates a thought process that carefully weighs
the implications of these decisions since greedy moves can easily be non-optimal in the long term.
Observations. Instead of directly processing raw pixel data of the game interface, we extract representative
 discrete information from graphics observed during human gameplay. These observations
encompass data related to the map, units, cities, government, technology, and diplomacy. The map
information includes whether a particular tile is observable, where a “tile” refers to a square space
on the grid-based map. The map provides details like the terrain type, owner of the tile, resource
output, additional infrastructure, and units present on the tile. The unit information provides insights
into a unit’s health, location, owner, attack/defense strength, remaining movement points (indicating
the actions the unit can take in a turn), maintenance costs, etc. The city information covers details
such as a city's location, owner, size, population, shield value, resource production, and more. The
government information indicates the current government type of the civilization, the tax rate, etc.
The technology information displays the technologies that have been researched and the technology
currently being researched. The diplomacy information comprises data regarding diplomatic rela-
tionships with other players. For a comprehensive list of these observations, please refer to § A.1.1.
Actions. We have implemented a rich set of action classes that encompass the five primary facets
of gameplay: unit, city, government, technology, and diplomacy. The unit actions are responsible
for controlling a player's units. They can be categorized into three main types: engineering actions,
which handle tasks like city construction, planting, mining, and more; movement actions, including
moving, transportation, embarking, and so on; and military actions, such as attacking, fortifying,
bribing, etc. The city actions pertain to the development and management of a city. They include unit
production, building construction, city worker assignment, and more. The government actions allow
players to change their government type to gain corresponding political benefits, adjust tax rates to
balance economic expansion and citizen happiness, etc. The technology actions enable players to set
immediate or long-term goals for their technology research. The diplomacy actions empower players
to initiate negotiations, such as trading technologies, negotiating ceasefires, forming alliances, etc.
For an exhaustive list of the implemented actions, please refer to § A.1.2.
Evaluation Metrics. CivRealm offers evaluation metrics to assess playing performance across var-
ious dimensions, including population, constructed cities, researched technologies, produced units,
explored land, etc. An aggregated score is provided for overall evaluation. Please refer to § A.1.3.


We introduce Heterogeneous Pre-trained Transformers (HPT), a family of architecture designed to scalably learn from data across heterogeneous embodiments. HPT modularizes a general policy network architecture (<PIC>) and pre-trains the policy representation of a latent transformer with supervised learning. Inspired by learning from multimodal data [1, 73, 19, 30], we use embodiment-specific tokenizers, dubbed “stem”, to align various sensor inputs such as camera views and pro-prioception inputs. The “trunk” is shared and pre-trained across datasets and is transferred when adapting to new embodiments and tasks that are unknown during the pre-training times. Moreover,we use task-specific action decoders, dubbed “head”, to produce the action outputs. Crucially, after
“tokenizing each embodiment”, HPT operates on a shared space of a short sequence of latent tokens.This hierarchy is motivated by how humans handle feedback loops between specific motor responses and perceived stimuli at the level of the spinal cord’s neural circuitry [68].
We extensively investigated the scaling behaviors and various designs of policy pre-training to the extent of more than 50 individual data sources (2 times more than [55]) and model size of over 1 billion parameters. Analogous to the scaling laws [26, 28], we found that to some extent, HPT scales with the dataset quantity and diversity as well as the model and training compute.
In addition, heterogeneity can occur in different embodiment domains, such as real robot hardware, simulation domains, and human videos. We incorporate many available embodied datasets in different embodiments such as real robots [14, 75, 38], simulation [81, 89, 49, 20, 85, 80] and internet human videos [15] in the pre-training process and demonstrate the generality of our framework including embodiments beyond expensive real-world on-robot teleoperations.
Through transfer learning experiments across multiple simulation benchmarks [89, 49, 81] and real-world dexterous tasks, we compare with several baselines and the from-scratch counterparts.Overall, based on the pre-training objectives, HPT can scale with the model, data, compute, and the heterogeneity of the robotic datasets across real robots, simulations, and human videos. These pre-training procedures and models can simplify building reliable robotic policies for new embodiments and new tasks in terms of data requirements and generalized performance. As an attempt to scale heterogeneous pre-training, our code and weights are open-sourced, and we hope that HPT can shed some light on learning robot representations from heterogeneous embodiments and tasks.
We experiment beyond real-world robot teleop data, which is expensive to collect and scale. For the additional datasets, we consider 7 simulation datasets across many popular simulators Drake [81], Mujoco [89, 49], Isaac Sim [20], and PyBullet [85, 80], as well as Sapien [52] and Flex [66], with image inputs and expert demonstrations. For the human datasets that lack proprioception and action information, we use poses and 2D positions as surrogates for the supervised policy learning objectives.We use in total 300 trajectories from EPIC kitchen [15] and PoCo [82] with a maximum trajectory length 1000. See Appendix <PIC> and Table 4 for more details on the dataset compositions.
In <PIC>, we use a maximum of 1000 trajectories for each dataset and compare against the baseline of 27 datasets with evaluation on all the pre-trained datasets. We show that pre-training on additional embodiment datasets such as simulation and human video datasets can be possible, despite the large embodiment gaps with real robots. These datasets provide complimentary embodiment data to pure teleop data, and they illustrate how much heterogeneity can be handled in the HPT framework.


This paper proposes a novel framework, shown in <PIC>, referred to as the Boosting of Thoughts (BoT), which achieves the boosting mechanism that embraces aggregation and experience, thereby enabling the progressive refinement of unreliable reasoning steps (weak thoughts) by learning from errors to eventually solve various problems. Starting with a simple prompt without human annotations for LLMs, BoT may get weak thoughts. With aggregation, BoT is capable of deriving a more logical and effective thought chain from them, thereby guiding the subsequent refinement. This guidance in our framework is achieved by tuning the prompt with experience, which is the detailed error reports, advice, and instructions of each reasoning step obtained by exploiting LLMs to analyze the aggregated chain. When such experience accumulates in the prompt, it gradually leads to stronger thoughts.
Specifically, BoT implements such a Boosting mechanism as an experience-driven iteration process,as shown in <PIC>. In each iteration, for a given prompt, BoT builds massive simplistic thought structures in parallel with the LLM. We select the tree structure as in ToT Yao et al. (2023) but significantly modify it to weighted binary trees with various growth strategies for our boosting purposes. After extracting the root-to-leaf branch with the highest score per tree, the aggregation component of BoT is performed to aggregate them into one single thought chain. Subsequently, this chain is evaluated by the same LLM to gain the experience, which is added to the prompt as guidance for the thought generation in the next iteration.
The primary experimental results are summarized in <PIC> and <PIC>, where we present insights into the overall performance of BoT. Our findings indicate that the proposed BoT with Boosting mechanism 1). obtains competitive problem-solving rates in most datasets without human annotations;2). is capable of reaching a new state-of-the-art on GSM8K and AQuA when provided with CoT examples. However, experimental results also demonstrate that BoT heavily relies on experience,thus is sensitive to the ability of LLMs.
Specifically, in <PIC>, BoT, starting from a simple initial prompt and performing basic chatting,eventually obtains a GSM8K solve rate 0.1% higher than the current state-of-the-art (SOTA) CSV Zhou et al. (2023), which heavily relies on code interpreter of GPT-4. Considering AQuA, BoT is 2.5% higher than SOTA. This demonstrates that by adding error analysis and advice to the prompt without human annotations, LLMs are able to perform well on complex reasoning. The main reason is that a simple prompt can be iteratively refined by accumulating prior experience towards accurate problem-solving. After including CoT examples in the prompt, BoT+CoT outperforms SOTA by
1.3% on average in GSM8K and AQuA datasets. We argue that the CoT examples can be regarded as the success cases in the experience, directly guiding the subsequent thought structures generation of BoT. Thus, cooperating with the iteration refinement, BoT+CoT reaches a new SOTA. It also deserves to show that because BoT can gradually collect analysis of various reasoning chains (bad or good) as experience, it is consistently close to the BoT+CoT. However, BoT and BoT+CoT, especially BoT, are at least 18% lower than SOTA in MATH. This observation means weak LLMs may not perform well with BoT due to their lower ability to analyze reasoning chains for an effective experience, as supported by <PIC>.
<PIC> presents that with BoT, GPT-4 and Llama2 are respectively improved by 11.6% and 4.4% on average in three datasets. The two numbers show a clear trend that when the LLM is weaker, BoT's performance drops significantly. With powerful GPT-4, as presented in <PIC>, BoT and BoT-CoT behave similarly to those shown in <PIC>. Additionally, their performance escalates along a similar trend as the number of trees varies from 1 to 20. As Llama2 is weaker, BoT is unable to benefit from its analysis to perform the experience-driven iteration process, which is particularly shown by <PIC> (a). When provided with valid success cases, i.e., 5-shots, BoT, through progressive refinement, can still help Llama2 to solve more problems than the baseline even though the improvement is limited.


In standard supervised HTC, there is an underlying assumption of abundant training samples(Zhao et al., 2023; Im et al., 2023; Song et al.,2023), which is often unattainable and expensive to construct manually. Moreover, HTC datasets are characterized by a complex hierarchical label structure, with leaf labels typically following a Zipfian distribution, resulting in very few data instances for these labels. As a result, the few-shot setting is more realistic, and has gained increasing interest recently (Ji et al., 2023; Bhambhoria et al., 2023; Wang et al., 2023b). Nevertheless, existing works often struggle with unsatisfactory performance in this setting. For example, BERT with the vanilla fine-tuning strategy performs extremely poorly in few-shot HTC.
Recently, large language models (LLMs) have achieved notable success on various NLP tasks (Wang et al., 2023a; Drozdov et al., 2023; Zeng et al., 2023), which have significantly enhanced the efficacy of in-context learning (ICL) with relevant demonstrations in the few-shot setting (Shome and Yadav, 2023; Dai et al., 2023; Zhang et al., 2023). However, the application of ICL on HTC faces unique challenges, diverging from traditional text classification scenarios. These challenges are primarily due to two distinct characteristics of HTC, as delineated in
<PIC>.
Firstly, HTC features a deep hierarchical labeling structure and expansive label sets, resulting in large label sets in ICL, which adversely impacts its performance. Secondly, as the hierarchy deepens, the semantic similarity between adjacent labels increases (Stein et al., 2019), making it very challenging to select relevant demonstrations that guide the learning process efficiently.
In this work, we introduce the first ICL-based framework for few-shot HTC. Specifically, we use a LLM as the foundation model for inference, and provide demonstrations to guide HTC label generation through ICL. Our success depends on finding suitable demonstrations for a given input. In order to achieve this, we build a retrieval database that can find the most-relevant demonstrations for the input. Further, in order to avoid providing an enormous set of multi-layer contextual HTC labels all at once, as is required for ICL, we suggest an iterative policy to infer the labels layer-by-layer, reducing the number of candidate labels greatly <PIC>
 illustrates our ICL-based framework for HTC. We first train a PLM-based indexer and build a retrieval database containing reference samples (the training data).
After that, we perform a similarity search in the retrieval database with the text to be inferred. Finally, we construct an ICL prompt with highly similar demonstrations for prediction.
We will introduce our ICL prompt policy for HTC (§ 3.1), and then detail the retrieval database construction (§ 3.2) and demonstration retrieval methods (§ 3.3). Indexer Training. For indexer training, we apply the objectives of mask language modeling Lmlm, and layer-wise classification Lcls. Lmlm is used to predict the words that fill the random mask tokens in the inputs. Lcls is to predict HTC labels through each hierarchical layer index vectors.
Additionally, we propose DCL for indexer training, which uses label text information to select positive and negative samples. For x, positive samples are chosen from sentences with the same label as x. Additionally, the corresponding label description d could be treated as a positive sample. Negative samples consist of two parts. First, based on the similarity between d and descriptions of other labels, negative examples are sampled from highly similar label categories. Similarly, their corresponding label descriptions could be also treated as negative samples. In addition, a few randomly selected sentences from other labels are used as negative samples of x. Thus, compared to traditional random sampling methods, our negative sample selection approach opts for more instances with semantically similar labels as hard negative samples.
Then the index vectors among the positive samples are pulled together and the negative ones are pushed apart. Taking x as an example, denote B = {x, x+, x− 1 , . . . , x− n } as a group of input data. The contrastive loss can be calculated as: In this paper, we proposed a retrieval-style ICL framework for few-shot HTC. We uniquely identify the most relevant demonstrations from a retrieval database to improve ICL performance and meanwhile designed an iterative policy to inference hierarchical labels sequentially, significantly reducing the number of candidate labels. The retrieval database is achieved by using a HTC labelaware representation for any given input, enabling the differentiation of semantically-closed labels (especially the leaf adjacent labels). The representation learning is implemented by continual training on a PLM with three carefully-designed objectives including MLM, layer-wise classification, and a novel DCL objective.
We conducted experiments on three benchmark datasets to evaluate our method. The results show that our method is highly effective, which is able to gain large improvements among a serious of baselines. Finally, our method can bring the stateof-the-art results in few-shot HTC on the three datasets. Further, we performed comprehensive analysis for deep understanding of our method, spreading various important factors.
This work still includes several unresolved problems, which might be addressed in the future.
Firstly, LLMs are currently confined to expanding text via label descriptions and their application to full training set expansion has not been effective.
In order to fully utilize LLMs in text expansion, we need further optimization. Second, the performance gap between supervised methods and our ICL-based approach appears to diminish with increasing training dataset size, suggesting the need for further analysis.


In this section, we elaborate on the details of the proposed LlamaDuo, which are depicted in <PIC>. This LLMOps pipeline aims to ensure service LLMs continuity by transitioning knowledge and abilities from service-oriented LLMs to smaller, locally manageable LLMs without the need for human intervention.
[<PIC>]Tasks and coverage dataset. We select four categories of downstream tasks-summarization, classification, coding, and closed QA-based on their prevalent use and relevance to the operational scope of service LLMs. We utilize the open-source “No Robots” [30] dataset as the coverage dataset. This coverage dataset consists of 10K high-quality prompt and response pairs across 10 categories, crafted by expert annotators. Specifically, we utilize four subsets of the coverage dataset, each corresponding to our targeted tasks. These subsets serve as seeds for generating synthetic data that can closely align with user expectations for LLM interactions. Service and local LLMs. Considering the API cost effectiveness, rate limit, and model utility, we select most popular service LLMs including GPT4o by OpenAI 4 , Claude 3 Sonnet by Anthropic 5 , and Gemini 1.5 Flash by Google 6 to serve as synthetic data generator and judges. As for the smallscale local LLMs to be fine-tuned, we opt for the open-source Gemma 2B and 7B [13], Mistral 7B [17], and LLaMA3 8B [25] as the base models. This selection is motivated by our aim to rigorously evaluate the efficacy and adaptability of our proposed pipeline across diverse settings. The varying scales of base models facilitate a nuanced comparison, allowing us to assess the impact of model scale on performance improvements. However, as a model-agnostic LLMOps pipeline, our LlamaDuo can be generalized to various forms of service and local LLMs beyond the aforementioned models. \subsection{Coverage Dataset}
Users interact with service LLMs through prompt engineering efforts.
The historical trials composed of the user input prompt and the responses of service LLMs, and potential errors will be recorded and saved in local storage.
Subsequently, users annotate and collect the most satisfied prompt and response pairs conformed with their real-world use cases.
The resulting instruction dataset is termed as coverage dataset, denoted as $\mathcal{D}^{(0)} \coloneq \{\mathcal{I}^{(0)}_i, \mathcal{R}^{(0)}_i\}^{N}_{i=1}$, and split as train and test subsets by ratio $\Phi$. Here, $\mathcal{I}^{(0)}_i$ denotes the $i$-th instruction (prompt) in $\mathcal{D}^{(0)}$, $\mathcal{R}^{(0)}_i$ is the corresponding response for the $i$-th instruction, and $N$ is the number of samples in $\mathcal{D}^{(0)}$.
Since coverage dataset is of high quality and satisfying the user's intent in real-world context, the train subsets $\mid\mathcal{D}^{(0)}_{train}\mid=\Phi\cdot N$ will be served as seeds for synthetic datasets generation, while the test subset $\mid\mathcal{D}^{(0)}_{test}\mid=(1-\Phi)\cdot N$ is reserved for performance evaluation of the fine-tuned local LLMs. \subsection{Fine-tuning}
To efficiently and effectively adapt the local model to specific downstream task(s), we finetune the local LLM with the supervised learning paradigm on high-quality instruction data.
At the initial cyclicality of the pipeline, the selected local LLM is fine-tuned on the train subsets $\mathcal{D}^{(0)}_{train}$ of the coverage dataset, obtaining the fine-tuned model $\pi^{(0)}$.
At subsequent cyclicality $t$, if the performance of fine-tuned model does not reach or surpass the predetermined evaluation threshold $\mathbf{\varepsilon}$ of specific tasks, the local LLM $\pi^{(t)}$ will be continuously fine-tuned on the increasing number of synthetic data $\{\mathcal{D}^{(1)}_{synth}, \mathcal{D}^{(2)}_{synth}, \ldots, \mathcal{D}^{(t-1)}_{synth}\}$ generated from service LLMs with $\mathcal{D}^{(0)}_{train}$ as seed dataset. Consequently, when $t\ge 1$, the objective of the fine-tuning phase can be formulated as
\begin{equation}
    \mathcal{L}_{\mathrm{SFT}}(\pi^{(t)}, \mathcal{D}^{(t)}) = -\mathbb{E}_{\mathcal{I}^{(t)}\sim\mathcal{D}^{(0)}_{train},\mathcal{R}^{(t)}\sim \{\mathcal{D}^{(0)}_{train},\{\mathcal{D}^{(\tau)}_{synth}\}_{\tau=1}^{t-1}\}}
    \left[\log P_{\pi^{(t-1)}}(\mathcal{R}^{(t)}\mid \mathcal{I}^{(t)})\right],
\end{equation}
\subsection{Batch Inference}
After the fine-tuning stage, the fine-tuned local model is prompted with prompts $\mathcal{I}^{(0)}$ sampled from the test subsets $\mathcal{D}^{(0)}_{test}$ of the coverage dataset to produce corresponding response $\mathcal{\hat{R}} \sim \pi^{(t)}(\mathcal{R}^{(0)}\mid \mathcal{I}^{(0)})$.
To improve the diversity and robustness of responses, the local model generates a batch of $K$ responses $\{\mathcal{\hat{R}}_1, \mathcal{\hat{R}}_2,\ldots,\mathcal{\hat{R}}_K\}$ for each given prompt $\mathcal{I}^{(0)}$.
Totally, it will construct prompt and responses pairs $\{(\mathcal{I}^{(0)}_i,\mathcal{\hat{R}}_i)\}_{i=1}^{(1-\Phi)\cdot N \cdot K}$. Formally,
\begin{equation}
\mathcal{\hat{R}}_k \sim \pi^{(t)}(\mathcal{R}^{(0)}\mid \mathcal{I}^{(0)}), \mathrm{for}\enspace k\in \{1,2,\ldots, K\},
\end{equation}
\begin{equation}
    \mathcal{I}^{(0)} \sim \mathcal{D}^{(0)}_{test}.
\end{equation} \subsection{Evaluation}
In the evaluation stage, we employ ``service LLMs-as-judge'', denoted as $\mathcal{E}_{\mathrm{LLM}}(\cdot)$, to conduct performance evaluation of local model on $\{(\mathcal{I}^{(0)}_i,\mathcal{\hat{R}}_i)\}_{i=1}^{(1-\Phi)\cdot N \cdot K}$.
Following the works \cite{zheng2024judging,yuan2024self}, the service LLMs evaluate each response triple $(\mathcal{I}^{(0)},\mathcal{\hat{R}},\mathcal{R}^{(0)})$, comprising prompt, the corresponding generated response, and the ground truth, by $M$ times with pairwise comparison and single answer grading strategies.
This evaluation process guarantees the trustworthy and reduces the inherent bias of results.
Moreover, when employing LLMs as evaluators, the evaluation metrics can be more flexibly adapted to specific tasks, along with a thorough evaluation guide.
In this paper, we measure the similarity between $\mathcal{\hat{R}}$ and $\mathcal{R}^{(0)}$, and how precise $(\mathcal{I}^{(0)},\mathcal{\hat{R}})$ the responses generated by the local LLM answer the given instructions.
Therefore, $\{(\mathcal{I}^{(0)}_i,\mathcal{\hat{R}}_i,\mathcal{R}^{(0)}_i)\}_{i=1}^{(1-\Phi)\cdot N \cdot K}$ invokes service LLMs to perform evaluation by $(1-\Phi)\cdot N \cdot K \cdot M$ times.  Subsequently, the evaluation results can be leveraged according to the intention of the operator performing this LLMOps pipeline.
For example, actions can be taken to increase the reliability of service LLM as an evaluator by calculating the mean or median.  In this study, we adopt the mean score $V_{\pi^{(t)}}$ and coverage percentage $C_{\pi^{(t)}}$ with $\zeta$ score as evaluation results.  Here, the coverage percentage $C_{\pi^{(t)}}$ indicates the proportion of responses that have met or exceeded the quality benchmark.
Formally, \begin{equation}
    V_{\pi^{(t)}} = \frac{1}{(1-\Phi)\cdot N \cdot K}\sum_{j=1}^{(1-\Phi)\cdot N \cdot K}V_{\pi^{(t)}}^j,
\end{equation}
\begin{equation}
    C_{\pi^{(t)}} = \frac{1}{(1-\Phi)\cdot N \cdot K}\sum_{j=1}^{(1-\Phi)\cdot N \cdot K}\mathds{1}(V_{\pi^{(t)}}^j\ge \zeta),
\end{equation}
\begin{equation}
   V_{\pi^{(t)}}^j = \frac{1}{M}\sum_{m=1}^{M}\mathcal{E}_{\mathrm{LLM}}(\mathrm{prompt}^{(eval)}, d_j),
\end{equation}
\begin{equation}
d_j \sim \{(\mathcal{I}^{(0)}_i,\mathcal{\hat{R}}_i,\mathcal{R}^{(0)}_i)\}_{i=1}^{(1-\Phi)\cdot N \cdot K},
\end{equation}
where $V_{\pi^{(t)}}$ and $C_{\pi^{(t)}}$ denote the performance of local LLM at $t$-th cyclicality, $\mathds{1}(\cdot)$ is an indicator function, $\zeta$ denotes a threshold score of $C_{\pi^{(t)}}$, $\mathrm{prompt}^{(eval)}$ is the system prompt used for LLM-as-a-Judge.
\subsection {Data Synthesis}
If the performance of fine-tuned local LLM $V_{\pi^{(t)}}$ or $C_{\pi^{(t)}}$ fails to reach or surpass the predetermined evaluation threshold $\mathbf{\varepsilon}$ of specific tasks, it indicates that fine-tuned local LLM's capabilities are insufficient for the tasks at hand.
Thus, the local LLM cannot yet serve as a substitute for the service LLM and necessitates further refinement.
To achieve this, we utilize service LLMs to generate additional synthetic datasets for fine-tuning local LLM in the next cyclicality.
To maintain the consistency of data distribution of coverage dataset $\mathcal{D}^{(0)}$ constructed from real-world scenarios, we employ the train subsets $\mathcal{D}^{(0)}_{train}$ as seeds and apply the same framework \cite{wang2023self,alpaca} for synthetic dataset generation.
During synthetic dataset generation, we perform data deduplication to exclude identical samples from $\{\mathcal{D}^{(0)}_{train}, \{\mathcal{D}^{(1)}_{synth}, \mathcal{D}^{(2)}_{synth}, \ldots, \mathcal{D}^{(t-1)}_{synth}\}\}$ and filter out low-quality samples based on carefully designed rules.
Finally, we conduct rigorous data decontamination for the synthetic dataset to remove samples that closely resemble those in the test subset $\mathcal{D}^{(0)}_{test}$ of the coverage dataset.
Formally, the data synthesis stage can be formulated as
\begin{equation}
\mathcal{D}^{(t)}_{synth} \leftarrow \bigcup \psi(\mathcal{D}^{(t)}_{synth},\{\mathcal{D}^{(0)}_{train}, \{\mathcal{D}^{(\tau)}_{synth}\}_{\tau=1}^{t-1}\},\mathcal{D}^{(0)}_{test}),
\end{equation}
\begin{equation}
    \mathcal{D}^{(t)}_{synth} \sim \mathcal{S}_{\mathrm{LLM}}(\mathrm{prompt}^{(synth)}, seed),
\end{equation}
\begin{equation}
    seed \sim \mathcal{D}^{(0)}_{train}, \mathrm{for}\enspace V_{\pi^{(t)}} < \mathbf{\varepsilon}\enspace \mathrm{or}\enspace C_{\pi^{(t)}} < \mathbf{\varepsilon},
\end{equation}
where $\bigcup\psi(\cdot,\cdot,\cdot)$ represent a series of data post-processing operations, $\mathcal{D}^{(t)}_{synth}$ denotes synthetic data generated from service LLMs at $t$-th cyclicality, $\mathcal{S}_{\mathrm{LLM}}$ and $\mathrm{prompt}^{(synth)}$ are the service LLM and system prompt used for the data synthesis, respectively. For GPT4o, as of August 2024, the pricing is \$5 and \$15 per million tokens for input and output, respectively. We estimate that a light workload, utilizing 10 million input tokens and 1 million output tokens per day, incurs a daily cost of \$65. Conversely, a heavy workload, consuming 50 million input tokens and 10 million output tokens per day, is estimated to cost \$325 daily.
The monthly operational cost comparison between Gemma 7B and GPT-4o under different workloads is summarized in Table \ref{tab:spec-cost-comparison}, demonstrating a significant advantage in fine-tuning and deploying a local LLM.  Moreover, as depicted in <PIC>, after the first two months, the cost of using GPT-4 under both light and heavy workloads exceeds that of setting up and running a local model deployed on $1\times\mathrm{L4}$ GPU and $8\times\mathrm{L4}$ GPU. After one year, GPT-4's costs surpass those of deploying a local model in all scenarios.
These findings highlight the substantial economic benefits of investing in local LLM fine-tuning and deployment for long-term use.
Meanwhile, avoiding recurring token-based charges and maintaining control over model customization further enhances the appeal of the \llamaduo for cost-conscious users and scenarios.


In this study, we advance minimally supervised hierarchical text classification by enriching the weak supervision with additional textual clues and leveraging LLMs tailored for the hierarchical label structure.
First, we enrich the existing label taxonomy with class-specific terms derived from two sources: LLM generation and automated extraction from the corpus.
For example, the ``conditioner'' class in Figure~\<PIC> is enriched with key terms like ``moisture'' and ``soft hair'', which distinguish it from other classes.
These terms enhance the supervision signal by combining knowledge from LLMs and text corpus and improve the pseudo label quality for classifier training.
Second, we further explore the application of LLMs in hierarchical text classification from two perspectives.
Specifically, we enhance LLM annotation efficiency and effectiveness through a taxonomy-guided candidate search and optimize LLM document generation to create more precise pseudo data by using taxonomy paths.
\subsection{Core Class Refinement with Enriched Taxonomy} \label{sec:class-refine} With the enriched class-indicative terms for each class, we propose to further utilize them to refine the initial core classes.
In this paper, we adopt an embedding-based document-class matching method.
Unlike previous methods in flat text classification~\cite{Wang2021XClass} that use keyword-level embeddings to estimate document and class representations, here, we are able to define class representations directly based on document-level embeddings thanks to the rough class assignments we created in the core class annotation step (c.f. Sec.~\ref{sec:core-class}). To obtain document representations, we utilize a pre-trained Sentence Transformer model~\cite{reimers-2019-sentence-bert} to encode the entire document, which we denote as $\vec{d}$. Then, for each class $c$, we identify a subset of its assigned documents that explicitly mention at least one of the class-indicative keywords and thus most confidently belong to this class, $D_c = \{d \in D^0_c | \exists w \in T_c, w \in d\}$. Then, we use the average of their document embeddings as the class representation, $\vec{c} = \frac{1}{|D_c|} \sum_{d \in D_c} \vec{d}$.
Finally, we compute the document-class matching score as the cosine similarity between their representations. Based on the document-class matching scores, we make an observation that the true core classes often have much higher matching scores with the document compared to other classes. Therefore, we use the largest ``similarity gap'' for each document to identify its core classes. Specifically, for each document $d_i \in \D$, we first get a ranked list of classes according to the matching scores, denoted as $[c^i_1, c^i_2, \dots, c^i_{|\C|}]$, where $\text{diff}^{\,i}(j) := \cos(\vec{d}_i, \vec{c}^i_j) - \cos(\vec{d}_i, \vec{c}^i_{j+1}) > 0$ for $j \in \{1, \dots, |\C|-1\}$. Then, we find the position $m_i$ with the highest similarity difference with its next one in the list.
After that, we treat the classes ranked above this position as this document's refined core classes $\mathbb{C}_i$, and the corresponding similarity gap as the confidence estimation $conf_i$.
% \vspace{-0.5em}
\begin{align}\label{eq:refine-class}
% \small
\begin{split}
    conf_i = \text{diff}^{\,i}(m_i),\quad
    \mathbb{C}_i &= \{c^i_1, \dots, c^i_{m_i}\},\\
    m_i = \argmax_{j \in \{1, \dots, |\C|-1\}}& \text{diff}^{\,i}(j).
\end{split}
% \vspace{-0.5em}
\end{align}
Finally, we select the top 75\% of documents $d_i$ and their refined core classes with the highest confidence scores $conf_i$, denoted as $\D^\text{core}$.
\subsection{Text Classifier Training with Path-Based Data Augmentation}
\label{sec:classifier} The final step of \Our is to train a hierarchical text classifier using the confident refined core classes. One straightforward way is to directly use the selected core classes as a complete set of pseudo-labeled documents and train a text classifier in a common supervised way. However, such a strategy is ineffective, because the core classes are not comprehensive enough and cannot cover all the classes in the taxonomy.
This is because the hierarchical label space naturally contains fine-grained and long-tail classes, and they are often not guaranteed to be selected as core classes due to their low frequency. Empirically, for the two datasets we use in our experiments, Amazon and DBPedia, the percentages of classes never selected as core classes are 11.6\% and 5.4\%, respectively. These missing classes will never be used as positive classes in the training process if we only train the classifier with the selected core classes. Therefore, to overcome this issue, we propose the idea of \emph{path-based document generation by LLMs} to generate a small number of augmented documents (e.g., $q=5$) for \emph{each} distinct path from a level-1 node to a leaf node in the taxonomy. By adding the generated documents to the pseudo-labeled data, we can ensure that each class of the taxonomy will be a positive class of at least $q$ documents. Moreover, we use a path instead of a single class to guide the LLM generation, because the meaning of lower-level classes is often conditioned on their parents. For example, in Figure~\<PIC>, a path ``hair care'' $\rightarrow$ ``shampoo'' can guide the LLM to generate text about hair shampoo instead of pet shampoo or carpet shampoo that are in different paths. To promote data diversity, we make one LLM query for each path and ask it to generate $q$ diverse documents. Appx.~\ref{appx:prompts} shows the prompts we used. We denote the generated documents as $\D^\text{gen}$.
[<PIC>] Now, with two sets of data, the pseudo-labeled documents $\D^\text{core}$ and LLM-generated documents $\D^\text{gen}$, we are ready to introduce the classifier architecture and the training process. \smallskip
\noindent\textbf{Classifier architecture.} We use a simple text matching network~\citet{shen-etal-2021-taxoclass} as our model architecture, which includes a document encoder initialized with a pre-trained BERT-base model~\cite{Devlin2019BERTPO} and a log-bilinear matching network. Class representations are initialized by class name embeddings (c.f. Sec.~\ref{sec:taxo-enrich}).
\smallskip
\noindent\textbf{Training process.} For each document with refined core classes, we construct its positive classes as the union of its refined core classes and their ancestors in the label taxonomy, and its negative classes are the ones that are not positive classes or descendants of any core class. This is because the ancestors of confident core classes are also likely to be true labels, and the descendants may not all be negative given that the automatically generated core classes are not optimal. For the LLM-generated documents, we are more confident in their labels, so we simply treat all the classes in the corresponding label path as positive classes and all other classes as negative.  We train the text-matching model with the standard binary cross-entropy loss. The loss terms of two sets of data are weighted by their relative size, $\frac{|\D^\text{core}|}{|\D^\text{gen}|}$. Notice that we do not continue training the classifier with self-training that is commonly used in previous studies~\cite{meng2019weakly, shen-etal-2021-taxoclass}. Using self-training may further improve the model performance, which we leave for future exploration. Algorithm~\ref{alg} in Appendix summarizes \Our.
Recently, large language models (LLM) such as GPT-4~\cite{OpenAI2023GPT4TR} and Claude 3~\cite{claude3} have demonstrated strong performance in flat text classification through zero-shot or few-shot prompting~\cite{sun-etal-2023-text}.
However, applying LLMs in hierarchical settings, with large and structured label spaces, remains challenging.
Directly including hundreds of classes in prompts is ineffective, leading to structural information loss, increased computational costs, and diminished clarity for LLMs at distinguishing class-specific information.
Along another line of research, \citet{meng2019weakly} propose to train a moderate-size text classifier by utilizing a small set of keywords or labeled documents for each class.
However, compiling keyword lists for hundreds of classes and obtaining representative documents for each specific and niche category still demand significant human efforts.
\citet{shen-etal-2021-taxoclass} study the hierarchical text classification with \emph{minimal supervision}, which takes the class name as the only supervision signal.
Specifically, they introduce TaxoClass which generates pseudo labels with a textual entailment model for classifier training.
However, this method overlooks additional class-relevant features in the text and suffers from the unreliable pseudo label selection due to domain shift of the entailment model.


We explore a range of techniques to guide LLaMA-2 models (Touvron et al, 2023) to improve MT outputs using fine-grained feedback derived from Multidimensional Quality Metric (MQM) annotations (Freitag et al, 2021), as shown in <PIC>. Following prior work on refinement, we start with evaluating the impact of such feedback when prompting LLMs in zero-shot and few-shot settings (§5). Different from prior work, we then explore fine-tuning the LLM to advance its ability to improve translations based on the feedback provided in the prompt, in an instruction following style (Taori et al, 2023) (§6). Through extensive experiments with three language pairs (Chinese-English, English-German, and English-Russian), we show that prompting LLMs to edit MT with feedback reliably improves translation quality as measured by automatic metrics, particularly in the few shot settings where the LLaMA-2 7B model achieves close peformance to the 13B version (§5). However, the models are unable to make the most of the fine-grained feedback which performs roughly on par with generic prompts for improvement. Instruction fine-tuning shows stronger improvements on translation quality based on both automatic and human evaluation (§6). Our analysis reveals that prompting the finetuned LLMs with fine-grained feedback not only helps fix the errors highlighted in the prompt (§7), but also leads to more natural outputs We ask bilingual human annotators to assess the post-edited outputs obtained by fine-tuning in the bilingual setting as it is the stronger approach based on automatic scores. We randomly select 50 instances for each language pair for annotation. Each instance is examined by 3 human annotators. For each instance of source text, original MT with MQM annotation, post-edited MT, the annotator is asked to rate on a 5-point Likert scale (1 strongly disagree to 5 strongly agree) whether the translation quality has improved, and to what extent the annotated errors are actually resolved through postediting. Ordinal Kripendorff’s alpha (Krippendorff, 2011) 6 , which measure the inter-annotator agreement is moderate for the Overall quality: 0.527, 0.479, 0.421 for Zh-En, En-De, and En-Ru. Annotators are also given the option to provide free form comments. Refer to Appendix F for further details on the annotation set-up.
As illustrated in <PIC>, our human evaluation results confirm that fine-tuning with error annotations enhances overall translation quality (Overall Quality) and effectively resolves errors in the initial translation (Resolve Errors). While this improvement is notably evident in Zh-En and En-De pair, for the En-Ru pair, approximately 40/150 annotations lean towards the Disagree category. Some of the feedback from En-Ru annotators who choose to Disagree state that there are cases when the output translation from the fine-tuned model is more precise in the target language, but loses some of the nuance in the source text. Further, feedback from the annotators support our own observation that the post-editing via finetuning does not only fix targeted errors in the original translation but rewrites for naturalness in the target language. They comment that the fine-tuning translation “better explains the context” and “flows better in the target language” compared to the original translation which seems to be directly translated without consideration of the context. We list further comments in Appendix Table 20.


How can we devise a fine-grained explanationbased text generation metric capable of pinpointing concrete error locations, identifying error types, assigning severity labels, and justifying the final score—all simultaneously without relying on human-annotated data. In this paper, we propose INSTRUCTSCORE, a method to learn an explainable text generation metric without using human annotated ratings. InstructScore provides both a numerical score and a natural language error explanation. To this end, we first extract latent evaluation knowledge from an instruction-following model, such as GPT-4 (OpenAI, 2023), to construct a synthetic dataset with a predetermined explanation structure. Next, we determine a range of explanation failure modes and devise automated feedback to meta-evaluate error explanations. Finally, we further fine-tune INSTRUCTSCORE model on self-generated outputs that optimize feedback scores, resulting in diagnostic reports that are better aligned with humans.
[<PIC>] We leverage GPT-4 to extract representative explainable knowledge that can greatly contribute to the subsequent Exp-Generator learning process. Specifically, we collected raw sentences in the target language from diverse domains and topics via GPT-4 (Details are included in Section 5.1 and Appendix A), resulting in data across diverse tasks. This corpus is used as the starting point to inject errors. Then, we prompt GPT-4 to synthesize designated generation errors, as shown in Table 1. For each text, we specify the number of errors, error types, and severity labels, and ask GPT-4 to generate a candidate output with the specified error descriptions and 2) an explanation for this error annotation. If an evaluation task is multi-dimensional, error types will be separately assigned to each dimension (An example is included in the Appendix). Benefiting from the large-scale pre-training process, GPT-4 is able to generate diverse errors and meet the requirements with specified instructions. To avoid the model’s over-reliance on the lexical and structural similarities between the candidate and raw text, we request GPT-4 to rephrase the raw text sentence to construct a pseudo-reference sentence. By specifying error type t, severity label se, and raw text, GPT-4 is able to generate a synthetic error sentence x with annotated error location l and a pseudo reference y with explanation e.
How can we devise a fine-grained explanationbased text generation metric capable of pinpointing concrete error locations, identifying error types, assigning severity labels, and justifying the final score—all simultaneously without relying on human-annotated data. In this paper, we propose INSTRUCTSCORE, a method to learn an explainable text generation metric without using human annotated ratings. InstructScore provides both a numerical score and a natural language error explanation. To this end, we first extract latent evaluation knowledge from an instruction-following model, such as GPT-4 (OpenAI, 2023), to construct a synthetic dataset with a predetermined explanation structure. Next, we determine a range of explanation failure modes and devise automated feedback to meta-evaluate error explanations. Finally, we further fine-tune INSTRUCTSCORE model on self-generated outputs that optimize feedback scores, resulting in diagnostic reports that are better aligned with humans.
INSTRUCTSCORE assesses the quality of generated texts based on an explainable diagnostic report. Building upon this report, INSTRUCTSCORE provides an intuitive way to comprehend a model’s generation capability, resulting in easier comparison among different models. In particular, we begin by extracting concise yet representative explainable knowledge from a large-scale instructionfollowing model, which is then utilized to train our Exp-Generator. After carefully analyzing the diagnostic reports produced by our Exp-Generator, we summarize common failure modes in diagnostic report and ask GPT-4 to identify them. Then we transform the GPT-4’s feedback into alignment scores using our predefined criteria. Finally, we select diagnostic reports that have the highest alignment scores, and further finetune our Exp-Generator on those self-refined outputs. The overall framework is illustrated in <PIC>. The quality score s for each candidate y is determined based on the number of errors and their severity labels in the diagnostic report. Minor errors are given a score of −1 and major errors are given a score of −5. These penalties for errors are weighted to calculate the final score. Similar to previous practices (Freitag et al, 2021a), our metric identifies up to five errors per sentence.


Impact of selective retrieval. When optimizing the utility model, we add an empty string es in the training process to leverage the knowledge of LLMs to achieve selective retrieval. Here, we conduct some case studies to examine this mechanism. As we can see from Table 3, there are example queries that the utility model deems no retrieval and in total there are 20.9% queries which the utility model ranks the empty string es higher than other documents. As we can see from Table 3, many of the listed queries are commonsense knowledge that has been memorized in LLMs’ parameters and we can easily call an LLM for the required answer instead of retrieval, which illustrates that in retrieval-augmented generation, despite the further knowledge external corpus introduces, the inherent knowledge of LLMs when deploying retrievalbased techniques is worth investigation and we believe that our work has provided a straightforward and plausible solution. However, this design still has certain limitations, i.e., the LLM used for end task needs to be the same or stronger than the one In this work, we argue that similarity is not always the “panacea” for retrieval-augmented generation and totally relying on similarity would sometimes degrade the performance. As is shown in the upper part of <PIC>, when a user types in the query “Tell me about author George RR Martin”, a similarity-driven retrieval system would rank the documents in a given corpus according to similarity metrics, i.e., the semantic relevance or TFIDF based metric (Robertson and Zaragoza, 2009). Even though the retrieved documents barely provide useful information, e.g., “George RR Martin is an author”, it would rank higher due to high similarity score and the document that states the publications “The Song of Ice and Fire” of George RR Martin with higher information gain would rank lower due to inadequate low similarity score. Besides, given the fact that the retrieved documents are often more than one, using them in isolation due to the context limitation of LLMs (Shi et al, 2023)or simply aggregating the Top-k document without considering the relationships between them makes it difficult to capture the commonalities and characteristics among them and even confuse LLMs due to excessive text length thus incurring information loss and probably performance degradation (Mallen et al, 2023). Passage window size matters in utility model performance! Since the utility model plays a key role in bring in the supervision of LLMs into passage selection process, we take a closer look at how the passage window size ∥D•∥ 8 would influence the final performance by directly incorporating the utility model trained under different settings to the final task. Due to the space limit, we average the metrics in four datasets and present the result in <PIC> and find that the model performance improves as the window size grows, which demonstrates that the growing passage window size endows more LLMs’ powers for distinguishing passage significance among diverse inputs to the utility model thus improving performance on downstream tasks. However, due to the computational burden (the model training time grows linearly with the passage window size) that a large passage window size imposes during training, there is need to balance the trade-off between performance and cost.


Recent advancements in neural audio codecs such as SpeechTokenizer [107], Encodec [22] and VALL-E [83] have provided evidence of the advantages of multi-layer residual vector quantizers (RVQs) in accurately representing speech with discrete speech tokens for high-quality and efficient audio transmission, regardless of sound type or language. 2. We aim to develop the neural codec architecture into an effective decoupling model that separates mixed speech tokens into standalone semantic and acoustic tokens. As illustrated in <PIC>, our proposed decoupling model based on the codec architecture (CDM) comprises three core components: an encoderdecoder architecture, a HuBERT-equipped RVQs module, and a discriminator. The encoder-decoder’s primary function of precisely reconstructing the original audio compels the encoder to extract the key features from speech signals. The HuBERT-equipped RVQs further decouple these features and hierarchically quantize them into discrete semantic and acoustic tokens. The discriminator enforces that the encoder and RVQs optimize their learned representations,aiming for comprehensive retention of the original audio’s details. Through this structure, we can achieve effective decoupling of speech signals. The decoupled semantic and acoustic audio samples can be found on our demo page [1].
The bottleneck layer aims to reduce the dimensions of acoustic tokens A from $R^{7𝐶×𝑇𝑛}$ to a more compact space $R^{𝐶×𝑇𝑛}$ by using 1D convolution and batch normalization. This layer serves a dual purpose: first, it enhances computational efficiency and reduces trainable parameters, facilitating subsequent layers to operate on a compact representation; second, it acts as a regularizer.By randomly rearranging the elements across the temporal dimension 𝑇𝑛, this layer nullifies speech comprehension that is highly dependent on the temporal order of phonemes and words [51]. We empirically set a shuffling window of 1 second, corresponding to 50 frames, to obscure word-level intelligibility (as each token representation is extracted from a 20ms waveform).In addition to decoupling speech information, the shuffle layer serves to augment content protection by further scrambling the
condensed acoustic tokens $A^𝑏$ . As shown in <PIC>, By randomly rearranging the elements across the temporal dimension 𝑇𝑛, this layer nullifies speech comprehension that is highly dependent on the temporal order of phonemes and words [51]. We empirically set a shuffling window of 1 second, corresponding to 50 frames, to obscure word-level intelligibility (as each token representation is extracted from a 20ms waveform). Thereby, the likelihood of
attackers deciphering and correcting these sequences is extremely low, given the sheer number of possible permutations for a 4-second audio ($50!^{4}$ , approximately $8.56 × 10^{257}$, details are discussed in §8). Our experiments also confirm the dual content protection by decoupling and shuffling, thwarting the advanced ASR techniques and human auditory analysis.


Large-scale SSL enables audio encoders to capture rich information from speech. Inspired by the BERT-based speech SSL framework [27, 2, 8, 10], we developed our audio encoder, a conformerbased [22] model that captures both global and local structures stored in audio signals. In this work, we primarily focus on speech signal. Since it is trained on large-scale unsupervised data, we term the trained audio encoder as LUISE, which represents Large-scale Unsupervised Iterative Speech Encoder
Adhering to the concept of BERT [14], LUISE adopts a learning paradigm of masked language prediction. The training procedure is illustrated in <PIC>. Specifically, the sequence of melfilterbank feature extracted from the waveform is first input to the tokenizer module to obtain discrete labels for each frame. Then, the training of LUISE is conducted using the cross-entropy criterion, with the loss function calculated only for the masked frames. After training, the softmax layer is removed, and the encoder part of LUISE is used for subsequent supervised fine-tuning.
We utilize an iterative fixed tokenizer method to obtain the corresponding discrete labels for each frame. In the first iteration, we apply a random-projection layer [10] to project speech feature to a randomly initialized codebook, and map them to discrete labels through finding the nearest vector in the codebook. In the second iteration, we perform K-means clustering on the representations of an intermediate layer of the previously trained encoder to obtain a new codebook. The discrete labels are then obtained by finding the closest vector in the new codebook to the representation from the same intermediate layer.  Based on the aforementioned motivation, we propose Seed-ASR, a large-scale speech recognition model built on the framework of audio conditioned LLM (AcLLM). By inputting encoded continuous speech representations together with a task instruction and relevant contexts into a pretrained LLM,Seed-ASR can leverage the rich text knowledge and the reasoning ability of the LLM to generate the corresponding text transcription of speech. The overall framework is shown in <PIC> Audio is a different modality from text. To enable LLMs better understand diverse speech inputs, we adopt the concept of large-scale pretraining in LLMs. Specifically, we construct an audio encoder with nearly 2 billion parameters and conduct self-supervised learning (SSL) on tens of millions of hours of data. The pre-trained audio encoder gains strong speech representation ability, which facilitates rapid convergence during supervised fine-tuning (SFT). After the large-scale SSL stage, we implement a simple and effective stage-wise training recipe within the framework of AcLLM (shown in [<PIC>]). In the stage of SFT, we establish the mapping relationship between speech and text by training on a large amount of speech-text pairs. In the stage of context SFT, we use a relatively small amount of context-speech-text triples to elicit the LLM’s ability to capture speech-relevant clues from context. These triple data can be customized according to specific scenarios. In the stage of reinforcement learning, we apply the training criteria of MWER [37] and some improvements to further strengthen the ability of our model. In the following subsections, we will introduce these methods in more detail.


As shown in <PIC>, the architecture for investigation is simply a speech encoder with an LLM. For each sample, the given prompt for transcribing (i.e., transcribe the speech), the speech utterance, and the corresponding transcript during training are denoted as P, S and T, respectively. We tokenize the prompt and the transcript using the tokenizer and embedding matrix of the LLM to obtain feature vector sequences Ep and Et as: $Ep = Embedding(Tokenizer(P)), (1)$ $Et = Embedding(Tokenizer(T)). (2)$ For the input audio S, we first extract features by passing the audio through a speech encoder to obtain encoder output Hs, denoted as: $Hs = Encoder(S). (3)$ Then, Hs is passed to a projector and further goes through a linear layer to obtain a feature sequence Es with the same dimensionality as the input to the LLM, denoted as: $Es = Linear(Projector(Hs)), (4)$ where the dimension of the feature output by the projector is the same as that of the speech encoder, and the linear layer is responsible for mapping the feature dimension to the embedding dimension of the LLM. Next, we concatenate Es, Ep, and Et to obtain the final feature and pass it to the LLM to obtain the output transcript Y , denoted as: $Y = LLM(Regulation(Ep, Es, Et)). (5)"$
We compare the effects of two types of projectors, Qformer and Transformer. In our experiment, we fix the encoder as HuBERT and LLM as Atom-7B and only unfreeze the projector module for one epoch training to compare the different effects of Qformer and Transformer, respectively. Although we expect Qformer to perform better than transformer similar to Blip [26] and SALMONN [19], results in <PIC> show otherwise. We hypothesize this is because Qformer was originally designed to accommodate the unique data structures in image processing, and should be redesigned to better adapt to speech for future research.