README.md

metadata

base_model:
  - TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
  - baichuan-inc/Baichuan2-7B-Base
  - Qwen/Qwen3-8B
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers

Model Card for HLLM

Paper Abstract

AI-generated content technologies are widely used in content creation. However, current AIGC systems rely heavily on creators' inspiration, rarely generating truly user-personalized content. In real-world applications such as online advertising, a single product may have multiple selling points, with different users focusing on different features. This underscores the significant value of personalized, user-centric creative generation. Effective personalized content generation faces two main challenges: (1) accurately modeling user interests and integrating them into the content generation process while adhering to factual constraints, and (2) ensuring high efficiency and scalability to handle the massive user base in industrial scenarios. Additionally, the scarcity of personalized creative data in practice complicates model training, making data construction another key hurdle. We propose HLLM-Creator, a hierarchical LLM framework for efficient user interest modeling and personalized content generation. During inference, a combination of user clustering and a user-ad-matching-prediction based pruning strategy is employed to significantly enhance generation efficiency and reduce computational overhead, making the approach suitable for large-scale deployment. Moreover, we design a data construction pipeline based on chain-of-thought reasoning, which generates high-quality, user-specific creative titles and ensures factual consistency despite limited personalized data. This pipeline serves as a critical foundation for the effectiveness of our model. Extensive experiments on personalized title generation for Douyin Search Ads show the effectiveness of HLLM-Creator. Online A/B test shows a 0.476% increase on Adss, paving the way for more effective and efficient personalized generation in industrial scenarios.

Model Overview

This repo is used for hosting HLLM and HLLM-Creator checkpoints.

For more details or tutorials see https://github.com/bytedance/HLLM.

Hierarchical Large Language Model (HLLM) architecture is designed to enhance sequential recommendation systems:

• HLLM significantly outperforms classical ID-based models on large-scale academic datasets and has been validated to yield tangible benefits in real-world industrial settings. Additionally, this method demonstrates excellent training and serving efficiency.
• HLLM effectively transfers the world knowledge encoded during the LLM pre-training stage into the recommendation model, encompassing both item feature extraction and user interest modeling. Nevertheless, task-specific fine-tuning with recommendation objectives is essential.
• HLLM exhibits excellent scalability, with performance continuously improving as the data volume and model parameters increase. This scalability highlights the potential of the proposed approach when applied to even larger datasets and model sizes.

HLLM-Creator is designed for personalized creative generation:

• HLLM-Creator enables precise user interest modeling and fine-grained content personalization.
• A Chain-of-Thought-based data construction pipeline is developed to expand personalization space and ensure factual consistency, effectively reducing hallucinations in generated titles.
• A flexible and efficient inference scheme is developed for large-scale industrial deployment, with significant positive results in Douyin search advertising demonstrating its real-world impact.

Comparison with state-of-the-art methods (HLLM)

Method	Dataset	Negatives	R@10	R@50	R@200	N@10	N@50	N@200
HSTU	Pixel8M	5632	4.83	10.30	18.28	2.75	3.94	5.13
SASRec	Pixel8M	5632	5.08	10.62	18.64	2.92	4.12	5.32
HLLM-1B	Pixel8M	5632	6.13	12.48	21.18	3.54	4.92	6.22
HSTU-large	Books	512	5.00	11.29	20.13	2.78	4.14	5.47
SASRec	Books	512	5.35	11.91	21.02	2.98	4.40	5.76
HLLM-1B	Books	512	6.97	14.61	24.78	3.98	5.64	7.16
HSTU-large	Books	28672	6.50	12.22	19.93	4.04	5.28	6.44
HLLM-1B	Books	28672	9.28	17.34	27.22	5.65	7.41	8.89
HLLM-7B	Books	28672	9.39	17.65	27.59	5.69	7.50	8.99

Cite our work

@article{HLLM,
title={HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling},
author={Junyi Chen and Lu Chi and Bingyue Peng and Zehuan Yuan},
journal={arXiv preprint arXiv:2409.12740},
year={2024}
}

@article{HLLM-Creator,
title={HLLM-Creator: Hierarchical LLM-based Personalized Creative Generation},
author={Junyi Chen and Lu Chi and Siliang Xu and Shiwei Ran and Bingyue Peng and Zehuan Yuan},
journal={arXiv preprint arXiv:2508.18118},
year={2025}
}

Thanks to the excellent code repository RecBole, VisRec, PixelRec and HSTU ! HLLM is released under the Apache License 2.0, some codes are modified from HSTU and PixelRec, which are released under the Apache License 2.0 and MIT License, respectively.