README.md

---
license: apache-2.0
pipeline_tag: video-text-to-text
library_name: transformers
---

# Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory

This repository contains a model checkpoint from the M3-Agent framework, which was introduced in the paper [Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory](https://arxiv.org/abs/2508.09736).

-   🌐 [**Project Page**](https://m3-agent.github.io)
-   💻 [**GitHub Repository**](https://github.com/hyc2026/M3-Agent-Training)

<div align=left>
    <img src="https://github.com/user-attachments/assets/c42e675e-497c-4508-8bb9-093ad4d1f216" width=40%>
</div>

## Abstract
We introduce M3-Agent, a novel multimodal agent framework equipped with long-term memory. Like humans, M3-Agent can process real-time visual and auditory inputs to build and update its long-term memory. Beyond episodic memory, it also develops semantic memory, enabling it to accumulate world knowledge over time. Its memory is organized in an entity-centric, multimodal format, allowing deeper and more consistent understanding of the environment. Given an instruction, M3-Agent autonomously performs multi-turn, iterative reasoning and retrieves relevant information from memory to accomplish the task. To evaluate memory effectiveness and memory-based reasoning in multimodal agents, we develop M3-Bench, a new long-video question answering benchmark. M3-Bench comprises 100 newly recorded real-world videos captured from a robot's perspective (M3-Bench-robot) and 929 web-sourced videos across diverse scenarios (M3-Bench-web). We annotate question-answer pairs designed to test key capabilities essential for agent applications, such as human understanding, general knowledge extraction, and cross-modal reasoning. Experimental results show that M3-Agent, trained via reinforcement learning, outperforms the strongest baseline, a prompting agent using Gemini-1.5-pro and GPT-4o, achieving 6.7%, 7.7%, and 5.3% higher accuracy on M3-Bench-robot, M3-Bench-web and VideoMME-long, respectively. Our work advances the multimodal agents toward more human-like long-term memory and provides insights into their practical design. Model, code and data are available at this https URL

## M3-Agent Demo Video
Explore M3-Agent's capabilities as a personal assistant in this demo video:

[![Watch the video](https://github.com/ByteDance-Seed/M3-Agent/raw/main/figs/demo.png)](https://www.youtube.com/watch?v=XUx31cBanfo)

## Usage

This model is designed to be a component of the larger M3-Agent framework, typically used for tasks related to memory generation or control. It can be loaded using the Hugging Face `transformers` library.

For detailed usage within the full M3-Agent pipeline, including processing video and audio inputs to generate memory graphs and performing reasoning tasks, please refer to the [official GitHub repository](https://github.com/hyc2026/M3-Agent-Training).

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Assuming this repository contains the 'M3-Agent-Memorization' or a similar Qwen-based checkpoint
# If your repository has a different ID, replace it below.
model_id = "ByteDance-Seed/M3-Agent-Memorization" # This might need to be adjusted to the actual repo ID

# Load tokenizer and model
# trust_remote_code=True is often required for custom architectures like Qwen3
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16, # Use torch.float16 if bfloat16 is not supported by your hardware
    device_map="auto",
    trust_remote_code=True
)

# Set model to evaluation mode
model.eval()

print(f"Model {model_id} loaded successfully.")
print("This model is a component of the M3-Agent framework. For full agent pipeline usage,")
print("including video processing for memory generation and control tasks,")
print("please refer to the official GitHub repository: https://github.com/hyc2026/M3-Agent-Training")

# Example for basic text generation (demonstrates model loading)
# The actual use case for this model is within the M3-Agent pipeline,
# involving multimodal inputs and structured outputs.
# messages = [
#     {"role": "user", "content": "Hello! How are you today?"},
# ]
# text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# input_ids = tokenizer(text, return_tensors="pt").to(model.device)
#
# with torch.inference_mode():
#     outputs = model.generate(input_ids, max_new_tokens=50, do_sample=True, temperature=0.7)
# generated_text = tokenizer.decode(outputs[0][len(input_ids[0]):], skip_special_tokens=True)
# print(f"
Generated response (general text generation): {generated_text}")
```

## Citation
If you find this model or the M3-Agent project helpful, please cite the following paper:

```bibtex
@misc{long2025seeing,
      title={Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory}, 
      author={Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan Lin, Hang Li, Junbo Zhao, Wei Li},
      year={2025},
      eprint={2508.09736},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```