base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
library_name: peft
pipeline_tag: text-generation
license: cc-by-4.0
datasets:
- SHARE
tags:
- long-form-dialogue
- dialogue-generation
Model Card for SHARE: Shared Memory-Aware Open-Domain Long-Term Dialogue Model
This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct based on the SHARE framework, designed to enhance long-term dialogue engagement by leveraging shared memories. It was developed using the new SHARE dataset, constructed from movie scripts, which provides rich explicit persona information, event summaries, and implicitly extractable shared memories between conversational participants. The associated EPISODE framework utilizes these shared experiences to facilitate more engaging and sustainable conversations.
Model Details
Model Description
Shared memories between two individuals strengthen their bond and are crucial for facilitating their ongoing conversations. This model aims to make long-term dialogue more engaging by leveraging these shared memories. To this end, it uses the new long-term dialogue dataset named SHARE, constructed from movie scripts, which are a rich source of shared memories among various relationships. The underlying research introduces the EPISODE framework, a long-term dialogue framework based on SHARE that effectively manages shared memories during dialogue. Experiments demonstrate that shared memories make long-term dialogues more engaging and sustainable.
- Developed by: The authors of the paper "SHARE: Shared Memory-Aware Open-Domain Long-Term Dialogue Dataset Constructed from Movie Script".
- Funded by [optional]: [More Information Needed]
- Shared by [optional]: [More Information Needed]
- Model type: LoRA adapter for Causal Language Model (
meta-llama/Meta-Llama-3.1-8B-Instruct) designed for long-term dialogue. - Language(s) (NLP): English
- License: Creative Commons Attribution 4.0 International (CC BY 4.0)
- Finetuned from model [optional]:
meta-llama/Meta-Llama-3.1-8B-Instruct
Model Sources [optional]
- Repository: https://github.com/share-dialogue/SHARE
- Paper [optional]: SHARE: Shared Memory-Aware Open-Domain Long-Term Dialogue Dataset Constructed from Movie Script
- Demo [optional]: [More Information Needed]
Uses
Direct Use
This model is intended for research and development in long-term dialogue systems. It can be used to generate conversational responses that effectively utilize shared memories between participants, leading to more coherent, engaging, and contextually rich interactions over extended conversations.
Downstream Use [optional]
The model can serve as a foundation for further research and applications in personalized dialogue, conversational AI agents with memory, and interactive storytelling where consistent character memory is crucial.
Out-of-Scope Use
This model is not intended for generating harmful, biased, or inappropriate content. Users should be aware that as a generative language model, it may produce outputs reflecting biases present in its training data or the base model. It should not be deployed in high-stakes applications without rigorous further testing, fine-tuning, and safety measures.
Bias, Risks, and Limitations
The model's performance and outputs are influenced by the characteristics of its training data (movie scripts) and the underlying Meta-Llama-3.1-8B-Instruct base model. Potential biases, stereotypes, or limitations present in these sources may be reflected in the generated dialogues. The abstract does not detail specific ethical considerations or limitations beyond the technical scope.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Continuous monitoring and evaluation of model outputs are recommended, especially when integrating into sensitive applications.
How to Get Started with the Model
Use the code below to get started with the model. This model is a PEFT (LoRA) adapter, meaning you need to load the base model (meta-llama/Meta-Llama-3.1-8B-Instruct) and then apply this adapter on top.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig
# Replace "your_repo_id/this_model" with the actual model ID on the Hub
model_id = "your_repo_id/this_model"
# Load PEFT config to get base model name
config = PeftConfig.from_pretrained(model_id)
# Load the base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained(
config.base_model_name_or_path,
torch_dtype=torch.bfloat16, # or torch.float16 or torch.float32 depending on your GPU/needs
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
# Load the PEFT adapter on top of the base model
model = PeftModel.from_pretrained(base_model, model_id)
# For seamless inference, you might want to merge the adapter weights into the base model
# This makes the model behave like a fully fine-tuned model and can improve performance on some hardware
# model = model.merge_and_unload()
# Example for text generation (using Llama 3.1 chat template)
messages = [
{"role": "user", "content": "Hello, do you remember our conversation about the movie 'Inception'?"},
{"role": "assistant", "content": "Yes, I do! We talked about its complex narrative structure and dream-within-a-dream concept. What specifically about it are you thinking about now?"},
{"role": "user", "content": "I was wondering, what was the name of the character who guided people through the dream layers?"}
]
# Apply chat template and tokenize
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
# Generate response
outputs = model.generate(
input_ids,
max_new_tokens=256,
eos_token_id=tokenizer.eos_token_id,
do_sample=True,
temperature=0.7,
top_p=0.9,
)
generated_text = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
print(generated_text)
Training Details
Training Data
The model was fine-tuned on the new SHARE dataset, which is constructed from movie scripts. This dataset contains the summaries of persona information and events of two individuals, as explicitly revealed in their conversation, along with implicitly extractable shared memories.
Training Procedure
The training procedure leverages the EPISODE framework, a long-term dialogue framework based on the SHARE dataset, which utilizes shared experiences between individuals. The model was fine-tuned using Parameter-Efficient Fine-Tuning (PEFT), specifically LoRA, as indicated by the adapter_config.json. The objective was to enable the model to effectively manage shared memories during dialogue and produce more engaging and sustainable long-term conversations.
Preprocessing [optional]
[More Information Needed]
Training Hyperparameters
- Training regime: [More Information Needed]
Speeds, Sizes, Times [optional]
[More Information Needed]
Evaluation
Testing Data, Factors & Metrics
Experiments using the SHARE dataset were conducted to evaluate the model. The evaluation focused on demonstrating that shared memories between two individuals make long-term dialogues more engaging and sustainable, and that the EPISODE framework effectively manages these shared memories during dialogue.
Testing Data
[More Information Needed]
Factors
[More Information Needed]
Metrics
Specific metrics for evaluation are not detailed in the abstract, but typical dialogue evaluation metrics (e.g., coherence, fluency, engagement, consistency) would likely be used.
Results
The research demonstrates that shared memories between two individuals lead to more engaging and sustainable long-term dialogues, and that the EPISODE framework effectively manages shared memories throughout conversations.
Summary
Model Examination [optional]
[More Information Needed]
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: [More Information Needed]
- Hours used: [More Information Needed]
- Cloud Provider: [More Information Needed]
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Technical Specifications [optional]
Model Architecture and Objective
The model is a PEFT (LoRA) adapted version of the meta-llama/Meta-Llama-3.1-8B-Instruct large language model. Its primary objective is to facilitate long-term, open-domain dialogue by explicitly leveraging shared memories and contextual information, aiming for more engaging and coherent conversations over time.
Compute Infrastructure
[More Information Needed]
Hardware
[More Information Needed]
Software
- PEFT 0.12.0
- Transformers (compatible with Llama 3.1)
Citation [optional]
BibTeX:
@article{share2024longterm,
title={SHARE: Shared Memory-Aware Open-Domain Long-Term Dialogue Dataset Constructed from Movie Script},
author={Authors Name(s) Here}, # Please replace with actual author list from the paper
journal={arXiv preprint arXiv:2410.20682},
year={2024},
url={https://huggingface.co/papers/2410.20682}
}
Glossary [optional]
[More Information Needed]
More Information [optional]
[More Information Needed]
Model Card Authors [optional]
[More Information Needed]
Model Card Contact
[More Information Needed]