If you already know T5 and Flan-T5, DialogStudio-T5 is better at many things. With the same number of parameters, the models are fine-tuned from a selected amount of dialogues from DialogStudio and also 1000 additional tasks.
Disclaimer: Content from this model card are modified from contents written by the Hugging Face team, and parts of it were copy pasted from the T5 model card and Flan-T5 model card.
Follow the DialogStudio GitHub repository for latest information.
Model Details
Data
We sample a small amount of dialogues from each commercial supported dataset under three categories of DialogStudio, i.e., KG-Dial, TOD and Open-Domain dialogues. Additionally, we sample at most 150 examples for each non-translation task from FLAN.
Note that this model version 1.0 does not incorporate datasets utilized for training large-scale models (>=7B) like Alpaca, ShareGPT, GPT4ALL, UltraChat from OpenAI's 'GPT-3.5/4', or other datasets such as OASST1 and WizardCoder.
We process dialogue data into below input format :
With instruction and external knowledge: Instruction: your instruction <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N <EXTERNAL KNOWLEDGE> your external knowledge
Without instruction: <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N <EXTERNAL KNOWLEDGE> your external knowledge
Without external knowledge: Instruction: your instruction <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N
Without both: <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N
Note: output is final the system response; <USER>, <SYSTEM> and <EXTERNAL KNOWLEDGE> are special tokens
For sampled FLAN data:
We follow their original data format, i.e., we did not set special tokens to separate in-context learning examples.
In summary:
We recommend you use our format and add our special tokens (such as <USER> and <SYSTEM> ) to get better performance. However, you may not necessary need to exactly follow our format if you do not observe random behavios.
We found that T5 model series such as Flan-t5 and DialogStudio-T5 may generate repetitive tokens during inference. If you find such repetition issues, you can set the repetition_penalty in model.generate(), such as 1.5, to mitigate them. Note that repetition_penalty=1.0 by default.
Usage
Find below some example scripts on how to use the model in transformers:
Using the Pytorch model
Running the model on a CPU
Click to expand
python
12from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
34tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-3b-v1.0")5model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-3b-v1.0")67input_text ="Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"8input_ids = tokenizer(input_text, return_tensors="pt").input_ids
910outputs = model.generate(input_ids, max_new_tokens=256)11print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Running the model on a GPU
Click to expand
python
1# pip install accelerate2from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
34tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-3b-v1.0")5model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-3b-v1.0", device_map="auto")67input_text ="Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"8input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")910outputs = model.generate(input_ids, max_new_tokens=256)11print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Running the model on a GPU using different precisions
FP16
Click to expand
python
1# pip install accelerate2import torch
3from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
45tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-3b-v1.0")6model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-3b-v1.0", device_map="auto", torch_dtype=torch.float16)78input_text ="Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"9input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")1011outputs = model.generate(input_ids, max_new_tokens=256)12print(tokenizer.decode(outputs[0], skip_special_tokens=True))
INT8
Click to expand
python
1# pip install bitsandbytes accelerate2from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
34tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-3b-v1.0")5model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-3b-v1.0", device_map="auto", load_in_8bit=True)67input_text ="Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"8input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")910outputs = model.generate(input_ids, max_new_tokens=256)11print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Uses
Direct Use and Downstream Use
The primary use is research on language models, including: research on zero-shot NLP tasks and in-context few-shot learning NLP tasks, such as dialogue response generation, reasoning, and question answering; advancing fairness and safety research, and understanding limitations of current large language models
Out-of-Scope Use
More information needed.
Bias, Risks, and Limitations
The information below in this section are copied and modified from Flan-T5's models card:
Language models, including DialogStudio-T5, can potentially be used for language generation in a harmful way, according to Rae et al. (2021). DialogStudio-T5 should not be used directly in any application, without a prior assessment of safety and fairness concerns specific to the application.
Ethical considerations and risks
DialogStudio-T5 is fine-tuned on a large corpus of text data that was not filtered for explicit content or assessed for existing biases. As a result the model itself is potentially vulnerable to generating equivalently inappropriate content or replicating inherent biases in the underlying data.
Known Limitations
DialogStudio-T5 has not been tested in real world applications.
Sensitive Use:
DialogStudio-T5 should not be applied for any unacceptable use cases, e.g., generation of abusive speech.
Training Details
Training Data
We sample a small amount of dialogues from each commercial supported dataset under three categories of DialogStudio, i.e., KG-Dial, TOD and Open-Domain dialogues. Additionally, we sample at most 150 examples for each non-translation task from FLAN.
Note:
Model Version 1.0 is built on small-scale pre-trained models, this version does not incorporate datasets utilized for training large-scale models (>=7B) like Alpaca, ShareGPT, GPT4ALL, UltraChat from OpenAI's 'GPT-3.5/4', or other datasets such as OASST1 and WizardCoder. As a result, it has certain limitations in terms of writing and creative capabilities. Our initial focus is to update the model versions to enhance existing abilities. Further improvements, including expansion of other capabilities, are part of our roadmap and will be responsive to community requests.
See above Training formats: for details of the training formats.
Training Procedure
These models are based on Flan-T5 and are fine-tuned with instructions for better zero-shot and few-shot performance. There is one fine-tuned DialogStudio model per T5 model size.
The model has been trained on 16 A100 GPUs, each with 40G memory, using public transformer codebase.
Evaluation
Testing Data, Factors & Metrics
The authors evaluated the model on several dialogue tasks and general tasks such as 0-shot/5-shot MMLU and 3-shot BBH.
Results
For full results for DialogStudio, see the research paper.
Environmental Impact
More information needed.
Ethical Considerations
This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP.
Citation
BibTeX:
bibtex
1@misc{zhang2023dialogstudio,
2 title={DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI},
3 author={Jianguo Zhang and Kun Qian and Zhiwei Liu and Shelby Heinecke and Rui Meng and Ye Liu and Zhou Yu and and Huan Wang and Silvio Savarese and Caiming Xiong},
4 year={2023},
5 eprint={2307.10172},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
