base_model: EleutherAI/pythia-70m-deduped library_name: peft
Model Card for Fin-Interpret
This model is a fine-tuned version of EleutherAI's Pythia-70m-deduped, specifically adapted to understand and interpret text from financial documents. It leverages Parameter-Efficient Fine-Tuning (PEFT) to achieve this specialization efficiently.
Model Details
Model Description
A financial model trained to interprete financial documents. This model is built upon the EleutherAI/pythia-70m-deduped base model and fine-tuned using the PEFT library. The goal is to provide a language model that is adept at tasks related to financial text analysis, such as summarization, question answering, and information extraction from financial reports, filings, and news articles.
- Training datasets: The model was trained on the Synthik Finance Narrative dataset, which contains a collection of financial documents. You can find more information about the dataset at https://www.synthik.io/datasets/36db6f48-0e63-43b5-a9d8-0252b8137835.
- Funded by [Synthik]: [More Information Needed]
- Shared by [optional]: [More Information Needed]
- Model type: Causal Language Model
- Language(s) (NLP): English
- License: Apache-2.0
- Finetuned from model: EleutherAI/pythia-70m-deduped
Model Sources [optional]
- Repository: [More Information Needed]
- Paper [optional]: [More Information Needed]
- Demo [optional]: [More Information Needed]
Uses
Direct Use
This model is intended for direct use in generating text-based responses to prompts related to financial documents. It can be used for tasks such as:
- Answering questions about a company's financial performance based on an annual report.
- Summarizing key points from a financial news article.
- Extracting specific financial metrics from a given text.
Downstream Use [optional]
The model can be further fine-tuned on more specific financial NLP tasks, such as:
- Sentiment analysis of financial news.
- Named entity recognition for financial entities.
- Classification of financial documents.
Out-of-Scope Use
This model is not a financial advisor and should not be used for making investment decisions. The outputs are generated based on patterns in the training data and may not be factually accurate. The model is not intended for use in high-stakes applications where incorrect information could lead to financial loss. Use in any of the following ways is out-of-scope:
- Generating financial advice or recommendations.
- Predicting stock market movements.
- Automated trading.
Bias, Risks, and Limitations
The model's knowledge is limited to the information present in the Synthik Finance Narrative dataset and the underlying Pythia-70m-deduped model. Therefore, it may exhibit biases present in the training data. The model may generate plausible-sounding but incorrect or misleading information. It is crucial to verify any factual claims made by the model with reliable financial sources.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. It is recommended to have a human-in-the-loop to review the model's outputs before they are used in any decision-making process.
How to Get Started with the Model
Use the code below to get started with the model. Make sure you have the transformers and peft libraries installed.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3
4# Load the base model and tokenizer
5base_model_id = "EleutherAI/pythia-70m-deduped"
6model = AutoModelForCausalLM.from_pretrained(base_model_id)
7tokenizer = AutoTokenizer.from_pretrained(base_model_id)
8
9# Load the PEFT model
10peft_model_id = "YOUR_MODEL_ID" # Replace with the actual model ID
11model = PeftModel.from_pretrained(model, peft_model_id)
12
13# Prepare your input
14prompt = "What were the key financial highlights from the attached report?"
15inputs = tokenizer(prompt, return_tensors="pt")
16
17# Generate a response
18outputs = model.generate(**inputs)
19print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Details
Training Data
The model was fine-tuned on the Synthik Finance Narrative dataset. This dataset is comprised of various financial documents and is designed to train models for financial text interpretation tasks.
Training Procedure
The model was fine-tuned using the Parameter-Efficient Fine-Tuning (PEFT) library, which significantly reduces the computational and storage costs associated with training large language models. This approach involves freezing the majority of the base model's parameters and only training a small number of additional parameters.
Preprocessing [optional]
[More Information Needed]
Training Hyperparameters
- Training regime: [More Information Needed]
Speeds, Sizes, Times [optional]
[More Information Needed]
Evaluation
Testing Data, Factors & Metrics
Testing Data
[More Information Needed]
Factors
[More Information Needed]
Metrics
[More Information Needed]
Results
[More Information Needed]
Summary
[More Information Needed]
Model Examination [optional]
[More Information Needed]
Environmental Impact
Carbon emissions can be estimated using the
Machine Learning Impact calculator presented in
Lacoste et al. (2019).
- Hardware Type: [More Information Needed]
- Hours used: [More Information Needed]
- Cloud Provider: [More Information Needed]
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Technical Specifications [optional]
Model Architecture and Objective
The model uses the same architecture as the EleutherAI/pythia-70m-deduped model, which is a GPT-NeoX-based transformer model. The objective of the fine-tuning was to minimize the causal language modeling loss on the financial document dataset.
Compute Infrastructure
Hardware
[More Information Needed]
Software
The fine-tuning was performed using the Hugging Face transformers and peft libraries.
Glossary [optional]
- PEFT (Parameter-Efficient Fine-Tuning): A set of techniques to adapt large pre-trained models to new tasks by fine-tuning only a small fraction of their parameters.
- Causal Language Model: A type of language model that predicts the next token in a sequence given the preceding tokens.
More Information [optional]
[More Information Needed]
Model Card Authors [optional]
[More Information Needed]
Model Card Contact
[More Information Needed]
Framework versions
