adityatandon94/

Llama-3.2-1B-peft-medical-specialty-classification

Model Card for Model ID: adityatandon94/Llama-3.2-1B-peft-medical-specialty-classification Model Details Model Description This is a fine-tuned version of the LLaMA-3.2-1B model, developed by Aditya Tandon, for medical specialty classification tasks. The model has been fine-tuned using domain-specific medical data to classify text related to various medical specialties.

Developed by: Aditya Tandon

Shared by: Aditya Tandon (Hugging Face: adityatandon94)

Model type: Transformer-based language model (LLaMA 3.2-1B)

Language(s) (NLP): English

License: Open-source (Details to be clarified)

Finetuned from model: LLaMA-3.2-1B (Meta)

Model Sources Repository: Hugging Face Model Page

Paper: LLaMA-3.2-1B Paper Link

Demo: [More Information Needed]

Uses Direct Use This model can be used directly for classifying medical texts into specific specialties, such as cardiology, dermatology, etc. It outputs the most likely medical specialty based on the input text.

Downstream Use When fine-tuned on additional domain-specific datasets, this model can be employed in healthcare applications, including electronic health record (EHR) classification, medical document categorization, or automating the triage of medical queries based on specialty.

Out-of-Scope Use The model is specialized for medical specialty classification and may not perform well in other domains. It is not intended for use in real-time decision-making systems or other high-stakes scenarios without human oversight.

Bias, Risks, and Limitations This model may reflect biases present in the training data, which could impact its predictions in sensitive medical contexts. It is essential to evaluate the model's output carefully and ensure it is used under appropriate guidance in clinical settings.

Recommendations Users should validate the model’s outputs for bias and limitations, especially in medical applications. Fine-tuning the model with diverse, high-quality medical datasets can help mitigate some of these issues. Human oversight is recommended for critical decisions.

How to Get Started with the Model To get started with the model, use the following code snippet:

Training Details Training Data The model was fine-tuned on medical text data related to various specialties. The dataset includes categorized text related to cardiology, dermatology, and other medical fields.

Training Procedure The model was trained with standard NLP training procedures using PyTorch and Hugging Face Transformers library. The training utilized mixed-precision training on a multi-GPU setup to improve efficiency.

Preprocessing Tokenization and padding were performed on the input text data. Special tokens were added to handle medical terminology and relevant context.

Training Hyperparameters Training regime: Mixed precision (fp16) for efficiency on multi-GPU setups.

Batch size: [More Information Needed]

Learning rate: [More Information Needed]

Epochs: [More Information Needed]

Speeds, Sizes, Times Training times varied depending on the hardware setup. The use of mixed-precision training helped to accelerate the process.

Evaluation Testing Data, Factors & Metrics The evaluation was done using a separate test set of medical text data to validate the model’s ability to classify text into relevant medical specialties.

Factors: Medical domain-specific language, term frequency, and specialty-specific context.

Metrics: Accuracy, F1-Score, Precision, Recall.

Results The model achieved high accuracy in classifying text into medical specialties. Further testing is necessary to assess its performance in real-world applications.

Summary This model has demonstrated strong performance on medical specialty classification tasks. It is suitable for integration into medical classification systems and could significantly aid in categorizing healthcare-related documents.

Model Examination The model has been examined for interpretability and bias. Techniques such as LIME or SHAP could be applied to increase transparency and provide insights into how the model makes its predictions.

Environmental Impact The environmental impact of training this model is significant due to the multi-GPU infrastructure used. Further estimates on CO2 emissions and energy consumption can be calculated using tools like the ML CO2 Impact calculator.

Hardware Type: Multi-GPU setup (8x RTX 4060 Ti GPUs)

Hours used: [More Information Needed]

Cloud Provider: [More Information Needed]

Compute Region: [More Information Needed]

Carbon Emitted: [More Information Needed]

Technical Specifications Model Architecture and Objective The model follows the LLaMA architecture, with optimization for text classification. It was fine-tuned specifically for medical specialty classification tasks.

Compute Infrastructure The model was trained on an 8x RTX 4060 Ti multi-GPU setup, allowing for efficient large-scale training.

Hardware GPU: 8x RTX 4060 Ti (16GB VRAM each)

CPU: [More Information Needed]

RAM: [More Information Needed]

Software Framework: PyTorch 2.6.0 with CUDA 12.4

Library: Hugging Face Transformers

Citation If you use this model in your research or applications, please cite it using the following formats:

BibTeX:

Glossary LLaMA: A transformer-based language model.

PEFT: Parameter Efficient Fine-Tuning, a technique used to fine-tune models more efficiently.

More Information For additional details, visit the GitHub repository and check out the LinkedIn Profile.

Model Card Authors Aditya Tandon: Creator and main contributor.

Model Card Contact For inquiries, please reach out to Aditya Tandon via LinkedIn.

CPU: [More Information Needed]

RAM: [More Information Needed]

Software Framework: PyTorch 2.6.0 with CUDA 12.4

Library: Hugging Face Transformers

Citation If you use this model in your research or applications, please cite it using the following formats: