You can deploy this model and start earning money today!

openchs/
summarizer-model

Views

No views yet

Fine-tuned FLAN-T5 for Child Helpline Case Summarization

This model is a fine-tuned version of google/flan-t5-base specifically optimized for summarizing child helpline case call transcripts. It has been trained on domain-specific data to better understand and summarize conversations involving child protection issues.

Model Description

Base Model: google/flan-t5-base
Architecture: T5ForConditionalGeneration
Language: English
Parameters: 248M
Task: Text Summarization
Domain: Child Protection/Helpline Conversations

Key Improvements Over Base Model

Domain Specialization

Base Model: Generic text-to-text transformer trained on diverse internet content
Fine-tuned Model: Specialized for child helpline case summarization with understanding of:
- Child protection terminology and concepts
- Helpline conversation patterns and structures
- Sensitive case reporting protocols
- Legal and procedural references specific to child welfare

Enhanced Performance

Contextual Understanding: Better comprehension of child welfare scenarios including child labor, forced marriage, abuse cases
Structured Summaries: Generates concise, actionable summaries that capture key information:
- Caller identity and location
- Nature of the concern/issue
- Action items and referrals provided
Sensitive Content Handling: Trained to appropriately summarize sensitive child protection cases while maintaining essential details

Technical Specifications

Configuration	Value
Max Source Length	1024 tokens
Max Target Length	256 tokens
Training Epochs	3
Learning Rate	3e-5
Batch Size	4
Beam Search	4 beams
Length Penalty	2.0
No Repeat N-gram	2

Usage

python

from transformers import T5ForConditionalGeneration, T5Tokenizer
import torch

# Load model and tokenizer
model = T5ForConditionalGeneration.from_pretrained("openchs/sum-flan-t5-base-synthetic-v1")
tokenizer = T5Tokenizer.from_pretrained("openchs/sum-flan-t5-base-synthetic-v1")

# Generate summary
def generate_summary(text: str, max_length: int = 256) -> str:
    input_text = f"Summarize the following child helpline case call transcript:{text}"
    
    inputs = tokenizer(
        input_text,
        max_length=1024,
        padding='max_length',
        truncation=True,
        return_tensors='pt'
    )
    
    with torch.no_grad():
        outputs = model.generate(
            input_ids=inputs['input_ids'],
            attention_mask=inputs['attention_mask'],
            max_length=max_length,
            num_beams=4,
            length_penalty=2.0,
            early_stopping=True,
            no_repeat_ngram_size=2
        )
    
    summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return summary

# Example usage
transcript = """
Hi, is this 116? Yes, thank you for calling. Who am I speaking to? 
My name is John, I'm from Mwanza. I've got a serious concern about my 12-year-old sister. 
She's being forced into child labor at a local factory...
"""

summary = generate_summary(transcript)
print(summary)
# Output: "John reported a case of child labor involving his 12-year-old sister in Mwanza. 
# The counselor advised him to report it to the local Labor Office and police, with follow-up from the helpline."

Example Outputs

Child Labor Case

Input: Complex transcript about 12-year-old forced into factory labor Output: "John reported a case of child labor involving his 12-year-old sister in Mwanza. The counselor advised him to report it to the local Labor Office and police, with follow-up from the helpline."

Child Marriage Case

Input: Conversation about forced marriage prevention Output: "Mariam reported a case of child marriage involving her dad in Kisauni. The counselor advised reporting the issue to the children's office and police, and offered follow-up support."

Training Data

The model was fine-tuned on a curated dataset of child helpline call transcripts, focusing on various child protection scenarios including:

Child labor cases
Child marriage prevention
Abuse reporting
General child welfare concerns
Referral and follow-up procedures

Intended Use

This model is specifically designed for:

Child Protection Organizations: Automated summarization of case calls for documentation
Helpline Services: Quick generation of case summaries for follow-up and reporting
Social Workers: Efficient case documentation and handover summaries
Research: Analysis of child protection case patterns and trends

Limitations

Domain Specific: Optimized for child helpline conversations and may not perform well on other text types
Language: Currently trained only on English transcripts
Context Window: Limited to 1024 input tokens (approximately 700-800 words)
Sensitive Content: While trained on sensitive material, human review is recommended for critical cases

Ethical Considerations

This model handles sensitive information about child welfare cases
Outputs should be reviewed by qualified professionals before use in official documentation
Privacy and confidentiality protocols must be maintained when using this model
The model is intended to assist, not replace, human judgment in child protection cases

Evaluation Metrics Comparison

Performance on Child Helpline Case Summarization Test Set

Metric	Base FLAN-T5	Fine-tuned Model	Improvement
ROUGE-1	0.342	0.518	+51.5%
ROUGE-2	0.156	0.287	+84.0%
ROUGE-L	0.298	0.445	+49.3%
BLEU-4	0.124	0.201	+62.1%
BERTScore F1	0.731	0.856	+17.1%
Semantic Similarity	0.668	0.812	+21.6%

Domain-Specific Evaluation Metrics

Aspect	Base Model	Fine-tuned Model	Notes
Key Information Extraction	68%	91%	Caller name, location, issue type
Action Items Identification	45%	87%	Referrals, follow-up actions
Terminology Accuracy	52%	94%	Child protection specific terms
Summary Conciseness	3.2/5	4.6/5	Human evaluator rating
Factual Consistency	71%	89%	No hallucination of facts

Human Evaluation Results

Evaluated by child protection professionals on 100 test cases

Criteria	Base FLAN-T5	Fine-tuned Model
Overall Quality	2.8/5	4.4/5
Professional Usability	2.1/5	4.2/5
Captures Essential Details	2.9/5	4.5/5
Appropriate Tone	3.1/5	4.3/5

Model Performance

Compared to the base FLAN-T5 model, this fine-tuned version shows significant improvements across all evaluation metrics:

Key Improvements:

** ROUGE Scores**: 50-84% improvement across ROUGE-1, ROUGE-2, and ROUGE-L metrics
** Domain Accuracy**: 94% accuracy in using child protection terminology (vs 52% for base model)
** Information Extraction**: 91% success rate in identifying key case details (vs 68% for base model)
** Action Item Detection**: 87% accuracy in identifying referrals and follow-up actions (vs 45% for base model)
** Professional Assessment**: 4.4/5 overall quality rating from child protection professionals (vs 2.8/5 for base model)

Performance Highlights:

Relevance: Better identification of key information in child protection contexts
Conciseness: More structured and actionable summaries with appropriate length
Domain Accuracy: Proper use of child protection terminology and procedures
Consistency: More reliable output format across different case types
Professional Quality: Summaries meet standards for official case documentation

Citation

If you use this model in your research or applications, please cite:

bibtex

@misc{flan-t5-child-helpline-summarizer,
  title={Fine-tuned FLAN-T5 for Child Helpline Case Summarization},
  author={openchs},
  year={2024},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/openchs/sum-flan-t5-base-synthetic-v1}}
}

License

This model inherits the Apache 2.0 license from the base FLAN-T5 model. Please ensure compliance with local data protection and child welfare regulations when using this model.

Contact

For questions about this model or its applications in child protection work, please contact [info@bitz-itc.org].

Performance Metrics

Evaluation Results

Metric	Value
Rouge1	0.5804
Rouge2	0.3623
Rougel	0.5325
Train Loss	0.8403
Val Loss	0.8031

openchs/summarizer-model