You can deploy this model and start earning money today!

sadisylarq/
gemma-2-9b-custom

Name: gemma-2-9b-custom
Availability: InStock

endpoints_compatible

Views

No views yet

Gemma-2-9b Custom Handler

Overview

This repository contains a custom handler for deploying the google/gemma-2-9b model as a HuggingFace Inference Endpoint.

Model Information

Base Model: google/gemma-2-9b
Task: Text Generation
Purpose: Medical diagnosis generation for the Dx²⁴ AI disease diagnosis system
Use Case: Primary diagnosis model for generating medical assessments

Handler Features

Automatic Device Detection: Uses GPU if available, falls back to CPU
Optimized Performance: Uses bfloat16 precision on GPU for faster inference
Flexible Parameters: Customizable generation parameters (temperature, top_p, max_tokens)
Medical-Optimized Defaults: Pre-configured parameters for medical text generation
Error Handling: Graceful error handling with informative messages
Memory Efficient: Uses low CPU memory usage mode

Input Format

Basic input:

json

{
  "inputs": "Patient presents with fever and cough. What are potential diagnoses?"
}

With custom parameters:

json

{
  "inputs": "Patient presents with fever and cough. What are potential diagnoses?",
  "parameters": {
    "max_new_tokens": 512,
    "temperature": 0.7,
    "top_p": 0.9,
    "do_sample": true,
    "repetition_penalty": 1.1
  }
}

Output Format

json

[
  {
    "generated_text": "Based on the symptoms of fever and cough, potential diagnoses include..."
  }
]

Default Parameters

max_new_tokens: 512 (maximum tokens to generate)
temperature: 0.7 (controls randomness, lower = more deterministic)
top_p: 0.9 (nucleus sampling threshold)
do_sample: true (enables sampling for varied outputs)
repetition_penalty: 1.1 (prevents repetitive text)

Deployment Instructions

Go to HuggingFace Inference Endpoints
Click "+ New Endpoint"
Configure:
- Model Repository: sadisylarq/gemma-2-9b-custom
- Endpoint Name: dx24-gemma-2-9b
- Region: us-east-1
- Instance Type: nvidia-a10g (recommended) or nvidia-l4
- Min Replica: 0
- Max Replica: 1
- Scale to Zero Timeout: 1800 seconds
Click "Create Endpoint"

Note: Gemma-2-9b requires more powerful GPUs. The nvidia-a10g instance is recommended for optimal performance.

Testing

Once deployed, test with:

python

import requests

API_URL = "https://your-endpoint-url.aws.endpoints.huggingface.cloud"
headers = {"Authorization": "Bearer YOUR_HF_TOKEN"}

data = {
    "inputs": "Patient presents with fever and cough. Provide diagnosis.",
    "parameters": {
        "max_new_tokens": 256,
        "temperature": 0.7
    }
}

response = requests.post(API_URL, headers=headers, json=data)
print(response.json())

Requirements

transformers>=4.30.0
torch>=2.0.0
GPU with at least 16GB VRAM recommended (A10G or better)

Performance Notes

First request may take 30-60 seconds as the model loads
Subsequent requests are much faster (1-5 seconds depending on output length)
Scale-to-zero saves costs during idle periods

License

Follows the license of the base model: google/gemma-2-9b

sadisylarq/gemma-2-9b-custom

Gemma-2-9b Custom Handler

Overview

Model Information

Handler Features

Input Format

Output Format

Default Parameters

Deployment Instructions

Testing

Requirements

Performance Notes

License

sadisylarq/
gemma-2-9b-custom