meta-llama.Meta-Llama-3-70B-Instruct_W8A8_FP8 – AI Model by espressor | AlphaNeural AI

You can deploy this model and start earning money today!

espressor/
meta-llama.Meta-Llama-3-70B-Instruct_W8A8_FP8

like

0

text-generation

HuggingFaceH4/ultrachat_200k

meta-llama/Meta-Llama-3-70B-Instruct

autotrain_compatible

text-generation-inference

endpoints_compatible

compressed-tensors

Views

No views yet

meta-llama/Meta-Llama-3-70B-Instruct - W8A8_FP8 Compression

This is a compressed model using llmcompressor.

Compression Configuration

Base Model: meta-llama/Meta-Llama-3-70B-Instruct
Compression Scheme: W8A8_FP8
Dataset: HuggingFaceH4/ultrachat_200k
Dataset Split: train_sft
Number of Samples: 512
Preprocessor: chat
Maximum Sequence Length: 8192

Sample Output

Prompt:

Could not generate output

Output:

CUDA error: unknown error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Evaluation