You can deploy this model and start earning money today!

RichardErkhov/
ssmits_-_Falcon2-5.5B-Polish-gguf

Name: ssmits_-_Falcon2-5.5B-Polish-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Falcon2-5.5B-Polish - GGUF

Model creator: https://huggingface.co/ssmits/
Original model: https://huggingface.co/ssmits/Falcon2-5.5B-Polish/

Name	Quant method	Size
Falcon2-5.5B-Polish.Q2_K.gguf	Q2_K	2.03GB
Falcon2-5.5B-Polish.IQ3_XS.gguf	IQ3_XS	2.29GB
Falcon2-5.5B-Polish.IQ3_S.gguf	IQ3_S	2.35GB
Falcon2-5.5B-Polish.Q3_K_S.gguf	Q3_K_S	2.35GB
Falcon2-5.5B-Polish.IQ3_M.gguf	IQ3_M	2.46GB
Falcon2-5.5B-Polish.Q3_K.gguf	Q3_K	2.56GB
Falcon2-5.5B-Polish.Q3_K_M.gguf	Q3_K_M	2.56GB
Falcon2-5.5B-Polish.Q3_K_L.gguf	Q3_K_L	2.72GB
Falcon2-5.5B-Polish.IQ4_XS.gguf	IQ4_XS	2.87GB
Falcon2-5.5B-Polish.Q4_0.gguf	Q4_0	2.99GB
Falcon2-5.5B-Polish.IQ4_NL.gguf	IQ4_NL	3.01GB
Falcon2-5.5B-Polish.Q4_K_S.gguf	Q4_K_S	2.99GB
Falcon2-5.5B-Polish.Q4_K.gguf	Q4_K	3.19GB
Falcon2-5.5B-Polish.Q4_K_M.gguf	Q4_K_M	3.19GB
Falcon2-5.5B-Polish.Q4_1.gguf	Q4_1	3.29GB
Falcon2-5.5B-Polish.Q5_0.gguf	Q5_0	3.6GB
Falcon2-5.5B-Polish.Q5_K_S.gguf	Q5_K_S	3.6GB
Falcon2-5.5B-Polish.Q5_K.gguf	Q5_K	3.8GB
Falcon2-5.5B-Polish.Q5_K_M.gguf	Q5_K_M	3.8GB
Falcon2-5.5B-Polish.Q5_1.gguf	Q5_1	3.9GB
Falcon2-5.5B-Polish.Q6_K.gguf	Q6_K	4.24GB
Falcon2-5.5B-Polish.Q8_0.gguf	Q8_0	5.41GB

Original model description:

base_model:

tiiuae/falcon-11B library_name: transformers tags:
mergekit
merge
lazymergekit
tiiuae/falcon-11B license: apache-2.0 language:
'pl'

Why prune?

Even though Falcon-11B is trained on 5T tokens, it is still undertrained, as can be seen by this graph:

This is why the choice is made to prune 50% of the layers. Note that ~1B of continued pre-training (~1M rows of 1k tokens) is still required to restore the perplexity of this model in the desired language. I'm planning on doing that for certain languages, depending on how much compute will be available.

sliced

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was pruned using the passthrough merge method.

Models Merged

The following models were included in the merge:

tiiuae/falcon-11B

Configuration

The following YAML configuration was used to produce this model:

yaml

slices:
  - sources:
      - model: tiiuae/falcon-11B
        layer_range: [0, 24]
  - sources:
      - model: tiiuae/falcon-11B
        layer_range: [55, 59]
merge_method: passthrough
dtype: bfloat16

PruneMe has been utilized using the wikimedia/wikipedia Polish (pl) subset by investigating layer similarity with 2000 samples. The layer ranges for pruning were determined based on this analysis to maintain performance while reducing model size.

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "ssmits/Falcon2-5.5B-Polish"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
)
sequences = pipeline(
   "Can you explain the concepts of Quantum Computing?",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

💥 Falcon LLMs require PyTorch 2.0 for use with transformers!

For fast inference with Falcon, check-out Text Generation Inference! Read more in this [blogpost]((https://huggingface.co/blog/falcon).

Direct Use

Research on large language models; as a foundation for further specialization and finetuning for specific usecases (e.g., summarization, text generation, chatbot, etc.)

Out-of-Scope Use

Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful.

Bias, Risks, and Limitations

Falcon2-5.5B is trained mostly on English, but also German, Spanish, French, Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish. It will not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it will carry the stereotypes and biases commonly encountered online.

Recommendations

We recommend users of Falcon2-5.5B to consider finetuning it for the specific set of tasks of interest, and for guardrails and appropriate precautions to be taken for any production use.

RichardErkhov/ssmits_-_Falcon2-5.5B-Polish-gguf