You can deploy this model and start earning money today!

Lambent/
Falcon3-Continued-0.3-10B-Base

Name: Falcon3-Continued-0.3-10B-Base
Availability: InStock

safetensors

llama

PleIAs/common_corpus

tiiuae/Falcon3-10B-Base

finetune

other

Views

No views yet

Falcon3-Continued-0.3-10B-Base is built using artificial intelligence technology from the Technology Innovation Institute.

This model uses qLoRA with UnSloth to continuously pretrain Falcon3-10B-Base on an additional 30,720 rows from PleIAs/common_corpus, cyclically.

Rows trained at a time varied between 2048, 4096, and 8192, using cosine decay. A merged model was saved and tested every 10240 rows.

Adapters ranged from rank 32 to rank 128, with ranks 64 and 128 being the most common. Weight decay was 0.01.

Trained context length ranged from 4096 to the full 32678, with 32678 being the most common. Sample packing was not used. Long documents, if present, were truncated.

Training continued until no improvement in eq_bench was demonstrated from this method. Most other benchmarks stayed similar.

hf (pretrained=Lambent/Falcon3-Continued-0.3-10B-Base,dtype=auto,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
eq_bench	2.1	none	0	eqbench	↑	64.2105	±	2.1413
		none	0	percent_parseable	↑	100.0000	±	0.0000

hf (pretrained=Lambent/Falcon3-Continued-0.3-10B-Base), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:4

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.8105	±	0.0108
		strict-match	5	exact_match	↑	0.8036	±	0.0109

hf (pretrained=Lambent/Falcon3-Continued-0.3-10B-Base), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:4 (4,64,64,64)

Tasks	Version	Filter	Metric		Value		Stderr
arc_challenge	1	none	acc	↑	0.5401	±	0.0146
		none	acc_norm	↑	0.5648	±	0.0145
piqa	1	none	acc	↑	0.7873	±	0.0095
		none	acc_norm	↑	0.7954	±	0.0094
sciq	1	none	acc	↑	0.9620	±	0.0060
		none	acc_norm	↑	0.9500	±	0.0069
winogrande	1	none	acc	↑	0.7332	±	0.0124

MuSR:

RUNNING | Lambent/Falcon3-Continued-0.3-10B-Base | murder mysteries | cot+ | 145 / 250 | 58.0 RUNNING | Lambent/Falcon3-Continued-0.3-10B-Base | object placements | cot+ | 83 / 256 | 32.4 RUNNING | Lambent/Falcon3-Continued-0.3-10B-Base | team allocation | cot+ | 112 / 250 | 44.8

Original under same conditions:

hf (pretrained=tiiuae/Falcon3-10B-Base,dtype=auto,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
eq_bench	2.1	none	0	eqbench	↑	60.9913	±	2.2402
		none	0	percent_parseable	↑	100.0000	±	0.0000

hf (pretrained=tiiuae/Falcon3-10B-Base), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:4

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.8188	±	0.0106
		strict-match	5	exact_match	↑	0.8105	±	0.0108

hf (pretrained=tiiuae/Falcon3-10B-Base), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:4 (4,64,64,64)

Tasks	Version	Filter	Metric		Value		Stderr
arc_challenge	1	none	acc	↑	0.5520	±	0.0145
		none	acc_norm	↑	0.5887	±	0.0144
piqa	1	none	acc	↑	0.7873	±	0.0095
		none	acc_norm	↑	0.7949	±	0.0094
sciq	1	none	acc	↑	0.9610	±	0.0061
		none	acc_norm	↑	0.9360	±	0.0077
winogrande	1	none	acc	↑	0.7364	±	0.0124

MuSR:

RUNNING | tiiuae/Falcon3-10B-Base | murder mysteries | cot+ | 140 / 250 | 56.0 RUNNING | tiiuae/Falcon3-10B-Base | object placements | cot+ | 139 / 256 | 54.3 RUNNING | tiiuae/Falcon3-10B-Base | team allocation | cot+ | 118 / 250 | 47.2