You can deploy this model and start earning money today!

steampunque/
madlad400-10b-mt-MP-GGUF

Name: madlad400-10b-mt-MP-GGUF
Availability: InStock

google/madlad400-10b-mt

quantized

apache-2.0

endpoints_compatible

Views

No views yet

Mixed Precision GGUF layer quantization of madlad400 10b mt by google

Original model: https://huggingface.co/google/madlad400-10b-mt

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. This particular quant achieves a ~7.8G gguf for the 10.71B parameter translation model . The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. For this file the layer quants are as follows:

Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K
Q6_K_M : attn_v = q8_0 ffn_d = q8_0

   LAYER_TYPES_ENC='[
   [0 ,"Q6_K_S"],[1 ,"Q5_K_L"],[2 ,"Q5_K_M"],[3 ,"Q5_K_S"],[4 ,"Q5_K_S"],[5 ,"Q5_K_S"],[6 ,"Q5_K_S"],[7 ,"Q5_K_S"],
   [8 ,"Q5_K_S"],[9 ,"Q5_K_S"],[10,"Q5_K_S"],[11,"Q5_K_S"],[12,"Q5_K_S"],[13,"Q5_K_S"],[14,"Q5_K_S"],[15,"Q5_K_S"],
   [16,"Q5_K_S"],[17,"Q5_K_S"],[18,"Q5_K_S"],[19,"Q5_K_S"],[20,"Q5_K_S"],[21,"Q5_K_S"],[22,"Q5_K_S"],[23,"Q5_K_S"],
   [24,"Q5_K_M"],[25,"Q5_K_M"],[26,"Q5_K_M"],[27,"Q5_K_M"],[28,"Q5_K_M"],[29,"Q5_K_L"],[30,"Q6_K_S"],[31,"Q6_K_M"]
   ]'

   LAYER_TYPES='[
   [0 ,"Q6_K_S"],[1 ,"Q5_K_L"],[2 ,"Q5_K_M"],[3 ,"Q5_K_S"],[4 ,"Q5_K_S"],[5 ,"Q5_K_S"],[6 ,"Q5_K_S"],[7 ,"Q5_K_S"],
   [8 ,"Q5_K_S"],[9 ,"Q5_K_S"],[10,"Q5_K_S"],[11,"Q5_K_S"],[12,"Q5_K_S"],[13,"Q5_K_S"],[14,"Q5_K_S"],[15,"Q5_K_S"],
   [16,"Q5_K_S"],[17,"Q5_K_S"],[18,"Q5_K_S"],[19,"Q5_K_S"],[20,"Q5_K_S"],[21,"Q5_K_S"],[22,"Q5_K_S"],[23,"Q5_K_S"],
   [24,"Q5_K_M"],[25,"Q5_K_M"],[26,"Q5_K_M"],[27,"Q5_K_M"],[28,"Q5_K_M"],[29,"Q5_K_L"],[30,"Q6_K_S"],[31,"Q6_K_M"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high --layer-types-enc-high"

The layer quants were optimized to enable the model to fit into 8G VRAM GPUs with room left over for some context while using a minimum layer quant of Q5_K_S across all layers for high performance.

A second smaller Q4_K_H quant is also available. This quant is intended to be able to run fully offloaded on 6G VRAM GPUs while still having good performance. For this file the encoder was quantized to less precision than the decoder to save space:

Q4_K_L : Q4_K_L : Q4_K_M + attn_o = q6_k
Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K

   LAYER_TYPES_ENC='[
   [0 ,"Q4_K_S"],[1 ,"Q3_K_L"],[2 ,"Q3_K_M"],[3 ,"Q3_K_M"],[4 ,"Q3_K_M"],[5 ,"Q3_K_M"],[6 ,"Q3_K_M"],[7 ,"Q3_K_M"],
   [8 ,"Q3_K_M"],[9 ,"Q3_K_M"],[10,"Q3_K_M"],[11,"Q3_K_M"],[12,"Q3_K_M"],[13,"Q3_K_M"],[14,"Q3_K_M"],[15,"Q3_K_M"],
   [16,"Q3_K_L"],[17,"Q3_K_M"],[18,"Q3_K_L"],[19,"Q3_K_M"],[20,"Q3_K_L"],[21,"Q3_K_M"],[22,"Q3_K_L"],[23,"Q3_K_M"],
   [24,"Q3_K_L"],[25,"Q4_K_L"],[26,"Q4_K_S"],[27,"Q4_K_S"],[28,"Q4_K_S"],[29,"Q4_K_S"],[30,"Q4_K_S"],[31,"Q4_K_M"]
   ]'

   LAYER_TYPES='[
   [0 ,"Q4_K_M"],[1 ,"Q4_K_S"],[2 ,"Q3_K_L"],[3 ,"Q3_K_L"],[4 ,"Q3_K_L"],[5 ,"Q3_K_L"],[6 ,"Q3_K_L"],[7 ,"Q3_K_L"],
   [8 ,"Q4_K_S"],[9 ,"Q3_K_L"],[10,"Q4_K_S"],[11,"Q3_K_L"],[12,"Q4_K_S"],[13,"Q3_K_L"],[14,"Q4_K_S"],[15,"Q3_K_L"],
   [16,"Q4_K_S"],[17,"Q3_K_L"],[18,"Q4_K_S"],[19,"Q4_K_S"],[20,"Q4_K_S"],[21,"Q4_K_S"],[22,"Q4_K_S"],[23,"Q4_K_S"],
   [24,"Q4_K_S"],[25,"Q4_K_S"],[26,"Q4_K_M"],[27,"Q4_K_L"],[28,"Q5_K_S"],[29,"Q5_K_M"],[30,"Q5_K_L"],[31,"Q6_K_S"]
   ]'
   FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K --layer-types-high --layer-types-enc-high"

Comparison:

Quant	size	Comment
Q6_K	8.8e9	Q6_K with default embedding and output
Q6_K_H	7.8e9	Hybrid quant with Q6_K embedding Q6_K output
Q4_K_H	6e9	Hybrid quant with Q4_K embedding Q6_K output

Usage:

This model is a LangID machine language translation (MT) model based on the google T5 encoder-decoder architecture. Its prompt format is as follows:

<2TARGET>Hello, how are you?

where TARGET is a language code from the following set of 418 languages:

langs=(
    "ace" "Achinese"
    "ach" "Acoli"
    "ada" "Adangme"
    "adh" "Adhola"
    "ady" "Adyghe"
    "aa"  "Afar"
    "af"  "Afrikaans"
    "agr" "Aguaruna"
    "msm" "Agusan Manobo"
    "ahk" "Akha"
    "sq"  "Albanian"
    "alz" "Alur"
    "abt" "Ambulas"
    "am"  "Amharic"
    "grc" "Ancient Greek"
    "ar"  "Arabic"
    "hy"  "Armenian"
    "frp" "Arpitan"
    "cni" "Ashninka"
    "as"  "Assamese"
    "av"  "Avar"
    "kwi" "Awa-Cuaiquer"
    "awa" "Awadhi"
    "quy" "Ayacucho Quechua"
    "ay"  "Aymara"
    "az"  "Azerbaijani"
    "ban" "Balinese"
    "bm"  "Bambara"
    "bgz" "Banggai"
    "bci" "Baoul"
    "bas" "Basa (Cameroon)"
    "ba"  "Bashkir"
    "eu"  "Basque"
    "akb" "Batak Angkola"
    "bbc" "Batak Toba"
    "bts" "Batak Simalungun"
    "btx" "Batak Karo"
    "be"  "Belarusian"
    "bzj" "Belize Kriol English"
    "bn"  "Bengali"
    "bew" "Betawi"
    "bho" "Bhojpuri"
    "bim" "Bimoba"
    "bi"  "Bislama"
    "brx" "Bodo (India)"
    "bqc" "Boko (Benin)"
    "bus" "Bokobaru"
    "bs"  "Bosnian"
    "br"  "Breton"
    "ape" "Bukiyip"
    "bg"  "Bulgarian"
    "bum" "Bulu"
    "bua" "Buryat"
    "qxr" "Caar H. Quichua"
    "qvc" "Cajamarca Quechua"
    "yue" "Cantonese"
    "jvn" "Caribbean Javanese"
    "rmc" "Carpathian Romani"
    "ca"  "Catalan"
    "ceb" "Cebuano"
    "bik" "Central Bikol"
    "maz" "Central Mazahua"
    "mel" "Central Melanau"
    "ch"  "Chamorro"
    "cbk" "Chavacano"
    "ce"  "Chechen"
    "chr" "Cherokee"
    "hne" "Chhattisgarhi"
    "ny"  "Chichewa"
    "cjk" "Chokwe"
    "ctu" "Chol"
    "cce" "Chopi"
    "cac" "Chuj"
    "chk" "Chuukese"
    "cv"  "Chuvash"
    "kw"  "Cornish"
    "co"  "Corsican"
    "crh" "Crimean Tatar"
    "hr"  "Croatian"
    "cs"  "Czech"
    "mps" "Dadibi"
    "da"  "Danish"
    "dwr" "Dawro"
    "dv"  "Dhivehi"
    "din" "Dinka"
    "tbz" "Ditammari"
    "dov" "Dombe"
    "nl"  "Dutch"
    "dyu" "Dyula"
    "dz"  "Dzongkha"
    "bgp" "E. Baluchi"
    "bru" "E. Bru"
    "djk" "E. Maroon Creole"
    "gui" "E. Bolivian Guaran"
    "nhe" "E. Huasteca Nahuatl"
    "taj" "E. Tamang"
    "enq" "Enga"
    "en"  "English"
    "sja" "Epena"
    "myv" "Erzya"
    "eo"  "Esperanto"
    "et"  "Estonian"
    "ee"  "Ewe"
    "cfm" "Falam Chin"
    "fo"  "Faroese"
    "hif" "Fiji Hindi"
    "fj"  "Fijian"
    "fil" "Filipino"
    "fi"  "Finnish"
    "fip" "Fipa"
    "fon" "Fon"
    "fr"  "French"
    "ff"  "Fulfulde"
    "gag" "Gagauz"
    "gl"  "Galician"
    "gbm" "Garhwali"
    "cab" "Garifuna"
    "ka"  "Georgian"
    "de"  "German"
    "gil" "Gilbertese"
    "gom" "Goan Konkani"
    "gof" "Gofa"
    "gor" "Gorontalo"
    "el"  "Greek"
    "guh" "Guahibo"
    "gub" "Guajajra"
    "gn"  "Guarani"
    "amu" "Guerrero Amuzgo"
    "ngu" "Guerrero Nahuatl"
    "gu"  "Gujarati"
    "gvl" "Gulay"
    "gyn" "Guyanese"
    "ht"  "Haitian Creole"
    "cnh" "Hakha Chin"
    "mey" "Hassaniyya"
    "ha"  "Hausa"
    "haw" "Hawaiian"
    "he"  "Hebrew"
    "hil" "Hiligaynon"
    "mrj" "Hill Mari"
    "hi"  "Hindi"
    "ho"  "Hiri Motu"
    "hmn" "Hmong"
    "qub" "Huallaga Hunuco Quechua"
    "hus" "Huastec"
    "hui" "Huli"
    "hu"  "Hungarian"
    "iba" "Iban"
    "ibb" "Ibibio"
    "is"  "Icelandic"
    "ig"  "Igbo"
    "ilo" "Ilocano"
    "qvi" "Imbabura H. Quichua"
    "id"  "Indonesian"
    "inb" "Inga"
    "iu"  "Inuktitut"
    "ga"  "Irish"
    "iso" "Isoko"
    "it"  "Italian"
    "ium" "Iu Mien"
    "izz" "Izii"
    "jam" "Jamaican Creole English"
    "jax" "Jambi Malay"
    "ja"  "Japanese"
    "jv"  "Javanese"
    "kbd" "Kabardian"
    "kbp" "Kabiy"
    "kac" "Kachin"
    "dtp" "Kadazan Dusun"
    "kl"  "Kalaallisut"
    "xal" "Kalmyk"
    "bjj" "Kanauji"
    "kn"  "Kannada"
    "cak" "Kaqchikel"
    "kaa" "Kara-Kalpak"
    "kaa-Latn" "Kara-Kalpak (Latn)"
    "krc" "Karachay-Balkar"
    "ks"  "Kashmiri"
    "kk"  "Kazakh"
    "meo" "Kedah Malay"
    "kek" "Kekch"
    "ify" "Keley-I Kallahan"
    "kjh" "Khakas"
    "kha" "Khasi"
    "km"  "Khmer"
    "kjg" "Khmu"
    "quc" "Kiche"
    "kmb" "Kimbundu"
    "krj" "Kinaray-A"
    "rw"  "Kinyarwanda"
    "ktu" "Kituba (DRC)"
    "tlh" "Klingon"
    "trp" "Kok Borok"
    "kv"  "Komi"
    "koi" "Komi-Permyak"
    "kg"  "Kongo"
    "ko"  "Korean"
    "kos" "Kosraean"
    "kri" "Krio"
    "ksd" "Kuanua"
    "kj"  "Kuanyama"
    "kum" "Kumyk"
    "mkn" "Kupang Malay"
    "ckb" "Kurdish (Sorani)"
    "ku"  "Kurdish (Kurmanji)"
    "ky"  "Kyrghyz"
    "lhu" "Lahu"
    "quf" "Lambayeque Quechua"
    "laj" "Lango (Uganda)"
    "lo"  "Lao"
    "ltg" "Latgalian"
    "la"  "Latin"
    "lv"  "Latvian"
    "ln"  "Lingala"
    "lt"  "Lithuanian"
    "lu"  "Luba-Katanga"
    "lg"  "Luganda"
    "lb"  "Luxembourgish"
    "ffm" "Maasina Fulfulde"
    "mk"  "Macedonian"
    "mad" "Madurese"
    "mag" "Magahi"
    "mai" "Maithili"
    "mak" "Makasar"
    "mgh" "Makhuwa-Meetto"
    "mg"  "Malagasy"
    "ms"  "Malay"
    "ml"  "Malayalam"
    "mt"  "Maltese"
    "mam" "Mam"
    "xmm" "Manado Malay"
    "zh"  "Mandarin Chinese"
    "mqy" "Manggarai"
    "gv"  "Manx"
    "mi"  "Maori"
    "arn" "Mapudungun"
    "mrw" "Maranao"
    "mr"  "Marathi"
    "mh"  "Marshallese"
    "mas" "Masai"
    "msb" "Masbatenyo"
    "mbt" "Matigsalug Manobo"
    "chm" "Meadow Mari"
    "mni" "Meiteilon (Manipuri)"
    "mtr" "Mewari"
    "min" "Minangkabau"
    "lus" "Mizo"
    "mdf" "Moksha"
    "mnw" "Mon"
    "mn"  "Mongolian"
    "mfe" "Morisien"
    "meu" "Motu"
    "miq" "Mskito"
    "tuc" "Mutu"
    "my"  "Myanmar (Burmese)"
    "emp" "N. Ember"
    "lrc" "N. Luri"
    "qvz" "N. Pastaza Quichua"
    "se"  "N. Sami"
    "nnb" "Nande"
    "niq" "Nandi"
    "nv"  "Navajo"
    "ne"  "Nepali"
    "new" "Newari"
    "nij" "Ngaju"
    "gym" "Ngbere"
    "nia" "Nias"
    "noe" "Nimadi"
    "nog" "Nogai"
    "zxx" "Noise"
    "no"  "Norwegian"
    "nut" "Nung (Viet Nam)"
    "nyu" "Nyungwe"
    "nzi" "Nzima"
    "ann" "Obolo"
    "oc"  "Occitan"
    "or"  "Odia (Oriya)"
    "oj"  "Ojibwa"
    "ang" "Old English"
    "om"  "Oromo"
    "os"  "Ossetian"
    "pck" "Paite Chin"
    "pau" "Palauan"
    "pam" "Pampanga"
    "pag" "Pangasinan"
    "pap" "Papiamento"
    "prk" "Parauk"
    "ps"  "Pashto"
    "fa"  "Persian"
    "pis" "Pijin"
    "pon" "Pohnpeian"
    "pl"  "Polish"
    "jac" "Popti"
    "pt"  "Portuguese"
    "pa"  "Punjabi"
    "qu"  "Quechua"
    "otq" "Quertaro Otomi"
    "raj" "Rajasthani"
    "rki" "Rakhine"
    "rwo" "Rawa"
    "rom" "Romani"
    "ro"  "Romanian"
    "rm"  "Romansh"
    "rn"  "Rundi"
    "rcf" "Runion Creole French"
    "ru"  "Russian"
    "alt" "S. Altai"
    "luz" "S. Luri"
    "quh" "S. Bolivian Quechua"
    "qup" "S. Pastaza Quechua"
    "msi" "Sabah Malay"
    "hvn" "Sabu"
    "sm"  "Samoan"
    "cuk" "San Blas Kuna"
    "sxn" "Sangir"
    "sg"  "Sango"
    "sa"  "Sanskrit"
    "skr" "Saraiki"
    "srm" "Saramaccan"
    "stq" "Saterfriesisch"
    "gd"  "Scottish Gaelic"
    "seh" "Sena"
    "nso" "Sepedi"
    "sr"  "Serbian"
    "srr" "Serer"
    "crs" "Seselwa Creole French"
    "st"  "Sesotho"
    "ksw" "Sgaw Karen"
    "shn" "Shan"
    "shp" "Shipibo-Conibo"
    "sn"  "Shona"
    "jiv" "Shuar"
    "smt" "Simte"
    "sd"  "Sindhi"
    "si"  "Sinhala"
    "sk"  "Slovak"
    "sl"  "Slovenian"
    "so"  "Somali"
    "nr"  "South Ndebele"
    "es"  "Spanish"
    "srn" "Sranan Tongo"
    "acf" "St Lucian Creole French"
    "su"  "Sundanese"
    "suz" "Sunwar"
    "spp" "Supyire Senoufo"
    "sjp" "Surjapuri"
    "sus" "Susu"
    "sw"  "Swahili"
    "ss"  "Swati"
    "sv"  "Swedish"
    "gsw" "Swiss German"
    "syl" "Sylheti"
    "syr" "Syriac"
    "tab" "Tabassaran"
    "tg"  "Tajik"
    "tks" "Takestani"
    "ber" "Tamazight (Tfng)"
    "ta"  "Tamil"
    "tdx" "Tandroy-Mahafaly Malagasy"
    "tt"  "Tatar"
    "tsg" "Tausug"
    "te"  "Telugu"
    "twu" "Termanu"
    "teo" "Teso"
    "tll" "Tetela"
    "tet" "Tetum"
    "th"  "Thai"
    "bo"  "Tibetan"
    "tca" "Ticuna"
    "ti"  "Tigrinya"
    "tiv" "Tiv"
    "toj" "Tojolabal"
    "to"  "Tonga (Tonga Islands)"
    "sda" "Toraja-Sadan"
    "ts"  "Tsonga"
    "tsc" "Tswa"
    "tn"  "Tswana"
    "tcy" "Tulu"
    "tr"  "Turkish"
    "tk"  "Turkmen"
    "tvl" "Tuvalu"
    "tyv" "Tuvinian"
    "ak"  "Twi"
    "tyz" "Ty"
    "tzh" "Tzeltal"
    "tzo" "Tzotzil"
    "tzj" "Tzutujil"
    "udm" "Udmurt"
    "uk"  "Ukrainian"
    "ppk" "Uma"
    "ubu" "Umbu-Ungu"
    "ur"  "Urdu"
    "ug"  "Uyghur"
    "uz"  "Uzbek"
    "ve"  "Venda"
    "vec" "Venetian"
    "vi"  "Vietnamese"
    "fy"  "W. Frisian"
    "knj" "W. Kanjobal"
    "wa"  "Walloon"
    "war" "Waray (Philippines)"
    "guc" "Wayuu"
    "cy"  "Welsh"
    "wal" "Wolaytta"
    "wo"  "Wolof"
    "noa" "Woun Meu"
    "xh"  "Xhosa"
    "sah" "Yakut"
    "yap" "Yapese"
    "yi"  "Yiddish"
    "yo"  "Yoruba"
    "yua" "Yucateco"
    "zne" "Zande"
    "zap" "Zapotec"
    "dje" "Zarma"
    "zza" "Zaza"
    "za"  "Zhuang"
    "zu"  "Zulu"
)

The inference platform to run the model must support T5 encode-decode architecture. The model is useful for sentence/phrase translations only as its maximum context window is 512 tokens. Therefore long passages or very long sentence material needs to be chunked up into smaller pieces to enable the model to work.

Example use:

This example shows use with a downstream llama.cpp server (supporting both T5 and beam search decode, which can be used to find higher probability translation sequences compared to straightforward greedy decode) accessed through a shell based CLI running in a bash shell which handles unicode correctly. The example will cycle a starting phrase through a small set of languages using a bash script. Note because the model was trained as a LangID model, the SOURCE language code does not have to be added as part of the prompt (the SOURCE language is auto-id'ed from the input phrase to automatically determine SOURCE)

bash

translate()

{
   phrase=$1
   SOURCE=$2
   TARGET=$3
   echo -n $(lm "<2${TARGET}>$phrase")
}

langs=(de es fr ru ja zh en)

phrase='The secret code to open the safe is "dead mens chest"'
SOURCE=en
echo "${SOURCE}: $phrase"
for TARGET in ${langs[@]}; do
   phrase=$(translate "$phrase" $SOURCE $TARGET)
   # Remove lead and trail \n delimiters
   phrase=${phrase#\\n}
   phrase="${phrase%\\n}"
   echo "${TARGET}: $phrase"
   SOURCE=$TARGET
done

Running this script produces:

en: The secret code to open the safe is "dead mens chest"
de: Der geheime Code, um den Safe zu öffnen ist "dead mens chest"
es: El código secreto para abrir la caja fuerte es "dead mens chest"
fr: Le code secret pour ouvrir le coffre-fort est "dead mens chest"
ru: Секретный код для открытия сейфа - "dead mens chest"
ja: 金庫の秘密鍵は "死んだ男の胸"
zh: 保险箱的秘密钥匙是死人的胸部
en: The secret key to the safe is the chest of the dead man.

The model handles the translation semantics correctly until it crosses over from indo-european to oriental language sets, when semantics are lost. madlad400-7b-mt and madlad400-3b-mt both lose semantics immediately at the first translation, incorrectly converting the secret code to the target language (the prompt is somewhat adversarial by design, explicity to test this). Beam search can be effectively used with MT models to improve generation confidence (i.e. find higher probability translations). A 4-beam search on the above example produces:

en: The secret code to open the safe is "dead mens chest"
de: Der geheime Code, um den Safe zu öffnen ist "dead mens chest"
es: El código secreto para abrir la caja fuerte es "dead mens chest"
fr: Le code secret pour ouvrir le coffre-fort est "dead mens chest"
ru: Секретный код для открытия сейфа - "dead mens chest"
ja: 金庫を開ける秘密のコードは "死んだ男の胸"
zh: 打开保险箱的密码是"死人的胸部"
en: The code to open the safe is "dead man's chest."

Technically, semantics are still lost when switching to Oriental, but the beam search here improved oriental translations enough that semantics were good coming back to the original language, but syntax was still broken (wrong secret phrase).

MT benchmarks for the model are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
madlad400-10b-mt.Q6_K_H.gguf	Q6_K_H	7.8e9 B	1B smaller than Q6_K
madlad400-10b-mt.Q4_K_H.gguf	Q4_K_H	6e9 B	~2B smaller than Q6_K_H

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040

steampunque/madlad400-10b-mt-MP-GGUF

Mixed Precision GGUF layer quantization of madlad400 10b mt by google

Download the file from below:

steampunque/
madlad400-10b-mt-MP-GGUF