GGUF, introduced by the llama.cpp team on August 21st, 2023, is a new format designed to replace the outdated GGML, which is no longer maintained by llama.cpp. GGUF boasts several improvements over GGML, such as enhanced tokenization, support for special tokens, and metadata capabilities. It is also designed with extensibility in mind.

Below is a partial list of clients and libraries known to support GGUF:

llama.cpp. The foundational project for GGUF, featuring both a command-line interface (CLI) and server options.
text-generation-webui, A highly utilized web UI offering extensive features and robust extensions, supporting GPU acceleration.
KoboldCpp, A comprehensive web UI with full GPU acceleration across all platforms and architectures, particularly effective for storytelling.
LM Studio, An intuitive and powerful local GUI designed for Windows and macOS (Silicon), featuring GPU acceleration for enhanced performance.
LoLLMS Web UI, A notable web UI with distinctive features, including a comprehensive model library for easy model selection.
Faraday.dev, A user-friendly character-based chat GUI for Windows and macOS (both Silicon and Intel), boasting GPU acceleration for smooth operation.
llama-cpp-python, A Python library featuring GPU acceleration, LangChain support, and an OpenAI-compatible API server.
candle, A Rust-based ML framework prioritizing performance, equipped with GPU support and designed for ease of use.

Prompt template: None

{prompt}

Compatibility

The NT-Java-1.1B GGUFs are supported by llama.cpp and are compatible with a range of third-party user interfaces and libraries. For a detailed list, please refer to the beginning of this README.

Provided files

Name	Quant method	Bits	Size	Max RAM required	Model Evaluation	Application Scenarios
NT-Java-1.1B_Q2_K.gguf	Q2_K	2	511 MB	764 MB	poor quality	not advised for usage
NT-Java-1.1B_Q3_K_M.gguf	Q3_K_M	3	663 MB	912 MB	moderate quality	suitable for environments with low RAM
NT-Java-1.1B_Q4_0.gguf	Q4_0	4	726 MB	1021 MB	moderate quality, prefer using Q3_K_M	not recommended, prefer Q3_K_M
NT-Java-1.1B_Q4_K_M.gguf	Q4_K_M	4	792 MB	1.1 GB	good quality	top recommendation due to optimal size and quality
NT-Java-1.1B_Q5_0.gguf	Q5_0	5	868 MB	1.08 GB	good quality, prefer Q4_K_M	not recommended, prefer Q4_K_M
NT-Java-1.1B_Q5_K_M.gguf	Q5_K_M	5	910 MB	1.13 GB	excellent quality	recommended, second-best choice
NT-Java-1.1B_Q6_K.gguf	Q6_K	6	1.02 GB	1.24 GB	excellent quality	generally not suggested due to size compared to Q5_K_M
NT-Java-1.1B_Q8_0.gguf	Q8_0	8	1.32 GB	1.54 GB	top-tier quality, near flawless	preferred in environments with sufficient RAM

Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.

How to download GGUF files

On the command line, including multiple files at once

The use of the Huggingface Hub Python library is recommended:

pip3 install huggingface-hub

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download infosys/NT-Java-1.1B-GGUF NT-Java-1.1B_Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False

More advanced huggingface-cli download usage (click to read)

You can also download multiple files at once with a pattern:

huggingface-cli download infosys/NT-Java-1.1B-GGUF --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

To accelerate downloads on fast connections (1Gbit/s or higher), install hf_transfer:

pip3 install hf_transfer

And set environment variable HF_HUB_ENABLE_HF_TRANSFER to 1:

HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download infosys/NT-Java-1.1B-GGUF NT-Java-1.1B_Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False

Windows Command Line users: You can set the environment variable by running set HF_HUB_ENABLE_HF_TRANSFER=1 before the download command.

Example `llama.cpp` command

Make sure you are using llama.cpp from commit d0cee0d or later.

./main -ngl 35 -m NT-Java-1.1B_Q4_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "{prompt}"

Change -ngl 32 to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.

Change -c 2048 to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically. Note that longer sequence lengths require much more resources, so you may need to reduce this value.

If you want to have a chat-style conversation, replace the -p <PROMPT> argument with -i -ins

For other parameters and how to use them, please refer to the llama.cpp documentation

How to use with Ollama

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Run the nt-java model:

ollama run infosys/nt-java

Building from `Modelfile`

Assuming that you have already downloaded GGUF files, here is how you can use them with Ollama:

Get the Modelfile:

huggingface-cli download infosys/NT-Java-1.1B-GGUF Modelfile_q4_k_m --local-dir /path/to/your/local/dir

Build the Ollama Model: Use the Ollama CLI to create your model with the following command:

ollama create NT-Java -f Modelfile_q4_k_m

Run the NT-Java model:

Now you can run the NT-Java model with Ollama using the following command:

ollama run NT-Java "Your prompt here"

Replace "Your prompt here" with the actual prompt you want to use for generating responses from the model.

How to use with Llamafile:

Assuming that you already have GGUF files downloaded. Here is how you can use the GGUF model with Llamafile:

Download Llamafile-0.7.3

wget https://github.com/Mozilla-Ocho/llamafile/releases/download/0.7.3/llamafile-0.7.3

Run the model with prompt:

public class HelloWorld {\n public static void main(String[] args) {

./llamafile-0.7.3 -ngl 9999 -m NT-Java-1.1B_Q4_K_M.gguf --temp 0.6 -p "public class HelloWorld {\n    public static void main(String[] args) {"

Run with a chat interface:

./llamafile-0.7.3 -ngl 9999 -m NT-Java-1.1B_Q4_K_M.gguf

Your browser should open automatically and display a chat interface. (If it doesn't, just open your browser and point it at http://localhost:8080)

How to run from Python code

You can use GGUF models from Python using the llama-cpp-python version 0.2.23 and later.

How to load this model in Python code, using llama-cpp-python

For full documentation, please see: llama-cpp-python docs.

First install the package

Run one of the following commands, according to your system:

shell

# Base ctransformers with no GPU acceleration
pip install llama-cpp-python
# With NVidia CUDA acceleration
CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python
# Or with OpenBLAS acceleration
CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python
# Or with CLBLast acceleration
CMAKE_ARGS="-DLLAMA_CLBLAST=on" pip install llama-cpp-python
# Or with AMD ROCm GPU acceleration (Linux only)
CMAKE_ARGS="-DLLAMA_HIPBLAS=on" pip install llama-cpp-python
# Or with Metal GPU acceleration for macOS systems only
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python

# In windows, to set the variables CMAKE_ARGS in PowerShell, follow this format; eg for NVidia CUDA:
$env:CMAKE_ARGS = "-DLLAMA_OPENBLAS=on"
pip install llama-cpp-python

Simple llama-cpp-python example code

python

from llama_cpp import Llama

# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = Llama(
  model_path="./NT-Java-1.1B_Q4_K_M.gguf",  # Download the model file first
  n_ctx=2048,  # The max sequence length to use - note that longer sequence lengths require much more resources
  n_threads=8,            # The number of CPU threads to use, tailor to your system and the resulting performance
  n_gpu_layers=35         # The number of layers to offload to GPU, if you have GPU acceleration available
)

# Simple inference example
output = llm(
  "{prompt}", # Prompt
  max_tokens=512,  # Generate up to 512 tokens
  stop=["</s>"],   # Example stop token - not necessarily correct for this specific model! Please check before using.
  echo=True        # Whether to echo the prompt
)

How to use with LangChain

Here are guides on using llama-cpp-python and ctransformers with LangChain:

LangChain + llama-cpp-python

Citation

@article{rathinasamy2024narrow,
  title={Narrow Transformer: Starcoder-Based Java-LM For Desktop},
  author={Kamalkumar Rathinasamy and Balaji A J and Rajab Ali Mondal and Ankush Kumar and Harshini K and Gagan Gayari and Sreenivasa Raghavan Karumboor Seshadri and Swayam Singh},
  journal={arXiv preprint arXiv:2407.03941},
  year={2024}
}

Original model card: Infosys's NT-Java-1.1B

NT-Java-1.1B

The Narrow Transformer (NT) model NT-Java-1.1B is an open-source specialized code model built by extending pre-training on StarCoderBase-1B, designed for coding tasks in Java programming. The model is a decoder-only transformer with Multi-Query Attention and with a context length of 8192 tokens. The model was trained with Java subset of the StarCoderData dataset, which is ~22B tokens.

InfosysEnterprise/NT-Java-1.1B-GGUF

NT-Java-1.1B-GGUF

Description

About GGUF