GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
Here is an incomplete list of clients and libraries that are known to support GGUF:
llama.cpp. The source project for GGUF. Offers a CLI and a server option.
text-generation-webui, the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
KoboldCpp, a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
GPT4All, a free and open source local running GUI, supporting Windows, Linux and macOS with full GPU accel.
LM Studio, an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration. Linux available, in beta as of 27/11/2023.
LoLLMS Web UI, a great web UI with many interesting and unique features, including a full model library for easy model selection.
Faraday.dev, an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
llama-cpp-python, a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
candle, a Rust ML framework with a focus on performance, including GPU support, and ease of use.
ctransformers, a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server. Note, as of time of writing (November 27th 2023), ctransformers has not been updated in a long time and does not support many recent models.
Original model card
license: llama2
Bailong: Bilingual transfer learning based on QLoRA and zip-tie embedding
This model card contains the information and the results of our released Bailong (白龍) project. Bailong, which stands for Bilingual trAnsfer learnIng based on qLOra and zip-tie embeddiNG, is our newest project aimed at enhancing the Traditional Chinese performance in open-source large language model (LLM). All the consequences are listed as follows:
Bailong 7B(not released): Bailong 7B is an autogressive language model with 7B parameters and decoder-only transformer architecture. It is derived from implementing secondary pretraining on Llama 2 7B with tied embedding and expanded vocabulary. The model is trained with context length of 2048 tokens and the training dataset is primarily composed of Traditional Chinese data with a minor portion of English one. Specially, motivated by the Chinese-LLaMA paper, we implemented QLoRA during the secondary pretraining stage to train the model, as opposed to the standard full-parameter training method. This approach significantly reduces the computational cost while achieving satisfactory model performance simultaneously.
Bailong-instruct 7B: Bailong-instruct 7B is the fine-tuned version of Bailong 7B optimized for multi-turn dialogue use case. Similar to secondary pretraining stage, we use QLoRA to fine-tune the model. To facilitate the development and communication within the research community in Traditional Chinese NLP, we decide to release this model on Hugging Face.
Bailong-bench: Most existing language models claiming to support Traditional Chinese are adapted from continuously pre-trained open-source models, primarily trained on English data. In certain cases, models fine-tuned with instructions using this approach may respond to Traditional Chinese instructions in English and vice versa. This could pose a significant problem when deploying the model for real-world applications. Consequently, it is essential to have a benchmark dataset specifically designed to assess a model's proficiency in following both English and Traditional Chinese instructions. To address this issue, we propose Bailong-bench, a benchmark dataset crafted not only to evaluate the model's performance in various real-world application scenarios but also to assess its ability to maintain language consistency.
Technical report: We intend to release a technical report in the future, providing a detailed overview of the Bailong project. Please stay tuned for further updates.
Bailong features
🚀 Fast and efficient tokenizer
We expand the vocabulary size of Llama 2 from 32000 to 59241 by merging original Llama 2's tokenizer with our self-made tokenizer. As shown in the following figure, with the fixed context length, Bailong's tokenizer generates less tokens and tokenize more efficiently compared to Breeze-7B's and Taiwan-LLM's tokenizers while tokenizing the Traditional Chinese sequences.
Tokenizers performance
💦 Aggresive cleaning
In addition to utilizing the conventional fuzzy deduplication, we also implement semantic deduplication such as SemDeDup during our data cleaning processes to improve the quality of the pretraining data.
🏃 Memory efficient training
Further saving more gpu memory than Chinese-LLaMA by using QLoRA during the secondary pretraining and supervised fine-tuning.
💪 Advanced methods for initiallizing embedding layer
We propose zip-tie embedding to initiallize embedding of appended vocabs. The proposed embedding method combined with appropriate learning rate save steps during the training. The detail of the zip-tie embedding can be found in the upcoming technical report.
🙋♂️ Advanced instruction tuning method for multi-turn dialogue
We leverage TargetLMLoss and training method proposed in FireFly project to endow model with multi-turn dialogue capability. For basic introduction of this method, please refer to zhihu article.
Applications (Bailong-instruct 7B)
The following tables present, but are not limited to, several possible scenarios for the applications of Bailong-instruct 7B.
根據以下的文本回答我，為何OpenAI會遭到控告？\n《紐約時報》（The New York Times）27日控告OpenAI及微軟公司（Microsoft）未經許可使用該報數以百萬篇文章，以幫助訓練聊天機器人為讀者提供資訊。\n\n《紐約時報》指出，該報是首個控告OpenAI及微軟的美國主要媒體機構，原因其作品相關版權問題。OpenAI是熱門人工智慧（AI）平台ChatGPT開發商，微軟則是OpenAI投資者及AI平台Copilot創造者。\n\n《紐時》在紐約曼哈頓聯邦法院提交訴狀，控告OpenAI及微軟試圖「免費使用《紐時》在新聞業的巨額投資」，利用《紐時》內容提供替代方式，對讀者傳遞訊息。\n\n此外，《紐時》還表示：「在不付費情況下使用《紐時》內容，來創造替代《紐時》並搶走其讀者的產品，並不具有『變革性』。」\n\n對此，OpenAI及微軟未立即作出回應。兩家公司曾指稱，使用受版權保護的作品來訓練AI產品，構成「合理使用」。
Do you think AI will totally replace human in the future?
As an AI language model, I don't have personal beliefs or opinions. However, it is difficult to predict whether AI will eventually become so advanced that it would completely replace humans as a workforce. While there are many benefits to using AI for tasks such as automation and efficiency, there are also concerns about ethical issues related to employment and job displacement. Ultimately, how AI becomes integrated into society depends on various factors such as policy, technology development, and public awareness. It remains to be seen what role AI will play in making progress towards reducing poverty, addressing environmental challenges, and improving health outcomes among other goals.
Write an article about how does climate change impact human society.
Climate Change is a rapidly evolving issue that affects every aspect of the world, including our daily lives and the way we operate as humans. The effects of climate change are not limited to just environmental harm but also have significant social and economic implications for all individuals and communities. Here are some ways in which climate change can influence human society:\n\n1. Health Issues: Changes in weather patterns and precipitation levels can lead to food shortages, water scarcity, and other health issues such as infectious diseases spreading more widely due to lack of access to clean drinking water or sanitary conditions. This can significantly increase the risk of malnutrition and promote disease outbreaks across subcontinental borders.\n2. Economic Impact: Climate changes can have significant financial repercussions on agriculture and livelihood sources, leading to reduced crop yields, increased cost of crops, and disruption to trade routes. These can lead to increased poverty rates and irregular income streams, causing inequality among populations based on their ability to cope with changing conditions.\n3. Refugee Crisis: As regions become less inhabitable due to extreme heat wave or flooding, people may be forced to migrate from their homes in search of safe refuge places. This can create new migration flows and conflict within countries, region, and cultural groups.\n4. Community Borders: Climate change can break down physical boundaries between regions, making it difficult for cultures to communicate and collaborate. For example, if a country experiences drought while another is experiencing rainfall, this could lead to political tensions over use of resources and distribution policies.\n5. Cultural Disputes: Climate change can cause differences in perceptions of nature, beliefs around natural governance, and sustainable development strategies. For instance, some nations may prioritize renewable energy while others may focus on fossil fuels. This can contribute to deep divisions in ideas of what constitutes sustainability and responsible resource usage.\nOverall, there are many ways through which climate change can impact human society. By addressing these challenges and developing resilience, we can work towards a better future where we can adapt effectively to changing conditions and stay secure during uncertain times. It's essential to take action now to reduce carbon emissions and improve sustainability practices, so that we can combat the impact of climate change and protect ourselves against its consequences.
What is a verb?
A verb is a word or a group of words used to describe an action, state, or condition, and is often used to complete a sentence. Examples of verbs in English include run, talk, walk, learn, and do.
Proofreading assistant(校對助理)
Prompt
Bailong-instruct 7B
幫我修正以下英語句子的文法錯誤\nIt cost less I expected
It cost less than I expected.
修正以下英語句子的文法錯誤。\nI have two hundreds dollars.
I have two hundred dollars. (已糾正後的句子)
修正以下英語句子中的錯誤\nOn the other words, making friends with smarter people can help people make money.
On the other hand, making friends with smarter people can help people make money.
Translation(中英語翻譯)
Prompt
Bailong-instruct 7B
請將以下英語文本翻譯成繁體中文\nMany studies have shown that time outdoors is good for our physical health. Three wilderness education experts explain why periods alone in nature also provide mental and spiritual benefits.
請將以下英語文章翻譯成繁體中文\nToday’s news is like fast food. They’re quick and easily accessible but poor in substance and value.\n\nOn the other hand, old news is like home-cooked meals prepared with heart. They are richer in substance and provide more long-term benefits.
With Taiwan's recovery from the pandemic, people's leisure lifestyles are starting to regain their vibrant nature again. Additionally, foreign tourism is beginning to return as well, and demand for services such as food, entertainment, and travel is growing exponentially. However, with a significant loss of workers in related industries during the pandemic, there has been increasing attention paid to the "manpower shortage" issues facing both the travel & hospitality sector (which includes hotels/motel, restaurants, and airlines) and the catering industry.
在全球氣候變遷日益嚴重的背景下，尋求減緩碳排放並實現永續發展的需求更加迫切。減碳轉型和永續發展已經成為當今全球議程的核心，共同為實現淨零碳排的目標而奮鬥。減碳轉型代表著我們對降低溫室氣體排放，並將再生能源廣泛推行的承諾，以及要保持地球上所有物理、化學或生物特性不會因人类活动引起窘境的可能性。Energy transition and sustainable development have become the core of today’s global agenda. Decarbonisation and a shift to renewables are our commitment, as well as ensuring that all physical, chemical or biological properties on Earth will not be affected by human activity in an unfavourable way.\n In this context, energy companies must play their part by developing innovative solutions for low-carbon transitions while at the same time taking into account the needs of society and preserving natural resources. Aware of its responsibility towards future generations and committed to playing its role within these transformations, Total is working every day to accelerate the pace of change through numerous initiatives around four main pillars: climate action, biodiversity, circular economy
南韓媒體報導，ChatGPT 引領的生成式人工智慧的興起，設備端人工智慧市場正在開展。由於設備端人工智慧是在智慧型手機等資訊科技設備中達到人工智慧功能，而不依賴伺服器和雲端的技術。因此有利減少使用者的安全性及個人隱私風險。【South Korea media report】, ChatGPT has led the rise of generative artificial intelligence. The equipment-end AI market is in full swing with set-top devices such as smartphones and information technology equipment achieving artificial intelligence functions without relying on servers or cloud computing technologies. Therefore, it will be beneficial to reduce users' security risks and privacy concerns. According to a report by Yonhap News Agency (YNA), South Korean companies are actively developing products that combine AI chips for use in mobile phones and other IT equipment. LG Electronics announced plans to apply its own deep learning AI chip "Vision Processing Unit" (VPU) to the next generation of TVs released this year. The VPU was develope
We show the evaluation results of models on Bailong-bench here. For judging, we implement LLM-as-a-judge and prompt GPT-4 to act as judges and assess the quality of the models' responses. Note that in our implementation, the consistency of the language used between the user and the chat assistant has been included in the evaluation criteria. Unless it is a translation task or specifically specified by the user, the language used by the model should be consistent with the language used by the user.
Incorrect facts and information: As with all LLMs, the model may generate incorrect responses to user's prompts. Users should consider the model's output as suggestions or references rather than definitive and sole answers.
Language limitation: The model is mainly designed to follow Traditional Chinese and English instructions. Therefore, the model may lack the capability to follow instructions composed in other languages including Simplified Chinese, leading to potential misinterpretations or errors in response.
Potential toxicity, bias, and uncontrollability: Due to the lack of further training through reinforcement learning from human feedback, there's a possibility that model may generate harmful and societal biased responses or deviate from the expected responses. As a result, before deploying any applications of our model, developers should perform safety testing and tuning tailored to their specific applications of the model.
Citation
@article{chen2024bailong,
title={Bailong: Bilingual transfer learning based on QLoRA and zip-tie embedding},
author={Chen Lung-Chuan and Li Zong-Ru},
journal={arXiv},
year={2024}
}
