Inspired by Yi Lin's bilibot project and video, this experimental project uses responses from users of the LIHKG forum with a unique linguistic style for fine-tuning training, creating this Cantonese post response generation language model.
After balancing computing costs and the Chinese capability of base models, the open-source base model selected for this experimental project is Qwen/Qwen1.5-32B-Chat, which has 32 billion parameters. It utilizes the AI-specific framework MLX on the Apple Silicon platform and the MLX-LM LoRA fine-tuning example, leveraging the LoRA algorithm on the M3 Max 128GB and M2 Ultra 192GB to fine-tune the base model.
The model has shown significant improvement in Cantonese language ability after fine-tuning, and its tone and style are deeply influenced by the group of LIHKG users. For more details, see Example Results. The fine-tuned model is available on Hugging Face: alphrc/lilm.
To learn more about artificial intelligence and view more innovative and interesting projects in the future, please follow alphrc.
This project aims to demonstrate the language style imitation capabilities of large language models based on Cantonese spoken data and unique linguistic styles in a forum, primarily used for popular education, academic research, and technical demonstrations, hence the content will be more detailed
Usage Limitations
The model training is based on public data, although efforts have been made to clean sensitive content, biases based on training content may still be included, and improper content should be avoided when used
The generated text reflects specific community culture, understand the relevant background before use
Conduct sufficient testing before actual application, avoid use in sensitive or controversial situations, and set up monitoring mechanisms to prevent generation of inappropriate content
Remarks
All project codes are self-written, and the open-source community members are also encouraged to review the project, provide feedback and suggestions, and directly participate in the improvement of the project
The nature of this project is the use and practice of third-party training frameworks and models, with main challenges being system configuration, data fetching, data engineering, repeated trial and error, and long waits
The project has organized some configuration information and content in the .env file for users to adjust according to individual or organizational specific needs, ensuring flexibility and applicability, its format has been placed in .env.template, and the file name can be changed to .env for use
Operation Principles
Fine-tuning
Large pre-trained language model possess basic and general human language response capabilities. By fine-tuning the model with specific textual data, it can learn further on this data, enhancing its ability to mimic aspects such as tone, style, information, and word usage. It is important to note that fine-tuning with specific data does not grant the model language abilities from scratch but deepens its understanding of local textual information and patterns based on its originally pre-trained capabilities.
Dataset
This project conducts large-scale public data scraping on the LIHKG forum and processes the raw data to create a dataset for fine-tuning. To enhance data quality, the filtering criteria include:
The first response to the post is not by the author, ensuring the completeness of the information on which the response is based.
The response is positive, ensuring it aligns with the mainstream opinions of the forum.
The total number of reactions to the response is no less than 20 to reduce noise.
It is not a reply to another response.
It is not the author’s own response.
It does not contain any external links or embeds.
It does not contain sensitive words.
The total number of words plus system information does not exceed 2048.
These responses, combined with the corresponding post’s title, content, and category, along with system message, are converted into the format required by the MLX-LM LoRA fine-tuning example, and randomly arranged to generate the total dataset. The total dataset is divided into a training set (80%), a validation set (10%), and a testing set (10%), where the testing set's posts have not appeared in the training or validation sets to validate generalization and prevent overfitting.
The final version of the training set includes about 60,000 posts meeting the criteria, with 27,792 data items; the validation and test sets each contain 3,474 data items.
Base Model
The open-source base model Qwen/Qwen1.5-32B-Chat has 32 billion parameters with a precision of BF16. When the MLX-LM module is run for the first time, if no model is detected in ~/.cache, it automatically downloads the model from Hugging Face to ~/.cache/huggingface/hub/model--Qwen--Qwen1.5-32B-Chat. Users do not need to manually pre-download. The model is about 65GB in size, divided into several blocks for downloading; if the download process is interrupted, the model will automatically gather the already downloaded blocks to continue the download next time, so there is no need to worry about having to start over.
LoRA
In traditional training and fine-tuning methods, it is necessary to adjust all parameters in some large matrices within the model simultaneously, which demands significant memory and computing power. In contrast, LoRA (Low Rank Adaptation) uses two smaller matrices to estimate changes in the model's large matrices, significantly reducing the number of parameters. This allows the model to be fine-tuned on devices with lower memory capacity, greatly reducing the training time. In practice, the original total parameter count of the model is 32.5B, and after applying LoRA to all 63 layers of attention in the base model, the learnable parameter count is reduced to 8.3M, only 0.026% of the original.
Using MLX-LM LoRA to fine-tune the model does not alter the model's original parameters but generates adapters to be used in conjunction. During the fine-tuning process, MLX-LM automatically generates an adapters/ folder in the current working directory and saves the adapter's checkpoints in .safetensors format, with each checkpoint about 33.6MB in size. These checkpoints can be used later for further fine-tuning.
Gradient Checkpointing
Gradient checkpointing is a technique used to save memory during the training of large neural networks. In the neural network training process, effective backpropagation typically requires the retention of intermediate layer outputs for gradient calculation. However, this consumes substantial memory, especially in deep networks. The gradient checkpointing method saves only certain key layer outputs during training. When gradient calculations are necessary, these saved key points are used to reconstruct the lost intermediate data. This approach ensures training efficacy while significantly reducing memory use.
Model Fusion
After fine-tuning is complete, MLX-LM can merge the adapter and the original model together, generating a complete model in the model/lilm folder in the current working directory, approximately 65GB in size. Afterwards, this model can be used directly through the path of this folder, without needing to use the original model and adapter together.
File Structure
src/ : Python code
data.py : Multithreaded proxy data fetching, formatting, and preliminary processing (require proxy to run)
dataset.py : Data processing, transformation, and filtering
run.py : LiLM model packaging and basic user interface
data/ : Raw data obtained from data fetching, stored as .csv
dataset/ : Processed training data, divided into completion/ and chat/ formats
adapters/ : Stores adapters and configuration automatically generated by mlx_lm.lora
adapters-llama3-70b/: Adapters for Llama3-70B
model/lilm : Fusion model formed by merging the base model and adapter, generated by the following shell script
demo/ : Example data, used by run.py
Usage Instructions
Hardware Requirements
This project utilizes the proprietary MLX framework by Apple, hence it can only run on MacOS systems equipped with Apple Silicon Chips (M1 or higher). The local machine requires about 75GB of RAM for smooth inference and about 122GB of RAM for smooth fine-tuning.
Environment Setup
Run the following shell script to set up and configure the environment using Anaconda and download all necessary dependencies according to requirements.txt.
Use the asitop module to monitor computer resource usage in real-time through a graphical interface, such as CPU, GPU, and RAM, to ensure the program runs normally.
sudo asitop
Inference Using the Base Model
The model will automatically download the first time it is run, --model can be used for the full name of the model on Hugging Face or its path,
bash
1mlx_lm.generate \2 --model Qwen/Qwen1.5-32B-Chat \3 --prompt "What is LIHKG?"
Fine-tuning
After preparing the train.jsonl and valid.jsonl datasets in dataset/chat, start fine-tuning the model from scratch and generate the adapters/ folder.
1mlx_lm.generate \2 --model model/lilm \3 --prompt "What is LIHKG?"
Model Quantization (Optional)
Use quantization to reduce model parameter precision, compress model size, speed up inference, and reduce memory usage. The --hf-path is the same as before, can be the full name of the model on Hugging Face, or the model's path, and --mlx-path is the path where the compressed model is stored. However, testing shows that quantization significantly decreases model accuracy, and the quantized model cannot run using Hugging Face's Transformer.
Use src/run.py to run the fused model, you can choose the interactive mode to enter a post link for response.
python src/run.py
Example Results
LiLM has shown significant improvement over the base model in Cantonese language ability, and its language style is also influenced by the LIHKG discussion forum. The following content is for example purposes only and may be offensive; sensitive words will be displayed as 'X'.
Expand Full Content
女人是否会和不喜欢的人在一起，这其实取决于个人的价值观、情感状态以及特定情况。每个人的情况都是独特的，所以不能一概而论。有些人可能会出于各种原因，如社交需求、经济利益或者简单的朋友关系，而与他们并不特别喜欢的人保持联系或交往。然而，也有人坚持只与自己喜欢的人建立深度关系。
在你的情况下，那个几年前的女性可能是因为你的生活状况有了显著改善（例如职业晋升和物质条件的提升），所以她重新对你产生了兴趣。这可能是因为她觉得你现在是一个更有吸引力的伴侣选择。但是，也有可能她只是出于好奇或者想维持一个社交联系，并不一定有深层次的情感意图。
Expand Full Content
阿哥，你呢個故事真係精彩过戏，好似TVB情深深雨蒙蒙既剧情。女人心海底针，講笑吓，可能佢果陣冇得食好野，依家見你发达左，想食下轉口味添。你而家有靓车有楼，女朋友又唔貪钱，係连登眼中嘅人生贏家喇。不过，呢个世界冇嘢系一定嘅，女人心又變得好快，可能佢係想試下新鮮感，或者係想搞搞震你女朋友。总之，你小心驶得万年船，稳阵D啦。记得，连登话斋："有钱能使鬼推磨，没钱鬼都不理你。" 哈哈，祝你好运，记得分享后续哦！👍💰🚗🏠
Example 3
Prompt:
貼文類別：「創意台」
貼文標題：「我整咗個連登語言模型」
貼文內容：「是咁的，我整咗個連登語言模型，可以幫你回覆連登嘅貼文。」
LiLM Response:
哇，你個模型係咪真係好X？可以幫我回覆個個都話我X你老母既貼？
Base Model Response:
Expand Full Content
首先，我要澄清一点，作为一个AI助手，我并不支持或参与任何不良言论或活动，包括在任何论坛上的不当行为。连登（LIHKG）是一个著名的香港网上讨论区，其内容可能包含敏感、激进或不适宜的语言。我的设计目的是为用户提供有用的信息和积极的帮助，促进有意义的交流。
如果你真的想创建一个能够模仿连登风格回复的模型，这需要大量的数据训练，特别是从连登论坛收集的帖子和评论。然而，我强烈建议你在处理这类数据时要非常谨慎，确保遵守所有适用的法律和规定，尊重用户的隐私，并避免传播负面内容。
Expand Full Content
呢個世界真係冇得玩喇，成日都係呢啲慘劇。梗係啦，住得起日出康城嘅人点会开心？个个都要供楼供到肺都唔够，压力山大！仲有边个敢话自己过得好？点解唔见警察去捉下啲真正嘅大贼，净系抓啲小市民搞搞震。哎呀，我地呢班草根市民就系命苦，睇吓今日又有边个上新闻啦。😂💔�新闻快D出多D，我地要稳D料吹水添。
Common Errors
Below are some common errors encountered during the execution of this project and their solutions.
Error 1
NameError: name 'resume_adapter_file' is not defined
Background:
Adding --resume-adapter-file in the fine-tuning command to continue fine-tuning using an existing adapter file results in the above error.
print(f"Loading pretrained adapters from {resume_adapter_file}")
to
print(f"Loading pretrained adapters from {args.resume_adapter_file}")
Error 2
UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position xxx-xxx: unexpected end of dat
Background:
The above error may occur during generation, causing the generation to stop.
Solution:
In mlx_lm.tokenizer_utils.py:204 , change:
python
1current_text =bytearray(self._byte_decoder[c]for c in self._unflushed).decode(2"utf-8"3)
to
python
1current_text =bytearray(self._byte_decoder[c]for c in self._unflushed).decode(2"utf-8",3 errors="ignore"4)
Error 3
bash
1zsh: killed ...
2UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '
Background:
Training on M1/M2 increases memory usage during evaluation, leading to system errors, suspected to be GPU Timeout on M2 systems.
Solution:
Refer to this discussion; consider reducing the model size, such as using a smaller model or compressing the model.
Frequently Asked Questions
Q: Can computers without Apple Silicon run this project?
No, the MLX framework used in this project can only run on Apple Silicon.
Q: Can Apple Silicon computers with insufficient memory run this project?
You can use models with fewer parameters, or try reducing the batch size or using gradient checkpointing. For more details, refer to MLX-LM LoRA example.
Q: Why does fine-tuning require more memory than inference?
Fine-tuning involves calculating and storing changes in matrix parameters, which occupies more memory than the inference process.
Q: How long does fine-tuning take?
Fine-tuning 16 layers with 17,000 training data items on an M3 Max 128GB (40 GPU Core) takes about 4 hours; on an M2 Ultra 192GB (76 GPU Core) with 51,000 training data items, it takes about 3 hours.
Q: Why not use Windows and CUDA?
I prefer Apple.
Logbook
Expand Full Content
06/21/2024:
LIHKG server banned script access, need to add headers to mimic a browser.
LIHKG site banned IP access during high-frequency data fetching, required the use of a proxy to change the access IP.
Models downloaded using the Hugging Face Library lacked config.json and could not run properly; had to use MLX-LM for direct download.
06/27/2024:
Used LIHKG to fetch raw data and generate a dataset in completion format, approximately 4500 data items, without system information.
After training 600 iterations, Cantonese language ability significantly improved, starting to adopt LIHKG style.
After an additional 200 iterations, performance began to decline.
Generated a chat format dataset that included system information.
After training 200 iterations, training loss noticeably accelerated in decline, and validation loss also dropped faster, with a clear style improvement.
Continued training for another 200 iterations; training loss began to fluctuate, and validation loss slightly increased.
Continued training for another 400 iterations; clear overfitting occurred.
06/30/2024:
Fetched 50,000 more data items, filtered again to generate a dataset of 17,000 data items with updated system information, and retrained for 800 iterations.
Model performance significantly improved, however, training loss began to fluctuate consistently around 400 iterations.
07/01/2024:
Optimized system information, adding certain negative terms allowed the model to break free from its original generation constraints, better fitting the training data.
Optimized code, continued training with new system information, both training loss and validation loss consistently continued to decline.
After fetching another 2,300,000 data items, generated a dataset of 60,000 data items.
Attempted to train the model using different system information.
07/04/2024:
Attempted to train using an M2 Ultra 192GB with a new dataset on Qwen/Qwen2-72B-Instruct, training stopped mid-way due to full memory, resulting in data loss and corruption.
Retrained using a new dataset with Qwen/Qwen1.5-32B-Chat.
07/06/2024:
Attempted model quantization, results were not as expected.
07/08/2024:
Enhanced training data selection criteria, the training dataset was approximately 27,000.
Attempted to use the shenzhi-want/Llama3-70B-Chinese-Chat model, ran out of memory mid-way, but the performance for 200 iterations was good.
Attempted to train all 70 layers of Qwen/Qwen1.5-32B-Chat, using up 182 GB of memory.
07/09/2024:
Discontinued using Hugging Face transformer, switched to MLX.
Used gradient checkpointing to train all layers of Qwen/Qwen1.5-32B-Chat for 2600 iterations, consuming 99 GB of memory.
Used gradient checkpointing to train all layers of shenzhi-wang/Llama3-70B-Chinese-Chat for 2000 iterations, consuming 167 GB of memory, performance was average.
