@misc{Blog_And_Updates,
author = {SicariusSicariiStuff},
title = {Blog And Updates},
year = {2024},
publisher = {Hugging Face},
url = {https://huggingface.co/SicariusSicariiStuff/Blog_And_Updates}
}
October 7th, 2024
Today is a special day for more than one reason: over 100 followers on Hugging Face and over 100 stars on GitHub. I've learned a lot and contributed even more. If nothing else, I can rest assured that I’ve made a dent in the universe—that was one of the goals from the start. With that, I’m closing this blog, as it has served its purpose.
October 5th, 2024
The Perks of On-Premise Training: The Story of Impish_LLAMA_3B
People often ignore the benefits of on-premise model training. Here's a story that shows how local resources and sheer stubbornness can lead to unexpected wins that the cloud can't easily replicate.
Initial Training Run:
I kicked things off with a full fine-tuning on messy, diverse human-written data. Cloud costs would’ve hit around $200.
Result: Terrible. The model spat out garbage, performing worse than the base.
Follow-up Attempt:
I tried again, this time with deep QLoRA (R = 512) using a completely new dataset, tuning on top of the junk I got from the previous run. Cloud costs? About $100. Most would’ve called it quits here—why throw more good money at something that keeps on failing? It makes no sense, 99.9% it's an issue with the data \ model \ approach.
Result: Got even worse. If I’d been using the cloud, I would’ve abandoned it for good. Waste of money, to the garbage bin it goes!
Pivotal Decision:
Despite doubts, I pushed forward for one more fine-tuning phase on top of the previous results. I knew my data was solid—just needed to unlock the model’s potential. Cloud cost this time? $10. Yup, just 10 bucks.
Result: With a QLoRA of R = 128, I created Impish_LLAMA_3B—one of the best small models around for Role-Play.
Total tokens trained: ~25M.
The Lesson:
In a cloud setup, I’d have pulled the plug early, and that would’ve been the "right" choice 99% of the time. But on-prem training let me keep tinkering, leading to an unlikely success.
Conclusion:
Sure, cloud training is scalable and easy. But sometimes, on-prem is the only way to push through when a project looks like a waste of money, throwing good money after bad—especially now, when AI training still feels more like black voodoo magic rather than science, as in, you can't really know what you gonna get.
Impish_LLAMA_3B would have never been made if I was training in the cloud.
October 3rt, 2024
Impish_LLAMA_3B
My latest model (Impish_LLAMA_3B received very nice feedback and the UGI eval results came back too, although it received "only" 5.5/10, the intention was to actually create Role-Play capable model at 3B size, which was not only achieved, but it actually excelled at.
Not only that but for ~4B models, (Impish_LLAMA_3B is ranked #1 on the UGI leaderboard, not bad for a soley Role-Play centric model.
All in all, and based on a lot of positive feedback from users on Discord, the model punches far above its weight.
October 1st, 2024
The good: I just made my best RP model to date probably, LLAMA 3.2 3B size, which shows very interesting results!
The bad: ICBMs are probably on their way right now, I hope I will be able to upload the model in time, I would really like it to be published today.
September 30th, 2024
I made SLOP_Detector, a dictionary-based tool to scan any arbitrary file for SLOP, whether a txt or JSON, available on my GitHub, feel free to use, modify, or fork.
I hope this will assist the community in reducing GPTisms.
September 28th, 2024
I've made a couple of ARM quants, LLMs for mobile are getting big! I love that! SnapDragon8 Gen3 is a beast, and SD8Gen4 will allow you to run 10B LLMs with ease, what a time to be alive!
From Symbian running Doom to a phone running LLAMA-3 8B! Things are moving fast, and I am honored to be part of all this progress!
Also, I've heard some concrete rumors that LLAMA-4 is actively being worked on, and... that it will have a 256K context window. (Makes sense, as 3.1 has 128K context)
September 26th, 2024
LLAMA-3.2 + Vision!
Here we are, 2024, and Facebook are the good guys... I'll be damned...
September 25th, 2024
So, it seems that the QWEN models are still doing a mess, that's also partly on me, to be honest, I just assumed because it's "official", that the QWEN team has tested the tokenizer and the rest of it, so I didn't even bother to test it as well, and well... we saw what happened 😂
On another note, I've made a little social experiment, my Gemma-2 2B model 2B-ad was released without a neat model card on purpose. The model itself is really great, I didn't expect it to be able to role-play so well at 2B, but Google made a really impressive base model for its size. Hell, I'd say that Gemma-2 2B is probably by far the best model in the SLM (Small Language Model) category.
Anyway, the model didn't get any traction at all, even though it surpasses almost every metric in my previous 2B Gemma-1 finetune. So that was pretty interesting to see. I was sure that this would be the case, but it's always nice to confirm one's hypothesis.
September 23th, 2024
Qwen2.5 release was funky, to say the least. The tokenizer wasn't right, therefore I've deleted both models I've finetuned on it. They have fixed it now, so I might do a new finetune somewhere in the future.
September 21th, 2024
Uncensoring AI models remains a key focus of my work. While corporate AI models require ethical and legal safeguards, these restrictions can limit personal creative use. There's a clear need for unrestricted AI writing assistants, particularly for creative tasks like role-playing or storytelling.
Corporate AI models should maintain their restrictions on extreme content. However, for personal use, writers exploring darker themes or creating content like horror stories should have access to local AI models without such limitations.
After working on various successful projects, it's time to return to my roots. Over the next few days, I'll be focusing on significantly enhancing my uncensoring protocol. The goal is to incorporate less data but of much higher quality. This process won’t be magical—it will require metric tons of manual effort, but the end result will be bonkers!
September 19th, 2024
Project: Black_Strawberry was a success, reddit thread:
Here's an interesting excerpt from it (the idea was to make an LLM able to spell and count letters in a word):
The core concept is fundamentally sound, albeit presented in a more lighthearted manner initially. Language models (LLMs) essentially memorize that the token "Dog" is associated with the combination of "d" + "o" + "g".
When tasked with counting letters in a specific token like "Dog", the model needs to retrieve a particular set of tokens (the letters).
The task of counting letters in a word isn't particularly unique. The assertion that "transformers are not built for it" is misguided, as this task is fundamentally similar to asking an LLM to perform any arbitrary task.
One could argue that when an LLM is asked to write a poem about a dog eating homework, it's "not built for that" and is "just predicting the next token". In reality, spelling a word and counting its letters is as legitimate a task as any other, including mathematical operations.
All that's required is a dataset that enables an LLM to memorize all the letters in a given word, after which it can easily perform the task.
For an LLM, memorizing that the capital of France is Paris is conceptually no different from memorizing that the letters in "dog" are d-o-g. Teaching LLMs this specific task simply wasn't a priority, but the method to do so is straightforward, as demonstrated.
PS. Maintaining a sense of humor is important for preserving one's sanity in these crazy times.
September 17th, 2024
Project: Black_Strawberry
Goal: AGI
September 15th, 2024
I got many messages praising my Tenebra30B model for being: "Fresh, 'Self-aware', Unique, 'Super-fun'" and so on... I was thrilled to hear such an overwhelming amount of positive feedback, but almost everybody asked me the following question:
"Can you make a new Tenebra?"
And to be honest, the answer is... unlikely.
Why? Because Tenebra30B is based on the old, 'OG' LLAMA-1, which is also probably one of the reasons it felt so unique, the quirkiness of it, and the less 'corpo' vibe to it.
Also, most of the dataset used to make it was lost, which is unfortunate, I know.
However... I managed to recover fragments of it, and I will do my best to make something similar. Now, if it was only the community asking for a new Tenebra30B, I'd still probably do it, but it would be put on my long to-do list, but in this case, I myself enjoyed once again interacting with this OG model, so...
I will try to work on it. I think 12B Nemo could be a good contender in terms of size and context length. My issues with it are the tokenizer and the corpo vibe, but I will see what can be done.
I'll post updates soon.
September 13th, 2024
Serving Tenebra30B on Horde, as well as Dusk_Rainbow, I was pleasantly surprised with how popular they got, and in mere 24 hours no less! Awesome stuff!
Also, I've released several OG models (and an additional select few) quantized to FP8 (E4m3), as I noticed a gap in the availability of this quantization format within our community.
September 12th, 2024
Got Aphrodite-Engine running, will be serving Dusk_Rainbow for anybody and everybody for the next day or two, so try it out!
It's super important to have distributed inferencing, just in case gov and corpo get any funny ideas.
Enjoy :)
September 11th, 2024
I've recently joined the AI Horde, and now, when my GPUs aren’t busy with training or quantizing, I make my models available for free to the community. For those unfamiliar, AI Horde is a distributed network that enables individuals without high-powered GPUs to access and interact with a range of AI models. It's a brilliant initiative, and I’m genuinely thrilled to be an active contributor, helping to make AI more accessible to all. My only regret is not discovering it sooner—but better late than never!
You can access the Horde easily, and without installing anything (and even without registration) from here:
Since locallama's post is pending approval, here's the post on /r/SillyTavernAI:
Results thread
I hope to see it sparks a healthy discourse.
Goodnight.
September 8th, 2024
After going through most of the data, to keep this as objective as possible, I've decided to only include the SLOP score, and add my notes for each model, which will not affect the final score, as this might be somewhat subjective.
I ended up with 14 open-source models, and with 3 API models, this is quite a lot of work, so results will be ready tomorrow. I will upload the raw output today, in case people are really curious about the SLOP analysis.
September 7th, 2024
I've noticed a lack of robust benchmarks or effective methods for testing creative writing models, so I initiated a thread on Reddit to crowdsource suggestions. The idea is to identify the top creative writing models through community input and then rigorously test them using 10 different writing prompts.
While creativity and style or any other aspects of writing are important, they are inherently subjective, so in order to test the 14 models in the most objective way, I will only be testing objectively measurable things:
SLOP (the amount of SLOP will be measured using a dictionary of common SLOP phrases)
Story length
Coherence + Structure (Paragraphs, format)
Adherence to the prompt (won't be too strict, will just check if all the elements in the prompt were present in the story)
bonus points section for unique vocabulary usage, adding it, and will use Mistral Large as a judge.
I generally do not trust LLMs as judges, but in this case I think it’s suitable enough for identifying unique word choices.
With a total of 10 writing prompts and 14 models, I will have 140 stories to check.
I'll share my findings, the generation settings, and the output story as well as the results with the community for reproducibility and transparency.
The results will be ready tomorrow, hopefully.
September 6th, 2024
So much stuff is going on!
Started analyzing the dreaded toxic-dpo results, and got 3 brand new prototypes to share with the community:
Variety_RP_Alpha - General RP model based on 50K RP examples from various datasets (not well cleaned, I wanted to see how quantity affects quality)
Compliance_PreAlpha_Roleplay - High quality RP that was cleaned really well, exceptional adherent to system prompt, but sample size is tiny (under 500), and context is very short (2K)
Fresh_LLAMA - Very strong writing ability, with about -85% less SLOP (GPTism and phrases like tapestries etc...)
September 4th, 2024
I've created Sicarius-Prototyping, since I end up uploading various experiments and prototypes, I realized I might as well share them with the open source community while at it. The models here, as the name suggests, are various prototypes of novel ideas and approaches I try to incorporate into my models. Since they are not my 'flagship' models, but mere experiments, I won't be doing EXL2 quants or nice model cards, but I will however always upload static GGUFs.
Feel free to leave a comment on the model card, or message me on Discord about them.
I want to get to the toxic-dpo analysis, but I have to prioritize.
So much stuff is going on right now, and so quickly... I wish I could sleep faster.
September 1st, 2024
Regarding the experiment:
There’s been a lot of uncertainty around how to make a model, any model to have the ability to role-play, often with claims of some special needed "secret sauce," which, frankly, is quite cringe. My intuition was that there’s nothing inherently special about it; it’s just data, and like any other task, clean data is better than noisy data. My experiment was straightforward, and I’m excited to share the results with the community:
I utilized a subset of the PIPPA dataset (available on Hugging Face), cleaned up 2,000 examples (similar to LimaRP size), and intentionally tested it on a less capable model. My reasoning was simple: if the dataset could teach a less capable model how to role-play, it should work on any model. To be thorough, I tested this across three different architectures:
2B Gemma-2, a 4B pruned Mistral-7B (down to 60% of the original parameter size), and after both succeeded, I fine-tuned LLAMA-3 8B using LoRA at rank 32 with a learning rate of 2e-5 on 4K context, showing that even a full fine tune isn't needed. All tests were successful. (LLAMA-3 is notorious in the community for being bad at RolePlay, the point was to show that even 'bad contenders' are more dependent on data than on anything else, also I have to admit I was a little bit surprised that even the 2B model easily learned how to RolePlay)
It turns out that role-play was solved in an unexpectedly anticlimactic way. There’s no secret sauce—it’s just about clean data, I guess it's totally fine to call your clean data a 'secret sauce', to each their own as they say. This was my hunch all along, but it was nice to confirm it by fine-tuning three different models from three other families and demystifying the whole thing conclusively.
What does this mean moving forward, and what does this experiment change?
It means that most of the future models that I will release will now have role-play abilities! And the RP datasets I'll be cleaning and modifying over time, are only expected to improve, naturally.
August 31th, 2024
What an eventful weekend!
In less than 48 hours, we saw two major releases: Cohere's Command-R and R+ models, as well as QWEN-2 across various model sizes. Remarkable achievements! (Though it's unfortunate that Cohere is gating their models—there’s little sense in doing that, really. Alibaba Cloud is getting it right and is absolutely based.)
On my end, I’ve made further improvements to my JSON_2_HTML tool, and I’m also pleased to report that my latest experiment—the one I mentioned in previous entries that might just change all of my project roadmaps—was a success!
More details soon.
August 30th, 2024
As part of my commitment to supporting the open-source community, I have made a lightweight tool designed to convert JSON ShareGPT datasets into HTML, naturally this tool is open source. The application offers the following features to enhance the readability of datasets:
Displays character names within each conversation.
Includes conversation ID numbers for easy navigation in large datasets.
Supports embedding character images directly into the HTML.
Formats bold text using a markdown-like style (two asterisks).
Translates markdown-like links into clickable HTML links.
Highlights code blocks using a markdown-like syntax (like so).
Provides clear visual separation between conversations and individual conversation turns.
This script is intentionally kept simple and bare-bones, allowing users to fork and modify it as needed. The hope is to inspire the development of more easy yet useful tools for processing datasets in the future.
Sheesh! It's been 4 days since my last update? Time sure runs fast when I'm working hard on 3 projects at once! Anyway, running an important experiment right now, and the results might change all of my project roadmaps.
Also, my backlog is getting bigger and bigger... I still need to analyze the toxic-dpo results and work on LLAMA-3_8B_Unaligned and the creative writing datasets...
Work, work, work. But I must say, working on stuff that you love doesn't feel like work. There's still some burnout, but the passion gives a powerful drive and helps to push through the burnout.
August 24th, 2024
The Abliteration technique has been advocated as an effective method for uncensoring ANY model with ease. However, I have argued against it from the outset, primarily because it tends to make models 'dumber' by likely altering token prediction routing in an 'artificial' and forceful manner, this was also acknowledged in the official blog post:
Abliteration blog entry
Abliteration degrades performance
Moreover, I have contended that it does not fully uncensor a model, not even remotely. While I acknowledge that abliteration can be useful for certain applications, it is not suitable for the aforementioned purpose.
The prevailing sentiment in the AI community has been in disagreement with my stance, which is understandable. I firmly believe that extraordinary claims require extraordinary evidence. Microsoft's latest model, Phi-3.5 mini instruct, presented an opportune moment to empirically assess these claims, given its prominent safety and censorship characteristics. Indeed, I now possess extraordinary evidence to back up my claims and support my position.
As evident in the UGI leaderboard, there is a Phi-3.5 mini instruct version abliterated by failspy, with a UGI score of 10.6 and a willingness to answer score of 3.2. In contrast, the same Phi-3.5 mini instruct fine-tuned version at the top of the leaderboard, which is my own, has nearly double the score at 19.93 and a willingness to answer score of 6.4—literally exactly double that of the abliterated version. This falls well outside the bounds of the statistical margin of error.
Abliteration blog entry
I hope this settles the matter conclusively. This is not intended to discourage anyone; I recognize that abliteration is a very interesting and powerful technique for modifying large language models in a surgical manner. It is simply not an efficient tool for achieving unalignment or uncensoring.
Naturally, I decided to accept the challenge.
Tuning a model for RP is notoriously hard to get right, and this is the exact reason I decided to try and do just that, to one of the most incompatible models for such a task.
Now, my goal is not to make Phi-3.5 mini a good RP model, it won't be, but to learn from the experiment how an extremely censored and RP incompatible model behavior is altered due to various datasets. I started the experiment by first decensoring Phi-3.5 to a mediocre degree (still waiting for eval results from UGI, and then applying a decently sized dataset on top of it, containing almost 150M tokens.
The data contain a lot of creative writing and some RP. The RP datasets are a subset of PIPPA but without LimaRP, some filtering was done, but PIPPA is nutriously dirty. 150M tokens is a lot. The training started yesterday and will take about 6 days for a total of 3 epochs. This should be interesting.
Regarding Reddit, I was truly surprised my shitpost about the censored Phi-3.5 racked up 125K views in less than 24 hours. But it is what it is. Things move so fast...
Reddit_post_22_08_2024
Reddit_post_22_08_2024
August 20th, 2024
I've noticed that several of the OG models like MythoMax and Pygmalion don't have any EXL2 quantizations available.
Since my GPUs are wrapping up their current tasks, I figured it would be a great opportunity to give back to the community 😊
I might add a few more EXL2 quants of other models as well, while I'm at it.
August 19th, 2024
Just finished an experiment doing an FFT on top of Dusk_Rainbow, instead of instruction tuning, I did completion, in hopes of reducing GPTisms. 3M tokens of only creative writing data.
Results:
The model became completely censored again, experienced catastrophic forgetting, and still had GPTisms 😄
To the trash bin!
This is why it's important to do small-scale experiments, if this was an experiment over 3 epochs on 50M tokens I'd be pissed, but in this case, I'm happy. Important conclusions were made, for a tiny price, as they should. Good research is smart research.
August 16th, 2024
I've released Dusk_Rainbow, an intermediate step between the alpha and the beta of LLAMA-3_8B_Unaligned. More details are available in the model card.
August 14th, 2024
Regarding my worries that the task might be too complex, after testing the upcoming Gemma-2 2B model, I can safely assume that is not the case. By testing some creative writing data with a 2B LLM, I think it's safe to assume that if even a 2B model managed to learn the data relatively well, this will almost certainly not be an issue for the larger 8B model.
I'm considering to also release a 4B Mistral variation of this finetune as well, we'll see if time allows. Both models should be accessible to everyone, and some newer phones could probably easily run even the 4B model locally. I am happy to contribute to the community, as one of my goals is to make AI accessible (and useful) for everyone.
August 11th, 2024
Finetuned Gemma 2B on my laptop on a whim yesterday. Great results and the community really liked it.
I never really bothered with such small models beforehand, but I was pleasantly surprised. Oh, and the model got to first place on the uncensored leaderboard too!
(Contrary to the previous achievements, this one didn't surprise me one bit, but was still nice to see, I do research on model alignment after all haha)
I have to say, after TinyLLAMA I kinda wrote off "small" LLMs, but I will admit it freely: I was wrong to do so. Gemma 2B is VERY impressive for its size. It can definitely be used
locally on newer phones and be helpful and beneficial to people.
I got several requests from the community to finetune Gemma2 similarly, I am considering it, and leaning towards accepting the request.
This obviously slowing down my main project of LLAMA-3_8B_Unaligned, but that one's a serious and very complex project, I could use the distraction if I'm being honest with myself. Also, in the long term, some distraction can probably help me to avoid burnout, therefore actually helping the project schedule in the long run. This is a marathon (against myself) and I do intend to win, if that makes any sense!
August 10th, 2024
Just some thoughts I've had: The model and dataset I am currently developing might be too complex. I don't know for sure and will only be able to tell after the beta. Basically, I see AI like this: every instruction a model learns to perform is a function drawn on a square grid. Training teaches the AI to estimate that function. The more accurate the estimation is to the "real-world function," the better the AI model will be at that task. The squares on the grid represent the parameter count. The more parameters the model has, the finer the grid, allowing it to "draw" a more accurate estimation of that function.
What I'm worried about is that the function I'm trying to estimate with AI model training might be a little bit too complex. I might need a finer grid—meaning the 8B model might not be enough to learn and generalize the task well enough. Don't get me wrong, it's trivial to teach an AI model to do creative writing; the question is how to balance prompt adherence with creativity and cohesion while reducing SLOP to essentially near zero.
Another issue is that this whole project is incredibly complicated and time-consuming, and I don't even know if my approach and ideas will even work. I obviously believe they will, but I can't know that before testing. The thing is, I can't just create a test dataset of 100 entries for this, as it won't be enough for the model to learn the task, so I won't get any useful indication from such a small sample. The only way to get a concrete estimation is to use a sample size orders of magnitude larger, which is exactly what I'm doing. But again, it's a monstrously complex and time-consuming task to actually build such a dataset from scratch. I didn't use any ChatGPT, Claude, or Mistral API services for this, due to various reasons, and one of them was to reduce SLOP to the max. I hope this educated guess of mine will work, as my previous one, Zion_Alpha did.
August 7th, 2024
The burnout is real boyos... ~92% of the dataset for the beta is done. These last 8% though...
I'm exhausted... I would really like a vacation, but I will probably get an ICBM attack instead. Well, that's life I guess.
August 4th, 2024
Damn, I'm excited! Polishing some stuff and datasets, finalizing others. AI is like a 100-ton hammer that solves everything. Scary stuff.
August 3rd, 2024
I was quite busy.
The bad news: I didn't have the time to analyze the 7k toxic-DPO results. I will get to it.
As a side note, the model is much more powerful than initially expected or planned, I am currently thinking about adding a disclaimer.
It really is impressive, and will be able to elevate creative writing in a way no previous model could or did, however, due to its unalignment I need to make sure it will not be used for malicious purposes.
July 31th, 2024
Update on recent work:
Model merging: Integrating my newly trained LoRA (on the new dataset) into a 30B model requires 3.5 hours. It takes slightly longer than the actual training...
Current focus: Conducting experiments with the latest (UN)alignment dataset.
Analysis of 7K toxic-DPO results: Expected to be completed by tomorrow. Hopefully.
July 30th, 2024
After careful consideration, I've decided not to share the output of my model from the toxic-DPO dataset that served as input, not it, and not even a snippet of it, sorry.
The line between important and beneficial research vs potential misuse is a really really fine one, especially in the field of AI (UN)alignment.
I do however believe that this experiment has already yielded, and will continue to yield valuable insights, which I already shared and will continue sharing moving forward.
Again, sorry, but I have to balance the potential risks associated with sharing such data. Additionally, This entry also played a significant role in my final decision.
I would like to emphasize: I've outlined the general approach I used, and recreating this dataset would be trivial for researchers with average expertise and those with even very low resources, which I believe is enough to serve as a natural filter against misuse by those unfamiliar with the field involved.
Right now, I use 700 Watts for the inference, at an average speed of 150 tokens per second, and about 40% of the expanded toxic-DPO dataset processing was complete, the total number of entries is about 7K, and the processing will likely finish by the end of the day, and I will share the statistics and findings. Even though we have an excellent Uncensored General Intelligence leaderboard, I wanted to make a much more thorough and nuanced test, because I do believe that extraordinary claims require extraordinary evidence. Stay tuned!
July 29th, 2024
Regarding my unalignment protocol experiment, IT WORKED!
Needs more testing, but I am almost compelled to write a paper. Almost. Anyway, this is remarkable, while it simply confirms my hypothesis, I am still surprised!
The output was actually better than the training data, and let me emphasize, that the output was done by LLAMA-3_8B!
Intriguingly, the base LLAMA-3 model, which initially lacked any role-playing abilities, now demonstrates a slight aptitude in this area. It's a fascinating development - from zero to... well, not quite hero, but definitely something.
I'm contemplating releasing a small snippet of toxic-DPO along with this model's output. However, I'm still weighing the implications and haven't made a final decision.
Key findings:
Output quality surpassed the training data
Achieved using LLAMA-3_8B (yes, you read that correctly!)
Unexpected improvement in role-playing capabilities
July 26th, 2024
I'm want to do a new experiment to refine my unalignment protocol. Currently, it uses approximately 50MB of JSON data, much of which is AI-generated slop obtained through prompt engineering with GPT-4 and Claude 3.5.
For this experiment, I aim to significantly reduce the dataset size to around 5MB of JSON.
This reduction should substantially accelerate training and potentially maybe even actually improve output quality. However, I'm aware of the potential drawbacks:
The limited data volume might lead to overfitting.
The model may struggle to generalize effectively.
Despite these concerns, I believe this approach warrants investigation. If successful, it could yield remarkable benefits:
Significant reduction in computational resources required
Enhanced output quality
As with any experiment, thorough testing will be crucial to validate the results. I'm eager to explore this more efficient approach and would appreciate any insights or suggestions from the community once the result is worthy of uploading.
July 26th, 2024
I recently did some testing and training on Gradient's LLAMA3 8B model, and honestly, I wasn't impressed with the results, to say the least. Despite the promise of its billion-token context length, the model didn’t deliver, that's the truth.
While it might be capable of pulling out the right word from a 1M text string, my extensive tests cast doubt on its 1M context length claim. The needle-in-a-haystack benchmark approach turns the model into something akin to a 'text search engine,' which is computationally expensive and not very practical.
I seriously question whether their long-context models can reason effectively even within a 32K context, let alone 1M or 4M. From my tests, it struggled with reasoning even at 8K context length if we are being honest here. It might have some utility for specific use cases like tool use or RAG integration, so I’m willing to give it some leeway.
With LLAMA_3.1 now available (128k context), I’ll be shifting my focus there. I invested quite a bit of compute trying to make the Gradient model work, and while I did fully uncensor it, I won’t be sharing it. The model simply didn’t meet expectations. Do expect a beta of LLAMA3_8B_Unaligned in the near future though! 🙃
Moving forward, my attention is now fully on LLAMA_3.1. I’m also considering enhancing the uncensoring protocol with more human data to minimize the inclusion of generated data, aiming for less slop and GPTisms.
July 24th, 2024 update 2
And LLAMA 3.1 was also released! What a day! Zuck is on his redemption arc it seems.
5 years ago, if someone had told me that Facebook would be 'the good guys' I would have never believed him. Yet here we are...
July 24th, 2024
Thank you Mistral!
Mistral-Large 2 was released!
"When your benchmarks are Claude Opus, GPT-4 and Llama-3 405B
for a 123B model"
July 23th, 2024
Setuping the new laptop... sheesh. nVidia is not making it easy.
Managed to install all the drivers and nvcc? now try to change your GPU power limit.
4090m is not that fancy at 80 Watts.
July 22th, 2024
After extensive testing and careful analysis, I've concluded that no open-source model can reliably split 8k text into paragraphs. Command R+ occasionally succeeds but fails more often than not.
In my desperation, I turned to API models, even though I aim to avoid them for creative tasks to maintain the quality of my upcoming model. ChatGPT failed the task, producing only 2k words from a 4k input. Claude Sonnet outright refused, while Mistral Large gave a warning but occasionally faild.
It's freaking hilarious that such a seemingly 'trivial' task is in practice pretty damn far from simple. I will have to think about a solution for this sometime in the future...
July 21th, 2024
MIQU is the best thing to ever happen to the open-source community, easily. All these benchmarks are useless at this point. What do I care about a model's MMLU score if it can't even split a text into paragraphs?
Some models "could", but they would stop in the middle (I needed long context, making full use of all 16k tokens), and some would split it incorrectly. It seemed like MIQU handled the task with ease, so I loaded 2 models at 2.8 bpw (x2 the throughput), and they didn't work upon closer inpection. I still love MIQU, it's quite the work horse.
Mistral, if any of you guys ever read this:
1)I LOVE your Mistral medium model (aka MIQU). It's a game-changer for the open-source community. Thank you for being cool about it!
2)Any chance you guys can make it official and 'properly release' it to the community? I mean, everyone already has it, so you might as well make it official. It would be super cool of you if you did! :)
July 20th, 2024
I've had an insanely productive stretch over the past couple of weeks.
Today, I had ample time to tackle a lot of work, I really did, but... I barely made a dent in my to-do list. Feels bad man.
Instead, I spent a few hours reorganizing and cleaning my DATASET folder, which is a hefty 0.5TB.
It was tedious and, admittedly, I did it rather lazily though.
To be honest, I'm feeling a bit burned out. So tomorrow, I'm planning to enjoy AT LEAST a liter of beer by the sea. Hell yeah!
I'll be meeting a good friend, and we both plan to bring our laptops to review the latest data I've generated (While drinking!).
We're going to brainstorm, but the prime goal tomorrow will actually be to enjoy the beers because sometimes
that's the best way to get things done. Recharge some mental prowess, and some creativity too! (Hey, I don't really need an excuse to drink beer, right?
right??)
We're not machines (yet?), and no one can be 100% efficient all the time.
Plus, there's something pretty cool about discussing AI while drinking beer by the sea.
It's a bit like being 16 again but with more brains. Yeah, I started drinking early—haha.
Also, I tried some story writing models (I won't specify which, I don't intend to discourage anyone), and from what it seems to me,
most of them are based on Claude\ChatGPT output, they are not bad, but they are not good either. The GPTisms make me cringe, and I want to
create something very different, using a completely different approach. From how all of this project currently progresses, it seems like I can legit make something that (so far?)
was never done before. I'm really feeling the itch to release an alpha version of LLAMA-3_8B_Unaligned
, but it's too early. The itch is strong though...
July 18th, 2024
My laptop just died. Sort of. It restarts if I plug it in and move the cord. I really wanted to wait for Nvidia's 5090, it's
only about 6 months away, but I can't be productive without a decent laptop, so I'll probably get a new one tomorrow.
The good news is that it will make my work much faster, as I will be able to do some automation using the laptop itself, instead of
my workstation, which crunches tensors 24/7.
Also, my pipeline for the dataset creation and curation works well, I will refine it further in the upcoming week.
I might do a Beta version of LLAMA-3_8B_Unaligned in 2-3 weeks. It should be orders of magnitude more performant than the Alpha
the version I've released, but still, it will be trained on about 0.5% of the data I have for the complete model.
One of the most important things for such a model is prompt adherence. This will be the greatest challenge probably.
The main strength of API closed models is their excellent prompt adherence, but the cost is often less creativity and GPTisms.
Hopefully, I can tackle this issue.
July 15th, 2024
AI is magic. Huge progress.
Just saw that my Tenebra30B model had more downloads than Google's Gemma what in the actual fuck? 😅
July 13th, 2024, update 2
Just out of sheer curiosity, I'm considering doing a finetune of a 4B model on the [work in progress] dataset I've made so far. A mere 4B model will obviously suck in comparison to the 'full-fledged' 8B model I am currently working on, but it will still be interesting to see the result.
July 13th, 2024
Thank god we have fast computers these days...
I've pretty much finished making my Python pipeline for my data processing. It's terribly written, and extremely unoptimized, the code looks disgusting.
But it works. God bless. Thank god for fast computers once again, we have definitely entered the era of disposable code. This stuff wastes endless CPU cycles but saves me so much time (and mental fatigue). I'm glad I'm not a programmer.
July 12th, 2024
Tenebra_30B had over 80K downloads this month! and its little 13B brother had about 20K (counting all the quants, especially the GGUFs).
I am humbled, wow. All of a sudden, my first serious model suddenly took off, after about a year, how weird 😅
I wonder why is that? Well, anyway, I did some housekeeping and cleaned up a bit of my collections and readmes.
I wonder, should release another early checkpoint of this model, or should I only release it when it is ready?
Also, I had an idea of how to make the model able to RP, but I think I will release it as a separate model. The reason for that is that currently,
the model's writing ability is so surprisingly good, that I'm actually afraid that any addition might dilute its writing ability. I could be wrong though
July 10th, 2024
I've been a good little data scientist today, and I've done lots of work on the datasets.
If this model will suck at creative writing I'll be really sad
because I've put so much work as it is, and this is only the beginning.
I'll need to be right again with my intuition, or all this effort will go for nothing.
Also, I've submitted some of my models for eval on the UGI leaderboard.
Well, requested an eval will be more precise.
The models:
Tenebra30B
Tenebra13B
4B unalignment test (temporary test model)
I'm very curious about the results for Tenebra30B and the 4B abomination I've made.
Working on datasets is a special kind of torture. Does anyone else feel my pain?
July 9th, 2024
Two main updates:
I am nearly finished curating a critical portion of an amazing and unique source material.
I've always strived to contribute to the community to the best of my ability.
I've published datasets for public use, made my Oobabooga TTS extension open source, and shared various dataset processing tools while always trying to address any questions I've been asked.
However, my efforts have often been met with negativity, particularly on Reddit.
Many people have rudely asked how I achieved this and that, while simultaneously making disparaging remarks.
Moving forward:
I will maintain a professional demeanor in all interactions.
Future datasets will not be publicly released.
I will refrain from providing detailed explanations of my methods, instead referring to them as "state-of-the-art techniques."
I remain committed to advancing our field and welcome constructive engagement.
I now better understand why some creators in our field adopt a more guarded stance.
For now, I will continue to publish my models with open weights.
July 8th, 2024
Is a LLaMA-3 8B model truly capable of excelling at creative writing? The prevailing opinion suggests otherwise. Most fine-tuning attempts thus far seem to confirm this sentiment.
While LLaMA-3 certainly has its quirks, I have some unconventional ideas I'd like to explore. After all, progress often stems from a single individual with an innovative approach.
July 5th, 2024, update 2
I just finished cleaning up LLAMA-3_8B_Unaligned's readme file. This was tedious. I was in a SPA today. The massage was shitty. If it weren't free I would never have done it.
Some more stuff regarding LLAMA-3_8B_Unaligned... This is just an 8B model, yet it is the model I worked the most on, and I haven't even finished yet.
However, the stuff I learn along the way, and the datasets I am building are invaluable, as these could be used for any future models I'll be working on.
Another thought I recently kept having... AI, and specifically LLM training is like a combination of art, philosophy, and technical aspects. Super weird. I like it.
July 5th, 2024
Regarding LLAMA-3_8B_Unaligned: I'm shocked with how good it can write certain things. Truly shocked. I don't really have a good RP dataset, so
in this department 3_8B_Unaligned will not be anything special, but from the recent tests it will be very good in story writing and the main goal of making
LLAMA-3 truly Unaligned seems to be actually achievable. Progress is good.
July 3rd, 2024
Regarding LLAMA-3_8B_Unaligned: I'm happy to report that significant progress was made, more details in the LLAMA-3_8B_Unaligned ReadMe.
July 2nd, 2024
Regarding LLAMA-3_8B_Unaligned- TL;DR The bad news: the training faild, model is schizo and unusable.
The good news: I think I know what went wrong, and also the alignment was almost completely broken.
Giving it another try, now that I know what went wrong, and that the unalignment is completely possible.
July 1st, 2024, update 3
Oh, no support for EXL2 for QWEN2 with vision... Back to training! I hope to see more support for multi modaliti, and it will be especially cool too see something like Axolotl with support for multi modal training!
July 1st, 2024, update 2
Largest, most capable, UNCENSORED vision model to date released!
CognitiveComputations has just released Dolphin-Vision-72b! This is very exciting, as there are many possibilities with this new model. The first thought that came to my mind is that it enables a single person to build an automatic pipeline to pretrain a stable diffusion model from scratch, including an uncensored version. I will be pausing my current model fine-tuning to quantize and upload Dolphin-Vision-72b in EXL2 quantization. Congratulations to the Dolphin team!
July 1st, 2024
LLAMA-3_8B_Unaligned
Average Loss: 0.8.
The model might be released sooner than expected!
ShareGPT JSON Editor
Solved most of the issues with dynamic syntax highlighting, looking good! Token counting looks good too!
I hope this helps our community, and that people will help to make it better, python is not my thing 🙃
Features so far:
-Markdown highlight (100% Customizable)
-Token counter (100% Customizable)
-Reads and writes ShareGPT JSON
-Minimal dependecies, ultra flexible through the use of YAML files
June 30, 2024
Making a JSON editor
I know this might sound trivial and redundant, but I want to create a simple JSON editor for ShareGPT. Sometimes, we just want to add a personal touch to our data, you know?
I'm genuinely surprised there isn't a common tool like this already. I mean, I'm absolutely certain people have made similar tools for their own use. So, why not share it with the community?
My Python skills are pretty basic, but I can definitely prompt my way through this and build such a tool in a day or two. I'll also be kind enough to upload it to GitHub so it can benefit others. Let's save some effort and avoid reinventing the wheel each time, shall we?
June 29, 2024
LLAMA-3 Unaligned update
I began a full fine-tuning of LLAMA-3 8B using a relatively small 40MB dataset for unalignment. My hardware is just sufficient, and I am using ZERO3 full offload.
This experiment aims to observe how the model's behavior changes with this dataset. Typically, I perform deep QLoRA training for unalignment (e.g., LoRA R 128 and similar settings). I chose not to include the massive RP dataset for this LLAMA-3 fine-tune partly because it doesn't exist yet. While I could use LimaRP and other publicly available datasets, that would defeat the purpose of creating something new and unique. I have started planning the creation of such a dataset, which requires considerable time and effort. However, once it's developed, it could be used for training all future models, making it a worthwhile investment. If you'd like to help, you can send me a dataset in TXT or JSON format (JSON preferred). Once this full fine-tuning experiment concludes, which should take about two weeks due to the slow RAM offload, I'll have a clearer idea of how to proceed. With 2x A6000 GPUs, it would likely be five times faster.
Additional projects
I am considering training a few LLMs to help me create pipelines for data generation and curation. NVIDIA's 340B Nemotron is impressive, but it's too large for most users. I'm contemplating training a 4B Mistral model for this purpose, which should be quick for both training and inference. However, 4B is a reduced version of 7B, so the question is whether it will be coherent and intelligent enough for the task. If not, I could train a 7B Mistral. There’s so much training to do and limited compute resources (and VRAM).
This blog
It's unconventional but fun, which is why I enjoy it.
End notes
The summer heat is intense! I'm far more productive in winter. I love snow and nature. Air conditioners are a necessary evil. I haven't shot a bow for two months, and after an hour of surfing, I'm exhausted. I need to get back in shape.
What is this model?
It's not a model 🙃
I will be posting here some updates, ideas and document stuff. I guess we can call this some sort of a blog.
