There wasn’t any huge news this week, but lots more minor news. I was especially impressed with the rate of progress in the open-source LLM community. It seems that some new LoRA technology is released weekly to either speed up LLMs, increase the context window, or reduce system requirements. This week, we got some of all of that!
Open-Source LLM Updates
The most exciting news for this past week is the multiple innovations made in the open-source LLM community. Here’s a breakdown of everything:
SuperHOT - A new LLM approach that allows for 8k and even 16k context windows, compared to the typical 2k. Even longer prompts and responses are possible without losing performance or quality. This all started with another research paper out of Meta, found here. I’ve been beyond impressed with Meta’s contribution to the open-source AI community and am thankful they have decided to take such an open approach. TheBloke has already been hard at work re-releasing models using this new technique.
Ex-llama - A new model loading technique called exllama was released that allows for vastly increased inference speeds and reduced VRAM requirements for the same models as compared to AutoGPTQ and other techniques.
Salesforce X Gen - Even Salesforce is getting into the open-source LLM game! X Gen is a new model from SF is called X Gen and is a 7B parameter model trained on 8k context lengths.
Training on longer contexts seems to be the key innovation to increase the context windows for LLMs. It turns out that if you train on smaller-length data, the context windows will be smaller. And the inverse is also true. Longer context training results in more extended possible contexts.
AI Generative Art
Generative AI Art had a huge week. First, SDXL 0.9 was released by Stability AI, the new version of Stable Diffusion, and a considerable improvement over their previous version in terms of quality. In my opinion, this puts StableDiffusion and open-source generative AI art models on par with Midjourney.
Draggan
Also from the world of generative AI art, Draggan’s code was released! This project gained a lot of attention a few weeks ago for how impressive the demo was, but only now is the actual code released and people are using it. I am in the process of setting it up on my computer, but it hasn’t been easy. I'll create a tutorial video as soon as I figure it out.
Big Acquisition - MosaicML
MosaicML, the company behind one of the best open-source LLMs MPT30b, was acquired by Databricks, another huge AI company. The transaction was valued at $1.3b and set up Databricks as a major AI player.
AI Video Games Set Back
Valve has banned any games that incorporate AI-generated content in a huge set back for AI in video games. Their main concern is that it’s challenging to contract copyright issues for AI-generated content, and they don’t want to be exposed to legal risk.
