A Ukrainian developer "crammed" a language model into a microcontroller for $10. How did it work out, and what can it do?
A Ukrainian developer launched a language model on a microcontroller for $10. It works without the Internet.
A Ukrainian developer launched a language model on a microcontroller for $10. It works without the Internet.
Ukrainian developer slvDev has launched a language model with almost 29 million parameters on an ESP32-S3 microcontroller, Highload reports. The board costs less than $10, and the model generates text entirely locally — without connecting to servers or cloud services.
The project is called ESP32-AI. According to the developer, the model generates about 9.88 tokens per second and displays text on a small screen connected to the board. In the video, we can see how the model creates a short story about a girl named Lily. «Yes! It works! It’s alive! It’s incredible!» exclaims its developer.
The ESP32-S3 has only 512 kilobytes of fast RAM. This is not enough to run a model with such a large number of parameters. Therefore, the developer compressed the model to a 4-bit format, reducing its weight to only 15 megabytes. In addition, not all of the model uses RAM. Most of its parameters are stored in flash memory, so during generation the model can simply load from there small fragments needed for the next token.
The model is built on the TinyStories dataset, so it can only create small stories. So it can’t answer questions or follow other instructions. But the developer says the most interesting thing about this experiment is that this big model was able to fit on a tiny chip, not what the model can do.
The developer also created another similar model. It’s called Barista and can answer questions about making espresso. slvDev shared a video where he tells the model that his espresso tastes burnt. The model concludes that it’s most likely due to over-extraction. Perhaps the coffee was in contact with hot water for too long, or it’s ground too finely. And then Barista asks about the dosage, weight of the finished coffee, and extraction time.
an espresso Q/A model running fully offline on an ESP32S3
by u/slvDev_ in LocalLLaMA
The developer also published the firmware code, tools for training the model, and instructions on Github so you can recreate the experiment yourself.
Open Ukrainian language model Lapa LLM has received a public release
Open Ukrainian language model Lapa LLM has received a public release
The Ukrainian language model «Syayvo» has begun to be tested in closed mode
The Ukrainian language model «Syayvo» has begun to be tested in closed mode
Read the country’s main IT news in our Telegram
Read the country’s main IT news in our Telegram
