Check out this port of microgpt to C, posted 5 months ago. It got a 2500x speedup over the python version. https://github.com/moebiusV/cugpt
pjmlp 53 minutes ago [-]
Of course, an interpreted language implementation, versus a compiled language designed to port UNIX.
Even if counting that it is basically calling into C or C++ libraries, there is lots of boilerplate that the CPU has to consume, that just won't happen on the pure C implementation.
Retr0id 17 hours ago [-]
> The most atomic way to train and inference a GPT in pure, dependency-free C.
What sense of the word "atomic" is meant here?
elromulous 17 hours ago [-]
No dependencies, self-contained
Retr0id 12 hours ago [-]
My read was that it meant "small", since they already say "dependency-free" in the second half of the sentence. But it's also a strange way to say "small" so I'm still not sure.
ilaksh 18 hours ago [-]
This is not an LLM obviously
, it's just for generating random names. But interesting to think of the possibilities of truly tiny language models if there were connected together.
alightsoul 17 hours ago [-]
It's an slm (small language model) due to number of parameters and it uses the same architecture as an llm, but llms have billions of parameters
nomel 16 hours ago [-]
Hasn't it been repeatedly shown that many small models perform worse than a large model of the same total parameters?
reilly3000 15 hours ago [-]
I think that is a sort of a reverse scaling fallacy. Given the right resources and environments, many small models can function together in an emergent way. I’ve been on the lookout for an SLM version of Conway’s game of life. SLM always reminds me of slime molds, which demonstrate a form of intelligence which is remarkable.
nomel 14 hours ago [-]
> Given the right resources and environments, many small models can function together in an emergent way.
What is this based on? Every researcher I've heard talk about this says it's exactly not true, as an uncontested rule, because the larger models will more effectively contain the smaller models, and use them together in ways that the connections between the smaller models can't. Remember, even MOE is to save compute/memory, not to help performance/parameter.
grumbelbart2 14 hours ago [-]
That is my understanding as well. Thousands of monkeys do not equal or surpass a man, intellectually, even if working together. There is some intrinsic super linear scaling in intelligence.
0x20cowboy 12 hours ago [-]
Does that mean an African elephant 2.57×10^11 (neurons) is smarter than a human 8.6×10^10 (neurons)?
nomel 8 hours ago [-]
You're comparing completely different training data, harness overhead, and cost function. You're also comparing brains, which aren't really related to this discussion at all.
But, it depends on what you're measuring. By spatial/navigational memory, yes, elephants are far far better. Reasoning, no. It would be interesting to see what an elephant or whale eugenics program could result in, since humans have that pesky (or maybe instrumental?) birth canal problem.
fl4regun 11 hours ago [-]
This is not a good comparison, because the brain doesn't only do "being smart" - it has to do things like innervate muscle and other tissue through the body, elephants will require more neurons given their larger size, just to be able to *walk*
ilaksh 11 hours ago [-]
This is not a normal small model. It's in a different category of tiny.
dcow 18 hours ago [-]
Is token rate a function of parameter size?
unrahul 16 hours ago [-]
Yes, a quick back of the envelope math is 0.65 * (memory bandwidth of the card / (model weights in bytes + kv cache in bytes) ~ practical decode tps. Below context around 32k (depends upon the model but again can be used as a placeholder number) you can ignore the kv cache in bytes and the math becomes just about memory bandwidth and model weights in bytes.
rbanffy 17 hours ago [-]
Not quite linear, but yes.
giantrobot 14 hours ago [-]
Parameter size and total number of parameters so ultimately the total size of the model in memory.
This leads to some interesting optimizations. You can quantize all the parameters (or certain layers) of a model and halve or quarter the memory requirement but maintain most of the model's intelligence. This increases the token rate inversely with the size reduction.
Popular quantizations for local models are 8-bit and 4-bit parameter sizes. The Blackwell series of nVidia chips now even support native FP4 math making 4-bit quantizations even faster.
elzbardico 8 hours ago [-]
LLMs are an imprecise, more of a marketing term, to define Transformer models based on the self-attention mechanism, trained with massives amounts of data.
And this implements a transformer. Actually it is a very cool didactic example.
16 hours ago [-]
api 17 hours ago [-]
Isn't a MoE model basically a cascading tree of smaller models or some variation of that?
unrahul 17 hours ago [-]
You could think of it as a standard decoder only LLM (almost all modern ones we use everyday), with some layers (experts) having parallel networks and conditionally based on the input token (per token) - the token is routed through some of these layers. In the case of a non MoE (dense) - each token goes through all layers, so the inference engine has to read all the layers and do a matrix (layer) times vector (token) computation, while in the case of MoE the number of layers per token that has to do the compute is substantially lesser, so one can expect much higher tps than a dense model at the same number of parameters (size - 7B, 27B etc)
hasteg 14 hours ago [-]
This is pretty cool. Implemented something similar myself (a really small language model with ~10M params) just to teach myself the ML behind the LLMs. Did not implement it in C obviously, just use PyTorch, but it's interesting to go through the c file and see how he has implemented stuff I took for granted in Python in C.
Anyway, I just tested this out myself on my AMD Ryzen 9 9800x3d. I got 7647173 tok/sec using karpathy's Shakespeare dataset https://raw.githubusercontent.com/karpathy/char-rnn/master/d.... Going to play around with it and see if I can get a CUDA kernal built to see what it could do on a 5090. Claude estimates with napkin math that we could get around 2B tok/s
pkilgore 17 hours ago [-]
Honestly not sure this is impressive. I ported microgpt to zig as a learning exercise, then moved scalar engines to NEON/metal just to see what happened. Besides metal being slower (I probably did something wrong, but it could be due to the fixed costs of memory transfer into the GPU not being worth it due to the small model).
Anyways, it was also stupid fast, particularly compared to the python version. But I was pretty sure that's irrelevant to real production architectures!
aa-jv 2 hours ago [-]
Is your zig microgppt port available anywhere for perusal? I'm interested in both at the moment, have some zig workshops with a few local folks on the horizon and would love to use that opportunity to boot up the noggin on a few things.
>fast .. irrelevant ..
I ask myself this question all the time, but I'm not sure I'll like the answer: have we really attained peak performance in the engines, if we're "using python for everything" .. ?
aa-jv 2 hours ago [-]
I often wonder if there aren't some low hanging fruit in the nature of the python ecosystem surrounding AI/ML these days. I'm sure there are a lot of eyeballs looking at the problem, but every time I fire up a chat and see the python loadout, it "feels" like there might be work yet still to be done to make things as fast as possible.
And then, while I'm waiting for a response on things, the mind wanders to thinking about what the rust camp are doing, with regards to AI/ML.
To my addled mind, it really seems like the first thing to be done when getting the models firing, is turn them on their own tools and optimize, optimize, optimize. I'd sure like to know what the frontier labs are doing to squeeze those ergs out of our substrates.
Does anyone have clues what the landscape looks like outside the python tooling, vis a vis higher-performance infrastructure? Being glib about it I admit, it just seems 'odd' that the C/C++/Rust camp are letting scripting languages drive the AI/ML ship.
pjmlp 49 minutes ago [-]
Mostly because many on that camp are averse to nice tooling, like REPLs, JITs, or incremental compilation.
They do exist, but again, not really loved enough.
Additionally, while in the past there are been nice high level libraries for C++, which is where Java took its inspiration from in first place[0], current culture seems to hate having such frameworks around, and they are mostly gone nowadays.
[0] - It isn't writing Java in C++, rather how C++ inspired by Smalltalk used to look like, 1983 - 2000's, before Java got released in 1996.
gok 15 hours ago [-]
It could probably go quite a bit faster using Arm SME. The entire network could fit into the ZA register.
throwa356262 18 hours ago [-]
And the 5 years old AMD Ryzen 5 5600H is doing 7M?
Am I reading this right? Then I need to try this on Strix Halo
rbanffy 17 hours ago [-]
And it's only using AVX-2 and not AVX-512, AMX or ACE. Or built-in GPUs and NPUs (the M series doesn't emphasize matrix multiplication on the CPU side because it already has matrix multiplication units on the GPU, which is always attached).
throwa356262 14 hours ago [-]
The NPU in previous AMD generation has unfortunately it's own dedicated RAM that is far too small for an LLM.
But even those should comfortably run an SLM of around 100-400M parameters at crazy high speeds and with minimal power usage.
rbanffy 13 hours ago [-]
> an SLM of around 100-400M parameters at crazy high speeds and with minimal power usage
Software support for that kind of usage, when available, will be immensely impactful.
ranger_danger 15 hours ago [-]
But do processors actually offload any CPU opcodes to their GPU? That could be quite useful if it can be used to improve execution speed.
rbanffy 13 hours ago [-]
No, but if you are guaranteed to have a GPU or NPU packaged together it becomes like the SPUs in the Cell processor - you can’t offload instructions (unless you use a trap mechanism to a subroutine) but you can have code that hides the setup, the different ISA, and the result retrieval, behind an API call.
bigyabai 17 hours ago [-]
Before the M5, there was no dedicated matrix multiplication hardware on the Apple Silicon GPU. Their solution was generally using the NPU and AMX coprocessors for tensor and matrix workloads.
saidnooneever 16 hours ago [-]
[dead]
oivaksef 12 hours ago [-]
How much of this translates to Asahi?
Always been disappointed by the answer to that question in the past - fingers crossed this time
fwip 18 hours ago [-]
Model is 4K parameters - I don't know enough about that size of model to know if this impressive or not.
altcognito 18 hours ago [-]
It's a trivial example. This won't be useful outside of a VERY specific domain without more parameters. Many people need to know about the bitter lesson.
Over time, I'm sure we'll be able to filter information better and get parameter counts down, but I wouldn't count on that within the next 6 months.
odo1242 17 hours ago [-]
The point here is that the library's overhead cost is very low. The fact that a tiny model can reach 10M tokens per second means that the overhead of token decode, memory allocation, calling the model, etc. is very low. The model doesn't actually need to be useful to prove that point.
voakbasda 17 hours ago [-]
It’s interesting and worthy of genuine applaud for being a good starting point for further work.
That said, I am more interested in what size model this could manage while producing “just enough” tokens per second to work at a conversational rate. What are models in that class capable of doing for me?
entrope 16 hours ago [-]
Is it that impressive? It is a model generating short strings from scratch, so I do not think there is significant token parsing or memory allocation going on.
In compute performance, 10M tok/s * 4096 parameter/tok * 4 byte/param = ~160 GiB/sec implied memory bandwidth, vs nominal ~300 GB/sec for the M5 Pro's RAM. That seems even less notable once we consider that 4096 parameter * 4 byte/param = 16 KiB of parameters fit easily within L1 data cache (the M5's efficiency cores each have 64 KiB of L1 data cache). 40 GFLOPs means ~10 FLOP/cycle, about 2.5 NEON instructions per cycle given that the core instructions are fused multiply-adds (FMAs). A random web page I found says the M5 family cores have a 4-wide SIMD block, meaning this gets about 63% utilization: respectable but not super high.
altcognito 16 hours ago [-]
Thank you for your kind reply. I appreciate your point completely and while I tried to moderate sounding dismissive of what was being done here, I think I could have done better.
I love "trivial" examples and everything you've said is tue.
alightsoul 17 hours ago [-]
I think the bitter lesson only talks about task performance but not computational efficiency. Could tiny models improve efficiency? Maybe by just using a general architecture on specialized data, so the artichecture itself is not task specific?
mcbuilder 16 hours ago [-]
Small # of parameters means no memory bottleneck, which means blazing fast performance.
andai 16 hours ago [-]
Wait, does this fit in L1?
fwip 13 hours ago [-]
L1 cache for a M5 is like 128KB, right? So I can't see how it wouldn't.
dyauspitr 7 hours ago [-]
Imagine 10 billion tokens per second even if we make no further improvement in frontier LLMs than what we have today that would be paradigm changing.
Rendered at 09:47:58 GMT+0000 (Coordinated Universal Time) with Vercel.
Even if counting that it is basically calling into C or C++ libraries, there is lots of boilerplate that the CPU has to consume, that just won't happen on the pure C implementation.
What sense of the word "atomic" is meant here?
What is this based on? Every researcher I've heard talk about this says it's exactly not true, as an uncontested rule, because the larger models will more effectively contain the smaller models, and use them together in ways that the connections between the smaller models can't. Remember, even MOE is to save compute/memory, not to help performance/parameter.
But, it depends on what you're measuring. By spatial/navigational memory, yes, elephants are far far better. Reasoning, no. It would be interesting to see what an elephant or whale eugenics program could result in, since humans have that pesky (or maybe instrumental?) birth canal problem.
This leads to some interesting optimizations. You can quantize all the parameters (or certain layers) of a model and halve or quarter the memory requirement but maintain most of the model's intelligence. This increases the token rate inversely with the size reduction.
Popular quantizations for local models are 8-bit and 4-bit parameter sizes. The Blackwell series of nVidia chips now even support native FP4 math making 4-bit quantizations even faster.
And this implements a transformer. Actually it is a very cool didactic example.
Anyway, I just tested this out myself on my AMD Ryzen 9 9800x3d. I got 7647173 tok/sec using karpathy's Shakespeare dataset https://raw.githubusercontent.com/karpathy/char-rnn/master/d.... Going to play around with it and see if I can get a CUDA kernal built to see what it could do on a 5090. Claude estimates with napkin math that we could get around 2B tok/s
Anyways, it was also stupid fast, particularly compared to the python version. But I was pretty sure that's irrelevant to real production architectures!
>fast .. irrelevant ..
I ask myself this question all the time, but I'm not sure I'll like the answer: have we really attained peak performance in the engines, if we're "using python for everything" .. ?
And then, while I'm waiting for a response on things, the mind wanders to thinking about what the rust camp are doing, with regards to AI/ML.
To my addled mind, it really seems like the first thing to be done when getting the models firing, is turn them on their own tools and optimize, optimize, optimize. I'd sure like to know what the frontier labs are doing to squeeze those ergs out of our substrates.
Does anyone have clues what the landscape looks like outside the python tooling, vis a vis higher-performance infrastructure? Being glib about it I admit, it just seems 'odd' that the C/C++/Rust camp are letting scripting languages drive the AI/ML ship.
They do exist, but again, not really loved enough.
https://clang.llvm.org/docs/ClangRepl.html
https://github.com/jupyter-xeus/xeus-cling
https://root.cern/cling
Additionally, while in the past there are been nice high level libraries for C++, which is where Java took its inspiration from in first place[0], current culture seems to hate having such frameworks around, and they are mostly gone nowadays.
[0] - It isn't writing Java in C++, rather how C++ inspired by Smalltalk used to look like, 1983 - 2000's, before Java got released in 1996.
Am I reading this right? Then I need to try this on Strix Halo
But even those should comfortably run an SLM of around 100-400M parameters at crazy high speeds and with minimal power usage.
Software support for that kind of usage, when available, will be immensely impactful.
Always been disappointed by the answer to that question in the past - fingers crossed this time
https://en.wikipedia.org/wiki/Bitter_lesson
Over time, I'm sure we'll be able to filter information better and get parameter counts down, but I wouldn't count on that within the next 6 months.
That said, I am more interested in what size model this could manage while producing “just enough” tokens per second to work at a conversational rate. What are models in that class capable of doing for me?
In compute performance, 10M tok/s * 4096 parameter/tok * 4 byte/param = ~160 GiB/sec implied memory bandwidth, vs nominal ~300 GB/sec for the M5 Pro's RAM. That seems even less notable once we consider that 4096 parameter * 4 byte/param = 16 KiB of parameters fit easily within L1 data cache (the M5's efficiency cores each have 64 KiB of L1 data cache). 40 GFLOPs means ~10 FLOP/cycle, about 2.5 NEON instructions per cycle given that the core instructions are fused multiply-adds (FMAs). A random web page I found says the M5 family cores have a 4-wide SIMD block, meaning this gets about 63% utilization: respectable but not super high.
I love "trivial" examples and everything you've said is tue.