Suppose an invoice service has used the same small model for months, and it still extracts the fields correctly. We could switch models, but have no reason to. Taalas has built a chip around one model, raising a question for services like this: how much GPU flexibility are we paying for but never using?
Why all the fuss about ChatJimmy?
ChatJimmy gives us a way to try HC1, the model-specific chip from Toronto-based Taalas, co-founded by Ljubisa Bajic. Underneath the chat window is Meta's Llama 3.1 8B. An unusual choice for a demo attracting this much attention, until we look at the response times.
During the February 2026 launch, Simon Willison, co-creator of Django and creator of the data-exploration tool Datasette, described responses appearing almost instantly on 20 February. His account concerns the wait for an answer. It doesn't mean this small model can do everything a frontier model can.
What happened in the tests?
Taalas advertises 17,000 tokens/s per user on Llama 3.1 8B. Here, we are looking at one user's output, not adding together everyone on the server.
In Eric Cappannelli's tests dated 20 February, a maths prompt registered 15,752 tokens/s in 0.032 seconds, and a creative-writing prompt 15,703 tokens/s in 0.082 seconds. He includes the prompts and answers. The maths answer, in his account, adds up to 41.82 hectares when the problem asks for 58. The demo's generation statistics show an extremely quick answer, but the answer still needs checking.
Sherman Yang's benchmark on 26 February measured 20 direct API calls. They averaged 0.857 seconds, with 1.526 seconds p95. That includes more of the request than generating the answer. The prompts differ from Cappannelli's too, so these aren't competing measurements of the same run.
Our invoice can spend time travelling to the server, waiting in a queue and having its text read before the answer starts. We need that whole wait, especially when many invoices arrive together. Neither report gives us a sustained production average.
Taalas' comparison chart combines its lab results with a NVIDIA baseline and Artificial Analysis figures under a 1k/1k caption. I can't reconstruct a matched test from the settings shown there. For a GPU comparison, we'd need the same model and precision, the same prompt/output lengths, and the same number of simultaneous requests.
HC1 also uses mixed 3-bit and 6-bit parameters, and Taalas acknowledges quality degradation. If our USD 100 invoice comes back as GBP 100, someone has to correct it. Paying for that correction can change which service is cheaper.
The GPU has the model. Why is it waiting?
The invoice text is available at the start, so the GPU can process many of its tokens together. This is prefill. It doesn't yet have the answer, though. During decode, each new output token normally depends on the tokens generated before it. With a small batch, there may be plenty of arithmetic capacity left while the GPU waits for data. NVIDIA's inference explanation describes this memory bandwidth bottleneck.
For this example, we've already loaded the model's learned weights into GPU memory, often high-bandwidth memory (HBM). We aren't reading the model from disk again for each token. The weights still have to move from memory to the units using them, with caches and reuse saving some of those trips.
Reading the invoice also produces attention keys and values. Later tokens need them, so we keep them in the KV cache rather than calculate them again. These belong to this request; they aren't the model's learned weights. Longer conversations leave more to store, and simultaneous requests each need their own state. NVIDIA discusses that memory pressure here.
Think of a chef collecting ingredients for several orders in one trip to the pantry. Batching makes a similar saving: several requests use the weights brought from memory. Buying a faster knife wouldn't shorten the walk; adding arithmetic capacity alone won't remove the memory traffic either.
PagedAttention helps by wasting less KV-cache space. Another option, disaggregated serving, puts prefill and decode on separate hardware. We can then allocate more capacity to reading long documents or producing long answers, depending on the workload. There is still useful optimisation to do on GPUs.
Put the model into the chip
In Taalas' architecture description, the base model is hardwired into a chip that combines model storage and computation without relying on HBM. The base weights no longer make those repeated trips from external DRAM. In our kitchen example, the counter has been built for one menu, with its ingredients close at hand.
Compiling a GPU program leaves us with software for an existing processor. Taalas takes the model into the chip design itself, before manufacture.
Gold shows model preparation and weights, purple shows request data and adapters, and green shows computation and output. The next invoice has different text. HC1 still needs memory for that changing request state.
Taalas says it can take a new model to hardware in around two months. Before booking a launch, I'd ask when our chip would be ready to use. The public write-up explains the idea, but not enough of the compiler or chip design to reproduce it.
HC1's published specifications list TSMC 6nm, an 815 mm2 die, 53 billion transistors and a 2.5 kW server. Here, 2.5 kW refers to the server, not just the HC1 chip. We shouldn't read a phone-sized implementation into these specifications.
What happens when we need a different model?
Suppose customers start sending invoices in a format our model handles badly. On a GPU, we can try another supported checkpoint that fits. We cannot load an arbitrary replacement into HC1's fixed base model. We either find a supported way to adapt it or move the work to other hardware.
There is some room to adapt. Taalas advertises configurable context and LoRA support, and its API documentation exposes an adapter state. A LoRA adds a small trained update alongside the base weights. The original base model remains, so this isn't the same as swapping one model family for another.
Or the model keeps working, but half the customers leave. There are now fewer paid jobs to recover what we spent designing and making the chip. We still own the same hardware.
Taalas' launch claims include 20x lower build cost and 10x lower power. For our budget, we'd need a quote for the service or hardware and a test with our volume of work. The launch figures don't give us that bill.
Lower energy per useful result could be important for power-constrained data centres. If we measure energy for the whole server, we must count the whole server's output over the same period and load. Dividing its power draw by one user's token rate leaves out the work it does for everyone else.
Aren't other companies already doing this?
Yes, but "not a GPU" covers very different designs. An ASIC, an application-specific integrated circuit, need not contain a permanently fixed model.
With NVIDIA GPUs or AMD Instinct, we buy programmable processors, HBM and a software stack. AMD's stack includes ROCm. If we later choose another supported model that fits, we can change the weights without ordering a chip made for it.
Google TPUs narrow the job to machine-learning computation, using matrix arrays and HBM. That still leaves a choice of models to run. A processor can be specialised for the maths without having one model's weights fixed during manufacture.
Keeping weights nearby is also central to Groq LPUs, with compiler-scheduled execution and on-chip SRAM. Cerebras takes this to wafer-scale hardware with large amounts of SRAM. Neither approach requires permanently hardwiring one checkpoint as HC1 does.
Then there are different ways of organising the work itself. SambaNova maps models onto reconfigurable dataflow hardware with tiered memory. Tenstorrent uses programmable Tensix compute and software-controlled data movement. Both leave room to change the mapped workload on existing hardware.
We'd still need matched tests to rank their speeds. Even equal token counts can represent different amounts of text if the tokenisers differ. The model and serving settings matter too.
So, is NVIDIA actually in danger?
A company could keep training on NVIDIA hardware while moving one busy inference service elsewhere. NVIDIA would still have a customer, but a smaller serving order. Taalas doesn't have to replace every GPU workload to compete for that spending.
On 6 August 2026, AMD announced a definitive agreement to acquire Taalas, subject to closing conditions and regulatory approvals. AMD plans to combine the technology with Instinct GPUs. It sees a place for both.
NVIDIA has made its own move. Groq announced a non-exclusive technology licensing agreement in December 2025. On 24 August 2026, NVIDIA announced Groq 3 LPX in full production, extending its Vera Rubin platform with specialised inference hardware.
NVIDIA can therefore offer specialised inference alongside its GPUs, software and networking. Customers who need to keep changing models still have that option. It could lose orders or face lower prices, but HC1 hasn't shown that it can replace frontier-model training, and NVIDIA is already selling an alternative to GPU-only serving.
Cheaper inference may also let a team build something it couldn't previously afford to run. That could grow total demand even if fewer of those requests go to GPUs. It doesn't tell us what will happen to NVIDIA's share price.
Back to our invoice service
Our invoice example has a catch. Millions of similar documents could keep one small model busy, but extracting a few fields produces a short answer. If most of the delay is in reading the document or waiting in a queue, faster decoding won't remove much of it.
An agent that writes long intermediate answers has more decoding to speed up, possibly over several steps. We are leaving its tools outside the chip comparison. A database query still takes time even if the model asks for it more quickly.
This is a proposed routing pattern. It does not show a commercially available HC1 deployment.
The business requirement comes first in Building Scalable Data Platforms too. A hyperscaler may already have enough recurring work for custom hardware. A startup still changing its product might replace the model before it recovers the cost of the chip.
An enterprise using the service needs isolation, support and a workable plan for changing versions. For an edge device, we'd need separate evidence about size, power and operating conditions. The server demo doesn't answer those questions.
Suppose we process 1,000 invoices. Some need another attempt, some go to a fallback model, and some need a person to correct them. We pay for all of that, but the extra attempts don't create extra completed invoices. Include operating and eventual hardware-replacement costs, then divide by the invoices we accept: our cost per accepted result. Alongside that cost, test p95 completion time, errors and energy with realistic simultaneous traffic. The manual recovery work is familiar from Building Durable Data Pipelines. We haven't removed it by putting a model into the pipeline.
What changes if this reaches 70B models?
Perhaps the 8B model can't handle our invoices, while a 70B model can. A fast implementation of that larger model would then be much more interesting to us. We'd need its own measurements, though; the 8B demo can't supply them.
There are more parameters to store and more computation to do. If the model spans chips, those chips must communicate. Longer contexts also need more space for request state. Yield and packaging affect the cost of manufacturing the system, and model lifetime affects how much use we get from it. The sources linked here don't provide a comparable, independently verified 70B+ Taalas result.
Someone still has to choose which part of the platform can stay fixed and which part needs to change. That is why I included architecture in 5 IT jobs AI will replace, and 5 it will not. Faster implementation doesn't make that decision for us.
Conclusion
I'd consider HC1 for work where the model already passes our checks and we expect to keep using it. The trial would need the difficult documents, the busy periods and the costs of correcting mistakes, not just a token counter. If the saving holds, that is inference work a GPU supplier could lose. Before committing, we still need an answer to one awkward question: what will we do if the next model is the one our customers actually need?
Resources
Architecture, quantisation and economic claims (Taalas); API documentation.
First-hand February demo account (Simon Willison); author background.
Inference optimisation (NVIDIA); KV-cache optimisation; Dynamo disaggregated serving.
Licensing agreement (Groq); Groq 3 LPX production announcement (NVIDIA).
TPU architecture (Google); Groq processor research, author-uploaded copy (Abts et al.).
Wafer-scale inference (Cerebras); dataflow architecture (SambaNova); Tensix architecture (Tenstorrent).
Individual ChatJimmy tests with prompts and generation timings (Eric Cappannelli).
Comments