The race to speed up AI inference is intensifying, and while custom silicon has captured the spotlight, a French startup is convinced that the GPUs already running in data centers have untapped potential.
Kog, a French company, is betting that conventional accelerators can be pushed far beyond their typical performance for inference workloads. The startup’s approach challenges the assumption that purpose-built chips are necessary for extremely fast response times.
Cerebras, a maker of specialized inference hardware, received a warm reception from investors when it went public in May. Its IPO debut underscored the market’s appetite for faster AI inference. But Kog argues that enterprises don’t necessarily need new hardware to achieve dramatic speedups.
In May, Kog’s tech preview climbed to the front page of Hacker News. The preview was designed to demonstrate that “extremely fast single-request decoding is possible on the standard datacenter GPUs enterprises already own.”
For the demonstration, Kog used AMD MI300X and Nvidia H200 GPUs, two of the most widely deployed datacenter accelerators. The claim is notable because it suggests that optimization and software innovation can extract performance gains without replacing infrastructure.
Kog’s bet is that the next wave of inference speedups will come from software and systems engineering rather than silicon alone. Whether the startup can deliver on that promise at scale remains to be seen, but the early buzz suggests there is room for improvement in how existing hardware is used.
