Microsoft’s new 1-bit AI models challenge the need for cloud-based large language models

Microsoft Research unveils BitNet, a set of 1-bit large language models designed to run efficiently on consumer CPUs, signalling a shift towards decentralised AI with lower memory, energy use, and latency.

The latest argument for on-device artificial intelligence is not that models have become smaller in an absolute sense, but that they may no longer need to be so large in the ways that matter most for deployment. For years, the default assumption has been that serious AI belongs in the cloud, where vast clusters of accelerators can absorb the memory and compute demands of modern large language models. That assumption is now being challenged by a growing body of work on quantisation, ternary weights and more efficient inference on ordinary CPUs.

At the centre of that shift is BitNet, Microsoft Research’s line of 1-bit large language models. In the company’s papers, the key idea is to represent each parameter as one of three values: -1, 0 or 1. That ternary approach reduces memory needs sharply and also cuts the movement of data between memory and processor, which is often a major bottleneck in token-by-token generation. Microsoft says its bitnet.cpp software stack can run these models efficiently on CPUs, with speed-ups of 2.37 times to 6.17 times on x86 systems and 1.37 times to 5.07 times on ARM hardware.

The broader significance is that this is not just post-training compression, where a model is trained in higher precision and then aggressively rounded down. Microsoft’s research argues that models can be designed and trained natively for this low-precision regime without collapsing performance. In its BitNet b1.58 paper, the company says such models can match full-precision transformers of similar size and training scale while using less memory, less energy and lower latency. That matters because it suggests the redundancy in today’s models is far greater than many deployments require.

The company has since pushed the idea further with BitNet b1.58 2B4T, which TechCrunch reported in April 2025 as the largest 1-bit AI model released by Microsoft Research. The model, with 2 billion parameters, was presented as runnable on consumer CPUs, including Apple’s M2 chip. Microsoft’s own GitHub repository for BitNet also points to continuing optimisation work, including parallel kernel implementations and embedding quantisation, which is a sign that the effort is moving beyond theory and into engineering meant for practical use.

A related trend is the rise of Mixture-of-Experts systems, in which a model may contain many parameters but activates only a small subset for each token. Combined with ternary weights, this can produce a model that retains a large nominal capacity while keeping per-query cost relatively low. That is why recent preview systems such as DeepGrove’s Maple-Preview are drawing attention: they point towards a hybrid future in which smaller local models handle routine work while cloud systems remain available for heavier reasoning and training. The likely outcome is not the disappearance of the data centre, but a more distributed AI stack in which privacy, latency, cost and energy use all improve when the model can stay close to the user’s device.