inkentry 1.1: GPU everywhere

inkentry 1.0 shipped with local embedding on CPU everywhere, and GPU acceleration on macOS through Metal. We wanted GPU support on Windows and Linux from the start, but we held it back deliberately rather than rush it.

GPU hardware is fragmented. Metal covers Mac. Nvidia needs CUDA. AMD needs ROCm. The embedding engine we shipped in 1.0 only supported CUDA and Metal. The CUDA version had an open bug that meant it couldn't detect support at runtime. Ship the CUDA version and it would be up to the user to know which version to run, and they would get no feedback if they picked the wrong option. That's not an experience we were willing to provide.

So we held off, rather than rush a fix before we knew what would actually work.

Vulkan changes the calculation

Vulkan is an open, cross-vendor GPU API. One target instead of one for each manufacturer. Most AI tooling skips it anyway, because every last bit of hardware-specific performance tends to matter more than portability. The embedding engine we used in 1.0 had no Vulkan support at all.

For inkentry, that tradeoff goes the other way. Indexing a full codebase is GPU-intensive, but it's also rare. Most usage is search, where the difference between CUDA and Vulkan is a few milliseconds, well under what anyone would notice. In our testing Vulkan trailed CUDA by only a few percentage points. While going from CPU to GPU there is a real gap: 8–100x faster, depending on the card. A CUDA-only build would have covered Nvidia users with a slightly higher ceiling; Vulkan covers effectively everyone, at nearly the same speed. We picked broad support over a narrow, marginally faster path.

A new engine

Committing to Vulkan meant leaving our 1.0 embedding engine behind, we had taken it as far as we could. We liked what we had: flexible, easy to build on, and shipped embedded in the binary with no external infrastructure to run. Replacing it was not an easy decision to make, but it was necessary.

We moved to an engine that matched what we needed: mature, embeddable in the binary the same way, already compatible with our model, and with support for CPU, Vulkan, CUDA, and Metal all in one codebase. We shipped it with Vulkan support for Windows and Linux, and, since the Metal path was already built and there's only one GPU architecture to target on Mac, Metal support too. A CUDA build remains an option if we decide it's worth it later.

What's in 1.1

The headline is GPU support on effectively any GPU shipped in the last decade, across macOS, Windows, and Linux. We still provide a graceful fallback to CPU for machines that don't meet the requirements.

The new engine also brought flash attention, which on our Mac benchmarks cut indexing wall-clock time by roughly 30%, alongside a meaningful drop in memory usage.

There's a smaller change worth mentioning too: interactive queries now run on their own dedicated lane, so a search no longer gets stuck behind a running index command.

What's next

We’re not stopping here, but the changes we’re adding in 1.1 is such a significant step forward for every user that we wanted it out as soon as it was ready. The next release, 1.2 won't be another performance-focused release. There will be more performance changes coming in the future, but we don’t expect the impact to be this big, and they won’t be our main priority for the immediate future.

Get inkentry 1.1 today

You can get the new release from get.inkentry.com.