I'm interested in using LLMs to create dynamic, unscripted dialogue for NPCs in my current project. However, calling an API every time a player speaks to a character introduces huge latency and cost issues. Is there a way to run small, quantized models locally within the game engine, or should I be looking at hybrid cloud solutions? I want to ensure the AI feels "alive" without making the player wait 5 seconds for a response.
3 answers
Running a full LLM locally is tough unless your target platform is high-end PC. For most games, you should look into 4-bit quantization using libraries like llama.cpp or ONNX Runtime. I’ve had success running a 3B parameter model locally on a mid-range GPU with under 2GB of VRAM. The trick is to "pre-warm" the prompt and use a streaming response so the text appears as it's generated. This masks the latency. Also, limit the AI to specific personality parameters to prevent it from "hallucinating" lore that doesn't exist in your game world.
Are you planning to use a middleware like Convai or Inworld AI, or are you trying to build your own pipeline using something like LangChain?
Use a "Small Language Model" (SLM) like Phi-2. They are much faster and can be fine-tuned on your game's specific script for better accuracy.
Great point, Susan. Phi-2 is surprisingly capable for its size. Fine-tuning it on a JSON of your game's lore is the best way to keep the dialogue "on-brand."
I was looking at Inworld, but the subscription costs for a small indie team are a bit intimidating. I’m leaning towards a local Python-based microservice that communicates with my Unity build via WebSockets. This way, I can keep the costs down and have more control over the data privacy, but I am worried about the extra CPU load interfering with the game's physics thread.