I want to build a private messaging app with an integrated assistant. Is it possible to run a small LLM locally within Flutter (AI apps) using something like Llama.cpp or are we strictly limited to cloud-based APIs for natural language?
3 answers
Yes, it is absolutely possible to run local LLMs in Flutter (AI apps), though it requires some advanced configuration. You can use Dart FFI to bridge with C++ libraries like llama.cpp. I managed to get a quantized 3B parameter model running on a high-end Android device recently. The performance was surprisingly usable for simple tasks. The key is to use highly compressed GGUF models and manage the memory manually through the FFI bridge. This approach is perfect if you are prioritizing user privacy or need the app to function without an internet connection for its core AI features.
Donna, what is the impact on the app's final binary size when you bundle a quantized model directly into the assets of your Flutter (AI apps)?
MediaPipe also offers a LLM Inference API that is much easier to implement in Flutter than manual FFI for most standard use cases.
Sandra makes a great point. MediaPipe simplifies the deployment process for Flutter (AI apps) immensely, especially if you don't want to dive deep into writing custom C++ wrappers for your AI.
Thomas, the binary size will definitely increase significantly. Even a small 3B model will add about 1.5GB to 2GB to your APK or IPA. For Flutter (AI apps), the best strategy is to have the user download the model on the first launch rather than bundling it. This keeps the initial download small and compliant with app store limits. You can use the path_provider package to store the model in the local application documents directory once the download is complete.