We see Apple and Google pushing on-device processing. Is local AI the future of mobile tech, or is it just a marketing gimmick to sell new phones? I want to know if these chips are actually doing heavy lifting or just basic photo edits.
3 answers
It's definitely more than a gimmick. On-device processing is the only way to enable features like real-time voice translation or advanced predictive text without the lag associated with a round-trip to a server. As privacy laws become stricter, manufacturers want to move as much processing as possible away from the cloud to limit their own liability. We are seeing neural engines becoming a core part of the SoC, taking up more die space every year. This isn't just for photos; it's about the phone learning your habits locally to provide a truly personalized experience that is both fast and private.
Scott here. Do you think battery life will become the main constraint as we push more LLM-style tasks to the phone's local processor?
I think it's the only way to make AI assistants useful when you have a spotty internet connection or you're traveling abroad.
Rebecca makes a solid point. Offline capability is a massive advantage for local models that cloud-based services just cannot compete with at all.
Scott, battery is definitely the big boss we have to fight. Running a large model is computationally expensive and generates heat. This is why most mobile AI right now is very "bursty"—it runs for a second and then stops. We need significant breakthroughs in efficiency or specialized low-power AI cores before we can have a persistent, always-on local assistant on our phones.