I'm trying to build a cloud-native app and want to use Chroma DB for my vector search. Has anyone successfully deployed it on AWS Lambda or Fargate? I am worried about the cold start times and how to manage the persistent storage of the vector index in a serverless environment. Would it be better to use an EC2 instance instead, or is there a way to make it work?
3 answers
Running it on AWS Lambda is tricky because of the deployment package size and the need for persistent storage. Lambda is ephemeral, so you’d have to load the index from S3 every time, which kills performance. AWS Fargate is a much better middle ground. You can mount an EFS (Elastic File System) volume to the Fargate container. This allows it to persist the data while still being "serverless" in terms of management. It handles the memory-intensive nature of vector searches much better than a small Lambda function ever could.
Does using EFS with Fargate introduce much latency compared to local EBS storage?
I ended up using an EC2 t3.medium instance. It was cheaper and more reliable for a 24/7 vector search service than trying to hack it into Lambda.
I second that, Gregory. For production workloads that require constant availability, a dedicated instance often provides the most consistent performance.
Scott, there is a slight increase in latency for I/O operations, but since the tool loads the active index into RAM, the impact on actual search queries is minimal once the service is warmed up. The flexibility of having a shared file system usually outweighs the few milliseconds of storage latency for most cloud-based AI projects I've managed recently.