We are looking to build a "crew" for each client to handle SEO, social copy, and image prompts. Can we run multiple instances of CrewAI in parallel on a serverless architecture, or is it too resource-heavy for a high-volume agency?
3 answers
We actually run a similar setup for our agency. CrewAI is surprisingly lightweight because it doesn't require a constant "always-on" connection between agents—they communicate via the task manager. We deploy our crews as individual Docker containers on AWS Fargate. When a client needs a weekly report, the container spins up, the crew does its thing, and then shuts down. The biggest "cost" isn't the memory, it's the tokens. We use smaller models like GPT-3.5 or Claude Haiku for the "Researcher" role and save GPT-4 for the final "Editor" to keep costs down. It’s very scalable if you manage your model selection wisely.
Have you run into any issues with rate limits when running multiple "crews" at the exact same time?
The modularity is the best part. We created a "Base Crew" and just swap the "Client Context" file for each new project. Takes minutes to set up.
That's a great tip, Lawrence. Templates are the secret to scaling an agency with CrewAI.
That's the main bottleneck, Arthur. We had to implement a custom queue using Celery. Instead of hitting the OpenAI API all at once, the queue staggers the starts. CrewAI handles the internal retries pretty well, but you definitely need a global rate-limiter at the infrastructure level if you're doing high-volume work for an agency.