Use Groq as your LLM provider

You can now use Groq as your LLM provider. It currently hosts the following models: llama2-70b-chat, mixtral-8x7b-32768, and Gemma-7b-it.

What is Groq and why should I try them out?

Groq has recently gained a lot of attention for being one of the top companies providing fast LLM inference solutions. There's a growing need among Gen AI application builders to decrease response times, as lowering latency is key for real-time AI applications. The field is getting crowded, with many companies competing in LLM inference. Groq being one of them, claims to offer inference speeds that are 18 times faster than some well-known cloud-based services. The models hosted by Groq utilize the Groq LPU™ Inference Engine, which is powered by their specialized hardware called Language Processing Units (LPUs) to enhance LLM performance.

Tokens per second