Meta’s new Code Llama 70B now available on OctoAI

We are excited to announce the availability of Meta’s newly released Code Llama 70B Instruct model on the OctoAI Text Gen Solution. Code Llama 70B Instruct adds to the catalog of popular large language models (LLMs) available on OctoAI - all on the same unified Text Gen API endpoint and all delivered through the same OpenAI consistent API. This addition reflects our commitment to speedily bring the latest models to OctoAI and to simplify how builders take advantage of the rapid evolution in the world of LLMs.

Sign up and get started with a free trial today on the OctoAI Text Gen Solution.

You can get started with Code Llama 70B Instruct using the same API and token key as other OctoAI Text Gen Solution models, passing “codellama-70b-instruct-fp16” as the model parameter. If your application already uses the OpenAI python SDK, you can easily swap and use Code Llama 70B Instruct (or any other model on OctoAI) with these three code changes.

OPENAI_API_KEY  = <your-ocotai-token>
OPENAI_URL_BASE = “https://text.octoai.run/v1”

## update the model parameter in the OpenAI chat completions API
model: “codellama-70b-instruct-fp16”

Open source moving closer to GPT-4 quality in code generation

The OctoAI Text Gen Solution serves the instruction following fine-tuned Code Llama 70B Instruct model, with applicability across both code generation and natural language instruction following use cases. The inclusion of Code Llama 70B marks the most recent in OctoAI’s history of Llama work — from the accelerated Llama 65B, to the early inclusion of the Llama 2 Chat and the broader Llama 2 herd, to the launch of the OctoAI Text Gen Solution and inference based pricing across the complete Llama 2 family.

Code Llama 70B is now the largest open code trained LLM. Based on results published by Meta, Code Llama 70B outperforms GPT 3.5 and is approaching GPT 4 in multiple code generation specific benchmarks including HumanEval and Mostly Basic Python Programming (MBPP). The Code Llama models on OctoAI are used by several OctoAI Text Gen customers - both for code generation use cases and for targeted text generation applications. Within hours of the Code Llama 70B announcement, we’ve had several customers reach out regarding the new model. And we’re excited to make this model available for your evaluations and your applications.

OctoAI includes built-in updates to the new chat prompt format

Code Llama 70B Instruct introduces a new format for chat prompts compared to other Llama 2 models. This change can result in intermittent outputs of unexpected text like [INST] or <<SYS>> in the text response, if the conversation template is not updated to use the new format. OctoAI has automatically enabled the correct conversation template for inferences against the Code Llama 70B Instruct model.

Get started with Code Llama 70B on OctoAI today

You can get started with Code Llama 70B today at no cost, with a free trial on OctoAI. You’re also welcome to join us on Discord to engage with the team and community, and to share updates about your new AI powered applications. We look forward to hearing from you on our channels!

If you are currently using a closed source LLM like GPT-3.5-Turbo or GPT-4 from OpenAI, we have a new promotion to help you accelerate integration of open source LLMs in your applications. Read more about this in the OctoAI Model Remix Program introduction.