Announcing the launch of Code Llama 70B, the most extensive and top-performing addition to the Code Llama lineage. Code Llama 70B is accessible in the same trio of versions as its predecessors, all freely available for both research and commercial purposes: CodeLlama – 70B, the foundational coding model; CodeLlama – 70B – Python, specializing in Python; and Code Llama – 70B – Instruct 70B, finely tuned for comprehending natural language directives.

Code Llama stands as a cutting-edge Language Model (LLM) with the prowess to generate code and articulate natural language about code, drawing inspiration from both code snippets and natural language cues. It is open for deployment in research and commercial domains. Built atop the Llama 2 framework, Code Llama offers three models: Code Llama, the foundational coding model; Codel Llama – Python, honed for Python; and Code Llama – Instruct, tailored for understanding natural language instructions.

In our proprietary benchmark assessments, Code Llama has surpassed prevailing publicly accessible LLMs in coding tasks. Today marks the release of Code Llama, a substantial LLM designed to employ textual prompts for code generation. Code Llama demonstrates state-of-the-art performance among publicly accessible LLMs in coding tasks, presenting potential efficiency enhancements for current developers and lowering barriers for coding novices. Positioned as a productivity and educational tool, Code Llama aids programmers in crafting robust, well-documented software.

The landscape of generative AI is swiftly advancing, and we advocate for an open strategy in today’s AI realm to foster the development of innovative, secure, and responsible AI tools.

Code Llama, a code-focused iteration of Llama 2, evolves through additional training on code-centric datasets. It extracts more data from these datasets for extended durations, endowing it with enhanced coding capabilities. Essentially, Code Llama amplifies coding proficiency, building upon the foundation of Llama 2. It can generate code and articulate natural language about code, responding to both code and natural language prompts. It is proficient in code completion and debugging, offering support for prevalent programming languages like Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash. We release four sizes of Code Llama with 7B, 13B, 34B, and 70B parameters, each trained with 500B tokens of code and code-related data, except for 70B, which is trained on 1T tokens.

The diverse models cater to varied serving and latency needs. The 7B model can be deployed on a single GPU, while the 34B and 70B models yield superior outcomes, providing enhanced coding assistance. The smaller 7B and 13B models prioritize speed and are more fitting for tasks demanding low latency, such as real-time code completion. Code Llama models offer consistent outputs with up to 100,000 tokens of context, trained on sequences of 16,000 tokens, showcasing improvements on inputs with up to 100,000 tokens.

Extended input sequences unlock novel use cases for a Code LLM. Users can furnish the model with more context from their codebase, enhancing relevance in the generated output. This proves advantageous in debugging scenarios within expansive codebases, where overseeing all code associated with a specific issue poses a challenge. When grappling with debugging large portions of code, developers can input the entire code length into the model.

In addition to the core Code Llama, we introduce two specialized variations: Code Llama – Python and Code Llama – Instruct.

Code Llama – Python focuses on Python code, undergoing further fine-tuning on 100B tokens of Python code. Given Python’s prominence in code generation benchmarks and its role in the AI community, this specialized model offers added utility.

Code Llama – Instruct is a finely tuned variant, aligning with instruction inputs. The model processes a “natural language instruction” and the anticipated output, enhancing its ability to comprehend user prompts. For code generation, particularly, we recommend using Code Llama – Instruct variants, as they are tailored to generate helpful and secure responses in natural language.

We discourage the usage of Code Llama or Code Llama – Python for general natural language tasks, as these models aren’t optimized for following natural language instructions. Code Llama excels in code-specific assignments and is unsuitable as a foundational model for other tasks.

Users engaging with Code Llama models must adhere to our license and acceptable use policy.

To assess Code Llama’s performance, we subjected it to two widely recognized coding benchmarks: HumanEval and Mostly Basic Python Programming (MBPP). HumanEval gauges the model’s capacity to complete code based on docstrings, while MBPP assesses its ability to generate code from textual descriptions.

Benchmark results showcase Code Llama’s superiority over open-source, code-centric LLMs, outperforming even Llama 2. Code Llama 34B achieved a notable score of 53.7% on HumanEval and 56.2% on MBPP, surpassing other state-of-the-art solutions and standing shoulder-to-shoulder with ChatGPT. Acknowledging the cutting-edge nature of this technology, Code Llama does carry certain risks. We prioritize responsible AI model development and execute extensive safety measures before the release. As part of our red teaming efforts, we quantitatively evaluated Code Llama’s risk of generating malicious code, comparing its responses to prompts with clear malicious intent against ChatGPT’s (GPT3.5 Turbo). Our findings indicate that Code Llama consistently provides safer responses.

Developers are already leveraging LLMs for diverse tasks, ranging from crafting new software to debugging existing code. The objective is to streamline developer workflows, enabling them to concentrate on the most human-centric facets of their work rather than mundane tasks.

At Meta, we advocate for an open approach to AI models, particularly LLMs for coding, recognizing their potential to drive innovation and enhance safety. Publicly accessible, code-centric models contribute to the evolution of technologies that enhance human lives. By releasing models like Code Llama, the entire community gains the opportunity to evaluate their capabilities, pinpoint issues, and address vulnerabilities.

For those interested, Code Llama’s training recipes are available on our GitHub repository, along with the model weights.

Our research paper delves into the development of Code Llama, detailing our benchmarking tests, elucidating the model’s limitations, highlighting challenges encountered, discussing mitigations applied, and outlining future challenges we aim to explore.

Our Responsible Use Guide has been updated, offering guidance on developing downstream models responsibly. This includes defining content policies and mitigations, preparing data, fine-tuning the model, evaluating and improving performance, addressing input and output-level risks, and incorporating transparency and reporting mechanisms in user interactions.

Developers are encouraged to evaluate their models using code-specific evaluation benchmarks and conduct safety studies on code-specific use cases, such as generating malware, computer viruses, or malicious code. Safety datasets for automatic and human evaluations, coupled with red teaming on adversarial prompts, are also recommended.

Code Llama is designed to support software engineers across various domains, from research and industry to open-source projects, NGOs, and businesses. However, the base and instruct models cater to a spectrum of use cases, and we anticipate that Code Llama will inspire the creation of innovative tools using Llama 2 in research and commercial product development.

1. How big is the Llama 70B?

2. What is CodeLlama?

3. Is Llama better than ChatGPT?

4. Is Llama 2 better than GPT-4?