stable-code:code

Model	Size	Python	C++	Javascript	Java	PHP	Rust
Stable Code	3B	32.4%	30.9%	32.1%	32.1%	24.2%	23.0%
CodeLLama	7B	30.0%	28.2%	32.5%	31.1%	25.7%	26.3%
Deepseek Coder	1.3B	28.6%	29.2%	28.7%	29.0%	23.6%	18.5%
Wizard Coder	3B	31.6%	25.6%	26.2%	25.8%	25.3%	20.4%
StarCoder	3B	21.6%	19.8%	21.5%	20.5%	19.0%	16.9%
Replit Code V1.5	3B	23.0%	25.9%	26.2%	23.6%	23.2%	21.5%
Deci Coder	1B	19.1%	6.8%	18.4%	16.7%	2.1%	1.7%

Model Details

Developed by: Stability AI
Model type: stable-code 模型是基於 transformer 解碼器架構的自迴歸語言模型。
Language(s): 英文、程式碼
Contact: 如有關於模型的問題和意見，請發送電子郵件至 lm@stability.ai

Model Architecture

該模型是一個僅解碼器的 transformer，類似於 LLaMA (Touvron et al., 2023) 架構，但進行了以下修改

Parameters	Hidden Size	Layers	Heads	Sequence Length
2,796,431,360	2560	32	32	16384

Position Embeddings: 旋轉位置嵌入 (Su et al., 2021) 應用於前 25% 的 head embedding 維度，以提高吞吐量，遵循 Black et al. (2022)。
Tokenizer: 我們使用 GPTNeoX Tokenizer 的修改版本。NeoX。我們添加了特殊 token 來訓練 Fill in the Middle (FIM) 功能，例如 <FIM_PREFIX> 和 <FIM_SUFFIX> 以及其他特殊 token。

Training

Training Dataset

該資料集包含在 HuggingFace Hub 上提供的開源大型資料集的過濾混合：Falcon RefinedWeb 提取物 (Penedo et al., 2023)，以及 CommitPackFT 和 Github Issues (BigCode., 2023) 和 StarCoder (Li et al., 2023)。我們進一步使用來自數學領域的資料來補充我們的訓練 (Azerbayev, Zhangir, et al., 2023 和 Yu, Longhui, et al., 2023)。

前 18 名程式語言訓練於：- C - CPP - Java - JavaScript - CSS - Go - HTML - Ruby - Rust - Markdown - Shell - Php - Sql - R - Typescript - Python - Jupyter-Clean - RestructuredText

Use and Limitations

Intended Use

該模型旨在用作特定應用程式微調的基礎模型。開發人員必須評估和微調模型，以確保在下游應用程式中的安全效能。

Limitations and Bias

作為基礎模型，此模型可能會表現出不可靠、不安全或其他不良行為，必須在部署前通過評估和微調來糾正。即使在應用資料清理篩檢程式後，預訓練資料集也可能包含冒犯性或不適當的內容，這可能會反映在模型產生的文字中。我們建議使用者在生產系統中使用這些模型時謹慎行事。如果模型不適合您的應用程式，或任何可能對他人造成有意或無意傷害的應用程式，請勿使用這些模型。

References

Hugging Face

Stable Code 3B is a 3 billion parameter Large Language Model (LLM), allowing accurate and responsive code completion at a level on par with models such as Code Llama 7b that are 2.5x larger.

**Key Features**

* **NEW** instruct model `ollama run stable-code`
* Fill in Middle Capability (FIM)
* Supports Long Context, trained with Sequences upto 16,384

![spiderchart](https://github.com/jmorganca/ollama/assets/3325447/6c3de7a5-5e10-4884-81fb-3a1b3f566609)

| Model            | Size | Python | C++  | Javascript | Java | PHP  | Rust |
|------------------|------|--------|------|------------|------|------|------|
| **Stable Code**  | 3B   | 32.4%  | 30.9%| 32.1%      | 32.1%| 24.2%| 23.0%|
| CodeLLama        | 7B   | 30.0%  | 28.2%| 32.5%      | 31.1%| 25.7%| 26.3%|
| Deepseek Coder   | 1.3B | 28.6%  | 29.2%| 28.7%      | 29.0%| 23.6%| 18.5%|
| Wizard Coder     | 3B   | 31.6%  | 25.6%| 26.2%      | 25.8%| 25.3%| 20.4%|
| StarCoder        | 3B   | 21.6%  | 19.8%| 21.5%      | 20.5%| 19.0%| 16.9%|
| Replit Code V1.5 | 3B   | 23.0%  | 25.9%| 26.2%      | 23.6%| 23.2%| 21.5%|
| Deci Coder       | 1B   | 19.1%  | 6.8% | 18.4%      | 16.7%| 2.1% | 1.7% |

## Model Details

* **Developed by**: [Stability AI](https://stability.ai/)
* **Model type**: stable-code models are auto-regressive language models based on the transformer decoder architecture.
* **Language(s)**: English, Code
* **Contact**: For questions and comments about the model, please email `lm@stability.ai`

### Model Architecture

The model is a decoder-only transformer similar to the LLaMA ([Touvron et al., 2023](https://arxiv.org/abs/2307.09288)) architecture with the following modifications:

| Parameters     | Hidden Size | Layers | Heads | Sequence Length |
|----------------|-------------|--------|-------|-----------------|
| 2,796,431,360  | 2560        | 32     | 32    | 16384            |

* **Position Embeddings**: Rotary Position Embeddings ([Su et al., 2021](https://arxiv.org/abs/2104.09864)) applied to the first 25% of head embedding dimensions for improved throughput following [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf).
* **Tokenizer**: We use a modified version of the GPTNeoX Tokenizer.[`NeoX`](https://github.com/EleutherAI/gpt-neox). We add special tokens to train for Fill in the Middle (FIM) capabilities like `<FIM_PREFIX>` and `<FIM_SUFFIX>` along with other special tokens.

## Training

### Training Dataset

The dataset is comprised of a filtered mixture of open-source large-scale datasets available on the [HuggingFace Hub](https://huggingface.co/datasets): Falcon RefinedWeb extract ([Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)), along with [CommitPackFT](https://huggingface.co/datasets/bigcode/commitpackft) and [Github Issues](https://huggingface.co/datasets/bigcode/the-stack-github-issues) (BigCode., 2023), and StarCoder ([Li et al., 2023](https://arxiv.org/abs/2305.06161)). We further supplement our training with data from mathematical domains ([Azerbayev, Zhangir, et al., 2023](https://arxiv.org/abs/2310.10631) and, [Yu, Longhui, et al., 2023](https://arxiv.org/abs/2309.12284)).

Top 18 programming languages trained on:
- C
- CPP
- Java
- JavaScript
- CSS
- Go
- HTML
- Ruby
- Rust
- Markdown
- Shell
- Php
- Sql
- R
- Typescript
- Python
- Jupyter-Clean
- RestructuredText

## Use and Limitations

### Intended Use

The model is intended to be used as a foundational base model for application-specific fine-tuning. Developers must evaluate and fine-tune the model for safe performance in downstream applications.

### Limitations and Bias

As a base model, this model may exhibit unreliable, unsafe, or other undesirable behaviors that must be corrected through evaluation and fine-tuning prior to deployment. The pre-training dataset may have contained offensive or inappropriate content, even after applying data cleansing filters, which can be reflected in the model-generated text. We recommend that users exercise caution when using these models in production systems. Do not use the models if they are unsuitable for your application, or for any applications that may cause deliberate or unintentional harm to others.

## References

[Hugging Face](https://huggingface.co/stabilityai/stable-code-3b)

貼上、拖放或點擊以上傳圖片 (.png, .jpeg, .jpg, .svg, .gif)