---
license: apple-amlr
base_model:
- Qwen/Qwen2.5-Coder-7B-Instruct
pipeline_tag: text-generation
tags:
- code
- diffusion
- Dream
- diffusion language model
---


### CADD-Base-7B

CADD-Base-7B is a masked diffusion language model for code generation, augmented with **Continuously Augmented Discrete Diffusion (CADD)** --- a continuous flow-matching signal that guides the discrete denoising process.

**Key idea:** At each diffusion step, a continuous embedding `z_continuous` is added to masked-token embeddings, following a linear flow-matching trajectory from noise to clean embeddings. This is orthogonal to the discrete unmasking strategy --- any MDM algorithm can be combined with CADD.

#### Usage

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_path = "apple/CADD-Base-7B"
model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = model.to("cuda").eval()

prompt = "def fibonacci(n):\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")

output = model.diffusion_generate(
    input_ids,
    max_new_tokens=512,
    steps=512,
    temperature=0.1,
    alg="entropy",
    alg_temp=0.0,
    use_cadd=True,
    cadd_sampling_mode="weighted",
)

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

#### CADD Sampling Parameters

| Parameter | Type | Default | Description |
|:---|:---:|:---:|:---|
| `use_cadd` | bool | `True` | Enable CADD continuous augmentation |
| `cadd_sampling_mode` | str | `"argmax"` | How to estimate z_0 from logits: `"weighted"` or `"argmax"` |
| `alg` | str | `"origin"` | Unmasking strategy: `"entropy"`, `"origin"`, `"maskgit_plus"`, `"topk_margin"` |
| `temperature` | float | `1.0` | Sampling temperature for token prediction |
| `steps` | int | `512` | Number of diffusion steps |

#### More details:

- Paper: [Continuously Augmented Discrete Diffusion Model for Categorical Generative Modeling](https://arxiv.org/abs/2510.01329) (ICLR 2026)
- GitHub: https://github.com/apple/ml-CADD

#### Citation

```bibtex
@article{zheng2025continuously,
  title={Continuously augmented discrete diffusion model for categorical generative modeling},
  author={Zheng, Huangjie and Gong, Shansan and Zhang, Ruixiang and Chen, Tianrong and Gu, Jiatao and Zhou, Mingyuan and Jaitly, Navdeep and Zhang, Yizhe},
  journal={arXiv preprint arXiv:2510.01329},
  year={2025}
}
```

#### Acknowledgment

To power this HuggingFace model release, we build upon and improve [DiffuCoder](https://github.com/apple/ml-diffucoder), reusing [Dream](https://huggingface.co/Dream-org/Dream-v0-Base-7B)'s modeling architecture and generation utils.