---
license: other
language:
- en
tags:
- 3d
- point-cloud
- multimodal
- multi-object
- pointllm
- modelnet40
pipeline_tag: text-generation
---

# Multi-3DLLM Checkpoints

This repository hosts the BeyondSingleObject checkpoints:

- `multi-3dllm/`: MO3D, Shape Mating, and Change Captioning
- `multi-3dllm-classification/`: ModelNet40 zero-shot classification

The task checkpoint uses the Patch-Interaction Transformer implementation and
was jointly fine-tuned on the three released task datasets. Code and training
scripts are available at:

```text
https://github.com/KohsukeIde/BeyondSingleObject
```

Dataset annotations are available at:

```text
https://huggingface.co/datasets/idekoh/BeyondSingleObject
```

Point-cloud preparation instructions and upstream download links are provided
in the GitHub repository README.

## Download

```bash
huggingface-cli download idekoh/Multi-3DLLM \
  --local-dir checkpoints \
  --include "multi-3dllm/**" "multi-3dllm-classification/**"
```

Expected local layout:

```text
checkpoints/
|-- multi-3dllm/
`-- multi-3dllm-classification/
data/
```

## Evaluation Snapshot

Our latest checkpoint and cleaned datasets further improve multi-object 3D
reasoning performance.

| Task | Metric | Current release |
| --- | --- | ---: |
| MO3D holistic QA | Binary accuracy `B` | **84.0%** (42/50) |
| Shape Mating | Selection accuracy `S` | **73.0%** (146/200) |
| Change Captioning verify | Binary accuracy `B` | **67.0%** (67/100) |

The separately released classification checkpoint follows the PointLLM
ModelNet40 evaluation protocol. See the repository README for its six
one-, two-, and three-object commands.

## Usage

Example inference:

```bash
MODEL_PATH=checkpoints/multi-3dllm \
ANNO_PATH=data/mo3d/test.json \
DATA_PATH=data/point_clouds \
OUTPUT_DIR=outputs/mo3d_eval \
scripts/eval/infer.sh
```

ModelNet40 classification:

```bash
MODEL_PATH=checkpoints/multi-3dllm-classification \
OUTPUT_DIR=outputs/modelnet40_eval \
LIMIT=0 \
PROMPT_MODE=paper \
NUM_OBJECTS=1 \
TARGET_POSITION=1 \
scripts/eval/eval_modelnet.sh
```

## Notes

LLM-judged reasoning and delta-caption scores depend on the judge model and
prompt configuration. Use the released evaluators and report the exact judge
configuration when comparing those metrics. The task and classification
checkpoints have different roles and should not be interchanged.

## License

These checkpoints are built with the BeyondSingleObject codebase and use
PointLLM-style initialization and data. They may inherit terms from upstream
model, code, and dataset components, including PointLLM, Vicuna/Llama,
Objaverse/Cap3D, ShapeTalk, Thingi10K, Neural Shape Mating, and ModelNet40.
Please check the corresponding upstream licenses before redistribution or
commercial use.

## Citation

```bibtex
@inproceedings{ide2026beyondsingleobject,
  title={BeyondSingleObject: Learning 3D Relations with Large Language Models},
  author={Ide, Kohsuke and Yamada, Ryousuke and Qiu, Yue and Ma, Xianzheng and Fukuhara, Yoshihiro and Kataoka, Hirokatsu and Satoh, Yutaka},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings},
  year={2026}
}
```