---
license: mit
tags:
- video-text-retrieval
- factor-graph-attention
- multimodal
library_name: transformers
---

# Video-text matching on YouCook2

Built on the attention layer of [Factor Graph
Attention](https://arxiv.org/abs/1904.05880) (CVPR 2019), following [Video and
Text Matching with Conditioned Embeddings](https://arxiv.org/abs/2110.11298)
(WACV 2022). A caption and a video are attended jointly and scored against each
other, trained with a max-margin contrastive loss against the hardest negative in
the batch.

Code: [github.com/idansc/fga](https://github.com/idansc/fga).

## Read the split before the numbers

This is **not** YouCook2's official train/val division, and these numbers do not
compare to published YouCook2 retrieval figures.

The feature dump available here covers YouCook2's validation videos only -- 436 of
them -- so the data was divided over those by video, no video appearing on two
sides:

| | videos | captioned segments |
| --- | --- | --- |
| train | 296 | 2,354 |
| val | 52 | 379 |
| test | 88 | 617 |

Retrieval over 617 segments is an easier problem than retrieval over the official
split's several thousand, and the numbers below should be read as an internal
comparison only.

## Results

Held-out test slice, 617 segments. Recall at k in both directions:

| | R@1 | R@5 | R@10 | median rank |
| --- | --- | --- | --- | --- |
| text to video | 0.418 | 0.723 | 0.818 | 2 |
| video to text | 0.476 | 0.786 | 0.882 | 2 |

The checkpoint is the one that scored best on the validation videos, not on the
test slice. Selecting on the test slice reads 0.438 / 0.468 instead; the
validation split exists so that choice is not made there.

R@10 here counts a hit at rank 10 or better. Worth stating because the reference
implementation's `evaluation.py` computes its `r10` as `ranks < 50` in all four of
its ranking functions -- a commented-out print in the same file labels that value
"Ranks@50" -- so an R@10 reported by that code is an R@50 and is not comparable to
the column above.

## Training

Hidden size 768, learning rate 2e-4, 2,000 epochs. Video features are the
InternVideo MM-L14 dump; captions are encoded with GloVe 300d.

## Usage

```python
from fga.tasks.video_retrieval import VideoMatchModel

model = VideoMatchModel.from_pretrained("Idan/fga-video-retrieval")
scores = model(caption_input_ids=text, video_features=video).logits
```

## Citation

```bibtex
@inproceedings{ali2022video,
  title={Video and Text Matching with Conditioned Embeddings},
  author={Ali, Ameen and Schwartz, Idan and Hazan, Tamir and Wolf, Lior},
  booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision},
  year={2022}
}

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}

@inproceedings{zhou2018towards,
  title={Towards Automatic Learning of Procedures from Web Instructional Videos},
  author={Zhou, Luowei and Xu, Chenliang and Corso, Jason J},
  booktitle={AAAI Conference on Artificial Intelligence},
  year={2018}
}
```