# Summarization taks, looking for clarifications before getting started

**URL:** <https://discuss.huggingface.co/t/summarization-taks-looking-for-clarifications-before-getting-started/3626>\
**Category:** Beginners\
**Created:** [February 13, 2021, 12:13pm UTC](https://discuss.huggingface.co/t/summarization-taks-looking-for-clarifications-before-getting-started/3626 "2021-02-13T12:13:39Z")\
**Posts on this page:** 1\
**Showing post:** 9

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [February 15, 2021, 3:09pm UTC](https://discuss.huggingface.co/t/summarization-taks-looking-for-clarifications-before-getting-started/3626/9 "2021-02-15T15:09:38Z")

</div>

Hi @neuralpat, no bother at all 🙂

If I understood your original aim, you’d like to perform summarization right? As far as I know, you won’t be able to use the xlm-r model fine-tuned on token classification since what you really need is a language modelling head to generate the summary.

How long are you documents? Depending on time / cost, I would be tempted to still run an experiment with the encode-decoder approach just to get a feel for how well this baseline performs on the dataset. For example, the CNN / DailyMail dataset has articles that are longer than most Transformer model’s context size, yet the summaries are not so bad.

If length is really an issue, then you might want to check out the LongFormer model: [allenai/led-base-16384 · Hugging Face](https://huggingface.co/allenai/led-base-16384) which can process 16k tokens 🤯

There’s also a long thread here with a discussion related to your issue, so you might find some relevant ideas there: [Summarization on long documents](https://discuss.huggingface.co/t/summarization-on-long-documents/920)

What is generally true is that the _pretrained_ checkpoints can be _fine-tuned_ on a variety of downstream tasks via transfer learning - perhaps this is what you had in mind?

---

_[View the full topic](https://discuss.huggingface.co/t/summarization-taks-looking-for-clarifications-before-getting-started/3626)._