# Git push too slow and interruptted with broken pipe and EOF problem

**URL:** <https://discuss.huggingface.co/t/git-push-too-slow-and-interruptted-with-broken-pipe-and-eof-problem/81949>\
**Category:** 🤗Datasets\
**Created:** [April 16, 2024, 1:02am UTC](https://discuss.huggingface.co/t/git-push-too-slow-and-interruptted-with-broken-pipe-and-eof-problem/81949 "2024-04-16T01:02:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jasmineChen](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jasmineChen](https://discuss.huggingface.co/u/jasmineChen)\
**Post date:** [April 16, 2024, 1:02am UTC](https://discuss.huggingface.co/t/git-push-too-slow-and-interruptted-with-broken-pipe-and-eof-problem/81949/1 "2024-04-16T01:02:34Z")

</div>

Hello, when I was trying to upload a dataset with over 10000+ files (totally around 600G) to my hugging face repository, I encountered the following problems:

1. The uploading speed was quite slow, around 2.7M/s, how can I improve that？
2. The uploading process was frequently interruptted by the following EOF error:

```auto
Uploading LFS objects: 0% (1/10000), 1.2 GB | 2.7 MB/s, 
write |1: broken pipe
write |1: broken pipe
EOF
EOF
error: failed to push some refs to 'https://<my_username>:<my_token>@huggingface.co/datasets/<my_username>/<my_dataset_name>

```

1. My shell comand was:

```shell
cd <my_dataset_directory>
git add .
git commit -m "some message"
git push 

```

Btw, I have enabled the LFS for large files uploading by the command `huggingface-cli lfs-enable-largefiles <my_dataset_directory>`  
Really hope that someone can give me some idea about this! Thank you~

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [April 18, 2024, 6:27pm UTC](https://discuss.huggingface.co/t/git-push-too-slow-and-interruptted-with-broken-pipe-and-eof-problem/81949/2 "2024-04-18T18:27:51Z")

</div>

Hi! We suggest using `huggingface_hub` rather than Git to interact with the Hub. You can find more info [here](https://huggingface.co/docs/huggingface_hub/guides/upload#upload-files-to-the-hub).

Splitting the files into chunks of 50-100 files and uploading them with `create_commit` should be the simplest/most robust approach for your use case.
