Hugging Face's logo

nrishabh
/
llama3-8b-instruct-qlora-large

PEFT
Safetensors
trl
sft
Generated from Trainer
Model card Files
xet
 Community
nrishabh commited on
Commit
de0f7bd
·
verified ·
1 Parent(s): 425f118

End of training

Browse files
Files changed (4) hide show
  1. README.md +32 -12
  2. adapter_config.json +2 -2
  3. adapter_model.safetensors +1 -1
  4. training_args.bin +1 -1
README.md CHANGED
@@ -18,7 +18,7 @@ should probably proofread and complete it, then remove this comment. -->
18
 
19
  This model is a fine-tuned version of [LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank](https://huggingface.co/LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank) on an unknown dataset.
20
  It achieves the following results on the evaluation set:
21
- - Loss: 0.9675
22
 
23
  ## Model description
24
 
@@ -43,22 +43,42 @@ The following hyperparameters were used during training:
43
  - seed: 42
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
  - lr_scheduler_type: cosine
46
- - num_epochs: 10
47
 
48
  ### Training results
49
 
50
  | Training Loss | Epoch | Step | Validation Loss |
51
  |:-------------:|:-----:|:----:|:---------------:|
52
- | 2.3454 | 1.0 | 158 | 1.2455 |
53
- | 2.1309 | 2.0 | 316 | 1.0947 |
54
- | 2.043 | 3.0 | 474 | 1.0464 |
55
- | 1.9615 | 4.0 | 632 | 1.0167 |
56
- | 1.8828 | 5.0 | 790 | 0.9873 |
57
- | 1.8103 | 6.0 | 948 | 0.9738 |
58
- | 1.7546 | 7.0 | 1106 | 0.9691 |
59
- | 1.7231 | 8.0 | 1264 | 0.9668 |
60
- | 1.7081 | 9.0 | 1422 | 0.9677 |
61
- | 1.6978 | 10.0 | 1580 | 0.9675 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
 
63
 
64
  ### Framework versions
 
18
 
19
  This model is a fine-tuned version of [LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank](https://huggingface.co/LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank) on an unknown dataset.
20
  It achieves the following results on the evaluation set:
21
+ - Loss: 0.8530
22
 
23
  ## Model description
24
 
 
43
  - seed: 42
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
  - lr_scheduler_type: cosine
46
+ - num_epochs: 30
47
 
48
  ### Training results
49
 
50
  | Training Loss | Epoch | Step | Validation Loss |
51
  |:-------------:|:-----:|:----:|:---------------:|
52
+ | 2.3454 | 1.0 | 158 | 1.2439 |
53
+ | 2.1288 | 2.0 | 316 | 1.0900 |
54
+ | 2.0335 | 3.0 | 474 | 1.0394 |
55
+ | 1.9315 | 4.0 | 632 | 0.9995 |
56
+ | 1.804 | 5.0 | 790 | 0.9605 |
57
+ | 1.6583 | 6.0 | 948 | 0.9411 |
58
+ | 1.4994 | 7.0 | 1106 | 0.9283 |
59
+ | 1.3388 | 8.0 | 1264 | 0.9158 |
60
+ | 1.1894 | 9.0 | 1422 | 0.9103 |
61
+ | 1.0616 | 10.0 | 1580 | 0.9027 |
62
+ | 0.9461 | 11.0 | 1738 | 0.8963 |
63
+ | 0.8447 | 12.0 | 1896 | 0.8922 |
64
+ | 0.7575 | 13.0 | 2054 | 0.8887 |
65
+ | 0.6817 | 14.0 | 2212 | 0.8803 |
66
+ | 0.6192 | 15.0 | 2370 | 0.8761 |
67
+ | 0.5669 | 16.0 | 2528 | 0.8715 |
68
+ | 0.5196 | 17.0 | 2686 | 0.8719 |
69
+ | 0.479 | 18.0 | 2844 | 0.8683 |
70
+ | 0.4473 | 19.0 | 3002 | 0.8662 |
71
+ | 0.4202 | 20.0 | 3160 | 0.8624 |
72
+ | 0.397 | 21.0 | 3318 | 0.8590 |
73
+ | 0.377 | 22.0 | 3476 | 0.8573 |
74
+ | 0.3622 | 23.0 | 3634 | 0.8558 |
75
+ | 0.3514 | 24.0 | 3792 | 0.8548 |
76
+ | 0.3434 | 25.0 | 3950 | 0.8543 |
77
+ | 0.3349 | 26.0 | 4108 | 0.8541 |
78
+ | 0.332 | 27.0 | 4266 | 0.8538 |
79
+ | 0.328 | 28.0 | 4424 | 0.8541 |
80
+ | 0.3286 | 29.0 | 4582 | 0.8532 |
81
+ | 0.3279 | 30.0 | 4740 | 0.8530 |
82
 
83
 
84
  ### Framework versions
adapter_config.json CHANGED
@@ -20,11 +20,11 @@
20
  "rank_pattern": {},
21
  "revision": null,
22
  "target_modules": [
23
- "q_proj",
24
  "o_proj",
25
  "k_proj",
26
  "all-linear",
27
- "v_proj"
28
  ],
29
  "task_type": "CAUSAL_LM",
30
  "use_dora": false,
 
20
  "rank_pattern": {},
21
  "revision": null,
22
  "target_modules": [
23
+ "v_proj",
24
  "o_proj",
25
  "k_proj",
26
  "all-linear",
27
+ "q_proj"
28
  ],
29
  "task_type": "CAUSAL_LM",
30
  "use_dora": false,
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5df0c3aeedd7de88a9e5a7477d9a878b5c6d3ea1a97496be99b422a27bbc64f6
3
  size 218138576
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2d78f98f66fae54ea02a26db118377f56bd46a4fad3e22a1c9380d770b22a02d
3
  size 218138576
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:56bb3002bb9eb9b76d5683f4c945445b2c56cf5f671b7bd03273380597f0f3bf
3
  size 5048
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1566104477e43dbd953449afb6f0974e2c7765e1042ea8d06d2a920964a1cde5
3
  size 5048