sbert

This is a sentence-transformers model finetuned from sentence-transformers/bert-base-nli-mean-tokens on the fin_modern_bert-pairs-sec-synthetic-v1 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: sentence-transformers/bert-base-nli-mean-tokens
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text
  • Training Dataset:
    • fin_modern_bert-pairs-sec-synthetic-v1
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    "What was the percentage growth in third sector product sales for Aflac Japan in 2014, and how did this compare to the company's sales target?",
]
documents = [
    'Sales of third sector products increased 6.1% in 2014 , compared with the same period in 2013 , achieving the high end of our 2% to 7% sales target.',
    'Gross margin decreased 1 percentage point in 2024 when compared to 2023. Gross margin was impacted by lower sales volume and higher infrastructure costs which were partially offset by lower salary expense related to workforce reduction activities and expenses attributed to business exit activities.',
    'As of December 31, 2015 and 2014 , the Company had aggregate unfunded commitments of $ 436.4 million and $ 187.9 million, respectively, of which $ 2.2 million and $ 5.6 million, respectively, is unfunded obligations recorded in relation to the acquisition of AREA Property Partners, L.P. (“AREA”).',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7479, 0.1490, 0.1153]])

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.6376
cosine_accuracy@3 0.7946
cosine_accuracy@5 0.8436
cosine_accuracy@10 0.8886
cosine_precision@1 0.6376
cosine_precision@3 0.2649
cosine_precision@5 0.1687
cosine_precision@10 0.0889
cosine_recall@1 0.6376
cosine_recall@3 0.7946
cosine_recall@5 0.8436
cosine_recall@10 0.8886
cosine_ndcg@10 0.765
cosine_mrr@10 0.7252
cosine_map@100 0.7293

Training Details

Training Dataset

fin_modern_bert-pairs-sec-synthetic-v1

  • Dataset: fin_modern_bert-pairs-sec-synthetic-v1
  • Size: 44,709 training samples
  • Columns: query and positive
  • Approximate statistics based on the first 100 samples:
    query positive
    type string string
    modality text text
    details
    • min: 16 tokens
    • mean: 28.12 tokens
    • max: 46 tokens
    • min: 20 tokens
    • mean: 63.47 tokens
    • max: 171 tokens
  • Samples:
    query positive
    Why are quarter-to-quarter comparisons of the company's revenues and operating results considered difficult or non-meaningful? Our revenues and operating results have varied significantly from quarter to quarter because our business experiences seasonal fluctuations, which reflect seasonal trends for the travel services offered by our websites. As a result, quarter-to-quarter comparisons of our revenues and operating results may not be meaningful.
    How does the company handle the amortization of DAC and DSI for interest-sensitive life, fixed annuities, and other investment contracts? For interest-sensitive life, fixed annuities and other investment contracts, DAC and DSI are amortized in proportion to the incidence of the total present value of gross profits, which includes both actual historical gross profits (“AGP”) and estimated future gross profits (“EGP”) expected to be earned over the estimated lives of the contracts.
    What are the specific risks associated with the company's strategy of making acquisitions and investments in businesses, software products, and technologies? The risks associated with such acquisitions include, among others, the difficulty of assimilating products, operations and personnel, inheriting liabilities such as intellectual property infringement claims, the failure to realize anticipated revenue and cost projections, the requirement to test and assimilate the internal control processes of the acquired business in accordance with the requirements of Section 404 of the Sarbanes-Oxley Act of 2002 and the diversion of management's time and attention.
  • Loss: CachedMultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "mini_batch_size": 4,
        "gather_across_devices": false,
        "directions": [
            "query_to_doc"
        ],
        "partition_mode": "joint",
        "hardness_mode": null,
        "hardness_strength": 0.0
    }
    

Evaluation Dataset

fin_modern_bert-pairs-sec-synthetic-v1

  • Dataset: fin_modern_bert-pairs-sec-synthetic-v1
  • Size: 9,581 evaluation samples
  • Columns: query and positive
  • Approximate statistics based on the first 100 samples:
    query positive
    type string string
    modality text text
    details
    • min: 19 tokens
    • mean: 27.96 tokens
    • max: 41 tokens
    • min: 16 tokens
    • mean: 63.97 tokens
    • max: 142 tokens
  • Samples:
    query positive
    How could the proposed 2024 lawsuit settlement agreement between Visa and Mastercard impact the company's profitability? For example, in 2024 Visa and Mastercard proposed a lawsuit settlement agreement that would have, among other things, required reductions and caps on interchange fees and streamlined requirements for merchants who wish to impose a surcharge on credit transactions. While the settlement agreement was not approved by the court, Visa and Mastercard may ultimately agree or be subject to changes in policies or practices that result in greater surcharging generally or downward pressure on our merchant discount rates from decreases in competitor pricing in connection with reductions and caps on interchange fees.
    What are the specific dates and terms for the Indentures relating to the New 1.00% Notes and New 2.25% Notes? 4.15(w) Indenture relating to New 1.00% Notes, dated as of November 6, 2006, between priceline.com Incorporated and American Stock Transfer and Trust Company, as Trustee. 4.16(w) Indenture relating to New 2.25% Notes, dated as of November 6, 2006, between priceline.com Incorporated and American Stock Transfer and Trust Company, as Trustee.
    What are the primary competitors in the external defibrillator market according to the company's report? Our competition in the external defibrillator market comes primarily from Physio Control Corporation (a subsidiary of Medtronic Inc.) and Zoll Medical Corporation.
  • Loss: CachedMultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "mini_batch_size": 4,
        "gather_across_devices": false,
        "directions": [
            "query_to_doc"
        ],
        "partition_mode": "joint",
        "hardness_mode": null,
        "hardness_strength": 0.0
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 16
  • learning_rate: 2e-05
  • warmup_steps: 0.1
  • bf16: True
  • logging_nan_inf_filter: False
  • per_device_eval_batch_size: 16
  • seed: 777
  • data_seed: 777
  • prompts: {'query': '', 'positive': ''}
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 16
  • num_train_epochs: 3
  • max_steps: -1
  • learning_rate: 2e-05
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.1
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: False
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 16
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 777
  • data_seed: 777
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • dataloader_multiprocessing_context: None
  • dataloader_in_order: True
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • local_rank: -1
  • prompts: {'query': '', 'positive': ''}
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • warmup_ratio: None

Training Logs

Click to expand
Epoch Step Training Loss Validation Loss finance-eval_cosine_ndcg@10
-1 -1 - - 0.2731
0.0179 50 1.0162 - -
0.0358 100 0.9645 - -
0.0537 150 0.8988 - -
0.0716 200 0.9452 - -
0.0894 250 0.9270 - -
0.1073 300 0.7601 - -
0.1252 350 0.7007 - -
0.1431 400 0.6563 - -
0.1610 450 0.5641 - -
0.1789 500 0.4359 - -
0.1968 550 0.4266 - -
0.2147 600 0.3230 - -
0.2326 650 0.2815 - -
0.2504 700 0.2642 - -
0.2683 750 0.2248 - -
0.2862 800 0.2430 - -
0.3041 850 0.2061 - -
0.3220 900 0.1926 - -
0.3399 950 0.1578 - -
0.3578 1000 0.1848 - -
0.3757 1050 0.1431 - -
0.3936 1100 0.1361 - -
0.4114 1150 0.1297 - -
0.4293 1200 0.1537 - -
0.4472 1250 0.1215 - -
0.4651 1300 0.1196 - -
0.4830 1350 0.1292 - -
0.5009 1400 0.1322 - -
0.5188 1450 0.1053 - -
0.5367 1500 0.1070 - -
0.5546 1550 0.0743 - -
0.5725 1600 0.1000 - -
0.5903 1650 0.1081 - -
0.6082 1700 0.0882 - -
0.6261 1750 0.1009 - -
0.6440 1800 0.1053 - -
0.6619 1850 0.0933 - -
0.6798 1900 0.1083 - -
0.6977 1950 0.0671 - -
0.7156 2000 0.0882 - -
0.7335 2050 0.0819 - -
0.7513 2100 0.0829 - -
0.7692 2150 0.0740 - -
0.7871 2200 0.0873 - -
0.8050 2250 0.0582 - -
0.8229 2300 0.0753 - -
0.8408 2350 0.0903 - -
0.8587 2400 0.0739 - -
0.8766 2450 0.0942 - -
0.8945 2500 0.0955 - -
0.9123 2550 0.0774 - -
0.9302 2600 0.0710 - -
0.9481 2650 0.0749 - -
0.9660 2700 0.0740 - -
0.9839 2750 0.0749 - -
1.0 2795 - 0.0554 0.7511
1.0018 2800 0.0693 - -
1.0197 2850 0.0553 - -
1.0376 2900 0.0575 - -
1.0555 2950 0.0493 - -
1.0733 3000 0.0537 - -
1.0912 3050 0.0710 - -
1.1091 3100 0.0586 - -
1.1270 3150 0.0732 - -
1.1449 3200 0.0729 - -
1.1628 3250 0.0579 - -
1.1807 3300 0.0702 - -
1.1986 3350 0.0788 - -
1.2165 3400 0.0765 - -
1.2343 3450 0.0632 - -
1.2522 3500 0.0506 - -
1.2701 3550 0.0563 - -
1.2880 3600 0.0696 - -
1.3059 3650 0.0602 - -
1.3238 3700 0.0790 - -
1.3417 3750 0.0578 - -
1.3596 3800 0.0800 - -
1.3775 3850 0.0627 - -
1.3953 3900 0.0711 - -
1.4132 3950 0.0711 - -
1.4311 4000 0.0557 - -
1.4490 4050 0.0698 - -
1.4669 4100 0.0663 - -
1.4848 4150 0.0768 - -
1.5027 4200 0.0678 - -
1.5206 4250 0.0641 - -
1.5385 4300 0.0543 - -
1.5564 4350 0.0764 - -
1.5742 4400 0.0594 - -
1.5921 4450 0.0447 - -
1.6100 4500 0.0630 - -
1.6279 4550 0.0771 - -
1.6458 4600 0.0478 - -
1.6637 4650 0.0561 - -
1.6816 4700 0.0628 - -
1.6995 4750 0.0579 - -
1.7174 4800 0.0631 - -
1.7352 4850 0.0472 - -
1.7531 4900 0.0505 - -
1.7710 4950 0.0764 - -
1.7889 5000 0.0549 - -
1.8068 5050 0.0764 - -
1.8247 5100 0.0782 - -
1.8426 5150 0.0633 - -
1.8605 5200 0.0638 - -
1.8784 5250 0.0724 - -
1.8962 5300 0.0688 - -
1.9141 5350 0.0567 - -
1.9320 5400 0.0688 - -
1.9499 5450 0.0570 - -
1.9678 5500 0.0666 - -
1.9857 5550 0.0513 - -
2.0 5590 - 0.0487 0.7643
2.0036 5600 0.0745 - -
2.0215 5650 0.0609 - -
2.0394 5700 0.0765 - -
2.0572 5750 0.0423 - -
2.0751 5800 0.0551 - -
2.0930 5850 0.0637 - -
2.1109 5900 0.0695 - -
2.1288 5950 0.0638 - -
2.1467 6000 0.0629 - -
2.1646 6050 0.0749 - -
2.1825 6100 0.0485 - -
2.2004 6150 0.0474 - -
2.2182 6200 0.0466 - -
2.2361 6250 0.0608 - -
2.2540 6300 0.0563 - -
2.2719 6350 0.0610 - -
2.2898 6400 0.0573 - -
2.3077 6450 0.0614 - -
2.3256 6500 0.0700 - -
2.3435 6550 0.0560 - -
2.3614 6600 0.0888 - -
2.3792 6650 0.0788 - -
2.3971 6700 0.0622 - -
2.4150 6750 0.0481 - -
2.4329 6800 0.0696 - -
2.4508 6850 0.0723 - -
2.4687 6900 0.0492 - -
2.4866 6950 0.0615 - -
2.5045 7000 0.0651 - -
2.5224 7050 0.0601 - -
2.5403 7100 0.0484 - -
2.5581 7150 0.0570 - -
2.5760 7200 0.0516 - -
2.5939 7250 0.0640 - -
2.6118 7300 0.0590 - -
2.6297 7350 0.0603 - -
2.6476 7400 0.0678 - -
2.6655 7450 0.0591 - -
2.6834 7500 0.0515 - -
2.7013 7550 0.0485 - -
2.7191 7600 0.0602 - -
2.7370 7650 0.0462 - -
2.7549 7700 0.0608 - -
2.7728 7750 0.0626 - -
2.7907 7800 0.0652 - -
2.8086 7850 0.0532 - -
2.8265 7900 0.0660 - -
2.8444 7950 0.0521 - -
2.8623 8000 0.0566 - -
2.8801 8050 0.0515 - -
2.8980 8100 0.0639 - -
2.9159 8150 0.0796 - -
2.9338 8200 0.0559 - -
2.9517 8250 0.0519 - -
2.9696 8300 0.0710 - -
2.9875 8350 0.0510 - -
3.0 8385 - 0.0483 0.7650
-1 -1 - - 0.7650

Training Time

  • Training: 58.3 minutes

Framework Versions

  • Python: 3.13.16
  • Sentence Transformers: 5.6.0
  • Transformers: 5.22.1
  • PyTorch: 2.14.1+cu130
  • Accelerate: 1.14.0
  • Datasets: 4.8.5
  • Tokenizers: 0.23.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
Downloads last month
24
Safetensors
Model size
0.1B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for finvector/sbert_raw_finmodern_pair_merged

Finetuned
(6)
this model

Papers for finvector/sbert_raw_finmodern_pair_merged

Evaluation results