Hugging Face's logo

sparse-encoder
/
example-splade-distilbert-base-uncased-msmarco-mrl

Feature Extraction
sentence-transformers
Safetensors
English
distilbert
sparse-encoder
sparse
splade
Generated from Trainer
dataset_size:90000
loss:SpladeLoss
loss:SparseMultipleNegativesRankingLoss
loss:FlopsLoss
Eval Results
text-embeddings-inference
Model card Files Community
example-splade-distilbert-base-uncased-msmarco-mrl / README.md
arthurbresnu's picture
arthurbresnu HF Staff
Add new SparseEncoder model
2e04eb9 verified
preview code
|
raw
history blame contribute delete
89.7 kB
metadata
language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sparse-encoder
  - sparse
  - splade
  - generated_from_trainer
  - dataset_size:90000
  - loss:SpladeLoss
  - loss:SparseMultipleNegativesRankingLoss
  - loss:FlopsLoss
base_model: distilbert/distilbert-base-uncased
widget:
  - text: what is chess
  - text: what is a hickman for?
  - text: >-
      Steps. 1  1. Gather your materials. Here's what you need to build two
      regulations-size horseshoe pits that will face each other (if you only
      want to build one pit, halve the materials): Two 6-foot-long treated wood
      2x6s (38mm x 140mm), cut in half. 2  2. Decide where you're going to put
      your pit(s).
  - text: who played at california jam
  - text: "To the Citizens of St. Bernard We chose as our motto a simple but profound declaration: â\x80\x9CWelcome to your office.â\x80\x9D Those words remind us that we are no more than the caretakers of the office of Clerk of Court for the Parish of St. Bernard."
datasets:
  - sentence-transformers/msmarco
pipeline_tag: feature-extraction
library_name: sentence-transformers
metrics:
  - dot_accuracy@1
  - dot_accuracy@3
  - dot_accuracy@5
  - dot_accuracy@10
  - dot_precision@1
  - dot_precision@3
  - dot_precision@5
  - dot_precision@10
  - dot_recall@1
  - dot_recall@3
  - dot_recall@5
  - dot_recall@10
  - dot_ndcg@10
  - dot_mrr@10
  - dot_map@100
  - query_active_dims
  - query_sparsity_ratio
  - corpus_active_dims
  - corpus_sparsity_ratio
co2_eq_emissions:
  emissions: 20.864216098626564
  energy_consumed: 0.05652200756224921
  source: codecarbon
  training_type: fine-tuning
  on_cloud: false
  cpu_model: AMD EPYC 7R13 Processor
  ram_total_size: 248
  hours_used: 0.179
  hardware_used: 1 x NVIDIA H100 80GB HBM3
model-index:
  - name: splade-distilbert-base-uncased trained on MS MARCO triplets
    results:
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoMSMARCO
          type: NanoMSMARCO
        metrics:
          - type: dot_accuracy@1
            value: 0.44
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.66
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.72
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.82
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.44
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.22
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.14400000000000002
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.08199999999999999
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.44
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.66
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.72
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.82
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.6223979987260191
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.5599444444444444
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.5701364200315813
            name: Dot Map@100
          - type: query_active_dims
            value: 25.260000228881836
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9991724002283965
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 89.06385040283203
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9970819785596348
            name: Corpus Sparsity Ratio
          - type: dot_accuracy@1
            value: 0.44
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.6
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.74
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.84
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.44
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.2
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.14800000000000002
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.08399999999999999
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.44
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.6
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.74
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.84
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.6241753240638171
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.5571349206349206
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.5639260419913368
            name: Dot Map@100
          - type: query_active_dims
            value: 20.5
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9993283533189175
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 81.87666320800781
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9973174541901578
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoNFCorpus
          type: NanoNFCorpus
        metrics:
          - type: dot_accuracy@1
            value: 0.4
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.52
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.54
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.66
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.4
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.3666666666666667
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.332
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.27
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.023282599806398227
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.07519782108259539
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.09254782270412643
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.12120665375595915
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.32050254842735026
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.4703888888888889
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.13331879084552362
            name: Dot Map@100
          - type: query_active_dims
            value: 17.639999389648438
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9994220562417387
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 165.31358337402344
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9945837892872674
            name: Corpus Sparsity Ratio
          - type: dot_accuracy@1
            value: 0.36
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.46
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.54
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.68
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.36
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.34
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.32799999999999996
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.27
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.02081925669789383
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.07064967781220355
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.09055307754310991
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.14403725441385476
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.3196380424829849
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.4414444444444445
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.13569627052041464
            name: Dot Map@100
          - type: query_active_dims
            value: 18.299999237060547
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9994004324999325
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 156.04843139648438
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9948873458031424
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoNQ
          type: NanoNQ
        metrics:
          - type: dot_accuracy@1
            value: 0.48
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.68
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.72
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.76
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.48
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.22666666666666668
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.14400000000000002
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.08199999999999999
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.46
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.65
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.68
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.74
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.6136977374010735
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.585079365079365
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.5730967720685111
            name: Dot Map@100
          - type: query_active_dims
            value: 24.299999237060547
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9992038529835181
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 103.79106140136719
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9965994672235972
            name: Corpus Sparsity Ratio
          - type: dot_accuracy@1
            value: 0.48
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.68
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.74
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.76
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.48
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.22666666666666668
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.15200000000000002
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.08
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.47
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.64
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.7
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.73
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.6150809765850531
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.5864999999999999
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.5841443871983568
            name: Dot Map@100
          - type: query_active_dims
            value: 22.200000762939453
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9992726557642704
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 103.72532653808594
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9966016209115365
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-nano-beir
          name: Sparse Nano BEIR
        dataset:
          name: NanoBEIR mean
          type: NanoBEIR_mean
        metrics:
          - type: dot_accuracy@1
            value: 0.44
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.6200000000000001
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.66
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.7466666666666667
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.44
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.27111111111111114
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.2066666666666667
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.14466666666666664
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.30776086660213275
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.4617326070275318
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.49751594090137546
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.5604022179186531
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.5188660948514809
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.5384708994708994
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.42551732764853867
            name: Dot Map@100
          - type: query_active_dims
            value: 22.399999618530273
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9992661031512178
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 112.03345893951784
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9963294194699063
            name: Corpus Sparsity Ratio
          - type: dot_accuracy@1
            value: 0.5241130298273154
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.6799372056514913
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.7415070643642072
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.8169230769230769
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.5241130298273154
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.3215384615384615
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.2547566718995291
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.17874411302982732
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.30856930592565196
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.4441119539769697
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.5092929381431597
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.5878231569460904
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.5577320367017354
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.6173593605940545
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.48084758588880655
            name: Dot Map@100
          - type: query_active_dims
            value: 38.07395960627058
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9987525732387698
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 105.05153383516846
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9965581700466821
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoClimateFEVER
          type: NanoClimateFEVER
        metrics:
          - type: dot_accuracy@1
            value: 0.24
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.42
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.56
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.64
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.24
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.14666666666666664
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.12
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.07400000000000001
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.11833333333333332
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.21166666666666664
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.26233333333333336
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.29966666666666664
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.25712162589613363
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.35861111111111116
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.20460406106488077
            name: Dot Map@100
          - type: query_active_dims
            value: 51.47999954223633
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9983133477641624
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 134.2989959716797
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9955999280528248
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoDBPedia
          type: NanoDBPedia
        metrics:
          - type: dot_accuracy@1
            value: 0.7
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.82
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.88
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.92
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.7
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.6133333333333333
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.58
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.52
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.05306233623739282
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.16391544714816778
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.23662708539883293
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.3543605851621492
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.6137764330075132
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.771888888888889
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.4604772150699302
            name: Dot Map@100
          - type: query_active_dims
            value: 20.520000457763672
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.999327698038865
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 111.07841491699219
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9963607098185902
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoFEVER
          type: NanoFEVER
        metrics:
          - type: dot_accuracy@1
            value: 0.74
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.9
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.92
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.98
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.74
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.3133333333333333
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.19599999999999995
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.10399999999999998
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.7066666666666667
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.8666666666666667
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.8933333333333333
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.9433333333333332
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.8368149756149829
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.8170000000000001
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.7993556466302367
            name: Dot Map@100
          - type: query_active_dims
            value: 44.84000015258789
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9985308957423306
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 154.09767150878906
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9949512590423697
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoFiQA2018
          type: NanoFiQA2018
        metrics:
          - type: dot_accuracy@1
            value: 0.34
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.5
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.58
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.68
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.34
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.21333333333333332
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.17600000000000002
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.11199999999999999
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.1770793650793651
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.3069920634920635
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.3936825396825397
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.48673809523809525
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.3901649596140352
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.4438809523809523
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.32670074884185174
            name: Dot Map@100
          - type: query_active_dims
            value: 18.920000076293945
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9993801192557403
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 75.49989318847656
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9975263779179453
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoHotpotQA
          type: NanoHotpotQA
        metrics:
          - type: dot_accuracy@1
            value: 0.88
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.92
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.94
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.96
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.88
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.4866666666666666
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.324
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.16999999999999996
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.44
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.73
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.81
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.85
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.8077539978128343
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.9041666666666667
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.74474463747389
            name: Dot Map@100
          - type: query_active_dims
            value: 43.880001068115234
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9985623484349612
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 120.78840637207031
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9960425789144856
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoQuoraRetrieval
          type: NanoQuoraRetrieval
        metrics:
          - type: dot_accuracy@1
            value: 0.84
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.92
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.94
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.96
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.84
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.32666666666666666
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.22
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.11999999999999998
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.7873333333333333
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.8540000000000001
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.898
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.9313333333333332
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.8841170132005264
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.8805555555555554
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.8625873756339163
            name: Dot Map@100
          - type: query_active_dims
            value: 18.760000228881836
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9993853613711787
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 20.381887435913086
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9993322230707059
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoSCIDOCS
          type: NanoSCIDOCS
        metrics:
          - type: dot_accuracy@1
            value: 0.42
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.6
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.64
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.76
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.42
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.2866666666666667
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.21999999999999997
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.152
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.086
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.17666666666666664
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.22466666666666665
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.3116666666666667
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.31329169156104253
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.5258253968253969
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.24015404586272074
            name: Dot Map@100
          - type: query_active_dims
            value: 38.599998474121094
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9987353384943936
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 120.28081512451172
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9960592092548158
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoArguAna
          type: NanoArguAna
        metrics:
          - type: dot_accuracy@1
            value: 0.1
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.34
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.46
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.66
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.1
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.11333333333333333
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.09200000000000001
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.06600000000000002
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.1
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.34
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.46
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.66
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.35624387960476495
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.2620238095238095
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.27408886435627244
            name: Dot Map@100
          - type: query_active_dims
            value: 121.0199966430664
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9960349912639058
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 107.16836547851562
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9964888157565521
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoSciFact
          type: NanoSciFact
        metrics:
          - type: dot_accuracy@1
            value: 0.6
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.72
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.72
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.78
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.6
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.24666666666666665
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.16399999999999998
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.088
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.565
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.68
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.71
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.77
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.6798182226611048
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.6625
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.6532896014216637
            name: Dot Map@100
          - type: query_active_dims
            value: 57.41999816894531
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9981187340879056
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 158.03323364257812
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9948223172255234
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoTouche2020
          type: NanoTouche2020
        metrics:
          - type: dot_accuracy@1
            value: 0.673469387755102
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.9591836734693877
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.9795918367346939
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 1
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.673469387755102
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.6666666666666667
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.5918367346938777
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.4836734693877551
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.04710668568549065
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.13289821324817133
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.20161215990326012
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.3205651054850781
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.5525193350177682
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.814139941690962
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.40124972048901353
            name: Dot Map@100
          - type: query_active_dims
            value: 18.12244987487793
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9994062495945587
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 84.7328109741211
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9972238774990461
            name: Corpus Sparsity Ratio

splade-distilbert-base-uncased trained on MS MARCO triplets

This is a SPLADE Sparse Encoder model finetuned from distilbert/distilbert-base-uncased on the msmarco dataset using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

  • Model Type: SPLADE Sparse Encoder
  • Base model: distilbert/distilbert-base-uncased
  • Maximum Sequence Length: 256 tokens
  • Output Dimensionality: 30522 dimensions
  • Similarity Function: Dot Product
  • Training Dataset:
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture 

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("arthurbresnu/splade-distilbert-base-uncased-msmarco-mrl")
# Run inference
queries = [
    "meaning of the name bernard",
]
documents = [
    'English Meaning: The name Bernard is an English baby name. In English the meaning of the name Bernard is: Strong as a bear. See also Bjorn. American Meaning: The name Bernard is an American baby name. In American the meaning of the name Bernard is: Strong as a bear.',
    'To the Citizens of St. Bernard We chose as our motto a simple but profound declaration: â\x80\x9cWelcome to your office.â\x80\x9d Those words remind us that we are no more than the caretakers of the office of Clerk of Court for the Parish of St. Bernard.',
    "Get Your Prior Years Tax Information from the IRS. IRS Tax Tip 2012-18, January 27, 2012. Sometimes taxpayers need a copy of an old tax return, but can't find or don't have their own records. There are three easy and convenient options for getting tax return transcripts and tax account transcripts from the IRS: on the web, by phone or by mail.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[18.6221, 10.0646,  0.0000]])

Evaluation

Metrics

Sparse Information Retrieval

  • Datasets: NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
  • Evaluated with SparseInformationRetrievalEvaluator
Metric NanoMSMARCO NanoNFCorpus NanoNQ NanoClimateFEVER NanoDBPedia NanoFEVER NanoFiQA2018 NanoHotpotQA NanoQuoraRetrieval NanoSCIDOCS NanoArguAna NanoSciFact NanoTouche2020
dot_accuracy@1 0.44 0.36 0.48 0.24 0.7 0.74 0.34 0.88 0.84 0.42 0.1 0.6 0.6735
dot_accuracy@3 0.6 0.46 0.68 0.42 0.82 0.9 0.5 0.92 0.92 0.6 0.34 0.72 0.9592
dot_accuracy@5 0.74 0.54 0.74 0.56 0.88 0.92 0.58 0.94 0.94 0.64 0.46 0.72 0.9796
dot_accuracy@10 0.84 0.68 0.76 0.64 0.92 0.98 0.68 0.96 0.96 0.76 0.66 0.78 1.0
dot_precision@1 0.44 0.36 0.48 0.24 0.7 0.74 0.34 0.88 0.84 0.42 0.1 0.6 0.6735
dot_precision@3 0.2 0.34 0.2267 0.1467 0.6133 0.3133 0.2133 0.4867 0.3267 0.2867 0.1133 0.2467 0.6667
dot_precision@5 0.148 0.328 0.152 0.12 0.58 0.196 0.176 0.324 0.22 0.22 0.092 0.164 0.5918
dot_precision@10 0.084 0.27 0.08 0.074 0.52 0.104 0.112 0.17 0.12 0.152 0.066 0.088 0.4837
dot_recall@1 0.44 0.0208 0.47 0.1183 0.0531 0.7067 0.1771 0.44 0.7873 0.086 0.1 0.565 0.0471
dot_recall@3 0.6 0.0706 0.64 0.2117 0.1639 0.8667 0.307 0.73 0.854 0.1767 0.34 0.68 0.1329
dot_recall@5 0.74 0.0906 0.7 0.2623 0.2366 0.8933 0.3937 0.81 0.898 0.2247 0.46 0.71 0.2016
dot_recall@10 0.84 0.144 0.73 0.2997 0.3544 0.9433 0.4867 0.85 0.9313 0.3117 0.66 0.77 0.3206
dot_ndcg@10 0.6242 0.3196 0.6151 0.2571 0.6138 0.8368 0.3902 0.8078 0.8841 0.3133 0.3562 0.6798 0.5525
dot_mrr@10 0.5571 0.4414 0.5865 0.3586 0.7719 0.817 0.4439 0.9042 0.8806 0.5258 0.262 0.6625 0.8141
dot_map@100 0.5639 0.1357 0.5841 0.2046 0.4605 0.7994 0.3267 0.7447 0.8626 0.2402 0.2741 0.6533 0.4012
query_active_dims 20.5 18.3 22.2 51.48 20.52 44.84 18.92 43.88 18.76 38.6 121.02 57.42 18.1224
query_sparsity_ratio 0.9993 0.9994 0.9993 0.9983 0.9993 0.9985 0.9994 0.9986 0.9994 0.9987 0.996 0.9981 0.9994
corpus_active_dims 81.8767 156.0484 103.7253 134.299 111.0784 154.0977 75.4999 120.7884 20.3819 120.2808 107.1684 158.0332 84.7328
corpus_sparsity_ratio 0.9973 0.9949 0.9966 0.9956 0.9964 0.995 0.9975 0.996 0.9993 0.9961 0.9965 0.9948 0.9972

Sparse Nano BEIR

  • Dataset: NanoBEIR_mean
  • Evaluated with SparseNanoBEIREvaluator with these parameters:
    {
    "dataset_names": [
        "msmarco",
        "nfcorpus",
        "nq"
    ]
}
Metric Value
dot_accuracy@1 0.44
dot_accuracy@3 0.62
dot_accuracy@5 0.66
dot_accuracy@10 0.7467
dot_precision@1 0.44
dot_precision@3 0.2711
dot_precision@5 0.2067
dot_precision@10 0.1447
dot_recall@1 0.3078
dot_recall@3 0.4617
dot_recall@5 0.4975
dot_recall@10 0.5604
dot_ndcg@10 0.5189
dot_mrr@10 0.5385
dot_map@100 0.4255
query_active_dims 22.4
query_sparsity_ratio 0.9993
corpus_active_dims 112.0335
corpus_sparsity_ratio 0.9963

Sparse Nano BEIR

  • Dataset: NanoBEIR_mean
  • Evaluated with SparseNanoBEIREvaluator with these parameters:
    {
    "dataset_names": [
        "climatefever",
        "dbpedia",
        "fever",
        "fiqa2018",
        "hotpotqa",
        "msmarco",
        "nfcorpus",
        "nq",
        "quoraretrieval",
        "scidocs",
        "arguana",
        "scifact",
        "touche2020"
    ]
}
Metric Value
dot_accuracy@1 0.5241
dot_accuracy@3 0.6799
dot_accuracy@5 0.7415
dot_accuracy@10 0.8169
dot_precision@1 0.5241
dot_precision@3 0.3215
dot_precision@5 0.2548
dot_precision@10 0.1787
dot_recall@1 0.3086
dot_recall@3 0.4441
dot_recall@5 0.5093
dot_recall@10 0.5878
dot_ndcg@10 0.5577
dot_mrr@10 0.6174
dot_map@100 0.4808
query_active_dims 38.074
query_sparsity_ratio 0.9988
corpus_active_dims 105.0515
corpus_sparsity_ratio 0.9966

Training Details

Training Dataset

msmarco

  • Dataset: msmarco at 9e329ed
  • Size: 90,000 training samples
  • Columns: query, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    query positive negative
    type string string string
    details
    • min: 4 tokens
    • mean: 9.02 tokens
    • max: 29 tokens
    • min: 16 tokens
    • mean: 79.88 tokens
    • max: 203 tokens
    • min: 20 tokens
    • mean: 77.8 tokens
    • max: 201 tokens
  • Samples:
    query positive negative
    yosemite temperature in september Here are the average temp in Yosemite Valley (where CV is located) by month: www.nps.gov/yose/planyourvisit/climate.htm. Also beginning of September is usually still quite warm. Nights can have a bit of a chill, but nothing a couple of blankets can't handle. Guide to Switzerland weather in September. The average maximum daytime temperature in Switzerland in September is a comfortable 18°C (64°F). The average night-time temperature is usually a cool 9°C (48°F). There are usually 6 hours of bright sunshine each day, which represents 45% of the 13 hours of daylight.
    what is genus Intermediate minor rankings are not shown. A genus (/ˈdʒiːnəs/, pl. genera) is a taxonomic rank used in the biological classification of living and fossil organisms in biology. In the hierarchy of biological classification, genus comes above species and below family. In binomial nomenclature, the genus name forms the first part of the binomial species name for each species within the genus. The composition of a genus is determined by a taxonomist. The genus is the first part of a scientific name. Note that the genus is always capitalised. An example: Lemur catta is the scientific name of the Ringtailed lemur and Lemur … is the genus.Another example: Sphyrna zygaena is the scientific name of one species of Hammerhead shark and Sphyrna is the genus. name used all around the world to classify a living organism. It is composed of a genus and species name. A sceintific name can also be considered for non living things, the … se are usually called scientific jargon, or very simply 'proper names for the things around you'. 4 people found this useful.
    what did johannes kepler discover about the motion of the planets? Johannes Kepler devised his three laws of motion from his observations of planets that are fundamental to our understanding of orbital motions. Little Street, Johannes Vermeer, c. 1658. New stop on Delft tourist trail after Vermeer's Little Street identified. Few artists have left such a deep imprint on their birthplace as Johannes Vermeer on Delft. In the summer, tour parties weave through the Dutch town’s cobbled streets ticking off Vermeer landmarks.
  • Loss: SpladeLoss with these parameters:
    {
    "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
    "lambda_corpus": 0.001,
    "lambda_query": 5e-05
}

Evaluation Dataset

msmarco

  • Dataset: msmarco at 9e329ed
  • Size: 10,000 evaluation samples
  • Columns: query, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    query positive negative
    type string string string
    details
    • min: 4 tokens
    • mean: 9.16 tokens
    • max: 26 tokens
    • min: 18 tokens
    • mean: 79.89 tokens
    • max: 256 tokens
    • min: 15 tokens
    • mean: 76.95 tokens
    • max: 220 tokens
  • Samples:
    query positive negative
    scarehouse cast The Scarehouse. The Scarehouse is a 2014 Canadian horror film directed by Gavin Michael Booth. It stars Sarah Booth and Kimberly-Sue Murray as two women who seek revenge against their former sorority. Nathalie Emmanuel joined the TV series as a recurring cast member in Season 3, and continued as a recurring cast member into Season 4. Emmanuel was later promoted to a starring cast member for seasons 5 and 6.
    population of bellemont arizona The 2016 Bellemont (zip 86015), Arizona, population is 300. There are 55 people per square mile (population density). The median age is 29.9. The US median is 37.4. 38.19% of people in Bellemont (zip 86015), Arizona, are married. • Arizona: A 2010 University of Arizona report estimates that 40% of the state's kissing bugs carry a parasite strain related to the Chagas disease but rarely transmit the disease to humans. The Arizona Department of Health Services reported one Chagas disease-related death in 2013, reports The Arizona Republic.
    does air transat check bag size • Weight must be 10kg (22 lb) in Economy class and in Option Plus and 15 kg (33lb) in Club Class. Checked Baggage Air Transat allows for multiple pieces, as long as the combined weight does not exceed weight limitations. • Length + width + height must not exceed 158cm (62 in). Bag-valve masks come in different sizes to fit infants, children, and adults. The face mask size may be independent of the bag size; for example, a single pediatric-sized bag might be used with different masks for multiple face sizes, or a pediatric mask might be used with an adult bag for patients with small faces.
  • Loss: SpladeLoss with these parameters:
    {
    "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
    "lambda_corpus": 0.001,
    "lambda_query": 5e-05
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • learning_rate: 2e-05
  • num_train_epochs: 1
  • warmup_ratio: 0.1
  • bf16: True
  • load_best_model_at_end: True
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 1
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: True
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • tp_size: 0
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Epoch Step Training Loss Validation Loss NanoMSMARCO_dot_ndcg@10 NanoNFCorpus_dot_ndcg@10 NanoNQ_dot_ndcg@10 NanoBEIR_mean_dot_ndcg@10 NanoClimateFEVER_dot_ndcg@10 NanoDBPedia_dot_ndcg@10 NanoFEVER_dot_ndcg@10 NanoFiQA2018_dot_ndcg@10 NanoHotpotQA_dot_ndcg@10 NanoQuoraRetrieval_dot_ndcg@10 NanoSCIDOCS_dot_ndcg@10 NanoArguAna_dot_ndcg@10 NanoSciFact_dot_ndcg@10 NanoTouche2020_dot_ndcg@10
0.0178 100 199.0423 - - - - - - - - - - - - - - -
0.0356 200 11.3558 - - - - - - - - - - - - - - -
0.0533 300 0.9845 - - - - - - - - - - - - - - -
0.0711 400 0.4726 - - - - - - - - - - - - - - -
0.0889 500 0.2639 0.2407 0.5514 0.3061 0.5649 0.4741 - - - - - - - - - -
0.1067 600 0.2931 - - - - - - - - - - - - - - -
0.1244 700 0.2301 - - - - - - - - - - - - - - -
0.1422 800 0.2168 - - - - - - - - - - - - - - -
0.16 900 0.1741 - - - - - - - - - - - - - - -
0.1778 1000 0.1852 0.1878 0.5868 0.2975 0.5648 0.4830 - - - - - - - - - -
0.1956 1100 0.1684 - - - - - - - - - - - - - - -
0.2133 1200 0.1629 - - - - - - - - - - - - - - -
0.2311 1300 0.1736 - - - - - - - - - - - - - - -
0.2489 1400 0.1813 - - - - - - - - - - - - - - -
0.2667 1500 0.1826 0.1382 0.5941 0.3251 0.5911 0.5035 - - - - - - - - - -
0.2844 1600 0.177 - - - - - - - - - - - - - - -
0.3022 1700 0.1568 - - - - - - - - - - - - - - -
0.32 1800 0.1707 - - - - - - - - - - - - - - -
0.3378 1900 0.1554 - - - - - - - - - - - - - - -
0.3556 2000 0.1643 0.1553 0.6157 0.2997 0.5807 0.4987 - - - - - - - - - -
0.3733 2100 0.1564 - - - - - - - - - - - - - - -
0.3911 2200 0.1334 - - - - - - - - - - - - - - -
0.4089 2300 0.1349 - - - - - - - - - - - - - - -
0.4267 2400 0.1228 - - - - - - - - - - - - - - -
0.4444 2500 0.1473 0.1239 0.6242 0.3196 0.6151 0.5196 - - - - - - - - - -
0.4622 2600 0.1506 - - - - - - - - - - - - - - -
0.48 2700 0.1436 - - - - - - - - - - - - - - -
0.4978 2800 0.1471 - - - - - - - - - - - - - - -
0.5156 2900 0.1378 - - - - - - - - - - - - - - -
0.5333 3000 0.1248 0.1328 0.6077 0.3073 0.6022 0.5057 - - - - - - - - - -
0.5511 3100 0.1672 - - - - - - - - - - - - - - -
0.5689 3200 0.1301 - - - - - - - - - - - - - - -
0.5867 3300 0.1325 - - - - - - - - - - - - - - -
0.6044 3400 0.1335 - - - - - - - - - - - - - - -
0.6222 3500 0.122 0.1163 0.6081 0.3302 0.6190 0.5191 - - - - - - - - - -
0.64 3600 0.1369 - - - - - - - - - - - - - - -
0.6578 3700 0.1651 - - - - - - - - - - - - - - -
0.6756 3800 0.1243 - - - - - - - - - - - - - - -
0.6933 3900 0.1122 - - - - - - - - - - - - - - -
0.7111 4000 0.1308 0.1307 0.6013 0.3232 0.5981 0.5075 - - - - - - - - - -
0.7289 4100 0.1708 - - - - - - - - - - - - - - -
0.7467 4200 0.1143 - - - - - - - - - - - - - - -
0.7644 4300 0.167 - - - - - - - - - - - - - - -
0.7822 4400 0.1119 - - - - - - - - - - - - - - -
0.8 4500 0.1128 0.1177 0.6082 0.3228 0.5866 0.5058 - - - - - - - - - -
0.8178 4600 0.125 - - - - - - - - - - - - - - -
0.8356 4700 0.1252 - - - - - - - - - - - - - - -
0.8533 4800 0.1066 - - - - - - - - - - - - - - -
0.8711 4900 0.1196 - - - - - - - - - - - - - - -
0.8889 5000 0.1291 0.1120 0.6134 0.3230 0.6115 0.5160 - - - - - - - - - -
0.9067 5100 0.1219 - - - - - - - - - - - - - - -
0.9244 5200 0.1492 - - - - - - - - - - - - - - -
0.9422 5300 0.1138 - - - - - - - - - - - - - - -
0.96 5400 0.1583 - - - - - - - - - - - - - - -
0.9778 5500 0.1516 0.1125 0.6224 0.3205 0.6137 0.5189 - - - - - - - - - -
0.9956 5600 0.1227 - - - - - - - - - - - - - - -
-1 -1 - - 0.6242 0.3196 0.6151 0.5577 0.2571 0.6138 0.8368 0.3902 0.8078 0.8841 0.3133 0.3562 0.6798 0.5525
  • The bold row denotes the saved checkpoint.

Environmental Impact

Carbon emissions were measured using CodeCarbon.

  • Energy Consumed: 0.057 kWh
  • Carbon Emitted: 0.021 kg of CO2
  • Hours Used: 0.179 hours

Training Hardware

  • On Cloud: No
  • GPU Model: 1 x NVIDIA H100 80GB HBM3
  • CPU Model: AMD EPYC 7R13 Processor
  • RAM Size: 248.00 GB

Framework Versions

  • Python: 3.13.3
  • Sentence Transformers: 4.2.0.dev0
  • Transformers: 4.51.3
  • PyTorch: 2.7.1+cu126
  • Accelerate: 0.26.0
  • Datasets: 2.21.0
  • Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss 

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

FlopsLoss 

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
    }