Update Prompt

.cache/chunks_6635d94cf9023c83521982b3043ec70c.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:a4cef2cc09ef9d4ef7d8649bb78ec868e356dcfecbcd6dde23442a90497d407e
-size 124546

.cache/embeddings_b24811e7d333cc7d5047e52b357abd7e.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:475523b57f8f6b89e62e668efef73309193b05f0f05bbeffb7f012ee952024f0
-size 347400

.cache/hub/models--sentence-transformers--all-MiniLM-L6-v2/refs/main

@@ -1 +0,0 @@
-c9745ed1d9f207416be6d2e6f8de32d1f16199bf

.cache/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/1_Pooling/config.json

@@ -1,7 +0,0 @@
-{
-  "word_embedding_dimension": 384,
-  "pooling_mode_cls_token": false,
-  "pooling_mode_mean_tokens": true,
-  "pooling_mode_max_tokens": false,
-  "pooling_mode_mean_sqrt_len_tokens": false
-}

.cache/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/README.md

@@ -1,173 +0,0 @@
----
-language: en
-license: apache-2.0
-library_name: sentence-transformers
-tags:
-- sentence-transformers
-- feature-extraction
-- sentence-similarity
-- transformers
-datasets:
-- s2orc
-- flax-sentence-embeddings/stackexchange_xml
-- ms_marco
-- gooaq
-- yahoo_answers_topics
-- code_search_net
-- search_qa
-- eli5
-- snli
-- multi_nli
-- wikihow
-- natural_questions
-- trivia_qa
-- embedding-data/sentence-compression
-- embedding-data/flickr30k-captions
-- embedding-data/altlex
-- embedding-data/simple-wiki
-- embedding-data/QQP
-- embedding-data/SPECTER
-- embedding-data/PAQ_pairs
-- embedding-data/WikiAnswers
-pipeline_tag: sentence-similarity
----
-
-
-# all-MiniLM-L6-v2
-This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.
-
-## Usage (Sentence-Transformers)
-Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
-
-```
-pip install -U sentence-transformers
-```
-
-Then you can use the model like this:
-```python
-from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
-
-model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
-embeddings = model.encode(sentences)
-print(embeddings)
-```
-
-## Usage (HuggingFace Transformers)
-Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
-
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-import torch.nn.functional as F
-
-#Mean Pooling - Take attention mask into account for correct averaging
-def mean_pooling(model_output, attention_mask):
-    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
-    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
-
-
-# Sentences we want sentence embeddings for
-sentences = ['This is an example sentence', 'Each sentence is converted']
-
-# Load model from HuggingFace Hub
-tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
-model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
-
-# Tokenize sentences
-encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
-
-# Compute token embeddings
-with torch.no_grad():
-    model_output = model(**encoded_input)
-
-# Perform pooling
-sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
-
-# Normalize embeddings
-sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)
-
-print("Sentence embeddings:")
-print(sentence_embeddings)
-```
-
-------
-
-## Background
-
-The project aims to train sentence embedding models on very large sentence level datasets using a self-supervised 
-contrastive learning objective. We used the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model and fine-tuned in on a 
-1B sentence pairs dataset. We use a contrastive learning objective: given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset.
-
-We developed this model during the 
-[Community week using JAX/Flax for NLP & CV](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), 
-organized by Hugging Face. We developed this model as part of the project:
-[Train the Best Sentence Embedding Model Ever with 1B Training Pairs](https://discuss.huggingface.co/t/train-the-best-sentence-embedding-model-ever-with-1b-training-pairs/7354). We benefited from efficient hardware infrastructure to run the project: 7 TPUs v3-8, as well as intervention from Googles Flax, JAX, and Cloud team member about efficient deep learning frameworks.
-
-## Intended uses
-
-Our model is intended to be used as a sentence and short paragraph encoder. Given an input text, it outputs a vector which captures 
-the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.
-
-By default, input text longer than 256 word pieces is truncated.
-
-
-## Training procedure
-
-### Pre-training 
-
-We use the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model. Please refer to the model card for more detailed information about the pre-training procedure.
-
-### Fine-tuning 
-
-We fine-tune the model using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch.
-We then apply the cross entropy loss by comparing with true pairs.
-
-#### Hyper parameters
-
-We trained our model on a TPU v3-8. We train the model during 100k steps using a batch size of 1024 (128 per TPU core).
-We use a learning rate warm up of 500. The sequence length was limited to 128 tokens. We used the AdamW optimizer with
-a 2e-5 learning rate. The full training script is accessible in this current repository: `train_script.py`.
-
-#### Training data
-
-We use the concatenation from multiple datasets to fine-tune our model. The total number of sentence pairs is above 1 billion sentences.
-We sampled each dataset given a weighted probability which configuration is detailed in the `data_config.json` file.
-
-
-| Dataset                                                  | Paper                                    | Number of training tuples  |
-|--------------------------------------------------------|:----------------------------------------:|:--------------------------:|
-| [Reddit comments (2015-2018)](https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit) | [paper](https://arxiv.org/abs/1904.06472) | 726,484,430 |
-| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Abstracts) | [paper](https://aclanthology.org/2020.acl-main.447/) | 116,288,806 |
-| [WikiAnswers](https://github.com/afader/oqa#wikianswers-corpus) Duplicate question pairs | [paper](https://doi.org/10.1145/2623330.2623677) | 77,427,422 |
-| [PAQ](https://github.com/facebookresearch/PAQ) (Question, Answer) pairs | [paper](https://arxiv.org/abs/2102.07033) | 64,371,441 |
-| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Titles) | [paper](https://aclanthology.org/2020.acl-main.447/) | 52,603,982 |
-| [S2ORC](https://github.com/allenai/s2orc) (Title, Abstract) | [paper](https://aclanthology.org/2020.acl-main.447/) | 41,769,185 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Body) pairs  | - | 25,316,456 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title+Body, Answer) pairs  | - | 21,396,559 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Answer) pairs  | - | 21,396,559 |
-| [MS MARCO](https://microsoft.github.io/msmarco/) triplets | [paper](https://doi.org/10.1145/3404835.3462804) | 9,144,553 |
-| [GOOAQ: Open Question Answering with Diverse Answer Types](https://github.com/allenai/gooaq) | [paper](https://arxiv.org/pdf/2104.08727.pdf) | 3,012,496 |
-| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 1,198,260 |
-| [Code Search](https://huggingface.co/datasets/code_search_net) | - | 1,151,414 |
-| [COCO](https://cocodataset.org/#home) Image captions | [paper](https://link.springer.com/chapter/10.1007%2F978-3-319-10602-1_48) | 828,395|
-| [SPECTER](https://github.com/allenai/specter) citation triplets | [paper](https://doi.org/10.18653/v1/2020.acl-main.207) | 684,100 |
-| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Question, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 681,164 |
-| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Question) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 659,896 |
-| [SearchQA](https://huggingface.co/datasets/search_qa) | [paper](https://arxiv.org/abs/1704.05179) | 582,261 |
-| [Eli5](https://huggingface.co/datasets/eli5) | [paper](https://doi.org/10.18653/v1/p19-1346) | 325,475 |
-| [Flickr 30k](https://shannon.cs.illinois.edu/DenotationGraph/) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/229/33) | 317,695 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles) | | 304,525 |
-| AllNLI ([SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) | [paper SNLI](https://doi.org/10.18653/v1/d15-1075), [paper MultiNLI](https://doi.org/10.18653/v1/n18-1101) | 277,230 | 
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (bodies) | | 250,519 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles+bodies) | | 250,460 |
-| [Sentence Compression](https://github.com/google-research-datasets/sentence-compression) | [paper](https://www.aclweb.org/anthology/D13-1155/) | 180,000 |
-| [Wikihow](https://github.com/pvl/wikihow_pairs_dataset) | [paper](https://arxiv.org/abs/1810.09305) | 128,542 |
-| [Altlex](https://github.com/chridey/altlex/) | [paper](https://aclanthology.org/P16-1135.pdf) | 112,696 |
-| [Quora Question Triplets](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs) | - | 103,663 |
-| [Simple Wikipedia](https://cs.pomona.edu/~dkauchak/simplification/) | [paper](https://www.aclweb.org/anthology/P11-2117/) | 102,225 |
-| [Natural Questions (NQ)](https://ai.google.com/research/NaturalQuestions) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/1455) | 100,231 |
-| [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) | [paper](https://aclanthology.org/P18-2124.pdf) | 87,599 |
-| [TriviaQA](https://huggingface.co/datasets/trivia_qa) | - | 73,346 |
-| **Total** | | **1,170,060,424** |

.cache/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/config_sentence_transformers.json

@@ -1,7 +0,0 @@
-{
-  "__version__": {
-    "sentence_transformers": "2.0.0",
-    "transformers": "4.6.1",
-    "pytorch": "1.8.1"
-  }
-}

.cache/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/modules.json

@@ -1,20 +0,0 @@
-[
-  {
-    "idx": 0,
-    "name": "0",
-    "path": "",
-    "type": "sentence_transformers.models.Transformer"
-  },
-  {
-    "idx": 1,
-    "name": "1",
-    "path": "1_Pooling",
-    "type": "sentence_transformers.models.Pooling"
-  },
-  {
-    "idx": 2,
-    "name": "2",
-    "path": "2_Normalize",
-    "type": "sentence_transformers.models.Normalize"
-  }
-]

.cache/hub/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/sentence_bert_config.json

@@ -1,4 +0,0 @@
-{
-  "max_seq_length": 256,
-  "do_lower_case": false
-}

.cache/models--sentence-transformers--all-MiniLM-L6-v2/refs/main

@@ -1 +0,0 @@
-c9745ed1d9f207416be6d2e6f8de32d1f16199bf

.cache/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/1_Pooling/config.json

@@ -1,7 +0,0 @@
-{
-  "word_embedding_dimension": 384,
-  "pooling_mode_cls_token": false,
-  "pooling_mode_mean_tokens": true,
-  "pooling_mode_max_tokens": false,
-  "pooling_mode_mean_sqrt_len_tokens": false
-}

.cache/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/README.md

@@ -1,173 +0,0 @@
----
-language: en
-license: apache-2.0
-library_name: sentence-transformers
-tags:
-- sentence-transformers
-- feature-extraction
-- sentence-similarity
-- transformers
-datasets:
-- s2orc
-- flax-sentence-embeddings/stackexchange_xml
-- ms_marco
-- gooaq
-- yahoo_answers_topics
-- code_search_net
-- search_qa
-- eli5
-- snli
-- multi_nli
-- wikihow
-- natural_questions
-- trivia_qa
-- embedding-data/sentence-compression
-- embedding-data/flickr30k-captions
-- embedding-data/altlex
-- embedding-data/simple-wiki
-- embedding-data/QQP
-- embedding-data/SPECTER
-- embedding-data/PAQ_pairs
-- embedding-data/WikiAnswers
-pipeline_tag: sentence-similarity
----
-
-
-# all-MiniLM-L6-v2
-This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.
-
-## Usage (Sentence-Transformers)
-Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
-
-```
-pip install -U sentence-transformers
-```
-
-Then you can use the model like this:
-```python
-from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
-
-model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
-embeddings = model.encode(sentences)
-print(embeddings)
-```
-
-## Usage (HuggingFace Transformers)
-Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
-
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-import torch.nn.functional as F
-
-#Mean Pooling - Take attention mask into account for correct averaging
-def mean_pooling(model_output, attention_mask):
-    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
-    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
-
-
-# Sentences we want sentence embeddings for
-sentences = ['This is an example sentence', 'Each sentence is converted']
-
-# Load model from HuggingFace Hub
-tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
-model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
-
-# Tokenize sentences
-encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
-
-# Compute token embeddings
-with torch.no_grad():
-    model_output = model(**encoded_input)
-
-# Perform pooling
-sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
-
-# Normalize embeddings
-sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)
-
-print("Sentence embeddings:")
-print(sentence_embeddings)
-```
-
-------
-
-## Background
-
-The project aims to train sentence embedding models on very large sentence level datasets using a self-supervised 
-contrastive learning objective. We used the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model and fine-tuned in on a 
-1B sentence pairs dataset. We use a contrastive learning objective: given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset.
-
-We developed this model during the 
-[Community week using JAX/Flax for NLP & CV](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), 
-organized by Hugging Face. We developed this model as part of the project:
-[Train the Best Sentence Embedding Model Ever with 1B Training Pairs](https://discuss.huggingface.co/t/train-the-best-sentence-embedding-model-ever-with-1b-training-pairs/7354). We benefited from efficient hardware infrastructure to run the project: 7 TPUs v3-8, as well as intervention from Googles Flax, JAX, and Cloud team member about efficient deep learning frameworks.
-
-## Intended uses
-
-Our model is intended to be used as a sentence and short paragraph encoder. Given an input text, it outputs a vector which captures 
-the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.
-
-By default, input text longer than 256 word pieces is truncated.
-
-
-## Training procedure
-
-### Pre-training 
-
-We use the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model. Please refer to the model card for more detailed information about the pre-training procedure.
-
-### Fine-tuning 
-
-We fine-tune the model using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch.
-We then apply the cross entropy loss by comparing with true pairs.
-
-#### Hyper parameters
-
-We trained our model on a TPU v3-8. We train the model during 100k steps using a batch size of 1024 (128 per TPU core).
-We use a learning rate warm up of 500. The sequence length was limited to 128 tokens. We used the AdamW optimizer with
-a 2e-5 learning rate. The full training script is accessible in this current repository: `train_script.py`.
-
-#### Training data
-
-We use the concatenation from multiple datasets to fine-tune our model. The total number of sentence pairs is above 1 billion sentences.
-We sampled each dataset given a weighted probability which configuration is detailed in the `data_config.json` file.
-
-
-| Dataset                                                  | Paper                                    | Number of training tuples  |
-|--------------------------------------------------------|:----------------------------------------:|:--------------------------:|
-| [Reddit comments (2015-2018)](https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit) | [paper](https://arxiv.org/abs/1904.06472) | 726,484,430 |
-| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Abstracts) | [paper](https://aclanthology.org/2020.acl-main.447/) | 116,288,806 |
-| [WikiAnswers](https://github.com/afader/oqa#wikianswers-corpus) Duplicate question pairs | [paper](https://doi.org/10.1145/2623330.2623677) | 77,427,422 |
-| [PAQ](https://github.com/facebookresearch/PAQ) (Question, Answer) pairs | [paper](https://arxiv.org/abs/2102.07033) | 64,371,441 |
-| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Titles) | [paper](https://aclanthology.org/2020.acl-main.447/) | 52,603,982 |
-| [S2ORC](https://github.com/allenai/s2orc) (Title, Abstract) | [paper](https://aclanthology.org/2020.acl-main.447/) | 41,769,185 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Body) pairs  | - | 25,316,456 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title+Body, Answer) pairs  | - | 21,396,559 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Answer) pairs  | - | 21,396,559 |
-| [MS MARCO](https://microsoft.github.io/msmarco/) triplets | [paper](https://doi.org/10.1145/3404835.3462804) | 9,144,553 |
-| [GOOAQ: Open Question Answering with Diverse Answer Types](https://github.com/allenai/gooaq) | [paper](https://arxiv.org/pdf/2104.08727.pdf) | 3,012,496 |
-| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 1,198,260 |
-| [Code Search](https://huggingface.co/datasets/code_search_net) | - | 1,151,414 |
-| [COCO](https://cocodataset.org/#home) Image captions | [paper](https://link.springer.com/chapter/10.1007%2F978-3-319-10602-1_48) | 828,395|
-| [SPECTER](https://github.com/allenai/specter) citation triplets | [paper](https://doi.org/10.18653/v1/2020.acl-main.207) | 684,100 |
-| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Question, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 681,164 |
-| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Question) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 659,896 |
-| [SearchQA](https://huggingface.co/datasets/search_qa) | [paper](https://arxiv.org/abs/1704.05179) | 582,261 |
-| [Eli5](https://huggingface.co/datasets/eli5) | [paper](https://doi.org/10.18653/v1/p19-1346) | 325,475 |
-| [Flickr 30k](https://shannon.cs.illinois.edu/DenotationGraph/) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/229/33) | 317,695 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles) | | 304,525 |
-| AllNLI ([SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) | [paper SNLI](https://doi.org/10.18653/v1/d15-1075), [paper MultiNLI](https://doi.org/10.18653/v1/n18-1101) | 277,230 | 
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (bodies) | | 250,519 |
-| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles+bodies) | | 250,460 |
-| [Sentence Compression](https://github.com/google-research-datasets/sentence-compression) | [paper](https://www.aclweb.org/anthology/D13-1155/) | 180,000 |
-| [Wikihow](https://github.com/pvl/wikihow_pairs_dataset) | [paper](https://arxiv.org/abs/1810.09305) | 128,542 |
-| [Altlex](https://github.com/chridey/altlex/) | [paper](https://aclanthology.org/P16-1135.pdf) | 112,696 |
-| [Quora Question Triplets](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs) | - | 103,663 |
-| [Simple Wikipedia](https://cs.pomona.edu/~dkauchak/simplification/) | [paper](https://www.aclweb.org/anthology/P11-2117/) | 102,225 |
-| [Natural Questions (NQ)](https://ai.google.com/research/NaturalQuestions) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/1455) | 100,231 |
-| [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) | [paper](https://aclanthology.org/P18-2124.pdf) | 87,599 |
-| [TriviaQA](https://huggingface.co/datasets/trivia_qa) | - | 73,346 |
-| **Total** | | **1,170,060,424** |

.cache/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/config.json

@@ -1,24 +0,0 @@
-{
-  "_name_or_path": "nreimers/MiniLM-L6-H384-uncased",
-  "architectures": [
-    "BertModel"
-  ],
-  "attention_probs_dropout_prob": 0.1,
-  "gradient_checkpointing": false,
-  "hidden_act": "gelu",
-  "hidden_dropout_prob": 0.1,
-  "hidden_size": 384,
-  "initializer_range": 0.02,
-  "intermediate_size": 1536,
-  "layer_norm_eps": 1e-12,
-  "max_position_embeddings": 512,
-  "model_type": "bert",
-  "num_attention_heads": 12,
-  "num_hidden_layers": 6,
-  "pad_token_id": 0,
-  "position_embedding_type": "absolute",
-  "transformers_version": "4.8.2",
-  "type_vocab_size": 2,
-  "use_cache": true,
-  "vocab_size": 30522
-}

.cache/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/config_sentence_transformers.json

@@ -1,7 +0,0 @@
-{
-  "__version__": {
-    "sentence_transformers": "2.0.0",
-    "transformers": "4.6.1",
-    "pytorch": "1.8.1"
-  }
-}

.cache/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:53aa51172d142c89d9012cce15ae4d6cc0ca6895895114379cacb4fab128d9db
-size 90868376

.cache/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/modules.json

@@ -1,20 +0,0 @@
-[
-  {
-    "idx": 0,
-    "name": "0",
-    "path": "",
-    "type": "sentence_transformers.models.Transformer"
-  },
-  {
-    "idx": 1,
-    "name": "1",
-    "path": "1_Pooling",
-    "type": "sentence_transformers.models.Pooling"
-  },
-  {
-    "idx": 2,
-    "name": "2",
-    "path": "2_Normalize",
-    "type": "sentence_transformers.models.Normalize"
-  }
-]

.cache/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/sentence_bert_config.json

@@ -1,4 +0,0 @@
-{
-  "max_seq_length": 256,
-  "do_lower_case": false
-}

.cache/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/special_tokens_map.json

@@ -1 +0,0 @@
-{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

.cache/models--sentence-transformers--all-MiniLM-L6-v2/snapshots/c9745ed1d9f207416be6d2e6f8de32d1f16199bf/tokenizer_config.json

@@ -1 +0,0 @@
-{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "name_or_path": "nreimers/MiniLM-L6-H384-uncased", "do_basic_tokenize": true, "never_split": null, "tokenizer_class": "BertTokenizer", "model_max_length": 512}

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/refs/main

@@ -1 +0,0 @@
-4ca70771034acceecb2e72475f72050fcdde4ddc

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/snapshots/4ca70771034acceecb2e72475f72050fcdde4ddc/1_Pooling/config.json

@@ -1,7 +0,0 @@
-{
-  "word_embedding_dimension": 384,
-  "pooling_mode_cls_token": false,
-  "pooling_mode_mean_tokens": true,
-  "pooling_mode_max_tokens": false,
-  "pooling_mode_mean_sqrt_len_tokens": false
-}

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/snapshots/4ca70771034acceecb2e72475f72050fcdde4ddc/README.md

@@ -1,114 +0,0 @@
----
-license: apache-2.0
-library_name: sentence-transformers
-tags:
-- sentence-transformers
-- feature-extraction
-- sentence-similarity
-- transformers
-datasets:
-- flax-sentence-embeddings/stackexchange_xml
-- s2orc
-- ms_marco
-- wiki_atomic_edits
-- snli
-- multi_nli
-- embedding-data/altlex
-- embedding-data/simple-wiki
-- embedding-data/flickr30k-captions
-- embedding-data/coco_captions
-- embedding-data/sentence-compression
-- embedding-data/QQP
-- yahoo_answers_topics
-pipeline_tag: sentence-similarity
----
-
-# sentence-transformers/paraphrase-MiniLM-L3-v2
-
-This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.
-
-
-
-## Usage (Sentence-Transformers)
-
-Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
-
-```
-pip install -U sentence-transformers
-```
-
-Then you can use the model like this:
-
-```python
-from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
-
-model = SentenceTransformer('sentence-transformers/paraphrase-MiniLM-L3-v2')
-embeddings = model.encode(sentences)
-print(embeddings)
-```
-
-
-
-## Usage (HuggingFace Transformers)
-Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
-
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-
-
-#Mean Pooling - Take attention mask into account for correct averaging
-def mean_pooling(model_output, attention_mask):
-    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
-    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
-
-
-# Sentences we want sentence embeddings for
-sentences = ['This is an example sentence', 'Each sentence is converted']
-
-# Load model from HuggingFace Hub
-tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-MiniLM-L3-v2')
-model = AutoModel.from_pretrained('sentence-transformers/paraphrase-MiniLM-L3-v2')
-
-# Tokenize sentences
-encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
-
-# Compute token embeddings
-with torch.no_grad():
-    model_output = model(**encoded_input)
-
-# Perform pooling. In this case, max pooling.
-sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
-
-print("Sentence embeddings:")
-print(sentence_embeddings)
-```
-
-
-
-## Full Model Architecture
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
-  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
-)
-```
-
-## Citing & Authors
-
-This model was trained by [sentence-transformers](https://www.sbert.net/). 
-        
-If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):
-```bibtex 
-@inproceedings{reimers-2019-sentence-bert,
-    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
-    author = "Reimers, Nils and Gurevych, Iryna",
-    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
-    month = "11",
-    year = "2019",
-    publisher = "Association for Computational Linguistics",
-    url = "http://arxiv.org/abs/1908.10084",
-}
-```

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/snapshots/4ca70771034acceecb2e72475f72050fcdde4ddc/config.json

@@ -1,24 +0,0 @@
-{
-  "_name_or_path": "old_models/paraphrase-MiniLM-L3-v2/0_Transformer",
-  "architectures": [
-    "BertModel"
-  ],
-  "attention_probs_dropout_prob": 0.1,
-  "gradient_checkpointing": false,
-  "hidden_act": "gelu",
-  "hidden_dropout_prob": 0.1,
-  "hidden_size": 384,
-  "initializer_range": 0.02,
-  "intermediate_size": 1536,
-  "layer_norm_eps": 1e-12,
-  "max_position_embeddings": 512,
-  "model_type": "bert",
-  "num_attention_heads": 12,
-  "num_hidden_layers": 3,
-  "pad_token_id": 0,
-  "position_embedding_type": "absolute",
-  "transformers_version": "4.7.0",
-  "type_vocab_size": 2,
-  "use_cache": true,
-  "vocab_size": 30522
-}

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/snapshots/4ca70771034acceecb2e72475f72050fcdde4ddc/config_sentence_transformers.json

@@ -1,7 +0,0 @@
-{
-  "__version__": {
-    "sentence_transformers": "2.0.0",
-    "transformers": "4.7.0",
-    "pytorch": "1.9.0+cu102"
-  }
-}

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/snapshots/4ca70771034acceecb2e72475f72050fcdde4ddc/model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:cf1e4e2d420c664973037c3c73125d7a8fc69952495093ef8f50596f8943a433
-size 69569488

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/snapshots/4ca70771034acceecb2e72475f72050fcdde4ddc/modules.json

@@ -1,14 +0,0 @@
-[
-  {
-    "idx": 0,
-    "name": "0",
-    "path": "",
-    "type": "sentence_transformers.models.Transformer"
-  },
-  {
-    "idx": 1,
-    "name": "1",
-    "path": "1_Pooling",
-    "type": "sentence_transformers.models.Pooling"
-  }
-]

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/snapshots/4ca70771034acceecb2e72475f72050fcdde4ddc/sentence_bert_config.json

@@ -1,4 +0,0 @@
-{
-  "max_seq_length": 128,
-  "do_lower_case": false
-}

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/snapshots/4ca70771034acceecb2e72475f72050fcdde4ddc/special_tokens_map.json

@@ -1 +0,0 @@
-{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

.cache/models--sentence-transformers--paraphrase-MiniLM-L3-v2/snapshots/4ca70771034acceecb2e72475f72050fcdde4ddc/tokenizer_config.json

@@ -1 +0,0 @@
-{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "name_or_path": "nreimers/MiniLM-L3-H384-uncased", "do_basic_tokenize": true, "never_split": null, "model_max_length": 512}

.cache/response_2ab720ffccd688afdc790db13e338c83.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:1c5853e52bd3fdc0bdf05ca5b73769bc17fe8f44fe56271a78a87f155c5de6da
-size 429

.gitattributes

@@ -1,35 +0,0 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

HUGGINGFACE_DEPLOYMENT.md

@@ -1,112 +0,0 @@
-# Hugging Face Spaces Deployment Guide
-
-This guide will help you deploy your HackRx Insurance Policy Assistant to Hugging Face Spaces.
-
-## Prerequisites
-
-1. A Hugging Face account (free at https://huggingface.co)
-2. A Google Gemini API key
-3. Your code pushed to a Git repository (GitHub, GitLab, etc.)
-
-## Step 1: Prepare Your Repository
-
-Your repository should contain the following files:
-- `app.py` - Main application entry point
-- `Dockerfile` - Docker configuration
-- `requirements.txt` - Python dependencies
-- `parser.py`, `embedder.py`, `retriever.py`, `llm.py` - Application modules
-- `.dockerignore` - Docker build optimization
-
-## Step 2: Create a Hugging Face Space
-
-1. Go to https://huggingface.co/spaces
-2. Click "Create new Space"
-3. Choose the following settings:
-   - **Owner**: Your username
-   - **Space name**: `hackrx-insurance-assistant` (or your preferred name)
-   - **Space SDK**: `Docker`
-   - **License**: Choose appropriate license
-   - **Visibility**: Public or Private (your choice)
-
-## Step 3: Connect Your Repository
-
-1. In your new Space, go to the "Settings" tab
-2. Under "Repository", click "Connect to existing repository"
-3. Select your Git provider (GitHub, GitLab, etc.)
-4. Choose your repository
-5. Click "Connect"
-
-## Step 4: Configure Environment Variables
-
-1. In your Space settings, go to the "Repository secrets" section
-2. Add the following secret:
-   - **Name**: `GOOGLE_API_KEY`
-   - **Value**: Your Google Gemini API key
-
-## Step 5: Deploy
-
-1. Push your code to your Git repository
-2. Hugging Face Spaces will automatically detect the changes and start building
-3. You can monitor the build progress in the "Logs" tab
-4. Once built successfully, your API will be available at `https://your-space-name.hf.space`
-
-## Step 6: Test Your Deployment
-
-### Health Check
-```bash
-curl https://your-space-name.hf.space/
-```
-
-### Test API Endpoint
-```bash
-curl -X POST https://your-space-name.hf.space/api/v1/hackrx/run \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer your_token_here" \
-  -d '{
-    "documents": "https://example.com/insurance-policy.pdf",
-    "questions": ["What is the coverage amount?"]
-  }'
-```
-
-## Troubleshooting
-
-### Common Issues
-
-1. **Build Fails**: Check the logs in the "Logs" tab for error messages
-2. **Environment Variable Not Set**: Ensure `GOOGLE_API_KEY` is set in Space secrets
-3. **Port Issues**: The application runs on port 7860 (default for Hugging Face Spaces)
-4. **Memory Issues**: If you encounter memory issues, consider optimizing the Dockerfile
-
-### Debugging
-
-1. Check the build logs in the "Logs" tab
-2. Monitor the application logs for runtime errors
-3. Test locally first to ensure everything works
-
-## API Documentation
-
-Once deployed, your API will have the following endpoints:
-
-- `GET /` - Health check
-- `GET /health` - API status
-- `POST /api/v1/hackrx/run` - Process PDF from URL
-- `POST /api/v1/hackrx/local` - Process local PDF file
-
-## Cost Considerations
-
-- Hugging Face Spaces offers free hosting for public spaces
-- Private spaces may have usage limits
-- Consider the cost of Google Gemini API calls
-
-## Security Notes
-
-- Keep your API keys secure
-- Use appropriate authentication for production use
-- Consider rate limiting for public APIs
-
-## Updates
-
-To update your deployment:
-1. Push changes to your Git repository
-2. Hugging Face Spaces will automatically rebuild and deploy
-3. Monitor the build process in the "Logs" tab 

README_HF.md

@@ -1,112 +0,0 @@
-# HackRx Insurance Policy Assistant
-
-A FastAPI application that processes PDF documents and answers questions using AI, deployed on Hugging Face Spaces.
-
-## Features
-
-- PDF document parsing and text extraction
-- Vector-based document search using FAISS
-- AI-powered question answering using Google Gemini
-- RESTful API endpoints for document processing
-
-## API Endpoints
-
-### Health Check
-- `GET /` - Root endpoint
-- `GET /health` - API status check
-
-### Process PDF from URL
-- `POST /api/v1/hackrx/run`
-- **Headers**: `Authorization: Bearer <your_token>`
-- **Body**:
-```json
-{
-  "documents": "https://example.com/document.pdf",
-  "questions": ["What is the coverage amount?", "What are the exclusions?"]
-}
-```
-
-### Process Local PDF File
-- `POST /api/v1/hackrx/local`
-- **Body**:
-```json
-{
-  "document_path": "/app/files/document.pdf",
-  "questions": ["What is the coverage amount?", "What are the exclusions?"]
-}
-```
-
-## Environment Variables
-
-Set these in your Hugging Face Space settings:
-
-- `GOOGLE_API_KEY` - Your Google Gemini API key
-
-## Usage Examples
-
-### Using curl
-
-```bash
-# Health check
-curl https://your-space-name.hf.space/
-
-# Process PDF from URL
-curl -X POST https://your-space-name.hf.space/api/v1/hackrx/run \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer your_token_here" \
-  -d '{
-    "documents": "https://example.com/insurance-policy.pdf",
-    "questions": ["What is the coverage amount?", "What are the exclusions?"]
-  }'
-```
-
-### Using Python
-
-```python
-import requests
-
-# Health check
-response = requests.get("https://your-space-name.hf.space/")
-print(response.json())
-
-# Process PDF
-url = "https://your-space-name.hf.space/api/v1/hackrx/run"
-headers = {
-    "Content-Type": "application/json",
-    "Authorization": "Bearer your_token_here"
-}
-data = {
-    "documents": "https://example.com/insurance-policy.pdf",
-    "questions": ["What is the coverage amount?", "What are the exclusions?"]
-}
-
-response = requests.post(url, headers=headers, json=data)
-print(response.json())
-```
-
-## Local Development
-
-To run the application locally:
-
-```bash
-pip install -r requirements.txt
-python app.py
-```
-
-The API will be available at `http://localhost:7860`
-
-## Deployment
-
-This application is configured for deployment on Hugging Face Spaces using Docker. The following files are included:
-
-- `app.py` - Main application entry point
-- `Dockerfile` - Docker configuration
-- `.dockerignore` - Docker build optimization
-- `requirements.txt` - Python dependencies
-
-## Model Information
-
-- **Framework**: FastAPI
-- **AI Model**: Google Gemini
-- **Vector Database**: FAISS
-- **Document Processing**: PyMuPDF 

embedder.py

@@ -1,48 +1,52 @@
 import faiss
-from sentence_transformers import SentenceTransformer
 import numpy as np
 import os
+from sentence_transformers import SentenceTransformer
 
-# Set up cache directory in a writable location
+# Use a local cache for transformer downloads
 cache_dir = os.path.join(os.getcwd(), ".cache")
 os.makedirs(cache_dir, exist_ok=True)
 os.environ['HF_HOME'] = cache_dir
 os.environ['TRANSFORMERS_CACHE'] = cache_dir
 
-# Initialize model as None - will be loaded lazily
+# Lazy-loaded model
 _model = None
 
-def preload_model():
-    """Preload the sentence transformer model at startup"""
+def preload_model(model_name="all-MiniLM-L6-v2"):
     global _model
-    if _model is None:
-        print("Preloading sentence transformer model...")
-        try:
-            _model = SentenceTransformer("all-MiniLM-L6-v2", cache_folder=cache_dir)
-            print("Model preloading completed")
-        except Exception as e:
-            print(f"Error loading model: {e}")
-            # Fallback to a different model if the first one fails
-            try:
-                _model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", cache_folder=cache_dir)
-                print("Fallback model preloading completed")
-            except Exception as e2:
-                print(f"Error loading fallback model: {e2}")
-                raise
+    if _model is not None:
+        return _model
+
+    print("Preloading sentence transformer model...")
+
+    try:
+        _model = SentenceTransformer(model_name, cache_folder=cache_dir)
+    except Exception as e:
+        print(f"Primary model load failed: {e}")
+        fallback_name = "sentence-transformers/" + model_name
+        print(f"Trying fallback: {fallback_name}")
+        _model = SentenceTransformer(fallback_name, cache_folder=cache_dir)
+
+    print("✅ Model ready.")
     return _model
 
 def get_model():
-    """Get the sentence transformer model, loading it lazily if needed"""
-    global _model
-    if _model is None:
-        print("Warning: Model not preloaded, loading now...")
-        return preload_model()
-    return _model
+    return preload_model()
 
-def build_faiss_index(chunks):
+def build_faiss_index(chunks, batch_size=128, show_progress_bar=False):
     model = get_model()
-    embeddings = model.encode(chunks)
-    dimension = embeddings.shape[1]
-    index = faiss.IndexFlatL2(dimension)
-    index.add(np.array(embeddings))
-    return index, chunks
+
+    # Encode using batching for speed
+    embeddings = model.encode(
+        chunks,
+        batch_size=batch_size,
+        show_progress_bar=show_progress_bar,
+        convert_to_numpy=True,
+        normalize_embeddings=True  # Helps FAISS L2 perform better
+    )
+
+    dim = embeddings.shape[1]
+    index = faiss.IndexFlatL2(dim)
+    index.add(embeddings)
+
+    return index, chunks

main.py

@@ -1,260 +0,0 @@
-import os
-import warnings
-import logging
-import time
-from datetime import datetime
-
-# Suppress TensorFlow warnings
-os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
-os.environ['TF_ENABLE_ONEDNN_OPTS'] = '0'
-os.environ['TF_LOGGING_LEVEL'] = 'ERROR'
-os.environ['TF_ENABLE_DEPRECATION_WARNINGS'] = '0'
-
-# Suppress specific TensorFlow deprecation warnings
-warnings.filterwarnings('ignore', category=DeprecationWarning, module='tensorflow')
-logging.getLogger('tensorflow').setLevel(logging.ERROR)
-
-from fastapi import FastAPI, Request, HTTPException, Depends, Header
-from fastapi.middleware.cors import CORSMiddleware
-from pydantic import BaseModel
-from pdf_parser import parse_pdf_from_url, parse_pdf_from_file
-from embedder import build_faiss_index, preload_model
-from retriever import retrieve_chunks
-from llm import query_gemini
-import uvicorn
-
-app = FastAPI(title="HackRx Insurance Policy Assistant", version="1.0.0")
-
-# Add CORS middleware
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=["*"],
-    allow_credentials=True,
-    allow_methods=["*"],
-    allow_headers=["*"],
-)
-
-# Preload the model at startup
-@app.on_event("startup")
-async def startup_event():
-    print("Starting up HackRx Insurance Policy Assistant...")
-    print("Preloading sentence transformer model...")
-    preload_model()
-    print("Model preloading completed. API is ready to serve requests!")
-
-@app.get("/")
-async def root():
-    return {"message": "HackRx Insurance Policy Assistant API is running!"}
-
-@app.get("/health")
-async def health_check():
-    return {"status": "healthy", "message": "API is ready to process requests"}
-
-class QueryRequest(BaseModel):
-    documents: str
-    questions: list[str]
-
-class LocalQueryRequest(BaseModel):
-    document_path: str
-    questions: list[str]
-
-def verify_token(authorization: str = Header(None)):
-    if not authorization or not authorization.startswith("Bearer "):
-        raise HTTPException(status_code=401, detail="Invalid authorization header")
-    
-    token = authorization.replace("Bearer ", "")
-    # For demo purposes, accept any token. In production, validate against a database
-    if not token:
-        raise HTTPException(status_code=401, detail="Invalid token")
-    
-    return token
-
-@app.post("/api/v1/hackrx/run")
-async def run_query(request: QueryRequest, token: str = Depends(verify_token)):
-    start_time = time.time()
-    timing_data = {}
-    
-    try:
-        print(f"\n=== INPUT JSON ===")
-        print(f"Documents: {request.documents}")
-        print(f"Questions: {request.questions}")
-        print(f"==================\n")
-        
-        print(f"Processing {len(request.questions)} questions...")
-        
-        # Time PDF parsing
-        pdf_start = time.time()
-        text_chunks = parse_pdf_from_url(request.documents)
-        pdf_time = time.time() - pdf_start
-        timing_data['pdf_parsing'] = round(pdf_time, 2)
-        print(f"Extracted {len(text_chunks)} text chunks from PDF")
-        
-        # Time FAISS index building
-        index_start = time.time()
-        index, texts = build_faiss_index(text_chunks)
-        index_time = time.time() - index_start
-        timing_data['faiss_index_building'] = round(index_time, 2)
-        
-        # Time chunk retrieval for all questions
-        retrieval_start = time.time()
-        all_chunks = set()
-        for i, question in enumerate(request.questions):
-            question_start = time.time()
-            top_chunks = retrieve_chunks(index, texts, question)
-            question_time = time.time() - question_start
-            all_chunks.update(top_chunks)
-        
-        retrieval_time = time.time() - retrieval_start
-        timing_data['chunk_retrieval'] = round(retrieval_time, 2)
-        print(f"Retrieved {len(all_chunks)} unique chunks")
-        
-        # Time LLM processing
-        llm_start = time.time()
-        print(f"Processing all {len(request.questions)} questions in batch...")
-        response = query_gemini(request.questions, list(all_chunks))
-        llm_time = time.time() - llm_start
-        timing_data['llm_processing'] = round(llm_time, 2)
-        
-        # Time response processing
-        response_start = time.time()
-        # Extract answers from the JSON response
-        if isinstance(response, dict) and "answers" in response:
-            answers = response["answers"]
-            # Ensure we have the right number of answers
-            while len(answers) < len(request.questions):
-                answers.append("Not Found")
-            answers = answers[:len(request.questions)]
-        else:
-            # Fallback if response is not in expected format
-            answers = [response] if isinstance(response, str) else []
-            # Ensure we have the right number of answers
-            while len(answers) < len(request.questions):
-                answers.append("Not Found")
-            answers = answers[:len(request.questions)]
-        
-        response_time = time.time() - response_start
-        timing_data['response_processing'] = round(response_time, 2)
-        print(f"Generated {len(answers)} answers")
-        
-        # Calculate total time
-        total_time = time.time() - start_time
-        timing_data['total_time'] = round(total_time, 2)
-        
-        print(f"\n=== TIMING BREAKDOWN ===")
-        print(f"PDF Parsing: {timing_data['pdf_parsing']}s")
-        print(f"FAISS Index Building: {timing_data['faiss_index_building']}s")
-        print(f"Chunk Retrieval: {timing_data['chunk_retrieval']}s")
-        print(f"LLM Processing: {timing_data['llm_processing']}s")
-        print(f"Response Processing: {timing_data['response_processing']}s")
-        print(f"TOTAL TIME: {timing_data['total_time']}s")
-        print(f"=======================\n")
-        
-        result = {"answers": answers}
-        print(f"=== OUTPUT JSON ===")
-        print(f"{result}")
-        print(f"==================\n")
-        
-        return result
-        
-    except Exception as e:
-        total_time = time.time() - start_time
-        print(f"Error after {total_time:.2f} seconds: {str(e)}")
-        raise HTTPException(status_code=500, detail=f"Internal server error: {str(e)}")
-
-@app.post("/api/v1/hackrx/local")
-async def run_local_query(request: LocalQueryRequest):
-    start_time = time.time()
-    timing_data = {}
-    
-    try:
-        print(f"\n=== INPUT JSON ===")
-        print(f"Document Path: {request.document_path}")
-        print(f"Questions: {request.questions}")
-        print(f"==================\n")
-        
-        print(f"Processing local document: {request.document_path}")
-        print(f"Processing {len(request.questions)} questions...")
-        
-        # Time local PDF parsing
-        pdf_start = time.time()
-        text_chunks = parse_pdf_from_file(request.document_path)
-        pdf_time = time.time() - pdf_start
-        timing_data['pdf_parsing'] = round(pdf_time, 2)
-        print(f"Extracted {len(text_chunks)} text chunks from local PDF")
-        
-        # Time FAISS index building
-        index_start = time.time()
-        index, texts = build_faiss_index(text_chunks)
-        index_time = time.time() - index_start
-        timing_data['faiss_index_building'] = round(index_time, 2)
-        
-        # Time chunk retrieval for all questions
-        retrieval_start = time.time()
-        all_chunks = set()
-        for i, question in enumerate(request.questions):
-            question_start = time.time()
-            top_chunks = retrieve_chunks(index, texts, question)
-            question_time = time.time() - question_start
-            all_chunks.update(top_chunks)
-        
-        retrieval_time = time.time() - retrieval_start
-        timing_data['chunk_retrieval'] = round(retrieval_time, 2)
-        print(f"Retrieved {len(all_chunks)} unique chunks")
-        
-        # Time LLM processing
-        llm_start = time.time()
-        print(f"Processing all {len(request.questions)} questions in batch...")
-        response = query_gemini(request.questions, list(all_chunks))
-        llm_time = time.time() - llm_start
-        timing_data['llm_processing'] = round(llm_time, 2)
-        
-        # Time response processing
-        response_start = time.time()
-        # Extract answers from the JSON response
-        if isinstance(response, dict) and "answers" in response:
-            answers = response["answers"]
-            # Ensure we have the right number of answers
-            while len(answers) < len(request.questions):
-                answers.append("Not Found")
-            answers = answers[:len(request.questions)]
-        else:
-            # Fallback if response is not in expected format
-            answers = [response] if isinstance(response, str) else []
-            # Ensure we have the right number of answers
-            while len(answers) < len(request.questions):
-                answers.append("Not Found")
-            answers = answers[:len(request.questions)]
-        
-        response_time = time.time() - response_start
-        timing_data['response_processing'] = round(response_time, 2)
-        print(f"Generated {len(answers)} answers")
-        
-        # Calculate total time
-        total_time = time.time() - start_time
-        timing_data['total_time'] = round(total_time, 2)
-        
-        print(f"\n=== TIMING BREAKDOWN ===")
-        print(f"PDF Parsing: {timing_data['pdf_parsing']}s")
-        print(f"FAISS Index Building: {timing_data['faiss_index_building']}s")
-        print(f"Chunk Retrieval: {timing_data['chunk_retrieval']}s")
-        print(f"LLM Processing: {timing_data['llm_processing']}s")
-        print(f"Response Processing: {timing_data['response_processing']}s")
-        print(f"TOTAL TIME: {timing_data['total_time']}s")
-        print(f"=======================\n")
-        
-        result = {"answers": answers}
-        print(f"=== OUTPUT JSON ===")
-        print(f"{result}")
-        print(f"==================\n")
-        
-        return result
-        
-    except Exception as e:
-        total_time = time.time() - start_time
-        print(f"Error after {total_time:.2f} seconds: {str(e)}")
-        raise HTTPException(status_code=500, detail=f"Internal server error: {str(e)}")
-
-if __name__ == "__main__":
-    port = int(os.environ.get("PORT", 10000))
-    uvicorn.run("main:app", host="0.0.0.0", port=port)
-

pdf_parser.py

@@ -2,42 +2,49 @@ import fitz  # PyMuPDF
 import requests
 from io import BytesIO
 from concurrent.futures import ThreadPoolExecutor
-import os
 
-def extract_page_text(page):
+def _extract_text(page):
     text = page.get_text()
-    return text if text.strip() else None
-
-def parse_pdf_from_url_multithreaded(url, max_workers=None):
-    # Automatically detect and use all available CPU cores if max_workers not set
-    if max_workers is None:
-        max_workers = os.cpu_count() or 8
+    return text.strip() if text and text.strip() else None
 
+def parse_pdf_from_url_multithreaded(url, max_workers=2, chunk_size=1):
+    """
+    Download PDF from URL, extract text in parallel, optionally chunk pages.
+    """
     res = requests.get(url)
-    doc = fitz.open(stream=BytesIO(res.content), filetype="pdf")
-    pages = [page for page in doc]
-    chunks = [None] * len(pages)
-    
-    # Process pages in parallel, preserving page order
-    with ThreadPoolExecutor(max_workers=max_workers) as executor:
-        results = list(executor.map(extract_page_text, pages))
-    
-    # Keep only non-empty page results, preserving order
-    doc.close()
-    return [r for r in results if r]
-
-def parse_pdf_from_file_multithreaded(file_path, max_workers=None):
-    if max_workers is None:
-        max_workers = os.cpu_count() or 8
+    with fitz.open(stream=BytesIO(res.content), filetype="pdf") as doc:
+        num_pages = len(doc)
+        pages = list(doc)
+        # Step 1: Parallel text extraction
+        with ThreadPoolExecutor(max_workers=max_workers) as executor:
+            texts = list(executor.map(_extract_text, pages))
+        # Step 2: Optional chunking
+        if chunk_size > 1:
+            chunks = []
+            for i in range(0, len(texts), chunk_size):
+                chunk = ' '.join([t for t in texts[i:i+chunk_size] if t])
+                if chunk:
+                    chunks.append(chunk)
+            return chunks
+        # Default: return one chunk per page
+        return [t for t in texts if t]
 
-    try:
-        doc = fitz.open(file_path)
-        pages = [page for page in doc]
-        chunks = [None] * len(pages)
-        
+def parse_pdf_from_file_multithreaded(file_path, max_workers=2, chunk_size=1):
+    """
+    Parse a local PDF file, extract text in parallel, optionally chunk pages.
+    """
+    with fitz.open(file_path) as doc:
+        num_pages = len(doc)
+        pages = list(doc)
+        # Step 1: Parallel text extraction
         with ThreadPoolExecutor(max_workers=max_workers) as executor:
-            results = list(executor.map(extract_page_text, pages))
-        doc.close()
-        return [r for r in results if r]
-    except Exception as e:
-        raise Exception(f"Error parsing PDF file {file_path}: {str(e)}")
+            texts = list(executor.map(_extract_text, pages))
+        # Step 2: Optional chunking
+        if chunk_size > 1:
+            chunks = []
+            for i in range(0, len(texts), chunk_size):
+                chunk = ' '.join([t for t in texts[i:i+chunk_size] if t])
+                if chunk:
+                    chunks.append(chunk)
+            return chunks
+        return [t for t in texts if t]

start.sh

@@ -1,12 +0,0 @@
-#!/bin/bash
-
-# Set up cache directory
-mkdir -p .cache
-export HF_HOME="$(pwd)/.cache"
-export TRANSFORMERS_CACHE="$(pwd)/.cache"
-
-echo "Cache directory set to: $(pwd)/.cache"
-echo "Starting application..."
-
-# Run the application
-python app.py 

test_deployment.py

@@ -1,75 +0,0 @@
-#!/usr/bin/env python3
-"""
-Test script for Hugging Face Spaces deployment
-"""
-
-import requests
-import json
-import sys
-
-def test_health_check(base_url):
-    """Test the health check endpoint"""
-    try:
-        response = requests.get(f"{base_url}/")
-        print(f"Health check status: {response.status_code}")
-        print(f"Response: {response.json()}")
-        return response.status_code == 200
-    except Exception as e:
-        print(f"Health check failed: {e}")
-        return False
-
-def test_api_endpoint(base_url, api_key):
-    """Test the main API endpoint"""
-    try:
-        url = f"{base_url}/api/v1/hackrx/run"
-        headers = {
-            "Content-Type": "application/json",
-            "Authorization": f"Bearer {api_key}"
-        }
-        data = {
-            "documents": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
-            "questions": ["What is this document about?"]
-        }
-        
-        response = requests.post(url, headers=headers, json=data)
-        print(f"API test status: {response.status_code}")
-        print(f"Response: {response.json()}")
-        return response.status_code == 200
-    except Exception as e:
-        print(f"API test failed: {e}")
-        return False
-
-def main():
-    if len(sys.argv) < 2:
-        print("Usage: python test_deployment.py <base_url> [api_key]")
-        print("Example: python test_deployment.py https://your-space-name.hf.space your_api_key")
-        sys.exit(1)
-    
-    base_url = sys.argv[1].rstrip('/')
-    api_key = sys.argv[2] if len(sys.argv) > 2 else "test_token"
-    
-    print(f"Testing deployment at: {base_url}")
-    print("=" * 50)
-    
-    # Test health check
-    print("1. Testing health check...")
-    health_ok = test_health_check(base_url)
-    
-    # Test API endpoint
-    print("\n2. Testing API endpoint...")
-    api_ok = test_api_endpoint(base_url, api_key)
-    
-    # Summary
-    print("\n" + "=" * 50)
-    print("DEPLOYMENT TEST SUMMARY")
-    print("=" * 50)
-    print(f"Health check: {'✅ PASS' if health_ok else '❌ FAIL'}")
-    print(f"API endpoint: {'✅ PASS' if api_ok else '❌ FAIL'}")
-    
-    if health_ok and api_ok:
-        print("\n🎉 Deployment is working correctly!")
-    else:
-        print("\n⚠️  Some tests failed. Check the logs above for details.")
-
-if __name__ == "__main__":
-    main() 

test_model_loading.py

@@ -1,34 +0,0 @@
-#!/usr/bin/env python3
-"""
-Test script to verify model loading works correctly
-"""
-import os
-import sys
-
-# Set up cache directory
-cache_dir = os.path.join(os.getcwd(), ".cache")
-os.makedirs(cache_dir, exist_ok=True)
-os.environ['HF_HOME'] = cache_dir
-os.environ['TRANSFORMERS_CACHE'] = cache_dir
-
-print(f"Cache directory: {cache_dir}")
-print(f"Current working directory: {os.getcwd()}")
-
-try:
-    from embedder import get_model, build_faiss_index
-    
-    print("Testing model loading...")
-    model = get_model()
-    print("✓ Model loaded successfully!")
-    
-    # Test with some sample text
-    test_chunks = ["This is a test document.", "Another test sentence."]
-    print("Testing FAISS index building...")
-    index, texts = build_faiss_index(test_chunks)
-    print("✓ FAISS index built successfully!")
-    
-    print("All tests passed!")
-    
-except Exception as e:
-    print(f"✗ Error: {e}")
-    sys.exit(1)