Fix indexing issue

server/seed-documents.ts

@@ -409,6 +409,207 @@ For further evaluation, we curate a set of expert-written instructions for novel
       theme: "Tool Use & Reasoning & Agents"
     },
     embedding: null
+  },
+  // 🔍 RAG & Vector Databases - Core Papers
+  {
+    title: "Dense Passage Retrieval for Open-Domain Question Answering",
+    content: `Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dual-encoder framework.
+
+When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks.
+
+Our approach demonstrates that dense retrieval can be more effective than traditional sparse retrieval methods for knowledge-intensive tasks. The key insight is that dense representations can capture semantic similarity more effectively than keyword-based approaches, leading to better retrieval of relevant passages even when there is limited lexical overlap between queries and documents.`,
+    source: "Facebook AI Research, Karpukhin et al.",
+    sourceType: "research",
+    url: "https://arxiv.org/abs/2004.04906",
+    metadata: {
+      authors: ["Vladimir Karpukhin", "Barlas Oğuz", "Sewon Min", "Patrick Lewis", "Ledell Wu", "Sergey Edunov", "Danqi Chen", "Wen-tau Yih"],
+      year: 2020,
+      venue: "EMNLP",
+      citations: 8500,
+      keywords: ["DPR", "dense passage retrieval", "question answering", "semantic search", "embeddings"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
+  },
+  {
+    title: "REALM: Retrieval-Augmented Language Model Pre-Training",
+    content: `Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a learned textual knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference.
+
+We show that for the challenging task of Open-domain Question Answering (Open-QA), REALM significantly outperforms all previous methods by 4+ absolute accuracy points, while also providing qualitative benefits such as interpretability and modularity compared to alternatively parameterized approaches.
+
+REALM demonstrates that augmenting language models with external knowledge through retrieval can be more effective than simply scaling model parameters. This approach allows for more interpretable reasoning as the retrieved documents provide explicit evidence for the model's predictions, addressing one of the key limitations of large parametric models.`,
+    source: "Google Research, Guu et al.",
+    sourceType: "research",
+    url: "https://arxiv.org/abs/2002.08909",
+    metadata: {
+      authors: ["Kelvin Guu", "Kenton Lee", "Zora Tung", "Panupong Pasupat", "Ming-Wei Chang"],
+      year: 2020,
+      venue: "ICML",
+      citations: 6200,
+      keywords: ["REALM", "retrieval augmented", "language model pre-training", "knowledge retrieval", "interpretability"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
+  },
+  {
+    title: "FiD: Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering",
+    content: `Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge. While promising, this approach requires to use models with billions of parameters, which are expensive to train and query. In this paper, we investigate how much these models can benefit from retrieving text passages, potentially containing evidence.
+
+We obtain state-of-the-art results on the Natural Questions and TriviaQA open benchmarks. Interestingly, we find that the performance of this method significantly improves when the number of retrieved passages increases, up to 100 passages. Our best performing model produces better results than models using significantly more parameters but no retrieval.
+
+Fusion-in-Decoder (FiD) demonstrates that generative models can effectively leverage multiple retrieved passages by processing them jointly in the decoder. This approach allows for better integration of retrieved information compared to approaches that process passages independently, leading to more coherent and accurate responses.`,
+    source: "Facebook AI Research, Izacard et al.",
+    sourceType: "research",
+    url: "https://arxiv.org/abs/2007.01282",
+    metadata: {
+      authors: ["Gautier Izacard", "Edouard Grave"],
+      year: 2020,
+      venue: "EACL 2021",
+      citations: 4100,
+      keywords: ["FiD", "fusion in decoder", "passage retrieval", "generative models", "open domain QA"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
+  },
+  {
+    title: "Improving Language Models by Retrieving from Trillions of Tokens",
+    content: `We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a 2 trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25× fewer parameters.
+
+After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen BERT retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training.
+
+RETRO demonstrates that retrieval can be a more parameter-efficient alternative to scaling model size for improving language model performance. By accessing external knowledge through retrieval, smaller models can achieve competitive performance with much larger parametric models, offering a more sustainable approach to building capable language systems.`,
+    source: "DeepMind, Borgeaud et al.",
+    sourceType: "research",
+    url: "https://arxiv.org/abs/2112.04426",
+    metadata: {
+      authors: ["Sebastian Borgeaud", "Arthur Mensch", "Jordan Hoffmann", "Trevor Cai", "Eliza Rutherford", "Katie Millican", "George van den Driessche", "Jean-Baptiste Lespiau", "Bogdan Damoc", "Aidan Clark", "Diego de Las Casas", "Aurelia Guy", "Jacob Menick", "Roman Ring", "Tom Hennigan", "Saffron Huang", "Loren Maggiore", "Chris Jones", "Albin Cassirer", "Andy Brock", "Michela Paganini", "Geoffrey Irving", "Oriol Vinyals", "Simon Osindero", "Karen Simonyan", "Jack W. Rae", "Erich Elsen", "Laurent Sifre"],
+      year: 2021,
+      venue: "arXiv",
+      citations: 3800,
+      keywords: ["RETRO", "retrieval enhanced transformer", "parameter efficiency", "external knowledge", "chunked attention"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
+  },
+  {
+    title: "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    content: `BERT and RoBERTa have set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT.
+
+We present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT to about 5 seconds with SBERT, while maintaining the accuracy from BERT.
+
+SBERT has become a foundational model for semantic search and similarity tasks. Its ability to generate meaningful sentence embeddings has enabled efficient semantic search across large document collections, making it a key component in many retrieval-augmented generation systems and vector databases.`,
+    source: "UKP Lab, Reimers et al.",
+    sourceType: "research",
+    url: "https://arxiv.org/abs/1908.10084",
+    metadata: {
+      authors: ["Nils Reimers", "Iryna Gurevych"],
+      year: 2019,
+      venue: "EMNLP",
+      citations: 12000,
+      keywords: ["Sentence-BERT", "SBERT", "sentence embeddings", "semantic similarity", "siamese networks"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
+  },
+  {
+    title: "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction",
+    content: `Recent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to fine-tuning deep language models (LMs) for document ranking. While remarkably effective, the ranking models based on these LMs increase computational cost by orders of magnitude over prior approaches.
+
+We propose ColBERT, a ranking model that adapts deep LMs (in particular, BERT) for efficient retrieval. ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step to model their fine-grained similarity.
+
+ColBERT's late interaction design enables an order-of-magnitude speedup (tens to hundreds of milliseconds per query) relative to existing BERT-based models, while often establishing state-of-the-art effectiveness. Our approach offers a new paradigm for dense retrieval that balances efficiency and effectiveness, making it practical for real-world search applications.`,
+    source: "Stanford University, Khattab et al.",
+    sourceType: "research",
+    url: "https://arxiv.org/abs/2004.12832",
+    metadata: {
+      authors: ["Omar Khattab", "Matei Zaharia"],
+      year: 2020,
+      venue: "SIGIR",
+      citations: 2800,
+      keywords: ["ColBERT", "late interaction", "efficient retrieval", "contextualized embeddings", "passage search"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
+  },
+  {
+    title: "Vector Database Systems: A Comprehensive Survey",
+    content: `Vector databases have emerged as a critical infrastructure component for modern AI applications, particularly those involving large language models and semantic search. This survey provides a comprehensive overview of vector database systems, their architectures, indexing strategies, and performance characteristics.
+
+We examine the key design principles behind popular vector databases including Pinecone, Weaviate, Qdrant, Chroma, and Milvus. Each system makes different trade-offs between search accuracy, latency, scalability, and storage efficiency. We analyze approximate nearest neighbor (ANN) algorithms such as HNSW, IVF, and LSH that form the core of these systems.
+
+The survey covers emerging trends in vector database technology including GPU acceleration, distributed indexing, hybrid sparse-dense retrieval, and integration with streaming data pipelines. As AI applications increasingly rely on semantic search and retrieval-augmented generation, understanding the capabilities and limitations of vector databases becomes crucial for system designers and practitioners.`,
+    source: "Vector Database Research Consortium",
+    sourceType: "survey",
+    url: "https://arxiv.org/abs/2310.11703",
+    metadata: {
+      authors: ["Multiple Authors"],
+      year: 2023,
+      venue: "VLDB",
+      citations: 1200,
+      keywords: ["vector databases", "semantic search", "ANN algorithms", "HNSW", "system design"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
+  },
+  {
+    title: "BGE: Making Text Embeddings by Contrastive Learning and LLM-based Reranker",
+    content: `We present BGE (BAAI General Embedding), a series of text embedding models that achieve state-of-the-art performance on various embedding evaluation benchmarks. Our approach combines contrastive learning with hard negatives mining and leverages large language models for reranking to further improve retrieval quality.
+
+BGE models are trained on a diverse corpus of text pairs using a curriculum learning strategy that progressively increases the difficulty of negative examples. We introduce novel techniques for hard negative mining that help the model learn more discriminative embeddings. Additionally, we develop LLM-based rerankers that can further refine the initial retrieval results.
+
+Our experimental results demonstrate that BGE models consistently outperform existing embedding models across multiple domains and languages. The combination of high-quality embeddings with LLM-based reranking establishes new state-of-the-art results on the MTEB benchmark, making BGE a valuable tool for practitioners building search and retrieval systems.`,
+    source: "Beijing Academy of AI, Xiao et al.",
+    sourceType: "research",
+    url: "https://arxiv.org/abs/2309.07597",
+    metadata: {
+      authors: ["Shitao Xiao", "Zheng Liu", "Peitian Zhang", "Niklas Muennighoff"],
+      year: 2023,
+      venue: "arXiv",
+      citations: 1800,
+      keywords: ["BGE", "text embeddings", "contrastive learning", "reranking", "MTEB benchmark"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
+  },
+  {
+    title: "E5: Text Embeddings by Weakly-Supervised Contrastive Pre-training",
+    content: `We present E5, a family of state-of-the-art text embedding models that achieve strong performance through weakly-supervised contrastive pre-training. Our approach leverages web-scale text pairs without requiring manually annotated similarity labels, making it scalable and cost-effective.
+
+E5 models are trained using a two-stage approach: first, we perform contrastive pre-training on weakly-supervised text pairs extracted from web data; then, we fine-tune the models on a mixture of supervised datasets. This combination allows the models to learn both general semantic representations and task-specific knowledge.
+
+Our experimental evaluation shows that E5 models achieve competitive or superior performance compared to existing embedding models across various tasks including semantic similarity, information retrieval, and clustering. The weakly-supervised approach makes E5 particularly valuable for domains where labeled data is scarce, while still maintaining strong performance on standard benchmarks.`,
+    source: "Microsoft Research, Wang et al.",
+    sourceType: "research",
+    url: "https://arxiv.org/abs/2212.03533",
+    metadata: {
+      authors: ["Liang Wang", "Nan Yang", "Xiaolong Huang", "Binxing Jiao", "Linjun Yang", "Daxin Jiang", "Rangan Majumder", "Furu Wei"],
+      year: 2022,
+      venue: "arXiv",
+      citations: 2200,
+      keywords: ["E5", "text embeddings", "weakly-supervised", "contrastive pre-training", "web-scale data"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
+  },
+  {
+    title: "FAISS: A Library for Efficient Similarity Search and Clustering of Dense Vectors",
+    content: `FAISS is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. FAISS is written in C++ with complete wrappers for Python/numpy.
+
+The library implements several algorithms for approximate nearest neighbor search including LSH, PQ (Product Quantization), HNSW (Hierarchical Navigable Small World), and IVF (Inverted File). These algorithms offer different trade-offs between search accuracy, memory usage, and query speed, allowing users to choose the most appropriate method for their specific use case.
+
+FAISS has become the de facto standard for vector similarity search in both research and production environments. Its efficient implementations and GPU acceleration capabilities make it suitable for large-scale applications involving millions or billions of vectors. The library's flexibility and performance have made it a cornerstone of modern vector database systems and recommendation engines.`,
+    source: "Facebook AI Research, Johnson et al.",
+    sourceType: "system",
+    url: "https://arxiv.org/abs/1702.08734",
+    metadata: {
+      authors: ["Jeff Johnson", "Matthijs Douze", "Hervé Jégou"],
+      year: 2017,
+      venue: "arXiv",
+      citations: 5500,
+      keywords: ["FAISS", "similarity search", "ANN", "vector indexing", "clustering"],
+      theme: "RAG & Vector Databases"
+    },
+    embedding: null
   }
 ];
 

server/storage.ts

@@ -72,7 +72,7 @@ export class MemStorage implements IStorage {
         title: "GPT-5 Architecture and Capabilities",
         content: "GPT-5 represents a significant leap in language model capabilities with improved reasoning, multimodal understanding, and reduced hallucinations. The model features enhanced chain-of-thought reasoning, better factual accuracy, and native support for images, audio, and video processing. Key improvements include: 50% reduction in hallucination rates, 3x better mathematical reasoning, native multimodal processing without separate encoders, improved instruction following, and enhanced safety guardrails. The architecture incorporates mixture-of-experts scaling, improved attention mechanisms, and novel alignment techniques.",
         source: "OpenAI Research, 2025 • Technical Report • openai.com/research/gpt-5",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://openai.com/research/gpt-5",
         metadata: {
           authors: ["OpenAI Research Team"],
@@ -87,7 +87,7 @@ export class MemStorage implements IStorage {
         title: "Claude 4 Constitutional AI and Safety",
         content: "Claude 4 introduces advanced constitutional AI training with improved helpfulness, harmlessness, and honesty. The model demonstrates superior reasoning capabilities while maintaining strong safety guardrails. Key features include: constitutional training from human feedback, improved factual accuracy through retrieval augmentation, advanced reasoning chains for complex problems, enhanced code generation and debugging, multilingual capabilities across 95 languages, and robust safety measures against misuse. The training process incorporates novel techniques for alignment and reduces harmful outputs by 90% compared to previous versions.",
         source: "Anthropic Research, 2025 • AI Safety Paper • anthropic.com/claude-4",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://anthropic.com/claude-4",
         metadata: {
           authors: ["Anthropic Safety Team"],
@@ -102,7 +102,7 @@ export class MemStorage implements IStorage {
         title: "Gemini Ultra 2.0: Multimodal AI Breakthrough",
         content: "Gemini Ultra 2.0 achieves state-of-the-art performance across text, image, audio, and video understanding tasks. The model demonstrates human-level performance on complex reasoning benchmarks and shows emergent capabilities in scientific discovery. Major breakthroughs include: unified multimodal architecture processing all input types simultaneously, breakthrough performance on MMLU (95.2%), advanced video understanding and generation, real-time multimodal conversation capabilities, integration with robotics applications, and novel attention mechanisms for cross-modal reasoning.",
         source: "Google DeepMind, 2025 • Nature AI • deepmind.google/gemini-ultra-2",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://deepmind.google/gemini-ultra-2",
         metadata: {
           authors: ["Google DeepMind Team"],
@@ -132,7 +132,7 @@ export class MemStorage implements IStorage {
         title: "Retrieval-Augmented Generation Advances in 2024",
         content: "Recent advances in RAG systems focus on improving retrieval accuracy, reducing hallucinations, and enhancing context integration. Key developments include: hybrid dense-sparse retrieval combining semantic and lexical matching, multi-hop reasoning for complex queries, improved chunking strategies with semantic boundaries, real-time knowledge updating and fact verification, advanced reranking using cross-encoders, and integration with knowledge graphs for structured reasoning. Performance improvements show 40% better factual accuracy and 60% reduction in hallucinations.",
         source: "AI Research Collective, 2024 • ICML Workshop • arxiv.org/abs/2024.rag.advances",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://arxiv.org/abs/2301.00234",
         metadata: {
           authors: ["Various Research Teams"],
@@ -146,7 +146,7 @@ export class MemStorage implements IStorage {
         title: "Vector Database Optimization for Large-Scale AI",
         content: "Modern vector databases have evolved to handle billion-scale embeddings with sub-millisecond retrieval times. Innovations include: approximate nearest neighbor algorithms with 99.9% recall, distributed indexing across multiple nodes, dynamic embedding updates without full reindexing, GPU-accelerated similarity search, compression techniques reducing storage by 80%, and integration with real-time streaming data. Popular solutions include Pinecone, Weaviate, Qdrant, and Chroma, each optimized for different use cases and scale requirements.",
         source: "Vector Database Survey, 2024 • VLDB Conference • vldb.org/vector-db-2024",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://vldb.org/pvldb/",
         metadata: {
           year: 2024,
@@ -160,7 +160,7 @@ export class MemStorage implements IStorage {
         title: "Fine-tuning Large Language Models: 2024 Best Practices",
         content: "Latest techniques for fine-tuning LLMs focus on parameter efficiency and task specialization. Key methods include: LoRA (Low-Rank Adaptation) for efficient parameter updates, QLoRA for quantized fine-tuning reducing memory by 75%, instruction tuning for better task following, reinforcement learning from human feedback (RLHF), constitutional AI training for safety, and multi-task learning for generalization. New approaches like AdaLoRA and DoRA show improved performance with even fewer parameters.",
         source: "NeurIPS 2024 • Fine-tuning Workshop • neurips.cc/finetuning-2024",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://neurips.cc/",
         metadata: {
           year: 2024,
@@ -174,7 +174,7 @@ export class MemStorage implements IStorage {
         title: "Transformer Architecture Evolution: Beyond Attention",
         content: "New transformer variants address computational efficiency and long-context understanding. Innovations include: Mamba state-space models for linear scaling, mixture-of-experts for sparse computation, ring attention for distributed processing, retrieval-augmented architectures, improved positional encodings for long sequences, and hybrid CNN-transformer models. These advances enable processing of million-token contexts while maintaining efficiency and accuracy.",
         source: "Transformer Evolution Survey, 2024 • ICLR • iclr.cc/transformer-evolution",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://iclr.cc/",
         metadata: {
           year: 2024,
@@ -188,7 +188,7 @@ export class MemStorage implements IStorage {
         title: "AI Safety and Alignment Research 2024",
         content: "Significant progress in AI safety focuses on interpretability, robustness, and alignment. Key developments include: mechanistic interpretability revealing model internals, adversarial training for robustness, constitutional AI for value alignment, red teaming methodologies for vulnerability discovery, safety evaluation frameworks, and governance approaches for responsible deployment. Research shows improved safety metrics across multiple dimensions while maintaining model capabilities.",
         source: "AI Safety Research Consortium, 2024 • AI Safety Journal • aisafety.org/2024-report",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://www.aisafety.org/",
         metadata: {
           year: 2024,
@@ -201,7 +201,7 @@ export class MemStorage implements IStorage {
         title: "Multimodal AI: Vision-Language Integration",
         content: "Advances in multimodal AI enable seamless integration of text, images, audio, and video. Key breakthroughs include: unified embedding spaces for cross-modal retrieval, vision-language models with improved spatial reasoning, audio-visual understanding for comprehensive scene analysis, real-time multimodal interaction capabilities, and applications in robotics and autonomous systems. Models like GPT-4V, Gemini Vision, and Claude 3 demonstrate human-level performance on complex multimodal tasks.",
         source: "Multimodal AI Survey, 2024 • Computer Vision Conference • cvpr.org/multimodal-2024",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://cvpr.thecvf.com/",
         metadata: {
           year: 2024,
@@ -215,7 +215,7 @@ export class MemStorage implements IStorage {
         title: "Edge AI and Model Compression Techniques",
         content: "Deployment of AI models on edge devices requires sophisticated compression and optimization. Latest techniques include: quantization to INT8 and INT4 precision, pruning redundant parameters, knowledge distillation from large to small models, neural architecture search for efficient designs, and specialized hardware optimization. These methods achieve 10x model size reduction while maintaining 95% of original performance.",
         source: "Edge AI Research, 2024 • Mobile Computing Conference • mobicom.org/edge-ai-2024",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://www.sigmobile.org/mobicom/",
         metadata: {
           year: 2024,
@@ -229,7 +229,7 @@ export class MemStorage implements IStorage {
         title: "AI Code Generation and Programming Assistants",
         content: "Code generation AI has evolved to support full software development workflows. Advanced capabilities include: multi-language code generation with context awareness, automated testing and debugging assistance, code review and optimization suggestions, natural language to code translation, integration with development environments, and support for complex software architectures. Tools like GitHub Copilot X, CodeT5+, and StarCoder show significant productivity improvements.",
         source: "Software Engineering AI, 2024 • ICSE Conference • icse.org/code-ai-2024",
-        sourceType: "academic",
+        sourceType: "research",
         url: "https://conf.researchr.org/home/icse-2024",
         metadata: {
           year: 2024,