arXiv RAG v2

AI/ML Research Paper Search & Chat

arXiv RAG v2: Hybrid Retrieval Benchmark

A comprehensive evaluation of 10 retrieval configurations across 9,544 synthetic benchmark queries. This study explores the performance trade-offs between dense embeddings (BGE-M3, OpenAI 3-large), sparse retrieval, hybrid fusion strategies, and cross-encoder reranking for AI/ML research paper search.

2,500

Research Papers

Benchmark Queries

Best NDCG@10

Avg Reranker Gain

📊 Data Collection & Benchmark Design

Data Collection Pipeline

arXiv API Collection - 14 months (2025.01 ~ 2026.02), 7 categories (cs.CL, cs.AI, cs.LG, cs.CV, stat.ML, cs.IR, cs.NE)

NG Keyword Filtering - 1,756 keywords across 9 categories (biomedical, chemistry, robotics, etc.)

Gemini Classification - SUITABLE/NOT SUITABLE via gemini-3-flash-preview

Semantic Filtering - Cosine similarity ≥ 0.55 with anchor queries

Multi-Score Ranking - Citation + Recency + Semantic + Stratified sampling → 2,500 papers

Benchmark Query Generation

4 Query Styles × ~2,400 papers = ~9,600 queries

KEYWORD

4-7 technical terms

NATURAL_SHORT

6-12 word questions

NATURAL_LONG

15-25 word research Qs

CONCEPTUAL

Paraphrased, no acronyms

Hard Negative Mining: BGE-M3 embedding similarity search to identify "similar but different" papers for robust evaluation.

🔧 Model Configurations

Dense (BGE-M3)

1024d vectors

Sparse (BGE-M3)

Lexical weights

Hybrid (BGE-M3)

RRF: dense + sparse

OpenAI 3-large

3072d vectors

Hybrid-3L

3-large + sparse

+Rerank Variants: Each base model tested with BGE-reranker-v2-m3 cross-encoder (Top-20 → Rerank → Top-10)

📈 Global Performance Overview

5 Metrics × 5 Models (Base + Reranker Gain/Loss)

Dense Sparse Hybrid 3-large Hybrid-3L +Reranker Gain +Reranker Loss

🔥 QueryType × Difficulty Heatmap

Loading heatmap data...

📊 Model Comparison Delta (MRR)

Baseline:

Delta values show MRR difference compared to baseline model (green = better, red = worse)

Loading delta data...

⚡ Reranker Impact Analysis

Avg NDCG@10 Gain

Max Gain (conceptual)

Avg Latency Overhead

⚖️ Model Comparison Tool

Model A:

Model B:

Select models to compare

⏱️ Search Latency

Detailed Results

Model	MRR	NDCG@5	NDCG@10	P@5	P@10	Latency

💡 Insights & Production Recommendations

Key Findings

Loading analysis...

Production Recommendations

🏆 Best Quality

⚡ Best Speed

🎯 Best Balance

💰 Best ROI

Chat with Research Papers

AI/ML 연구 논문에 대해 질문하세요

안녕하세요! AI/ML 연구 논문에 대해 질문해 주세요. 다음과 같은 주제에 대해 답변드릴 수 있습니다:

Research Papers Dataset

#	Title	Date	arXiv
Loading papers...