hybrid-search-implementation 技能详解面向 LLM 应用与 RAG 的向量 关键词混合检索实战模板【免费下载链接】agentsMulti-harness agentic plugin marketplace for Claude Code, Codex, Cursor, OpenCode, GitHub Copilot, and Google Antigravity项目地址: https://gitcode.com/GitHub_Trending/agents24/agents本文是 agents 仓库中llm-application-dev插件下hybrid-search-implementation技能的深度技术指南。该技能解决纯向量检索与纯关键词检索各自的短板语义召回 vs 精确匹配提供 Reciprocal Rank FusionRRF、线性融合、pgvector 全文检索、Elasticsearch kNN/BM25 以及端到端 Hybrid RAG 流水线四套可直接落地的 Python 模板。读完本文你将掌握混合检索的架构选择、四种融合方法的适用场景与完整可运行代码并能结合仓库内rag-implementation、vector-index-tuning等相邻技能搭建生产级检索系统。技能定位为什么需要混合检索在 agents 仓库中hybrid-search-implementation是 llm-application-dev 插件 八大技能之一其 SKILL.md 明确定义了适用时机见 SKILL.md构建对召回率recall要求更高的 RAG 系统需要同时兼顾语义理解与精确匹配如人名、产品代码、型号等专有名词处理领域专属词汇或纯向量检索遗漏关键词匹配的场景。从仓库的 Agent 编排看hybrid-search-implementation与 vector-database-engineer 的能力清单高度对应Vector BM25 keyword search fusion / Reciprocal Rank Fusion (RRF) scoring / Reranking with cross-encoders同时与 rag-implementation 中的 Hybrid Search: Combine dense sparse with weighted fusion 策略互为表里。其完整模板库位于 references/details.md本文即以其为主体展开。混合检索架构SKILL.md 给出的典型架构如下Query → ┬─► Vector Search ──► Candidates ─┐ │ │ └─► Keyword Search ─► Candidates ─┴─► Fusion ─► Results核心思想是同一查询并行走两条检索通道——向量检索语义相似度与关键词检索BM25/全文检索各取前 N 个候选再通过融合策略合并为最终排序。四种融合方法对比方法说明最佳场景RRFReciprocal Rank Fusion基于排名的倒数融合通用场景无需调参Linear分数加权求和线性插值可调平衡需按数据调权重Cross-encoder用神经模型重排追求最高质量Cascade先过滤再重排追求效率下文四个模板分别覆盖了 RRF/Linear模板一、数据库内融合与 Cross-encoder 重排模板二、Elasticsearch 原生融合模板三以及完整流水线模板四。模板一纯 Python 实现 RRF 与线性融合details.md 的第一个模板不依赖任何外部存储给出了两个可独立复用的函数reciprocal_rank_fusion与linear_combination。Reciprocal Rank Fusionfrom typing import List, Dict, Tuple from collections import defaultdict def reciprocal_rank_fusion( result_lists: List[List[Tuple[str, float]]], k: int 60, weights: List[float] None ) - List[Tuple[str, float]]: Combine multiple ranked lists using RRF. Args: result_lists: List of (doc_id, score) tuples per search method k: RRF constant (higher more weight to lower ranks) weights: Optional weights per result list Returns: Fused ranking as (doc_id, score) tuples if weights is None: weights [1.0] * len(result_lists) scores defaultdict(float) for result_list, weight in zip(result_lists, weights): for rank, (doc_id, _) in enumerate(result_list): # RRF formula: 1 / (k rank) scores[doc_id] weight * (1.0 / (k rank 1)) # Sort by fused score return sorted(scores.items(), keylambda x: x[1], reverseTrue)关键参数与原理k默认 60RRF 常数。它决定了低排名文档获得分数的衰减速度——k越大排名靠后的文档权重越接近排名靠前的文档k越小排序靠前的文档优势越明显。经典论文与实践中k60是稳妥默认值这也是 Elasticsearch 官方rank_constant的默认值见模板三。weights每个结果列表的可选权重。当不同检索通道可信度不同时例如向量通道质量更高可传入[1.5, 1.0]之类权重默认等权[1.0, ...]。公式score(doc) Σ weight_i * 1 / (k rank_i 1)。RRF 只依赖排名而非原始分数因此天然免疫向量分数与 BM25 分数量纲不同、无法直接相加的问题这是它在混合检索中被广泛使用的最重要原因。线性组合Linear Combinationdef linear_combination( vector_results: List[Tuple[str, float]], keyword_results: List[Tuple[str, float]], alpha: float 0.5 ) - List[Tuple[str, float]]: Combine results with linear interpolation. Args: vector_results: (doc_id, similarity_score) from vector search keyword_results: (doc_id, bm25_score) from keyword search alpha: Weight for vector search (1-alpha for keyword) # Normalize scores to [0, 1] def normalize(results): if not results: return {} scores [s for _, s in results] min_s, max_s min(scores), max(scores) range_s max_s - min_s if max_s ! min_s else 1 return {doc_id: (score - min_s) / range_s for doc_id, score in results} vector_scores normalize(vector_results) keyword_scores normalize(keyword_results) # Combine all_docs set(vector_scores.keys()) | set(keyword_scores.keys()) combined {} for doc_id in all_docs: v_score vector_scores.get(doc_id, 0) k_score keyword_scores.get(doc_id, 0) combined[doc_id] alpha * v_score (1 - alpha) * k_score return sorted(combined.items(), keylambda x: x[1], reverseTrue)与 RRF 的关键差异需要先归一化线性融合直接对分数加权求和但向量相似度如余弦相似度约在 [-1, 1]与 BM25 分数无上界量纲完全不同因此模板先用 min-max 归一化把各通道分数压到 [0, 1] 区间。注意range_s的保护性处理——当max_s min_s时置为 1避免除零。alpha默认 0.5向量通道权重关键词通道权重为1 - alpha。调大alpha偏向语义检索调小偏向精确匹配。SKILL.md 的最佳实践特别提醒不同查询需要不同权重Different queries need different weights这正是指alpha应根据数据与查询分布经验性调优并通过 A/B 测试验证。模板二PostgreSQL pgvector 数据库内混合检索第二个模板展示了向量索引 全文索引在单数据库内完成的方案PostgresHybridSearch类基于asyncpg连接池将 HNSW 向量索引与 GIN 全文索引放在同一张documents表上用一条 SQL 完成混合检索。建表与索引setup_schemaimport asyncpg from typing import List, Dict, Optional import numpy as np class PostgresHybridSearch: Hybrid search with pgvector and full-text search. def __init__(self, pool: asyncpg.Pool): self.pool pool async def setup_schema(self): Create tables and indexes. async with self.pool.acquire() as conn: await conn.execute( CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE IF NOT EXISTS documents ( id TEXT PRIMARY KEY, content TEXT NOT NULL, embedding vector(1536), metadata JSONB DEFAULT {}, ts_content tsvector GENERATED ALWAYS AS ( to_tsvector(english, content) ) STORED ); -- Vector index (HNSW) CREATE INDEX IF NOT EXISTS documents_embedding_idx ON documents USING hnsw (embedding vector_cosine_ops); -- Full-text index (GIN) CREATE INDEX IF NOT EXISTS documents_fts_idx ON documents USING gin (ts_content); )值得注意的实现细节vector(1536)维度与 OpenAItext-embedding-3-small的输出维度一致该模型维度信息可见 rag-implementation/SKILL.md 的模型表若改用 Voyage AI 的voyage-3-large1024 维或text-embedding-3-large3072 维需同步调整此处维度声明。ts_content生成列GENERATED ALWAYS AS (to_tsvector(english, content)) STORED让全文索引随content自动维护写入时无需手动同步这是 PostgreSQL 12 的生成列特性。HNSW 索引hnsw (embedding vector_cosine_ops)适合中等规模数据、召回率约 95-99%与 similarity-search-patterns/SKILL.md 中HNSW for most cases的建议一致超大规模场景可参考 vector-index-tuning 切换 IVF/PQ 等索引策略。GIN 索引加速ts_content to_tsquery(...)全文匹配。hybrid_searchCTE FULL OUTER JOIN RRFasync def hybrid_search( self, query: str, query_embedding: List[float], limit: int 10, vector_weight: float 0.5, filter_metadata: Optional[Dict] None ) - List[Dict]: Perform hybrid search combining vector and full-text. Uses RRF fusion for combining results. async with self.pool.acquire() as conn: # Build filter clause where_clause 11 params [query_embedding, query, limit * 3] if filter_metadata: for key, value in filter_metadata.items(): params.append(value) where_clause f AND metadata-{key} ${len(params)} results await conn.fetch(f WITH vector_search AS ( SELECT id, content, metadata, ROW_NUMBER() OVER (ORDER BY embedding $1::vector) as vector_rank, 1 - (embedding $1::vector) as vector_score FROM documents WHERE {where_clause} ORDER BY embedding $1::vector LIMIT $3 ), keyword_search AS ( SELECT id, content, metadata, ROW_NUMBER() OVER (ORDER BY ts_rank(ts_content, websearch_to_tsquery(english, $2)) DESC) as keyword_rank, ts_rank(ts_content, websearch_to_tsquery(english, $2)) as keyword_score FROM documents WHERE ts_content websearch_to_tsquery(english, $2) AND {where_clause} ORDER BY ts_rank(ts_content, websearch_to_tsquery(english, $2)) DESC LIMIT $3 ) SELECT COALESCE(v.id, k.id) as id, COALESCE(v.content, k.content) as content, COALESCE(v.metadata, k.metadata) as metadata, v.vector_score, k.keyword_score, -- RRF fusion COALESCE(1.0 / (60 v.vector_rank), 0) * $4::float COALESCE(1.0 / (60 k.keyword_rank), 0) * (1 - $4::float) as rrf_score FROM vector_search v FULL OUTER JOIN keyword_search k ON v.id k.id ORDER BY rrf_score DESC LIMIT $3 / 3 , *params, vector_weight) return [dict(row) for row in results]SQL 层面的工程要点参数化安全filter_metadata的过滤条件通过$N占位符拼接值走参数绑定避免 SQL 注入where_clause默认11保证无过滤时语法合法。过采样每条通道LIMIT $3即limit * 3取候选最终LIMIT $3 / 3收敛回limit为融合与后续重排留出余量。websearch_to_tsquery比to_tsquery更宽容支持类搜索引擎的语法引号、OR/AND、-排除适合直接解析用户查询。算子pgvector 的余弦距离算子1 - distance得到相似度分数。RRF 融合vector_weight控制向量通道权重关键词通道为1 - vector_weight与模板一公式完全一致且FULL OUTER JOIN保证只在单通道命中的文档也能进入结果集。search_with_rerankCross-encoder 二阶段重排async def search_with_rerank( self, query: str, query_embedding: List[float], limit: int 10, rerank_candidates: int 50 ) - List[Dict]: Hybrid search with cross-encoder reranking. from sentence_transformers import CrossEncoder # Get candidates candidates await self.hybrid_search( query, query_embedding, limitrerank_candidates ) if not candidates: return [] # Rerank with cross-encoder model CrossEncoder(cross-encoder/ms-marco-MiniLM-L-6-v2) pairs [(query, c[content]) for c in candidates] scores model.predict(pairs) for candidate, score in zip(candidates, scores): candidate[rerank_score] float(score) # Sort by rerank score and return top results reranked sorted(candidates, keylambda x: x[rerank_score], reverseTrue) return reranked[:limit]该方法的模式是典型的召回 → 精排两级架构召回阶段混合检索放宽limit如取 50 个候选保证召回精排阶段用cross-encoder/ms-marco-MiniLM-L-6-v2对 (query, doc) 逐对打分——Cross-encoder 让 query 与文档在同一个 Transformer 中交互建模相关性判断远优于双塔式 embedding 余弦相似度但计算成本高因此只对少量候选执行。这与 rag-implementation/SKILL.md 中列出的 Cross-Encoders: BERT-based reranking (ms-marco-MiniLM) 方法一一对应。模板三Elasticsearch 混合检索原生能力 RRF第三个模板基于官方elasticsearchPython 客户端提供三种检索能力索引创建、bool查询内融合向量与 BM25脚本打分、以及 Elasticsearch 8.x 原生的sub_searches rank.rrf。索引映射from elasticsearch import Elasticsearch from typing import List, Dict, Optional class ElasticsearchHybridSearch: Hybrid search with Elasticsearch and dense vectors. def __init__( self, es_client: Elasticsearch, index_name: str documents ): self.es es_client self.index_name index_name def create_index(self, vector_dims: int 1536): Create index with dense vector and text fields. mapping { mappings: { properties: { content: { type: text, analyzer: english }, embedding: { type: dense_vector, dims: vector_dims, index: True, similarity: cosine }, metadata: { type: object, enabled: True } } } } self.es.indices.create(indexself.index_name, bodymapping, ignore400)要点dense_vector字段开启index: True后 ES 会为向量构建 ANN 索引similarity: cosine指定余弦相似度content使用english分析器做分词/词干化支撑 BM25。方式一bool should script_score向量与 BM25 同场打分def hybrid_search( self, query: str, query_embedding: List[float], limit: int 10, boost_vector: float 1.0, boost_text: float 1.0, filter: Optional[Dict] None ) - List[Dict]: Hybrid search using Elasticsearchs built-in capabilities. # Build the hybrid query search_body { size: limit, query: { bool: { should: [ # Vector search (kNN) { script_score: { query: {match_all: {}}, script: { source: fcosineSimilarity(params.query_vector, embedding) * {boost_vector} 1.0, params: {query_vector: query_embedding} } } }, # Text search (BM25) { match: { content: { query: query, boost: boost_text } } } ], minimum_should_match: 1 } } } # Add filter if provided if filter: search_body[query][bool][filter] filter response self.es.search(indexself.index_name, bodysearch_body) return [ { id: hit[_id], content: hit[_source][content], metadata: hit[_source].get(metadata, {}), score: hit[_score] } for hit in response[hits][hits] ]实现说明script_score中的cosineSimilarity(...) * boost_vector 1.0将余弦相似度范围约 [-1,1]平移为正分数使其能与 BM25 分数在bool.should的求和打分中共存boost_vector/boost_text分别是两个通道的加权系数等效于模板一的线性融合权重minimum_should_match: 1保证至少命中一个通道才返回。方式二Elasticsearch 8.x 原生 RRFdef hybrid_search_rrf( self, query: str, query_embedding: List[float], limit: int 10, window_size: int 100 ) - List[Dict]: Hybrid search using Elasticsearch 8.x RRF. search_body { size: limit, sub_searches: [ { query: { match: { content: query } } }, { query: { knn: { field: embedding, query_vector: query_embedding, k: window_size, num_candidates: window_size * 2 } } } ], rank: { rrf: { window_size: window_size, rank_constant: 60 } } } response self.es.search(indexself.index_name, bodysearch_body) return [ { id: hit[_id], content: hit[_source][content], score: hit[_score] } for hit in response[hits][hits] ]这是把模板一的 RRF 公式下沉到搜索引擎内部sub_searches分别跑 BM25match与近似最近邻knnrank.rrf指定rank_constant: 60与模板一默认k60一致与window_size每个子查询取前 N 名参与融合num_candidates: window_size * 2为 ANN 探索候选数。相比手写融合该方案无需自行归一化分数且融合在 ES 内部完成、返回即最终排序。模板四自定义 Hybrid RAG 流水线第四个模板把前面所有能力封装为一个完整的、可插拔的异步检索流水线HybridRAGPipeline适合在 RAG 系统中直接复用与 rag-implementation/SKILL.md 中 LangGraph 的retrieve节点天然衔接。数据结构与初始化from typing import List, Dict, Optional, Callable from dataclasses import dataclass dataclass class SearchResult: id: str content: str score: float source: str # vector, keyword, hybrid metadata: Dict None class HybridRAGPipeline: Complete hybrid search pipeline for RAG. def __init__( self, vector_store, keyword_store, embedder, rerankerNone, fusion_method: str rrf, vector_weight: float 0.5 ): self.vector_store vector_store self.keyword_store keyword_store self.embedder embedder self.reranker reranker self.fusion_method fusion_method self.vector_weight vector_weight设计要点vector_store/keyword_store/embedder/reranker全部为鸭子类型接口可注入任意实现如 pgvector、Elasticsearch、Pinecone、sentence-transformers是典型的依赖注入设计SearchResult.source字段标记结果来源vector/keyword/hybrid便于调试与日志——对应 SKILL.md 最佳实践中的 Log both scoresfusion_method支持rrf与线性融合切换。主流程 search()async def search( self, query: str, top_k: int 10, filter: Optional[Dict] None, use_rerank: bool True ) - List[SearchResult]: Execute hybrid search pipeline. # Step 1: Get query embedding query_embedding self.embedder.embed(query) # Step 2: Execute parallel searches vector_results, keyword_results await asyncio.gather( self._vector_search(query_embedding, top_k * 3, filter), self._keyword_search(query, top_k * 3, filter) ) # Step 3: Fuse results if self.fusion_method rrf: fused self._rrf_fusion(vector_results, keyword_results) else: fused self._linear_fusion(vector_results, keyword_results) # Step 4: Rerank if enabled if use_rerank and self.reranker: fused await self._rerank(query, fused[:top_k * 2]) return fused[:top_k]四步流水线清晰可循Embedding用注入的embedder生成查询向量并行召回asyncio.gather同时执行向量检索与关键词检索各取top_k * 3个候选过采样为融合/重排留余量并可透传filter做元数据过滤融合按fusion_method选择 RRF 或线性融合内部实现与模板一的公式一致重排若配置了reranker且use_rerankTrue对融合后前top_k * 2的结果用 Cross-encoder 打分重排最终截断到top_k。其余私有方法_vector_search、_keyword_search、_rrf_fusion、_rerank分别封装了单通道查询统一包装为SearchResult并标记source、按1/(krank1)累加融合k60、以及(query, content)对批量打分排序。整体设计把召回多样性 融合鲁棒性 精排质量三个阶段解耦可独立替换任一组件的实现。工程最佳实践Dos 与 DontsSKILL.md 在模板之外给出了经过提炼的工程纪律与四个模板的实现细节相互印证Dos应当遵守经验性调权Tune weights empirically无论alpha模板一线性融合、vector_weight模板二还是boost_vector/boost_text模板三权重都要基于你自己的数据集测试而非拍脑袋优先用 RRFUse RRF for simplicityRRF 不依赖分数归一化、无需调参即可获得稳定效果是通用场景的首选模板二、三、四都以 RRF 为默认路径增加重排Add reranking模板二的search_with_rerank与模板四的_rerank都展示了 Cross-encoder 二阶段精排带来的显著质量提升记录双侧分数Log both scores模板四的SearchResult.source字段即为该实践的落点便于排查哪条通道贡献了哪些结果A/B 测试以真实用户指标验证检索改动而非只看离线指标。Donts应当避免不要假设一套权重通吃不同查询类型精确术语 vs 语义开放问题需要不同融合权重甚至可做查询路由不要省略关键词检索专有名词、代码、编号等精确匹配场景下 BM25/全文检索不可替代不要过度取数Dont over-fetch候选量要在召回率与延迟之间权衡模板中top_k * 3、limit * 3属于可复用的合理起点不要忽略边界情况空结果、单词查询、分数全相等的退化场景——模板一normalize中range_s的除零保护即为示例。与仓库相邻技能/资源的衔接hybrid-search-implementation是 llm-application-dev 检索体系的一环配套查阅以下仓库资源可构成完整知识链embedding-strategies/SKILL.mdembedding 模型选型与维度如voyage-3-large1024 维、text-embedding-3-small1536 维直接决定模板二/三中的vector(1536)、vector_dims1536参数similarity-search-patterns/SKILL.md距离度量Cosine/L2/Dot与索引类型Flat/HNSW/IVFPQ选型支撑模板二 HNSW 索引的调优背景vector-index-tuning/SKILL.mdHNSW 的M、ef_construction、ef_search等参数的召回/延迟权衡rag-implementation/SKILL.md把本技能的search()结果接入 LangGraphretrieve → generate工作流构成端到端 RAGvector-database-engineer承载混合检索实现的专项 Agent其 Workflow 第 6 步 Implement hybrid search: If keyword matching improves results 与本技能直接对应。结语混合检索的价值不在于某一算法的先进而在于把两种互补的检索信号——语义相似度与词法匹配——通过可控的方式融合。本技能在 references/details.md 中给出的四个模板覆盖了从纯 Python 算法原型模板一、单库 SQL 融合模板二、搜索引擎原生能力模板三到可插拔完整流水线模板四的全部层级同一套 RRF 公式贯穿始终k60Cross-encoder 重排作为质量上限的兜底vector_weight/alpha/boost则提供了按业务调优的旋钮。在仓库的 RAG 体系内本技能与rag-implementation、embedding-strategies、vector-index-tuning等技能协同即可支撑从原型验证到生产部署的完整检索链路。【免费下载链接】agentsMulti-harness agentic plugin marketplace for Claude Code, Codex, Cursor, OpenCode, GitHub Copilot, and Google Antigravity项目地址: https://gitcode.com/GitHub_Trending/agents24/agents创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考