Papers with Code Revamps Search Using Hugging Face Jobs, Buckets, and Inference Endpoints

Papers with Code Revamps Search Using Hugging Face Jobs, Buckets, and Inference Endpoints Papers with Code rebuilt its research discovery engine into a hybrid search system covering over 110,000 papers by decoupling offline batch vector generation from online query serving across Hugging Face infrastructure and PostgreSQL. The search engine combines PostgreSQL full-text search with pgvector semantic retrieval using Reciprocal Rank Fusion to balance exact keyword matches and conceptual recall. Batch vector generation runs via Hugging Face Jobs on NVIDIA L4 GPUs using Qwen3-Embedding-0.6B truncated to 256 dimensions with Matryoshka Representation Learning. Hugging Face Storage Buckets serve as an immutable, checksummed storage layer linking database snapshots, compute jobs, and index importers. Online query vectors and hourly catalog deltas are handled by a Text Embeddings Inference endpoint featuring scale-to-zero economics and an immediate fallback to lexical search on latency spikes. Strict Embedding Contracts and Batch Corpus Generation To prevent subtle pipeline errors from model revisions, prompt mismatches, or altered text, Papers with Code enforces an immutable embedding contract that records model revisions, prompt roles, dimensions, normalization, and content hashes. Full corpus processing runs as Hugging Face Jobs on single NVIDIA L4 GPUs. An exporter streams papers from a PostgreSQL snapshot into bounded JSONL shards within a private Storage Bucket mounted directly into the job container. The worker verifies manifests, sorts text by length to minimize padding, generates document embeddings via Qwen3-Embedding-0.6B, truncates vectors to 256 dimensions using Matryoshka Representation Learning, and saves verified Parquet shards atomically. Storage Buckets as System Decoupling Boundaries Hugging Face Storage Buckets act as the explicit boundary between the production database, batch GPU workloads, and search indexing. Organizing runs under immutable prefixes with SHA-256 checksums ensures reproducibility, allows interrupted jobs to resume from verified shards, and enables low-cost dimension experiments without re-running data extraction. Vectors are imported into PostgreSQL only after complete validation of schemas, normalization, and content hashes. An HNSW index is then built in pgvector and activated atomically once coverage of all eligible papers is confirmed, leaving previous index generations available for instant rollbacks. Low-Latency Query Serving with Lexical Fallbacks For user searches, a Hugging Face Inference Endpoint powered by Text Embeddings Inference converts incoming queries into normalized 256-dimensional vectors for cosine-distance queries against pgvector. Pilot testing showed that reducing dimensions from 1024 to 256 achieved 0.9955 Recall@20 with a 1.31 ms p50 lookup latency while consuming only 27% of the original storage. To control costs, the endpoint scales to zero during idle periods. To manage cold-start delays, the client implements a one-second timeout, concurrency limits, and circuit breakers, immediately dropping the semantic search branch in favor of fast PostgreSQL lexical search if the endpoint is slow or unreachable. Hybrid Reciprocal Rank Fusion and Incremental Updates The system fuses up to 50 candidates from the lexical branch and 50 from the semantic branch using reciprocal rank fusion with equal weights and a rank constant of k=60. Deterministic rules override fused ranks to guarantee top placement for exact titles, arXiv IDs, and navigational queries like taxonomy-recognized papers. Rather than triggering batch GPU jobs for routine catalog updates, an hourly incremental worker routes deltas of up to 500 new or modified papers through the existing query endpoint in batches of 16 using document prompts, validating row locks and content hashes to preserve consistency.

Papers with Code 基于 Hugging Face 构建混合检索系统工程实践 Papers with Code 重构了论文搜索引擎，通过结合 PostgreSQL 全文检索与 pgvector 语义检索构建混合搜索系统，并借助 Hugging Face 的 Jobs、Buckets 和 Inference Endpoints，将高吞吐离线语料构建与低延迟在线服务解耦，维护超 11 万篇学术论文的检索能力。 搜索架构采用 PostgreSQL 全文检索与 pgvector 稠密向量互补，并通过倒数排名融合算法合并排序结果。 选用 Qwen3-Embedding-0.6B 模型，利用 Matryoshka 表示学习截断至 256 维，在维持高召回率的同时节省约 73% 的存储空间。 离线全量构建由 Hugging Face Jobs 在 L4 GPU 上完成批处理，产物通过 Storage Buckets 实现带校验的持久化交接与断点续跑。 在线查询由托管 Text Embeddings Inference 的 Endpoint 处理，配置了缩容到零、1 秒超时与快速降级至全文检索的兜底策略。 混合检索与嵌入契约 学术论文搜索既需要精确匹配标题或 arXiv 编号，又需要理解概念性查询。系统采用 PostgreSQL 全文检索提供词法基准，结合 pgvector 扩展语义召回，并使用 RRF 算法平衡两路候选排序。团队将向量生成定义为版本化 API，锁定 Qwen3-Embedding-0.6B 的具体版本，通过 Matryoshka 表示学习将向量维度降为 256 维，并严格区分文档与查询前缀及标准化规则，确保各环节嵌入标准严格一致。 Jobs 与 Buckets 离线构建体系 全量论文嵌入属于高吞吐批处理任务，系统使用挂载 NVIDIA L4 GPU 的 Hugging Face Jobs 运行。导出程序将数据库快照切片上传至私有 Storage Buckets，计算任务直接挂载存储桶，按文本长度排序以降低填充损耗并生成 Parquet 分片。Storage Buckets 作为数据库与计算任务的缓冲边界，通过清单与 SHA-256 校验和保证产物不可变性，支持任务失败后无损断点续跑。 在线推理与冷启动弹性降级 在线查询端部署了基于 Text Embeddings Inference 的推理终端节点。该节点可缩容至零以节省闲置成本，但冷启动也会带来延迟挑战。为此，查询客户端配置了 1 秒超时、并发控制、结果格式校验及熔断机制。当节点处于唤醒冷启动或故障状态时，系统会跳过向量检索，直接返回全文搜索结果，确保前端体验不被外部依赖阻塞。 增量更新与系统演进经验 针对持续流入的新论文与勘误，系统通过每小时执行的轻量级增量任务调用推理节点处理不超过 500 篇论文，避免为了小批量数据重复触发 GPU Job。此外，单篇论文详情页的“相关论文”直接复用已存向量执行近邻查询，无需在请求时再次调用模型。新一代向量在独立构建 HNSW 索引并通过完整性校验后才会原子化激活，极大简化了灰度发布与回滚流程。

3 months ago, we started a revival of Papers with Code (see also the announcement tweet ). Its goal is to make open AI research accessible and digestible, so that people can easily find the artifacts related to a paper, find state-of-the-art (SOTA) across the various domains of AI, share interesting research and build on top of each other's work. In other words, its goal is to power the wave of research that leads to the next Transformer . Of course, making AI research accessible requires a powerful search engine, so that humans and agents can quickly find relevant and related work, either through the website or the pwc search CLI command , which agents can use via the Skill . It's important to note that searching for research is not quite the same as searching for regular text. A useful paper search engine should find an exact title or arXiv identifier, but it should also understand a query such as “small language models for code generation” even when those words do not appear together in a paper. It needs to recognize that “the original BERT paper” is a navigational request, tolerate an incomplete title or typos, and still respond quickly when a model service is cold or temporarily unavailable. Search results on Papers with Code for the query DINO. For Papers with Code , we built this as a hybrid search system. This is also based on our prior experience at ML6 , where we developed RAG -based systems for clients. It turned out that hybrid search typically outperforms keyword- and vector-based search systems, as it combines the best of both worlds (see also this blog for more info). Keyword search finds exact mentions, whereas vector search finds more fuzzy, semantically similar terms. Note that rerankers (also called cross-encoders) can further improve the results, although they also come with additional overhead and latency. Hybrid retrieval outperforms keyword- and vector-only search. Figure from Microsoft, Azure AI Search: Outperforming vector search with hybrid retrieval and reranking (2023). Papers with Code relies on a PostgreSQL database, hence its full-text search capabilities provide a fast lexical baseline. For dense embeddings, pgvector is used to add semantic recall, and the reciprocal rank fusion (RRF) algorithm combines the two. Three Hugging Face services are used for the dense embeddings: Hugging Face Jobs gives us burstable GPU compute for embedding the paper corpus. Hugging Face Storage Buckets provides the durable handoff between our database, experiments, and Jobs. Hugging Face Inference Endpoints serves low-latency embeddings for live queries and incremental updates. Today, the system maintains embeddings for more than 110,000 current papers sourced from arXiv and Daily Papers . This post explains the architecture, the design decisions behind it, and the lessons we learned while taking it to production. TL;DR We deliberately split search into an offline corpus build and an online search service: Architecture of the offline corpus build and online hybrid search pipeline. The expensive, throughput-oriented work runs as Jobs. Durable artifacts live in a Bucket. Only the small query-embedding step sits on the request path, behind a protected Inference Endpoint, to power the online search. If that endpoint is cold, busy, or unhealthy, search immediately falls back to full-text retrieval. This separation makes the system both powerful and fast. Start with a strict embedding contract Embedding pipelines often fail in subtle ways: a model revision changes, query and document prompts are mixed up, vectors are truncated differently, or an updated abstract no longer matches its stored vector. We avoid this by treating the embedding format as a versioned API. Every paper is encoded as: normalized title + "\n\n" + normalized abstract For each vector generation, we record: the model repository and exact revision; the output dimension; the input-format version; whether the input is a query or a document; the normalization method; a content hash for the source title and abstract. Our production generation uses Qwen/Qwen3-Embedding-0.6B , pinned to an exact revision, with 256-dimensional L2-normalized vectors. We selected the model with help from the MTEB leaderboard , the go-to benchmark for comparing embedding models. Note that newer embedding models like Qwen3 allow for 2 new features: one can specify a dynamic embedding size , which allows to trade-off quality with speed/storage costs. Qwen models call this "MRL" which is short for Matryoshka Representation Learning . You can learn all about it here . We chose an embedding size of 256 to make the search fast. one can provide an instruction prompt . Qwen embedding models support a document prompt (which we use to embed the papers) and live searches use their query prompt (to embed the user query). This contract follows an embedding from export, through GPU inference, into PostgreSQL, and finally into online retrieval. Jobs turn a database snapshot into a vector corpus Full-corpus embedding is a classic batch workload. It needs a GPU for a relatively short period, benefits from high throughput, and should not consume resources between runs. Hugging Face Jobs fits that shape well: a Job is defined by a command, a hardware flavor , and optionally a Docker image, and can run uv scripts with their dependencies declared inline. Our corpus build starts by exporting the latest version of every paper from a repeatable-read PostgreSQL snapshot. The exporter streams rows rather than loading the catalog into memory, writes bounded JSONL shards, and creates a manifest containing row counts and SHA-256 checksums. We sync that immutable run directory to a private Storage Bucket and mount the Bucket directly (using hf-mount ) into an l4x1 Job (an NVIDIA L4 GPU, which has 24GB of VRAM). From the worker's perspective it is simply a filesystem: hf jobs uv run \ --flavor l4x1 \ -- timeout 6h \ --volume hf://buckets/OWNER/pwc-paper-embeddings:/bucket \ embed_papers_job.py \ --input /bucket/runs/RUN_ID/input \ --output /bucket/runs/RUN_ID/output \ --model Qwen/Qwen3-Embedding-0.6B \ --revision MODEL_REVISION \ --dimensions 256 \ --allow-matryoshka The worker: verifies the input manifest and every shard checksum; loads the pinned model revision; sorts texts by length to reduce padding; calls encode_document in batches (as noted in the model card ); reduces the batch size automatically if the GPU runs out of memory; truncates the Matryoshka representation to 256 dimensions and normalizes it; writes float16 Parquet shards atomically; and records throughput, package versions, hardware, peak VRAM, row counts, and output checksums. Each completed shard has its own marker, so a restarted Job can skip verified work. This is useful for a large corpus: retrying should just resume work rather than overwriting existing embeddings. In our 5,000-paper pilot, the Qwen Job encoded about 75 papers per second at 1024 dimensions on an L4 GPU. The same pass could be deterministically materialized at 512 and 256 dimensions, so we could compare the storage and retrieval trade-offs without paying for more inference. Buckets are the connective tissue Storage Buckets are mutable, S3-like object storage on the Hub, optimized for AI workloads. They can be accessed through hf://buckets/... paths and mounted read-write in Jobs without building a separate storage integration. For us, the Bucket is more than a place to put vectors. It is the boundary between three systems with different lifecycles: the production database exports source records; ephemeral Jobs consume those records and produce vectors; the importer validates the results before touching the search index. We organize artifacts under immutable run prefixes: runs/<run-id>/ ├── input/ │ ├── manifest.json │ └── papers-*.jsonl └── output/ ├── manifest.json ├── embeddings-*.parquet └── embeddings-*.complete.json Buckets themselves are intentionally mutable, so immutability is an application-level rule: a run ID is never overwritten, and every artifact is covered by a manifest and checksum. This gives us several useful properties: Reproducibility: we can trace a database generation back to an exact corpus snapshot, model revision, and set of artifacts. Safe retries: Jobs can resume from completed shards in the same run prefix. Cheap experiments: several models or dimensions can reuse one verified input snapshot. Controlled rollout: importing a generation does not activate it. We first validate coverage and build its index. Simple rollback: the previous generation and its artifacts remain available until the new one is proven stable. Only after the importer rechecks schemas, checksums, dimensions, normalization, unique paper IDs, and current content hashes do we load the vectors into PostgreSQL. We then build a separate HNSW index for the new generation and atomically mark it active only when every eligible current paper is covered (HNSW is the graph-based algorithm that enables fast vector search). Inference Endpoints put semantic search on the request path Batch embeddings solve the document side of retrieval. A user query still needs to be embedded at request time using the same model contract. We deploy the pinned model as an authenticated Inference Endpoint backed by Text Embeddings Inference (TEI) . The endpoint accepts the query text and returns a normalized 256-dimensional vector using the model's query prompt. Note that one could also leverage vLLM or SGLang here. Hugging Face Inference Endpoint for the Papers with Code query embedding model. The API then performs a cosine-distance search over the active pgvector generation: SELECT paper_id, embedding <=> CAST (:query_vector AS halfvec( 256 )) AS distance FROM paper_embeddings WHERE generation_id = :active_generation ORDER BY embedding <=> CAST (:query_vector AS halfvec( 256 )) LIMIT 50 ; The HNSW index keeps this lookup fast. On our 5,000-paper pilot, the 256-dimensional Qwen index achieved 0.9955 Recall@20 against exact search, with 1.31 ms p50 and 2.21 ms p95 HNSW lookup latency. Its table and index used about 27% of the storage of the 1024-dimensional version while retaining essentially the same ANN recall in that test. The Endpoint is configured with a maximum of one replica and can scale to zero when idle. That is a useful cost lever, as this means you're not paying when there's no usage. However, this also means cold starts must be part of the application design rather than treated as an exceptional event, as it takes some time for the endpoint to spin up and serve traffic. Our query client therefore has deliberately strict behavior: a one-second production timeout; a non-blocking concurrency limit; response dimension, finiteness, and norm validation; a short cache keyed by the query and embedding generation; a circuit breaker after repeated failures; and no raw query text in logs, only a normalized fingerprint. If the endpoint is scaling up, times out, returns a malformed vector, or has no concurrency available, we skip the semantic branch immediately. Users still receive lexical results instead of waiting for an unreliable dependency. Inference Endpoints works really reliably, and includes a nice dashboard so you can quickly see key analytics. Inference Endpoint analytics showing request volume, errors, latency, and replica state. Hybrid retrieval is stronger than either branch alone For every query, the lexical branch retrieves up to 50 candidates using weighted PostgreSQL full-text search. The semantic branch retrieves up to 50 candidates from pgvector. We combine their ranks using weighted reciprocal rank fusion (RRF): score ( d ) = ∑ r ∈ { lexical , semantic } w r k + rank r ( d ) \text{score}(d) = \sum_{r \,\in\, \{\text{lexical},\, \text{semantic}\}} \frac{w_r}{k + \text{rank}_r(d)} RRF is simple and robust, because it combines ranks rather than scores from two systems with different scales. Basically, if a paper is ranked high both by the lexical branch and the semantic branch, it has a higher chance of being ranked high by the hybrid search. We currently use equal branch weights and (k=60) (k is the "rank constant", a hyperparameter of the RRF algorithm). Dense retrieval improves recall for conceptual queries. Full-text retrieval remains excellent for exact terminology, identifiers, and rare names. We also preserve deterministic identity behavior on top of the fused ranking: exact titles and arXiv IDs stay at the top; the method taxonomy recognizes navigational searches such as “the original BERT paper”; incomplete titles and bounded spelling mistakes use conservative trigram candidates; and ambiguous fuzzy matches abstain rather than forcing a bad result. Note: hybrid search isn't always the best option, it is recommended to start with keyword search as a cheap and fast baseline, and only adding semantic and/or hybrid search when it turns out those give a reasonable boost in retrieval quality. One could further improve the search by adding a reranker after keyword/semantic/hybrid retrieval, using a model like Qwen3-Reranker . One Endpoint, two update paths The large initial corpus is embedded with Jobs, but Papers with Code changes continuously. New papers arrive, abstracts are corrected, and new arXiv versions become current. Launching a GPU Job for a handful of changed rows would add unnecessary startup and orchestration overhead. Instead, an hourly incremental process selects missing or content-changed papers and sends a bounded delta to the same TEI Endpoint, this time with the document prompt. Each run processes at most 500 papers in batches of 16. Before an embedding is written, the source row is locked and its content hash is checked again. If a paper changed during inference, that vector is discarded and picked up by the next run. This gives us a useful division of labor: Jobs handle full rebuilds, new model generations, and large backfills. Inference Endpoints handle interactive query embeddings and small incremental document updates. Buckets preserve the large-build artifacts and make those builds resumable and auditable. The hourly path keeps the active index close to the live catalog without turning an online endpoint into an unbounded batch processor. Related papers become almost free online The same document embeddings also power related-paper recommendations on each paper page. Related papers for SenseNova-U1 . Because the source paper already has a stored vector, related-paper retrieval requires no model call at request time. It is a single nearest-neighbor query over the active generation. If a vector is temporarily missing, the application can use a previous arXiv version or fill results from the existing task- and citation-based fallback. We fetch citation data through the Semantic Scholar API and also built s2-cli , a command-line interface for querying its citation graph. The latter is used by the agent at https://paperswithcode.co/chat . What we learned 1. Separate throughput work from latency-sensitive work Corpus embedding and query embedding use the same model, but they are different infrastructure problems. Jobs optimize for throughput and bounded cost; Inference Endpoints optimize for availability and request latency. 2. Make storage the explicit contract between compute and production Buckets provide an explicit handoff between compute and production. Checksummed artifacts create a reviewable boundary before data enters the production index. 3. Pin more than the model name The revision, dimension, prompt, normalization, and input formatter all affect retrieval. Store them together and validate them everywhere. 4. Design for cold starts Scale-to-zero is valuable when traffic is intermittent, but only if the product has a fast fallback. Hybrid search gave us that fallback naturally: lexical search is always useful on its own. 5. Smaller vectors can be a systems feature Matryoshka embeddings let us evaluate quality, memory, index size, and latency as one trade-off. In our pilot, 256 dimensions preserved ANN recall while materially shrinking storage compared with 1024 dimensions. 6. Activation should be boring New generations are imported beside the current one, indexed independently, checked for complete and current coverage, and then activated atomically. Rollback is a configuration change, not an emergency recomputation. Feel free to try out the search at https://paperswithcode.co or the chat interface at https://paperswithcode.co/chat , and let us know any feedback!