Meta Slashes LLM Use in Data Privacy Infrastructure via Rule Distillation
@Scale
Meta builds an AI-native privacy engine that uses LLMs to generate auditable rules, reducing latency and compute costs while ensuring automated data classification.
## Background
Published on YouTube by @Scale on June 26, 2026, this presentation features Meta's Rituraj Kirti (delivered by Amrit Raj) discussing privacy-aware infrastructure in the AI-native era. According to the page description, the talk covers a hybrid deterministic rules engine with LLM fallback for asset classification, highlighting self-improvement loops, LLM-as-a-Judge evaluation, and distilling model behavior into auditable rules.yaml files.
## TL;DR
Meta addresses the massive scale of data classification by evolving from human-assisted AI to fully autonomous, replayable AI-native privacy infrastructure. To combat extreme signal noise and token bloat, the system transforms multi-source context into structured evidence briefs while employing a three-LLM judge panel with Cohen's Kappa scoring to detect drift and route edge cases to human review. High-confidence LLM decisions are distilled into deterministic, human-readable rules.yaml files, drastically lowering wall-clock latency, slashing compute costs, and reserving expensive LLM calls exclusively for ambiguous cases.
## Atmosphere & style
The presentation carries a pragmatic, engineering-focused tone that balances high-level architectural vision with transparent candor about real-world failures. The speaker presents complex technical challenges with grounded expertise, using evocative metaphors like 'scars behind the principles' to discuss initial missteps. It feels like an authoritative, insightful technical case study that candidly exposes operational friction to deliver actionable architectural lessons.
## Key takeaways
- [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) Privacy serves as a driving force for building better, more adaptable, and replayable software architecture rather than merely being compliance red tape.
- [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) Meta's four-part recipe for purpose-limitation in privacy infrastructure includes Understand, Discover, Enforce, and Demonstrate, with asset classification serving as the foundation.
- [02:42](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=162s) A single data table column can branch into over 100,000 downstream nodes across logs, ML features, and pipelines, making manual human review impossible at scale.
- [06:16](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=376s) Dumping raw context into LLMs causes token usage to explode and dilutes model attention; Meta resolves this by building structured evidence briefs prior to routing.
- [06:16](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=376s) Circular context signals are masked system-wide to ensure models and rule engines do not predict labels from data that already contains those labels.
- [08:39](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=519s) To handle signal drift, an automated evaluation panel uses three distinct LLM judge framings and Cohen's Kappa score to evaluate batch samples.
- [08:39](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=519s) When LLM judges disagree, assets route to human review, which serves as the sole mechanism for updating the frozen ground truth reference set.
- [10:44](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=644s) Meta uses a 5-step distillation pipeline to progressively convert high-confidence LLM decision patterns into human-readable rules in rules.yaml.
- [10:44](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=644s) Distilling LLM behavior into deterministic rules reduces execution latency to milliseconds, minimizes compute costs, and ensures repeatable logic for high-volume patterns.
- [13:26](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=806s) Context quality caps model performance far more than prompt phrasing, proving that structuring context is more impactful than automated prompt optimization.
- [13:26](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=806s) To prevent prompt optimization loops from oscillating endlessly and wasting compute, Meta integrated a Kappa-based controller into the architecture.
- [15:48](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=948s) The core design principles of AI-native infrastructure—context-first design, decoupled evaluation, and deterministic distillation—apply directly to agent observability and oversight.
## Detailed walkthrough
### [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) Privacy as a Driving Force for Architecture
Amrit Raj opens the presentation with a thesis that might seem counterintuitive at first: privacy is not merely a tax on engineering, a burden of process, or a layer of compliance red tape. Instead, in the AI-native era, privacy acts as a powerful driving force for building better, more adaptable, deterministic, and highly reliable software architecture. Tracking Meta’s trajectory in this domain, Raj notes that in 2025 the team shared how they accelerated privacy workflows using Generative AI (GenAI) to help human reviewers evaluate data faster through ground truth and continuous validation. Later, they published a blog post on scaling privacy infrastructure for rapid GenAI-driven product iteration. Today's talk represents the next evolutionary step: transitioning from "AI helps humans" to a fully AI-native infrastructure that autonomously produces robust, replayable enforcement decisions.
To ground the discussion, Raj introduces asset classification as the core case study. He outlines Meta's four-part "recipe for purpose limitation" in privacy-aware infrastructure:
1. **Understand:** Identify where the sensitive data that needs protection resides.
2. **Discover:** Map the data flow graph to understand where protected data flows and where it originated.
3. **Enforce:** Prevent regressions against privacy requirements across the complete data graph.
4. **Demonstrate:** Maintain an audit trail and system assurance to prove ongoing compliance.
The talk focuses squarely on the first workflow—**Understand**—and specifically on how to accurately classify assets so that downstream enforcement and demonstration remain rock-solid.
- The shift moves Meta from human-in-the-loop assistance to autonomous AI-native infrastructure capable of replayable decision-making.
- "Privacy as a driving force for better architecture."
- Asset classification forms the absolute foundation; if classification fails, all downstream enforcement and demonstration workflows break.
---
[02:42](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=162s) The Scale of the Problem & The Classification Pipeline
Visualizing a data flow graph derived downstream from a single Hive table column reveals an intricate web of 100,000 nodes—where each node represents a table column, log field, ML feature, or pipeline stage carrying derived data. Raj emphasizes that while this 100,000-node graph pushes the absolute limits of human visual comprehension, it represents only a microscopic fraction of the scale Meta operates at. Modern AI-native products introduce rapid new data modalities like audio, computer vision, embeddings, sensor streams, and feedback loops that ship far faster than manual human review can keep up.
Misclassifying these assets carries high stakes. A wrong label can lead to incorrect enforcement—either blocking legitimate data flows and breaking product features, or improperly permitting restricted data. Missing a category means failing a core privacy requirement, and lacking a trace destroys auditability. To tackle this, Meta built an end-to-end classification pipeline triggered by schema modifications, code changes, or configuration updates. The pipeline pulls multi-source context (lineage, ownership metadata, annotations, code search results), sanitizes and masks circular signals, and routes the asset to deterministic rules first, falling back to an LLM only when rules cannot handle it. Both paths emit a category, a confidence score, and a full decision path saved as versioned evidence.
- A single source column can branch into over 100,000 downstream nodes across logs, ML features, and pipelines.
- Modern modalities like sensor data and embeddings ship too quickly for human manual review.
- The pipeline combines noisy inputs (ambiguous names, partial lineage, false-positive scanners) to produce crisp, auditable output traces.
---
[06:16](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=376s) Challenge 1: Signal Noise & Structured Evidence Briefs
Raj introduces three central technical challenges: extreme signal noise, severe drift/variability, and massive compute/latency scale. Addressing the first challenge, Raj explains that fetching dozens of raw context types—lineage, code references, schema definitions, sample data—and dumping them raw into an LLM causes catastrophic failure. The model is forced to rediscover what matters on every single API call, causing token usage to balloon, diluting attention across noise, and blurring decision boundaries so that misleading or circular fields slip in unnoticed.
Meta solves this by abandoning raw context dumps in favor of structured **evidence briefs**—ranked summaries of signals created before hitting the decision engine. Each incoming context field is evaluated against a strict policy: whether it is useful, conditionally useful, circular (requiring masking), or prone to mispredictions (requiring blocklisting). Circular information is masked system-wide so the model isn't predicting a label from data that already contains that label. By explicitly presenting top supporting and contradicting signals, the system tells the model what matters rather than asking it to guess.
- Ingesting raw context dilutes LLM attention and inflates token costs significantly.
- Context fields are classified as useful, conditionally useful, circular, or blocked.
- Evidence briefs surface explicit top supporting and top contradicting signals to guide decision-making.
The second challenge stems from drift and variability across data batches and over time. When optimization targets constantly shift due to noisy ground truth labels, optimizers end up chasing a moving target—tracking drift rather than true model progress—which allows dangerous regressions to slip into production.
To overcome this, Meta implements an automated evaluation loop on sample assets from every single batch using a panel of three LLM judges running distinct prompt framings:
1. Direct classification.
2. Critique-then-classify.
3. Metadata-only classification.
Raj notes that three is the smallest odd-numbered panel capable of producing a clear majority vote without complex tie-breaking logic. All three judges use a model architecture deliberately distinct from the main classifier model to ensure framing and structural diversity. The system measures inter-judge agreement using **Cohen's Kappa**. When judges disagree, the asset routes to a human review gate where human decisions resolve ties and update the frozen ground truth reference set. When judges unanimously agree, the asset is flagged for potential promotion. Raj candidly notes the "honest limit": if all three judges agree on an incorrect label, the system won't catch it on that individual asset, but will detect the systemic anomaly through macro-level Kappa drift.
- Evaluates batch samples using 3 distinct LLM judge framings to reach a majority vote.
- Uses Cohen's Kappa score to measure statistical agreement among judges.
- Disagreements route directly to human reviewers, who serve as the sole source of updates to the frozen reference set.
---
[10:44](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=644s) Challenge 3: Distillation to Deterministic Rules
The third challenge is sheer scale: processing many millions of assets through raw LLM calls is impossible. LLM calls take seconds, consume costly tokens, and yield probabilistic outputs that make auditors ask, "Why did it behave differently when I tested it?" Meta requires millisecond latency, deterministic repeatability for common cases, and zero LLM overhead for standard traffic.
Meta's solution is to "use LLMs to put LLMs out of business" through a 5-step rule distillation pipeline:
1. **Identify Patterns:** Scan for stable, high-confidence LLM decision patterns across assets.
2. **Propose Rule:** Generate an explicit deterministic rule (regex, heuristic, or field match).
3. **Validate:** Test the proposed rule against a holdout dataset; if precision and support fall short, blocklist the rule pattern.
4. **Shadow Mode:** Run validated rules in production in logging-only mode to compare output directly against LLM predictions.
5. **Promote:** Once confirmed stable, graduate the rule to `rules.yaml`.
This distillation cycle continually shrinks the LLM surface area. The end state leaves standard, high-volume patterns running at millisecond latency with zero LLM dependency, reserving expensive model calls strictly for genuinely ambiguous cases. Applying this distilled boundary classification drastically reduces the problem surface area on the 100,000-node graph shown earlier.
- "Use LLMs to put LLMs out of business."
- Promoted rules live in a human-readable, auditable `rules.yaml`.
- Drastically lowers wall-clock latency to milliseconds and slashes compute costs toward zero.
---
[13:26](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=806s) Lessons Learned: Masking, Context, and Oscillation
Sharing the "scars behind the principles," Raj candidly breaks down three major architectural missteps and how the team corrected them:
- **Masking Misalignment:** Initially, masking circular fields was treated as a prompt-level concern for the LLM, while deterministic rules processed unmasked fields. Because rules and LLMs evaluated different data, rule coverage plateaued. Making masking a system-wide invariant so both engines see identical masked context caused rule coverage to jump and LLM usage to drop.
- **Prompt Optimization Ceiling:** The team spent many hours running state-of-the-art automated prompt optimizers, yet accuracy hit an unbreakable ceiling. Investigation revealed the prompt wasn't the problem—the input data wasn't clean or structured enough. Improving context quality and code resolution broke through the performance ceiling using the exact same prompt and model.
- **Prompt Oscillation:** Without tracking feedback dynamics, the prompt optimization loop bounced back and forth between two candidate prompts for dozens of iterations, burning massive compute without converging. Meta resolved this by building a Kappa-based tuning controller into the architecture to detect and halt prompt oscillation.
- Masking must be a system-wide invariant across both rule engines and LLMs.
- "Context > Prompts": Quality of context limits model accuracy far more than prompt phrasing.
- Built a Kappa-based controller directly into the architecture to catch prompt oscillation loops.
Raj summarizes the engineering takeaways into three core principles for AI-native infrastructure:
1. **Context > Prompts:** Invest heavily in structuring evidence briefs before tweaking prompt phrasing.
2. **Decouple Eval from Optimization:** Run evaluations using independent models, separate prompts, and frozen reference sets; sharing loops measures drift, not progress.
3. **Distill to Deterministic:** Progressively shrink the LLM surface area into versioned, auditable rules.
Looking ahead, Meta is actively migrating legacy hand-coded classifiers into this context-first distillation framework and extending the pattern to other privacy workflows like lineage improvement, safe enforcement rollouts, and evidence generation. Beyond privacy, Raj emphasizes that these patterns apply directly to agent observability and agent oversight—domains that face identical challenges with noisy signals, deterministic enforcement requirements, and replayable auditability needs. He closes by reiterating that privacy infrastructure isn't engineering friction, but a catalyst for cleaner data contracts, richer context, and disciplined architectural design.
- Core principles: Context over prompts, decoupled evaluation, and continuous distillation to deterministic rules.
- Expanding the AI-native approach to lineage, enforcement rollouts, and evidence generation.
- The pattern directly translates to broader AI engineering domains like agent observability and oversight.
## Notable quotes
> [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) privacy is not merely a tax on engineering, a burden of process, or a layer of compliance red tape. — Amrit Raj
> [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) Privacy as a driving force for better architecture. — Amrit Raj
> [00:00](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=0s) Privacy-by-design infrastructure that converges, exports human-readable logic, and monitors drift in production. — Rituraj Kirti
> [10:44](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=644s) Use LLMs to put LLMs out of business. — Amrit Raj
Amrit Raj argues that privacy requirements act as a primary catalyst for robust software architecture rather than an engineering tax. Focusing on Meta's asset classification infrastructure, he details a system that transitions from human-assisted review to an AI-native pipeline that progressively distills probabilistic model decisions into deterministic, auditable rules.
Scaling Classification Across Complex Data Graphs (2:42)
At Meta's operational scale, a single source column in a Hive table can propagate into more than 100,000 downstream nodes spanning derived logs, machine learning features, and processing steps. Rapidly emerging AI-native modalities—including audio, sensor streams, and embeddings—generate data faster than manual human review can audit. Raj stresses that misclassifying these nodes creates high operational risks: wrong labels can either inappropriately block functional product features or permit restricted data flows, while unrecorded lineage invalidates regulatory compliance audit trails.
To manage this scale, Meta built an automated classification pipeline triggered by schema revisions, code modifications, or configuration updates. The system gathers multi-source lineage and metadata, routing each asset to deterministic rules first. When deterministic checks cannot classify an asset, the system falls back to a large language model (LLM). Both execution paths output a category, a confidence score, and a versioned decision path to preserve end-to-end auditability.
Structuring Evidence Briefs and System-Wide Masking (6:16)
Supplying raw context—such as unparsed schema files, code references, and sample data—directly to an LLM leads to operational failure. Raj notes that raw context dumps inflate token usage, dilute model attention across background noise, and allow circular or misleading metadata to distort predictions. Meta encountered an accuracy ceiling when relying solely on automated prompt optimizers, demonstrating that context structure governs model performance far more than prompt phrasing.
Meta addressed signal noise by pre-processing context into structured evidence briefs containing ranked supporting and contradicting signals. Incoming attributes are explicitly designated as useful, conditionally useful, circular, or blocklisted. Addressing an earlier architectural error where circular signals were masked only within prompts, Meta enforced masking as a system-wide invariant across both rule engines and LLMs. Raj reports that aligning masking across both layers raised rule coverage and significantly lowered LLM query volume.
Multi-Judge Evaluation and Drift Detection (8:39)
Variable data batches and fluctuating optimization targets can lead classifiers to track data drift rather than genuine accuracy gains. To decouple evaluation from model training, Meta samples assets from every production batch and evaluates them using a panel of three distinct LLM judges. The judges employ three separate prompt framings—direct classification, critique-then-classify, and metadata-only—and operate on a different model architecture than the primary classifier to ensure structural diversity.
The system evaluates inter-judge consensus using Cohen's Kappa. Disagreements among the judges route to human reviewers, who resolve the ambiguity and update a frozen ground truth reference set. Unanimous evaluations qualify assets for automatic promotion. Raj highlights an operational limit: if all three judges unanimously agree on an erroneous classification, the system cannot detect the error on that single asset, relying instead on aggregate Kappa metric drift to expose systemic anomalies.
Distilling Model Decisions into Deterministic Rules (10:44)
Running high volumes of recurring data through LLMs is unsustainable due to compute costs, multi-second latency, and probabilistic variations that complicate regulatory audits. To enforce millisecond-level execution and repeatable outcomes, Meta uses LLMs to generate deterministic logic through a five-stage distillation pipeline: identifying stable decision patterns, proposing explicit heuristic rules, validating precision on holdout data, running rules in production shadow mode, and promoting validated rules to a human-readable `rules.yaml` file.
This continuous distillation progressively reduces the active LLM operational footprint, running standard patterns on zero-overhead deterministic rules while reserving model calls for novel or ambiguous edge cases. To prevent optimization loops from oscillating between candidate prompt variations without converging, Meta integrated a Kappa-based tuning supervisor directly into the pipeline.
Meta is expanding this context-first distillation framework to adjacent workflows such as data lineage enhancement, safe policy rollouts, and broader agent observability. Raj acknowledges an ongoing structural limitation: because validation relies on multi-judge consensus, unanimous judge errors on individual assets bypass immediate human review and can only be caught through macro-level statistical drift.
## Background
本视频由技术会议频道 @Scale 于 2026 年 6 月 26 日在 YouTube 发布。视频记录了来自 Meta 的 Rituraj Kirti 所作的精彩演讲，内容聚焦于 AI 原生时代隐私感知基础设施（PAI）的构建，深入拆解了如何通过“确定性规则优先 + LLM 兜底”的混合架构对数以千万计的数据资产进行高精度分类，并展示了借助评测闭环与知识蒸馏将大模型能力沉淀为可审计规则 `rules.yaml` 的端到端实践。
## TL;DR
Rituraj Kirti 开篇指出“隐私合规不仅不是业务阻碍，更是驱动架构跃迁的核心动力”，并将资产分类确立为目的限制（Purpose Limitation）全链路的底座。针对十万级衍生血缘节点与严重的上下文噪声，Meta 团队设计了“结构化证据摘要”（Evidence Briefs）与三判官评估体系（结合 Cohen's Kappa 一致性度量），精准阻断循环推论与数据漂移。系统核心奉行“用大模型革大模型的命”的理念，经由模式提取、保留集校验和影子模式验证，持续将高置信度的 LLM 推理蒸馏为毫秒级响应的确定性规则。团队基于踩坑经验总结出“上下文重于 Prompt”、“评估与优化物理隔离”以及“向确定性逻辑收敛”三大架构铁律，并展望了这套方法论在未来通用 AI Agent 行为监管与可观测性中的广阔应用前景。
## Atmosphere & style
整场演讲流淌着极具 Meta 工程风格的务实、克制与硬核严谨，没有对前沿模型的盲目狂热，而是充满了对生产级落地指标的冷静敬畏。演讲者带着工程团队在真实战壕中换来的“伤疤记忆”，坦率复盘诸如脱敏未对齐、盲目打磨 Prompt 措辞导致算力空转横跳等失败教训，极具真诚感与说服力。从展示令人头皮发麻的十万节点血缘膨胀，到用精简的 `rules.yaml` 将非确定性模型调用压缩至近乎为零，叙事层层推进，让听众深刻体会到一种用严密系统工程驯服概率大模型的理性之美。
## Bulletin
- [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) Meta 提出将隐私保护视为推动架构演进的核心动力，系统从传统的“AI 辅助人工”跃迁至自主决策、可精确重放的 AI 原生隐私感知基础设施（PAI）。
- [02:42](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=162s) 针对单个数据表列下游衍生膨胀出十万级节点的图谱规模，Meta 采用“规则优先、LLM 兜底”的混合决策架构，结合脱敏前置处理以规避循环推论。
- [10:44](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=644s) 提出“用 LLM 革 LLM 的命”，通过保留集校验与生产影子模式，将模型沉淀的高置信度逻辑持续蒸馏为确定性的 rules.yaml，实现毫秒级零 LLM 依赖执行与审计可重放。
- [15:48](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=948s) 工程团队总结出三大核心架构原则：输入模型的上下文结构远重于 Prompt 优化措辞、评测体系必须与优化过程物理解耦、运行层应坚持向确定性逻辑持续蒸馏。
## Key takeaways
- [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) 隐私工程不应被视为被动合规的阻碍税负，反而能倒逼系统建立更清晰的数据契约与更具确定性的底层软件架构。
- [01:33](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=93s) 落实数据“目的限制”需要经历理解、发现、执行和举证四大工作流，其中资产分类是支撑后续链路的核心基石。
- [02:42](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=162s) 面对十万级下游节点衍生网络及多模态新型资产，低容错率使得纯人工标注彻底失效，必须依赖全自动资产分类流水线。
- [04:29](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=269s) 端到端分类流水线将变更事件触发、上下文聚合脱敏与“确定性规则优先 + LLM 兜底”相结合，输出带版本号的审计证据。
- [07:01](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=421s) 直接向 LLM 倾倒未经筛选的原始元数据会导致 Token 暴增、注意力稀释与循环推论，必须引入结构化证据摘要（Evidence Briefs）。
- [08:39](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=519s) 采用由 3 个不同视角组成的判官面板配合 Cohen's Kappa 系数评估一致性，遇到分歧时交由人工审查关口仲裁以应对数据漂移。
- [10:44](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=644s) 大模型在千万级资产规模下存在单次调用延迟高、算力成本高昂及结果非确定性的瓶颈，无法胜任全量常规生产吞吐。
- [12:02](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=722s) 规则蒸馏闭环通过在保留集上校验精确度与支持度，并经过生产影子模式比对，将稳定模式自动沉淀至 rules.yaml。
- [12:02](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=722s) 边界分类策略大幅压缩了数据流图中下游需深度合规分析的节点数量，促使隐私感知基础设施具备横向扩展能力。
- [13:26](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=806s) 数据脱敏不能仅作为 Prompt 内部事务，必须升级为全局系统级不变量，确保规则引擎与 LLM 在完全一致的脱敏上下文中评估。
- [13:26](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=806s) 在劣质或缺乏结构的数据上花费大量算力优化 Prompt 措辞回报极低，应优先投资输入上下文的质量与代码语义解析。
- [13:26](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=806s) 自动化调优流程中必须引入基于 Cohen's Kappa 的断路器机制，防止 Prompt 候选方案陷入无序振荡并白白烧掉算力。
- [15:48](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=948s) 评估体系必须采用独立模型、独立 Prompt 及完全冻结的基准参考集，与优化流水线彻底解耦以避免测量自我漂移。
- [15:48](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=948s) LLM 擅长归纳学习但直接承载规模化吞吐表现糟糕，架构设计应致力于向确定性规则蒸馏以主动缩减大模型的调用暴露面。
- [16:54](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=1014s) 从噪声中提取微弱信号、确定性边界管控以及可重放审计的蒸馏治理架构，可直接外推至通用的 AI Agent 可观测性与合规监管。
## Detailed walkthrough
### [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) 核心论点：隐私是推动架构演进的源动力
来自 Meta 的 Rituraj Kirti 开篇抛出了一个初听起来颇具反直觉的论点——"Privacy as a driving force for better architecture"（将隐私视为构建更优架构的核心驱动力）。长期以来，工程团队往往将隐私保护视作被动的流程合规、安全红线或是层层审批的繁文缛节。但在 AI 原生时代，为隐私而设计不仅不是技术创新的阻碍，反而从底层促成了更具适应性、更具确定性以及高度可靠的软件系统架构。
回顾演进历程，团队在 2025 年曾分享过如何利用生成式 AI（GenAI）加速隐私审查工作流，当时的核心模式是借助大模型辅助人工审核，配合真实基准值（Ground Truth）与持续验证体系。随后，团队发布了关于如何扩展隐私基础设施以支持 GenAI 驱动产品高速迭代的技术实践。而本次分享标志着工程范式的进一步跃迁：彻底跨越"AI helps humans"（AI 辅助人工）的辅助阶段，迈向真正的 AI 原生基础设施——让系统完全自主做出高鲁棒性的决策，并实现可精确重放（replayable）的策略执行。作为这一范式的具体落脚点，资产分类（Asset Classification）成为了本次深度剖析的端到端案例。
### [01:33](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=93s) 目的限制的四大工作流与资产分类定位
在隐私工程领域，确保数据被合规使用的核心原则在于"目的限制"（Purpose Limitation）。为了让这一原则落到实处，Meta 提炼出了四个环环相扣的关键工作流：
- **理解（Understand）：** 精确识别需要保护的数据资产到底存放在何处。
- **发现（Discover）：** 在明确定位敏感数据后，持续追踪数据的全链路流向，厘清数据的上游来源与下游衍生去向。
- **执行（Enforce）：** 在掌握完整的受保护数据血缘图谱后，构建阻断机制，防止任何违反隐私合规要求的行为发生。
- **举证（Demonstrate）：** 提供完整且具备审计能力的证据链（Audit Trail），向审计团队和监管方持续证明系统始终稳定运转并满足隐私标准。
本次演讲聚焦于最前端也是最根本的底座工作流——**理解（Understand）**。只有首先把千差万别的海量数据资产准确分类，下游的数据流向阻断、隐私策略执行以及合规举证才可能做到精准无误与高度可信。
### [02:42](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=162s) 资产规模与分类挑战：从十万节点图谱到多模态数据
为了具象化理解面临的技术鸿沟，Rituraj 展示了一张令人震撼的数据流向图：仅仅从单一 Hive 表的某一列出发，其下游衍生出的依赖网络就膨胀出了整整 100,000 个节点。每一个节点都可能是一个下游数据表字段、日志埋点属性、机器学习特征（ML Feature）或是某一阶段的数据处理管道。这种复杂度已经达到了人类肉眼和大脑视觉认知的极限，但在 Meta 的整体业务体量下，这甚至仅仅是一个微不足道的数据子集。
```mermaid
flowchart LR
accTitle: 数据血缘衍生膨胀示意图
accDescr: 展示单个 Hive 列如何在下游衍生出表字段、日志属性、ML 特征等十万级资产节点。
subgraph Source["数据源头"]
S["单个 Hive 数据表列"]
end
subgraph Scale["十万级下游衍生网络"]
N1["下游表列字段"]
N2["日志埋点属性"]
N3["机器学习特征 (ML Feature)"]
N4["数据处理管道阶段"]
end
S --> N1
S --> N2
S --> N3
S --> N4
```
所谓"资产"（Asset），在工程定义上是指任何承载或传递数据的载体。伴随着 AI 原生产品的爆发，系统中的数据形态发生了剧变，涌现出海量全新模态：音频片段、视觉图像、高维向量嵌入（Embeddings）、传感器时序信号以及各类交互反馈循环。这些新型资产的上线与迭代速度极快，传统依靠人工审查填报标签的模式彻底宣告失效，唯有依赖高吞吐的自动化资产分类才能跟上业务步伐。
分类环节的容错率极低：
- **错误标记标签：** 触发错误的执行策略。若误判为违规，可能会错误拦截核心数据流，直接导致线上产品功能瘫痪；若漏判违规，则会放行非法数据流，造成严重的合规事故。
- **遗漏敏感类别：** 意味着直接击穿隐私合规基线。
- **缺失审计追踪：** 无法在面对内外部审查时举证系统决策依据。资产分类是整个隐私大厦的基石，基石一旦松动，整条下游防线将全线崩塌。
### [04:29](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=269s) 端到端分类架构：规则优先与大模型兜底的混合设计
针对海量资产分类，Meta 构建了一套兼顾确定性与智能推理的端到端分类流水线。系统由模式变更（Schema Change）、代码提交或配置更新等事件自动触发启动。随后，上下文引擎会从多方源头并行抓取上下文元数据，包括血缘图谱、归属权（Ownership）信息、人工注解、ML 特征定义以及代码搜索引用。
为了防止有害干扰，数据在喂入决策引擎前必须经过前置的脱敏与数据清洗（Masking and Sanitization）。其中最关键的原则是消除**循环推论（Circular Reasoning）**风险——必须从输入特征中严格剔除待预测的目标标签本身，避免模型自欺欺人。清洗完成后，流量首先通过确定性规则引擎（Deterministic Rules Engine）进行初筛；仅在确定性规则无法匹配覆盖时，流量才会兜底降级至大语言模型（LLM）进行深度推理。无论走规则还是走大模型，系统都会规范输出标准化的分类类别、置信度分数以及可追溯的决策路径，并将其作为带版本号的证据存入存储层，供后续重放与审计验证。
```mermaid
flowchart TD
accTitle: 端到端资产分类流水线架构
accDescr: 从事件触发、上下文抓取与清洗脱敏，到规则优先、LLM 兜底的双轨执行路径，最终输出带审计证据的决策结果。
subgraph Ingestion["输入与上下文准备"]
T["变更触发事件<br/>(Schema / 代码 / 配置变更)"]
C["多源上下文聚合<br/>(血缘 / 归属 / 特征 / 代码搜索)"]
M["数据脱敏与掩蔽<br/>(消除循环推论与泄露)"]
T --> C --> M
end
subgraph Engine["混合决策引擎"]
R{"确定性规则匹配<br/>(Deterministic Rules)"}
L["大模型推理兜底<br/>(LLM Fallback)"]
M --> R
R -- "未命中规则" --> L
end
subgraph Output["标准输出与归档"]
O["统一输出结构<br/>(类别 + 置信度 + 决策路径)"]
A[("版本化审计证据存储<br/>(供审计与回放)")]
R -- "命中规则" --> O
L --> O
O --> A
end
```
整个架构的本质任务，就是将输入端充斥着歧义命名、碎片化血缘、人员频繁更迭的归属元数据以及误报频出的扫描器信号，收敛凝聚为右侧清晰、稳健的决策输出。系统必须在长期的迭代中辨别出哪些信号具备高度统计可靠性，将稳定的指示特征与无意义的随机噪声彻底剥离。
### [07:01](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=421s) 挑战一：噪声信号与结构化证据摘要（Evidence Briefs）
在资产分类落地过程中，团队遭遇了三大工程挑战，首当其冲的便是输入信号的严重噪声。资产周边的上下文极其繁杂，包含代码引用、表结构声明、变量命名、上下游数据采样等数十种来源。如果直接将这些未经筛选的原始文本囫囵吞枣地塞进大模型，模型在每一次推理调用中都必须从头摸索哪些线索才是核心关键。
这种无序输入带来了三大负面后果：
1. **Token 消耗暴增：** 上下文急剧膨胀，计算成本成倍飙升。
2. **注意力被过度稀释：** 模型在海量背景噪声中迷失，决策边界变得模糊不清。
3. **循环推论暗度陈仓：** 带有误导性甚至包含答案倒推的循环字段与真实有效特征杂糅在一起，模型自身根本无法准确识别和剔除。
为了破解这一困局，团队摒弃了原始上下文倾倒的做法，设计了**结构化证据摘要（Evidence Briefs）**机制。工程团队为每个字段预先定义了特征策略：标记其属于高价值有用字段、特定场景下有条件的有用字段、因包含答案倒推必须前置屏蔽的循环字段，还是历史上经常引发误判需要显式阻断的毒性特征。在证据摘要中，系统会结构化提炼出得分最高的“正面支持信号”与“反面矛盾信号”，形成精炼聚焦的事实摘要送交决策引擎。正如 Rituraj 所强调的那样，"The model no longer has to rediscover what matters. We tell it."（模型不必再去费力重新摸索重点，我们直接明确告诉它）。
### [08:39](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=519s) 挑战二：数据漂移与基于 Cohen's Kappa 的三判官评估机制
第二大挑战是严重的分布漂移与数据易变性（Drift and Variability）。资产分类所依赖的基准标签本身伴随着业务逻辑持续发生变动，每一批待处理的数据分布都可能存在剧烈波动。当评估框架在一个持续位移的真实基准值（Ground Truth）上做优化时，优化器本质上是在追逐一个移动靶标，模型自以为指标上升，实则只是拟合了数据漂移，极易导致严重的回归缺陷被悄悄带入生产环境。
为了构筑坚固的评估防线，团队引入了基于批次采样的多智能体评估体系：
- **三判官提示工程面板（Three LLM Judges）：** 针对抽样资产，系统采用 3 个不同 Prompt 视角的 LLM 判官独立评估，包括“直接分类”（Direct Classification）、“先批判再分类”（Critique then Classify）以及“仅依据元数据分类”（Metadata-only）。选择 3 个判官是因为这是能形成有效简单多数投票机制（Majority Vote）的最小奇数规模，无需额外设计复杂的平局仲裁逻辑。这三个判官共享一个独立的裁判模型，该裁判模型在架构上刻意与用于线上分类的主模型不同，确保了推理视角的客观性与多样性。
- **Cohen's Kappa 一致性度量：** 系统通过批次级别的 Cohen's Kappa 系数计算判官间的一致性。当判官意见出现分歧时，该样本会流转至人工审查关口（Human Review Gate）进行仲裁，人工核定后的结果才会更新进入基准参考集；当三个判官完全一致时，样本会被标记为参考集候选，并在冻结期结束后统一晋级。
- **坦率的系统局限与防线：** Rituraj 坦诚指出，如果三个判官全部判断一致却同时犯了错，系统在单个资产颗粒度上确实无法拦截；但这种系统性偏差必然会在宏观统计层面表现为 Cohen's Kappa 的异常漂移，从而及时触发报警。
### [10:44](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=644s) 挑战三：千万级资产规模与大模型蒸馏规则引擎
第三大挑战是系统吞吐量与成本的现实天花板。生产环境中存在数以千万计的海量数据资产，大模型的单次调用延迟通常以秒为单位计算，不仅严重消耗 Token 算力，其概率输出的特质还导致结果难以严格复现。如果全量资产无差别调用大模型，不仅会导致耗费极长的物理运行时间，引发天文数字级的算力账单，还会面临内审团队最直白的灵魂拷问："Why did it behave differently when I tested it?"（为什么我测试的时候系统行为变了？）。
合规系统在绝大多数高频常规场景下，必须提供毫秒级的极低延迟与绝对的确定性结果，只有真正复杂的边缘长尾案例才应该交给大模型处理。面对这一矛盾，工程团队的核心策略是**"use LLMs to put LLMs out of business"**（用大模型来革大模型的命）——通过持续的知识蒸馏，将大模型的推理能力沉淀转化为确定性的代码规则。
演讲在蒸馏闭环的前两个步骤处切入展开：
1. **寻找高置信模式（Find Stable Patterns）：** 观察大模型在做出高置信度决策时展现出的稳定行为模式。如果模型在特定上下文组合下持续且高度一致地将某类资产划归为特定类别，底层往往存在某种清晰的结构化业务规律。
2. **提出精准规则（Propose an Exact Rule）：** 进而由系统自动提炼并建议精确可审计的规则定义，为消除大模型调用开辟路径。
### [12:02](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=722s) 规则蒸馏五步法与边界分类收敛
将大语言模型（LLM）在高难度样本中习得的分类能力固化为确定性系统，依赖于严密的规则蒸馏机制。该流程包含明确的准入与晋升阶段：从候选模式中提取出正则表达式、启发式逻辑或特定字段值匹配规则后，系统会立即在独立的保留集（holdout set）上进行离线验证。如果提议的规则未能达到预设的精确率（precision）和样本支持度（support）阈值，它就会被直接列入黑名单，彻底阻断其再次被评估的可能，防止低质逻辑反复消耗算力。
通过保留集检验的规则并不会直接接管流量，而是进入“影子模式”（shadow mode）。在此模式下，规则在生产环境中仅以日志形式伴随运行，系统实时比对规则输出与 LLM 预测结果的一致性。唯有当影子模式充分证实该规则在真实流量中保持稳定且高精度时，它才会被正式晋升（promote）并写入可版本化管理的 `rules.yaml`。每一次新规则的晋升，都在实质上削减 LLM 的运行时调用表面积。最终系统的稳态呈现出高度分流的格局：绝大多数通用常见模式直接以毫秒级延迟在规则引擎中运行，实现零 LLM 依赖；LLM 仅作为回退兜底，专门处理极少数真正模糊复杂的长尾案例，保证了系统完全的确定性、可重放性（replayable）与审计透明度。
```mermaid
flowchart TD
accTitle: 规则蒸馏与执行流水线
accDescr: 展示从模式提取、保留集验证、影子模式比对到最终晋升为确定性规则并回退至LLM的完整闭环。
subgraph Distill ["规则蒸馏闭环"]
A["候选模式提取<br/>(正则/启发式/字段匹配)"] --> B{"保留集校验<br/>(精确度 & 支持度)"}
B -- "未达标" --> C["列入黑名单<br/>(Blocklist)"]
B -- "达标" --> D["生产影子模式<br/>(Shadow Mode 记录)"]
D --> E{"稳定性校验<br/>(比对 LLM 输出)"}
E -- "稳定" --> F["晋升至 rules.yaml"]
end
subgraph Runtime ["运行时执行"]
G["资产元数据输入"] --> H{"匹配 rules.yaml?"}
H -- "命中规则" --> I["确定性执行<br/>(毫秒延迟 / 零 LLM)"]
H -- "未命中 / 歧义" --> J["LLM 兜底推理<br/>(复杂长尾案例)"]
end
F -.-> H
```
这种蒸馏机制与 LLM 兜底相结合的“边界分类”（boundary classification）策略，对整体系统拓扑产生了立竿见影的精简效果。演讲者回顾了前文展示的底层数据流图（data flow graph）：对于同一个源资产，在引入边界分类逻辑之后，下游需要展开深度合规分析的节点数量出现了断崖式下跌。这一优化大幅缩减了问题暴露面，使得隐私感知基础设施（PAI, Privacy-Aware Infrastructure）真正具备了横向扩展能力，足以应对 Meta 最严苛、最错综复杂的隐私合规挑战。
- 保留集评估标准：未同时满足精确率与支持度双重阈值的规则会被永久拉黑。
- 最终目标：大模型仅处理“真正具有歧义的个案（genuinely ambiguous cases）”，常见流量实现纯确定性代码执行。
- 运行收益：毫秒级延迟响应、完全可重放、支持合规审计，且彻底摆脱对高昂模型调用的实时依赖。
---
### [13:26](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=806s) 踩坑复盘：构建系统过程中遭遇的三大盲区
在系统从设计走向落地的过程中，工程团队踩过了不少坑，其中最典型的三大失败尝试深刻重塑了团队的认知。第一个教训发生在数据脱敏（masking）的设计边界上。最初，团队简单地将脱敏视为一种“Prompt 内部事务”，为了防止 LLM 产生循环论证或自指推理，在喂给模型的 Prompt 中对循环依赖字段进行了遮掩屏蔽；然而，上游的规则引擎却依然在包含未脱敏字段的原始上下文中运行。由于规则与大模型观察到的上下文不一致，匹配规则的输出与 LLM 的推理决策始终无法对齐，导致规则引擎的覆盖率很快遭遇瓶颈。团队随后将脱敏重构为一项“系统级不变量（system-wide invariant）”，强制要求规则引擎与 LLM 必须在完全相同的脱敏上下文中进行评估，规则覆盖率随即迎来跃升，模型调用量同步骤降。
第二个教训是“脱离上下文的 Prompt 优化陷阱”。团队曾搭建起一套专用的自动化优化流水线，引入了业界前沿的提示词优化算法，投入了海量算力与漫长的时间进行多轮迭代，但分类器的准确率却在达到某个上限后彻底停滞。深入排查后团队才发现，瓶颈根本不在提示词的措辞上，而是输入模型的基础数据本身缺乏足够的结构化表达与清晰度，使得大模型根本无法建立有效的推理链条。团队随后转变重心，转而提升上下文质量并引入更精准的代码语义解析（code resolution）；在模型未变、初始 Prompt 完全相同的情况下，分类准确度获得了大幅突破。演讲者为此给出忠告：“如果你的分类器性能停滞不前，在去动 Prompt 之前，务必先投资优化输入的上下文。”
第三个教训则是“振荡检测机制的缺失”。在早期的自动化提示词调优循环中，由于缺乏收敛监控，调优算法在两个互有优劣的候选 Prompt 之间反复横跳（oscillation），盲目迭代了数十次而未能收敛，在无人察觉的情况下白白烧掉了巨额算力。针对这一教训，团队引入了基于 Cohen's Kappa 一致性系数的调优控制器，将振荡检测直接内建为架构的核心基础能力，而非事后修补的附加插件。
- 掩码机制对齐：掩码不能只存在于 LLM 的输入端，必须作为系统级约束贯穿规则与推理全局。
- 上下文投资回报：结构化上下文与具体代码解析带来的收益，远超在劣质数据上打磨 Prompt 措辞的微小边际增益。
- 调优控制器：引入 Kappa 一致性检测算法作为架构断路器，彻底阻断 Prompt 搜索时的空转振荡与算力浪费。
---
### [15:48](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=948s) 伤疤背后的三大架构原则
上述踩坑经历所留下的“工程伤疤”，凝结为了指引 AI-Native 基础设施建设的三大核心原则：
首先，**上下文重于 Prompt（Context > Prompts）**。团队以大量算力浪费换来的惨痛教训证明，对提问方式修修补补只能换来微不足道的提升；真正实现准确率质的飞跃的，是通过“证据简报（evidence briefs）”对输入上下文进行的严密结构化改造。“如果本场演讲你只能带走一件事，那就是在优化你提问的方式之前，先全力投资那些输入给模型的内容。”
其次，**解耦评估体系与优化流程（Decouple eval from optimization）**。评估框架与评审委员会（eval panel）必须同受训的分类器保持物理级隔离：二者必须采用互不相同的底层模型、互不相同的 Prompts，并且基于完全冻结的基准参考集（frozen reference sets）运行。如果评估机制与参数调优混杂在同一个闭环内，系统最终测得的根本不是真实的优化进展，而仅仅是指标的自我漂移（measuring drift, not progress）。
最后，**向确定性逻辑持续蒸馏（Distill to deterministic）**。演讲者强调：“LLM 在学习归纳方面表现出色，但直接拿来承载规模化吞吐却非常糟糕（LLMs are great for learning, but they are terrible at scale）。”一个健壮的架构应该促使系统具备“自我净化”能力：每一个被 LLM 验证稳定的模式，都必须提炼为具备版本控制、可严密审计的确定性规则，从而主动且持续地压缩大语言模型在整个生产链路中的调用暴露面。
| 架构原则 | 核心实践要求 | 规避的反模式 |
|:--- |:--- |:--- |
| **上下文重于 Prompt** | 引入结构化证据简报（evidence briefs）与高解析度代码语义 | 仅靠调整 Prompt 措辞试图弥补输入数据质量的不足 |
| **评估与优化解耦** | 采用独立模型、独立 Prompt 及完全冻结的参考基准集 | 评测与优化共用同一逻辑环路，把分布漂移误当能力提升 |
| **向确定性逻辑蒸馏** | 将稳定推断转化为版本化的 `rules.yaml`，追求毫秒级零 LLM 执行 | 在生产高频链路上无节制全量调用昂贵且非确定的 LLM |
---
### [16:54](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=1014s) 未来演进与总结：从隐私合规走向通用 Agent 监管
立足当前成果，团队勾勒了下一阶段的三大演进路线。首先是存量资产改造：逐步将遗留的各类旧版分类器全面迁入这套新系统，彻底淘汰过去那些脆弱死板、由人工硬编码堆砌的旧逻辑，代之以“上下文优先 + 自动蒸馏”的现代化链路。其次是业务域扩展：将这套边界分类与评估闭环横向推广至 PAI（隐私感知基础设施）的其他核心领域，涵盖数据血缘自动化改进（lineage improvement）、安全执行策略的分级灰度发布（safe enforcement rollout）以及合规审计证据的自动生成。
第三个方向则是演讲者最富激情的展望——将这套方法论外推至隐私治理之外的广阔场景。随着 AI Agent 的全面落地，**智能体的可观测性与合规监管（Agent observability and oversight）** 正面临着完全同构的技术痛点：如何在充斥着海量噪声的环境中捕获微弱信号？如何在动态变幻的行为中落地确定性的边界管控？又如何构建能够支持合规回溯的重放审计机制？这一系列挑战都能从这套蒸馏架构中直接汲取解法。
在演讲尾声，Rituraj Kirti 给出了对隐私工程的终极定义：“隐私感知基础设施绝非施加于工程团队之上的阻碍税负，相反，它是倒逼系统迈向更优架构的核心驱动力。”它强迫系统建立更清晰的数据契约（data contracts）、提供更丰富的上下文支撑、引入更严密的评测机制，并推行更有纪律的变更管理。在 AI 原生时代，技术团队应当最大化借助大模型强大的归纳学习能力迅速探索未知，同时坚持将沉淀下来的知识转化为确定、高效、可重放的规则去执行底层约束。这一整套经受了海量噪声考验、能够抵御抖动漂移并持续自愈净化的系统工程模式，为大规模 AI 原生应用落地提供了极具参考价值的工业样本。
## Notable quotes
> [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) Privacy as a driving force for better architecture — Rituraj Kirti
> [00:05](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=5s) AI helps humans — Rituraj Kirti
> [07:01](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=421s) The model no longer has to rediscover what matters. We tell it. — Rituraj Kirti
> [10:44](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=644s) Why did it behave differently when I tested it?
> [10:44](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=644s) use LLMs to put LLMs out of business — Rituraj Kirti
> [13:26](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=806s) 如果你的分类器性能停滞不前，在去动 Prompt 之前，务必先投资优化输入的上下文。 — Rituraj Kirti
> [15:48](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=948s) 如果本场演讲你只能带走一件事，那就是在优化你提问的方式之前，先全力投资那些输入给模型的内容。 — Rituraj Kirti
> [15:48](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=948s) measuring drift, not progress — Rituraj Kirti
> [15:48](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=948s) LLMs are great for learning, but they are terrible at scale — Rituraj Kirti
> [16:54](https://www.youtube.com/watch?v=q9RSv2ltZtU&t=1014s) 隐私感知基础设施绝非施加于工程团队之上的阻碍税负，相反，它是倒逼系统迈向更优架构的核心驱动力。 — Rituraj Kirti
## Mentioned
Rituraj Kirti, Meta, @Scale, Hive, GenAI, LLM, Cohen's Kappa, rules.yaml, PAI (Privacy-Aware Infrastructure), AI Agent
## Video description
Building privacy-aware infrastructure in the AI-native era requires systems that can learn, evaluate, and operationalize improvements continuously—at the scale of millions of data assets. This talk presents an AI-native approach through a case study of AI-native asset classification: a hybrid deterministic rules engine with LLM fallback that transforms messy context (metadata, lineage, code references, scan signals) into enforceable privacy controls.
We’ll highlight ground truth (GT) + evaluation (eval) self-improvement loops, where active learning focuses human review on high-value examples, an LLM-as-Judge provides scalable review signal (including kappa reliability), and evaluation gates prevent regressions and circular reinforcement.
Finally, we’ll show how high-performing LLM behavior is distilled into auditable rules.yaml, driving LLM usage toward near-zero while improving determinism, latency, and cost. Attendees will leave with practical patterns for building privacy-by-design infrastructure that converges, exports human-readable logic, and monitors drift in production.
Learn more about the @Scale conferences here: atscaleconference.com
[Source](https://www.youtube.com/watch?v=q9RSv2ltZtU)
Source quotes
[0:05] Rituraj Kirti: Privacy as a driving force for better architecture
[0:05] Rituraj Kirti: AI helps humans
[7:01] Rituraj Kirti: The model no longer has to rediscover what matters. We tell it.
[10:44] Why did it behave differently when I tested it?
[10:44] Rituraj Kirti: use LLMs to put LLMs out of business
[13:26] Rituraj Kirti: 如果你的分类器性能停滞不前，在去动 Prompt 之前，务必先投资优化输入的上下文。
[15:48] Rituraj Kirti: 如果本场演讲你只能带走一件事，那就是在优化你提问的方式之前，先全力投资那些输入给模型的内容。
[15:48] Rituraj Kirti: measuring drift, not progress
[15:48] Rituraj Kirti: LLMs are great for learning, but they are terrible at scale
[16:54] Rituraj Kirti: 隐私感知基础设施绝非施加于工程团队之上的阻碍税负，相反，它是倒逼系统迈向更优架构的核心驱动力。
Source quotes
[0:05] Amrit Raj: privacy is not merely a tax on engineering, a burden of process, or a layer of compliance red tape.
[0:05] Amrit Raj: Privacy as a driving force for better architecture.
[0:00] Rituraj Kirti: Privacy-by-design infrastructure that converges, exports human-readable logic, and monitors drift in production.
[10:44] Amrit Raj: Use LLMs to put LLMs out of business.
[15:48] Amrit Raj: Context > Prompts
[15:48] Amrit Raj: Decouple Eval from Optimization
[15:48] Amrit Raj: Distill to Deterministic
Original transcript
[0:05] Welcome everyone. I'm Amrit Raj.
[0:08] Today we are going to talk about privacy
[0:10] aware infrastructure in the AI native
[0:12] era.
[0:13] But I want to start with a thesis that
[0:16] might seem a bit counterintuitive at
[0:18] first.
[0:19] Privacy as a driving force for better
[0:22] architecture.
[0:24] Building for privacy doesn't have to be
[0:26] just about process or compliance
[0:29] or adding red tape.
[0:31] In the AI native era,
[0:33] it fundamentally enables us
[0:35] to build more adaptable, more
[0:38] deterministic, and highly reliable
[0:40] systems.
[0:42] Let's take a look at how we did that.
[0:45] Quick context on how we got here. In
[0:47] 2025, we shared about how we are
[0:50] accelerating privacy workflows using
[0:53] GenAI.
[0:55] GenAI was used to help humans review
[0:58] faster
[0:59] with ground truth and continuous
[1:01] validation.
[1:03] Then we published a blog on scaling
[1:05] privacy infrastructure for rapid
[1:08] GenAI-driven product iteration.
[1:11] Today, we take the next step.
[1:14] We're going from AI helps humans to AI
[1:17] native infrastructure that produces
[1:20] robust decisions on its own
[1:23] for replayable enforcement.
[1:26] Asset classification is the concrete
[1:28] example we will use for the case study
[1:30] today.
[1:33] First, let's do a quick primer on what
[1:36] privacy aware infrastructure is and
[1:38] what's our recipe for purpose
[1:40] limitation.
[1:41] This recipe has four key workflows.
[1:45] Understand.
[1:47] This allows us to figure out where is
[1:48] the data that we need to protect.
[1:52] Discover.
[1:53] Once we know what data we want to
[1:54] protect, we also need to understand
[1:57] where the data goes from there or what
[2:00] was the origin of the data that we are
[2:02] protecting.
[2:03] Enforce.
[2:04] Once we know the complete graph of data
[2:07] that we need to protect, how do we
[2:08] actually prevent regressions for our
[2:10] privacy requirements?
[2:13] And finally, demonstrate.
[2:15] What is our system for ensuring that the
[2:17] system continues to operate and we keep
[2:19] meeting the privacy requirements and
[2:21] provide an audit trail for that.
[2:24] In today's talk, we are focused squarely
[2:27] on the very first workflow here,
[2:29] understand. Specifically, how we
[2:32] classify different assets.
[2:35] So that downstream enforcement and
[2:37] demonstration can be correct and
[2:39] reliable.
[2:42] Let's ground the problem.
[2:44] Here is a visualization of a data flow
[2:47] graph of 100,000 nodes downstream from a
[2:50] single hive table column. Each node here
[2:55] presents either a table column, log
[2:57] field, ML feature, pipeline stage, and
[3:00] that carries carries data derived from
[3:02] the source.
[3:03] This picture is at the limits of human
[3:06] visual comprehension.
[3:08] But it's a small subset of the problem
[3:10] scale that we are working at.
[3:13] Zooming in, you can actually see the
[3:15] nodes.
[3:16] Now, if you shrink this whole graph into
[3:18] what each of these nodes is, that gives
[3:20] us the full scale of what we need to
[3:23] protect.
[3:26] What are we classifying? I talked about
[3:28] assets. What is an asset? An asset is
[3:31] anything that carries data. It could be
[3:33] a database column. It could be a field
[3:35] that gets logged. It could be an ML
[3:37] feature.
[3:39] AI native products also bring new
[3:41] modalities. We have audio, we have
[3:43] vision, we have embeddings, we have
[3:45] sensor data, we have feedback loops.
[3:48] These now ship faster than manual review
[3:51] can follow.
[3:53] Classification is how we keep up.
[3:56] The stakes are high.
[3:57] On the slide, you will see why it
[3:59] matters.
[4:00] Wrong label could mean incorrect
[4:02] enforcement. You could block the wrong
[4:05] data flow and break a product feature
[4:08] or permit what you shouldn't be doing.
[4:11] Missing a category means you're missing
[4:13] the privacy requirement.
[4:15] And no trace means no auditability. You
[4:18] cannot demonstrate compliance.
[4:20] Classification sits at the foundation of
[4:23] these privacy workflows. Get it wrong
[4:25] and everything downstream can break.
[4:29] Here's what our end-to-end
[4:30] classification pipeline looks like.
[4:34] A trigger, a schema change, code change,
[4:36] configuration change kicks off the
[4:38] classification run.
[4:40] We fetch context from multiple different
[4:42] sources, lineage graphs, ownership
[4:45] metadata, annotations, ML features, code
[4:48] search results, and so on.
[4:51] Before anything reaches the classifier,
[4:53] we apply masking and sanitization. For
[4:56] example, to remove information that
[4:58] could cause circular reasoning.
[5:00] One key thing is we want to make sure
[5:02] the label we are trying to predict is
[5:03] not present in the data that we are
[5:05] trying to predict it from.
[5:07] More on that shortly. First,
[5:09] deterministic rules get the first pass
[5:11] at classification. And whatever these
[5:13] rules cannot handle falls to the
[5:16] element.
[5:17] Both paths produce the same output.
[5:22] This output has a category confidence
[5:24] score and the decision path that was
[5:26] used.
[5:27] All of this is stored
[5:29] as versioned evidence for replay and for
[5:31] audit.
[5:33] Here's the core problem
[5:35] framed more simply.
[5:37] On the left, we have noisy sync signals.
[5:40] Names can be ambiguous. Lineage is often
[5:43] partial.
[5:44] Ownership can change over time.
[5:47] Scanners can produce false positives.
[5:50] On the right, what we want is a crisp
[5:52] output that leads to a robust decision
[5:55] with a category confidence score and a
[5:58] trace that tells us exactly how this
[6:00] decision was arrived.
[6:02] The system's job is to combine these
[6:05] individually weak signals into robust
[6:07] decisions
[6:09] and track which signals are reliably
[6:10] informative across iterations,
[6:13] separating stable indicators from noise.
[6:16] With that introduction out of the way,
[6:18] let's take a look at what we're going to
[6:19] talk about today.
[6:21] We will review three key challenges,
[6:24] our key learnings, and next steps for
[6:26] the future.
[6:28] First challenge,
[6:30] signals can be incredibly noisy.
[6:33] Second,
[6:34] there is severe drift and variability on
[6:36] how data changes day-to-day and
[6:38] batch-to-batch.
[6:40] Third, the sheer scale.
[6:42] You cannot just throw raw LLMs at
[6:43] everything.
[6:45] The compute will ruin you and the
[6:47] latency will break your SLAs.
[6:50] And finally, we're going to take a look
[6:52] at what were our learnings from the
[6:54] process and some of the future next
[6:56] steps.
[6:57] Let's take a look at them one by one.
[7:01] Challenge number one,
[7:02] we fetch
[7:04] dozens of different types of context,
[7:06] lineage, code references, schema
[7:09] definitions, names, samples of data.
[7:12] If we dump all of that into the model,
[7:15] it has to rediscover what matters every
[7:18] single call.
[7:19] Token usage balloons, attention dilutes
[7:22] across noise, and worst of all, decision
[7:25] boundaries get blurred, misleading or
[7:28] circular fields slip right alongside
[7:30] useful ones.
[7:32] The model cannot tell these apart
[7:33] without help.
[7:35] So, what's the solution?
[7:37] Instead of dumping all that raw
[7:38] information to the decision engine,
[7:41] we produce structured evidence briefs,
[7:44] that is ranked signal summaries.
[7:46] On the left, you will see the various
[7:48] context sources,
[7:50] lineage, ownership, annotations, ML
[7:52] features, code usage,
[7:54] embeddings.
[7:56] Every field gets a policy,
[7:58] whether this field was useful, whether
[8:01] it is conditionally useful sometimes,
[8:03] whether it's a circular field and should
[8:05] be masked,
[8:06] whether there is something that
[8:08] mispredicts and it should be blocked.
[8:11] Circular information is masked before it
[8:13] reaches the decision engine.
[8:15] We will see why that matters again in
[8:17] the learning section.
[8:19] The evidence brief surfaces top
[8:21] supporting signals, top contradicting
[8:24] signals, and not a raw context dump.
[8:28] This gives the decision engine focused
[8:30] high-quality input.
[8:32] The model no longer has to rediscover
[8:35] what matters. We tell it.
[8:39] Challenge number two.
[8:41] Drift and variability.
[8:43] Our iteration metrics rely on labels
[8:46] that are noisy and shift over time.
[8:49] Each batch that we are classifying can
[8:51] have a lot of variation.
[8:53] And over time, the system changes.
[8:57] When we optimize against shifting ground
[8:59] truth, we are tracking drift, not real
[9:01] progress.
[9:02] The optimizer thinks it's improving, but
[9:05] it's chasing a moving target.
[9:07] Worst case, regressions ship to
[9:09] production because the eval site no
[9:11] longer reflects reality.
[9:15] How do we handle it?
[9:17] We set up a system to perform evals
[9:21] on samples from every batch.
[9:24] In this eval,
[9:26] we have multiple, in this case three LLM
[9:28] judges, evaluate each asset with
[9:31] different framings, classify directly,
[9:34] critique then classify, and metadata and
[9:36] only.
[9:37] Why three? That's the smallest odd
[9:39] number panel for a
[9:41] real majority vote, no extra
[9:43] tie-breaking knowledge needed.
[9:46] All three share a judge model that is
[9:48] deliberately different from the original
[9:51] classifier model.
[9:53] This allows us diversity from the prompt
[9:55] framing and does not rely on different
[9:57] model architectures.
[10:00] We check agreement between these judges
[10:03] using Cohen's Kappa for a batch.
[10:05] When judges disagree, top path,
[10:08] it goes to a human review gate.
[10:11] Humans review and break ties, and their
[10:13] decisions update the ground truth
[10:15] reference set.
[10:16] When judges agree, bottom path, we mark
[10:19] it for potential promotion to the
[10:21] reference set, which stays frozen until
[10:24] the next cycle.
[10:25] The honest limit. If all three judges
[10:29] agree, but are wrong, we won't find it
[10:32] on that particular asset. But we can
[10:35] catch it level through Kappa drift.
[10:38] Only reviewed outcomes are ever used to
[10:41] update the reference set.
[10:44] Challenge number three.
[10:46] We have many millions and millions of
[10:48] assets.
[10:49] An LLM can call can take seconds per
[10:52] call,
[10:53] consumes tokens, produces probabilistic
[10:56] outputs
[10:57] that are hard to reproduce exactly.
[11:00] If we classify everything with LLMs, we
[11:02] are looking at hours of wall clock time,
[11:04] massive compute bills,
[11:06] and audit teams asking,
[11:08] "Why did it behave differently when I
[11:11] tested it?"
[11:13] We need millisecond latency,
[11:15] deterministic decisions for the common
[11:17] cases, and LLMs only for the genuinely
[11:20] challenging hard ones.
[11:22] That's the scaling challenge.
[11:25] So, how do we solve it?
[11:27] Our approach is to use LLMs to put LLMs
[11:31] out of business.
[11:33] That is, use LLMs to generate the
[11:35] deterministic rules, what we call
[11:37] distillation.
[11:38] Here's the distillation loop in five
[11:40] steps.
[11:41] First,
[11:43] look for stable patterns in confident
[11:45] LLM decisions.
[11:47] If the LLM is consistently classifying
[11:50] certain assets in a particular way,
[11:52] there is a high likelihood that there is
[11:55] some structural rule underneath.
[11:58] Second, propose an exact rule,
[12:02] a regular expression, a heuristic, or
[12:04] field value match.
[12:07] Third, validate the proposed rule
[12:10] against a holdout set.
[12:12] If it does not meet the precision and
[12:13] support thresholds,
[12:15] we
[12:16] block list it from being considered
[12:19] again.
[12:21] Fourth,
[12:22] shadow mode.
[12:24] The rule runs in production
[12:26] in just logging mode, and we compare the
[12:29] outputs against the LLM's prediction.
[12:32] Fifth,
[12:34] promote it if the shadow mode confirms
[12:36] the rule is stable.
[12:38] It graduates to our rules.m. Every
[12:40] promoted rule shrinks the LLM surface
[12:43] area.
[12:44] The end state, LLMs handle only
[12:47] genuinely ambiguous cases. Common
[12:50] patterns run at millisecond latency with
[12:52] zero LLM dependency.
[12:54] Fully deterministic and replayable and
[12:57] auditable.
[12:59] What is the result of all this? Remember
[13:02] the data flow graph from earlier?
[13:04] After applying boundary classification
[13:07] using the distilled rules and LLM
[13:09] fallback we just discussed, the number
[13:11] of nodes that require analysis drops
[13:14] dramatically for the same source asset.
[13:18] That's a massive reduction in problem
[13:19] surface area, and it means that PAI
[13:22] scales for our most complex privacy
[13:24] requirements.
[13:26] Now that we have gone over the
[13:28] challenges and seen how the system
[13:30] works, let's review some of the things
[13:32] that did not work or did not work as
[13:35] expected.
[13:38] Number one,
[13:39] we treated masking as a prompt concern.
[13:43] We masked circular fields from the LLM
[13:45] to prevent self-referential reasoning.
[13:48] Good.
[13:49] But our rules were processing unmasked
[13:51] fields
[13:53] instead of evaluating unmasked context.
[13:56] Rule coverage plateaued because matching
[13:58] rules and LLM decisions did not track.
[14:01] The fix,
[14:03] masking is a system-wide invariant.
[14:05] Rules and LLMs now both evaluate on the
[14:08] same masked context.
[14:11] Rule coverage jumped.
[14:13] LLM usage dropped.
[14:15] Learning number two,
[14:17] prompt optimization without context.
[14:20] This was very surprising.
[14:22] We ran the dedicated optimization
[14:24] pipeline with state-of-the-art prompt
[14:26] optimizer algorithms.
[14:28] Many, many hours and iterations later,
[14:31] we could not improve the accuracy beyond
[14:33] the ceiling.
[14:35] Then we investigated
[14:37] and figured out what was going on.
[14:40] It turned out the data that was going
[14:42] into the LLM
[14:44] was not good enough or structured enough
[14:46] for it to make a better reasoning
[14:48] decision.
[14:50] Investing in improving the quality of
[14:52] the context
[14:54] and giving it more specific code
[14:55] resolution
[14:57] helped us break beyond that barrier.
[15:00] Same model, same prompt
[15:02] to start with, and our classification
[15:04] accuracy improved significantly.
[15:07] The lesson, if your classifier seems to
[15:09] have stalled,
[15:11] invest in better context before touching
[15:13] the prompt.
[15:16] Learning number three, no oscillation
[15:18] detection.
[15:20] Our optimization loop bounced between
[15:23] two candidate prompts for dozens of
[15:26] iterations, never converging.
[15:28] Burning through significant compute
[15:31] before anyone noticed.
[15:34] The fix.
[15:36] Kappa based tuning controller that
[15:38] detects oscillation. It's now an
[15:40] architectural feature, not a bolt on.
[15:43] These are the really the scars behind
[15:44] the principles on the next slide.
[15:48] Number one, context is greater prompts.
[15:52] We proved this the hard way, burning
[15:54] through hours of optimization to give it
[15:56] that gave us marginal gains.
[15:58] Structuring context with evidence briefs
[16:01] gave us the big accuracy improvement.
[16:04] If you take one thing away from this
[16:06] talk, invest in what goes into the model
[16:08] before you optimize how you ask.
[16:12] Principle number two, decouple eval from
[16:15] optimization.
[16:16] Your eval setup and panel should be
[16:18] independent of your classifier.
[16:21] Different model, different prompts,
[16:23] frozen reference sets.
[16:24] If eval and optimization share the same
[16:27] loop, you're measuring drift, not
[16:29] progress. This is necessary hygiene.
[16:32] Number three, distill to deterministic.
[16:35] LLMs are great for learning,
[16:38] but they are terrible at scale.
[16:40] Every stable pattern should become a
[16:43] versioned auditable rule that runs
[16:45] without any LLM.
[16:47] The system should progressively shrink
[16:49] its own LLM surface area.
[16:53] So, what's next?
[16:55] Here are some future directions.
[16:58] First, migrating our list legacy
[17:00] classifiers into the system. Replacing
[17:03] the brittle
[17:04] hand-coded logic into context first
[17:07] distillation based approach.
[17:10] Second, expanding beyond classification
[17:13] to the other PI workflows we discussed
[17:15] earlier. Lineage improvement, safe
[17:18] enforcement rollout, and evidence
[17:20] generation, for example.
[17:22] Number three,
[17:24] and this is the one I'm most excited
[17:25] about. Techniques that we discussed
[17:28] apply way beyond privacy.
[17:31] Agent observability and oversight face
[17:34] the same challenge.
[17:35] Noisy signals, need for deterministic
[17:38] enforcement, need for replayable
[17:40] auditability.
[17:41] If you're working in this space, I would
[17:43] love to connect.
[17:46] I'll leave you with this.
[17:48] Privacy-aware infrastructure isn't a tax
[17:50] on engineering, it's a driving force for
[17:52] better architecture.
[17:54] Clearer data contracts, richer context,
[17:56] rigorous evaluation.
[17:58] Disciplined change management.
[18:01] The AI-native opportunity and era allows
[18:04] us to learn quickly with LLMs while
[18:06] enforcing with deterministic replayable
[18:09] logic where possible.
[18:11] The system we built handles noisy
[18:13] signals, iterates safely in spite of
[18:16] variation, and distills stable patterns
[18:19] into efficient rules.
[18:21] If you're working on classification,
[18:23] lineage enforcement, or agent oversight,
[18:26] come find me.
[18:28] Thank you for listening.
