Ricardo Signes Urges Engineers to Halt Custom Email Parsers

Building compliant mail software is fundamentally impossible because obsolete RFC standards conflict with widespread real-world client bugs, forcing parsers to tolerate malformed inputs to survive.

AI-assisted summary of Email Hates the Living — original source. Check the original for full context.

Source published: 2012-05-06

Key ideas

Email protocol specifications and real-world client implementations systematically diverge, making completely bug-free and standards-compliant email handling software impossible to implement. (0:04)
Spammers routinely bypass email filters not by breaking standards, but by constructing strictly valid messages using obscure RFC edge cases such as comments in date fields and silent discarding of non-Base64 bytes. (2:23)
Header folding rules (dating back to RFC 724) permanently destroy consecutive whitespace sequences during line unwrapping and create major truncation hazards in poorly implemented intermediate mail handlers. (4:20)
Syntax-only email address validation is fundamentally ineffective; robust validation requires pragmatic multi-tiered checks, including live DNS MX record lookups and contextual abuse screening. (45:57)

Detailed notes

## Background This presentation was delivered by Ricardo Signes (rjbs) on June 16, 2008, and uploaded to YouTube by channel m on May 6, 2012. According to the page description, the talk covers the eldritch horrors, devilish ambiguities, and historical cruft buried within internet email protocols, offering a humorous and cautionary exploration of why writing email-handling software is inherently doomed.

## TL;DR Ricardo Signes breaks down the architectural insanity of internet email, arguing that flawed standards and non-compliant real-world implementations make it fundamentally impossible to write completely correct email-processing software. Tracing the evolution of email from RFC 724 and RFC 822 through MIME and modern standards, he demonstrates how arcane features—such as header folding, comment-folding whitespace (CFWS), parameter value continuations, and ambiguous Base64 handling—create immense parsing traps and security risks. He details the bizarre complexities of the `addr-spec` grammar, including quoted local-parts and address groups, while highlighting how spammers exploit specification edge cases to bypass naive filters. Ultimately, Signes urges developers to rely on battle-tested libraries, embrace practical pragmatism over strict RFC perfection, and avoid writing bespoke email code whenever possible.

## Atmosphere & style The presentation carries the frantic, darkly comedic energy of a seasoned engineer who has stared directly into an eldritch technical abyss and survived to tell the tale. Signes combines sharp Lovecraftian hyperbole and self-deprecating wit with deep technical expertise, delivering punchlines about broken specifications to an engaged, laughing audience of fellow programmers. The tone oscillates between exasperated horror and hard-earned pragmatic cynicism, creating an entertaining atmosphere of shared developer catharsis.

## Bulletin - [00:04](https://www.youtube.com/watch?v=4s9IjkMAmns&t=4s) Email protocol specifications and real-world client implementations systematically diverge, making completely bug-free and standards-compliant email handling software impossible to implement. - [02:23](https://www.youtube.com/watch?v=4s9IjkMAmns&t=143s) Spammers routinely bypass email filters not by breaking standards, but by constructing strictly valid messages using obscure RFC edge cases such as comments in date fields and silent discarding of non-Base64 bytes. - [04:20](https://www.youtube.com/watch?v=4s9IjkMAmns&t=260s) Header folding rules (dating back to RFC 724) permanently destroy consecutive whitespace sequences during line unwrapping and create major truncation hazards in poorly implemented intermediate mail handlers. - [45:57](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2757s) Syntax-only email address validation is fundamentally ineffective; robust validation requires pragmatic multi-tiered checks, including live DNS MX record lookups and contextual abuse screening.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=4s9IjkMAmns&t=4s) Writing completely standard-compliant email software is doomed to fail because real-world internet traffic routinely violates the specifications, forcing parsers to accept non-compliant input. - [02:23](https://www.youtube.com/watch?v=4s9IjkMAmns&t=143s) The Perl email module ecosystem reflects the deceptive complexity of the format: lightweight modules only function under strict constraints, while robust solutions like Mail::Message require massive codebases. - [04:20](https://www.youtube.com/watch?v=4s9IjkMAmns&t=260s) Header folding was introduced in RFC 724 to address display limits on hardware terminals, introducing lexical equivalence that allows arbitrary line breaks and indentation across message headers. - [06:25](https://www.youtube.com/watch?v=4s9IjkMAmns&t=385s) RFC 2822's Comment or Folding White Space (CFWS) permits recursively nested parenthetical comments between tokens, all of which semantically collapse into a single space. - [08:45](https://www.youtube.com/watch?v=4s9IjkMAmns&t=525s) Date headers permit obsolete military single-letter time zones and interspersed nested comments between date tokens, which modern parsers must still support for backwards compatibility. - [10:52](https://www.youtube.com/watch?v=4s9IjkMAmns&t=652s) The BNF grammars of RFC 2822 codify that obsolete constructs must be actively parsed and accepted rather than rejected, compounding maintenance overhead. - [12:22](https://www.youtube.com/watch?v=4s9IjkMAmns&t=742s) Standard SMTP enforces a strict 1,000-character line length limit and lacks native 8-bit transport, necessitating the development of the MIME suite (RFCs 2045–2049). - [13:59](https://www.youtube.com/watch?v=4s9IjkMAmns&t=839s) Base64 specifications mandate that non-Base64 characters must be silently ignored, creating an exploit vector where spammers inject garbage bytes to evade naive string filters. - [16:36](https://www.youtube.com/watch?v=4s9IjkMAmns&t=996s) In Quoted-Printable streams, escaping carriage return and line feed bytes as '=0D=0A' forces conforming clients to treat them as literal data bytes rather than visual line breaks. - [19:01](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1141s) RFC 2047 encoded-words allow message headers to mix multiple distinct character sets and encodings within a single header line using the '=?charset?encoding?text?=' syntax. - [24:39](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1479s) RFC 2184 (RFC 2231) introduced asterisk-based parameter continuation syntax ('param*0*='), creating a separate encoding system distinct from standard MIME encoded-words. - [28:28](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1708s) Multipart message preambles appear before the first boundary and are completely unrendered by MIME-capable clients, allowing hidden notes or legacy fallback messages. - [33:43](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2023s) The local-part of an email address must preserve case and permits quoted strings with backslash escaping, which malicious actors can exploit to inject CRLF control characters into headers. - [39:39](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2379s) The empty address group construct ('undisclosed-recipients:;') is the only syntactically valid method to send an RFC-compliant email without exposing recipient addresses. - [43:46](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2626s) Folding and unfolding headers collapses arbitrary whitespace into a single space, permanently stripping repeated whitespace unless it was explicitly octet-encoded. - [45:57](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2757s) Email address validation cannot rely on grammar parsing alone; practical implementations must balance DNS MX lookups, internal blocklists, and contextual strictness to avoid delivery and bounce floods. - [47:36](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2856s) Attachment filenames defined in Content-Disposition headers require path traversal sanitization by the receiver to prevent malicious senders from overwriting local files.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=4s9IjkMAmns&t=4s) Introduction: Email Hates You and Wants You to Suffer

Ricardo Signes introduces himself as Rick Signes, known online as `rjbs` across CPAN, IRC, mailing lists, and Flickr. He explains that he maintains an extensive collection of email modules on CPAN and works for Pobox, a company whose entire business revolves around handling email. Signes sets expectations for the talk right away: he is not there to present an uplifting educational lecture on becoming a better programmer or mastering the construction of RFC 2183 `Content-Disposition` headers. Instead, he highlights the grim reality encoded within the foundational RFCs. Showing a favorite quote from the RFC introducing modern multipart messages—*"This is strange and broke, and compatibility is always favored over elegance"*—he declares it the understatement of the century.

The core thesis of the talk is that any attempt to write email-handling software is inherently doomed. Whether an engineer writes code to ingest raw messages from the outside world or to generate new messages to transmit out, the code is almost guaranteed to be wrong. This failure is not a matter of individual incompetence; rather, the underlying standards are flawed, and real-world implementations routinely violate those standards. As a result, code written strictly to specification fails to process actual internet traffic.

Signes delivers a direct warning to the audience about dealing with email code:

> “I’m not saying probably because I know you did it wrong and I can tell you how I know: it's impossible to get it right. The standards get it wrong, the main implementations get it wrong, and that means that doing it right is going to be wrong because you can't actually receive real email. For God's sake, stop writing code that deals with email.”

```mermaid flowchart TD accTitle: The Impossibility of Correct Email Handling accDescr: Diagram illustrating why standard and real-world email implementations cause email code to fail. subgraph SpecFailures["Standards & Reality"] A["RFC Standards Defective"] --> C["Strict Conformance Breaks"] B["Real-World Clients Non-Compliant"] --> C end subgraph Outcome["Resulting Failure"] C --> D["Your Email Code Fails"] end ```

[02:23](https://www.youtube.com/watch?v=4s9IjkMAmns&t=143s) CPAN Modules and Why Spammers Love Email

Signes details his involvement with the Perl Email Project, an effort dedicated to maintaining `Email::*` modules alongside several legacy `MIME::*` and `Mail::*` distributions. Having given a presentation the previous year about the bugs across the project's codebase, he explains that no module can fully conquer the underlying insanity. He breaks down the CPAN ecosystem options:

- `Email::Simple`: Marginally acceptable only under exceptionally narrow constraints: creating a new message from scratch rather than parsing one from disk or an external source, containing an exclusively 7-bit ASCII body, with zero chance of receiving 8-bit variable input, never functioning as a reply to an existing message, and addressing recipients solely by bare email address without human-readable display names. - `Email::MIME`: Capable of generating more varieties of valid messages than `Email::Simple`, yet still prone to producing broken output that fails depending on the recipient's software. - `MIME::Entity`: Offers slightly better standards compliance, but carries a bizarre interface and significant historical baggage. - `Mail::Message`: The "10-ton elephant" of Perl mail handling. It gets the vast majority of edge cases right, but developers routinely avoid it because its sheer bulk clashes with the persistent illusion that email is a simple domain.

Signes notes that the Byzantine complexity of message parsing creates an enormous exploit surface for spammers. Because compliant parsing is so counterintuitive, a sender wishing to conceal malicious payloads does not need to send malformed data. By crafting completely standards-compliant email constructed out of obscure RFC edge cases, spammers easily bypass standard parsers and security scanners that choke on the input.

[04:20](https://www.youtube.com/watch?v=4s9IjkMAmns&t=260s) Header Folding and Lexical Equivalence (RFC 724)

Addressing the history of this complexity, Signes pushes back against the belief that the rot began in 1982 with RFC 821 and RFC 822. Early specifications like RFC 561 and RFC 680 were simple and straightforward to parse. The real madness began in 1977 with RFC 724. RFC 724 introduced "header folding" to solve a presentation problem at the data layer: long headers did not fit on narrow Wyse terminals, and authors avoided writing a presentation wrapping layer by embedding line breaks directly into the transport format. Under header folding, a field could be broken across lines using a newline followed by indentation, creating lexical equivalence with an unfolded single line.

```mermaid flowchart LR accTitle: Header Folding Lexical Equivalence accDescr: RFC 724 allows unfolding a header across multiple indented lines while preserving equivalence. A["Subject: This is a really long subject line"] -->|"Lexically Equivalent (RFC 724)"| B["Subject: This is a\n really long\n subject line"] ```

RFC 724 permits line breaks and whitespace to be inserted wherever a terminal requires wrapping. While this allows wrapping long headers, it also permits splitting headers across an arbitrary number of single-word lines. In real-world systems, encountering such bizarre formatting is common. Half the time it reflects a completely non-compliant client, and the other half it is strictly legal under ancient specifications. Because parsers cannot reliably distinguish between intentional standards exploitation and accidental bugs, receiving systems are forced to accept virtually everything.

[06:25](https://www.youtube.com/watch?v=4s9IjkMAmns&t=385s) CFWS: Folding White Space and Nested Comments

Signes examines how the specifications define valid locations for linear whitespace. RFC 724 states that whitespace may be freely inserted in "reasonable places," dryly noting that "the existence of another lexical analyzer is assumed." In practical terms, this rule means whitespace may be inserted anywhere two tokens meet. Furthermore, any arbitrary whitespace is treated as equivalent to a comment, and comments are allowed to nest inside parentheses. When a comment acts as a delimiter between tokens, the specification dictates that the entire comment structure collapses into the semantic equivalent of a single space.

Signes illustrates this by showing how an ordinary header can be progressively obfuscated with whitespace and nested commentary while remaining strictly identical in meaning:

> “If you ever want to see an email programmer cry and see his eyes glaze over with rage and desperation, just ask him about comment or folding whitespace.”

RFC 2822 later codified this concept as CFWS (Comment or Folding White Space). While RFC 2822 attempted to rein in the damage, its remedies were minimal. It outlawed lines composed entirely of whitespace—a syntax permitted under RFC 822 and RFC 724—but left the rest of the parsing nightmare intact.

- An RFC comment is wrapped in matching parentheses: `(this is a comment)`. - Comments can nest recursively: `(outer comment (nested comment))`. - For semantic understanding, any comment acting as a token separator is evaluated as a single space character. - RFC 2822 strictly prohibits empty lines consisting solely of whitespace characters, eliminating one historical edge case from RFC 822.

[08:45](https://www.youtube.com/watch?v=4s9IjkMAmns&t=525s) Date Headers, Military Time Zones, and RFC 2822 Obsoletion

Signes shifts focus to the ubiquitous `Date` header, which is mandatory on every email message to denote composition and transmission time. While modern protocols favor ISO 8601, email dates allow a vast array of legacy variations. Standard forms include conventional `Day, DD Mon YYYY HH:MM:SS ±ZZZZ` formats, but the specification also permits arbitrary whitespace padding and military single-letter time zone indicators. In military time zones, letters A through M (omitting J) represent time zones east of UTC, Z represents Greenwich Mean Time, and N through Y span time zones west of UTC.

Because date fields consist of discrete tokens, comments can technically be interspersed throughout the date components. Signes demonstrates an extreme date header filled with nested comments between the day, month, year, and time tokens. He jokes that a particular comment might look invalid until realizing empty comments `()` are strictly forbidden by the grammar.

RFC 2822 attempted to restrict comments to the end of the date field. However, robust mail systems cannot enforce RFC 2822 rules in isolation. Because modern software must remain backwards-compatible with incoming RFC 822 mail, parsers must continue to support deeply nested, interspersed comments embedded inside timestamp components. What appears to be a tidy grammar in newer RFC documents is undermined by the requirement to support legacy pseudo-grammars that tolerate complete madness.

- Every email is formally required to include a `Date` header. - Military time zones (such as `Z` for Zulu/GMT, or single-character zone offsets like `D`) remain valid within legacy date parsing rules. - Comments cannot be completely empty; `()` without content violates the specification. - RFC 2822 relegates mid-token date comments to obsolete syntax sections, but parsers must still retain the legacy logic to function on the live internet.

[10:52](https://www.youtube.com/watch?v=4s9IjkMAmns&t=652s) BNF Grammars, Obsolete Syntax, and the Structure of Email

RFC specifications gradually adopted Backus–Naur Form (BNF) to define email message structures, establishing formal rules for components such as dates, days, months, and years. In a straightforward world, a date would consist of a day (one or two digits), a month (one of the English three-letter abbreviations), and a four-digit year. However, standards like RFC 2822 complicate this by declaring that a day can be either standard digits or an "obsolete day," and a month can be a normal name or an "obsolete month." An obsolete month is defined simply as a standard month name flanked by folding white space and comments. This design choice creates a maintenance nightmare because any parser following the standard cannot merely reject non-conforming data.

> "The term is anything that is obsolete must be accepted and parsed." — Ricardo Signes

The core structure of an Internet text message appears deceptive in its simplicity: a message consists solely of a block of headers, optionally followed by a blank line and the message body, all represented in plain ASCII characters. Because the body is just an unformatted stream of text, early implementers assumed the complexity of email was entirely confined to the headers. This assumption proved completely false once messages needed to cross real networks.

```mermaid flowchart TD accTitle: Email Message Anatomy accDescr: Diagram showing the basic breakdown of an RFC 822 email message into headers and an ASCII body.

subgraph Msg["RFC 822 Message Structure"] H["Header Fields\n(Complex BNF, Folding White Space)"] SEP["Blank Line (CRLF CRLF)"] B["Message Body\n(Plain 7-bit ASCII)"] H --> SEP SEP --> B end ```

- RFC date specifications define months as three-letter abbreviations: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec. - Obsolete constructs in the grammar mandate full parsing support rather than deprecation or outright rejection. - Standard email messages enforce an absolute separation between the header section and the body text via a single empty line (`\r\n\r\n`).

---

[12:22](https://www.youtube.com/watch?v=4s9IjkMAmns&t=742s) The "Simple" Mail Transfer Protocol and the Birth of MIME

RFC 821 superseded early experimental mechanisms that sent mail across networks, including Unix-to-Unix Copy (UUCP) and File Transfer Protocol (FTP) extensions. Dubbed the Simple Mail Transfer Protocol (SMTP), its name stands as a historical contradiction. Under standard RFC 821 SMTP, lines are hard-capped at a maximum length of 1,000 characters (including the trailing carriage return and line feed). Unlike headers, message bodies offer no mechanism for folding excessively long lines, and the protocol provides no built-in way to transmit 8-bit characters. The only native protocol-level solution to non-ASCII text was an implicit demand that international users avoid using 8-bit characters entirely.

To solve the limitations of transmitting rich media and international alphabets over 7-bit transport, Multipurpose Internet Mail Extensions (MIME) was developed. Early precursors to MIME attempted to establish encapsulation rules; RFC 934 defined message forwarding using explicit finite-state automata, while RFC 1049 supported specialized body types like `troff`. RFC 1341 formally defined MIME, but its initial iteration attempted to introduce an SGML-compatible markup language for rich text in Section 7. The modern MIME architecture was ultimately standardized across a cohesive suite of documents: RFC 2045 through RFC 2049.

> "821 calls this the simple mail transfer protocol and like all the best names it is a lie." — Ricardo Signes

- RFC 821 line length restriction: 1,000 characters maximum, inclusive of the terminating `<CRLF>`. - RFC 934 formalized encapsulation behavior by specifying explicit finite-state automata for forwarding agents. - RFC 1049 defined early content-type mechanisms, famously supporting types such as `troff`. - RFC 1341 was superseded by the core MIME suite spanning RFCs 2045, 2046, 2047, 2048, and 2049.

---

[13:59](https://www.youtube.com/watch?v=4s9IjkMAmns&t=839s) Base64 Encoding, Character Sets, and Lenient Parsing Flaws

Sending an 8-bit message over a 7-bit SMTP channel requires declaring an encoding format in the message headers. Using `Content-Transfer-Encoding: base64` transforms arbitrary 8-bit byte sequences into a 6-bit printable ASCII representation. However, turning text into bytes introduces fundamental character encoding problems. If a sender types text in UTF-8, the system emits specific byte sequences; if a receiving client lacks character set metadata and guesses ISO-8859-1 (Latin-1) or Mac OS Roman, it produces completely garbled strings rather than readable text. Resolving this requires an explicit `Content-Type: text/plain; charset="utf-8"` header.

Base64 implementations contain a serious parser loophole: the specification mandates that any character encountered in a Base64 stream that is not a valid Base64 digit must be silently ignored. This permissive handling predated RFC 2119's standardized keyword requirements (`MUST`, `SHOULD`, `MAY`), leaving the handling of malformed input ambiguous across mail clients. While some implementations attempt to reject malformed input, standard-compliant implementations strip invalid characters, allowing hidden data to slip past basic filters. Spammers exploit this behavior to bypass naive content inspection filters (such as `procmail` recipes checking for strings like "Viagra") by splitting keywords with arbitrary non-Base64 bytes that mail user agents silently discard during decoding.

> "When you write email code that parses the body by splitting up the lines of the body you have failed, and everyone does this." — Ricardo Signes

```mermaid flowchart LR accTitle: Base64 and Charset Decoding Pipeline accDescr: Pipeline displaying how input text converts to bytes, Base64 transport, and displays differently based on client charset detection.

subgraph Encoding["Sender Side"] T["Text Input\n(e.g., UTF-8)"] --> BY["Raw Byte Stream"] BY --> B64["Base64 Encoding\n(7-bit ASCII Safe)"] end

subgraph Transport["SMTP Transport"] B64 --> NET["Network Transfer\n(Max 1000 chars/line)"] end

subgraph Decoding["Receiver Side"] NET --> D64["Base64 Decoding\n(Discards Non-Digits)"] D64 --> C{"Declared Charset?"} C -->|"utf-8"| OK["Readable Original Text"] C -->|"iso-8859-1"| MOJ["Mojibake Output"] C -->|"macintosh"| MAC["Corrupted Mac Glyphs"] end ```

- Base64 maps 8-bit binary input into a safe 64-character 7-bit ASCII alphabet (`A–Z`, `a–z`, `0–9`, `+`, `/`). - Omitting `charset` declarations forces mail clients to guess the encoding, yielding mojibake across platforms. - Non-Base64 characters must be silently ignored according to early specifications, breaking simple line-by-line body parsers.

---

[16:36](https://www.youtube.com/watch?v=4s9IjkMAmns&t=996s) Quoted-Printable Traps and Literal Newline Semantics

As an alternative to Base64, MIME offers `Content-Transfer-Encoding: quoted-printable`, designed to keep primarily ASCII text human-readable. Only characters outside the standard ASCII range—or characters that have syntactic significance—must be escaped as an equals sign followed by a two-digit hexadecimal byte value (such as `=E9`). Some overzealous encoders aggressively escape standard characters, including spaces (`=20`), creating noisy but valid text.

A severe semantic trap exists regarding line breaks in quoted-printable streams. In standard quoted-printable text, an actual line break is indicated by a literal hard carriage-return/line-feed (`CRLF`). If an encoder instead represents a newline by escaping the individual bytes as hex strings (`=0D=0A`), the specification dictates that these are literal CR and LF control characters rather than visual line endings. Mail user agents like Mutt strictly adhere to this rule: instead of rendering a line break on screen, they render raw, unparsed `^M` carriage return symbols or literal character blocks, corrupting the message layout.

> "If I put a literal CRLF there it means that I want a new line when you display the message, but if I encode the CRLF it means that I want the CR byte and the LF byte." — Ricardo Signes

- Quoted-printable encodes bytes using hexadecimal pairs preceded by `=`. - Escaping carriage return and line feed bytes as `=0D=0A` changes their semantic meaning from structural line breaks to raw byte data. - User agents such as Mutt intentionally render encoded `=0D=0A` sequences as literal control characters (`^M`) to conform to spec semantics.

---

[19:01](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1141s) RFC 2047 Encoded-Words and Multi-Charset Header Chaos

While `Content-Transfer-Encoding` solves encoding issues within the message body, it does not apply to RFC 822/2822 message headers. Headers are strictly restricted to 7-bit ASCII, meaning non-ASCII characters in display names within the `From:` or `To:` fields (such as "Vlad" spelled with Cyrillic or accented characters) cause immediate transport and parsing failures. Because `Content-Transfer-Encoding` cannot be applied to headers, the IETF created RFC 2047 to define "encoded-words."

An encoded-word is a specialized token that represents non-ASCII data within headers using the syntax `=?charset?encoding?encoded_text?=`. The encoding token is a single character: `Q` represents a header-specific variant of quoted-printable (where spaces can be represented by underscores), while `B` represents standard Base64. Because the specification permits individual adjacent encoded-words to declare entirely different character sets and encoding formats within the same header field, a single human name can legally be composed of a UTF-8 Q-encoded first name, an unencoded 7-bit ASCII middle token, and an EBCDIC B-encoded surname concatenated together.

> "Encoded-word is a new token introduced by RFC 2047 and it's just like a word, but it's encoded: that means it starts with equals question mark, it ends with question mark equals, and there's two question marks in the middle." — Ricardo Signes

- Encoded-word format: `=?<charset>?<encoding>?<data>?=`. - In RFC 2047, `Q` denotes the header variant of Quoted-Printable, and `B` denotes Base64. - RFC 2047 allows multiple adjacent tokens with different character sets (e.g., UTF-8, ISO-8859-1, EBCDIC) to construct a single logical display string.

[21:32](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1292s) MIME Encoded Words, Underscore Escaping, and Structured Comments

Ricardo Signes explains that even when dealing with entirely 7-bit ASCII strings—such as a sender's display name—senders or spammers might choose to encode them anyway just for safety or obfuscation. However, standard quoted-printable encoding cannot simply be dropped into an encoded-word token (`=?charset?encoding?encoded-text?=`) without modification. Because spaces cannot be permitted inside encoded words (a parser would not know where the token terminates), an underscore (`_`) is substituted for spaces. As Signes dryly points out, this creates an absurd dialect discrepancy: “in normal quoted printable underscore is underscore in this quoted printable underscore is space great let's put it under the rug and keep going.”

```mermaid flowchart TD accTitle: Quoted-Printable Encoding vs MIME Encoded-Word Q Encoding accDescr: Comparison of character handling between standard quoted-printable bodies and RFC 2047 header tokens.

subgraph StandardQP["Standard Quoted-Printable (RFC 2045 Body)"] A1["Underscore '_'"] --> B1["Represents literal '_'"] A2["Space ' '"] --> B2["Represents literal ' ' or '=20'"] end

subgraph HeaderQ["MIME Header 'Q' Encoding (RFC 2047)"] C1["Underscore '_'"] --> D1["Represents literal Space ' '"] C2["Space ' '"] --> D2["Illegal: breaks token boundary"] end ```

The resulting headers can quickly degenerate into an unreadable tangle containing multiple character sets within a single string. Signes notes that prior to Unicode, reliably decoding a single header with two or more character sets was essentially impossible. Furthermore, email standards permit comments inside structured header fields. These comments can appear between almost any two components and are dropped by parsers, yet they are strictly structured whitespace: - Comments can nest arbitrarily deep inside matching parentheses. - Comments can themselves contain encoded words. - All of this must be parsed properly without breaking header boundaries.

[22:55](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1375s) File Attachments, Missing RFCs, and the Parameter Hack

Moving on to message bodies, Signes looks at a reply to a "Weekly Newsletter" containing an attachment. Attachments do not arrive naturally as files; they must be structured as discrete email payloads using headers, Base64 encoding, and MIME semantics. Signes walks through the historical mess of attaching a binary document (like a Microsoft Word `.doc` file):

When RFC 2046 was drafted, it indicated that to provide an attachment filename, implementations should use a header called `Content-Disposition`, but the specification noted that `Content-Disposition` would only be formally defined in a future RFC. RFC 2046 also strictly warned that the field could not contain encoded words. Because that defining specification did not emerge for nine months, developers improvised by stuffing arbitrary parameters (like `title`) into `Content-Type`. This introduced a new conundrum: if a document title contains 8-bit non-ASCII characters, how does one encode it? - `Content-Transfer-Encoding` does not apply because parameters sit in the header, not the body. - RFC 2047 encoded-word syntax (`=?charset?Q?...?=`) was explicitly forbidden in MIME parameter values.

As Signes summarizes the dilemma: “so right encoded words um we can't use those in parameters so we can't use that equals question mark question mark question mark question mark equals thing so yeah there's no solution to this problem um you you just got to suck it up.”

[24:39](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1479s) RFC 2184/2231 Parameter Value Continuations and Header Folding

Nine months later, RFC 2184 (later RFC 2231) arrived to specify how parameter values should encode non-ASCII text and handle folding. Rather than reusing existing mechanisms, it invented completely new syntax: 1. An asterisk (`*`) is appended to the parameter name (e.g., `title*=`) to indicate the value is encoded. 2. The value begins with the character set, a single quote, the language tag (e.g., `utf-8'en'`), another single quote, followed by percent-encoded octets. 3. Quoted-printable was rejected because equal signs look too much like standard parameter assignments (`foo=bar`), and Base64 was rejected for poor human readability.

```mermaid flowchart LR accTitle: Parameter Value Splitting and Encoding via RFC 2184 accDescr: Sequence showing how long parameter lines are split into indexed parts with selective encoding flags.

P0["title*0*=utf-8'en'First%20Part%20"] --> P1["title*1*=Second%20Part%20"] P1 --> P2["title*2*=Third%20Part%20"] P2 --> P3["title*3=FinalPartASCII"]

subgraph Assembly["Parsed Full Parameter"] Result["Full Title: 'First Part Second Part Third Part FinalPartASCII'"] end

P0 -.-> Result P1 -.-> Result P2 -.-> Result P3 -.-> Result ```

This collided head-on with header line folding. While RFC 822 introduced folding (inserting CRLF followed by whitespace) to keep lines under 78 or 80 characters for human readability, mail clients evolved to demand it unconditionally: “it's gotten point at this at this time that uh if you don't fold your header stuff breaks so now you must fold.” But because parameters cannot have arbitrary whitespace injected into the middle of continuous tokens, RFC 2184 introduced parameter segmenting: - Parameter names append a number and an optional asterisk: `title*0*=...`, `title*1*=...`, `title*2*=...`, and `title*3=...`. - If an individual segment contains pure 7-bit ASCII, the asterisk is omitted (as in `title*3=`), while encoded segments keep the asterisk. - The language and charset are established once in part zero and must remain consistent across subsequent parts.

[26:54](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1614s) Content-Disposition, Advisory Metadata, and RFC Grammar Bugs

When `Content-Disposition` was finally specified, it allowed senders to declare `attachment`, provide a filename (using the asterisk-encoded parameter format), and theoretically treat mail messages like a remote filesystem. Senders could attach filesystem metadata: - Parameter `size`: strictly advisory and approximate, meaning parsers must never rely on it as an exact byte count. - Parameter `modification-date`: storing file timestamps. - Legal comments scattered directly between the parameter declarations.

Signes highlights a widespread trap regarding trailing semicolons in parameter lists. Under the grammar, parameters are defined strictly as delimited pairs: `value; param=val; param=val` with no trailing semicolon allowed at the end of the header. Yet implementations consistently violate this rule. Signes points out why this cannot be enforced strictly: “if you wonder why I making a point of this because it's not that crazy it's because half the time the RFC that defines this mentions it it gets it wrong in the examples the RFCs get it wrong everyone gets this wrong so you can't reject email because it's wrong which means when you parse email you have to get it right it's just it's a fact of life.”

[28:28](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1708s) Multipart Architecture, Boundaries, and the Preamble

To actually assemble the multipart message containing both text and attachments, the original top-level message body is scooped out and placed inside a nested MIME entity. The container message is declared with `Content-Type: multipart/...` and specifies an arbitrary text `boundary` string to delineate each subpart.

```mermaid flowchart TD accTitle: Multipart Email Structure and Preamble accDescr: Structural layout of a multipart MIME email highlighting the unrendered preamble and bounded parts.

TopHeaders["Top-level Headers: Content-Type: multipart/mixed; boundary=..."] Preamble["Preamble: Unrendered ASCII text outside boundaries"] B1["--boundary"] Part1["Part 1: Text message (text/plain)"] B2["--boundary"] Part2["Part 2: Word Attachment (application/msword; Content-Disposition)"] B3["--boundary--"] Epilogue["Epilogue: Trailing ignored text"]

TopHeaders --> Preamble Preamble --> B1 B1 --> Part1 Part1 --> B2 B2 --> Part2 Part2 --> B3 B3 --> Epilogue ```

Signes breaks down the structural rules of multipart messages: - Between the boundary delimiters lie complete, standalone MIME entities, each with their own headers and payload. - Any text appearing before the very first boundary delimiter is the **preamble**, and text after the final delimiter (`--boundary--`) is the **epilogue**. - Because the top-level `Content-Type` is multipart, `Content-Transfer-Encoding` is forbidden on the outer message; the preamble is therefore strictly 7-bit US-ASCII and invisible to modern MIME-compliant clients. - Senders frequently use the preamble to leave warning text for legacy clients. Signes recounts his favorite spam encounter using this feature: an HTML part selling pills, a plain text part selling pills, and a preamble that read simply, “get a capable email client.”

Finally, Signes notes that language tags made their way back to encoded words in RFC 2231/RFC 5646 formats as well, allowing headers to declare things like `utf-8*en`, before wrapping up the composition phase: “okay so we've composed a message it's got from it's got to it's got an attachment it's got a body it's great this is the best message we've ever had now we're going to send it.”

[31:41](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1901s) The Grammar of `addr-spec` and Domain Literals

Ricardo Signes turns to his favorite topic: email addresses. He recounts an email he received from someone testing the Perl module `Email::Valid`, complaining that it marked an absurd address as valid and asking how that could possibly be true. To demonstrate why, Signes walks through the formal grammar governing `addr-spec`, the specification defining an email address.

```mermaid flowchart TD accTitle: Email Address Grammar Decomposition accDescr: Diagram showing how addr-spec breaks down into local-part, @, and domain rules.

subgraph AS["addr-spec"] LP["local-part"] AT["@"] DOM["domain"] end

subgraph DOM_RULES["domain Parsing"] SUB["subdomain"] DOT_SUB["(. subdomain)*"] ATOM["atom (non-specials, non-control)"] end

LP --> AT --> DOM DOM --> SUB --> ATOM DOM --> DOT_SUB --> ATOM ```

He simplifies the specification on the slide by stripping out the `domain-literal` syntax (used for routing directly by IP, like `rjbs@[1.2.3.4]`, bypassing MX lookups) and removing unused definitions like `dtext`. Looking strictly at `domain`, the grammar defines it as a `subdomain` followed by zero or more sequences of a dot and another `subdomain`. Each `subdomain` is a `domain-ref`, which is an `atom`: one or more characters excluding spaces, control characters, and defined specials. Because the rules for domain literals permit quotes and spaces, technically bizarre constructs remain entirely syntactically valid according to the specifications.

[33:43](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2023s) The Local-Part: Preserving Case and the Threat of Injection

Signes explains that weird domain syntaxes are largely a theoretical problem ("shooting fish in a barrel") because the specification left room for future protocols beyond DNS, whereas in practice, software simply resolves hostnames via DNS. Instead, the true nightmare lies on the left side of the `@` sign: the `local-part`.

Signes reflects on how the specifications treat these boundaries: “The real hate in email messages is the local part and the local part is everything to the left side of the at sign.” A `local-part` is composed of words separated by dots, and crucially, it must be case-preserved. Violating this rule is one of the most common bugs in web applications. Signes admits to his own early blunder writing email software that assumed all incoming addresses were lowercase, causing him to drop legitimate incoming mail sent to uppercase variants like `RJBS`.

```mermaid flowchart LR accTitle: Anatomy of local-part Syntax accDescr: Flowchart illustrating the components of a local-part including words, atoms, and quoted strings.

LP["local-part"] --> W["word (. word)*"] W --> ATOM["atom"] W --> QS["quoted-string"] QS --> ESC["backslash escaping (\)"] QS --> SP["embedded spaces"] ```

Because a word can be either an `atom` or a `quoted-string` (where backslashes escape characters and double quotes enclose spaces), the grammar permits extreme local parts: - Standard identifiers like `rjbs` - Subaddressing variants like `rjbs+tag` - Unquoted or quoted unconventional punctuation, such as `first.last` or `!#$%&'*+-/=?^_{|}~` - Embedded spaces and escaped characters inside quotes

The danger escalates when quoted strings contain literal control characters: - Signes highlights an address containing embedded carriage returns and line feeds: `"Ricardo\r\nTo: rjbs"@example.com`. - Parsers that fail to respect quoting rules will split the header prematurely, interpreting the carriage return as the end of the header and the remainder as a whole new header (`To: rjbs`). - Signes points out the operational risk: “That is two message heads no matter what your software wants to tell you, it's just another vector for abuse.”

[36:50](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2210s) Phrases, Address Groups, and the "How Many Addresses" Quiz

Moving beyond the bare `addr-spec`, Signes discusses the `phrase`, which is the display name placed before an address enclosed in angle brackets (e.g., `Ricardo Signes <rjbs@cpan.org>`), and mentions that comments in parentheses can appear alongside them. He then introduces a lesser-known specification feature: the address group. An address group consists of a group name, a colon, a comma-separated list of addresses, and a terminating semicolon (such as `Technical Staff (R&D): rjbs@example.com, user@example.com;`).

Signes challenges the audience to spot how many valid email addresses appear on an example slide:

```text Team: rjbs.@cpan.org, http://example.com, "Van Helsing"@vampires.com, (Invisible Man), Alito@justice.gov.; ```

When an audience member guesses two, Signes walks through the slide token by token to show why almost every address fails: - `rjbs.@cpan.org` is invalid because dots cannot be the trailing (or leading) character of a local-part. - `http://example.com` fails because colons are illegal unquoted characters in local-parts. - `"Van Helsing"@vampires.com` fails in this specific slide because an explicit eight-bit character is present without proper encoding. - `(Invisible Man)` is merely a comment inside parentheses, not an address at all. - Commas without addresses between them are legally skipped, functioning similarly to empty list separators in Perl. - `Alito@justice.gov.` is invalid because the fully qualified domain name includes a trailing root dot; while valid in DNS lookups, a trailing dot is forbidden in the email grammar. - Consequently, only one actual address on the entire slide satisfies the RFC criteria.

[39:39](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2379s) Nested Groups, RFC Discrepancies, and Undisclosed Recipients

The complexity deepens when evaluating nested address groups across conflicting specifications. RFC 733 / RFC 822 treated nested groups unclearly, whereas RFC 2822 made nesting explicitly illegal. Signes highlights the real-world headache of standard revisions: while developers prefer to target newer RFCs like 2821 and 2822, older systems deployed across the Internet continue to operate under RFC 821 and RFC 822 rules. Parsers are forced to balance strict adherence against the messy reality of lenient implementations.

Signes reveals the single most practical takeaway of the entire presentation: - “That is the only way to produce a valid email message that does not have any recipients.” - An empty group—conventionally written as `undisclosed-recipients:;`—provides a syntactically valid `To:` header containing a name, a colon, and an immediate semicolon with zero addresses. - This allows bulk senders or BCC-only mailers to satisfy the parser requirement for a destination header without exposing private recipient addresses. - Signes quips: “If you are trying to send a message and you are doing it with something other than Mutt or Mail.app, you are losing the game already.”

[41:53](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2513s) Q&A: Header Line Folding and Length Limits

Noticing he only has about eight minutes remaining, Signes decides to postpone his planned segment on MIME until next year's conference and opens the floor for questions: “Who dares ask a question?”

An audience member named Todd explains that his workplace purchased an email list management application that began misbehaving, leading him to inspect the RFCs regarding header line folding and maximum line lengths. Todd asks whether line length restrictions apply before or after a line has been unfolded.

Signes reframes the question (“What's the deal with header line folding?”) and explains the folding mechanics: - Folding header lines (inserting a carriage return/line feed followed by whitespace) is generally optional for the sender, except when dealing with specific rigid parameter values. - Senders are encouraged to break lines around the 72- or 79-character mark. - The recommended line length limits apply to the full logical length of the line, including the header field name prefix (such as `Foo-Bar-Baz:`). - Senders have broad discretion over where whitespace breaks occur. - Signes concludes with his personal preference on folding headers: “If it were up to me and everything accepted everything, I would never fold it.”

[43:46](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2626s) Header Folding, Whitespace Loss, and Truncation Bugs

Ricardo Signes explains a subtle and frustrating flaw with email header line folding. When a line is folded, an application finds a point where there is whitespace, inserts a carriage return followed by a space, and continues. However, an arbitrary amount of whitespace can be placed on either side of that carriage return, and upon processing, the whole sequence is compressed down into a single space. Consequently, if a header required multiple consecutive spaces for any reason, that information is permanently destroyed unless encoded as literal octets.

An audience member raises a related scenario: an incoming message is addressed to 500 recipients on a `To:` header line, neatly folded into short individual lines by the sender's client. However, an application their company purchased ingests the message, unfolds the header into a single massive string of 500 addresses, decides the result exceeds an internal maximum header line length, and truncates it. Signes replies that while an official length ceiling might exist in some specification, he does not know of one, and very little worthwhile software enforces arbitrary limits on header length. Addressing the practical reality of email engineering, Ricardo Signes emphasizes: “Another point to to weigh in pragmatic for a moment, the point with email, the specs are all crap, you can't trust them, you can barely deal with the specs and expect anything to work, what you really need to do is do whatever works.” He hates resorting to such hacks in professional enterprise code, but in email handling, survival demands doing whatever prevents software from destructively unfolding and truncating valid headers.

```mermaid flowchart TD accTitle: Header Folding and Unfolding Truncation Issue accDescr: Flowchart illustrating how folded email headers can be improperly truncated by receiving software. subgraph SENDER["Sender Application"] A["500 Email Addresses"] --> B["Fold across lines with CR + Space"] end subgraph RECEIVER["Faulty Receiving Application"] B --> C["Unfold all lines into single long string"] C --> D{"Exceeds internal max length?"} D -- "Yes" --> E["Truncate header & destroy recipients"] end ```

[45:57](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2757s) Email Address Validation and Practical Pragmatism

An audience member asks for the appropriate snappy response to give a manager who demands adding email address validation to an application. Signes answers that the real response must simply be “Yes, sir.” He points out: “Email address validation is impossible, but that doesn't mean it's not worth attempting.” Signes maintains the Perl module `Email::Valid` (noting with self-deprecating humor that he did not author it, but merely applies patches like he does for a lot of email software), and he considers it quite good.

Signes explains that merely parsing local syntax against the formal RFC grammar is technically possible, but yields very little useful information. Pragmatic validation requires multi-tiered checks:

- Verifying whether the domain name actually exists in DNS. - Checking for valid MX records associated with that domain. - Screening against internal blocklists of known bogus addresses. - Applying contextual strictness: at Pobox, if incoming mail is simply being forwarded along, they assume it is good enough and pass it through; conversely, if an end-user attempts to register an address as a brand-new username, they enforce a far stricter set of rules.

Skipping address validation entirely causes catastrophic downstream consequences. Signes cautions that if you refuse to validate, your mail servers accept raw garbage, futile outbound delivery attempts clog up the outgoing mail exchangers (MXs), and engineers end up having to build extensive bounce parsing systems—an operational ordeal so painful that Signes calls it “a whole other sequel to this.”

[47:36](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2856s) MIME Filename Encoding and Path Traversal Hazards

An audience member asks whether multi-line header folding rules also apply to file names in attachments. Signes clarifies that the most standard, canonical place to locate an attachment's name is inside the `filename` or `name` parameter of a structured `Content-Disposition` header. Because it is defined as a parameter, it relies on MIME parameter value and line continuations:

- Non-ASCII 8-bit octets must use the asterisk character encoding syntax (e.g., `filename*="utf-8''..."`). - Long parameter values split across multiple sections utilize continuation numbers with asterisks (such as `filename*0*`, `filename*1*`, `filename*2*`). - The parameter must only specify the leaf file name, not directory paths. Because email clients cannot strictly enforce this on the wire, receiving software must actively sanitize input to prevent malicious path traversal from blowing away files on the local filesystem.

As the session wraps up, Signes notes that he has run out of time, thanking the room to warm applause.

## Notable quotes > [45:57](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2757s) Email address validation is impossible, but that doesn't mean it's not worth attempting. — Ricardo Signes

> [00:04](https://www.youtube.com/watch?v=4s9IjkMAmns&t=4s) The standards get it wrong, the main implementations get it wrong, and that means that doing it right is going to be wrong because you can't actually receive real email. — Ricardo Signes

> [13:59](https://www.youtube.com/watch?v=4s9IjkMAmns&t=839s) When you write email code that parses the body by splitting up the lines of the body you have failed, and everyone does this. — Ricardo Signes

> [39:39](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2379s) If you are trying to send a message and you are doing it with something other than Mutt or Mail.app, you are losing the game already. — Ricardo Signes

> [33:43](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2023s) The real hate in email messages is the local part and the local part is everything to the left side of the at sign. — Ricardo Signes

## Mentioned Ricardo Signes, Pobox, CPAN, Perl Email Project, Email::Simple, Email::MIME, MIME::Entity, Mail::Message, Email::Valid, RFC 561, RFC 680, RFC 724, RFC 733, RFC 821, RFC 822, RFC 934, RFC 1049, RFC 1341, RFC 2045, RFC 2046, RFC 2047, RFC 2048, RFC 2049, RFC 2119, RFC 2183, RFC 2184, RFC 2231, RFC 2821, RFC 2822, RFC 5646, Mutt, Mail.app, UUCP, FTP, SMTP, IMAP, MIME

## Video description By Ricardo SIGNES (‎rjbs‎) from ABE.pm, Philadelphia.pm Date: Monday June 16, 2008 14:50 Duration: 45 minutes Target audience: Lightning Tags: brains email hate horror lovecraft mail rant warning wtf you.are.all.going.to.die zombie

Email: you see it every day. It's on your desktop. It's in your servers. Through the magic of modern technology, it flows invisibly through the air and into your PDA! Your cellular phone conducts silent and arcane conversations with distant servers, speaking the ancient language of SMTP and the unknowable dialects of IMAP. Surely all this technology means progress of mankind... or does it?

Now you can learn the truth: buried deep within the eldrich documents describing the slow accretion of features in the Elder Protocols lie subtle recursions and devilish ambiguities -- grammars that are not grammars, strings quietly stripped of their higher bits -- designed to slowly bend the mind of all who implement them.

Now you can hear of the horrors perpetrated by those who became thrall to these documents -- and by those, too, who fought to oppose them for the sake of their own dwindling sanity.

Now it can be told: Email Hates the Living!

WARNING: This talk is low in career-sustaining educational content.

[Source](https://www.youtube.com/watch?v=4s9IjkMAmns)

中文简报

Ricardo Signes 呼吁工程师停用自研邮件解析器由于过时的 RFC 标准与普遍存在的实际客户端缺陷相互冲突，迫使解析器不得不包容畸形输入才能正常运作，因此从根本上无法构建出完全合规的邮件软件。 Ricardo Signes 深入剖析了互联网电子邮件在架构层面的混乱，认为存在缺陷的标准以及不合规的实际实现，使得编写完全正确的邮件处理软件从根本上变得不可能。在梳理了电子邮件从 RFC 724、RFC 822 到 MIME 及现代标准的演进历程后，他展示了诸如标头折叠、包含注释的折叠空白符（CFWS）、参数值续行以及含糊不清的 Base64 处理等晦涩特性，如何带来了巨大的解析陷阱和安全隐患。他详细阐述了 `addr-spec` 语法令人费解的复杂性（包括带引号的 local-part 和地址组），同时指出了垃圾邮件发送者如何利用规范中的极端情况绕过简易过滤器。最终，Signes 敦促开发者依托久经考验的现有库，注重实际应用而非执着于严苛的 RFC 完美标准，并尽可能避免自研邮件处理代码。电子邮件协议规范与现实中的客户端实现存在系统性脱节，导致完全无缺陷且符合标准的邮件处理软件无法被实现。 (0:04) 垃圾邮件发送者通常并非通过违背标准来绕过邮件过滤器，而是利用日期字段中的注释以及静默丢弃非 Base64 字节等晦涩的 RFC 边缘情况，构建出严格合规的消息。 (2:23) 可追溯至 RFC 724 的标头折叠规则在展开行时会永久破坏连续的空白符序列，并在实现不良的中间邮件处理程序中引发严重的截断风险。 (4:20) 仅基于语法的电子邮件地址验证在根本上是无效的；稳健的验证需要务实的多层级检查，包括实时 DNS MX 记录查询以及语境化的滥用筛查。 (45:57)

Reading summary

Pobox engineer Ricardo Signes argues that writing correct email software is practically impossible because foundational Internet standards are contradictory, real-world implementations routinely violate specifications, and systems must tolerate obsolete syntax to function on the live Internet. Standards Compliance vs. Real-World Interoperability (0:04) The core dilemma of email software design is that strict adherence to standards breaks compatibility with real network traffic. Ricardo Signes asserts that while early specifications like RFC 561 were straightforward, subsequent revisions introduced deep ambiguities. Beginning with RFC 724 in 1977, header folding permitted line breaks with indentation to accommodate narrow terminals, embedding presentation hacks directly into the transport format. When later specifications like RFC 2822 attempted to clean up grammar, they relegated older behaviors to 'obsolete syntax' rather than dropping them, requiring modern parsers to accept legacy constructs such as interspersed comments inside date tokens or nested whitespace. Because parsers cannot reliably distinguish between intentional standards exploitation and accidental bugs, receiving systems are forced to accept virtually everything. Spammers take advantage of this permissive architecture by crafting technically valid messages utilizing obscure RFC edge cases—such as inserting arbitrary non-Base64 bytes that decoders silently discard—thereby evading security filters and regular expression matchers that choke on standard-compliant complexity. MIME Patchwork and Parameter Encoding Quirks (12:22) Because RFC 821's Simple Mail Transfer Protocol restricted messages to 7-bit ASCII and capped lines at 1,000 characters, Multipurpose Internet Mail Extensions (MIME) had to be layered on top to support binary data, attachments, and international character sets. However, MIME introduced conflicting dialects across different message contexts. For message bodies, RFC 2045 established Base64 and Quoted-Printable encodings. In Quoted-Printable, standard line breaks require literal carriage-return/line-feed bytes; escaping newlines as hex values (`=0D=0A`) instead forces compliant clients like Mutt to display raw control characters like `^M` rather than visual line endings. Headers required an entirely different mechanism. Because `Content-Transfer-Encoding` does not apply to headers, RFC 2047 introduced encoded-words (`=?charset?encoding?data?=`), in which Quoted-Printable substitutes underscores for spaces to avoid breaking token boundaries. For parameter values such as file attachment titles in `Content-Disposition`, RFC 2184 (later RFC 2231) created a third, isolated mechanism using asterisks and percent-encoding (`title*0*=utf-8'en'part`), alongside numeric continuations for line folding. RFCs themselves frequently published incorrect examples, such as illegal trailing semicolons in parameter lists, forcing software authors to adopt lenient parsing to survive. Addressing Grammars, Injection Risks, and Pragmatic Handling (31:41) The formal grammar for email addresses demonstrates a significant divergence between specification and implementation. While domain definitions allow obsolete formats and domain literals, real-world systems strictly rely on DNS resolution. Conversely, the `local-part` (left of the `@`) permits complex quoted strings, escaped characters, and case preservation. Dropping case sensitivity can cause dropped mail, while failing to sanitize quoted strings containing literal carriage returns (`"user\r\nTo: other"@example.com`) introduces header injection vulnerabilities where receivers split the address into unintended additional headers. Signes also highlights lesser-known features of address syntax, such as address groups terminated by semicolons. The construct `undisclosed-recipients:;` is the only standard-compliant way to format a destination header without exposing recipient addresses. Because RFC-level address validation is functionally unhelpful on its own, Signes advocates for pragmatic, layered checks in production—validating DNS existence, checking MX records, filtering against known abuse blocklists, and sanitizing filename parameters against directory traversal attacks. Signes concludes that relying strictly on formal RFC specifications will inevitably cause production systems to break or truncate data. Instead, engineers must prioritize pragmatic survival strategies—handling messy edge cases, validating domains through external infrastructure rather than pure regex grammars, and adopting robust, established libraries over custom parsers.

精炼阅读摘要

资深邮件系统开发者 Ricardo Signes 认为，由于底层 RFC 标准本身存在缺陷且现实软件普遍违规实现，编写完全符合规范且能稳定处理真实流量的邮件软件在技术上几乎注定失败。历史遗留语法与注释折叠陷阱 (4:20) 邮件格式的解析困局并非始于 1982 年的 RFC 821 与 RFC 822，而是早在 1977 年的 RFC 724 引入标头折叠（Header Folding）时便已埋下隐患。RFC 724 为了适配狭窄终端的排版限制，允许在任意词汇间插入换行与缩进，使多行文本在词法上等同于单行。RFC 2822 虽将此类规则归纳为注释或折叠空白（CFWS）并禁止纯空白行，但仍保留了极其宽松的遗留语法。在规范中，括号内的注释不仅可以任意嵌套，且作为词元分隔符时会被直接折算为单个空格。由于 RFC 2822 等后续规范明确要求解析器必须接受并解析所有被废弃的过时语法（Obsolete Syntax），现代邮件系统无法直接拒绝历史异常格式。例如日期标头（Date Header）在规范中允许混入包含嵌套注释的组件，甚至支持军用单字母时区（如 Z 代表格林威治标准时间，A 至 M 与 N 至 Y 分别对应东西时区，且排除 J）。一旦标头被展开，多重连续空格往往被压缩为单个空格，导致原始排版信息被永久破坏，部分老旧客户端甚至会因展开后单行过长而直接截断标头，造成收件人丢失。传输限制与 MIME 编码混杂危机 (12:22) RFC 821 规定的 SMTP 协议存在硬性限制：单行长度不可超过 1,000 个字符（含 CRLF），且原生通道仅支持 7 位 ASCII 字符。为了在不原生支持 8 位字符的通道上传输富文本与国际化内容，IETF 相继制定了 RFC 1341 以及后续的 RFC 2045 至 2049 系列 MIME 规范。但在此机制下，Base64 规范要求解码器必须静默忽略任何非 Base64 字符，这使得垃圾邮件发送者可以通过插入无关字节拆分敏感关键词，从而轻易规避简易内容扫描器；Quoted-Printable 编码若将换行符转义为十六进制的“=0D=0A”，在 Mutt 等严格遵守规范的客户端中会被呈现为原始控制字符（如 ^M）而非真正的换行。更严峻的冲突出现在标头区域。由于 Content-Transfer-Encoding 无法作用于标头，RFC 2047 引入了编码词（Encoded-Words）语法（形如 =?charset?encoding?encoded_text?=）。该机制允许在同一标头字段内并列使用不同字符集与编码模式，甚至允许在局部使用下划线代替空格的特殊 Q 编码。在 Unicode 普及之前，解析此类混合了不同编码标准的字符串极为棘手，同时也为邮件安全检测带来了庞大的规避攻击面。附件参数修补与多部分实体结构 (22:55) 在处理文件附件时，规范演进的脱节进一步加剧了实现混乱。RFC 2046 建议使用 Content-Disposition 标头指定附件文件名，却延后了该标头的具体规范，且明文禁止在参数值中使用 RFC 2047 编码词。开发者曾临时将 title 等参数塞入 Content-Type，直到九个月后 RFC 2184（后演变为 RFC 2231）才确立参数连续与编码语法：通过在参数名后追加星号与索引编号（例如 title*0*=utf-8'en'First%20Part%20 与 title*1=SecondPart），使用语言标签与百分号编码拆分超长字符串。在包含附件的多部分结构（multipart）中，最外层容器标头声明 Content-Type: multipart 并定义边界字符串（boundary）。规范禁止在最外层设置 Content-Transfer-Encoding，因此首个边界之前的前言（Preamble）和末尾边界之后的后记（Epilogue）仅能使用 7 位纯 ASCII 文本，现代邮件客户端通常会忽略这些区域，但常被垃圾邮件发送者用于向老旧客户端显示备用文本。此外，Content-Disposition 规范中的 size 参数纯属参考值，且关于参数末尾是否允许分号的分歧屡见不鲜，导致严格校验格式的解析器极易崩溃。地址规范暗礁与工程防御实践 (31:41) 电子邮件地址的语法规则（addr-spec）极度复杂，核心风险集中在 @ 符号左侧的 local-part。根据规范，local-part 由点号分隔的词元组成，并且严格区分大小写；一旦系统盲目将其转换为小写，便会导致合法邮件丢失。若采用带双引号的 quoted-string 形式，local-part 不仅可以包含空格、反斜杠转义字符与特殊标点，甚至能嵌入回车换行符（如 "Ricardo\r\nTo: rjbs"@example.com），未妥善处理引号边界的解析器会将其错误识别为两个独立的标头，引发标头注入攻击。地址组（Address Group）语法还允许通过不含具体地址的空组形式（如 undisclosed-recipients:;）生成完全合规却不暴露收件人明细的 To 标头。面对复杂的语法陷阱，Signes 指出纯粹依靠 RFC 文法进行地址验证毫无业务意义。工程上的务实做法必须依靠多层防御：核查域名的 DNS 解析与 MX 记录、维护内部黑名单、根据业务上下文（如单纯中继转发还是新用户注册）实施差异化严格度校验，并在处理 Content-Disposition 的 filename 参数时强制剔除目录路径以防止路径遍历攻击，避免无效投递阻塞外发队列并引发更加痛苦的退信解析。 Signes 强调，在邮件工程领域，教条化地追求 RFC 规范纯洁性不仅无法应对充斥着违规实现与古老特例的真实互联网环境，还会导致系统崩溃；开发者在工程实践中应采取务实折衷策略，优先确保程序在非标准现实流量下稳定运行。

Detailed notes — zh

## Background 该视频由 YouTube 频道“m”于 2012 年 5 月 6 日发布，记录了资深 Perl 开发者 Ricardo Signes（网名 rjbs，来自 Pobox 及 ABE.pm、Philadelphia.pm）在 2008 年 6 月 16 日所做的一场技术闪电演讲。页面简介带着洛夫克拉夫特式的恐怖色彩，警告观众潜藏在古老电子邮件协议（如 SMTP、MIME 等 RFC 规范）深处的递归、歧义与历史包袱，正在不断摧残实现者的理智，并戏称本场演讲“对职业生涯毫无教育意义”。 ## TL;DR Ricardo Signes 以其维护 CPAN 邮件模块与在专业邮件服务公司 Pobox 的深厚经验为切入点，痛陈“电子邮件憎恨活人”的工程现实。他从早期 RFC 724 引入的头部折叠（Header Folding）与词法等价性展开，深入拆解了嵌套注释、废弃日期语法（obs-day/obs-month）、7-bit 传输限制下的 Base64 与 Quoted-Printable 陷阱，以及 RFC 2047 编码词与 RFC 2231 参数星号拆分的荒谬打补丁机制。随后，他分析了允许换行注入的畸形邮箱地址（local-part）和地址组语法，并在问答环节指出协议规范漏洞百出、标准自身充满矛盾的事实，强调在现实工程中盲信 RFC 必死无疑，开发者必须靠极度宽容的实用主义与防御性编程才能在邮件系统的深渊中勉强维系理智。 ## Atmosphere & style 整场演讲宛如一场程序员的集体宣泄与黑色幽默脱口秀，充斥着克苏鲁式绝望与辛辣的讽刺。Ricardo 语气激昂、妙语连珠，带着一种与恶劣标准鏖战多年后的创伤后应激障碍（PTSD）式的狂放，将枯燥严谨的 RFC 规范讲述得如同一部技术恐怖片。现场笑声与哀鸣不断，听众在共鸣与惊诧中体会着被历史遗留协议折磨的窒息感，整个过程既荒诞滑稽又充满冷酷清醒的工程现实主义。 ## Bulletin - [00:04](https://www.youtube.com/watch?v=4s9IjkMAmns&t=4s) Ricardo Signes 直指电子邮件标准与生态充斥着历史兼容包袱，任何试图按照纯理论规范开发邮件处理软件的程序员，其代码在现实世界中百分之百都会出错。 - [04:20](https://www.youtube.com/watch?v=4s9IjkMAmns&t=260s) 早期 RFC 724 将显示层的排版换行硬编码进数据协议层，导致折叠白空格与无限递归嵌套注释（CFWS）成为合法的词法等价物，严重破坏了语法解析的确定性。 - [19:01](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1141s) MIME 在处理非 ASCII 字符与附件文件名时设计支离破碎，迫使开发者依赖 RFC 2047 编码词与 RFC 2231 参数带星号索引分片等极其扭曲的方案。 - [34:14](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2054s) 邮箱地址规范（addr-spec）中 local-part 的双引号转义机制允许嵌入空格甚至回车换行，沦为潜在的邮件注入载荷，使理论上的严格邮箱校验在现实中成为不可能的任务。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=4s9IjkMAmns&t=4s) 电子邮件标准的演进奉行兼容性压倒优雅的原则，标准与主流实现漏洞百出，严格守规往往无法在现实中正常收发邮件。 - [04:20](https://www.youtube.com/watch?v=4s9IjkMAmns&t=260s) RFC 724 为适配极窄的物理终端，把换行机制强行塞进数据层，开启了标头折叠与词法等价的混乱源头。 - [06:25](https://www.youtube.com/watch?v=4s9IjkMAmns&t=385s) 注释被协议视为带结构的空白符，支持任意深度的嵌套，且在 RFC 2822 之前甚至允许出现纯空白的折叠空行。 - [08:45](https://www.youtube.com/watch?v=4s9IjkMAmns&t=525s) Date 标头允许穿插军用单字母时区、折叠空白与嵌套注释，规范为了兼容老旧标准创造了大量必须解析的“已废弃”（obsolete）BNF 语法分支。 - [12:22](https://www.youtube.com/watch?v=4s9IjkMAmns&t=742s) RFC 821（SMTP）限制每行最多 1000 字符且仅支持 7 位 ASCII，早期面对国际化需求的态度极其消极，迫使 MIME 体系诞生。 - [13:59](https://www.youtube.com/watch?v=4s9IjkMAmns&t=839s) Base64 解码规范要求静默忽略非法字符，被垃圾邮件发送者利用向正文掺杂噪声以规避简单的文本过滤规则。 - [16:36](https://www.youtube.com/watch?v=4s9IjkMAmns&t=996s) Quoted-Printable 区分物理回车换行 CRLF 与字面编码的 =0D=0A，后者会导致遵照规范的客户端（如 Mutt）渲染出原始控制符乱码。 - [19:01](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1141s) 邮件标头无法直接应用 Content-Transfer-Encoding，必须通过 RFC 2047 编码词机制传递非 ASCII 文本，且允许不同编码格式的词块粗暴拼接。 - [21:32](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1292s) RFC 2047 编码词内部禁止出现真实空格，强制用下划线替代空格，颠覆了标准 Quoted-Printable 的原本规则。 - [22:55](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1375s) 由于 RFC 2046 承诺的标准拖延 9 个月才出台，附件参数经历了标准空白期，最终在 RFC 2231 中催生出百分号编码与带星号索引的分片折行机制。 - [26:54](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1614s) Content-Disposition 的规范文档中连官方示例都写错了尾随分号，迫使接收端解析器必须具有高度的容错性，照单全收格式错误的邮件。 - [28:28](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1708s) Multipart 结构中的前言（Preamble）和结语（Epilogue）脱离了编码声明的管辖，常被垃圾邮件制造者用作戏谑受害者的免责声明区。 - [34:14](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2054s) 邮件地址的 local-part 要求保留大小写，允许带引号的字符串包含转义字符、空格甚至物理换行符，构成邮件头注入漏洞。 - [39:39](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2379s) 向未公开收件人发信时，空地址组 undisclosed-recipients:; 是 RFC 协议标准中唯一合法且合规的表达方式。 - [43:46](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2626s) 标头折叠还原会将换行符两侧的连续空白压缩为一个空格，造成原始空白排版数据不可逆丢失。 - [45:57](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2757s) 邮箱地址校验在理论语法层面无法完全闭环，工程实践中必须分层结合 DNS MX 记录查询，并根据中转或注册等业务场景采取不同宽松度的妥协策略。 - [47:36](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2856s) 附件文件名参数在协议层缺乏防路径穿越的机制，软件写盘时必须主动清洗恶意路径字符串以防文件被覆盖破坏。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=4s9IjkMAmns&t=4s) 讲者背景与电子邮件协议的残酷现实演讲者自我介绍为 Ricardo Signes（网络上常用网名“rjbs”），是 CPAN、IRC、邮件列表及 Flickr 上的活跃开发者。他维护着 CPAN 上大量的电子邮件相关模块，并在一家名为 Pobox 的专业邮件服务公司工作，日常工作完全围绕电子邮件展开。他本次演讲的核心议题是剖析电子邮件标准如何规范其工作与格式，并重点揭示处理邮件时那些令人崩溃的极端边界情况。 ```mermaid flowchart TD accTitle: CPAN 邮件处理模块对比与权衡 accDescr: 展现从简易轻量到复杂笨重的 Perl 邮件处理模块，以及它们各自的局限性。 subgraph SimpleModules["轻量级模块（局限明显）"] ES["Email::Simple 仅支持最简单的纯 7-bit ASCII，无上下文"] EM["Email::MIME 生成能力略强，但接收端仍频繁出错"] end subgraph HeavyModules["重量级模块（功能完整但复杂）"] ME["MIME::Entity 老旧复杂、接口怪异，准确率略高"] MM["Mail::Message “10 吨重的庞然大物”，规则最全却极度臃肿"] end ES --> EM EM --> ME ME --> MM ``` Ricardo 直截了当地指出，了解电子邮件最关键的事实就是：“电子邮件憎恨你，它想要你痛苦”（“email hates you and it wants you to suffer”）。他引用了引入现代多部分邮件格式（RFC 2045 等）的 RFC 文档中最讽刺的一句话：系统本身既古怪又残缺，但兼容性永远压倒优雅。他强调这绝非危言耸听，任何试图读取外部邮件或生成外发邮件的程序员，其代码百分之百都是写错了的，因为标准本身错漏百出，主流实现同样漏洞百出，严格按照标准反而无法正常接收现实中的邮件。 - 讲者来自 Pobox，拥有极深厚的电子邮件工程经验，同时是 Perl Email Project 的维护者。 - 他直言：“为了上帝的缘故，别再写处理电子邮件的代码了”（“for God's sake stop writing code that deals with email”）。 - **Email::Simple**：仅适用于最极端简化的环境——手工构建消息（而非从磁盘读取外部传入的数据）、纯 7-bit ASCII 正文、没有任何 8-bit 变量输入、不是对已有邮件的回复、收件人只有裸邮箱地址而没有“真实姓名”。 - **Email::MIME**：比 Email::Simple 稍好，能生成更多格式合法的邮件，但依旧频繁出错。 - **MIME::Entity**：接口古怪、带有大量历史陈旧代码，但在准确性上有所提升。 - **Mail::Message**：处理邮件模块中的“10 吨重的大象”，虽然极其庞大可怕，但多数细节处理得最为准确。 - **垃圾邮件制造者（Spammers）的利器**：由于邮件语法解析极端复杂，恶意发送者只需构建完全合规却极其冷门刁钻的邮件，就能轻松击垮绝大多数容错性不足的解析器，掩盖恶意载荷。 --- ### [04:20](https://www.youtube.com/watch?v=4s9IjkMAmns&t=260s) 混乱的根源：从早期 RFC 到 RFC 724 引入的折叠白空格不少人误以为电子邮件的疯狂源于 1982 年发布的 RFC 821 与 RFC 822，但事实并非如此。在更早期的 RFC 561 和 RFC 680 时代，电子邮件协议简单直接，任何人都能轻松手写解析器。真正的梦魇始于 RFC 724，该标准引入了“标头折叠”（Header Folding）。当时推出折叠机制的初衷十分狭隘：用户的终端屏幕（如早期的 Wyse 终端）非常狭窄，长标头无法单行放下。设计者不想在终端的表现层（Presentation Layer）实现自动换行算法，反而决定将换行硬编码进底层数据层（Data Layer）。规范允许在标头中插入换行符加缩进，且声明折叠后的版本与单行版本在词法上完全等价。这种折叠不仅可以发生在常规词语之间，甚至可以被极端切分，形成在极窄终端上支离破碎的格式。 ```mermaid flowchart LR accTitle: 词法等价性转换过程 accDescr: 展示 RFC 724 体系下正常标头、折叠标头以及极端折叠/带注释标头的等价转换。 H1["单行标头 Header: value"] H2["折叠标头 Header: CRLF TAB value"] H3["极端折叠/嵌套注释 Header: (nested (comment)) CRLF SPACE value"] H1 ---|"RFC 724 词法等价"| H2 H2 ---|"CFWS 等价规范"| H3 ``` Ricardo 强调，面对这种荒谬的格式，程序员无法简单地拒绝接收，因为现实世界中的邮件系统既可能合法地生成这种内容，也可能因为 Bug 误用它；无法判断其有效性时，解析器往往被迫全部照单全收。 - RFC 561 / RFC 680 时代的邮件设计清晰自然、容易解析。 - RFC 724 首次为了解决终端物理宽度显示问题，而将排版换行引入数据层协议。 - 邮件标头中只要存在换行与后续空白缩进，就被视为与原单行标头“词法等价（lexically equivalent）”。 - 讲者总结现实解析难题：“你根本搞不清楚什么是合法的、什么是不合法的，你基本上必须接收所有东西”（“you can't figure out what's valid and what's not you basically have to accept everything”）。 --- ### [06:25](https://www.youtube.com/watch?v=4s9IjkMAmns&t=385s) 词法等价的失控：任意位置的空白与嵌套注释如果仅限于词间断行，问题尚且有限，但 RFC 724 对“何处可以折叠”做出了极其宽松的定义。标准规定：线性空白（Linear White Space）可以在“合理的位置”（reasonable places）自由插入。甚至标准原文还煞有介事地假定除了常规词法分析器外，还存在另一个词法分析器在工作。在实际实现中，这意味着**只要两个 Token 相遇，彼此相邻，就可以在它们之间任意插入空白**。更灾难性的是，标准将“注释”（Comment，用圆括号包围）也视为一种特殊的空白，并允许注释无限嵌套。在解析内容语义时，一个或多个嵌套注释的作用仅仅被当成一个普通空格。 Ricardo 感叹道：“如果你想看到一个写电子邮件程序的程序员当场痛哭，眼中满含愤怒和绝望，只要问他关于注释或折叠空白的事就行了”（“if you ever want to see an email programmer cry and see his eyes glaze over with rage and desperation just ask him about comment or folding white space”）。在 RFC 2822 中，这种组合被称为 **CFWS**（Comment or Folding White Space）。RFC 2822 试图限制 CFWS 的泛滥，但收效甚微：虽然它明令禁止出现纯空白的空行（这在 RFC 822 和 RFC 724 中甚至是被允许的），但在其他语境下，嵌套注释与杂乱换行依然具有完全的合法性。 - RFC 724 语出惊人，明确写道“线性空白可以自由插入在合理的位置”。 - 注释由 `(` 和 `)` 界定，不仅本身代表空格，还支持任意深度的递归嵌套，例如 `(outer (inner (deep)))`。 - 带有极多嵌套注释和跨行折叠的标头，在 RFC 体系内与最普通的单行标头严格等价。 - RFC 2822 唯一的进步是明确禁止了“整行全由空白字符组成”的折叠行。 --- ### [08:45](https://www.youtube.com/watch?v=4s9IjkMAmns&t=525s) 日期标头的乱象与无法摆脱的历史包袱根据标准规范，每封电子邮件都必须包含一个 `Date` 标头，标明邮件创建和发送的时间。虽然现代开发者通常期望类似 ISO 8601 的整洁格式，但在邮件演进史上，日期格式经历了高度碎片化和极端的自由度扩展。标准的日期格式包含星期、日、月、年、时间以及时区。然而，RFC 允许诸如军用时区单字母代码（Military Time Zone，如字母 `D`、`Z` 代表格林威治，`A` 到 `M`、`N` 到 `Y` 各自代表格林威治东西方向的偏移）。不仅如此，由于日期标头也是由一系列独立 Token 组成，CFWS 规则全面适用于日期标头——任何两个 Token 之间都可以随意插入折叠空白与注释。更反直觉的是，在日期字段内插入注释形式的说明是合法的，但圆括号注释不能为空 `()`，因为标准规定注释必须包含内容。虽然 RFC 2822 试图亡羊补牢，声称注释只应该出现在日期末尾，但由于现代系统为了兼容性依然必须能够解析符合旧标准 RFC 822 的历史邮件，导致解析器不仅无法简化，反而必须将新旧两套语法的最坏情况同时兼容进来。 - 邮件协议强制要求包含 `Date` 标头。 - 日期支持各种奇特写法，包括军用单字母时区代码（如 GMT 标记为 `Z`，其他字母代表经度时区偏移）。 - 因为日期字段内部各元素均为独立 Token，所以日期中间允许穿插大量换行、空格及嵌套注释（如 `Date: 12 (twelve) Jun 2008...`）。 - **标准语法陷阱**：在日期标头中，虽然可以随意加注释，但空注释 `()` 在语法上是非法的。 - RFC 2822 试图规定注释只能置于日期尾部，但为了向后兼容 RFC 822，解析器必须继续承受早期在日期各个组成部分之间任意乱插注释的现实。 ### [10:52](https://www.youtube.com/watch?v=4s9IjkMAmns&t=652s) 荒谬的 BNF 语法与“废弃”月份的噩梦 Ricardo SIGNES 指出，随着时间推移，描述电子邮件格式的规范开始越来越多地采用巴科斯范式（BNF）来定义数据结构。例如对于日期格式，直觉上规则应该非常清晰：一个日期由“日”、“月”、“年”组成，“日”是一到两位数字，“月”是英文中固定的 3 个字母月份缩写，而“年”则是四位数字。然而在规范里，实际情况远比想象中扭曲：一个“日”可以是“一到两位数字”或者“已废弃的日”（obsolete day）；一个“月”可以是“一月、二月等标准缩写”或者“已废弃的月”（obsolete month）。那么，究竟什么是所谓的“已废弃的月份”？Ricardo 解释道，在规范的世界里，一个“已废弃的月份”不过就是在月份缩写的两侧带上了可折叠的空白符（folding white space）。关于“废弃”（obsolete）到底意味着什么，规范给出的定义令人绝望：“任何被标记为已废弃的内容，都必须被接收并解析。”Ricardo 带着辛辣的讽刺调侃道，这简直就是“干得漂亮”的典范。此时投影屏幕甚至闪烁了一下，他顺势打趣称“连广告投影系统都在试图保护大家免受这些糟糕 RFC 的精神污染”。到目前为止，大家讨论的甚至仅仅只是邮件头（Headers）而已，后续的内容只会更加恐怖。 - 规范通过不断向后兼容打补丁，创造出了诸如 `obs-day` 和 `obs-month` 之类的怪异规则。 - 邮件的基本整体结构看似简单：先是一堆邮件头（Headers），紧接着一行空行，随后是邮件正文（Body），全部由 ASCII 字符构成。 - Ricardo SIGNES 针对这套糟糕的规范评价道：“The term is anything that is obsolete must be accepted and parsed.” --- ### [12:22](https://www.youtube.com/watch?v=4s9IjkMAmns&t=742s) SMTP 的虚假谎言与 MIME 的艰难诞生如果说邮件头充满了疯狂的语法，那么负责传输消息的 RFC 821 同样充满了虚伪。RFC 821 取代了曾经利用 UUCP 或 FTP 发送邮件的原始机制（早期甚至有多个 RFC 专门讨论如何用 FTP 发送邮件），并将该协议命名为“简单邮件传输协议”（Simple Mail Transfer Protocol，即 SMTP）。Ricardo 毫不留情地讽刺：“821 calls this the Simple Mail Transfer Protocol, and like all the best names, it is a lie.” SMTP 的限制极其死板：单行长度不得超过 1000 个字符（包括末尾的 CR 和 LF 回车换行符），并且对于超长文本行，它完全不像邮件头那样提供任何折叠换行机制；更严重的是，它完全不支持任何 8 位字符（8-bit characters）的编码与传输。面对无法传输扩展字符的问题，早期的协议实际上给出了粗暴的“解决方案”——别用 8 位字符。Ricardo 对此对着台下大声喊话吐槽：“听到没有，欧洲？快住手吧，别再用你们那些带重音的奇怪字符了！” 为了解决多部分消息（Multipart）和 8 位内容等各种疯狂需求，MIME 应运而生。虽然通常大家认为 RFC 1341 是 MIME 的代表，但它绝非首创： - 先行者 RFC 934 表面看似简单，却在正文中堂而皇之地写着“当转发代理封装消息时，应应用以下有限状态自动机”这样的严苛要求。 - 另一个前身 RFC 1049 甚至将 Unix 下排版工具 troff 作为顶层 MIME 类型之一，Ricardo 笑称冲着这一点就能原谅它的大部分罪过。 - RFC 1341 大体还算正常，直到其第 7 节试图定义一种全新的、兼容 SGML 的标记语言用于富文本邮件格式。这套设计很快被废弃，最终由 RFC 2045 至 2049 这整整五份标准文档重新奠定了现代 MIME 体系。 ```mermaid flowchart TD accTitle: 邮件协议与标准演进路径 accDescr: 展示从早期的FTP和RFC 821到MIME标准体系的发展历程与限制 subgraph Early["早期传输机制"] FTP["FTP 邮件传输 / UUCP"] --> SMTP["RFC 821 (SMTP) 严格限制：7-bit ASCII、每行 ≤ 1000 字符"] end subgraph PreMIME["MIME 前身探索"] SMTP --> RFC934["RFC 934 引入有限状态自动机进行封装"] SMTP --> RFC1049["RFC 1049 甚至包含 troff 媒体类型"] end subgraph ModernMIME["现代 MIME 规范"] RFC934 --> RFC1341["RFC 1341 定义多部分与兼容 SGML 的富文本"] RFC1049 --> RFC1341 RFC1341 --> RFC204X["RFC 2045 - 2049 五大标准 确立现代 MIME、Base64 与编码机制"] end ``` --- ### [13:59](https://www.youtube.com/watch?v=4s9IjkMAmns&t=839s) Base64、字符集乱码与垃圾邮件过滤的漏洞要在仅支持 7 位 ASCII 的 SMTP 环境下发送包含非 ASCII 字符（例如带特殊音标的名字）的邮件，直接发送必然失败；即使传输层面强行允许，消息格式本身也不合法。为此，必须在邮件头中显式声明传输编码：`Content-Transfer-Encoding: base64`。Base64 的本质是把 8 位的字节流重新映射为人类可打印的安全 ASCII 字符序列。然而，仅仅把字节打包还远远不够。北美和使用类似字母表的欧洲开发者经常会忽视一个残酷事实：电脑里存的不是抽象字符，而是具体的底层字节。如果发信人以 UTF-8 编码输入文本并存入这些字节，而收信人的邮件客户端缺失字符集信息，一旦盲猜为 Latin-1，展示出来的就是一堆乱七八糟的乱码；如果客户端运行在经典的 Mac 系统上，用 MacRoman 编码去解析，看到的又会是另一套匪夷所思的符号。为了避免读者看到宛如“呼唤邪神降临”般的文字，邮件头还必须额外堆叠元数据：`Content-Type: text/plain; charset=utf-8`。更荒诞的是 Base64 规范本身的宽松容错机制。按照规定，解码器在解析 Base64 数据时，如果遇到任何非合法的 Base64 字符，应当“静默忽略”。这意味着发信方可以在正文的 Base64 编码中随意掺杂无关的垃圾字符，而在标准看来这依然是一封完全合法的邮件。虽然规范随后又含糊其辞地提到“在某些情况下也许可以拒绝解析”，但这发生在统一必须使用 “MUST” 和 “MUST NOT” 的 RFC 2119 之前。其导致的现实后果是： - 垃圾邮件发送者完全掌握了这一漏洞。如果编写简单的 procmail 过滤器去正则匹配某违禁词（如伟哥），若发信者在 Base64 编码的行中间隔插入噪声字符，这种简陋的过滤器就会彻底失效。 - 许多程序员在解析邮件正文时习惯简单地按行切分（split by lines），Ricardo 严肃指出：“When you write email code that parses the body by splitting up the lines of the body, you have failed, and everyone does this.” --- ### [16:36](https://www.youtube.com/watch?v=4s9IjkMAmns&t=996s) Quoted-Printable 的回车换行陷阱与 Mutt 的真实灾难如果不想使用完全不可读的 Base64，另一种更直观的编码方式是 `binary`，即直接传输原始字节。但这无法在普通的 7 位 SMTP 链路上传输。折中的方案是引入 `Quoted-Printable`（引用可打印编码）。在 Quoted-Printable 中，绝大部分普通的 ASCII 文本和标点符号均保持原貌，只有 8 位字符会被转义为类似 `=E9` 的十六进制形式。一些软件为了保险，甚至会把空格转义为 `=20`，整封邮件依然大致处于“人眼能辨认一部分”的半可读状态。但 Quoted-Printable 隐藏着一个极其危险的逻辑陷阱：它不仅仅编码底层字节，还关乎语意。正文中的真实换行符是由物理的 `CRLF`（回车换行）构成的；如果邮件作者自作聪明，将行末的物理换行转义编码成了字面的十六进制字符串 `=0D=0A`，从纯语法上看这并没有违规，但这会导致灾难性的显示差异： - 字面的物理 `CRLF` 代表展示层面的“换行”。 - 转义的 `=0D=0A` 则代表发信人想要的是原始的“CR 字节”和“LF 字节”。按照规范，邮件客户端在遇到编码的 `=0D=0A` 时，不应该将其渲染为换行，而应该忠实地把它作为一个回车控制符打印出来。知名的终端邮件客户端 Mutt 就会严格遵循规范，在屏幕上把它们活生生地渲染成 `^M` 乱码，而不是另起一行。Ricardo 叹息道，这绝对不是什么理论推演的搞笑段子：“I've encountered this bug numerous times; there is a lot of software out there in the wild that will do this to your email.” 这类设计缺陷使得处理邮件规范的工程师必须学会妥协与装糊涂，否则精神迟早崩溃。 --- ### [19:01](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1141s) 邮件头里的非 ASCII 梦魇：RFC 2047 编码词与疯狂拼贴处理完了正文的编码，新的绝望在邮件头（Headers）中降临。发信人通常希望在 `From` 和 `To` 中附带收发件人的真实姓名（例如使用双引号包裹姓名，后接尖括号包裹的邮箱地址），以便在收件箱列表中美观呈现。然而，一旦姓名中包含非 7 位 ASCII 字符（例如东欧名字中的特殊拉丁字母），整个解析链条就会立刻断裂。此时根本无法套用 `Content-Transfer-Encoding`，因为该邮件头的作用域被严格限制在邮件正文（Body），对邮件头自身完全无效。为了在邮件头中传递非 ASCII 文本，RFC 2047 发明了一种名为“编码词”（Encoded-Word）的机制。Ricardo 调侃道，这是一种由该 RFC 发明的特殊标记，形式上长得就像一个词，只不过被牢牢编码住了。其标准语法由一系列问号定界符包裹：`=?charset?encoding?encoded-text?=`。在定义编码方式的字段中： - 如果是字母 `B`，代表使用的是 Base64 编码，机制与正文的 Base64 完全一致。 - 如果是字母 `Q`，代表使用的是专用于邮件头的 Quoted-Printable 变体。Ricardo 特别警告，这个邮件头里的 Q 编码与正文里提到的 Quoted-Printable 并不完全相同，它有着自己独有的一套转义规则。更令人啼笑皆非的是现实数据库中的数据拼贴场景。假设用户数据库中某个人的姓名分成了三个字段，由于系统演化，各个字段的编码可能截然不同：名可能是 UTF-8 并采用 Q 编码的 Encoded-Word；中间名是纯 7 位 ASCII 字符，无需编码；而姓氏可能又是来自 EBCDIC 字符集并用 Base64 编码的 Encoded-Word。根据标准，把这些由完全互不兼容的字符集和编码方式拼凑在一起的词连缀成一行完整的姓名，在语法上居然是“完全合法”（totally legal）的。规范允许在词法层面上将多个互不兼容的编码碎片无缝黏合在一起，最终交由痛苦的客户端去还原，这再次坐实了整套邮件体系对开发者的折磨本质。 ### [21:32](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1292s) 编码词的空格陷阱与被破坏的 Quoted-Printable 规则 Ricardo Signes 指出，如果为了保证安全或者单纯像个“人渣垃圾邮件发送者”（scumbag spammers）一样，即使人名全是纯粹的 7-bit ASCII 字符，我们也可以选择对其进行编码。然而，直接将带空格的字符串塞进编码词（Encoded-word）是行不通的。因为在 RFC 规范中，编码词内部绝对不允许出现空格；一旦出现空格，邮件解析器就根本无法判定编码词究竟该在哪里截断。为了绕过这个限制，标准规定必须在编码词内部用下划线（`_`）来代替空格。Ricardo 调侃道，这正是这种机制极其荒唐的地方——它根本不是标准的 Quoted-Printable 编码。在标准的 Quoted-Printable 中，下划线就代表下划线自身；但到了邮件头部的编码词里，下划线却凭空变成了空格。对于这种打补丁式的妥协，Ricardo 嘲讽地说：“太棒了，让我们把它扫到地毯下面去，然后继续。” 紧接着他展示了一封被各种标准层层折磨的邮件示例：一封邮件里塞了四种不同的字符集编码，有时甚至在同一个字符串里混杂了两种字符集。在 Unicode 普及之前，要正确解码这种邮件几乎是不可能的任务。更可怕的是，邮件头还允许在字符串的任意两个部分之间插入注释（Comments），这些注释在最终渲染时会被丢弃。但邮件注释名义上是空白符（Whitespace），实质上却是有结构的空白符（Structured Whitespace），不仅支持嵌套，甚至还能在注释里面继续嵌套编码词。 - 编码词内禁止包含真实空格，必须使用下划线（`_`）代替空格。 - 标准 Quoted-Printable 中 `_` 表示下划线，但在编码词的 Q-encoding 中 `_` 被强制解释为空格。 - 邮件注释属于“有结构的空白符”，支持递归嵌套，且注释内部依然允许嵌入编码词。 --- ### [22:55](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1375s) 附件的困境：从 RFC 2046 到 9 个月的标准空白期演讲者切换到另一封邮件，展示这是一封包含“Re:”回复的原始邮件——一份附带了附件的每周通讯（Weekly Newsletter）。但在底层邮件编程的世界里，附件并不会像现代化图形界面那样自动以精美文档图标的形式弹出来。由于邮件传输协议本质上只处理文本，二进制文件必须经过编码才能传输，最常见的方式是将 Microsoft Word 文档进行 Base64 编码，并赋予其像普通邮件一样的头部结构。然而，如何为附件命名成了巨大的灾难。RFC 2046 曾提到，如果要指定附件的文件名，应当将其放入名为 `Content-Disposition` 的头部字段中，但该字段的详细规范却写着“将在未来的 RFC 中定义”，同时冷冰冰地规定该字段内“禁止使用编码词”。更荒谬的是，那个承诺中的未来 RFC 整整拖延了 9 个月才发布。在这漫长的 9 个月空白期内，开发者为了给附件起名字，只能在 `Content-Type` 头部不断增加自定义参数（例如 `title`）。这种权宜之计很快撞墙：如果文档标题中含有非 ASCII 的 8-bit 字符，现有的机制全盘瘫痪。`Content-Transfer-Encoding` 只作用于邮件正文，管不到头部参数；而参数内部又明确被禁止使用形如 `=?charset?encoding?...?=` 的标准编码词。面对这种技术死局，Ricardo 刻薄地指出当时开发者的绝望处境：“是的，这个问题毫无解决办法……你只能咬牙硬吞下去，直到 9 个月之后。” > “There's no solution to this problem, you just got to suck it up.” —— Ricardo Signes --- ### [24:39](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1479s) RFC 2231 / 2184 的折磨：参数星号编码与荒谬的拆分折行 9 个月后，RFC 2184（后演进为 RFC 2231）终于出炉，试图解决头部参数的编码与国际化问题。这套新方案引入了更加晦涩的语法：在参数名后追加一个星号（`*`），例如 `title*`，以此向解析器宣告该参数的值是经过编码的。等号后必须紧跟着字符编码（如 `utf-8`）、可选的语言标记（如 `en`），然后再跟随实际内容。更怪异的是，它没有采用标准的 Quoted-Printable（因为等号容易和参数赋值符 `foo=bar` 产生混淆），也没有使用 Base64（因为无法让人类直接阅读），而是发明了一种百分号八位组（Percent-octet）编码，空格也必须被百分号转义。随后，另一个由历史遗留机制引发的连环车祸爆发了。早期邮件为了人类易读性引入了“折行”（Folding），但随着生态发展，折行从一种可选的展示特性变成了强制约束——如果一封邮件的头部超过了 78 或 83 个字符而不折行，许多邮件中间件就会直接崩溃损坏。然而，经过百分号转义的附件参数往往极长，常规的折行规则却要求只能在空白字符处插入换行，参数内部根本没有可供折行的空白符。为了解决这个自相矛盾的僵局，标准制定者不得不推出了更加丑陋的折分机制：将一个参数手动切碎成多个带有数字索引的分片参数。Ricardo 展示了一段令人啼笑皆非的规范示例： ```text title*0*=utf-8'en'%E2%80%A6 title*1*=%20part%20one title*2*=%20part%20two title*3=plain 7bit text ``` 分片参数中，带星号的表示该分段经过了百分号编码，不带星号的（如 `title*3=`）则表示纯 7-bit 文本；只有在第一个分片（`*0*`）中声明字符集与语言，且规范明确“要求”后续分段中途不得篡改编码。Ricardo 吐槽这种拼凑出来的语法简直是对人类心智的摧残。 ```mermaid flowchart TD accTitle: RFC 2231 参数编码与分段拆分流程 accDescr: 展示长文件名参数如何通过星号标记编码类型、首段声明语言和编码，并切分为多个带索引的分段 A["长文件名或标题参数"] --> B{"行长是否超标且无空白符?"} B -- 是 --> C["切分成带索引的多个分段"] C --> D["分段 0: title*0*='utf-8'en'编码内容 (声明字符集与语言)"] C --> E["分段 1: title*1*=编码内容 (末尾星号代表百分号编码)"] C --> F["分段 2: title*2=纯7位文本 (无末尾星号代表普通文本)"] D & E & F --> G["邮件客户端拼接并还原完整参数"] ``` - RFC 2184 / 2231 采用 `param*=` 语法声明参数包含编码与语言标签。 - 使用类似 URL 的百分号编码（Percent-octet），放弃 Base64 和 Quoted-Printable 以避免混淆。 - 引入参数拆分规则：若无法在空白处通过常规机制折行，必须拆分为 `param*0*=`、`param*1*=` 等多个带序号的独立属性。 --- ### [26:54](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1614s) Content-Disposition 的荒唐扩展与 RFC 文档自相矛盾的 Bug 伴随着参数编码规范的落地，`Content-Disposition` 规范终于正式公布。它不仅能让发件人声明某部分数据是“附件”（attachment）还是行内展示内容，还允许塞入文件名（`filename*=`）。然而，规范的蔓延并没有止步于此，设计者们甚至开始试图将邮件附件变成一个分布式文件系统，允许在头部塞入文件大小（`size`）、创建时间与修改时间等各种元数据。 Ricardo 特别提醒大家注意规范中关于文件大小的极具讽刺意味的条款：头部里声明的 `size` 严格来说只是“近似值”，任何接收程序都绝对不允许将其作为真实文件大小的可靠依据，它纯粹是“建议性质”（advisory）的。甚至直接在大小字段后面跟一个括号注释也是合法的。更令人抓狂的是语法中的分号规则。按照严格语法，参数列表由分号分隔，最后一个参数的末尾绝对不能有尾随分号。然而 Ricardo 愤怒地揭露道，制定这些规范的 IETF RFC 文档本身，在一半以上的示例代码中都写错了——RFC 示例里赫然带有非法分号。面对这种滑稽的现实，Ricardo 道出了邮件系统开发最根本的无奈法则：“RFC 自己都写错了，全世界所有人都写错了，因此你绝对不能因为邮件格式写错就直接拒信；这意味着当你在解析邮件时，你必须把它们都解析正确，这就是现实生活。” > “The RFCs get it wrong. Everyone gets this wrong. So you can't reject email because it's wrong, which means when you parse email, you have to get it right. It's just a fact of life.” —— Ricardo Signes --- ### [28:28](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1708s) Multipart 邮件的内脏、边界线与垃圾邮件的终极嘲讽为了将正文和附件组装在一起，邮件系统引入了 `multipart`（多部分）类型。Ricardo 打比方说，这个过程就像是“把原来邮件的所有内脏全部掏空”，将其放进一个独立的附件分块中，然后在被掏空的外层邮件头部打上 `Content-Type: multipart/...` 标记，并声明一个用于划分不同实体的边界字符串（Boundary）。在实际运作中，边界字符串被用来界定正文实体和附件实体的开始与结束。理论上，在第一个边界之前（Preamble）和最后一个边界之后（Epilogue），是可以填充任意内容的。这部分内容游离在多部分类型之外，无法享有 `Content-Type` 定义的属性，也不能使用 `Content-Transfer-Encoding`，因此通常被假定为纯 US-ASCII 文本。在早期的理想设想中，前言部分是用来提醒那些使用远古客户端的用户：“如果你看到了这段文字，说明你的客户端不支持 MIME 多部分邮件”。 Ricardo 借此分享了他收到的“有史以来最棒的垃圾邮件”作为这一机制的荒谬注脚：那是一封推销药物的垃圾邮件，它的 HTML 分块里写着“请来买我们的伟哥”，Plain Text 纯文本分块里也写着“请来买我们的伟哥”，而在最前面的 Preamble 区域，垃圾邮件发送者留下了一句对受害者的无情嘲讽：“换一个能用的正经邮件客户端吧”（Get a capable email client）。最后，Ricardo 总结道，通过在 RFC 2231 参数中附加语言标识，以及给各种头部打上星号编码，开发者们终于拼凑出了一封五脏俱全的“完美”现代邮件：有发件人、收件人、多语种主题、格式扭曲的附件、嵌套的正文以及垃圾邮件前言。“这是我们有史以来做出的最棒的邮件，”Ricardo 带着黑色幽默长舒一口气，“现在，我们终于可以把它发送出去了。” - `multipart` 实体必须依赖唯一的 Boundary 字符串划分内容，分为前言（Preamble）、多个实体部分和结语（Epilogue）。 - `multipart` 整体顶层明确禁止声明 `Content-Transfer-Encoding`。 - Preamble 部分常被早期客户端用来提示兼容性问题，但常年被垃圾邮件发送者当做自由发挥的调侃区域。 ### [31:41](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1901s) 邮件地址规范与复杂的域名语法讲者 Ricardo Signes（rjbs）转向了他最热衷的话题——电子邮件地址。他提到曾收到一封来自用户的反馈邮件，对方在使用 Perl 模块 `Email::Valid` 进行测试时发现了一个极其古怪的地址，便质疑道：“它说这个地址是合法的，但这怎么可能是合法的电子邮件地址？”为了解答这个疑惑，Signes 现场拆解了规范邮件地址（`addr-spec`）的形式文法。他首先剔除了 `domain-literal`（域文字，形如 `rjbs@[1.2.3.4]`，用来绕过 MX 记录直接指定 IP），随后去掉了 `dtext`。在文法定义中，所谓 `atom`（原子）是指除特殊字符（`specials`）之外的任意字符集合。 ```mermaid flowchart TD accTitle: addr-spec 文法拆解流程 accDescr: 展示 addr-spec 如何拆解为 local-part 与 domain，并逐层细化到原子字符 subgraph ADDR ["addr-spec 结构"] AS["addr-spec"] --> LP["local-part (@ 符号左侧)"] AS --> AT["@"] AS --> DOM["domain (@ 符号右侧)"] end subgraph DOMAIN_TREE ["domain 文法推导"] DOM --> SD["subdomain"] DOM -.->|"0个或多个"| DOT_SD[".subdomain"] SD --> DREF["domain-ref"] DREF --> ATOM["atom"] ATOM --> CHARS["非空格 / 非控制字符 / 非 specials 字符"] end ``` 根据文法推导，一个合法的域名由一个 `subdomain` 以及零个或多个由句点分隔的 `subdomain` 组成。而每个 `subdomain` 本质上是一个 `domain-ref`，也就是一个 `atom`——即一个或多个非空格、非控制字符且非特殊字符的字符序列。Signes 指出，正因文法如此宽松，诸如带有奇怪字符甚至内部包含带引号和空格的域文字全都是完全合规的。尽管这种语法在现实中像是在“桶里射鱼”一样滑稽，但 Signes 认为规范制定者的初衷或许是希望未来如果取代了 DNS，新系统的主机名能拥有更大的灵活性。 Signes 评价道：“DNS doesn't really hate the living, it's more mild distaste.”真正的灾难并非来自域名部分，而是隐藏在 `@` 符号左侧的 `local-part`。 - `domain-literal` 形式如 `rjbs@[1.2.3.4]`，指示邮件传输直接投递而忽略 MX 记录查询。 - `atom` 由除 `specials`、空格以及 ASCII 控制字符之外的任意字符构成。 - DNS 系统对人类只是“温和的反感”，真正的恶意集中在邮件协议对 `local-part` 的处理上。 --- ### [34:14](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2054s) 恶意满满的 Local-Part 与破坏规则的奇葩地址 Signes 指出，邮件协议对活人真正的憎恨完全体现在 `@` 符号左边的 `local-part`（本地部分）上。依据标准规范，`local-part` 由一个 `word` 以及零个或多个句点连接的 `word` 组成，并且规范明确要求**保留大小写**（case-preserved）。Signes 分享了一桩令人啼笑皆非的往事：许多网站在注册时会粗暴地把用户输入的纯小写邮箱转换成全大写，而他早年写邮件软件时曾死板地假设发给自己的邮件一定是小写的 `rjbs`；当服务器收到大写的 `RJBS` 时，程序判定“这不是我”并直接把信件丢弃，导致他丢失了大量邮件。更可怕的是文法中对 `word` 的定义：一个 `word` 可以是一个 `atom`，也可以是一个 `quoted-string`（带引号的字符串）。在双引号内部，反斜杠可以转义任何字符，且允许包含空格等任意内容。这导致了许多荒谬但完全合法的地址形式： - 常规的纯字母组合； - 大多数服务支持的标签投递格式，如 `rjbs+something`； - 包含感叹号或特定特殊字符的地址； - 带引号并嵌入空格的地址（例如带有空格的用户名，这在亚马逊等网站下订单时往往会导致系统直接崩溃）； - 最恶意的形式：在双引号中直接塞入换行符（CRLF）与伪造的邮件头字段！ ``` "Ricardo To: someone-else"@example.com ``` Signes 嘲弄地指出，如果解析软件面对这样一个地址，试图判断它究竟发自何人，就会被中间换行的 `To:` 彻底玩弄：“That is two message heads no matter what your software wants to tell you, it's just another vector for abuse.”地址里仿佛包含了一个中间名叫“回车符”（Carriage Return）的人，直接构成了邮件注入与攻击载荷。 --- ### [36:53](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2213s) 地址组、完全限定域名陷阱与合规性辨析随着地址复杂度的上升，Signes 引入了前导短语（`phrase`）和鲜为人知的“地址组”（`address group`）机制。标准的完整地址形式通常是 `phrase <angle-addr>`，例如带人名的尖括号地址，多组地址间可以通过逗号和括号注释隔开。而地址组的结构则是由组名、冒号、逗号分隔的地址列表以及结尾的分号构成（如 `Technical Staff: user1@example.com, user2@example.com;`），其中甚至可以穿插类似 `(R&D)` 的注释。为了测试在场听众对 RFC 822 文法的理解，Signes 现场展示了一页充满陷阱的幻灯片，让大家找出其中到底有几个合法的邮件地址： ```text Technical Staff (R&D): rjbs.@cpan.org, http://work; Van Helmont <helmont@example.com> (The Invisible Man),, Alito <judge@domain.com.> ``` Signes 逐行剖析了其中的玄机： 1. `rjbs.@cpan.org`：不合法。句点虽然可以出现在本地部分，但**绝不能作为首字符或末尾字符**。 2. `http://work`：不合法。地址中非法混入了冒号。 3. `Van Helmont`：看似合法，但仔细看其域名部分缺失了句点，字符构成错误。 4. `(The Invisible Man)`：括号内纯属注释，根本不算地址；随后的连续逗号在规范中完全被允许（就像 Perl 一样，多余的逗号只相当于分隔了零个地址）。 5. `Alito <judge@domain.com.>`：不合法。虽然末尾带句点的 FQDN（完全限定域名）在 DNS 系统中完全有效，但在**电子邮件规范中是非法的**，禁止带有末尾的句点。 6. 整页幻灯片中，最终只有极少数严格符合语法的地址能够幸存。 --- ### [39:39](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2379s) 嵌套组噩梦与“undisclosed-recipients”的唯一合法之道在探讨了地址组后，Signes 指出了规范演进中的历史包袱：地址组的嵌套解析在 RFC 724 中是合法的，但在 RFC 822 中定义模糊，而在 RFC 2822 中则被明令禁止。尽管人们普遍认为 RFC 2821 和 RFC 2822 是更现代、更理智的标准，但 Signes 提醒大家，在 IETF 的严苛定义下它们很大程度上仍处于草案（Draft）状态，真正在现实世界中被广泛部署的坚固基石依然是古老的 RFC 821 和 RFC 822。为了保证邮件不丢失，开发者往往不得不故意宽容甚至错误地解析那些残缺的嵌套组（比如缺少了闭合分号的组结构）。紧接着，Signes 给出了全场讲座中“唯一真正实用”的技术干货：**如何发送一封没有任何收件人的合法邮件**。许多人在群发 BCC（密送）或不希望暴露收件人列表时都会遇到难题，而正确的做法正是利用空地址组： ```text To: undisclosed-recipients:; ``` 只要组名后紧跟冒号与分号，中间不放任何地址，就是 RFC 规范下生成完全合规、且不含任何可见收件人信息的唯一合法途径。Signes 戏谑地补充道：“Of course, I would suggest that if you are trying to send a message and you are doing it with something other than Mutt or Mail.app, you are losing the game already.” - 只有 RFC 821/822 是互联网广泛部署的事实基石，RFC 2821/2822 在很多老旧系统面前仍然不能完全替代前者。 - 空地址组 `Group-Name:;` 是协议标准中唯一合法允许的不包含任何具体收件人地址的邮件头表示法。 --- ### [41:53](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2513s) 提问环节：折叠邮件头（Header Folding）的未解之谜由于时间只剩最后 8 分钟，不足以展开 MIME 等庞大议题，Signes 决定将未尽内容留到下一年，并向现场观众抛出挑战：“Who dares ask a question?”一位名叫 Todd 的观众起立提问。Todd 提到自己在工作中购买了一款商业邮件列表软件，在排查故障时由于“怯生生地把脚伸进了 RFC 规范的泥潭”，结果被邮件头折叠（Header Line Folding）及行长度限制的描述彻底搞懵了，尤其是搞不清长度限制究竟适用于折叠前还是解折叠（Unfold）之后的行。 Signes 将该问题重新提炼为通俗的表达：“折叠邮件头到底是怎么一回事？”随后他解答道：在邮件头中，除了参数值等特殊场景外，邮件头的物理折叠在技术层面上是**可选的**（Optional）。虽然规范期望或建议行长保持在 72 或 79 个字符以内，但该长度计算的是整个逻辑行的长度（包含了如 `Foo-Bar-Baz:` 这样的头部字段名）。至于在何处插入回车换行和空格进行折叠，完全由实现者自行决定。Signes 在结语中表达了对这一机制的极端厌倦：“If it were up to me and everything accepted everything, I would never fold.”如果现实世界所有的客户端和服务器都能不加限制地接收超长文本，他根本绝不会去执行任何邮件头折叠操作。 ### [43:46](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2626s) 头部折叠中的空白丢失与超长头部截断困境 Ricardo Signes（rjbs）首先补充了一个关于邮件头部折叠（header line folding）的恶心细节：在对邮件头部进行换行折叠时，通常的做法是在原来有空格的地方插入回车换行符（CRLF）加一个空格，然后继续书写。然而，规范允许在换行符两边放置任意数量的空格，而解析时这些空白全部会被压缩合并成一个空格。这意味着，如果原本的内容里包含多个连续空格，折叠后再还原就会彻底丢失原始的空白格式；除非将内容逐字节进行字面八位元组（octets）编码，否则根本无法在头部中保留宽空白。现场一名听众提出了一个相关的现实案例：外部有人群发邮件，在 `To:` 头部里用逗号分隔塞了 500 个非常简单的邮箱地址。发送方客户端把这个头部折叠得漂漂亮亮，每行地址都不长。但他们公司购买的商业软件接收邮件后，执行了解析还原（unfold），把所有行合并成一条包含 500 个地址的超长字符串，紧接着软件判断“这行字符超过了我设定的单行头部最大长度限制”，随即强行将其截断，导致大量收件人丢失。听众向 Ricardo 询问协议中到底有没有单行头部长度的上限规定。 Ricardo 直言自己并不知道规范中是否存在头部的硬性最大长度限制，即使存在，在现代邮件体系中也有更核心的生存之道。他尖锐地指出，与邮件协议打交道绝不能盲信书面规则，必须完全以实用主义为准：“邮件的那些规范全都是垃圾，你不能信任它们，你几乎不可能指望完全照着规范做还能把事情办成。”在企业级或专业代码里，大家通常讨厌“管它规不规范只要能跑就行”的做法，但在邮件世界里，这就是唯一的行动法则。他建议提问者想办法阻止那款软件去破坏性地展开和截断头部，因为现实中几乎没有任何正经、有价值的邮件软件会对头部总长度施加这种机械的硬性截断。 - 头部折叠规则允许在换行符两侧添加任意空格，但还原时会压缩为单空格，导致连续空白信息丢失。 - 邮件地址列表过长时，软件若先做展开（unfold）再去校验行长度限制并截断，就会引发灾难性的数据破坏。 - Ricardo 指出协议规范在现实落地中漏洞百出，工程实现应当以实际兼容为首要考量，避免无谓的强制截断。 ### [45:57](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2757s) 邮箱地址校验：从不可能到必须妥协的工程现实现场另一位听众提问：“当老板或经理要求你给系统添加‘邮箱地址校验’功能时，最合适、最利落的反驳是什么？”这个问题引发了会场的共鸣。Ricardo 给出的第一句回应既清醒又现实：“邮箱地址验证在理论上是不可能的，但这并不意味着它不值得去尝试。”作为 Perl 社区核心模块 `Email::Valid` 的维护者（他自嘲自己并没有写这个模块，只是跟大部分邮件软件一样负责合并补丁），他结合工程经验给出了深入拆解。如果仅基于 RFC 语法（Grammar）去验证一个地址，虽然可以写出解析器，但其实并没有什么实际用途。真正的校验必须分层深入： 1. **域名层校验**：检查域名是否符合规范、该域名在现实中是否存在、DNS 查询是否有 MX 记录，以及该域名是否在已知的恶意/伪造地址黑名单中。 2. **用户名层（Local-part）校验**：这取决于系统的业务场景。Ricardo 介绍，在他当时任职的公司 Pobox（处理大量邮件中转业务），标准的策略是：如果是中转邮件，只要地址看起来勉强过得去，系统就会假定有效并直接转发；但如果是一个新用户注册并试图将该地址作为自己的账号，系统就会套用一套严格得多的规则去限制。 Ricardo 总结道，面对经理时的标准回答其实是“是的，先生，我们必须做邮箱地址校验”。因为如果你彻底放弃校验，系统就会全盘接收垃圾数据；随后系统尝试向这些荒谬的地址投递邮件，它们会一直堵塞在自己的 MX 服务器队列中，最终导致团队必须处理海量的退信解析（Bounce parsing）——而退信解析的混乱与恐怖，足以专门再开一场演讲作为本场分享的续集。因此，即便邮箱校验是一件极其恶心棘手的麻烦事，在工程上也必须硬着头皮去做。 ```mermaid flowchart TD accTitle: 邮箱地址分层校验与处理决策流程 accDescr: 展示从语法检查、域名与MX记录查询，到依据业务场景（中转或新用户注册）分流的邮箱校验机制。 A["待验证邮箱地址"] --> B{"RFC 语法规则校验"} B -->|"语法完全畸形"| F["直接拒绝 / 抛弃"] B -->|"语法合规"| C{"域名与 DNS 校验"} C -->|"无 MX 记录 / 黑名单域名"| F C -->|"域名与 MX 记录有效"| D{"业务应用场景判定"} subgraph S_BIZ ["业务处理分流"] D -->|"邮件转发 / 中继 (Pobox)"| E["宽松放行 (保障可送达性)"] D -->|"新用户注册 / 绑定账号"| G["严格限制策略 (校验格式与真实性)"] end G -->|"校验通过"| H["允许注册并投递验证"] G -->|"不合规"| F ``` ### [47:36](https://www.youtube.com/watch?v=4s9IjkMAmns&t=2856s) 附件文件名编码陷阱与演进悲剧的终章在最后几分钟的问答中，有听众询问多行折叠规则是否同样适用于附件文件名（File naming）。Ricardo 解释称，查找文件名称最标准、最规范的位置是在结构化的 `Content-Disposition` 头部中的 `filename`（或 `name`）参数里。由于它是一个参数，它必须遵循 MIME 参数编码那一套极其扭曲的机制——即使用星号标记编码（如 `filename*=`），以及使用带星号递增数字的形式（`filename*0*`, `filename*1*`, `filename*2*`）来将其强行拆分成多个部分跨行存放。如果一个文件名包含非 ASCII 的 8-bit 字符、或者非常长且无法在常规空白字符处自然换行折叠，就必须强制使用这种带星号的分片语法来拆分编码。除此以外，`filename` 参数在规范上还存在另一个极其危险的设计缺陷：规范规定它理应只能指向一个“叶子文件名”（Leaf file，即不包含路径的文件本身），但在协议传输层面，规范根本没有机制限制发送端塞入带有绝对路径或相对路径（如 `../`）的恶意字符串。因此，接收端软件必须在代码里极其小心地做清洗，防止写盘时发生路径遍历，把系统原有的重要文件直接覆盖抹除。在紧张而快速地剖析完这些怪异标准后，现场工作人员提醒 Ricardo 演讲时间已到。Ricardo 迅速结束了对邮件协议深渊的控诉，向观众致谢，全场在对这一连串“邮件历史遗留恐怖故事”的感慨与掌声中落下帷幕。 - 附件文件名应置于 `Content-Disposition` 的 `filename` 或 `name` 参数中，长文件名与非 ASCII 字符必须依赖 `*0*`, `*1*` 语法分片与编码。 - 规范中 `filename` 仅代表叶子文件，但因缺乏协议层强制约束，软件必须自行做安全防护，防御恶意路径注入攻击。 - Ricardo 用一系列血淋淋的技术细节说明了邮件协议历史积累带来的复杂性，宣告整场关于“邮件憎恨活人”的技术控诉正式结束。 ## Notable quotes > [26:54](https://www.youtube.com/watch?v=4s9IjkMAmns&t=1614s) The RFCs get it wrong. Everyone gets this wrong. So you can't reject email because it's wrong, which means when you parse email, you have to get it right. It's just a fact of life. — Ricardo Signes > [12:22](https://www.youtube.com/watch?v=4s9IjkMAmns&t=742s) 821 calls this the Simple Mail Transfer Protocol, and like all the best names, it is a lie. — Ricardo Signes > [13:59](https://www.youtube.com/watch?v=4s9IjkMAmns&t=839s) When you write email code that parses the body by splitting up the lines of the body, you have failed, and everyone does this. — Ricardo Signes ## Mentioned Ricardo Signes, Pobox, CPAN, Email::Simple, Email::MIME, MIME::Entity, Mail::Message, Email::Valid, RFC 561, RFC 680, RFC 724, RFC 821, RFC 822, RFC 934, RFC 1049, RFC 1341, RFC 2045, RFC 2046, RFC 2047, RFC 2119, RFC 2184, RFC 2231, RFC 2821, RFC 2822, SMTP, MIME, Mutt, Mail.app, Wyse, troff, IETF ## Video description By Ricardo SIGNES (‎rjbs‎) from ABE.pm, Philadelphia.pm Date: Monday June 16, 2008 14:50 Duration: 45 minutes Target audience: Lightning Tags: brains email hate horror lovecraft mail rant warning wtf you.are.all.going.to.die zombie Email: you see it every day. It's on your desktop. It's in your servers. Through the magic of modern technology, it flows invisibly through the air and into your PDA! Your cellular phone conducts silent and arcane conversations with distant servers, speaking the ancient language of SMTP and the unknowable dialects of IMAP. Surely all this technology means progress of mankind... or does it? Now you can learn the truth: buried deep within the eldrich documents describing the slow accretion of features in the Elder Protocols lie subtle recursions and devilish ambiguities -- grammars that are not grammars, strings quietly stripped of their higher bits -- designed to slowly bend the mind of all who implement them. Now you can hear of the horrors perpetrated by those who became thrall to these documents -- and by those, too, who fought to oppose them for the sake of their own dwindling sanity. Now it can be told: Email Hates the Living! WARNING: This talk is low in career-sustaining educational content. [Source](https://www.youtube.com/watch?v=4s9IjkMAmns)

Source quotes

[26:54] Ricardo Signes: The RFCs get it wrong. Everyone gets this wrong. So you can't reject email because it's wrong, which means when you parse email, you have to get it right. It's just a fact of life. [12:22] Ricardo Signes: 821 calls this the Simple Mail Transfer Protocol, and like all the best names, it is a lie. [13:59] Ricardo Signes: When you write email code that parses the body by splitting up the lines of the body, you have failed, and everyone does this.

Source quotes

[45:57] Ricardo Signes: Email address validation is impossible, but that doesn't mean it's not worth attempting. [0:04] Ricardo Signes: The standards get it wrong, the main implementations get it wrong, and that means that doing it right is going to be wrong because you can't actually receive real email. [13:59] Ricardo Signes: When you write email code that parses the body by splitting up the lines of the body you have failed, and everyone does this. [39:39] Ricardo Signes: If you are trying to send a message and you are doing it with something other than Mutt or Mail.app, you are losing the game already. [33:43] Ricardo Signes: The real hate in email messages is the local part and the local part is everything to the left side of the at sign.

Original transcript

[0:04] so this is a talk about email um for [0:06] those of you who don't know me I'm Rick [0:08] Rick sigis uh you might know me by rjbs [0:11] on cpan or IRC or mailing lists or [0:14] flicker or I don't know on the internets [0:17] um and uh oh sorry um that's me I [0:22] maintain a bunch of email modules on the [0:24] Sean um uh a lot a lot of em modules and [0:29] uh I work for a company called Poots and [0:32] all we do is email uh and we've been [0:34] doing email since a long time ago and I [0:38] deal a lot with email so I'm going to [0:40] talk to you today about some of the [0:42] email standards that exist um the way [0:45] they govern the way that email works and [0:46] it's formatted and paying special [0:49] special attention to some of the more [0:52] Troublesome cases in handling email but [0:54] the most important thing that you need [0:56] to know about [0:57] email is that email hates you and it [1:00] wants you to suffer um this is my [1:04] favorite quote I think from all the [1:05] email [1:06] rfc's this is from the RFC that [1:08] introduces modern multiart messages um [1:13] this is strange and broke and [1:16] compatibility is always favored over [1:18] Elegance this is the understatement of [1:20] the [1:20] century um if you are wondering why you [1:23] should care about this [1:26] um you don't really need to know any of [1:29] this in this CL isn't going to teach you [1:31] uh how to do this effectively it's not [1:33] going to make you a better programmer [1:35] it's not going to teach you the right [1:37] way to construct an RFC 2183 content [1:40] disposition header um the one important [1:43] thing you might learn here is that your [1:46] code [1:47] sus and I'm I'm really serious about [1:49] this and I mean it about my own code too [1:51] if you have written code that handles [1:53] email that gets email in from the [1:55] outside world and tries to read it if [1:56] you've heard a code that makes email and [1:58] sends it to the outside world you can [1:59] did it wrong um I'm not saying probably [2:03] because I know you did it wrong and I [2:05] can tell you how I know it's impossible [2:07] to get it right the standards get it [2:09] wrong the main implementations get it [2:11] wrong and that means that doing it right [2:13] is going to be wrong because you can't [2:15] actually receive real email for God's [2:17] sake stop writing code that deals with [2:19] email um there exists a small initiative [2:23] called The Pearl email project uh we [2:26] maintain most of the code that starts [2:28] with email colon colon and bunch of the [2:30] code that starts with other stuff like [2:31] mime and mail um and last year I gave a [2:36] big talk about uh the various problems [2:38] that the proail projects code has but [2:41] the basic fundamental issue is it's it's [2:44] never going to work um so here are the [2:47] things you need to know up front there's [2:49] something called email simple it's [2:51] should you use it probably not uh email [2:54] simple is okay if you're just building a [2:55] new message and not reading one off dis [2:57] or that someone else has produced if [2:59] it's got a plain asy 7 bit body and [3:03] there's no chance that any variable [3:05] you're putting in there any input you've [3:06] gotten from anywhere has 8 Bits and it's [3:09] not a reply to an existing message and [3:11] the people you're sending it to you only [3:13] have their email address and not their [3:14] real name to put in there so if if these [3:17] are all true maybe this is a good tool [3:20] then there's email mine which is a [3:21] little better it can produce more forms [3:23] of valid email but it's still going to [3:24] get them wrong all the time and you know [3:26] what's going to happen at the other [3:28] end um there's my entity M entity is a [3:31] little more accurate than that it's kind [3:33] of got a weird interface and it's got a [3:34] lot of old scariness in it but it's a [3:36] little better and then there's male [3:38] message which is the 900lb elephant of [3:41] it's pretty life around it's a it's the [3:44] 10 ton elephant if you will of male male [3:46] handling modules it uh it it gets most [3:50] things pretty Dar right um but it's big [3:52] and scary and people object to it [3:54] because they think that email is simple [3:55] but it's not there is one more reason [3:57] you might care about this and that's [3:59] spammers [4:00] um email is so hard to parse properly [4:04] that it's really easy to abuse parsers [4:07] with malicious input so if you want to [4:09] obscure malicious mail that you're [4:10] sending to somebody all you have to do [4:12] is make completely valid mail that most [4:15] tolerable parsel still can't figure out [4:18] um I'll Point some of these out as we [4:20] go so the question is why is email so [4:24] crazy why is it so completely horrible [4:25] and insane some people think that it [4:28] goes all the way back to R RC 822 and [4:30] RFC 821 um which uh brought this madness [4:34] into the public eye in about 1982 but [4:37] this isn't true um no one's really sure [4:40] how long email has been completely [4:41] insane but if you look back to rfc's 561 [4:45] 680 email actually made a lot of sense [4:48] it was simple anybody could write a [4:49] parser there was nothing to it but then [4:52] we got RFC [4:55] 724 um and this is where things really [4:58] got crazy [5:01] it introduced this thing called header [5:02] folding and header folding was a way of [5:03] saying look sometimes I've got a header [5:05] on my message we all know what an email [5:07] header is right um so we got a header on [5:09] this message and it's really long it [5:10] won't fit on my little uh my little wise [5:12] terminal but I want to be able to read [5:14] it without scrolling over and I don't [5:16] want to write a presentation layer to [5:18] figure out how to wrap it so we're going [5:20] to put it in the data layer so if you've [5:22] got a header like this and let's pretend [5:24] now and for the rest of these slides [5:27] that uh this is a really narrow terminal [5:30] so we need to wrap this header holding [5:32] lets us say we can write this we can [5:36] find a place to put a line break and [5:38] some indentation and it means the same [5:40] thing so this and this are in the terms [5:43] of RSC 724 lexically [5:46] equivalent um but you know we don't need [5:48] to stop there if we've got a really [5:50] narrow terminal this is fine too um [5:53] there's no reason we can't write that [5:54] that's that's completely legal um is [5:56] this [5:57] ridiculous well yeah it's completely [5:59] ridiculous but it doesn't matter because [6:01] you're going to find email that contains [6:03] this you're going to find content that [6:05] contains all this kind of nonsense all [6:07] the time and what's worse half the time [6:11] you see this the person will be [6:13] completely in the wrong and half the [6:15] time you see it it'll be completely [6:17] valid and you won't be able to figure it [6:19] out because you can't figure out what's [6:21] valid and what's not you basically have [6:23] to accept [6:25] everything so going back to folding when [6:27] can we fold well we can fold basically [6:30] whenever we want uh as long as it's a [6:33] valid place for linear white space so [6:35] when is linear white space valid RFC 724 [6:38] says linear white space can be freely [6:41] inserted in reasonable [6:44] places I'm it really says that it also [6:48] says that in addition to the parser that [6:50] that reads the tokens in your email the [6:52] existence of another lexal analyzer is [6:54] assumed of [6:56] course um so anywhere you want and what [7:00] this actually turns out to mean anytime [7:02] two tokens meet anytime there are two [7:04] tokens next to each other in an email [7:06] message you may add space and that SP [7:10] any kind of space is equivalent and [7:12] there's a special kind of space called a [7:14] comment and comments [7:17] Nest when a comment is used to act as a [7:19] bimer between any two things it's [7:22] treated just like one space for the [7:24] purpose of understanding what happened [7:26] so when you're trying to understand the [7:27] content the content [7:31] this is just the same as [7:36] this or [7:41] this [7:43] this is basically and I shouldn't even [7:46] say basically this is for all intents [7:48] and purposes lexically equivalent to [7:52] that it's really good thing we added [7:54] folding right because I mean could you [7:55] imagine having to scroll to the right to [7:57] read that this is this is a lot clearer [8:01] if you ever want to see an email [8:03] programmer cry and see his eyes glaze [8:06] over with rage and [8:08] Desperation just ask him about Comet or [8:11] folding whites [8:13] space Comet or folding Whit space is uh [8:16] the name given to this monstrosity by [8:18] RFC 2822 the modern email format now it [8:22] tries to limit the effects of a cfws but [8:25] it's not very effective so for example [8:28] this is still completely allowed um but [8:31] helpfully this is not this was valid in [8:34] RC 822 and [8:36] 724 but no longer RFC 2822 says no lines [8:40] that are only Whit space so thank [8:43] you but that that's enough of that right [8:45] let's talk about some other headers we [8:47] see a lot um email gets date headers in [8:50] fact every email message is required to [8:51] have a date header that tells you when [8:53] the mail was composed in sent uh here's [8:56] a pretty standard date header you know [8:58] that's a a day in a month in a year at a [9:01] time in the time zone that's that's [9:04] pretty reasonable um you know these days [9:06] we probably won't use ISO a601 but [9:09] nobody can say that's completely [9:11] insane uh you can also write this uh [9:13] this is another form that's completely [9:15] acceptable uh you can write this if you [9:17] want too uh we added a little more space [9:19] because hey why not um this is valid too [9:22] A lot of people think this is really [9:24] bizarre and I guess I'm one of them D [9:26] there is uh the military time zone which [9:29] is something like and I'm sure I've [9:31] going get this wrong a through n are [9:35] increasing hours West of Grenich and Z [9:38] is grenwich and N through y or something [9:42] are ours to the other side of grenwich [9:44] uh so this is completely valid because [9:46] you know the military wants to use the [9:47] internet um and here's another form [9:49] totally valid we can use these things [9:52] pretty pretty completely so okay it's [9:54] silly uh it's a little too flexible but [9:56] you know writing a parser for this in [9:58] Pearl is not a huge EX [10:00] the only problem is that in addition to [10:01] these forms we can write this um because [10:04] right they're just tokens um we can we [10:08] can put comments in there um and this is [10:11] totally valid right no this isn't valid [10:14] at all comments can't be [10:20] empty okay 2822 once again steps in and [10:24] it says look this comment folding white [10:26] space thing is crazy you guys are it was [10:28] a bad idea we're really sorry you can [10:30] still have comments I guess you might [10:32] need them but you can only put them at [10:33] the end okay right they're only at the [10:36] end the only problem is if you want to [10:38] parse 2822 you still need to parse 822 [10:41] which [10:42] means yeah that's still [10:44] good and the reason is if you look at [10:46] 2822 you're going to expect a grammar [10:48] that looks something like this all the [10:49] old rfc's use sort of pseudo grammar [10:52] some of them start to become more like [10:54] BNF over time and we say you know a date [10:57] is a day and a month and a year and a [10:58] day is one or two digits and a month is [11:01] one of these three character month [11:02] abbreviations in English and a year is [11:04] four digits okay the problem is the way [11:06] it actually looks as like this a day is [11:09] one or two digits or the obsolete day [11:12] and the month is January February or the [11:14] obsolete month so you know what's how do [11:16] you obsolete a month you know what's an [11:18] obsolete well an obsolete month is a [11:20] month with common folding white space on [11:21] beside of [11:23] it and if you want to know you know is [11:27] this like how what does obsolete mean [11:29] well the term is anything that is [11:31] obsolete must be accepted and [11:35] pars way to [11:37] go um so this is a pretty bad [11:42] RFC [11:44] wow the ad system is trying to protect [11:47] you um this is a pretty bad RFC and all [11:50] we've talked about so far is the header [11:53] you know things are going to get pretty [11:55] bad aren't they this isn't going to be [11:57] good well [11:59] this is what a message looks like a [12:01] message is the headers or maybe the [12:04] headers and then a blank line and the [12:05] message text sorry that's that's going [12:07] to be really really [12:09] bad oh okay I can occy that right it's [12:13] it's a bunch of asking characters that's [12:14] that's all you need to understand so so [12:17] everything crazy is in the headers right [12:22] right 821 is all about how to send a [12:24] mail it replaces things like the UCP uh [12:27] FTP there's a lot of RFC to talk about [12:29] how to use FTP to send mail which is [12:31] pretty great uh 821 calls this the [12:33] simple male transfer protocol and like [12:35] all the best names it is a [12:40] lie INTP lines can't be more than a [12:42] thousand characters long and that's [12:44] including the trail in CR there's no [12:47] folding for along lines like we had for [12:48] headers there's no encoding for apic [12:51] characters so the solution obviously is [12:53] don't use any big characters hear me [12:56] Europe stop [12:57] it well then the ince MIM um MIM is a [13:01] way to do multi multi-art messages it's [13:03] a way to do messages that are in 8 Bits [13:04] it's a way to do all kinds of crazy [13:06] stuff um there it's it's RFC [13:11] 1341 which isn't really the first thing [13:13] that gives us M [13:17] um RC 934 comes before uh which is [13:21] fairly simple I mean it only includes [13:22] text like when the forwarding agent [13:24] encapsulates a message it should apply [13:26] the following finite State [13:28] automaton [13:29] seriously um 1049 is other prior art for [13:33] this and I'd love to say bad things [13:35] about it but one of its top level mind [13:36] Parts is trough and That Forgives a lot [13:40] um 1341 is not too bad until you get to [13:44] section 7 where they Define a new sgml [13:46] compatible markup language for making [13:48] formatted email uh this gets replaced [13:51] pretty quickly with RFC 2045 through [13:54] 2049 five rfc's defining the new world [13:57] of M [13:59] so let's say you want to send this [14:02] message well we can't right it's got [14:05] apic characters in it uh we can't we [14:07] can't send this over SMTP and even if we [14:09] could it's not a valid message because a [14:11] message body is set in bits so what we [14:14] need to do is declare that we've encoded [14:16] the body for transport and we say that [14:19] by saying here's a Content transfer [14:21] encoding we're using Bas 64 here uh Bas [14:24] 64 is just a way of taking a 8bit string [14:28] more or less and turning it into sees so [14:32] this isn't quite enough though right [14:33] because all this has given us bites and [14:35] this is something that uh a lot of us [14:38] Americans and even other Europeans who [14:40] use the Fairly similar alphabets uh can [14:42] ignore a lot because we think that it's [14:44] all the same but really this is what we [14:45] put in there we didn't put that text in [14:47] there we put these bites in um what [14:50] character said is this because if I was [14:52] typing in [14:53] utfa what hat gets put in an email are [14:55] these bites and if your email program [14:58] doesn't know the characters set and [14:59] guesses that it's La one you're going to [15:00] see this but if you're on a Mac you're [15:03] going to see this and neither of those [15:06] is a very good exhortation toward right [15:08] God so we're going to just need thr more [15:11] metadata in here and we're going to say [15:13] there's a content type which is plain [15:14] text and that it's in utf8 so now you [15:16] know how to get things back out and this [15:18] is fairly reasonable um there is one [15:20] little Port which is that this is [15:22] exactly the same [15:23] message because anytime you find [15:26] characters in your Bas 64 that are not [15:29] valid base 64 digits you quietly ignore [15:32] them um so you can just you know you can [15:35] put all the AP data you want in there as [15:36] long as you have some apent transport [15:38] mechanism and uh and it's still a [15:39] completely valid email message kind of [15:42] because it says that we have to ignore [15:44] any of these nonb 64 characters except [15:47] maybe we want to reject them under some [15:50] circumstances um you know to be fair [15:52] this does predate RFC 2119 which is [15:55] where they tell us please please say [15:57] must or must not [15:59] um what does this really mean in [16:01] practice nobody knows uh if you send a [16:03] message that has some crap in it maybe [16:05] it'll work maybe it won't maybe somebody [16:08] can read it maybe they [16:10] can't um spammers can really really get [16:15] behind this system because let's say [16:16] they want to send you this message well [16:17] any you know crappy content filter your [16:19] little prop mail filter is a drops mail [16:21] looks for lines that contained by Ager [16:23] well it's going to catch that but it [16:24] won't catch this um and and when you [16:27] write email code that parses the body by [16:30] slipping up the lines of the body you [16:32] have failed uh and everyone does this [16:36] but let's get back to the subject hand [16:37] we've got this message and we want to [16:38] encode it another way this is a really [16:40] human readable way this doesn't have the [16:42] problem of Bas 64 it has a different [16:44] problem uh We've encoded this in binary [16:47] which means it's the literal bites of [16:49] the message so this is basically what we [16:51] had before except we've explicitly said [16:52] we didn't encode it and now we can't [16:54] send it over as DP so that's no good so [16:57] let's compromise we're going to [16:57] introduce another kind of en coding [16:59] called quoted [17:02] PR this is you know we can kind of read [17:04] some of this message uh you know we can [17:06] see there's there's some words here we [17:07] see the the quote signs um only things [17:11] that we've escaped are the eight bit [17:14] characters some encoding systems or [17:17] programs that are need encoding don't [17:18] know exactly what's legal and what's not [17:20] so they go a step further and they just [17:23] you know they encode um Bunches of stuff [17:26] so it starts it's still somewhere [17:28] between base 64 and and legible uh [17:31] there's some crap in there we've encoded [17:32] the spaces now really there's no reason [17:34] we can't encode any bites we want but [17:36] there's one thing we can't do we can't [17:38] do this um I've got a message here and [17:42] we've got a bunch of line breaks you see [17:44] there's a line break uh right there [17:46] after the equals 20 bracket and a line [17:48] bre is just a crlf so I've got crlf crlf [17:52] and then back to text so you think I [17:54] could encode those right and say well [17:55] I've got CR LF CR LF [17:59] it's wrong it's not illegal and it's not [18:01] invalid it's just kind of wrong and the [18:05] reason is that in quoted principle it's [18:08] not just the bats that matter it's it's [18:10] the meaning of what you're saying see if [18:12] I put a literal crlf there it means that [18:14] I want a new line when you display the [18:15] message but if I encode the crlf it [18:17] means that I want the CR bite and the LF [18:20] bite so if your mail program is going to [18:22] get this message well this is going to [18:25] display like this because it knows a new [18:27] line but if I send you this message your [18:30] mail program is expected to display [18:34] this m mut does that and just to take a [18:38] second uh before you think that I'm sort [18:40] of having a lot of fun here I've [18:42] encountered this bug numerous times um [18:46] there is a lot of software out there in [18:48] the wild that will do this to your [18:51] email okay but let's you know let's just [18:55] take a step back these email specs suck [18:57] they have to be ignored let's [18:59] take a breath ignore this problem and [19:00] move [19:01] on Okay so we've got this message we've [19:05] encoded it it's quoted printable only [19:07] the bites have to be [19:09] coded this is this is not so bad right [19:11] now um now one issue here is all we've [19:14] got in the from and two headers are the [19:16] email addresses that we're sending to [19:17] but we want to include the people's real [19:19] names so that things show up nicely in [19:21] inbox listings so that's going to be [19:23] easy um all we basically do here is we [19:26] put quotes around the name and we put [19:28] some angle around the email address and [19:29] that's perfectly valid and we just keep [19:32] going oh [19:34] um that's not going to work because um [19:38] vlad's name there that's a that's not [19:41] seven bit characters so how do we fix [19:44] that we can't use content transfer [19:46] encoding right because that only refers [19:47] to what's in the body so we need another [19:50] way to encode stuff so we're going to [19:52] use encoded words encoded word is a new [19:55] token introduced by RFC 247 and it's [19:58] just like a word but it's [20:00] encoded that means it starts with equals [20:03] question mark it ends with equals [20:04] question mark and there's two question [20:06] marks in the [20:11] middle okay so let's deal with this [20:14] right here's this one email how every [20:16] want to send let's let's just encode it [20:21] so there we go equals question mark then [20:24] the character set then a question mark [20:27] then a single character indicating how [20:29] it has been encoded and a question mark [20:31] and then the encoded string then [20:33] question mark equals Q here uh between [20:36] some of those question marks means [20:38] quoted principle um remember I mentioned [20:41] quoted principle before yeah this is [20:43] different um this is this is the other [20:45] quoted [20:48] princial and here um we've got another 8 [20:52] bit name and we're going to encode it [20:54] some other way a different character set [20:56] different encoding the B here is base 64 [20:59] this Bas 64 is the same one so great [21:03] unfortunately um the name we actually [21:04] had in our database for this guy um we [21:08] have it in three fields and they're [21:09] encoded differently so we're going to [21:12] say well his first name is this encoded [21:14] word and then we've got this other word [21:17] which doesn't need to be encoded It's s [21:18] bit and then we've got something in [21:20] epiic um and then his email address this [21:23] is totally legal um you know his name [21:26] which is lexically one [21:29] can be in a bunch of different [21:30] incompatible formats we can stick them [21:31] all together and uh it becomes the [21:32] reader agent's [21:34] problem another thing we can do is we [21:36] can take this guy's name and even though [21:38] it's completely seven bit for the sake [21:40] of Sanity or perhaps because we're [21:41] scumbag spammers we can encode that too [21:44] um but this actually is no good anybody [21:48] have a guess it's because you can't have [21:51] a space in [21:53] there encoded words can't contain spaces [21:56] because then you don't know when the [21:58] encoded words could stop so we use an [22:01] underscore instead of a space and that [22:04] is one of the reasons that this is not [22:05] normal quoted principle in normal quoted [22:07] printable underscore is underscore in [22:09] this quoted printable underscore is [22:10] space great let's put it under the rug [22:13] and keep [22:15] going so here's our message got four [22:19] encodings in this message sometimes [22:20] there little or four car sets sometimes [22:23] two car sets in one string before [22:25] Unicode you can never decode this but [22:27] okay this F fine it's not going to get [22:29] any [22:30] worse okay um we can put comments of [22:34] course between any two parts of a string [22:35] and those will get dropped out and did I [22:37] mention that comments uh while they are [22:39] whites space are structured Whit space [22:41] um I mean obvious right how do you know [22:44] that you can mest them um and you can [22:46] put encoded words inside of comments [22:50] so okay um here's our message and uh we [22:55] can tell if I look at this message that [22:56] it's a reply to something right we see [22:58] that little subject says Ray blah blah [23:00] blah blah so what was it in reply to [23:02] here's the original message um it's the [23:05] Weekly Newsletter and as you can see uh [23:07] there's going to be an attachment but [23:10] unfortunately uh in the world of email [23:12] programming attachments don't just pop [23:14] onto your screen like little documents [23:15] they have to be encoded because you know [23:17] it's going to be B um so what does this [23:19] look like as an encoded document well we [23:22] encoded to look just like an email [23:23] message there's headers and there's a [23:24] body uh so these headers we put it in [23:27] Bas 64 it's a Microsoft Word document [23:30] and [23:32] uh RFC 2046 is going to tell us how to [23:36] give a file name um it's going to say [23:38] that if you want to say the file name of [23:40] this attachment so we kind of know what [23:41] it is we're going to put that in a [23:42] header called content [23:44] disposition to be defined in a future [23:47] RFC it does tell us that that field [23:49] can't contain encoded words but that's [23:51] about it uh that RFC won't come out for [23:54] 9 months so in the meantime uh we'll [23:57] just add some more parameters content [23:59] type now sure we we've added title here [24:02] and title is um is only going to apply [24:05] to this one content type so we have to [24:07] add it to the definition of every [24:08] content type but at least we can put a [24:09] title on this document of course there's [24:12] at least one problem here which is that [24:15] we've got eight bit characters again um [24:18] so obviously content transfering coding [24:20] is not going to help it's not in the [24:21] body and right encoded words um we can't [24:25] use those in parameters so we can't use [24:27] that equals question mark question mark [24:29] question mark question mark equals thing [24:32] so yeah there's no solution to this [24:34] problem um you you just got to suck it [24:37] up [24:39] uh until 9 months later uh 9 months [24:43] later we get RC 248 and it tells us how [24:45] to encode this first thing we need to do [24:49] is add a star is that nice put a little [24:51] star after title and that says this is [24:53] this is going to be encoded and then [24:55] after the equal sign we're going to say [24:58] it's encoded utf8 the language is [25:01] English in case you want to translate it [25:02] or something um and then we're going to [25:06] have to U basically you encoded um we're [25:10] not going to use quoted printable [25:12] because those equal signs they look a [25:13] whole lot like parameter values you know [25:15] like Fu equals bar and that's what we're [25:17] doing you want to get confused and we [25:18] don't really want to use base 64 because [25:20] you can't read that easily so we're [25:22] going to we're going to use percent [25:24] octet and of course there's another [25:26] problem right we need to encode those [25:27] spaces too so we got to take care those [25:30] spaces it's good it's stay human [25:32] readable [25:33] but another problem has come up in the [25:35] meantime um remember how we introduced [25:38] folding because we wanted things to be [25:39] human readable well everybody got so [25:43] expecting that to happen you know [25:44] everybody thought well everybody folds [25:46] headers nobody's sending me you know [25:47] headers that are 79 characters long 83 [25:51] characters long they're going to be [25:52] folded so we added that so that things [25:54] would be more human readable if you want [25:56] you can fold things well it's gotten [25:58] point at this at this time that uh if [26:00] you don't fold your header stuff breaks [26:03] so now you must fold and if we must fold [26:07] that's a pretty long line after having [26:09] to escape those [26:10] spaces but we can't fold this like a [26:13] header because you can only do folding [26:15] inside of white space header folding is [26:17] when you take a white space and you make [26:19] it bigger by putting a new line in there [26:20] and we we can't do that here okay so [26:24] let's introduce a new mechanism for [26:25] folding it's going to look like this [26:29] this title is given to you in four parts [26:31] title part zero encoded title part one [26:34] encoded title part two encoded title [26:36] part three 7 [26:39] bit if if you notice that the three it's [26:42] title star three equals there's no star [26:44] that's because it's s bit and you see [26:46] the first one the first one tells us the [26:48] encoding the language and we are we are [26:51] requested not to change that in the [26:53] process of these [26:54] parts uh finally the same right about [26:57] the same time we've got gotten this [26:58] solution content disposition is finally [27:00] explained to us we can put content [27:03] disposition say this is an attachment [27:05] it's not some other kind of content we [27:06] put in the message um the final name is [27:09] here it is it's a utf8 got the Little [27:11] Star there telling us [27:13] that you know as long as we're putting [27:15] this header in [27:15] here we could put a lot more information [27:18] in there um and frankly I think that [27:20] there's no reason that we can't just say [27:22] we're going to use bile attachments as a [27:24] file system so let's put the file size [27:26] on there modification seems a good idea [27:30] um oh it's important to note that the [27:32] size um of the message that's strictly [27:37] approximate um you're you're not allowed [27:39] to count on it it's completely advisory [27:40] and that comment that I put there is [27:42] totally legal [27:45] um now one thing I I can't do here is [27:49] put that semicolon you see this that [27:50] there's a semicolon now after that size [27:52] the variance that's not legal um your [27:55] list of parameters is Val [27:58] semicolon parameter setting semicolon [28:01] parameter setting you're done semicolon [28:04] proceeded if you wonder why I making a [28:06] point of this because it's not that [28:07] crazy it's because half the time the RFC [28:10] that defines this mentions it it gets it [28:12] wrong in the [28:13] examples the rfc's get it [28:16] wrong everyone gets this wrong so you [28:18] can't reject email because it's wrong [28:21] which means when you parse email you [28:23] have to get it [28:25] right it's just it's a fact of life so [28:28] all right let's go back to this email [28:29] let's get back to this guy here so we [28:32] wanted to attach that document right and [28:33] we've we've turned that attached doc [28:36] file into another email message so now [28:39] we need to put those two things together [28:40] somehow so we're going to we're going to [28:42] scoop at all the guts of this message [28:44] we're going to pull them all out of [28:45] there and we're going to put them in [28:46] another [28:48] attachment it's going to look just like [28:49] what we pulled out and then we're going [28:52] to put a content type on the message we [28:54] have left over which frankly doesn't [28:55] seem to have a lot in it we're going to [28:57] say that's going to be a multi-part [28:59] message and a multi-part message just [29:01] means that it's made up of one or more [29:05] other messages that form its content and [29:07] to explain how those messages are going [29:09] to show up in here we declare a [29:13] boundary so we're going to use that [29:15] boundary to say content starts here [29:18] another one here and it ends here for [29:20] example it's real simple a lot of people [29:21] use very complicated boundaries I'm [29:23] using a simple one for the sake of [29:25] explanation between the boundaries we [29:28] have email messages so we're going to [29:29] put that main message in there first [29:31] it's our plain text message containing [29:33] the the actual message the person is [29:35] sending and then we're going to put [29:37] another boundary and the attachment [29:40] right so I'd be willing to call this [29:42] pretty reasonable you know say I've got [29:44] multiple Parts there's a delimiter and [29:46] uh there's you know going to be a way of [29:48] saying that this line isn't really the [29:50] delimer if it looks like it so that's [29:52] that's not too bad of course you can put [29:55] stuff before the first delimer and after [29:57] the the last one so what is that stuff [30:00] well content type doesn't apply CU right [30:03] content type applies to the overall [30:05] things say it's multiart so we'll assume [30:07] it's plain text [30:09] and it's not going to be 8bit and in [30:12] fact you're not allowed to use content [30:14] transfer encoding on multiart messages [30:16] so it's probably us asky uh a lot of [30:19] people use that to try and do something [30:21] useful like say you know if you're [30:24] seeing this you can't really receive [30:26] email messages um the best message I [30:28] ever got that used this was a Spam and [30:31] it had an HTML part that said you know [30:33] please come byy oura and the text part [30:35] that said please come byy AR and the [30:37] Preamble which said get a capable email [30:41] client [30:43] okay um one last thing so you see in [30:47] this one of the fields that we can put [30:48] in our RFC 28 2184 encoded parameter [30:53] multiart [30:54] value is um language we're saying this [30:57] header has been written in United States [30:59] English um that's a really great idea is [31:02] that you wish you could use that [31:02] everywhere well now you can uh if we're [31:05] going to go ahead and take that message [31:07] top level message we had and let's [31:08] encode that so we don't need to so we're [31:10] just going to say this is us asky and [31:12] it's been sort of quoted print encoded [31:15] and we really want people to be able to [31:16] automatically figure out what language [31:18] the subject was well now we add this [31:20] it's great there's a star means you know [31:23] what language it [31:26] is okay so we' we've composed a message [31:29] it's got from it's got to it's got an [31:31] attachment it's got a body it's great [31:33] this is the best message we've ever had [31:35] now we're going to send [31:37] it okay so we're going to send some mail [31:41] we have a message we need to send to [31:43] someone so we need their sender and [31:45] recipient [31:47] addresses okay um I I have to digress [31:51] here because we're going to talk about [31:53] my favorite [31:54] thing [31:56] addresses email addresses I love you [32:02] so I got this an email a while ago um [32:05] guy says I was doing some testing with [32:07] email valid and I found this it says [32:09] that this address is valid but come [32:12] on come on that's not a valid email [32:15] address right well here's the grammar um [32:18] don't worry I'll walk you through this [32:21] is the grammar that governs adder speec [32:24] adder speec is the specification for an [32:25] email address um [32:28] we can ignore domain literal uh so we'll [32:31] pull that right out there domain literal [32:32] is for like to rjbs bracket 1.2.3.4 [32:35] bracket it means don't look at the MX [32:37] record follow [32:39] um so we pull that out and we basically [32:43] get rid of this we get rid of detex [32:45] which we don't need anymore and uh atom [32:48] what's an atom it's something other than [32:49] specials so here are the specials clean [32:51] that up okay so let's look at domain a [32:56] domain is a sub domain and then zero or [33:00] more sequences of a DOT and a subdomain [33:03] that seems right that mentally that [33:05] seems to me like that's about what a [33:06] domain is a subdomain is a domain ref a [33:10] domain ref is an atom and an atom is one [33:13] or more of any character other than [33:15] spaces control characters or specials [33:19] and specials are those ones we see down [33:21] there [33:23] so these are all [33:26] Val [33:31] uh if you are wondering uh why that uh [33:33] that bottom one is valid it domain [33:35] literals kind of have their own rules so [33:37] I can put some spaces in quotes and Stu [33:39] it's fine it's cool [33:43] um okay so that's it's shooting fish in [33:46] the barite um it's not really a huge [33:49] problem in practice I guess there's [33:50] probably some idea that if you replace [33:52] DNS someday you want to leave the new [33:54] host names more flexibility than the old [33:56] host names and I I can appreciate that [33:59] um so you really always do want to look [34:02] at the thing after they at sign and [34:04] figur it out so you end up using DNS and [34:08] DNS doesn't really hate the living it's [34:10] more mild distaste um the real hate in [34:14] email messages is the local part and the [34:15] local part is everything to the left [34:17] side of the app sign um so let's bring [34:20] up the right spec it's something like [34:22] this so a local part is a word followed [34:26] by zero or more sequences of a DOT and a [34:29] word and it has to be case preserved um [34:33] another great email bug look how many [34:35] times you sign up for something on the [34:37] web and I'm pretty sure everybody here [34:39] probably types their email address in [34:40] all lowercase letters look at the mail [34:42] you get back in and see how many times [34:44] is in all caps that's a violation I used [34:47] to lose mail because of that because I [34:49] had written email software to my eternal [34:52] shame and it assumed that the email [34:54] addresses were going to be to me you [34:56] know lowercase r L case J lower case b [34:58] lowercase s and here I was getting [35:00] nailed to r j BS and it was like that's [35:03] not me threw [35:04] away great okay a word all right and we [35:09] local parts are made up of words a word [35:11] is an atom or a quoted [35:13] string um a quoted string is basically a [35:17] quote and then anything other than a [35:20] quote or [35:21] backslash uh which are used to escape [35:23] the backslash escapes anything else and [35:25] the quote ends in scen and that includes [35:28] space so you can have double quotes with [35:30] spaces in the middle anything you want [35:31] in the [35:32] middle so this is valid right great this [35:37] is I I use that email address you could [35:39] send mail and I'd get it um this is [35:41] valid too A lot of people use this kind [35:43] of system and say my address plus [35:45] something else and it kind of goes to [35:46] you into a little [35:47] bucket this is valid too um I'd be [35:52] interested to see who accepts that [35:54] totally valid this is valid um I've been [35:58] places where that was your email address [36:00] and you know good luck trying to order [36:02] anything from Amazon um and if you want [36:05] to be a real [36:06] jerk yeah that's valid [36:12] too imagine you get email from that guy [36:17] and your program is going to look at [36:18] that message and try to figure out who [36:20] it's from well it's from Ricardo right [36:24] Ricardo slash buddy oh and it was to [36:28] this guy but really it was to this other [36:30] guy now there's two two heads I mean [36:32] there's not really right it's from [36:33] Ricardo rjbs which isuse me Ricardo 2 [36:36] rjbs one of the few people I've ever met [36:38] with a carriage return is his middle [36:40] name um that is two message heads no [36:45] matter what your software wants to tell [36:47] you it's just another Vector for [36:49] abuse here's some more um that's valid [36:53] right now we've introduced the phrase [36:56] the phrase is it's this thing that goes [36:59] in front of the address and it tells you [37:02] usually who the person is you're sending [37:05] to so first one a phrase in an angle [37:08] Adder totally totally reasonable next [37:11] one is just showing us two of them you [37:12] can put them together with comments I [37:14] think that's fine a third and [37:16] infrequently seen form of email [37:18] technology is the address [37:20] Group address group is a [37:23] name and then a colon and then a list of [37:26] email addresses and and then a [37:28] semicolon so this is going to a bunch of [37:31] people who are known as the technical [37:33] staff and the technical staff because [37:35] that R&D in the middle that's a common [37:37] right that goes away and it contains two [37:40] people okay here's another one let's [37:44] play a [37:45] game everybody here has been paying [37:47] attention right you've all implemented [37:48] RFC 22 compliant email address parsers [37:51] by this point iume um how many valid [37:54] email addresses are on this [37:56] slide [38:01] anyone see [38:04] two two okay what are [38:15] they okay let's walk through them first [38:19] thing we see there is an address group [38:20] ands seems to contain two addresses well [38:23] that first address that's that's no good [38:25] rjbs do at cpan you can have dots in [38:28] your local part but they can't be the [38:30] last character also can't be the first [38:33] character HTP HTP Colon work can't have [38:37] colons Right Out characters there was [38:40] Van hson van hson looks pretty [38:41] reasonable right [38:43] but what's there's no dot here that's an [38:47] apit character [38:51] sorry um that's that's not valid dude [38:54] you you can't even send this message [38:55] anymore um The Invisible Man who's not [38:58] there he's really not there he's inside [39:00] parentheses he's he's just a comment so [39:03] so what do we have now we have an [39:04] address that's kind of bogus and then a [39:06] comma and a comment and another comma [39:09] well that's cool it's just like Pearl [39:10] you can have as many commas as you want [39:12] they don't mean anything they separate [39:14] you know zero or one addresses um then [39:17] we've got a phrase for Alito um and [39:21] Alito is not valid because that fully [39:23] qualified domain name well that might be [39:25] valid in DNS but it's not valid an email [39:28] you can't have that trailer done so the [39:30] only valid email address on this slide [39:32] is that last [39:39] one you can also Nest address groups [39:43] only in RFC [39:45] 724 rfc822 doesn't really address it [39:48] very well and in 2822 it's not legal um [39:52] so this would be completely valid email [39:55] because something else you might be [39:56] interested in is people talk about RC [39:57] 2822 and RC 2821 as being the modern [40:00] more sane forms this is what we'll [40:02] adhere to um but they're actually not [40:04] accepted standards um I know the ITF is [40:08] weird some things they call accept it [40:09] some they call draft but 2822 is really [40:12] sort of pretty draft status even though [40:14] we try to stick to it you really need to [40:16] stick to 821 and 822 because those are [40:18] really widely deployed standards so this [40:22] seems like something we actually need to [40:23] try and parse and we're going to fail [40:26] cuz it has mold multiple failures but we [40:28] need to keep going because somebody's [40:30] going to send you this message [40:31] somebody's going to send you this [40:32] message you're going to think you're [40:34] supposed to accept it because you're [40:35] going to mispar the thing intentionally [40:38] because you want the mail of course this [40:42] actually isn't valid because we've [40:44] nested those address groups right an [40:46] address group is a name and colon and [40:48] then a bunch of stuff and then a [40:50] semicolon and we' got a nested group but [40:52] only ended with one semicolon there so [40:54] we need to have that or we're [40:58] uh and that means there's one other [41:00] group and this is actually here's the [41:02] one useful thing you guys can all take [41:04] out of this [41:07] talk that's valid uh it's valid and it's [41:11] actually useful um module of the name of [41:14] the group undisclosed recipients call [41:16] whatever you want that is the only way [41:18] to produce a valid email message that [41:21] does not have any [41:24] recipients so if you're trying to send a [41:26] message and you want you know BCC [41:28] everybody in the message or you are [41:30] trying to send a message and you know [41:33] you don't want to have a header that has [41:35] a person's name or it's expected that's [41:37] what you have to do of course I would [41:39] suggest that if you are trying to send a [41:40] message and you are doing it with [41:42] something other than M or mail app you [41:46] are losing the game [41:48] already okay so back [41:53] top I like what 8 minutes I I don't I [41:58] don't think that's enough so I think you [41:59] guys are going to have to wait until [42:00] next year um we'll talk about& then and [42:04] I'm going to get it short [42:07] um who dares ask a [42:09] question [42:11] yes to Todd is it Todd it is Todd [42:15] um so at work we bought an application [42:20] email list and I was having a problem [42:24] with it so I timidly stuck my toe into [42:27] rfc's got thoroughly confused maybe you [42:31] can don't [42:34] bite [42:37] go header line [42:40] folding um there also seems to be some [42:43] statement about length of header but [42:46] it's confusing as to whether that length [42:48] applies after you've unfolded the [42:52] line okay so I'm going to I'm going to [42:55] restate your question completely [42:56] differently please what's the deal with [42:58] header line folding um [43:02] sorry it's something like this to to [43:05] give you more specifics on how it works [43:07] folding header [43:09] lines other than parameter [43:12] values is [43:14] optional you are expected to do it I [43:18] think the spec probably says something [43:19] like you know wherever um something like [43:23] at 72 characters or 79 characters and [43:26] that refers to the entire logical length [43:29] of the line which means that if you have [43:31] a header name that's fu- bar- baz I'd [43:35] count that um but like I said it's [43:37] totally optional so where you put the [43:38] break is up to you [43:41] um if it were up to me and everything [43:45] accepted everything I would never folded [43:46] my header lines another issue that comes [43:49] up that I that I didn't mention is that [43:52] when you fold a header line uh you [43:54] generally do that by finding a place [43:56] where there wasn't space and you put a [43:58] character turn and then a space and you [44:00] keep going but you can put as much space [44:02] on either side of that character turn as [44:03] you want and it gets compressed down [44:06] into one space and that means that if [44:08] you had something where you needed more [44:10] than one space you've lost it uh there's [44:14] no way to ensure wide amounts of space [44:16] and headers other than by encoding the [44:18] literal [44:20] octets did you there was the more of [44:22] that um the what I'm interested in is [44:25] slightly different um somebody somewhere [44:29] out there in the world CS an email [44:31] message to a very you know a simple [44:34] uncomplicated email address comma [44:37] another one and they've got 500 email [44:41] addresses on the two line and their [44:43] application has folded it so there's [44:46] email address fold address and so this [44:49] this goes on down so nice short Lines no [44:52] whatever and the application we bought [44:54] buys it brings it in and says oh [44:57] this is folded header line I'm going to [44:59] unfold it and so it unfolds and gets one [45:02] long string 500 addresses in it and then [45:05] it says oh this is longer than something [45:07] I believe to be the maximum length of a [45:09] header line so I'm going to truncate [45:11] that's an interesting question I don't [45:12] know of any maximum length for a header [45:14] there might there very well may be one [45:16] um but if there is one I don't know of [45:19] it and I would suggest that in modern [45:21] email another point to to Wayne [45:23] pragmatic for a moment um the point with [45:27] email the specs are all crap you can't [45:29] trust them you can't you can barely deal [45:30] with the specs and expect anything to [45:32] work what you really need to do is do [45:34] whatever works and I kind of hate doing [45:36] that in like you know Enterprise [45:37] professional code but with email that's [45:40] that's the Mandate so what I would [45:41] suggest is do something to keep that [45:44] software from unfolding it and breaking [45:46] it I don't think there's any real limit [45:49] to length of headers and if there is I [45:51] would suggest that very little [45:53] worthwhile software enforces it anybody [45:57] yes what is the appropriate Snappy to a [46:01] manager who asks you to add email [46:03] address validation to [46:05] an yes [46:07] sir um email email address validation is [46:13] impossible but that doesn't mean it's [46:15] not worth attempting um I maintain email [46:18] col colon Val I didn't write it I just a [46:21] lot of email software I just I just [46:22] apply the patches um but I can tell you [46:24] that it's quite good um like I I said if [46:27] you try to validate based on the grammar [46:30] you can you can do it but it doesn't [46:32] tell you a lot but you can go further [46:33] right you can say well what is the [46:34] domain okay is that a valid domain does [46:36] that domain name exist uh does it have [46:37] an MX record uh is it in our list of you [46:40] know bogus known addresses and for the [46:42] local part you know I like I said where [46:44] I work at poo we deal with a lot of mail [46:47] and the the general standard is whatever [46:50] comes in if we're just going to forward [46:51] it on we will generally try and assume [46:53] okay it's it looks good enough we'll [46:55] keep sending it and if somebody says I [46:57] want this email address like I want to [46:59] use this as my username we apply a much [47:02] stricter set of rules um so like with [47:05] everything else it depends on what [47:06] you're going to do if you're getting a [47:07] user who wants to sign up and you think [47:08] their address is crap and you're not [47:09] going be able to send to it you know [47:12] limit limit as much as you can within [47:14] reason uh but I think that this the snap [47:17] responsibility is yes sir you have to [47:18] you have to validate email addresses if [47:20] you don't validate them you will accept [47:22] garbage you will try sending a mail it [47:24] will sit on your mx's you'll have to [47:27] much more bounce parsing uh which is a [47:29] whole other sequel to this uh so it's [47:32] definitely worth doing even though it's [47:33] pain in the butt anyone [47:36] else yes [47:55] justs so is multi-line toe apply to file [47:58] naming file [48:00] names the most correct place to find a [48:02] file name is in the file name or name [48:06] file name parameter to the structured [48:08] content disposition header it is a [48:11] parameter which means it does the [48:13] encoded with an asterisk thing and it [48:15] does the split into multiple Parts with [48:16] an asterisk thing so if you have a very [48:18] long header uh you must cancode it if it [48:21] has 8 bit you must split it across lines [48:23] if it is splitable on white space and [48:26] it's not you must do the star zero star [48:28] one star two um other interesting things [48:30] about the file name field it's only [48:32] supposed to refer to the [48:34] leaf it's only supposed to refer to the [48:36] Lea file um and it's uh it can't be [48:40] restricted that way so your software has [48:42] to make sure you don't blow things away [48:44] okay I'm being told like over time uh [48:47] thank you very [48:55] much