OpenAI Demos GPT-4 Turning Paper Sketch Into Functional Website

GPT-4 Developer Livestream — OpenAI

OpenAI

OpenAI President Greg Brockman argues that GPT-4 functions as an augmenting tool rather than an autonomous replacement, requiring developers to provide fresh context and oversee execution.

AI-assisted summary of GPT-4 Developer Livestream — original source. Check the original for full context.

Source published:

Key ideas

  • OpenAI showcased GPT-4 in a live developer demo, highlighting improvements in complex reasoning, dynamic steerability via Chat Completions, and an expanded 32,000-token context window. (0:08)
  • The model features multimodal vision capabilities demonstrated in preview, including converting a hand-drawn paper wireframe into functional HTML and JavaScript code. (15:12)
  • GPT-4 parsed approximately 16 pages of dense 2018 statutory tax code to correctly calculate standard deductions and tax liability using internal arithmetic rather than an external calculator. (18:54)
  • OpenAI open-sourced its Evals testing framework, inviting the developer community to contribute benchmarks and failure cases to guide future model development. (4:40)

Detailed notes

## Background Published on YouTube by OpenAI on March 14, 2023, this developer livestream features OpenAI President and Co-Founder Greg Brockman introducing the newly released GPT-4. According to the page description, the session serves as a practical developer demonstration highlighting the model's capabilities and limitations, incorporating real-time audience prompt suggestions from Discord.

## TL;DR Greg Brockman demonstrates the structured Chat Completions interface and contrasts GPT-4 against GPT-3.5 using challenging linguistic constraints and synthesis tasks. He showcases GPT-4 as a collaborative programming partner by building and iteratively debugging a Discord bot live, leveraging the expanded 32,000-token context window to process raw documentation and terminal stack traces. The demo also previews GPT-4's multimodal vision capabilities by converting a photographed, hand-drawn paper sketch into a functional website and interpreting visual humor. Brockman concludes by having the model parse 16 pages of complex US tax statutes to perform multi-step arithmetic, framing GPT-4 as an amplifying tool and inviting community contributions through the open-source OpenAI Evals framework.

## Atmosphere & style The presentation carries the candid, unpolished energy of an authentic live engineering workshop rather than a slick corporate keynote. Brockman speaks with grounded warmth and relatable developer honesty, recounting late-night 2:00 a.m. pager alerts during training runs and taking runtime exceptions, Jupyter event-loop bugs, and mobile Discord hiccups in stride. Watching the demo feels intimate, participatory, and genuinely experimental as Brockman fields live audience suggestions, marvels openly at the model's internal step-by-step reasoning, and champions human oversight alongside AI acceleration.

## Bulletin - [00:08](https://www.youtube.com/watch?v=outcGtbnMuQ&t=8s) OpenAI showcased GPT-4 in a live developer demo, highlighting improvements in complex reasoning, dynamic steerability via Chat Completions, and an expanded 32,000-token context window. - [15:12](https://www.youtube.com/watch?v=outcGtbnMuQ&t=912s) The model features multimodal vision capabilities demonstrated in preview, including converting a hand-drawn paper wireframe into functional HTML and JavaScript code. - [18:54](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1134s) GPT-4 parsed approximately 16 pages of dense 2018 statutory tax code to correctly calculate standard deductions and tax liability using internal arithmetic rather than an external calculator. - [04:40](https://www.youtube.com/watch?v=outcGtbnMuQ&t=280s) OpenAI open-sourced its Evals testing framework, inviting the developer community to contribute benchmarks and failure cases to guide future model development.

## Key takeaways - [00:08](https://www.youtube.com/watch?v=outcGtbnMuQ&t=8s) Bringing GPT-4 to release required a two-year push involving rebuilding OpenAI's distributed training stack and conducting extensive safety and capability evaluations. - [00:08](https://www.youtube.com/watch?v=outcGtbnMuQ&t=8s) The Chat Completions interface uses structured roles (system, user, assistant) to allow developers to establish persistent behavioral guardrails resistant to user overrides. - [02:42](https://www.youtube.com/watch?v=outcGtbnMuQ&t=162s) GPT-4 successfully satisfies strict token-level combinatorial constraints, such as alliterative summarization, where GPT-3.5 fails or aborts the task. - [04:40](https://www.youtube.com/watch?v=outcGtbnMuQ&t=280s) The model incorporates qualitative and conversational negative feedback dynamically without losing the context of the underlying task. - [04:40](https://www.youtube.com/watch?v=outcGtbnMuQ&t=280s) OpenAI open-sourced the Evals framework so the broader developer community can submit evaluation suites and help stress-test model behavior. - [06:22](https://www.youtube.com/watch?v=outcGtbnMuQ&t=382s) Instructing GPT-4 to write out pseudo-code before generating executable code breaks complex architectural logic into verifiable, smaller steps. - [06:22](https://www.youtube.com/watch?v=outcGtbnMuQ&t=382s) Providing new documentation and API specifications directly in the prompt context allows GPT-4 to code against software libraries released after its training cutoff. - [08:21](https://www.youtube.com/watch?v=outcGtbnMuQ&t=501s) GPT-4 can parse raw terminal stack traces and runtime errors to resolve bugs without requiring explicit natural-language explanations. - [10:23](https://www.youtube.com/watch?v=outcGtbnMuQ&t=623s) GPT-4 handles interleaved text and image inputs, with the multimodal capability developed in a closed preview alongside launch partner Be My Eyes. - [12:34](https://www.youtube.com/watch?v=outcGtbnMuQ&t=754s) An expanded context window of up to 32,000 tokens enables the model to locate and extract specific implementation details across sprawling technical documentation. - [15:12](https://www.youtube.com/watch?v=outcGtbnMuQ&t=912s) GPT-4 Vision can interpret the layout and handwriting of a photographed paper sketch to synthesize working front-end code and interactive scripts. - [18:54](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1134s) The system navigates complex statutory law and reconciling amendments across dozens of pages of tax code to compute correct legal deduction figures. - [21:24](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1284s) Multi-step arithmetic and numerical problem-solving are performed directly within the model's weights rather than through an external code execution tool or calculator. - [22:19](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1339s) A single unified model seamlessly transitions between specialized analytical domains, programming, and creative generation like rhyming poetry.

## Detailed walkthrough ### [00:08](https://www.youtube.com/watch?v=outcGtbnMuQ&t=8s) The Long Road to GPT-4 and the Chat Completions Interface

Opening the livestream, OpenAI President and Co-Founder Greg Brockman reflects on how surreal the moment feels. While OpenAI had been laying the technical groundwork for this class of system since the company’s inception, the team spent the prior two years locked in a concentrated push specifically to deliver GPT-4. Bringing the model to life required rebuilding their entire distributed training stack from the ground up, executing the training run, mapping its capabilities and safety risks, testing behaviors under real-world conditions alongside external partners, and heavily tuning and optimizing the system for developer release. Brockman emphasizes that the goal of the demo is not to present a flawless artifact, but to show developers how to make the model shine, navigate its residual weaknesses, and treat it as a collaborative partner. Viewers are invited to drop prompt ideas into the `#gpt4-demo-suggestions` channel on the OpenAI Discord server.

To illustrate how GPT-4 is controlled, Brockman opens the Chat Completions playground, an interface OpenAI had launched just two weeks earlier. Rather than taking an undifferentiated wall of raw text, the Chat Completions format divides inputs into explicit conversational roles: the `system` message, `user` messages, and `assistant` outputs. Brockman explains that the system prompt allows developers to steer model behavior with unprecedented fidelity, establishing guardrails and personas that the model will actively prioritize over user overrides.

> “And the way to think of it is that we're moving away from sort of just raw text in, raw text out, where you can't tell where different parts of the conversation come from, but towards this much more structured format that gives the model the opportunity to know, well, this is the user asking me to do something that the developer didn't intend, I should listen to the developer here.” — Greg Brockman

- **Timeline context:** The push to build, harden, and evaluate GPT-4 spanned two full years of continuous infrastructure overhauls and post-training refinement. - **Interaction model:** Moving to explicit `system`, `user`, and `assistant` channels structurally differentiates developer configuration from end-user inputs, helping protect system guidelines against adversarial hijacking.

---

[02:42](https://www.youtube.com/watch?v=outcGtbnMuQ&t=162s) The Constrained Summarization Challenge: GPT-3.5 vs. GPT-4

Brockman recalls the grueling atmosphere of training runs: sitting up at 2:00 a.m. when the pager alerts go off, patching issues, and constantly worrying whether the immense compute investment will actually yield breakthroughs. During that period, every researcher on the team had a personal "pet task"—a benchmark or prompt that previous systems failed at, which they would run against fresh checkpoints to see if the model had turned the corner. For Brockman, that litmus test was constrained summarization under arbitrary linguistic rules.

```mermaid flowchart TD accTitle: Constrained Summarization Comparison accDescr: Diagram contrasting how GPT-3.5 and GPT-4 respond to alliterative summarization constraints.

subgraph Input["Input Prompt"] A["Summarize blog post in one sentence<br>where every word starts with 'G'"] end

subgraph Execution["Model Execution"] B["GPT-3.5"] -->|"Fails constraint"| C["Outputs unconstrained summary or aborts"] D["GPT-4"] -->|"Adheres strictly"| E["Generates valid alliterative sentence"] end

Input --> B Input --> D E --> F["Audience Stress-Test: 'Q'<br>Successfully synthesizes constrained response"] ```

Copying the opening section of the newly published GPT-4 blog post into the playground, Brockman instructs GPT-3.5 to summarize the article into a single sentence where every single word begins with the letter "G." The older model completely ignores the alliteration rule and simply returns standard prose. Brockman remarks, “Yeah, it kind of didn't even try. Just gave up on the task. This is pretty typical for 3.5 trying to do this particular kind of task.”

Swapping the engine to GPT-4 with the identical prompt produces an immediate, grammatically sound sentence of G-words. When the model includes the acronym "AI," Brockman playfully rejects it in the follow-up prompt: “AI doesn't count. That's cheating.” GPT-4 accepts the critique without fuss and regenerates a clean sentence without using the abbreviation. Turning to Discord for an audience-selected letter, Brockman runs a test with the letter "A" (allowing the term "GPT-4"), followed by a difficult curveball selected by chat moderators: the letter "Q." Despite the narrow vocabulary pool of English words starting with Q, GPT-4 quickly constructs a cohesive, fully alliterative summary sentence.

- **System steerability:** GPT-4 dynamically incorporates negative conversational feedback ("AI doesn't count") on the fly, regenerating output without losing context. - **Constrained reasoning:** The alliterative summary benchmark demonstrates GPT-4's ability to maintain high semantic relevance while simultaneously solving hard token-level combinatorial constraints.

---

[04:40](https://www.youtube.com/watch?v=outcGtbnMuQ&t=280s) Synthesizing Cross-Article Themes, Rhymes, and OpenAI Evals

Expanding beyond single-text transformations, Brockman tests GPT-4’s capacity for cross-document synthesis. He pastes the text of a trending Hacker News post from the day before—discussing Pynecone, an open-source Python framework for building full-stack web apps—directly into the ongoing playground session. He then prompts the model to find a single unifying theme that connects the Pynecone announcement with the GPT-4 release post.

GPT-4 initially identifies the shared objective of making advanced technology more accessible and user-friendly. Pushing the model harder, Brockman submits a bare prompt: “That was not insightful enough.” Without any additional instructions, GPT-4 reflects on its prior answer and elevates its analysis, noting that both projects focus on bridging the gap between raw, complex underlying technology and practical, real-world utility. Brockman then directs the model to distill the core GPT-4 blog post into a rhyming poem. As the model streams out couplets, it writes a line referencing OpenAI's open-source evaluation suite, prompting Brockman to give a direct shoutout to the developer community.

> “Which by the way, if you'd like to contribute to this model, please give us evals. We have an open source evaluation framework that will help us guide and all of our users understand what the model's capable of and to take it to the next level.” — Greg Brockman

- **OpenAI Evals:** OpenAI open-sourced the `evals` framework alongside GPT-4 so outside developers can contribute challenging test suites and edge cases to shape model evaluation. - **Unsupervised critique handling:** GPT-4 interprets qualitative complaints like "not insightful enough" contextually, refining its analytical depth rather than failing or repeating itself.

---

[06:22](https://www.youtube.com/watch?v=outcGtbnMuQ&t=382s) Live Coding a Discord Bot: Prompting with Pseudo-Code and Fresh Documentation

Brockman shifts from content generation to using GPT-4 as an interactive pair-programmer, demonstrating how to build a complete Discord bot live. To set up the system for success, he sets the system prompt to define the model as an AI programming assistant and issues a critical workflow constraint: the model must write out its algorithmic thinking in pseudo-code before writing the executable implementation. Brockman explains that forcing the model to draft pseudo-code first prevents it from attempting to solve multi-layered architectural problems in a single forward pass, while giving the developer a human-readable trace to inspect and guide.

> “And this approach is very helpful to let the model break down the problem into smaller pieces. And then that way, you're not kind of asking it to just come up with a super hard solution to a problem all in one go.” — Greg Brockman

The programming objective is to build a Discord bot capable of accepting both text and images using the GPT-4 API. This introduces an immediate hurdle: GPT-4’s training data cutoff was in 2021, meaning it possessed no baseline knowledge of the Chat Completions API format announced two weeks prior, nor the multimodal image input schema. Brockman solves this zero-shot by pasting the raw API documentation, response payloads, and minimal image specifications directly into the prompt context. Leveraging this in-context documentation, GPT-4 writes the pseudo-code breakdown and produces an end-to-end Python script implementing the bot.

```mermaid flowchart TD accTitle: In-Context Pairing and Debugging Loop accDescr: Workflow showing how out-of-distribution documentation and runtime errors are fed back into GPT-4.

subgraph PromptEngineering["Context Assembly"] SP["System Prompt:<br>Draft pseudo-code first"] DOCS["Pasted API Specs:<br>Chat Completions & multimodal extensions"] REQ["Bot Requirement:<br>Discord bot processing images & text"] end

subgraph Generation["Code Generation"] PC["Step 1: Explanatory Pseudo-code"] CODE["Step 2: Executable Python Script"] end

subgraph DebuggingLoop["Runtime Debugging"] ERR1["Discord Intents Exception"] -->|"Paste raw traceback"| FIX1["GPT-4 adds discord.Intents.default()"] ERR2["Jupyter Asyncio Event Loop Conflict"] -->|"State environment constraint"| FIX2["GPT-4 injects nest_asyncio & pip command"] end

PromptEngineering --> Generation CODE --> DebuggingLoop ```

- **Prompting technique:** Enforcing a two-step structure (pseudo-code first, concrete code second) improves reasoning quality on complex software engineering tasks and increases human interpretability. - **Handling cutoff limitations:** Models can operate effectively with software libraries and APIs released well after their training cutoff if the new specifications are provided directly in the prompt context.

---

[08:21](https://www.youtube.com/watch?v=outcGtbnMuQ&t=501s) Iterative Debugging: Discord Intents and Jupyter Event Loops

Brockman pastes the generated bot code into a local environment to run it, pausing to issue a candid reminder to the audience: “Don't run untrusted code from humans or from AIs.” Upon execution, the bot immediately crashes with an `AttributeError` regarding missing Discord `intents`. Brockman notes that while the model recognized the concept of bot intents (which Discord introduced in 2020), it lacked context on the specific version of the `discord.py` client installed in the local environment.

Rather than diagnosing the issue manually or writing an explanatory prompt, Brockman simply pastes the raw terminal stack trace into the chat window without any surrounding commentary. GPT-4 parses the error, identifies the missing `discord.Intents.default()` configuration, and outputs a revised, patched script.

When Brockman attempts to run the patched script inside a Jupyter Notebook, a second runtime crash occurs: `RuntimeError: This event loop is already running`. This error frequently catches Python developers off guard because Jupyter runs its own underlying `asyncio` event loop that conflicts with standard Discord bot initialization. Brockman explains the issue directly to the model: “Hey, I'm using Jupyter and would like to make this work. Can you fix it?”

GPT-4 diagnoses the nested event loop conflict, imports the `nest_asyncio` library, applies `nest_asyncio.apply()` to monkey-patch the active loop, and provides the command `!pip install nest_asyncio` directly in the notebook cell format.

- **Stack trace ingestion:** GPT-4 can diagnose and resolve programmatic bugs directly from raw runtime tracebacks without requiring natural language explanation. - **Environment adaptation:** When alerted that code is running within a Jupyter Notebook, the model understands the underlying asynchronous lifecycle constraints and injects third-party utilities (`nest_asyncio`) to bridge the runtime gap.

[10:23](https://www.youtube.com/watch?v=outcGtbnMuQ&t=623s) GPT-4 Vision Capabilities and Discord Demo Preview

Greg Brockman transitions to demonstrating GPT-4’s multimodal features by pasting a screenshot of their active Discord channel directly into the model's interface. He reminds viewers that GPT-4 is not merely a text model, emphasizing that it can flexibly accept interspersed image and text inputs much like a rich document. This visual understanding feature is shown in preview and is not yet publicly available; OpenAI is actively collaborating with an accessibility partner, Be My Eyes, to refine and mature the system before a broader rollout. Greg enters the prompt, “Hello, world. Can you describe this image in painstaking detail?”, and switches over to inspect the raw API request formatting while the model processes the query.

While waiting for the generation, Greg acknowledges that inference speed remains an engineering focus: “And now we wait um because, you know, one of the things we have to do is we have to make the system faster.” He invites the livestream viewers to submit their own images and tasks into the Discord server for the moderators to select for a live run. The model returns an intricate description of the screenshot, identifying it as the Discord application interface, noting server icons labeled GPT-4, outlining user notifications, and amusedly detailing comments from audience members telling Greg to press “Q”.

- **Input Modality**: GPT-4 handles arbitrary combinations of interspersed image and text inputs. - **Launch Partner**: Be My Eyes is working with OpenAI during the closed preview to prepare the vision feature for general release. - **Audience Participation**: Viewers are invited to submit test images and prompts directly to the `#gpt4-demo-suggestions` Discord channel.

[12:34](https://www.youtube.com/watch?v=outcGtbnMuQ&t=754s) Long Context Window and Debugging Discord Bot Intents

The demo hits a deliberate hurdle when a subsequent Discord message fails to register with the model, returning an empty string for the content payload. Greg explains that this behavior is due to an intentional curveball introduced into the setup: Discord updated its gateway requirements in September 2022 to mandate a specific `MessageContent` intent for bots to read messages that do not explicitly mention them. Because this API requirement postdates the model's training cutoff, GPT-4 could not inherently know about the change without external reference material.

```mermaid flowchart TD accTitle: Discord Bot Debugging Flow with GPT-4 accDescr: Flow diagram showing how 15000 words of Discord API documentation are ingested into GPT-4's 32k context window to resolve an unhandled intent bug.

subgraph ErrorState["Bug Encountered"] A["Incoming Discord Event"] --> B["Empty Message Content Received"] end

subgraph GPT4Resolution["Context Processing"] B --> C["Paste 10k-15k Words of Raw Discord Docs"] C --> D["GPT-4 32k Context Window Analysis"] D --> E["Identifies Missing Message Content Intent"] end

subgraph Fix["Implementation"] E --> F["Update Intent Configuration in Code"] F --> G["Bot Reconnects & Reads Full Content"] end ```

Rather than manually scouring through the sprawling documentation, Greg copies and pastes a raw, unformatted dump of 10,000 to 15,000 words of Discord documentation into the prompt alongside the bug description: “I am receiving blank message contents. Why could this be happening? How do I fix it?” Highlighting GPT-4's expanded 32,000-token context window, Greg explains that the model can parse extensive, messy documents to locate needle-in-a-haystack specifications. GPT-4 pinpointed that the message content intent was missing and explained how to enable it. Greg reflects on the balance between automated assistance and developer oversight: “I think that this is a augmenting tool, makes you much more productive, um but it's still important that you are in the driver seat and are the manager and knows what's what's going on.”

- **Context Window**: GPT-4 supports up to 32,000 tokens for long-context comprehension. - **Document Scale**: The copied documentation spanned an estimated 10,000 to 15,000 words of unformatted text. - **Root Cause**: Discord's September 2022 requirement requiring the explicit `MessageContent` privileged intent to capture unmentioned messages.

[15:12](https://www.youtube.com/watch?v=outcGtbnMuQ&t=912s) Visual Humor and Hand-Drawn Mockup to Functional Code

With the Discord bot fixed and reconnected with help from team member Boris, Greg tests the vision system on user-submitted media. The bot evaluates an image—evidently generated with DALL-E—depicting a squirrel holding a camera. Prompted to explain why the image is funny, GPT-4 breaks down the comic juxtaposition, noting that squirrels typically eat nuts and are not expected to manipulate camera hardware or imitate human behavior.

```mermaid flowchart LR accTitle: Sketch-to-Code Pipeline accDescr: Pipeline showing a hand-drawn website mockup being photographed, passed to GPT-4 Vision, and output as working HTML and JavaScript.

A["Hand-Drawn Paper Wireframe"] -->|"Smartphone Photo"| B["Discord Upload"] B -->|"Vision API Request"| C["GPT-4 Neural Network"] C -->|"Generate HTML/JS"| D["Working Interactive Joke Website"] ```

Greg then moves to an ambitious prototype: converting a crude, hand-drawn wireframe sketch of a joke website into functional software. Holding up a piece of paper with handwritten layout blocks and labels, he snaps a photograph with his smartphone and uploads it to Discord. While wrestling briefly with mobile Discord navigation, he jokes that the non-AI logistics are always the trickiest part of live presentations: “It's funny. It's always the the sort of non-AI parts of these demos that are the hardest part to do.”

Once received, the image is passed to GPT-4 with a prompt asking for the corresponding HTML code. The model interprets the layout boxes, interprets the handwritten headings, invents jokes to populate the content, and outputs fully working HTML and JavaScript. Greg renders the markup on screen to show functional interactive elements running directly from the quick paper sketch.

- **Meme Interpretation**: Correctly breaks down visual satire and incongruity in surreal or synthetic imagery. - **Paper to Software**: Converts a rough, pen-and-paper mock-up into interactive HTML and functional JavaScript. - **Core Mechanism**: Leverages next-token prediction across text and vision tokens trained on web-scale datasets.

[18:54](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1134s) Navigating Dense Regulations: 2018 Tax Code Calculation

For the final demonstration of the segment, Greg tackles a practical, real-world challenge: parsing complex statutory language from the US tax code. He emphasizes that neither GPT-4 nor he is a certified tax professional, advising that users should always consult a qualified tax advisor. Setting up the playground, Greg provides a system message framing the assistant as "TaxGPT," demonstrating how persona prompting steers model behavior toward specialized analytical tasks.

Greg pastes in approximately 16 pages of raw statutory tax code concerning standard deductions, presenting the model with a scenario regarding a married couple, Alice and Bob, who file jointly with specific income parameters for tax year 2018. To stress-test the model's reasoning, Greg works through the statutory logic manually on the fly: calculating the basic standard deduction at 200% of a baseline figure, checking exceptions, and navigating the Tax Cuts and Jobs Act cross-references that substitute $12,000 for $3,000 during tax years 2018 through 2025, ultimately resulting in $24,000.

GPT-4 outputs the exact same $24,000 figure alongside a step-by-step statutory justification. Greg reveals that when he first tried solving the puzzle himself, he spent half an hour wrestling with nested subparagraphs and circular statutory cross-references, only understanding the full legal chain after reading the model's step-by-step breakdown. He frames this synergy as the true value of foundation models: “It's not perfect, but neither are you. And together, it's this amplifying tool that lets you just reach new heights.”

- **Document Volume**: Approximately 16 pages of legal tax code ingested into the prompt. - **Target Calculation**: 2018 joint standard deduction for Alice and Bob, requiring reconciliation of multiple legislative amendments. - **Calculation Chain**: Applying 200% to the adjusted $12,000 base figure established under temporary 2018–2025 rules to reach the correct $24,000 deduction.

[21:24](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1284s) Calculating Tax Liability and GPT-4's Mental Math

Continuing the tax calculation demonstration, Greg Brockman prompts the model: “Okay, now calculate their total liability.” As GPT-4 streams its output and works through the math step by step, Brockman watches the computation unfold in real time with genuine admiration. He remarks on how impressive the model's internal computational abilities are, highlighting that it is performing arithmetic directly in its weights rather than relying on external tools. As Brockman observes, “It's not hooked up to a calculator. Like, that's another way that you could really try to enhance these systems. But, it has these raw capabilities that are so flexible.”

Brockman emphasizes the sheer versatility of having a single unified model that refuses to be siloed into a single domain. Whether interpreting statutory tax rules, generating code, or handling natural language, the system maneuvers across all of them seamlessly. As Brockman puts it, “All of these capabilities in one system that can be applied towards the problem that you care about, towards your application, towards whatever you build.”

- GPT-4 performs multi-step arithmetic purely through internal generation rather than delegating calculations to a plug-in or code interpreter. - The demonstration highlights the model's ability to interleave statutory tax law reasoning, data extraction, and numerical execution in a single prompt session.

[22:19](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1339s) A Rhyming Tax Poem and Closing Remarks

To close out the technical demonstration, Brockman adds a final creative flourish to show how easily the model pivots between rigid logic and artistic expression. He instructs GPT-4: “now summarize this problem into a rhyming poem.” Without hesitation, the model generates verses summarizing the entire complex tax scenario, prompting Brockman to joke about having generated “a beautiful, beautiful poem about doing your taxes.”

Wrapping up the livestream, Brockman thanks the audience for tuning in and expresses OpenAI's eagerness to see what developers will create with the newly released model. He gives a final call to action regarding OpenAI Evals, urging the developer community to contribute benchmarks and test cases. Making the model better and pushing it to the next tier is a collective effort, with Brockman concluding: “We think that this model, improving it, bringing it to the next level, is something that everyone can contribute to, and that we think it can really benefit a lot of people, and we want your help to do that.”

```mermaid flowchart LR accTitle: Unified Multimodal Model Capabilities accDescr: Diagram illustrating how GPT-4 processes diverse domains like code, tax logic, and poetry into end-user applications. subgraph Inputs["Task Domains"] A["Tax Code & Logic"] B["Mental Math"] C["Creative Writing"] end subgraph Core["GPT-4 Foundation"] D["Unified Model"] end subgraph Output["Outputs & Community"] E["Calculated Liability & Rhyming Summary"] F["Developer Applications & OpenAI Evals"] end A --> D B --> D C --> D D --> E D --> F ```

- Brockman encourages open-source evaluation contributions through the OpenAI Evals repository to help identify failure modes and improve future iterations of the model.

## Notable quotes > [18:54](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1134s) It's not perfect, but neither are you. And together, it's this amplifying tool that lets you just reach new heights. — Greg Brockman

> [08:21](https://www.youtube.com/watch?v=outcGtbnMuQ&t=501s) Don't run untrusted code from humans or from AIs. — Greg Brockman

## Mentioned Greg Brockman, OpenAI, GPT-4, GPT-3.5, Chat Completions API, Discord, Hacker News, Pynecone, OpenAI Evals, Jupyter Notebook, asyncio, nest_asyncio, Be My Eyes, DALL-E, Boris, Tax Cuts and Jobs Act

## Video description Join Greg Brockman, President and Co-Founder of OpenAI, at 1 pm PT for a developer demo showcasing GPT-4 and some of its capabilities/limitations.

Join the conversation on Discord here: discord.gg/openai. We'll be taking audience input from #gpt4-demo-suggestions.

[Source](https://www.youtube.com/watch?v=outcGtbnMuQ)

中文简报

OpenAI演示GPT-4将纸质草图转化为可用网站 OpenAI总裁格雷格·布罗克曼表示，GPT-4是一种增强工具而非自主替代品，需要开发者提供最新上下文并监督执行。 格雷格·布罗克曼演示了结构化的Chat Completions接口，并通过高难度的语言约束和综合任务将GPT-4与GPT-3.5进行了对比。他展示了GPT-4作为协作编程伙伴的能力，现场构建并反复调试了一个Discord机器人，利用扩展至32,000 token的上下文窗口来处理原始文档和终端调用栈轨迹。演示还预览了GPT-4的多模态视觉能力，将一张拍摄的手绘纸质草图转化为功能健全的网站，并解读了视觉幽默。布罗克曼最后让该模型解析了16页复杂的美国税法以执行多步算术运算，将GPT-4定性为一种能力放大工具，并邀请社区通过开源的OpenAI Evals框架做出贡献。 OpenAI在现场开发者演示中展示了GPT-4，重点突出了其在复杂推理、通过Chat Completions实现的动态可引导性，以及扩展至32,000 token的上下文窗口等方面的改进。 (0:08) 该模型具备在预览版中展示的多模态视觉能力，包括将手绘纸质线框图转换为可运行的HTML和JavaScript代码。 (15:12) GPT-4解析了大约16页晦涩密集的2018年法定税法，使用内部算术而非外部计算器正确计算了标准扣除额和应纳税额。 (18:54) OpenAI开源了其Evals测试框架，邀请开发者社区贡献基准测试和失败案例，以指导未来的模型开发。 (4:40)

Reading summary

During the developer launch livestream, OpenAI President Greg Brockman presented GPT-4 as an adaptable, multimodal partner that pairs an expanded context window and structured steering with improved reasoning across coding, statutory logic, and visual comprehension. Structured Steering and Constrained Reasoning (0:08) OpenAI transitioned from unformatted text inputs to a structured Chat Completions format with distinct system, user, and assistant roles. Brockman noted that this separation allows developers to establish authoritative guardrails and personas that the model prioritizes over user attempts to override instructions. This steering fidelity was shown through constrained summarization: while GPT-3.5 ignored prompts requiring every word of a summary to begin with the letter 'G', GPT-4 adhered to the constraint and successfully adapted to follow-up negative feedback, including audience-selected challenges with letters such as 'Q'. Pair Programming and Traceback Debugging (6:22) In a live software engineering demonstration, GPT-4 built a functional Discord bot that handled both text and multimodal inputs. To guide model reasoning, Brockman instructed it to output pseudo-code before drafting executable Python, preventing single-pass failures on complex architectures. Because the system's training cutoff preceded newer API specifications and Discord's privileged intent policies, Brockman provided relevant API documentation and raw runtime tracebacks directly in context. GPT-4 resolved runtime failures, including a Discord intent error and a nested Jupyter asyncio event loop collision patched with nest_asyncio, without requiring natural language explanations of the stack traces. Long-Context Synthesis and Complex Regulations (12:34) GPT-4 demonstrated an expanded 32,000-token context window capable of extracting critical specifications from dense, unstructured references. When a bot encountered an unhandled gateway update, the model parsed between 10,000 and 15,000 words of Discord documentation to pinpoint the missing MessageContent intent. Similarly, when presented with 16 pages of statutory tax code, the model adopted a 'TaxGPT' persona, correctly traced cross-references within the Tax Cuts and Jobs Act, and accurately computed a $24,000 standard deduction and subsequent tax liability for a 2018 joint return. Brockman noted the arithmetic was solved directly within the model's weights rather than delegated to an external calculator. Visual Comprehension and Multimodal Prototyping (10:23) The previewed multimodal capabilities accept arbitrary mixtures of images and text within the same input document. GPT-4 accurately interpreted visual nuance, explaining the comic incongruity of a synthetic image showing a squirrel operating a camera. In a rapid software prototyping test, the model processed a photograph of a hand-drawn website wireframe on paper and generated interactive HTML and JavaScript code that successfully rendered the joke site layout. Brockman framed GPT-4 as an augmenting tool that requires human developers to remain in the driver's seat to oversee execution, navigate residual weaknesses, and verify code safety. He also pointed out that visual capabilities remain in a limited preview with partners such as Be My Eyes while latency optimizations continue, and he urged the community to contribute edge cases to the open-source OpenAI Evals suite.

精炼阅读摘要

OpenAI总裁兼联合创始人Greg Brockman在开发者直播中展示了GPT-4的多模态理解、长上下文分析及受限逻辑推理能力，强调该系统并非无懈可击，而是旨在作为放大人类生产力的协作工具。 结构化控制与受限推理表现 (0:08) 在模型控制方式上，OpenAI推出了Chat Completions接口，将原本无差别的纯文本输入拆解为system、user和assistant三种明确角色。Brockman解释称，这种结构化格式赋予开发者更高保真度的引导能力，使系统提示设定的防护边界与角色特征能够优先于用户的覆盖指令，从而防范提示注入。在推理与遵循限制方面，Brockman以严苛的语言规则进行对比测试：面对将博客首段概括为“所有单词均以特定字母开头的一句话”的要求，GPT-3.5直接放弃规则并输出常规文本，而GPT-4不仅成功生成了符合G字母全头韵的合法句子，还能在无额外解释的负反馈下剔除缩写词，并顺利完成了针对生僻字母Q的同类限制挑战。此外，当被要求提炼跨领域文章的共同主题时，GPT-4能理解“见解不够深刻”等抽象质询，自主深化分析层次。 长上下文摄入与上下文内编程调试 (6:22) GPT-4展现了作为交互式结对编程助理的能力，但在演示中也遭遇了知识库截止于2021年的客观限制。为了构建支持文本与图像输入的Discord机器人，Brockman直接将新推出的API文档和多模态规范粘贴至提示上下文中，结合“先输出伪代码、后生成具体代码”的提示策略，促使模型将复杂架构拆解为可审查的步骤并完成开发。当程序因缺少Discord intents配置以及Jupyter内置asyncio冲突相继崩溃时，GPT-4能够直接解析终端原始报错追踪，精准补齐配置并引入nest_asyncio工具进行动态修复。在后续排查机器人读取消息为空的问题时，Brockman利用GPT-4支持高达32,000个token的长上下文窗口，直接喂入约10,000至15,000词未经整理的Discord官方文档；模型从中准确定位出2022年9月新增的MessageContent特权意图要求，展现出在海量文本中定位关键技术细节的能力。 视觉理解与草图转代码实现 (10:23) 在多模态能力的测试中，Brockman展示了尚处于预览阶段的视觉功能。该功能允许交错输入图像与文本，OpenAI正与无障碍合作伙伴Be My Eyes合作对其进行完善。在解析Discord界面截屏时，模型能详尽辨识出服务器图标、通知区域以及聊天室内观众的互动文字；在解读一幅松鼠手持相机的生成图像时，模型能准确剖析出动物习性与操纵人类精密硬件之间的荒谬反差，展现出对视觉幽默与讽刺的理解力。最为显著的跨模态应用是将手绘草图转化为功能性软件：Brockman使用手机拍摄了一张绘制在纸上的笑话网站线框图并上传，GPT-4不仅解析了手绘布局框和手写标题，还自主编写笑话文本以填充内容，最终生成了可直接在浏览器中交互运行的完整HTML与JavaScript代码。 法定税则推理与内部算术运算 (18:54) 针对现实世界中高度复杂的专业文本，Brockman将约16页有关标准扣除额的美国原始税法法条输入模型，并构建了名为“TaxGPT”的角色提示。面对2018纳税年度已婚联合申报人Alice与Bob的扣除额计算场景，GPT-4成功理清了法条中错综复杂的交叉引用与法案修正案，准确识别出《减税与就业法案》在2018至2025年期间将3,000美元基数临时替换为12,000美元的规定，并按照200%计算得出正确的24,000美元扣除额。在随后的总纳税义务计算中，模型全程依赖内部权重进行多步心算，未接入外部计算器或代码解释器。Brockman指出，尽管仍需专业税务顾问核验，但模型将法定规则推导、数据提取与数值计算融为一体的泛化表现，证明了单一通用底座跨领域解决实际问题的潜力。 Brockman强调，GPT-4在推理速度和偶发错误上仍面临工程挑战，开发者在使用代码与专业建议时仍需保持最终把控与审查。为推动系统的持续迭代与安全性评估，OpenAI呼吁社区通过开源的OpenAI Evals框架贡献测试用例与边缘场景，共同发掘模型缺陷并提升其能力极限。

Detailed notes — zh

## Background 该视频由 OpenAI 于 2023 年 3 月 14 日在 YouTube 平台发布。页面简介指出，OpenAI 总裁兼联合创始人 Greg Brockman 主持了这场面向开发者的演示直播，展示 GPT-4 的核心能力与局限性，并通过 Discord 实时收集观众的互动创意与测试建议。 ## TL;DR 在直播中，Greg Brockman 全面演示了 GPT-4 的多维能力跃迁，包括更强的可控性与严苛的首字母生成约束、长达 32k tokens 的超长上下文处理，以及跨文本的概念整合。演示核心涵盖了利用最新外部文档与 GPT-4 进行结对编程排障构建 Discord 机器人，并首次公开展示了其视觉多模态能力——直接将草稿纸上的手绘线框图转化为带 JavaScript 交互的可用网页。最后，Brockman 通过投喂 16 页复杂税法文本展示了“TaxGPT”的分步推导与原生心算能力，并呼吁全球开发者共同为开源评测项目 OpenAI Evals 贡献用例。 ## Atmosphere & style 整场直播洋溢着极客探索与实战开发交织的兴奋氛围，Brockman 的讲解真诚、从容且脚踏实地。面对代码报错和误发图片的突发状况，他以幽默自嘲化解尴尬，坦然展现系统尚需完善的真实切片。整场演示既有见证手绘原型秒变网页的惊艳瞬间，又始终贯穿着“人类坐在驾驶座、AI 充当能力放大器”的理性务实基调，让观众在极具参与感的互动中深切体会到技术落地的无限可能。 ## Bulletin - [00:08](https://www.youtube.com/watch?v=outcGtbnMuQ&t=8s) OpenAI 举办 GPT-4 开发者直播演示，展示了历时两年打磨的全新多模态大模型，涵盖视觉理解、代码协同、长文本分析与逻辑心算等核心突破。 - [00:08](https://www.youtube.com/watch?v=outcGtbnMuQ&t=8s) GPT-4 引入高达 32,000 tokens 的上下文窗口与全新的 Chat Completions 结构化消息体系（System/User/Assistant），极大提升了全局约束与抗注入可控性。 - [16:11](https://www.youtube.com/watch?v=outcGtbnMuQ&t=971s) 视觉功能处于预览阶段，演示中展示了将草稿纸上的手绘线框涂鸦一键转化为具备交互功能的完整 HTML/JS 网页应用。 - [18:54](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1134s) 在无需外挂计算器或微调的情况下，GPT-4 仅凭 System 提示词设定 TaxGPT 角色，即可消化 16 页复杂税法条款并准确推导出 2018 年 24,000 美元的标准扣除额。 ## Key takeaways - [00:08](https://www.youtube.com/watch?v=outcGtbnMuQ&t=8s) GPT-4 历经两年研发，团队彻底重构了底层训练栈，并通过真实场景测试与行为调优将其推向实用化。 - [00:08](https://www.youtube.com/watch?v=outcGtbnMuQ&t=8s) Chat Completions API 引入 System 角色消息，使开发者能够确立不可逾越的全局规则，显著提升模型可控性。 - [02:42](https://www.youtube.com/watch?v=outcGtbnMuQ&t=162s) 相比 GPT-3.5 面对极端指令容易直接放弃，GPT-4 展现出严苛遵循首字母（如全 G、全 Q）约束生成完整总结的能力。 - [04:40](https://www.youtube.com/watch?v=outcGtbnMuQ&t=280s) 在结对编程中倡导“先伪代码后实现”的思维链工程原则，将复杂任务拆解以提高准确性与逻辑可解释性。 - [07:31](https://www.youtube.com/watch?v=outcGtbnMuQ&t=451s) 凭借出色的上下文学习能力，GPT-4 能直接消化超出其训练截断期的外部 API 新文档并编写可用代码。 - [07:31](https://www.youtube.com/watch?v=outcGtbnMuQ&t=451s) 面对 Discord intents 报错与 Jupyter 异步事件循环冲突，GPT-4 无需提示核心逻辑即可通过错误日志自主定位并精准排错。 - [10:23](https://www.youtube.com/watch?v=outcGtbnMuQ&t=623s) GPT-4 支持图文任意穿插的复合输入，不仅能拆解 UI 界面细节与读取图像文字，还能敏锐理解幽默反常识图片背后的笑点。 - [10:23](https://www.youtube.com/watch?v=outcGtbnMuQ&t=623s) 多模态视觉功能尚在预览阶段，OpenAI 正与公益团队 Be My Eyes 合作优化打磨，并在积极推进推理速度提升。 - [12:35](https://www.youtube.com/watch?v=outcGtbnMuQ&t=755s) 32,000 tokens 的超长上下文允许用户一把喂入 10,000 至 15,000 词的原始技术文档，模型能如大海捞针般迅速定位接口权限变更并解决空内容报错。 - [12:35](https://www.youtube.com/watch?v=outcGtbnMuQ&t=755s) AI 是人类工作者的“放大工具”与助手，人类仍需保持在驾驶座上负责审查代码与把控全局方向。 - [16:11](https://www.youtube.com/watch?v=outcGtbnMuQ&t=971s) 多模态能力突破软件原型开发边界，手机抓拍的简陋草稿纸线框原型可瞬间生成包含完整前端逻辑与真实笑话内容的单页应用。 - [18:54](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1134s) 无需额外微调，通过 System Message 即可引导模型化身 TaxGPT 深入消化 16 页繁琐税法原文，准确处理时间限制与交叉引用条款。 - [21:24](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1284s) 在不连接外部计算器或代码沙盒的前提下，GPT-4 凭借其底层神经网络表征即可完成复杂的税额逻辑心算推演。 - [22:19](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1339s) OpenAI 呼吁开发者社区积极向开源评估基准 OpenAI Evals 贡献多样化测试用例，共同推动模型的迭代进化。 ## Detailed walkthrough ### [00:08](https://www.youtube.com/watch?v=outcGtbnMuQ&t=8s) GPT-4 的历程与可控性系统消息 OpenAI 总裁兼联合创始人 Greg Brockman 开启了这场 GPT-4 开发者演示直播。他由衷感叹道：“Honestly, it's kind of hard for me to believe that this day is here.” OpenAI 从创立伊始就在构建这类技术，但在过去的两年里，全团队全神贯注于交付 GPT-4。这项工程包括彻底重构整套训练体系（training stack）、实际训练模型、摸索其能力边界与潜在风险、与合作伙伴在真实场景中联调测试、反复调优模型行为，直至最终将其实用化上线。Brockman 明确了直播目标：不仅要展现如何让 GPT-4 绽放光彩并榨取其最大潜能，还要坦率展示其薄弱之处和尚未完善的部分，帮助开发者将其视为一个得力的搭档与工具。 为了说明模型的飞跃，Brockman 引入了研发过程中的一个内部小故事。在漫长的训练阶段，工程师们总是在凌晨两点面对警报（pager）排查修复，心里不断犯嘀咕：这些付出究竟会不会奏效？每个人心中都有一个用来检验模型是否真正蜕变的“专属考题”（pet task）。随后，Brockman 打开了两周前刚上线的 Chat Completions Playground，展示新的交互架构。与以往直接“纯文本进、纯文本出”的黑盒机制不同，新的聊天补全接口引入了结构化的消息分层（System、User、Assistant），让模型能够清晰界定指令来源，极大地提高了模型的可控性（steerability）。 ```mermaid flowchart TD accTitle: Chat Completions 消息交互架构 accDescr: 展示 System、User 与 Assistant 角色在 Chat Completions API 中的流转关系 subgraph PromptStructure["Chat Completions 结构化消息"] SYS["System Message<br/>定义角色行为与全局约束"] USR["User Message<br/>输入文章正文与具体任务"] end subgraph LLM["GPT-4 推理引擎"] STEER["遵循开发者系统约束<br/>优先执行全局规则"] end subgraph Output["输出层"] AST["Assistant Message<br/>返回严格符合约束的生成结果"] end SYS --> STEER USR --> STEER STEER --> AST ``` - 训练与优化周期长达两年，涵盖重构训练栈、能力探索、安全风险排查及真实场景调优。 - 直播开放了 Discord 互动频道（discord.gg/openai 中的 `#gpt4-demo-suggestions`）收集观众灵感。 - 结构化 API 允许开发者通过 System 消息建立牢固规则，防止用户输入恶意覆盖开发者意图。 ### [02:42](https://www.youtube.com/watch?v=outcGtbnMuQ&t=162s) 严苛首字母约束总结与 Discord 观众互动 Brockman 演示了那个“GPT-3.5 屡屡受挫，但 GPT-4 首次攻关成功”的经典任务：将 GPT-4 发布的官方博客文章首段总结为一句话，同时要求“每个单词都必须以字母 G 开头”。在 GPT-3.5 上运行时，模型几乎放弃抵抗，直接生成了普通句式。正如 Brockman 评价的那样：“Yeah, it kind of didn't even try. Just gave up on the task. This is pretty typical for 3.5 trying to do this particular kind of task.” 随后，同样的 System 指令与文章输入切换到 GPT-4。GPT-4 迅速给出了全 G 开头的句子，但其中夹带了“AI”缩写。Brockman 随即在聊天中追问纠正：“That's cheating.” GPT-4 欣然接受反馈，重写出了一句完全剔除 AI、纯粹由 G 字母单词组成的流畅概括。为了证明这并非针对特定字母的偶然拟合，Brockman 先是自己测试了字母 A（允许保留 GPT-4 专有名词），模型表现优异；接着，他转向 Discord 观众征集挑战，挑中了难度极高的字母 Q。面对近乎苛刻的词汇限制，GPT-4 依然成功构思出了一句全部以 Q 开头的总结，令现场惊叹。 - 任务要求：输入长文本，仅用一句话总结，且句中每一个单词必须严格以指定字母开头。 - GPT-3.5 表现：在复杂文本上直接放弃约束，输出常规总结；GPT-4 表现：精准捕捉全局拼写与词法约束。 - 字母 Q 挑战：观众在 Discord 频道提出极度罕见的首字母 Q 限制，GPT-4 顺利完成合成。 ### [04:40](https://www.youtube.com/watch?v=outcGtbnMuQ&t=280s) 跨文章主题融合与多模态代码构建准备 证明了单一文章的内容提炼能力后，Brockman 开始展示跨文本的灵活概念综合与风格重塑。他随手复制了一篇当天 Hacker News 上关于 Pinecone（一个 Python Web 应用开发框架）的热门文章，与先前的 GPT-4 博客拼接在一起，要求模型提炼两者之间的共通主题。初次生成强调了技术易用性，Brockman 进一步提示“That was not insightful enough”，GPT-4 随即深化了论点，精准提炼出“弥合强大底层技术与实际落地应用之间的鸿沟”（bridging the gap between powerful technology and practical applications）。随后，他又指令模型将 GPT-4 博客改编成押韵诗歌，诗中模型甚至自发提到了 OpenAI 最新开源的模型评估框架 OpenAI Evals，Brockman 也借此向社区呼吁贡献评测用例。 随后，演示进入重头戏——让 GPT-4 成为全流程结对编程搭档，现场从零开发一个兼具文本和图像识别能力的 Discord 机器人。Brockman 设定系统消息，要求模型在给出具体实现前必须先编写伪代码（pseudo code）。关于这种工程实践，Brockman 强调：“And this approach is very helpful to let the model break down the problem into smaller pieces. And then that way, you're not kind of asking it to just come up with a super hard solution to a problem all in one go.” 伪代码不仅将大任务分解，还大幅提升了逻辑的可解释性，便于人类开发者中途纠偏。 - 跨文本整合：GPT-4 能够跨越两个完全不同领域的长文，准确提炼出抽象层面相通的技术哲学与产业价值。 - 诗歌重构与社区号召：模型能够以押韵形式重述技术发布内容，并提及开源评测工具 OpenAI Evals。 - 思维链与伪代码原则：强制模型“先伪代码后实现”，既降低复杂编码的出错率，又增强了代码审计的可解释性。 ### [07:31](https://www.youtube.com/watch?v=outcGtbnMuQ&t=451s) 引入外部文档突破知识截断与调试实战 构建这个 Discord Bot 伴随着一个棘手的真实挑战：GPT-4 的训练数据截止至 2021 年，它完全不知道 OpenAI 两周前才发布的 Chat Completions API 规范，更不可能了解刚刚发布的多模态图像扩展协议。Brockman 直接将两周前官方博客里的 API 响应格式文档，连同自己手写的极简图片参数扩展描述直接粘贴到 Prompt 中。GPT-4 展现了出色的上下文学习能力（in-context learning），在完全没有预训练记忆的情况下，准确消化了新接口协议并写出了完整的机器人实现。Brockman 提醒观众：绝对不要盲目运行无论来自人类还是 AI 的不受信任代码（“Don't run untrusted code from humans or from AIs.”），必须逐行审查。 在实际运行代码时，演练环境接连遭遇了两个经典故障。首先是 Discord API 在 2020 年后引入的必填关键字参数 `intents` 报错。Brockman 没有给出任何指导性提示，只是原封不动地把控制台报错信息贴回模型，GPT-4 立刻道歉并补全了 `intents` 相关配置。紧接着，由于代码是在 Jupyter Notebook 中运行异步代码（asyncio），遇到了环境默认自带事件循环冲突导致的报错。面对复杂的嵌套事件循环异常，Brockman 仅提示自己是在 Jupyter 环境下运行，GPT-4 便准确定位出核心矛盾所在，并引入 `nest_asyncio` 库进行修复： ```mermaid flowchart LR accTitle: 交互式纠错与环境适配流程 accDescr: 展示代码执行报错后，GPT-4 自主定位原因并修复 Discord API 与 Jupyter 异步冲突的过程 subgraph Issue1["问题一：Discord 权限缺失"] E1["运行时报错：<br/>Missing intents keyword"] --> P1["原样回贴控制台错误日志"] P1 --> F1["补全 Discord intents 实例化代码"] end subgraph Issue2["问题二：Jupyter 异步循环冲突"] E2["运行时报错：<br/>Event loop already running"] --> P2["提示运行环境为 Jupyter"] P2 --> F2["引入 nest_asyncio 库<br/>调用 nest_asyncio.apply()"] end F1 --> E2 F2 --> RUN["Bot 成功启动监听"] ``` 正如 Brockman 所解释的，问题在于 Jupyter 本身已经在运行一个事件循环：“And the specific problem is that there's already an event loop running, so you need to use this nest async io library. You need to call nest async io.apply.” GPT-4 不仅精准插入了 `nest_asyncio.apply()` 调用，还贴心地提示如果环境中未预装该库，可以直接执行 `!pip install nest_asyncio`，完整展现了与 AI 协同排错、抹平环境差异的开发体验。 ### [10:23](https://www.youtube.com/watch?v=outcGtbnMuQ&t=623s) GPT-4 视觉多模态预览与 Discord 截图解析 Greg Brockman 切换到 Discord 界面，向频道中发送了一张 Discord 界面本身的截图，开始展示 GPT-4 的多模态视觉能力。他向观众强调，GPT-4 不仅是一个纯语言模型，也是一个视觉模型，能够灵活接收任意穿插图像与文本的输入，其形式就像处理一份图文混排的复合文档。不过，图像输入功能目前仍处于预览阶段（preview），尚未对公众完全开放；OpenAI 正在与名为 Be My Eyes 的伙伴团队展开独家合作，持续打磨这项技术以备正式上线。 在测试中，Greg 在聊天框输入：“GPT-4，Hello, world. Can you describe this image in painstaking detail?”（你能极其详尽地描述这张图片吗？）。转到后台代码查看，系统确实已成功接收到消息并封装成了标准的 API 请求。在等待返回时，Greg 坦言团队目前的一大核心任务就是提升系统的推理速度，同时呼吁直播间的观众向 Discord 的 `#gpt4-demo-suggestions` 频道提交图片与任务设想，稍后由管理员挑选进行现场演示。很快，模型给出了精准的描述，一眼认出这是一张 Discord 界面截图，甚至识别出了写着“GPT-4”的服务器图标、界面各功能区细节、通知信息，以及频道内正在刷屏提醒 Greg 输入“q”退出的热心观众留言。 - GPT-4 支持文本与图片任意穿插作为输入提示词。 - 视觉功能处于早期技术预览阶段，尚未公开发布，首个落地合作方为 Be My Eyes。 - 模型不仅能精准识别 UI 构图与组件，还能读取图像内的各种文字细节。 ### [12:35](https://www.youtube.com/watch?v=outcGtbnMuQ&t=755s) 超长上下文排障：诊断 Discord Message Content Intent 缺失 随后演示现场出现了一个小插曲：机器人虽然在 Discord 频道收到了新消息事件，但模型拿到的 `content` 却是一个空字符串（empty string）。面对这一异常，Greg 抛出疑问：这究竟是模型本身的缺陷，还是外部包裹系统的逻辑漏洞？他解释道，这其实是他们故意给模型出的一个“下马威”难题：Discord 官方在 2022 年 9 月引入了强制更新，开发者必须在代码中显式声明 `message_content` intent（消息内容权限），否则机器人就无法读取未显式 `@` 提及自己的消息内容。这项改动发生的时间远晚于模型的训练数据切断期，GPT-4 不可能凭先验记忆直接获知。 为了化解困境，Greg 直接全选复制了长达 10,000 到 15,000 词的未经排版的 Discord 原始开发文档，一把粘贴给 GPT-4，并提问自己收到的消息内容为空的原因与修复方案。他借此介绍了 GPT-4 的全新特性——高达 32,000 tokens 的上下文长度窗口（context length），这使得模型能够从汪洋大海般的杂乱文档中准确定位微小变动。GPT-4 毫无悬念地快速命中了核心原因，指出正是未开启 `message_content` intent 所致。重新配置并连线后，即使不直接 `@` 机器人，它也能丝滑抓取并解析频道图片了。针对这种协同流程，Greg Brockman 指出：“I think that this is a augmenting tool, makes you much more productive, but it's still important that you are in the driver seat and are the manager and knows what's what's going on.”（我认为这是一个增强工具，能让你效率倍增，但更重要的是你必须坐在驾驶座上充当管理者，清楚掌握实际发生的一切。） ```mermaid flowchart TD accTitle: Discord 消息排障与修复流程 accDescr: 展示因缺少权限导致空内容，随后借助 GPT-4 检索长文档完成修复的闭环流程 subgraph Problem ["异常出现"] A["Discord 频道收到新消息"] --> B["获取到空字符串 content: ''"] end subgraph Diagnosis ["GPT-4 诊断"] C["复制 10k-15k 词 Discord 官方文档"] --> D["利用 32k 上下文窗口检索"] D --> E["命中规则: 2022年9月新增 Message Content Intent 限制"] end subgraph Fix ["修复与验证"] E --> F["开启对应 Intent 重新连接"] F --> G["无需 @ 即可正常接收并解析消息与图像"] end Problem --> Diagnosis Diagnosis --> Fix ``` - GPT-4 支持高达 32,000 tokens 的上下文长度，能够消化并处理整篇长文档。 - 现场演示的 Discord 开发文档篇幅约为 10,000 至 15,000 词。 - 观众随后提交了一张疑似 DALL-E 生成的“松鼠手持相机拍照”的趣味图片，GPT-4 准确指出笑点在于松鼠通常只会吃坚果，拿相机模仿人类行为违背了常规认知。 ### [16:11](https://www.youtube.com/watch?v=outcGtbnMuQ&t=971s) 从草稿纸涂鸦到可用网页：手绘原型一键转 HTML/JS 紧接着，Greg 展示了视觉多模态能力最为惊艳的实用场景之一：纸上原型直接转代码。他拿出一张亲手绘制在草稿纸上的简陋涂鸦，上面画着一个讲笑话的网页布局，并自嘲这是“值得贴在我家冰箱上的大作”。他掏出手机对着草稿纸拍了一张照片，随后上传到演示的 Discord 频道。现场还发生了把照片误发到其他频道的乌龙，Greg 笑着打趣说，在所有的技术演示中，往往不是 AI 的部分出岔子，那些最平常的非 AI 环节才是最容易手忙脚乱的。 当模型接收到这张照片以及“输出图中对应的 HTML 代码”的指令后，GPT-4 直接生成了一套带有完整 JavaScript 逻辑的功能性网页代码。网页不仅完美还原了手绘图中的按钮和区域布局，甚至还自作主张为页面填充了真正的幽默笑话内容。Greg 将生成的网页与原始草稿纸同屏对比，直观展现了草图瞬间变为可用软件的过程。他感叹整个系统背后的本质其实是一个经过海量内容训练、学会预测下一个词（predict what comes next）的神经网络，而正是这种朴素的机制演化出了极其深厚且灵活的应用技能。 - 演示输入：一张用手机直接拍摄的手绘线框原型草图。 - 演示产出：包含完整布局样式与交互逻辑（JavaScript）的单页应用。 - 模型在缺乏文本说明的情况下，自主理解图中的按键含义并补全了真实的笑话文本。 ### [18:54](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1134s) 挑战复杂报税规则：化身“TaxGPT”推导标准扣除额 在演示的最后一部分，Greg 决定带模型攻克每个人都头疼但又避不开的硬核痛点——报税计算。他打出免责声明，明确指出 GPT-4 和他本人都不是持牌专业税务师，重要税务事项务必咨询专业顾问。随后，他利用 System Message（系统提示词）给模型赋予角色，将其命名为“TaxGPT”。他特别解释称，TaxGPT 并非官方经过针对性微调的特殊模型，而是纯粹依靠 System Message 的引导，让模型快速进入对应的专业思维状态。 Greg 随即将长达 16 页的美国税法复杂法条直接喂给 GPT-4，并输入了关于虚构夫妻“爱丽丝（Alice）和鲍勃（Bob）”的计算场景：两人在特定年份结婚，采用合并申报（married filing jointly），需要计算他们在 2018 纳税年度的标准扣除额（Standard Deduction）。为了突出题目的繁琐，Greg 在模型运行的同时亲自对着法条肉身解题：纳税人需依据基础扣除额计算，合并申报对应 C 分段的 200%；但随后法条出现交叉引用与时间限制，在 2018 至 2025 年度的特别条款中，基础参数由 3,000 美元临时调增为 12,000 美元，因此最终结果是 12,000 美元的 200%，即 24,000 美元。 GPT-4 不仅迅速给出了完全一致的 24,000 美元正确答案，而且分步骤详细列出了引用条款的递进推导过程。Greg 感慨道，自己第一次读这 16 页税法时被各种交叉引用搞得一头雾水，枯坐半小时都没理清脉络；最后反而是靠着让 GPT-4 拆解逻辑链条，自己才彻底弄懂。关于人机协作，Greg 评价道：“It's not perfect, but neither are you. And together, it's this amplifying tool that lets you just reach new heights.”（它并不完美，但人类同样不完美。二者结合在一起，它就成了一个能让你攀登全新高度的放大工具。） - 提示词工程技巧：通过 System Message（如设为 TaxGPT）为大模型设定专业人格与行为准则。 - 测试输入文档：涵盖交叉引用的 16 页美国税法原文。 - 关键税法推导细节：2018 纳税年度合并申报适用特殊修正条款，基准数值从 3,000 美元调整为 12,000 美元，经 200% 倍率折算后得出正确的标准扣除额为 24,000 美元。 ### [21:24](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1284s) GPT-4 的心算与综合能力展示 Greg Brockman 随后向 GPT-4 发出指令：“Okay, now calculate their total liability.”（好，现在计算他们的总纳税义务）。随着模型逐步展开运算，Brockman 看着屏幕上的实时生成由衷感叹道：“Honestly, every time it does it, it's just amazing.” 他直言这个模型非常擅长心算，水平“远比自己的心算能力强得多”。同时他特别强调，此时的模型并没有接入任何外部计算器，尽管连接计算器等工具确实是未来进一步增强此类系统的一种有效方式，但 GPT-4 凭借自身的原始计算能力（raw capabilities）就已经表现得极其灵活。 Brockman 总结指出，GPT-4 最强大的地方在于它的通用与统一性——它并不介意输入的是代码、自然语言还是复杂的税务条款，所有的能力都集成在同一个系统之中。这种跨领域的通用能力，可以直接应用到开发者所关心的具体问题、业务应用或正在构建的任何产品之中。 - **原生心算能力**：模型无需外挂真实计算器或 Python 执行器，仅依靠神经网络的内部表征与生成过程即可推演计算。 - **跨模态与跨领域适应性**：系统能够无缝处理自然语言理解、代码编写、税法逻辑判断和数学计算。 ### [22:19](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1339s) 押韵诗结语与开发者号召 在演示的收尾阶段，Brockman 决定给模型注入一点创意的调味剂，给出了最后一个提示词：“now summarize this problem into a rhyming poem”（现在把这个报税问题总结成一首押韵的诗）。GPT-4 随即迅速输出了一首工整押韵的诗歌，将繁琐枯燥的报税计算变成了一篇生动的文学表达，Brockman 笑着称赞道：“A beautiful, beautiful poem about doing your taxes.”（一首关于报税的极其优美的诗）。 紧接着，Brockman 向所有观看直播的开发者致谢，希望大家通过这场演示了解到了 GPT-4 的能力边界以及如何与模型进行有效交互。他再次呼吁社区积极参与开源评测项目 OpenAI Evals，并真挚地说道：“We think that this model, improving it, bringing it to the next level, is something that everyone can contribute to, and that we think it can really benefit a lot of people, and we want your help to do that.” 他热切期待看到全球开发者在 GPT-4 的基础上构建出令人惊叹的全新应用。 ```mermaid flowchart LR accTitle: GPT-4 演示闭环与社区协同 accDescr: 展示 GPT-4 如何从税务计算流转到创意诗歌，并最终呼吁开发者接入 OpenAI Evals 贡献测试数据。 subgraph Capability["GPT-4 核心能力演示"] A["税法逻辑与心算"] --> B["多领域统一处理"] B --> C["创意总结: 押韵诗"] end subgraph Community["社区协作与生态"] C --> D["开发者生态构建"] D --> E["贡献 OpenAI Evals"] E --> F["推动模型迭代进化"] end ``` ## Notable quotes > [18:54](https://www.youtube.com/watch?v=outcGtbnMuQ&t=1134s) It's not perfect, but neither are you. And together, it's this amplifying tool that lets you just reach new heights. — Greg Brockman > [07:31](https://www.youtube.com/watch?v=outcGtbnMuQ&t=451s) Don't run untrusted code from humans or from AIs. — Greg Brockman ## Mentioned OpenAI, GPT-4, Greg Brockman, Chat Completions API, GPT-3.5, Discord, Hacker News, Pinecone, OpenAI Evals, Jupyter Notebook, nest_asyncio, Be My Eyes, DALL-E, TaxGPT ## Video description Join Greg Brockman, President and Co-Founder of OpenAI, at 1 pm PT for a developer demo showcasing GPT-4 and some of its capabilities/limitations. Join the conversation on Discord here: discord.gg/openai. We'll be taking audience input from #gpt4-demo-suggestions. [Source](https://www.youtube.com/watch?v=outcGtbnMuQ)

Source quotes

[18:54] Greg Brockman: It's not perfect, but neither are you. And together, it's this amplifying tool that lets you just reach new heights. [7:31] Greg Brockman: Don't run untrusted code from humans or from AIs.

Source quotes

[18:54] Greg Brockman: It's not perfect, but neither are you. And together, it's this amplifying tool that lets you just reach new heights. [8:21] Greg Brockman: Don't run untrusted code from humans or from AIs.

Original transcript

[0:08] to the GPT-4 developer demo live stream. [0:12] Honestly, it's kind of hard for me to [0:13] believe that this day is here. OpenAI [0:15] has been building this technology really [0:18] since we started the company, but for [0:19] the past 2 years, we've been really [0:21] focused on delivering GPT-4. [0:24] That started with rebuilding our entire [0:26] training stack, actually training the [0:28] model, [0:29] and then seeing what it it was capable [0:30] of, trying to figure out its [0:32] capabilities, its risks, working with [0:33] partners in order to test it in [0:35] real-world scenarios, really tuning its [0:37] behavior, optimizing the model, getting [0:40] it available [0:41] so that you can use it. And so today, [0:43] our goal is to show you a little bit of [0:45] how to make GPT-4 shine, [0:48] how to really get the most out of it, [0:49] you know, where it's kind of, you know, [0:51] weaknesses are, where it's where still [0:52] working on it, and just how to really [0:54] use it as a good tool, a good partner. [0:56] Um so, if you're interested in [0:58] participating in the stream, uh that if [1:00] you go to our Discord, that's [1:01] discord.gg/openai, [1:03] uh there's comments in there and we'll [1:05] take a couple of audience suggestions. [1:08] So, the first thing I want to show you [1:09] is the first task that GPT-4 could do [1:13] that we never really got 3.5 to do. [1:16] And the way to think about this is all [1:17] throughout training that, you know, [1:18] you're constantly doing all this work. [1:20] It's 2:00 a.m., pager goes off, you fix [1:22] the model, [1:23] and you're always wondering, is it going [1:25] to work? [1:27] Is all of this effort actually going to [1:28] pan out? And so, we all had a pet task [1:31] that we really liked, and that we would [1:32] all individually be trying to see [1:35] is the model capable of it now? [1:37] And I'm going to show you the first one [1:39] that we had a success for four, but [1:41] never really got there for 3.5. [1:43] So, I'm just going to copy the top of [1:44] our blog post from today, going to paste [1:46] it into our playground. Now, this is our [1:50] new chat completions playground that [1:52] came out 2 weeks ago. I'm going to show [1:54] you first with GPT-3.5. Four has the [1:57] same API to it, the same playground. [1:59] The way that it works is you have a [2:00] system message where you explain to the [2:02] model what it's supposed to do, and [2:05] we've made these models very steerable. [2:07] So, you can provide it with really any [2:08] instruction you want, whatever you dream [2:10] up. And the model will adhere to it [2:11] pretty well, and in the future it will [2:13] get increasingly increasingly powerful [2:15] at at steering the model very reliably. [2:20] You can then paste whatever you want as [2:22] a user, the model will return messages [2:24] as an assistant. And the way to think of [2:25] it is that we're moving away from sort [2:28] of just raw text in, raw text out, where [2:29] you can't tell where different parts of [2:31] the conversation come from, but towards [2:32] this much more structured format that [2:34] gives the model the opportunity to know, [2:36] well, this is the user asking me to do [2:38] something that the developer didn't [2:39] intend, I should listen to the developer [2:41] here. [2:42] All right. So, now time to actually show [2:44] you the task that I'm referring to. So, [2:46] everyone's familiar with summarize [2:49] this, let's say, article into a [2:51] sentence, okay? Getting a little more [2:53] specific, uh but where every word begins [2:57] with G. [2:58] So, this is 3.5, let's see what it does. [3:02] Yeah, it kind of didn't even try. [3:04] Just gave up on the task. This is pretty [3:06] typical for 3.5 trying to do this [3:08] particular kind of task. If it's, you [3:11] know, sort of a very kind of stilted [3:13] article or something like that, maybe it [3:15] can succeed, but for the most part [3:17] 3.5 just gives up. [3:19] But, let's try the exact same prompt [3:22] the exact same system message [3:25] in GPT-4. [3:29] So, kind of borderline whether you want [3:31] to count AI or not, uh but so let's say [3:34] AI doesn't count. [3:36] That's cheating. [3:41] So, fair enough, the model happily [3:43] accepts my feedback. [3:44] So, now to make sure it's not just good [3:46] for G's, I I'd like to turn this over to [3:48] the audience. I'll take a suggestion on [3:49] what letter to try next. [3:51] In the meanwhile, while I'm waiting for [3:52] our moderators to pick the the lucky [3:54] lucky letter, [3:55] I will give a try with A. [4:02] Um but in this case, I'll say that GPT-4 [4:03] is fine. [4:04] Why not? [4:07] Also, pretty good summary. [4:09] So, I'll hop over to our Discord. [4:12] All right. [4:13] Wow. Uh people are are being a a little [4:15] ambitious here. I'm really trying to put [4:17] the model through the paces. We're going [4:18] to try Q. Uh which If you think about [4:20] this for a moment, I want the audience [4:22] to really think about how would you do a [4:24] summary of this article that all starts [4:26] with Q. [4:27] It's not easy. [4:37] It's pretty good. [4:38] That's pretty good. [4:40] All right. So, [4:41] I've shown you summarizing an existing [4:44] article. I want to show you how you can [4:46] flexibly combine ideas between different [4:48] articles. So, I'm going to take this [4:50] article that was on Hacker News [4:52] yesterday, [4:54] copy paste it [4:56] into the same conversation so it has all [4:58] the context of what we were just doing. [5:00] I'm going to say uh find one common [5:03] theme between this article and the GPT-4 [5:06] blog. [5:10] So, this is an article about Pinecone, [5:13] which is a Python web app development [5:14] framework, and it's making the [5:16] technology more accessible, [5:17] user-friendly. If you don't think that [5:18] was insightful enough, you can always [5:20] give some feedback and say, "That was [5:22] not insightful [5:25] enough. [5:26] Please No, I'll just even just leave it [5:28] there. Leave it up to the model to [5:29] decide. [5:30] So, bridging the gap between powerful [5:31] technology and practical applications. [5:34] Seems not bad. Um and of course, you can [5:36] ask for any other kind of task you want [5:38] using its flexible language [5:39] understanding um and synthesis. You can [5:42] ask for something like [5:43] "Now, turn the GPT-4 blog post into a [5:48] rhyming [5:49] poem. [5:57] Big up on [5:58] Big up on opening IE valves, open source [6:00] for all, helping to guide, answering the [6:02] call. Which by the way, if you'd like to [6:03] contribute to this model, please give us [6:06] IE valves. We have an open source [6:07] evaluation framework that will help us [6:08] guide and all of our users understand [6:11] what the model's capable of and to take [6:13] it to the next level. [6:15] So, there we go. This is consuming [6:17] existing content using GPT-4 with a [6:19] little bit of creativity on top. [6:22] But next, I want to show you how to [6:25] build with GPT-4. What it's like to [6:27] create with it as a partner. [6:30] And so, the thing we're going to do [6:32] is we're going to actually build a [6:34] Discord bot. [6:36] I'll build it live and show you the [6:38] process, show you debugging, show you [6:40] what the model can do, where its [6:41] limitations are, and how to work with [6:43] with them in order to sort of achieve [6:44] new heights. So, the first thing I'll do [6:46] is tell the model that this time it's [6:48] supposed to be an AI programming [6:50] assistant. Its job is to write things [6:52] out in pseudo code first and then [6:54] actually write the code. And this [6:56] approach is very helpful to let the [6:58] model break down the problem into [7:00] smaller pieces. And then that way, [7:02] you're not kind of asking it to just [7:04] come up with a super hard solution to a [7:06] problem all in one go. [7:08] It also makes it very interpretable [7:10] because you can see exactly what the [7:11] model was thinking and you can even [7:13] provide corrections if you'd like. [7:15] So, uh here is the prompt that we're [7:18] going to ask it. Uh this is the kind of [7:20] thing that 3.5 would totally choke on if [7:22] you if you tried anything like it. Um [7:23] but so, we're going to ask for a Discord [7:25] bot that uses the GPT-4 API to uh read [7:29] images and text. [7:31] Now, there's one problem here, which is [7:33] this model's training cutoff is in 2021, [7:37] which means it has not seen our new chat [7:39] completions format. So, I literally just [7:41] went to the blog post from 2 weeks ago, [7:43] copy pasted from the blog post including [7:45] the response format. It has not seen the [7:47] new image extension to that, and so I [7:49] just kind of wrote that up in, you know, [7:51] just [7:52] very minimal detail about how to include [7:54] images. [7:55] And now, the model can actually leverage [7:57] that doc- that documentation that it did [8:00] not have memorized that it does not [8:01] know. [8:09] And in general, these models are very [8:11] good at using information that it's been [8:13] trained on in new ways and synthesizing [8:15] new content. And you can see that right [8:17] here that it actually wrote an entirely [8:19] new bot. [8:21] Now, let's [8:23] actually see if this bot is going to [8:24] work in practice. So, you should always [8:26] look through the code to get a sense of [8:28] what it does. Don't run untrusted code [8:30] from humans or from AIs. Um [8:33] and one thing to note is that the [8:35] Discord API has changed a lot over time. [8:38] And particularly, that there's one [8:40] feature that has changed a lot since [8:42] this model was trained. [8:44] Give it a try. [8:46] In fact, yes, we are missing the intents [8:48] keyword. This is something that came out [8:50] in 2020. [8:52] So, the model does know it exists, but [8:53] it doesn't know which version of the [8:55] Discord API we're using. [8:57] So, are we out of luck? Well, not quite. [9:00] We can just simply paste to the model [9:02] exactly the error message. Not even [9:03] going to say, "Hey, this is from running [9:05] your code. Could you please fix it?" [9:08] We'll just let it run. [9:10] And the model says, "Oh, yeah. Whoops, [9:11] the intents argument. Here's the correct [9:13] Here's the correct code." [9:17] Now, let's give this a try. Once again, [9:19] kind of making sure that we understand [9:21] what the code is doing. [9:27] Now, a second issue that can come up is [9:29] it doesn't know what environment I'm [9:30] running in. And if you notice, it says, [9:32] "Hey, [9:33] here's this inscrutable error message." [9:36] Which, if you've not used Jupyter [9:37] Notebook a lot with async io before, you [9:39] probably have no idea what this means. [9:42] But, fortunately, [9:45] once again, you can just sort of say to [9:47] the model, "Hey, [9:48] I I'm using Jupiter [9:52] and would like to make this work. [9:56] Can you fix it?" [9:58] And the specific problem is that there's [10:00] already an event loop running, so you [10:01] need to use this nest async io library. [10:05] You need to call nest async io.apply. [10:07] The model knows all of this, [10:09] correctly instantiates all of these [10:11] these pieces into the bot. It even helps [10:14] you helpfully tells you, "Oh, you're [10:15] running in Jupiter. Well, you can do [10:16] this bang pip install in order to [10:18] install the package if you don't already [10:19] have it." That was very helpful. [10:23] So, now we'll run [10:24] and it looks like something happened. [10:26] So, the first thing I'll do [10:28] is [10:30] go over to our Discord [10:32] and I will paste in [10:35] a screenshot [10:36] of our Discord itself. So, remember, [10:38] GPT-4 is not just a language model. It's [10:42] also a vision model. In fact, it can [10:45] flexibly accept inputs that intersperse [10:48] images and text arbitrarily, kind of [10:50] like a document. Now, [10:52] the image feature is in preview. So, [10:54] this is going to be a little sneak peek. [10:56] It's not yet publicly available. It's [10:58] something we're working with one partner [10:59] called Be My Eyes in order to really [11:01] start to develop it and get it ready for [11:02] primetime. [11:04] But, you can ask anything you like. For [11:05] example, I can't you know, I'll say [11:08] GPT-4, [11:11] "Hello, world. [11:13] Can you describe this image [11:17] in painstaking detail?" [11:21] All right. Which, first of all, think of [11:22] how you would do this yourself. Uh [11:24] there's a lot of different things you [11:25] could latch onto, a lot of different [11:26] pieces of the system you could describe. [11:28] And we can go over to the actual code [11:30] and we can see that, yep, we in fact [11:31] received the message, have formatted an [11:34] appropriate request for our API. [11:37] And now we wait um because, you know, [11:39] one of the things we have to do is we [11:40] have to make the system faster. That's [11:42] one of the things that we're working on [11:43] optimizing. Um in the meanwhile, I just [11:45] want to say to the audience that's [11:46] watching, we'll take an audience request [11:48] next. So, if you have an image, and a [11:50] task you'd like to accomplish, please [11:53] submit that to the Discord. Our [11:54] moderators will pick one that we'll that [11:56] we'll run. [12:01] So, we can see that the Discord Oh, it [12:03] looks like we have a response. [12:05] Perfect. [12:06] So, it's a screenshot of a Discord [12:07] application interface. Pretty good. Did [12:09] not even describe it. It knows that that [12:11] it's Discord. It's probably Discord [12:12] written there somewhere where it just [12:13] kind of knows this from from prior [12:15] experience. Um [12:17] server icon labeled GPT-4, describes the [12:19] interface in great detail, talks about [12:22] all the people telling me that I'm [12:23] supposed to do Q. Uh [12:25] Very very kind audience. Uh and [12:28] describes a bunch of the the [12:29] notification messages and the users [12:31] there in the channel. And so, there you [12:32] go. That's some that's some pretty good [12:34] understanding. Now, this next one, if [12:37] you notice, first of all, we got a post, [12:39] but [12:41] the model did not actually see the [12:43] message. So, is this a failure of the [12:44] model or of the system around the model? [12:47] Well, we can take a look. [12:49] And if you notice, here content is an [12:51] empty string. We received a blank [12:53] message contents. [12:55] The reason for this is a dirty trick [12:57] that we played on the AI. [12:59] So, if you go to the Discord [13:01] documentation, [13:04] and you scroll through it, [13:06] all the way down to [13:08] I It's It's hard for me to even find, [13:10] honestly, to the message content [13:13] intent. You'll see this was added [13:15] as of September 2022 as a required [13:17] field. So, in order to receive a message [13:19] that does not explicitly tag you, you [13:22] now have to include this new intent in [13:24] your code. Remember I said, intents have [13:26] changed a lot over time. This is much [13:29] newer than the model is possible is [13:30] possibly able to know. So, maybe we're [13:33] out of luck. We have to debug this by [13:34] hand. But once again, we can try to use [13:37] GPT-4's language understanding [13:38] capabilities [13:41] to solve this. Now, keep in mind this is [13:42] a document of like I think this is like [13:45] 10,000, 15,000 words, something like [13:47] that. It's not formatted very well. This [13:49] is literally a command A copy paste. [13:51] Like this is what it's supposed to parse [13:52] through to find in the middle of that [13:54] document that Oh, yeah, a message [13:56] content. That's required now. But let's [13:58] see if it can do it. [13:59] So, we will ask for [14:02] I am receiving blank message contents. [14:06] Can you [14:08] Why could this be happening? [14:12] How do I fix it? [14:16] So, one thing that's new about GPT-4 is [14:18] context length. [14:20] 32,000 tokens is kind of the upper limit [14:22] that we support right now. Uh and the [14:24] model is able to flexibly use long [14:27] documents. Uh it's something we're still [14:30] optimizing. So, [14:31] we recommend trying it out um but not [14:34] necessarily sort of really really [14:35] scaling it up just yet um unless you [14:37] have an application that really benefits [14:39] from it. So, if you're really interested [14:41] in long context, please let us know. We [14:42] want to see what kinds of applications [14:44] it unlocks. [14:46] But if you see, [14:47] it says, "Oh, yeah, mess message content [14:48] intent was not enabled." And so, you can [14:50] either ask the model to write some code [14:51] for you or you could [14:54] I actually just, you know, [14:57] do it the old-fashioned way. [14:59] Either way is fine. [15:01] I think that this is a augmenting tool, [15:03] makes you much more productive, um but [15:04] it's still important that you are in the [15:06] driver seat and are the manager and [15:08] knows what's what's going on. So, now [15:10] we're connected once again. [15:12] And uh Boris, would you like to rerun [15:14] the message? [15:26] Once again, we can see that we have [15:28] received it, even though the bot was not [15:30] explicitly tagged. [15:32] Seems like a pretty good [15:36] pretty good description. Interesting. [15:38] This is an interesting image actually. [15:39] Looks like it's a DALL-E generated one. [15:41] Um and let's actually try this one as [15:43] well. [15:50] What's funny about this image? Oh, it's [15:52] already been submitted. [15:56] So, once again, we can verify this is [15:58] making the right API calls. [16:02] Squirrels do typically eat nuts. We [16:04] don't expect them to use a camera or act [16:06] like a human. So, I think that's that's [16:08] a pretty good explanation of why that [16:09] image is funny. [16:11] So, I'm going to show you one more [16:13] example of what you can do with this [16:16] model. [16:17] So, I have here [16:19] a nice hand-drawn mock-up of a joke [16:21] website. Uh definitely worthy of being [16:23] put up on my refrigerator. [16:26] So, I'm just going to take out my phone, [16:29] literally take a photo [16:33] of this mock-up. [16:35] And then, I'm going to send it [16:38] to our Discord. [16:49] All right. Going to send it to our [16:50] Discord. [17:01] This is of course the rockiest part, [17:02] making sure that we actually send it to [17:04] the right channel. [17:08] Which in fact, I think maybe I did not. [17:14] Sent it to the wrong channel. [17:16] It's funny. It's always the the sort of [17:19] non-AI parts of these demos that are the [17:21] hardest part to do. [17:29] And here we go. [17:31] Technology is now solved. [17:35] And now we wait. [17:39] So, the thing that's amazing in my mind [17:41] is that [17:43] what's going on here is we're talking to [17:44] a neural network. [17:46] And this neural network was trained to [17:48] predict what comes next. Right? It [17:50] played this this game of sort of being [17:52] shown a partial document and then [17:53] predicted what comes next across an [17:55] unimaginably large amount of content. [17:57] And from there, it learns all of these [17:59] skills that you can apply in all of [18:01] these very flexible ways. [18:03] And so, we can actually take now this [18:05] output, so literally we just said to [18:09] output the HTML from that picture. [18:13] And here we go. [18:15] Actual working JavaScript. [18:18] Filled in the jokes. [18:19] For comparison, [18:22] this was the original [18:27] of our mock-up. [18:29] And so, there you go. Going from [18:31] hand-drawn [18:32] beautiful art, [18:33] if I do say so myself, to working [18:36] website. [18:38] And this is all just potential, right? [18:39] We you can see lots of different [18:41] applications. We ourselves are still [18:43] figuring out new ways to use this. Um [18:45] so, we're going to work with our [18:46] partner. We're going to scale up from [18:47] there, but please be patient because [18:49] it's going to take us some time to [18:50] really make this available for everyone. [18:54] So, I have one last thing to show you. [18:56] I've shown you reading existing content. [18:59] I've shown you how to [19:01] build with the system as a partner. [19:04] The last thing I'm going to show [19:06] is how to work with the system to [19:08] accomplish a task that none of us like [19:10] to do, but we all have to. [19:12] So, you may have guessed, the thing [19:13] we're going to do is taxes. [19:17] Now, note that [19:18] GPT is not a certified tax professional, [19:20] nor am I, so you should always check [19:22] with your your tax advisor. Um but, it [19:24] can be helpful to understand some dense [19:26] content to just be able to empower [19:28] yourself to to be able to sort of solve [19:30] problems and get a get a handle on [19:31] what's happening uh when you could not [19:33] otherwise. So, once again, I'll do a [19:35] system message. In this case, I'm going [19:37] to tell it that it's tax GPT, uh which [19:39] is not a specific thing that we've [19:41] trained into this model. You can be very [19:42] creative if you want with the system [19:44] message to really get the model in the [19:45] mood of what is your job? What are you [19:47] supposed to do? [19:49] So, I paste it in [19:51] the tax code. This is about 16 pages [19:53] worth of of tax code. Um and there's [19:55] this question about Alice and Bob. They [19:57] got married at one point, uh and that [19:59] here are their their incomes, and they [20:00] take a standard deduction. They're [20:01] filing jointly. So, first question, what [20:03] is their standard deduction for 2018? [20:07] So, while the model is chugging, I'm [20:09] going to solve this problem by hand to [20:11] show you what's involved. So, the [20:12] standard deduction uh is the basic [20:15] standard deduction plus the additional. [20:16] The basic one is 200% uh for a joint [20:19] return uh of subparagraph C, which is [20:22] here. Okay, so additional doesn't apply. [20:24] The limitation doesn't apply. Um okay. [20:27] None of these apply. Oh, wait. Special [20:29] rules for taxable year 2018, which is [20:31] the one we care about, through 2025, you [20:34] have to substitute 12,000 for 3,000. So, [20:36] 200% of 12,000, 24,000 is the final [20:39] answer. [20:41] If you notice, the model got to the same [20:44] conclusion. [20:45] And you can actually read through its [20:49] explanation. [20:50] And to tell you the truth, the first [20:52] time I tried to approach this problem [20:54] myself, I could not figure it out. I [20:56] spent half an hour reading through the [20:58] tax code trying to figure out this like [21:00] back reference and why there's [21:01] subparagraph like just what's even going [21:02] on. [21:03] It was only by asking the model to spell [21:05] out its reasoning, and then I followed [21:07] along, but I was like, "Oh, I get it [21:10] now. I understand how this works." And [21:12] so, that, I think, is where the power of [21:14] the system lies. It's not perfect, but [21:16] neither are you. And together, it's this [21:18] amplifying tool that lets you just reach [21:20] new heights. [21:22] And you can go further. You can say, [21:24] "Okay, now calculate their total [21:27] liability." [21:33] And here we go. It's doing the [21:35] calculation. [21:51] Honestly, I [21:52] every time it does it, it's just it's [21:54] amazing. Uh this model's so good at [21:56] mental math. Uh it's way, way better [21:58] than I am at mental math. It's not [21:59] hooked up to a calculator. Like, that's [22:01] another way that you could really try to [22:02] enhance these systems. But, it has these [22:05] raw capabilities that are so flexible. [22:07] It doesn't care if it's code. It doesn't [22:08] care if it's language. It doesn't care [22:10] if it's tax. All of these capabilities [22:12] in one system that can be applied [22:14] towards the problem that you care about, [22:16] towards your application, towards [22:17] whatever you build. [22:19] And so, to end it, the final thing that [22:21] I will show is I a little other dose of [22:24] creativity, which is now summarize this [22:27] problem into a rhyming [22:29] poem. [22:36] And there we go. [22:38] A beautiful, beautiful poem about doing [22:40] your taxes. [22:41] So, thank you, everyone, for tuning in. [22:43] I hope you learn something about what [22:45] the model can do, how to work with it. [22:48] And honestly, we're just really excited [22:50] to see what you're going to build. I [22:52] I've talked about OpenAI Evals. Please [22:53] contribute. We think that this model, [22:55] improving it, bringing it to the next [22:57] level, is something that everyone can [22:58] contribute to, and that we think it can [23:01] really benefit a lot of people, and we [23:02] want your help to do that. So, thank you [23:04] very much. We're so excited to see what [23:05] you're going to build.