Anthony Scopatz Unveils Xonsh to Replace Bash With Python ASTs

PyCon 2016

Anthony Scopatz argues Python's dynamic AST manipulation and signal handling make it uniquely qualified to replace fragile Unix shells without sacrificing existing POSIX workflows.

AI-assisted summary of Anthony Scopatz - xonsh - PyCon 2016 — original source. Check the original for full context.

Source published: 2016-05-31

Key ideas

Anthony Scopatz introduces xonsh at PyCon 2016, a cross-platform command-line shell combining Python with features from Bash, Zsh, and Fish that relies solely on Python's standard library and PLY. (0:01)
The xonsh language operates as a strict superset of Python that compiles into standard Python AST nodes, allowing seamless intermixing of shell commands and pure Python code. (17:01)
A foreign shell execution bridge enables sourcing existing Bash, Zsh, and cmd.exe scripts to load environment variables, aliases, and shell functions directly into the xonsh runtime. (13:14)
The shell features native Jupyter notebook hooks, importable .xsh script files, in-memory tab completions, and terminal-rendered Matplotlib graphics. (15:33)

Detailed notes

## Background Published on YouTube by the PyCon 2016 channel on May 31, 2016, this talk features speaker Anthony Scopatz presenting xonsh. The page description details xonsh as a general-purpose shell combining Python with the best features of Bash, Zsh, and Fish, compiling directly to a Python AST with zero external dependencies beyond Python and PLY.

## TL;DR Scopatz introduces xonsh, an interactive command-line shell and general-purpose scripting language that functions as a strict superset of Python while integrating native POSIX shell primitives. He demonstrates how xonsh seamlessly bridges Python objects and subprocess execution using syntax like `@()` argument expansion, `!()` command capture, callable Python aliases, and regex-based file globbing. The architecture relies on PLY and a two-phase syntax tree transformation that compiles ambiguous commands into standard Python AST nodes, allowing direct execution via Python's native runtime. Addressing practical integrations, Scopatz outlines foreign shell bridging for Bash and Zsh environments, script execution via `.xsh` files, and dynamic integrations with packages like Matplotlib and NLTK before answering audience questions about prompt customization, startup performance, and line-editing backends.

## Atmosphere & style The presentation is energetic, lighthearted, and deeply pragmatic, driven by Scopatz's self-deprecating humor and candid technical admissions. Despite an early on-stage display hiccup, he maintains effortless rapport with the audience, drawing frequent laughs when poking fun at Bash's clunky conventions and confessing that his slick terminal demonstrations were actually simulated keystroke playbacks. The session feels like an enthusiastic open-source workshop where complex language parsing and compiler concepts are demystified with clarity, wit, and contagious enthusiasm.

## Bulletin - [00:01](https://www.youtube.com/watch?v=uaje5I22kgE&t=1s) Anthony Scopatz introduces xonsh at PyCon 2016, a cross-platform command-line shell combining Python with features from Bash, Zsh, and Fish that relies solely on Python's standard library and PLY. - [17:01](https://www.youtube.com/watch?v=uaje5I22kgE&t=1021s) The xonsh language operates as a strict superset of Python that compiles into standard Python AST nodes, allowing seamless intermixing of shell commands and pure Python code. - [13:14](https://www.youtube.com/watch?v=uaje5I22kgE&t=794s) A foreign shell execution bridge enables sourcing existing Bash, Zsh, and cmd.exe scripts to load environment variables, aliases, and shell functions directly into the xonsh runtime. - [15:33](https://www.youtube.com/watch?v=uaje5I22kgE&t=933s) The shell features native Jupyter notebook hooks, importable .xsh script files, in-memory tab completions, and terminal-rendered Matplotlib graphics.

## Key takeaways - [00:01](https://www.youtube.com/watch?v=uaje5I22kgE&t=1s) Xonsh requires no dependencies beyond Python and PLY, enabling seamless cross-platform execution on Linux, macOS, and Windows. - [01:11](https://www.youtube.com/watch?v=uaje5I22kgE&t=71s) A primary architectural motivation was replacing fragile Bash scripting patterns—like mutating $IFS—with clean, memorable Python APIs. - [03:49](https://www.youtube.com/watch?v=uaje5I22kgE&t=229s) The shell evaluates pure Python expressions natively while integrating shell primitives like $NAME and ${...} for environment variable manipulation. - [07:04](https://www.youtube.com/watch?v=uaje5I22kgE&t=424s) Subprocess outputs can be captured as Python strings via $(cmd) or as metadata-rich command objects via !(cmd) and streamed ![cmd]. - [09:23](https://www.youtube.com/watch?v=uaje5I22kgE&t=563s) The @() operator evaluates arbitrary Python expressions and flattens iterable structures into individual command-line arguments for subprocesses. - [11:41](https://www.youtube.com/watch?v=uaje5I22kgE&t=701s) File globbing using regular expressions enclosed in backticks returns lists of strings in Python mode and passes expanded arguments in subprocess mode. - [13:14](https://www.youtube.com/watch?v=uaje5I22kgE&t=794s) Aliases can be defined as callable Python functions and lambdas accepting command-line arguments and standard input. - [13:14](https://www.youtube.com/watch?v=uaje5I22kgE&t=794s) Foreign shell sourcing parses Bash, Zsh, and cmd.exe scripts in background subshells to import environment variables and functions without rewriting code. - [15:33](https://www.youtube.com/watch?v=uaje5I22kgE&t=933s) Custom import hooks allow .xsh script files to be imported and executed like standard Python modules across projects. - [17:01](https://www.youtube.com/watch?v=uaje5I22kgE&t=1021s) Ambiguous statements are resolved by checking if the leftmost token exists in the Python scope; if absent, the line executes as a subprocess command. - [20:15](https://www.youtube.com/watch?v=uaje5I22kgE&t=1215s) Command output from utilities like curl can be directly piped into standard library modules like json.loads() or third-party libraries like nltk. - [23:46](https://www.youtube.com/watch?v=uaje5I22kgE&t=1426s) Interactive runtime execution and tab completion are highly responsive due to in-memory caching, despite admitted startup latency bottlenecks in early releases. - [25:31](https://www.youtube.com/watch?v=uaje5I22kgE&t=1531s) Xonsh includes built-in integration with Jupyter notebooks and provides dedicated commands to manage virtual environments. - [28:38](https://www.youtube.com/watch?v=uaje5I22kgE&t=1718s) Python was chosen over alternatives like JavaScript and Ruby because of its robust OS-level signal handling and dynamic syntax tree manipulation capabilities. - [28:38](https://www.youtube.com/watch?v=uaje5I22kgE&t=1718s) Line editing supports both GNU readline and prompt_toolkit backends, including native support for Vi keybindings.

## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=uaje5I22kgE&t=1s) Introducing xonsh: A Python Superset Shell

Anthony Scopatz introduces xonsh, a general-purpose command-line shell designed to replace traditional shells like Bash, Fish, or Zsh while simultaneously operating as a strict superset of Python. The project requires no external dependencies outside of Python itself and the standard library; consequently, any system capable of running Python—including Linux, macOS, and Windows—can run xonsh out of the box.

Beyond general scripting convenience, Scopatz explains that his underlying goal was scientific tooling: “And sort of the purpose of xonsh, at least in my mind, is to trick scientists and engineers into being more reproducible, but I don’t have time to go into that into this talk, so you'll just have to talk to me about that later.” He notes that users can unlock additional fancy features if they are willing to install companion packages, which he previews throughout the presentation.

[01:11](https://www.youtube.com/watch?v=uaje5I22kgE&t=71s) The Motivation: Bash Frustrations vs. Python Cleanliness

Scopatz motivates the creation of xonsh by contrasting everyday shell operations in sh-based languages with their Python counterparts. While a basic `for` loop in sh requires extra boilerplate, standard Python syntax is universally familiar and clean. During this explanation, Scopatz pauses to resolve a display resolution issue on stage before laughing off the hiccup with the audience: “The best-laid technical plans. Woo! All right. Live debugging. This is what we're all about.”

Returning to comparisons, he points out how cumbersome checking if a file exists is in shell languages: they rely on strange, overlapping flags and commands that users almost invariably have to look up in documentation, whereas Python provides a clear, memorable API. Even worse is splitting a string by a non-whitespace delimiter. In Bash, the user must mutate global shell state by modifying the internal field separator (`$IFS`) and then remember to reset it afterward, which Scopatz labels “an absolute disaster.”

He humorously charts the emotional stages of building xonsh: - **Conceptualization**: Realizing that the current shell situation cannot stand, prompting the thought of merging the best parts of Python with the best parts of Bash. - **Research**: Investigating parsers like PLY and realizing that implementing it might not be so difficult after all. - **Compulsion**: Seeing the path forward clearly and feeling an irresistible drive to build the library. - **The Future**: Scopatz quips, “And the following stages of anger, regret, and acceptance are sure to follow any day now.”

[03:49](https://www.youtube.com/watch?v=uaje5I22kgE&t=229s) Python Mode and Shell Primitives

Opening a live shell demonstration, Scopatz illustrates how xonsh functions as an everyday interactive shell. The prompt presents standard terminal contextual information, displaying the current user and path in the file system. However, the environment evaluates pure Python directly: users can add numbers, import standard or third-party modules, access attributes, instantiate and query dictionary literals, write multi-line `if`, `for`, or `while` blocks, and define or call functions inside the current execution context:

```mermaid flowchart TD accTitle: Dual-Mode Architecture in xonsh accDescr: Diagram showing how xonsh routes commands between Python mode and Subprocess mode.

Input["xonsh Command Input"] --> ModeCheck{"Evaluation Mode?"}

subgraph PyMode["Python Mode"] ModeCheck -->|"Python Syntax"| PyExec["Python Statements / AST"] PyExec --> PyObj["Direct Object Manipulation / Types"] end

subgraph SubMode["Subprocess Mode"] ModeCheck -->|"Shell Command"| SubExec["Subprocess Execution"] SubExec --> Streams["Pipes, Redirection, Metadata"] end

PyMode <-->|"@() / $() /!() Bridges"| SubMode ```

Scopatz stresses that xonsh is more than vanilla Python because it introduces native shell primitives to the language. An environment lookup operator `$NAME` retrieves variables just like in Bash, but variables can be set using standard Python assignments and immediately consumed by subprocesses. Furthermore, environment variables like `$PATH` are exposed not as rigid, colon-delimited strings, but as native Python lists of strings. Arbitrary custom objects and class instances can be injected straight into the environment.

Key environment operators include: - `$NAME`: Accesses an individual environment variable by identifier. - `${<expr>}`: Dynamically evaluates the inner Python expression to determine which environment variable name to fetch. - `${...}`: References the entire environment object itself, allowing dictionary-like operations such as testing containment with `in` or invoking environment object methods directly.

[07:04](https://www.youtube.com/watch?v=uaje5I22kgE&t=424s) Subprocess Capture and Git Integrations

Navigating the filesystem demonstrates developer-centric features. Changing directories into the xonsh source repository instantly renders the active branch name in color within the prompt, an integration that works seamlessly for both Git and Mercurial. Tab completion brings up a navigable drop-down menu for commands such as `git status`. Rather than reinventing completions for every binary, xonsh parses and reuses existing Bash completion scripts, functioning even on Windows.

To capture command output back into Python, xonsh provides specialized syntax: - `$(cmd)`: Behaves identically to standard POSIX shells, capturing a subprocess's standard output as a Python string. This can be assigned to a variable and manipulated using standard Python string methods in a single expression. - `!(cmd)`: Returns a completed command object containing not only standard output, but complete metadata such as process start and end timestamps, any alias used, and boolean evaluation mapping a exit status of `0` to `True`. - `![cmd]`: Functions identically to `!(cmd)`, but actively streams command output to the console during execution instead of buffering it silently.

[09:23](https://www.youtube.com/watch?v=uaje5I22kgE&t=563s) Injecting Python into Subprocesses

Scopatz details the reverse pathway: feeding Python expressions directly into subprocess execution using the `@()` operator. Placing `@(<expr>)` within a shell command evaluates the Python expression inside the parentheses, converts the outcome to a string, and supplies it as a single argument to the target command (e.g., `echo @(2+2)` outputs `4`).

```mermaid flowchart LR accTitle: Python Expression Expansion with @() accDescr: Flow demonstrating how Python data structures are expanded into command-line arguments.

PData["Python Expression: @(list_or_val)"] --> Eval["Evaluate Expression"] Eval --> TypeCheck{"Type?"} TypeCheck -->|"Single Value"| SingleArg["Pass as single string argument"] TypeCheck -->|"Iterable / List"| MultiArg["Unpack into separate CLI arguments"] SingleArg --> Exec["Subprocess Call"] MultiArg --> Exec ```

If the expression inside `@()` evaluates to a list or iterable, xonsh automatically flattens each item into its own distinct argument in the subprocess argument array. The syntax is fully nestable: subprocess captures can encompass `@()` expressions, which can themselves invoke subprocesses. This operator can even be used in the command position to dynamically resolve which executable to invoke. As Scopatz highlights: “But the place where this @() stuff really shines for me is the fact that you can now use normal Python looping semantics to loop over a set of data, and then run over subprocess commands very, very easily, like we see in this example here.”

Rounding out classic shell requirements, xonsh supports: - **Pipes and Redirections**: Standard syntax for piping between processes (`|`) and redirecting file streams (including standard error to standard out) works identically to traditional shells. - **Chaining Subprocesses**: By popular demand, commands can be chained using Python's `and` and `or` keywords, which automatically wraps the underlying subprocesses in `![]` checks to evaluate process exit codes. - **POSIX Muscle Memory**: For users accustomed to Bash conventions, standard `&&` and `||` operators are fully supported. Scopatz reassures the audience: “And don't worry if you have a lot of muscle memory and you want to use && ||. You're welcome to do so. I won't hold it against you.”

[11:41](https://www.youtube.com/watch?v=uaje5I22kgE&t=701s) Regex Globbing and IPython-Style Object Inspection

Building on shell logic, xonsh allows standard Python boolean operations for file system checks. Testing whether a file exists, or chaining conditions with `and`, `or`, parentheses groupings, and negation semantics, simply evaluates to a native Python boolean value. In addition to standard wildcard expansion, xonsh introduces regular expression file globbing enclosed in backticks. Any pattern inside backticks is compiled as a regular expression; in subprocess mode, the matched paths are forwarded directly to the underlying subprocess command, while in Python mode, the expression returns a standard list of strings. This enables complex path matching, such as selecting all Python files in a directory that do not contain an underscore, or running Python functions like `len()` directly on the matched glob list.

To assist with interactive exploration, xonsh adopts inspection operators popularized by IPython. Appending a single question mark (`?`) to any object opens contextual documentation. Appending a double question mark (`??`) triggers "super help," which inspects the object, locates its source definition, applies syntax highlighting to the code, and prints it directly in the terminal.

- Regular expression globbing using backticks (`` `regex` ``) yields lists of strings in Python context and expanded arguments in subprocess context. - Single `?` displays docstrings and object metadata. - Double `??` fetches and syntax-highlights original source definitions before rendering.

---

[13:14](https://www.youtube.com/watch?v=uaje5I22kgE&t=794s) Callable Aliases and Foreign Shell Execution

Command aliasing in xonsh extends far beyond standard POSIX string replacements. While traditional list-of-string aliases familiar from Bash are supported, xonsh allows aliases to be defined as native Python functions and lambdas. A callable alias accepts command-line arguments and standard input, and can return a string, or optionally a tuple containing standard error and an exit return code. Anthony Scopatz demonstrates this by defining a lambda that takes standard input, applies standard Python string formatting, and outputs an ominous greeting after piping text into it from `echo`.

```mermaid flowchart LR accTitle: Foreign Shell Execution Pipeline accDescr: Diagram showing how foreign shell scripts are sourced and imported into xonsh environment

subgraph Foreign ["Foreign Shell Script (owl.sh)"] ENV["Environment Variables"] AL["Shell Aliases"] FN["Shell Functions"] end

subgraph Parser ["xonsh Foreign Bridge"] SRC["source-bash / foreign parser"] end

subgraph XonshEnv ["xonsh Runtime Context"] XENV["os.environ / ${...}"] XAL["aliases['...']"] XCALL["Callable Function Bridge"] end

ENV --> SRC AL --> SRC FN --> SRC SRC --> XENV SRC --> XAL SRC --> XCALL ```

Xonsh also introduces a formal mechanism for interacting with foreign shells. Sourcing shell setups—such as a Bash script named `owl.sh` that exports environment variables, creates aliases, and defines shell functions—traditionally forces users to rewrite configurations. In xonsh, running the built-in foreign sourcing command parses the Bash file within a background Bash subshell, extracts the resulting modifications, and loads the environment variables, aliases, and functions directly into the live xonsh environment. Scopatz notes that this bridge is implemented for Bash, Zsh, and `cmd.exe`, while quipping about future support: “And clearly there's still a big fish in the room here.”

- Callable aliases accept `(args, stdin=None)` and can return stdout, or `(stdout, stderr, exit_code)`. - Foreign shell integration natively captures environment variables, command aliases, and functions from Bash, Zsh, and Windows `cmd.exe`. - Sourced bash functions can be invoked directly from xonsh command prompts without modification.

---

[15:33](https://www.youtube.com/watch?v=uaje5I22kgE&t=933s) The `.xsh` File Format and Modular Imports

To serve as a general-purpose programming language and automation tool, xonsh defines its own script file format with the `.xsh` extension. Sourcing an `.xsh` file—such as an updated version of `owl.xsh` containing a Python function that times network calls using `ping`—evaluates both shell expressions and Python statements, retaining definitions in the active session. Furthermore, xonsh leverages Python's modular import machinery by implementing import hooks. Scopatz explicitly credits core developers like Brett Cannon for making custom import hooks accessible. Users can place `.xsh` files on the Python path and write standard `import owl` statements, subsequently invoking module functions such as `owl.time_ping()` within pure Python workflows.

Beyond script execution, xonsh incorporates an extensive history backend designed for session auditing and reproducibility. Rather than storing a simple flat file of executed commands, the shell records structured metadata for commands, outputs, and environments. Due to time constraints, Scopatz directs attendees to consult the online documentation for the full architecture of the history subsystem.

- Script files use the `.xsh` extension, mixing arbitrary Python and subprocess commands. - Python import hooks allow `.xsh` scripts to be imported like standard Python modules (`import owl`). - Structured session history preserves rich command context to support scientific reproducibility.

---

[17:01](https://www.youtube.com/watch?v=uaje5I22kgE&t=1021s) Architecture: Resolving Syntax Ambiguities

Like traditional compilers, xonsh relies on tokenization, lexing, parsing, compilation, and execution stages. However, bridging shell syntax with Python requires a custom syntax tree transformation phase. In pure Python, whitespace within expressions carries no semantic weight, meaning varying spaces around an expression like `ls -l` compile to identical abstract syntax tree (AST) structures representing subtraction. In shell subprocess mode, whitespace acts as an argument delimiter, transforming `ls -l` into distinct invocations depending on how tokens are separated.

```mermaid flowchart TD accTitle: xonsh Compilation Pipeline accDescr: Step-by-step pipeline from source code to execution showing the AST transformation phase

A["xonsh Source Code"] --> B["Tokenizer & Lexer"] B --> C["PLY LALR Parser"] C --> D["Python AST Nodes"] D --> E{"Syntax Tree Transformation"}

subgraph Transformation ["Transformation Phases"] E --> F["Phase 1: Context-Insensitive"] F --> G["Phase 2: Context-Sensitive"] G --> H{"Leftmost token in Python scope?"} H -- "Yes" --> I["Parse as Pure Python"] H -- "No (valid command)" --> J["Wrap in Subprocess Mode![ ]"] H -- "Fails both" --> K["Raise SyntaxError"] end

I --> L["Python compile() / exec()"] J --> L ```

The lexer and parser (built with PLY) generate standard Python AST nodes, identical to the output of `ast.parse`. Because standard Python AST nodes are produced, xonsh simply feeds the transformed tree directly into Python’s native `compile()` and `exec()` built-ins. To resolve mode ambiguity, the syntax tree transformation uses a context-sensitive heuristic: if the leftmost identifier in an ambiguous statement does not exist in the active Python scope and the line successfully parses as a valid subprocess command, xonsh wraps the statement in subprocess delimiters (`![]`). If the identifier is present in Python scope, “Python always wins and we parse it as Python.” Users who prefer deterministic parsing can manually wrap commands using the `![]` subprocess literal syntax, Scopatz observing, “It's your fingers.”

- PLY generates standard Python AST instances that hand off directly to native `exec` and `compile`. - Subprocess resolution inspects the leftmost symbol: unassigned identifiers fallback to subprocess execution. - Shared runtime helpers and shell hooks are injected directly into Python's `builtins` namespace.

---

[20:15](https://www.youtube.com/watch?v=uaje5I22kgE&t=1215s) Practical Ecosystem Integration, Terminal Graphics, and Wrap-Up

Because xonsh retains immediate access to Python's package ecosystem, terminal commands can be directly piped into complex data processing pipelines. Scopatz showcases fetching remote text over `curl` and feeding the stream into the Natural Language Toolkit (`nltk`) to tokenize and extract sentences in a one-liner. Similarly, outputs from HTTP APIs fetched with `curl` can be piped directly into `json.loads()` and parsed into formatted terminal output without separate intermediary parsing utilities like `jq`.

```python # Streaming JSON API responses straight into Python dictionary parsing !(curl -s https://api.example.com/data) parsed = json.loads(__xonsh__.history[-1].out) print(f"Status: {parsed['status']}") ```

Scopatz highlights the shell's dedicated color management system, noting that customizable color schemes are essential for readability across terminal profiles and provide crucial accessibility for colorblind users. To conclude the demonstrations, he presents a third-party extension hook for Matplotlib and NumPy that renders graphical figures straight into the terminal grid using colored ANSI background cells, generating enthusiastic laughter and applause from the audience. Scopatz closes by highlighting version 0.3.2, urging the community to adopt and contribute to the shell, and referencing the previous year's proposal: “And just remember that a vote for the shell in 2015 yields xonsh in 2016.”

- Web utilities like `curl` pipe directly into standard library modules (`json.loads`) and third-party tools (`nltk`). - Terminal styling supports custom color palettes tailored for visual accessibility and colorblindness. - Matplotlib figures can render directly inside text terminals using ANSI terminal background pixel blocks. - The talk concludes at version 0.3.2, transitioning directly into attendee Q&A.

[23:46](https://www.youtube.com/watch?v=uaje5I22kgE&t=1426s) Prompt Customization, Python Syntax, and Execution Performance

An audience member opens the Q&A by asking whether xonsh includes something equivalent to a `PROMPT_COMMAND`—a hook to re-run logic and construct the prompt every time the user presses Enter (similar to Bash's `PS1`). Anthony Scopatz clarifies that xonsh features a fully customizable `$PROMPT` environment variable documented extensively in the official manuals. Users can configure colors, select desired metadata components, set up right-aligned prompts (`$RIGHT_PROMPT`), and style multi-line prompts.

Another attendee checks whether xonsh is truly a strict superset of Python syntax and whether it includes process substitution from shells like Bash and ksh. Scopatz confirms both points, though he caveats with a laugh that “the process substitution may be buggy still, but it has been implemented.”

When asked how xonsh performs against traditional Unix shells, Scopatz gives an honest assessment of its strengths and current bottlenecks. He admits that “startup times currently suck, but we're working on that,” noting that startup latencies actually worsened in the most recent release, though the core team understands the root cause and knows how to resolve it. On the other hand, interactive runtime performance is solid. The AST compilation and execution pipeline is fast, and interactive features like tab completion outpace traditional shells because xonsh caches completion components directly in memory rather than reading them from disk repeatedly.

```mermaid flowchart TD accTitle: xonsh Interactive Loop and Environment Structure accDescr: Diagram showing how prompt evaluation, user input, AST execution, and shell completions interact.

subgraph Config ["Prompt & Interface"] P["$PROMPT & $RIGHT_PROMPT"] ML["Multi-line Prompt Engine"] end

subgraph Runtime ["Execution Pipeline"] Input["User Command / Script"] --> Parser["PLY Parser (Superset AST)"] Parser --> Exec["Compilation & Execution"] end

subgraph Memory ["In-Memory Caching"] Comps["Tab Completions & Components"] end

Config --> Input Exec -.->|"Fast interactive evaluation"| Input Comps -->|"Instant lookup"| Input ```

- `$PROMPT` controls dynamic prompt generation, with built-in styling, color formatting, and right-hand prompt support. - Strict Python superset syntax allows blending standard Python expressions directly with shell commands. - Process substitution (e.g., `<(command)`) is implemented. - Startup latency is the primary known performance issue, whereas runtime interactive typing and in-memory tab completion remain very responsive.

---

[25:31](https://www.youtube.com/watch?v=uaje5I22kgE&t=1531s) Integrations: Jupyter Notebooks, Virtual Environments, and Bash Profiles

Attendees query xonsh's interoperability with common Python workflow tools, starting with Jupyter notebooks. Scopatz confirms that xonsh ships out of the box with native Jupyter notebook hooks. When asked about `virtualenv`, Scopatz explains that supporting virtual environments was one of the primary historical drivers behind building xonsh's foreign shell interface. Because standard `virtualenv` activation scripts execute unorthodox shell hacks, xonsh bundles dedicated virtual environment commands to provide seamless alternative pathways if standard activation scripts misbehave.

A question is raised about why a tool like xonsh took until 2016 to exist, and when it might become an industry standard. Scopatz quips that it required a “moral compulsion to act, and probably the fact that no one wanted to re-implement the Python language spec, which I did as a stress reliever project while writing a book.” When asked when it will become standard, he jokes, “Hopefully tonight.”

Regarding startup configuration files, Scopatz explains that xonsh's foreign shell interface automatically reads standard files like `.bash_profile` and `.bashrc` by default. Users have full flexibility: they can configure xonsh to source profiles from any supported foreign shell or disable foreign profile loading altogether.

- Jupyter notebook integration is included stock with the xonsh distribution. - Foreign shell interfaces bridge the gap for legacy tools like `virtualenv`, `.bash_profile`, and `.bashrc`. - Built-in fallback commands exist for virtual environments if Bash-specific activation scripts fail.

---

[27:24](https://www.youtube.com/watch?v=uaje5I22kgE&t=1644s) Real vs. Fake Demos, Third-Party Packages, and Magics

An audience member asks about a slight delay observed during an `ls` command in the slide presentation, wondering whether it was caused by terminal latency or a browser extension. Scopatz reveals the secret behind the presentation: “Oh, this was not live. This was me fake typing. Yeah, I was not going to do that live. Did you see that URL for the Weather Underground? Come on.”

The questions shift to extending xonsh with the broader Python ecosystem. Scopatz confirms that users can `pip install` third-party libraries such as `requests` and import them directly on the command line: “Anything—it is Python, so anything that you can import in Python you can import in xonsh directly and just start using it.”

When asked whether xonsh implements IPython-style magic commands or dedicated debugger hooks, Scopatz explains that while an integrated debugger has been requested, none is built-in yet. Furthermore, xonsh avoids blanket magic syntax; Scopatz invites the community to specify exactly which individual magic functions they actually need so they can be implemented purposefully.

- Presentation demos were recorded and simulated via keystroke playback rather than executed live over network APIs. - Any package installed in the active Python environment (such as `requests`) can be imported directly into the shell prompt. - IPython magics are not universally mirrored; additions depend on user demand for specific workflows.

---

[28:38](https://www.youtube.com/watch?v=uaje5I22kgE&t=1718s) Why Python, Reasons Not to Switch, and Line Editing Backends

The discussion turns to architectural language choice. Scopatz is asked whether Python is objectively the best language for building this type of hybrid shell or merely his personal favorite. Scopatz makes the case that Python is uniquely qualified: > “I do feel that Python is the best language for this because it's a high-level dynamic language and you can do this syntax tree transformation stuff dynamically.”

He contrasts Python with alternatives: - **Ruby:** Lacks the broad ecosystem and scoped design needed for such shell integration. - **JavaScript / Node.js:** Lacks critical OS-level primitives: “You can't really manage signals very well. And Python has this beautiful—maybe not beautiful but functional interface for handling signals.” - **Scala / Ammonite:** While the Ammonite shell does something similar in the Scala ecosystem, Scopatz believes xonsh offers a superior developer experience.

When asked if there is any reason *not* to adopt xonsh as a primary daily driver, Scopatz answers: “I don't think so, but if you find them, please report them.”

The session concludes with a question regarding multi-line input and Vi keybindings. Scopatz explains that xonsh supports two distinct line-editing backends: classic GNU `readline` and `prompt_toolkit`. Both backends support Vi mode natively. While setting `$VI_MODE = True` in xonsh engages the functionality, refining a completely unified configuration interface across both backends remains ongoing work. With that, the host wraps up the session to applause, announcing the next talk in ten minutes.

## Notable quotes > [00:01](https://www.youtube.com/watch?v=uaje5I22kgE&t=1s) the purpose of xonsh, at least in my mind, is to trick scientists and engineers into being more reproducible — Anthony Scopatz

## Mentioned Anthony Scopatz, xonsh, Python, Bash, Fish, Zsh, cmd.exe, PLY, Git, Mercurial, IPython, Brett Cannon, Natural Language Toolkit (nltk), curl, Matplotlib, NumPy, Jupyter, virtualenv, requests, Ruby, Node.js, Ammonite, Scala, GNU readline, prompt_toolkit

## Video description Speaker: Anthony Scopatz

Xonsh is general purpose shell that combines Python and the best features of Bash, zsh, and fish. Relying only the standard library and PLY, the xonsh language is a strict superset of Python that compiles to a Python AST. The shell provides exciting features such as a rich history, tab completion from bash and man pages, syntax highlighting, auto-suggestion, foreign-function aliases and more!

Slides can be found at: https://speakerdeck.com/pycon2016 and https://github.com/PyCon/2016-slides

[Source](https://www.youtube.com/watch?v=uaje5I22kgE)

中文简报

Anthony Scopatz 发布 Xonsh：利用 Python AST 替代 Bash Anthony Scopatz 认为，Python 的动态 AST 操作和信号处理使其具备独特优势，能够在不牺牲现有 POSIX 工作流的情况下替代脆弱的 Unix shell。 Scopatz 介绍了 xonsh，这是一种交互式命令行 shell 和通用脚本语言，它作为 Python 的严格超集运行，同时集成了原生 POSIX shell 原语。他展示了 xonsh 如何利用诸如 `@()` 参数展开、`!()` 命令捕获、可调用的 Python 别名以及基于正则表达式的文件 glob 通配等语法，无缝衔接 Python 对象与子进程执行。该架构依赖 PLY 和两阶段语法树转换，将有歧义的命令编译为标准 Python AST 节点，从而允许通过 Python 原生运行时直接执行。在谈及实际集成时，Scopatz 概述了针对 Bash 和 Zsh 环境的外部 shell 桥接、通过 `.xsh` 文件的脚本执行，以及与 Matplotlib 和 NLTK 等包的动态集成，随后回答了听众关于提示符自定义、启动性能和行编辑后端的问题。 Anthony Scopatz 在 PyCon 2016 上推出了 xonsh，这是一款结合了 Python 与 Bash、Zsh 及 Fish 特性的跨平台命令行 shell，它仅依赖 Python 标准库和 PLY。 (0:01) xonsh 语言作为 Python 的严格超集运行，可编译为标准 Python AST 节点，从而实现 shell 命令与纯 Python 代码的无缝混合。 (17:01) 外部 shell 执行桥接允许 source 现有的 Bash、Zsh 和 cmd.exe 脚本，将环境变量、别名和 shell 函数直接加载到 xonsh 运行时中。 (13:14) 该 shell 具有原生 Jupyter notebook 钩子、可导入的 .xsh 脚本文件、内存中制表符补全以及终端渲染的 Matplotlib 图形。 (15:33)

Source quotes

[0:01] Anthony Scopatz: the purpose of xonsh, at least in my mind, is to trick scientists and engineers into being more reproducible

Original transcript

[0:01] (host) Hello everyone. [0:02] Our next speaker will be Anthony Scopatz, and he will be speaking to us [0:05] about a new command-line shell that you might be interested in. [0:08] So please join me in welcoming Anthony. Thank you. [0:10] [applause] [0:12] (Anthony Scopatz) All right. [0:15] All right. Hello everyone. [0:17] I'm Anthony Scopatz and this is xonsh. [0:20] So what is xonsh? [0:22] Xonsh is meant as a general purpose shell language [0:26] to replace things like bash or fish or Z shell, [0:29] but it also is simultaneously a Python superset language. [0:34] And some of the features of this are that we have no external dependencies [0:38] other than the Python library, Python itself. [0:41] And so if you have Python, you also have the ability to have xonsh. [0:46] And because of that, we support, you know, [0:48] the big three major platforms, Linux, Mac, and Windows. [0:51] And sort of the purpose of xonsh, at least in my mind, [0:54] is to trick scientists and engineers into being more reproducible, [0:59] but I don't have time to go into that into this talk, [1:01] so you'll just have to talk to me about that later. [1:03] There's a lot of fancy features, especially if you're willing [1:06] to install some other code, and we'll see some of that in this talk. [1:11] But before that, I just wanted to give some motivating examples [1:14] between bash and other sh languages and what we have available in Python. [1:20] So, you know, here's a simple example of a 'for' loop, right? [1:23] In an sh-based language, you have to do a little more [1:26] for a 'for' loop, not a big deal. [1:29] Our Python version we all know and love. [1:31] [audience member inaudible] [1:35] How do I --? OK. [1:38] Uh, yeah. OK. Well... [1:42] I can do this, maybe? [1:47] No. All right. They promised me something. [1:51] There we go. Yes? No? [1:54] All right. [1:56] Yeah, there is no other resolution that works, so... [2:01] That works? OK. All right. We'll go from here. All right. [2:10] All right. Sorry. [2:13] The best-laid technical plans. [2:15] Woo! All right. Live debugging. This is what we're all about. [2:19] So, another terrible bash example is testing whether a file exists. [2:23] There's a beautiful Python API for doing this. [2:25] And in shell-based lang-- er, in sh-based languages, [2:29] there's a lot of syntax and some weird commands, [2:31] and they overlap with a bunch of other syntax, [2:34] and it's hard to keep straight. [2:36] And you almost always have to look this up. [2:38] And the last example is just tokenizing a string by a non-whitespace character. [2:42] In Python this is super easy, and in shell-based languages, [2:45] you have to set global state and then reset it back [2:48] after you're done tokenizing the string. It's an absolute disaster. [2:50] OK, so you're all motivated right now. [2:54] So the stages of xonsh development sort of went as follows. [2:59] There was this initial conceptualization stage [3:01] where, you know, "This situation cannot stand," [3:04] so what if you had the ability to take the best parts of Python [3:08] and the best parts of bash or some other language? [3:12] And then that, inevitably, led into actually looking up how to do this, [3:16] you know, researching things like ply and other parser libraries out there, [3:21] and coming to the realization that, you know, [3:25] maybe it actually can't be that hard, right? [3:28] And that, inevitably, once you have all the pieces in your toolbox [3:31] and you know how to do it, you see the way to do it, [3:34] you go ahead and you feel that you must build this library. [3:38] And that's kind of where we are at emotionally with xonsh right now. [3:42] And the following stages of anger, regret, and acceptance [3:45] are sure to follow any day now. [3:49] Without further ado, let's actually dive into [3:52] what the language actually looks like in some examples. [3:55] This is best done in a demo, so prepare yourself [3:57] for some really fast typing here. [4:00] All right, so the first thing you'll notice about xonsh [4:02] is that it's really meant to be used as a general-purpose shell, [4:06] so the prompt looks like what you would see in other languages, right? [4:09] It has the -- who you are and where you are in the file system, etc. [4:14] But on the other hand, it really is Python, [4:16] so you can do things like add two numbers together. [4:18] What a novelty! [4:20] Or you can import modules -- that's kind of nice -- [4:23] and access members of that module. [4:26] Or -- whoops. Ah, OK, I'm all messed up here. [4:30] Sorry. [4:32] Yeah, sorry. The re-res messed me up. [4:35] Or you can create dictionary literals [4:37] and access members of those dictionary literals, [4:42] or do more complex statements like 'if' statements, etc., [4:47] or 'for' loops or 'while' loops, [4:49] or define functions in your current execution context, [4:52] and then go ahead and call those functions, right? [4:55] In some very real sense, xonsh is Python. [4:59] And it's all -- but it's a little more than Python [5:01] because we add a bunch of shell-like primitives [5:05] to the Python language. [5:07] So for instance, we add a $ operator [5:10] that works like it would in bash, [5:13] where it looks up variable names in the environment. [5:16] And this is very similar, but the difference here [5:19] is that we're able to go ahead and set variables [5:23] like you would in Python, [5:25] set environment variables like you would in Python, [5:27] and then access those from subprocess commands like you would in bash. [5:32] So xonsh keeps a very clear notion [5:36] of whether you're in Python mode or subprocess mode, [5:40] and we'll talk more about this in the coming slides. [5:43] But you can access and modify the environment, [5:46] you know, in a sort of more Pythonic way. [5:50] And additionally, because it's Python, [5:52] things like your path are really a list of strings, right? [5:56] They're not some crazy colon-separated list of strings. [5:59] And this works for any type. You can have any structure. [6:01] You can have your own custom class that you stick an instance of [6:04] into your environment and then access it that way. [6:07] OK. So that's easy peasy. [6:11] Additionally, we add a ${ } operator, [6:15] and what this does, it's distinct from the same operator in bash. [6:20] So here we look up the environment variable [6:23] based on an expression that's been evaluated inside of the curly braces. [6:28] And so, you know, you can use this to programmatically look up variables [6:32] if that's something that you want to do. [6:34] And of course all of this stuff is available in subprocess mode as well, [6:38] so you can echo "~ cows come $HOME". [6:41] Additionally, if you look up an ellipsis, [6:44] so if you do ${...}, [6:47] this is a reference to the environment itself. [6:50] And so you're able to do things like test for containment [6:54] or access methods of the environment [6:58] just by using ${...}. [7:01] OK. [7:04] Moving right along, it wouldn't be a very good shell [7:06] if you could change directories, so here we've just changed directory [7:09] into the xonsh repository. [7:11] And as you see, when you do that, [7:13] the branch name pops up appropriately colored. [7:17] This works with both mercurial and git, [7:20] and so it's really sort of a developer-facing tool in that regard. [7:24] It's meant to have all of these things that you would like and use. [7:28] Additionally, if we go ahead and type "git sta" and then hit tab, [7:32] you'll see that we have this drop-down menu [7:34] of all the tab completion options that are available. [7:37] And in particular, this is actually just eating [7:41] the bash completions that have already been written. [7:43] So we don't have to necessarily rewrite [7:46] all of the completers for all the commands out there. [7:48] They're just accessible to xonsh. [7:50] And this works even on Windows, which is kind of cool. [7:55] So, let's go back to the home directory. [8:01] Additionally we have this $( ) operator [8:03] which does pretty much exactly the same thing [8:06] that it does in an sh-based language in that it returns to you [8:09] the result of a subprocess as a string. [8:12] No big deal there. [8:14] Additionally, you can -- this is available [8:16] in both Python mode and subprocess mode, [8:18] so you can do $( ) to subprocess -- [8:23] to capture the subprocess as a Python variable, [8:25] and then use Python string manipulation [8:28] to modify that string however you want. [8:32] And that's just a single expression. [8:34] Additionally, we also add an !() operator. [8:38] And what this does is it returns to you a special completed command object [8:42] that contains not only the standard out but all of the metadata [8:46] about how that command was run and what its results were. [8:50] And furthermore, we add a ![ ] operator [8:54] that does pretty much the same thing except that it streams the results to you. [8:58] And then after that, you know, [9:00] after you have this X, you're able to access things [9:03] like the timestamp for when the command started and stopped, [9:06] the alias that the command was run with, [9:08] and we even implement appropriate Boolean semantics. [9:12] So we look at the return value of that completed command [9:17] and test whether it's equal to 0 and return that. [9:21] OK. [9:23] So, so far what we've seen is subprocess mode in Python mode. [9:26] But of course we -- xonsh allows us to go the other way. [9:30] And so, say we have two Python variables here. [9:32] If we do "echo @( )", [9:36] we evaluate what's inside of those parentheses [9:38] as a Python expression, and then pass that -- [9:41] convert that to a string and then pass that down as a single argument [9:44] to any subprocess that we want to call. [9:48] And additionally, you know, this works for anything [9:50] that's string convertible, so you could "echo @(2+2)" [9:52] and actually get the result that you're interested in. [9:56] And furthermore, if you pass in a list of expressions, [10:00] each of those expressions will get sent down [10:03] to the subprocess as individual arguments in the argument list. [10:09] And obviously this stuff is perfectly nestable, [10:12] and so if you're in Python mode and then you go into subprocess mode, [10:15] you can go back into Python mode and everything works out just fine. [10:20] Moreover, you can use this @( ) stuff [10:24] to programmatically determine [10:27] what command you want to run. Whether or not that's useful [10:31] or how much obfuscation you like is up to you. [10:34] But the place where this @() stuff really shines for me [10:37] is the fact that you can now use normal Python looping semantics [10:41] to loop over a set of data, and then run over subprocess commands [10:44] very, very easily, like we see in this example here. [10:48] OK. All right. [10:51] Next, it wouldn't be a very good shell, of course, [10:53] if you couldn't pipe things, so we implemented piping. [10:57] Similarly, it wouldn't be a very good shell if you couldn't redirect things, [11:00] so redirection to a file or to -- from standard error to standard out [11:06] and all those things are available as well. [11:10] OK. [11:14] Additionally, we added, by popular demand, [11:17] the ability to 'and' and 'or' subprocesses together [11:21] using Python's 'and' and 'or' keywords. [11:24] And this -- all this is doing is it's wrapping those subprocess commands [11:28] in ![ ] before it runs them. [11:32] And don't worry if you have a lot of muscle memory [11:34] and you want to use &&||. [11:37] You're welcome to do so. I won't hold it against you. [11:41] And the place that this is particularly useful [11:43] is if you want to test if a file exists and that a file doesn't exist -- [11:47] and that another file doesn't exist. [11:49] If you do that, you'll just get back the appropriate Boolean. [11:53] And this is all, you know, [11:55] using the normal kind of Python 'and' and 'or', [11:58] and parentheses groupings and negation semantics [12:01] that we all know from Python. [12:05] Additionally, we add [12:08] regular expression file globbing using backticks. [12:11] So anything in the backtick [12:14] is compiled as a regular expression [12:17] and then sent down to either the subprocess, [12:20] or it returns a list of strings in Python mode. [12:24] So we could write more complicated regular expressions. [12:26] So if we -- initially we were just looking at all the Python files [12:30] in a directory, but if, you know, we wanted to look at [12:31] all the Python files that didn't include an underscore, [12:35] that's easy to represent as well. [12:37] And to just show that this works in Python mode, [12:39] we can go ahead and take the length of that regular expression. [12:45] OK. Like IPython, we add a '?' operator [12:49] to give you help on whatever object you ask for help on. No big deal there. [12:55] We also add a super help operator, so you can do '??', [12:59] and what this does is, it goes and tries to find the source code [13:03] that the object was defined with, and in xonsh we go ahead and [13:07] syntax highlight that source code for you as well before we print it. [13:12] OK. [13:14] Additionally we have an aliases mapping, right. [13:16] So this just maps command names [13:20] to the command that you want to run. [13:23] Nothing super complex there, except for the fact [13:26] that we have a few different forms of aliases. [13:29] So we have the normal diff -- the normal list of strings-based aliases [13:34] that you would see in a bash or in other kinds of shells. [13:39] But since we're Python, we also have the ability [13:41] to take functions as aliases. [13:43] So here let's just define a little alias [13:47] that takes a lambda, that accepts the standard args -- [13:51] or, accepts an argument list and a standard in. [13:54] And here we're just going to ignore that and return a string no matter what. [13:58] You can also return standard error and return code if you desire. [14:03] And so when we go ahead and run this command on the command line, [14:05] we just get that string printed out. [14:07] But this -- so maybe doing something a little more interesting. [14:10] If we create another lambda that actually reads from standard in [14:13] and maybe formats a string using Python string formatting, [14:18] and then returns that result. So here, once we have this set up, [14:22] we can go ahead and echo a name and then pipe that to our new alias, [14:26] and we see the ominous message that we would expect. [14:31] All right. [14:34] Additionally, [14:37] xonsh has a very well-defined notion of what it means to be a foreign shell, [14:42] and so just as an example here, here's a little bash script [14:46] called owl.sh that does a few different things. [14:48] So the first thing that it does is it exports an environment variable, [14:52] it defines an alias, and it also defines a bash function. [14:56] Now in xonsh, we add a source bash command [15:00] that allows you to source this bash file [15:02] in exactly the same way that you would source it in bash [15:05] And so if we look, we can see that this environment variable [15:08] is loaded into the xonsh environment when we source that file. [15:11] Additionally, we get the alias, as you would expect, [15:14] and we can even call this bash function [15:16] in exactly the same way that you would call it in bash. [15:19] And this all just works. And so this is implemented currently [15:22] for bash, Z shell, and cmd.exe. [15:27] And clearly there's still a big fish in the room here. [15:33] Of course, you know, another one -- another thing that makes -- [15:38] would make xonsh not be very good language is if it didn't have a file format. [15:42] So here we are showing just that same owl.sh script [15:46] written in the .xsh xonsh format. [15:49] And additionally we've added a Python function [15:53] that times a ping command and then returns the result. [15:57] So if you go ahead and source owl.xsh, [16:00] you get that timed ping command loaded into your current Python context [16:04] and it runs just fine. [16:06] Additionally xonsh also has import hooks for any .xsh file. [16:10] So you can go ahead and those are easy to load [16:13] thanks to Brett Cannon and other people. [16:16] And you can just -- if you prefer to import owl [16:19] and then do owl.time_ping, you're welcome to run code that way. [16:23] OK. [16:25] It's worth mentioning that xonsh has a very rich and detailed notion [16:30] of what history is that I don't have time to go into here. [16:33] So you're just going to have to look up the docs on your own, [16:36] or talk to me afterwards, and this was kind of put in there [16:38] for some of these reproducibility concerns that I mentioned before. [16:41] OK. [16:43] That all being said, [16:47] xonsh -- the purpose of that was to show you that xonsh really is Python, [16:51] but it also is meant to be used as a general-purpose shell, [16:53] and so there's a more detailed translation guide [16:57] on the website that you're welcome to check out. [17:01] And how does all of this stuff work? [17:02] Well, it works like most languages work. [17:04] So we go through a number of common phases. [17:06] We have a tokenizer, a lexer, a parser, compilation and execution stages. [17:11] But the thing that's different is that xonsh inserts [17:13] the syntax tree transformation phase. [17:16] And this is super important. [17:18] So, just to go through these one by one, [17:22] the tokenizer and lexer are a little more complicated than you would think [17:25] because xonsh has to keep track at a very deep level [17:29] of whether it thinks you're in Python mode or in subprocess mode. [17:35] Python is not whitespace-sensitive inside of an expression. [17:39] So the three "ls -l" examples up here [17:41] all compile to the exact same Python code. [17:45] However, subprocess mode is exactly whitespace-sensitive. [17:51] So these three "ls -l" expressions [17:54] appear compiled to three distinct pieces of code [17:59] and have different representations and would do different things. [18:02] So we need some way of resolving the ambiguity [18:05] between Python's whitespace-insensitiveness [18:07] and subprocess's whitespace-sensitive aspects, [18:13] which we'll get to in a couple slides. [18:15] The parser is written in ply, and what the parser does is [18:19] it produces a syntax tree of normal Python AST nodes. [18:24] And so when I say xonsh is Python, [18:27] it actually does produce the same exact syntax tree [18:30] that if you called ast.parse you would get [18:34] from the Python standard library. [18:36] And this is really useful because we can just [18:38] hand that syntax tree off to functions like exec and compile. [18:42] And it's then the Python interpreter's job [18:44] and we don't have to care about it anymore. [18:46] And to make all this work, we shove a bunch of things [18:49] into builtins. So sue us. [18:54] The syntax tree transformation phase is really here [18:57] to resolve this ls -l ambiguity. [19:00] And so this has two subphases. [19:01] It has a first phase which is context-insensitive, [19:06] and then the second phase which is context-sensitive. [19:09] And so, to keep this clear, we try to reduce this -- [19:13] we've reduced this to just a couple of simple rules [19:16] for what it means to do context-sensitive parsing [19:18] so that you can gain an intuition [19:20] about how a command's gonna work or when it's gonna run. [19:26] If the -- the rules are basically: if the leftmost name is in -- [19:30] is not in the Python context or Python scope, [19:33] and the expression as a whole can be parsed in subprocess mode, [19:37] so if the ls is not a Python variable, basically, [19:41] then we will go ahead and wrap it in ![]. [19:44] And if the leftmost name is in the Python scope, [19:49] then Python always wins and we parse it as Python. [19:52] And if it doesn't compile [19:55] with either Python mode or in subprocess mode, [19:57] then it's a syntax error and we raise it as such. [20:01] And if you want to avoid this, you're welcome to always type ![ ]. [20:06] It's your fingers. [20:08] Compilation and execution, nothing new and special here. [20:11] So we just hand it off to the Python interpreter. [20:15] So that being said, I wanted to show you some more intense examples [20:20] about sort of the things that you can do with xonsh [20:22] now that you have this language available. [20:24] So here is a ridiculous way to -- [20:28] really obfuscated way of typing ls -l [20:31] using all the xonsh syntax. [20:36] Sort of a silly example. I guess my thing is lost here. [20:40] But another motivating piece [20:43] is that the... [20:47] You have the full Python library -- [20:50] you have all the Python ecosystem and libraries available to you, [20:53] and you can directly pipe things. [20:55] Wow, I'm gonna have to make this a little bigger. I'm sorry. [21:01] Yeah. So, sorry if it gets cut off. Oops. [21:05] So you can directly pipe things from the internet. [21:08] So, say you wanted to go grab the xonsh license. [21:10] You can take nltk and pass the result directly into nltk [21:14] and get the first seven sentences or so as a list, [21:18] just as a single Python expression. [21:22] OK. [21:25] This is also useful in -- wow, this is not working out. I'm sorry. [21:32] Sorry if it gets cut off. [21:37] So, you can also use this to grab -- [21:41] say you have some JSON API from the internet [21:44] and you want to go grab that, you can pipe -- [21:47] use curl to grab that from the internet [21:49] and then directly pass that into JSON.loads [21:52] and use that to format a string. [21:55] So if you deal with a lot of JSON APIs, this is, you know, [21:58] all done as a potentially single expression [22:00] and it will print you that message right away. [22:05] That's something cool that can be done. Additionally, [22:08] xonsh has a very well-developed notion of colors and color management. [22:14] This was put in for a couple of reasons, like perhaps [22:16] you're on a terrible system with really bad color styles, [22:20] but also sort of the more important aspect [22:23] is for people who may be colorblind, [22:26] this is really helpful because you are able to -- [22:29] they're able to select a color palette that works for them. [22:33] All right. So, finishing up here, [22:35] while I'm on the subject of colors, just going to jump ahead here. [22:39] If you import numpy and matplotlib, [22:42] we have a third-party extension system that lets you -- [22:45] that we have written hooks for a matplotlib in, [22:48] and will allow you to just dump out the current figure [22:52] as -- via background colors. [22:55] So, bam. [22:57] [laughter, applause] [23:03] If you like really pixelated graphics, I guess. [23:07] Just to finish up, [23:09] we believe that xonsh is a usable tool right now as a shell. [23:12] There's obviously more to do. We're at version 0.3.2. [23:15] Please, we love to have a community of users and contributors, [23:19] so please, you know, try it out and let us know how you feel. [23:23] And just remember that a vote for the shell in 2015 [23:27] yields xonsh in 2016. Thank you very much. [23:31] [applause] [23:36] (host) Thank you very much, Anthony. [23:38] We do have time for some questions. [23:40] So there are two microphones on either side of the room. [23:42] Please feel free to line up and Anthony can answer. [23:44] We'll start on this side. [23:46] (audience member) Hey, great talk. I think I'm looking [23:47] at my new shell. But one question: [23:50] does there -- is there the prompt command? [23:54] (Anthony Scopatz) The prompt command? [23:56] (audience member) Yeah, so like, every time you hit Enter [23:58] it can run a command as it -- like, to create your prompt? [24:02] Like PS1 command. [24:04] (Anthony Scopatz) Yeah. So you can mod -- [24:05] there's actually a prompt environment variable [24:07] that you can totally customize and modify, [24:10] and this is all the docs. And so, yeah. [24:12] (audience member) OK, so as I go, it could -- [24:14] (Anthony Scopatz) Yeah, it's just called $ prompt, [24:16] and there's huge documentation on how you color that and what aspects you want. [24:20] There's even a write prompt if that's something you're interested in. [24:22] you can customize the multi-line prompt and everything. [24:24] (audience member) OK, awesome. Thank you. [24:29] (audience member) Thank you. [24:32] OK, I've got two questions. [24:34] First, is this a strict superset of Python syntax? [24:38] And second would be, have you implemented [24:41] process substitution from bash and Korn shell? [24:44] (Anthony Scopatz) Yes to both of those. [24:46] (audience member) Awesome. [24:48] (Anthony Scopatz) Now the process substitution [24:49] may be buggy still, but it has been implemented. [24:52] [laughter] [24:53] (audience member) So, just a quick question on performance [24:56] compared to other shells, a few words you can say about that? [25:00] (Anthony Scopatz) Yeah, so startup times currently suck, [25:03] but we're working on that, [25:05] and they got worse on the last release, sadly, but we know how to fix it. [25:09] The actual performance of different pieces, [25:11] so the compilation/execution stage is actually relatively quick, [25:13] and even doing things like tab completing [25:16] is actually much faster than other shells [25:18] because we keep in memory sort of all of these components. [25:22] So in general the actual execution while you type is not so bad. [25:26] It's just the startup time that we need to deal with. [25:28] (audience member) Thank you. [25:31] (audience member) Does it work inside a Jupiter notebook? [25:33] (Anthony Scopatz) Yes it does. We come -- [25:35] we ship stock with a Jupiter notebook hook. [25:39] (audience member) Does xonsh play nicely with virtualenvs? [25:42] (Anthony Scopatz) It does, and we also have a command -- [25:46] so there's been a lot of talk about this, and so worst case scenario, [25:49] we also ship with some virtualenv commands as well. [25:53] But a lot of the foreign shell interface stuff [25:55] was actually motivated by virtualenv. [25:58] Now, virtualemv does some ridiculous things, and that makes it difficult, [26:01] so if you're running into those, we have alternatives for you. [26:04] (audience member) Perfect, thank you. [26:06] (audience member) Amazing product. [26:09] Why do you think it is that it took us until the year 2016 [26:11] to get to something like this? [26:13] And how long do you think it'll be before this becomes standard, [26:16] and then before we can settle on something [26:19] a little bit less Frankenstein's-monster-feeling [26:21] and more just, like, pure shell awesomeness, [26:24] which is where this is taking us? [26:28] (Anthony Scopatz) Why do I think it took until 2016? [26:31] It was that moral compulsion to act, and probably the fact [26:33] that no one wanted to re-implement the Python language spec, [26:36] which I did as a stress reliever project while writing a book. [26:40] (audience member) So you're a hero. [26:42] (Anthony Scopatz) And then the -- [26:44] see, writing a book can be effective. [26:46] And then, how long does it take till this becomes standard? [26:49] Hopefully tonight. [26:51] [laughter, applause] [26:55] (audience member) Yeah, so this is insanely awesome. [26:58] What's the configuration scheme and does it work with, like, [27:01] bash profiles, bash rc, any of that? [27:03] (Anthony Scopatz) Yeah, actually, so another reason [27:05] for the foreign shell interface is that by default, [27:08] it will load your bash profile and everything. [27:11] But you can set it up to load your profile [27:14] from any other shell that we have hooks for, [27:16] or set it up to load profiles from no shell. [27:19] And so, yes, it's -- we've rounded that out. [27:24] (audience member) I noticed when -- well, [27:26] when you ran ls earlier, the output was a little bit slow to come through. [27:29] Is that because of the Chrome extension [27:31] that you're using, or is that a performance thing? [27:33] (Anthony Scopatz) Oh, this was not live. [27:35] This was me fake typing. [27:37] Yeah, I was not going to do that live. [27:39] Did you see that URL for the weather underground? Come on. [27:42] [laughter] [27:44] (audience member) Thank you. [27:47] (audience member) I was wondering if, [27:49] in addition to standard libraries, can I install, like, requests [27:52] and then suddenly start using requests as part of the command line? [27:56] (Anthony Scopatz) Yes, absolutely. Yeah. [27:57] Anything -- it is Python, so anything that you can import [28:01] in Python you can import in xonsh directly and just start using it. [28:05] (audience member) Perfect, thanks. [28:07] (audience member) Hi, I was wondering if you have support [28:09] for something similar to some of the IPython magic functions, [28:13] where you can do, like, debugging and that kind of stuff. [28:17] (Anthony Scopatz) So, we don't implement [28:19] any particular debugger support, although that has been requested [28:22] and hopefully someone will do that. [28:25] The magic function stuff we don't have support for, [28:28] but my question for the magic stuff is, [28:30] what magics are people interested in, right? [28:34] And then we can talk about how to implement those features specifically. [28:38] (audience member) Do you feel that Python is the best [28:40] language choice for this, or is it just the one [28:42] that you happen to like the most of time? [28:44] (Anthony Scopatz) I do feel that Python is the best language [28:46] for this because it's a high-level dynamic language [28:49] and you can do this syntax tree transformation stuff dynamically. [28:53] You know, just the other dynamic languages out there, like, [28:57] I don't think there's a good scope. So Ruby is an example. [29:00] In Ruby, they don't have the same kind of scope and broad ecosystem. [29:04] And languages like JavaScript, even if you're using node, [29:09] they don't have the -- [29:12] they don't really have the OS-level things that you would need, right? [29:16] You can't really manage signals very well. [29:18] And Python has this beautiful -- maybe not beautiful [29:21] but functional interface for handling signals. [29:24] And so, you know, there is another project [29:27] called ammonite out there that does something similar for scala, [29:30] but I think we're doing a little better than them currently. [29:36] (audience member) Seems like an awesome system. [29:37] Are there reasons I should not switch to xonsh as the shell I always use? [29:42] (Anthony Scopatz) I don't think so, [29:44] but if you find them, please report them. [29:48] (audience member) Wait, is there time for one more? [29:51] (host) Last question, sure. [29:53] (audience member) So how does the -- [29:55] do you have, like, multi-line line editing? [29:57] (Anthony Scopatz) Uh-huh. [29:59] (audience member) And is there, like, a VI mode, or not yet? [30:02] (Anthony Scopatz) Yes, there is VI mode. [30:04] So this is an important point. We have two back ends for line editing, [30:07] so we use readline with all of its problems and glory. [30:11] And then we also use prompt toolkit. [30:14] And both of those have VI modes available. [30:17] But the interface to that, we haven't -- [30:19] we don't have a unified interface for -- [30:21] in xonsh you can just set VI mode -- you know, $ VI mode = true, [30:25] and then you get VI mode. [30:29] (host) OK, let's thank Anthony again. [30:31] [applause] [30:34] And the next talk will start in this room in about 10 minutes. Thank you.