The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019
GOTO Conferences
AI-assisted summary of The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 — original source. Check the original for full context.
Source published:
Key ideas
- The true foundation uniting Erlang and Elixir is the BEAM virtual machine, which acts essentially as an operating system for code capable of spawning up to 130 million lightweight processes inside a single OS process. (0:00)
- BEAM enforces preemptive context switching every millisecond via reduction counting, allowing a single-core instance to maintain an uninterrupted 10,000 ops/sec throughput even when a concurrent task triggers an infinite CPU-bound loop. (15:36)
- Operators can attach a live remote IEx shell to a saturated production node to inspect stack traces, stream runtime function calls, and execute unconditional process termination (Process.exit(pid, :kill)) without dropping user connections. (20:11)
- By providing native concurrency, distributed location transparency, and hot code reloading within a single runtime, BEAM eliminates the traditional trade-off between rapid MVP development and long-term enterprise scalability. (39:58)
Detailed notes
## Background Published on YouTube by GOTO Conferences on May 22, 2019, this presentation from GOTO Chicago 2019 features Saša Jurić, author of Elixir in Action. According to the session description, the presentation looks past syntax to explore the concurrency model of Erlang and Elixir, using theory and live demonstrations to explain why the underlying BEAM runtime is uniquely suited for building highly available, scalable server-side systems.
## TL;DR Dedicated to the memory of Erlang co-inventor Joe Armstrong, the presentation explores how the BEAM virtual machine delivers fault tolerance and high availability through lightweight process isolation, preemptive scheduling, and message passing. Through live demonstrations running on a single CPU core, Jurić shows how BEAM isolates unhandled exceptions, preempts tight CPU-bound loops, and enables live remote introspection and zero-downtime hot code reloading while sustaining 10,000 background operations per second. He also showcases distributed clustering via libraries like Swarm, contrasting BEAM's unified runtime with the fragmented operational overhead of mainstream multi-tier architectures.
## Atmosphere & style The presentation carries an enthusiastic, fast-paced, and pragmatic energy, underpinned by Jurić's dry humor and obvious passion for the technology. Watching the live demonstrations feels thrilling and confident as Jurić deliberately stresses the system with runaway infinite loops and hard kills, calmly navigating live shells and production charts to prove BEAM's resilience. The narrative balances a reverent tribute to Joe Armstrong with a bold, refreshing critique of modern infrastructure sprawl, creating an engaging and persuasive experience for backend engineers.
## Bulletin - [00:00](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=0s) The true foundation uniting Erlang and Elixir is the BEAM virtual machine, which acts essentially as an operating system for code capable of spawning up to 130 million lightweight processes inside a single OS process. - [15:36](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=936s) BEAM enforces preemptive context switching every millisecond via reduction counting, allowing a single-core instance to maintain an uninterrupted 10,000 ops/sec throughput even when a concurrent task triggers an infinite CPU-bound loop. - [20:11](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1211s) Operators can attach a live remote IEx shell to a saturated production node to inspect stack traces, stream runtime function calls, and execute unconditional process termination (Process.exit(pid, :kill)) without dropping user connections. - [39:58](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2398s) By providing native concurrency, distributed location transparency, and hot code reloading within a single runtime, BEAM eliminates the traditional trade-off between rapid MVP development and long-term enterprise scalability.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=0s) Erlang and Elixir serve primarily as distinct language interfaces to harness the underlying capabilities of the BEAM virtual machine. - [02:46](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=166s) A BEAM process is an isolated sequential runtime execution context with its own stack, heap, and garbage collection cycle, communicating purely through asynchronous message passing. - [06:00](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=360s) The entire BEAM system runs inside a single operating system process, scheduling up to tens of millions of lightweight processes across CPU cores using dedicated scheduler threads. - [13:09](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=789s) Structuring applications with dedicated processes per connection and per task establishes failure isolation, preventing unhandled exceptions from terminating the broader system. - [15:36](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=936s) Preemptive scheduling based on reduction slices ensures fair CPU allocation, isolating system latencies so that computationally intensive jobs do not block background traffic. - [17:39](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1059s) High availability in BEAM focuses on resilience and user service continuity under partial system failure rather than chasing theoretical uptime metrics. - [20:11](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1211s) Live remote introspection via interactive shells allows on-call operators to diagnose performance bottlenecks and runaway processes directly within production environments without node restarts. - [24:48](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1488s) BEAM provides first-class process cancellation mechanisms that terminate rogue workers while allowing supervising processes to capture exit signals and report structured errors. - [26:29](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1589s) Hot code reloading enables deploying compiled patches directly to active nodes to resolve boundary bugs and improve algorithmic complexity without service downtime. - [29:51](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1791s) BEAM natively exposes cluster primitives that treat process spawning and messaging identically across local cores and remote network nodes via location-transparent PIDs. - [33:20](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2000s) Embedding distributed coordination libraries like Swarm runs orchestration logic as internal supervision trees, eliminating dependencies on external coordination daemons. - [36:30](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2190s) Consolidating caching, queuing, and real-time mechanics into a single language and OS process drastically lowers the architectural and onboarding complexity barrier. - [38:20](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2300s) Technical uniformity across the development lifecycle removes the necessity for fragmented technology silos and brittle external configuration tooling. - [39:58](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2398s) The BEAM runtime dismantles the conventional compromise between fast early-stage prototyping and robust, distributed high-scale production systems.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=0s) Honoring Joe Armstrong and the Core Focus of the Talk
Saša Jurić opens the presentation by dedicating it to the legacy of Joe Armstrong, the principal inventor of Erlang, who had recently passed away. Reflecting on Armstrong’s impact, Jurić highlights that everything presented exists first and foremost because of his immense contribution to the field of software engineering. Speaking from personal experience, Jurić describes working with Erlang and Elixir as by far the favorite part of his own professional career, and he hopes to spread some of that happiness by explaining why these technologies are so exciting.
He frames the presentation as a journey toward discovering the "soul" of Erlang and Elixir. While audience members might feel they are getting two languages for the price of one—and Jurić notes he could easily toss a few more BEAM languages into the mix—his real focus is the shared foundation underneath them: the Erlang virtual machine, commonly known as the BEAM. Because Jurić views the languages primarily as interfaces used to communicate with and harness the power of the BEAM, he intends to treat them interchangeably. Although most code samples and demos will be in Elixir, the concepts apply equally to Erlang or any other BEAM language.
Jurić outlines the talk in three main segments: - A basic introduction to the principles and mechanics of BEAM concurrency. - A central, demo-driven exploration of BEAM concurrency in practice, examining how and why systems are split into concurrent parts and the tangible benefits gained. - A final, somewhat controversial think piece examining an alternative architectural style for constructing software systems—one that runs counter to mainstream industry trends, yet offers a simpler, more powerful, and flexible foundation.
Warning the audience that his plan is ambitious and time is short, Jurić admits he will speak quickly: “Basically whatever I'm about to show you exists first and foremost owing to Joe's immense contribution to our field and speaking personally for myself I'm extremely grateful for that.”
[02:46](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=166s) The BEAM Concurrency Model and Processes
Jurić introduces BEAM-style concurrency as Joe Armstrong originally envisioned it, centered on the foundational abstraction called a process. In BEAM terminology, a process is an isolated runtime execution context. All runtime code, such as Elixir code, runs inside some process. Crucially, a BEAM process is neither an operating system process nor an operating system thread.
At its core, a process is strictly a sequential program with zero internal concurrency. Inside a single process, execution follows standard sequential mechanics: invoking functions, passing parameters, handling return values, branching, and executing loops. Within this sequential scope, Elixir exhibits functional characteristics. Concurrency is only achieved when multiple processes run at the same time. The most fundamental mechanism to start a process is the standard library function `spawn`, which takes an anonymous function (a lambda) and begins executing it in a newly created process. As soon as the new process is spawned, the originating process immediately resumes its own work.
```mermaid flowchart LR accTitle: Message Passing Concurrency in BEAM accDescr: Process A spawns and sends a message to Process B, which retrieves it from its mailbox queue.
subgraph Sender["Process A (Sequential)"] PA["Run code"] --> SP["spawn(fn ->... end)"] SP --> SND["send(target_pid, message)"] end
subgraph Receiver["Process B (Sequential)"] MB["Mailbox (FIFO Queue)"] --> RC["receive do... end"] RC --> PR["Process Message"] end
SP -.->|"creates"| Receiver SND -->|"delivers term"| MB ```
Once running, these processes exist as two completely independent programs. They share nothing: they have separate execution flows, separate memory spaces (each with its own stack and heap), and separate garbage collection cycles occurring at independent times. The only mechanism they have to coordinate or cooperate is message-passing concurrency. A process sends a message to another process using the primitive `send` function, addressing the target process directly. Messages can consist of any arbitrary Elixir data term. The receiving process retrieves messages sequentially from its internal mailbox, which behaves like a FIFO queue.
An audience member interjects to ask how processes find or address each other. Jurić clarifies that process addressing is handled through process identifiers (PIDs) or process registration, noting that it is a rich topic in its own right and that concrete examples will follow shortly in the demos. He then draws an analogy between BEAM processes and microservices architecture: both models feature independent entities that share no state and communicate exclusively via asynchronous messages. However, with BEAM processes, there is no network overhead, no HTTP protocol overhead, and no need for explicit serialization or deserialization; the runtime natively provides the entire mechanism with extreme efficiency.
As Jurić summarizes the essence of this isolation: “We have two completely independent programs running completely separately from each other they have nothing in common with each other at all.”
[06:00](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=360s) Under the Hood: The BEAM Runtime and the First Live Demo
Jurić explains the physical execution of a BEAM system on standard hardware. A typical project, consisting of regular source code files in a folder structure, is started via a single command. That command boots exactly one operating system process: an instance of the BEAM virtual machine. The entire application runs inside this single OS process.
Within that single OS process, developers can spawn thousands, hundreds of thousands, or even millions of lightweight BEAM processes. The virtual machine imposes an architectural upper limit of around 130 million concurrent processes. Because their memory overhead and initial startup costs are negligible, spawning large numbers of them is standard practice. To schedule these processes across hardware resources, the BEAM starts a small pool of OS threads called schedulers—by default allocating one scheduler per available CPU core. Conceptually, one can imagine a central queue of processes ready to execute: a scheduler pulls a process from the queue, runs it for a small slice of work, places it at the end of the queue, and picks up the next one. While the actual runtime implementation is considerably more advanced, this mental model accurately conveys how massive concurrency is managed without heavy OS thread context switching.
```mermaid flowchart TD accTitle: BEAM Runtime Architecture accDescr: A single OS process hosts schedulers mapped to CPU cores, running thousands of lightweight BEAM processes.
subgraph OSProc["Single Operating System Process (BEAM VM Instance)"] subgraph Hardware["CPU Core Mapping"] S1["Scheduler 1 (OS Thread)"] --- C1["CPU Core 1"] S2["Scheduler 2 (OS Thread)"] --- C2["CPU Core 2"] SN["Scheduler N (OS Thread)"] --- CN["CPU Core N"] end
subgraph RunQueue["Execution Scheduling"] Q["Processes Pending Execution"] S1 --> Q S2 --> Q SN --> Q end
subgraph LightweightProcs["Thousands to Millions of BEAM Processes"] P1["Process 1\n(Isolated Heap/Stack)"] P2["Process 2\n(Isolated Heap/Stack)"] P3["Process 3\n(Isolated Heap/Stack)"] PX["Process 10,000+\n(Isolated Heap/Stack)"] end
Q -.-> P1 Q -.-> P2 Q -.-> P3 Q -.-> PX end ```
Moving to the practical implications, Jurić transitions into a series of live demonstrations to answer how systems should be split into processes and what concrete architectural benefits emerge. He brings up an example system implemented in Elixir that is already running on his laptop as a headless background daemon, started identically to how it would run in a production environment. Everything shown on screen runs strictly inside that single OS process, with no external auxiliary processes or dependencies.
Navigating to an internal administrative dashboard, Jurić initiates synthetic load by triggering background jobs. Each individual job is simply an independent sequential process. Within a single second of clicking the button, 10,000 separate processes are spun up and actively running inside the VM. Each of these 10,000 processes executes an infinite loop that runs a short CPU-bound calculation to simulate realistic system work, after which it transmits an update message to a designated, well-known process in the system.
- Operating system footprint: exactly one OS process runs the entire BEAM node. - Concurrency scale: BEAM supports up to approximately 130 million lightweight processes inside that single OS process. - Hardware scheduling: by default, the BEAM runs one scheduler per available CPU core to distribute the lightweight processes. - Demo workload: 10,000 distinct processes were launched simultaneously in less than one second, each running an infinite loop containing a CPU-bound payload and message passing.
[08:45](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=525s) Benchmarking Architecture and Single-Scheduler Simulation
Saša Juric walks through the live dashboard metrics that monitor the demonstration system. The workload consists of thousands of synthetic operations: a singleton process acts as a well-known, addressable reference point that marks a success for each iteration, sleeps for a single second, and repeats the cycle. Other processes aggregate this reporting data and transmit it over to the browser page. The dashboard displays a real-time line chart on the left, indicating an expected baseline of roughly 10,000 successes per second. Juric notes that the line is not perfectly flat: small oscillations appear because the operational logic is not rigidly synchronized, and the modest machine running the demo is also handling heavy client-side SVG rendering inside the browser.
To expose the inner workings of the runtime under severe constraints, Juric explains that he deliberately constrained the virtual machine's concurrency settings. While the host machine has four CPU cores and the BEAM runtime would normally run four scheduler threads, Juric downgraded the configuration to run on a single scheduler thread. This simulates running on a single-core CPU. Despite the synthetic load of 10,000 operations per second, the single scheduler on the right-hand dashboard remains relaxed, running at a moderate workload of only around 20% CPU capacity.
```mermaid flowchart LR accTitle: System Process Architecture and Client Connection accDescr: Flow diagram showing how synthetic metrics, client WebSockets, and calculation tasks interact within BEAM.
subgraph BackgroundLoad ["Background Metric Pipeline"] A["Singleton Process"] -->|"Records success"| B["Aggregator Processes"] B -->|"Publishes stats"| C["Dashboard SVG Chart (~10k ops/sec)"] end
subgraph ClientFlow ["User Request Flow"] Browser["User Browser"] <-->|"WebSocket"| Conn["Connection Process"] Conn -->|"Spawns"| Task["Calculation Process (Task)"] Task -->|"Returns result or crash notice"| Conn end ```
[10:30](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=630s) Web Interface and Process Separation per Request
With the background load humming along, Juric navigates to a simple user-facing web interface. He describes it as capturing what he views as the fundamental essence of any web application: an input field where a user supplies data, followed by backend processing that returns an output. In this case, the application takes a natural number $n$ and computes the sum of the first $n$ integers (for example, entering $3$ yields $1 + 2 + 3 = 6$, and entering $4$ yields $10$).
Behind the interface sits a deliberate process design. Whenever a browser visits any page in this system, a persistent WebSocket connection is opened. All communication flows asynchronously over that socket. Critically, each individual WebSocket connection is owned and managed by its own dedicated BEAM process: - If three users connect, exactly three connection processes exist. - If a million users connect simultaneously, there will be one million connection processes. - If everyone disconnects, the count drops back to zero.
The internal architecture within a single connection goes one step further: the math calculation is offloaded to a separate, one-off, ephemeral process. When the connection process receives a calculation request, it spawns a calculation worker process (an Elixir `Task`), receives the computed integer asynchronously, forwards the response over the WebSocket to the browser, and lets the calculation process terminate normally. Juric points out that this structure was chosen for a deliberate architectural goal: to provide fault tolerance and maintain high availability under production stress.
[13:09](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=789s) Separating Failures: Surviving Unhandled Exceptions
To test the system's fault isolation, Juric demonstrates the most common production problem: an edge case leading to an unhandled runtime exception (analogous to dividing by zero, passing a negative number to a square root, or encountering `undefined is not a function`). In this application, submitting the input `13` triggers a deliberate crash. The user interface immediately displays an error badge. Tailing the background service logs reveals an unhandled runtime error where a `Task` crashed and terminated abnormally.
Despite the crash, the wider system remains completely unaffected. When Juric repeatedly spams input `13` to generate a stream of crashes, the dashboard line chart remains steady at 10,000 successes per second. Juric emphasizes the core BEAM design philosophy: > “Now an important thing about BEAM is that when a process crashes everything else progresses right, so most of the system still works.”
Furthermore, failures in BEAM are transparent rather than silent or fatal to the host node. When a process terminates—normally or abnormally—any interested process linked or monitoring it receives a failure message explaining that a particular process died due to a specific error reason. Because the parent connection process receives that notification, it does not drop the WebSocket connection; instead, it catches the termination event, maps it to the user's action, and reports a clean error message back to the web UI.
- Unhandled exceptions are isolated strictly within the single-use calculation process. - Background throughput does not dip below its 10,000 operations per second baseline during repeated crashes. - Sibling processes monitor workers via asynchronous messages to format and handle errors gracefully without crashing the connection.
[15:36](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=936s) Separating Latencies: Preemption and Fair Scheduling
Juric introduces a second major rationale for process isolation: latency separation. The application's summing algorithm is written in an intentionally inefficient, brute-force manner, computing $\sum_{i=1}^{n} i$ via a CPU-bound loop from $1$ up to $n$. Submitting an enormous input—specifically the nine-digit number `999,999,999`—triggers heavy, continuous CPU calculation.
When Juric submits this input, the calculation takes about ten seconds to complete, pushing the single scheduler thread straight to 100% CPU utilization. Yet, throughout that computation, the dashboard continues to record an uninterrupted 10,000 successes per second. Even though the application is pinned to a single operating system thread with no explicit `yield` calls or concurrency trickery in the user code, the background traffic does not stall.
Juric explains that this fairness is enforced directly by the BEAM virtual machine scheduler, which executes true preemptive context switching: > “The thing is that BEAM scheduler performs very frequent context switching like in less than 1 millisecond intervals and it does it with proper preemption.”
Every process, whether doing network I/O or tight CPU arithmetic, is allocated a short reduction slice before being preempted and swapped out. The runtime intentionally prioritizes overall system throughput and fair resource distribution over maximizing the raw execution speed of any single, isolated task—an architectural trade-off fine-tuned specifically for resilient multi-tenant server software.
[17:39](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1059s) Surviving Stress and the True Meaning of High Availability
Saša Jurić highlights that separating failure domains and separating latencies are the two most fundamental reasons to split work across Erlang processes. To demonstrate this in the running system, he submits an intentionally CPU-intensive input (`456789`) through the UI. Because the calculation runs in its own dedicated process, the client connection remains responsive, allowing him to immediately issue subsequent inputs and trigger intentional errors without locking up the interface. Despite these artificial stresses, the overall workload across the cluster continues to progress steadily, and the individual user still receives their intensive computation result once it completes.
```mermaid flowchart LR accTitle: User Connection and Isolated Calculation Flow accDescr: The user connection process delegates a heavy task to a dedicated calculation process while remaining free to handle other requests. subgraph Client ["Client Session"] C["Connection Process"] end subgraph Isolation ["Process Isolation"] Calc["Calculation Process<br/>(CPU-intensive task)"] Other["Subsequent Input /<br/>Error Processing"] end C -->|"spawns & monitors"| Calc C -->|"processes immediately"| Other ```
Jurić draws an analogy to a ship that stays afloat even when taking on water: minor leaks should never sink the entire vessel. In his view, true reliability is not an abstract chase for arbitrary metrics, but rather making sure real users can continue getting work done. As he remarks, “this in my mind is what high availability is about it's not about chasing some mythical amount of nines of uptime it's about having a system which is there for its users which is ultimately its primary if not the only purpose.”
[18:44](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1124s) The Infinite Loop Rogue Activity
While the system is resilient against isolated crashes and high latency, certain bugs can pose an existential threat to the runtime if left unmitigated. Jurić demonstrates one of the most insidious failure modes: feeding an invalid domain input, such as the negative number `-1`. The software fails to reject the value upfront, nor does it return an error. Instead, the computation hangs indefinitely while the scheduler utilization chart shoots up to a sustained 100%.
Under the hood, this rogue activity is running an unyielding, infinite CPU-bound loop—the functional equivalent of `while (true);`. Yet because of preemptive scheduling in the BEAM virtual machine and careful process design, the background throughput does not crash; it holds at its baseline rate of 10,000 successful operations per second. However, Jurić warns that this grace period is finite: if users retry failed requests or more clients hit the same codepath, those runaway loops will compound until the server grinds to a total halt. Diagnosing and resolving the issue before a widespread outage occurs becomes urgent.
- Invalid negative input `-1` bypasses domain validation and triggers an infinite CPU-bound loop. - Preemptive scheduling keeps the system operating at ~10,000 requests per second despite the runaway process. - Multiple unconstrained loops would eventually saturate all available scheduler capacity, making quick operator intervention essential.
[20:11](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1211s) Investigating the Live BEAM via Remote Console
Jurić steps into the shoes of an on-call operator who has just received a high-CPU alert. Connecting over SSH, he first inspects the system log files. The logs are entirely static—aside from earlier recorded errors caused by testing the number `13`, there are no traces or warnings explaining where the CPU cycles are going. Traditional logging falls flat when an application is stuck in an active, non-crashing loop.
```mermaid flowchart TD accTitle: Remote Operator Diagnostics Flow accDescr: An operator connects to the BEAM node via remote shell and queries runtime introspection functions. Operator["Operator (SSH)"] --> Shell["Remote IEx Shell<br/>(Node Context)"] Shell --> Top["runtime.top()<br/>List Top CPU Spenders"] Top --> Suspect["Identify Culprit PID<br/>0.12605.0"] Suspect --> Inspect["Process Introspection<br/>Stack Trace & runtime.trace()"] Inspect --> Terminate["Process.exit(pid,:kill)<br/>First-Class Runtime Cancellation"] ```
Fortunately, BEAM provides built-in introspection without requiring preconfigured diagnostic flags or process restarts. Jurić launches a remote `iex` shell that connects straight into the running node, executing commands within the live environment. Because the single-threaded scheduler guarantees fair execution time, the debugging shell itself remains fully responsive alongside the active 10k ops/sec background workload. Jurić notes that the BEAM's cooperative guarantees extend directly to the operator: “going to beams scheduling I am actually able to even do this to debug the system itself.”
- Log inspection yields zero data about an infinite loop since no exceptions are being thrown. - A remote IEx console attaches to the live production instance without restarting the application. - BEAM scheduler fairness allows diagnostic shell interactions even while a core is pinned at 100% capacity.
[21:42](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1302s) Hunting the Culprit with Reductions and Tracing
To find the specific process devouring compute resources, Jurić inspects the runtime process table. Calling `Process.list()` returns a list of PIDs (process identifiers), each representing an isolated unit of execution. Passing a PID into `Process.info/1` exposes core runtime metrics maintained by BEAM, most notably the *reductions* count. Reductions are a non-negative counter that acts as a proxy for the total instructions a process has executed since its creation.
By sampling reduction deltas across all processes over time, an operator can generate a `top`-like ranking of CPU consumption. Jurić invokes a pre-written helper function, `Runtime.top()`, which queries reductions over a one-second window and returns the highest spenders alongside their percentage of CPU usage. A single process—with PID `<0.12605.0>`—stands out as the clear outlier. Running the helper a second time confirms that this exact PID continues to consume nearly all the processor time.
```elixir # Conceptual representation of identifying the rogue process iex> Runtime.top() [%{cpu: 98.4, pid: #PID<0.12605.0>},...] iex> pid = hd(Runtime.top()).pid #PID<0.12605.0> iex> Process.info(pid,:current_stacktrace) ```
With the offending PID assigned to a variable, Jurić pulls its active stack trace using `Process.info(pid,:current_stacktrace)`. Even though the process is trapped in an infinite loop and cannot voluntarily yield or process incoming messages, the runtime itself inspects the memory directly. To supplement the static snapshot, he executes `Runtime.trace()`, which dynamically hooks into the process and streams the next 50 function invocations (or up to one second of execution) to show precisely what code is being run.
[24:48](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1488s) Killing the Rogue Process and Runtime Cancellation
Having captured the stack trace and execution traces, Jurić terminates the rogue process by invoking `Process.exit(pid,:kill)`. This sends an unconditional exit signal—the BEAM equivalent of a UNIX `kill -9` directed at a specific lightweight process. The moment the function executes, the CPU scheduler graph on the monitoring dashboard plummets back to normal levels, while the continuous 10,000 requests per second remain undisturbed.
Crucially, the client UI does not hang; it immediately surfaces an error notification. Because the user's connection was governed by a separate connection process monitoring the calculation worker, the violent destruction of the worker process was captured and translated into a clean, structured failure message. As Jurić explains, “the calculation process itself was brutally killed and you can truly try catch your way out of that one but because we have another process which gets the information about it this is the connection process we are able to consistently report error information back to the user interface.”
He emphasizes that BEAM's exit signaling delivers true first-class cancellation at the VM level. This mechanism can be triggered manually from an interactive shell or wired programmatically into timeouts and client disconnections, wiping out runaway work without leaving lingering threads or corrupting state. With the system stabilized, Jurić turns back to the gathered stack traces and module traces to analyze the root cause of the bug.
[26:29](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1589s) Tracking the Infinite Recursion and Hot-Fixing Production
```mermaid flowchart TD accTitle: Hot Code Reloading and Fix Pipeline accDescr: A flowchart depicting the diagnosis of the infinite loop, testing of the mathematical fix, and runtime hot patch without restarting the node.
subgraph Edge ["System Edge"] Input["User Input (-1, 0, 13, 9999...)"] --> Guard["Input Validation & Boundary Guards"] end
subgraph Engine ["Deep System Logic"] Guard -->|"Positive Integers Only"| Math["Gauss Sum: n * (n + 1) / 2"] Guard -.->|"Pre-fix Invalid Input"| InfiniteLoop["Unchecked Recursion: arg1 + 1, arg2 - 1"] end
subgraph Deploy ["Runtime Hot Reload"] Math --> Tests["Local Unit Tests Pass"] Tests --> HotPatch["Recompile & Inject Bytecode"] HotPatch --> RunningNode["Production BEAM Node (Zero Downtime / 10k req/s sustained)"] end ```
Looking deeper into the dynamic runtime inspection view, Saša Jurić examines the module name, file path, and exact line number to place himself in the right context within the codebase. The runtime inspection reveals what the runaway process was actually doing: it was stuck in a constant tail-recursion—the functional programming equivalent of an infinite loop. Inspecting the function arguments at each successive frame of the call stack tells the whole story: the first parameter incremented by one on every step, the second parameter decremented by one, and the third parameter grew by an irregular amount.
With the stack trace and the loop arguments laid bare, the underlying design bug becomes clear. The function running deep inside the system was written under the rigid assumption that its argument would always be a positive integer, failing completely when given any other input. As Jurić explains, that internal assumption is actually fine for the business domain, because the core domain models natural numbers. The real fault lies at the edge of the system, where incoming user input must be sanitized and rejected before it can contaminate internal components. While the edge already rejected certain inputs, it failed to cover every boundary condition: “It's a classical kind of bug, you know, I haven't thought of all possible scenarios.”
Jurić repairs the code live, joking that “all is fair in demos” as he uncomments the missing guard clauses and fixes all three flaws at once: - **Input Validation**: Edge cases like negative numbers and zeros are caught and cleanly rejected with an explicit error. - **The Number 13**: The synthetic bug triggered by this specific edge-case value is removed. - **Algorithmic Complexity**: Instead of naively summing numbers one by one, he substitutes the standard closed-form formula from high school math: $\frac{n(n + 1)}{2}$.
He runs the test suite to verify the logic, and everything passes. He then executes a deployment task to inject the updated compiled modules straight into the running production system without restarting the BEAM node. The dashboard proves zero downtime occurred: the throughput line never dips below its steady 10,000 successful operations per second. Testing the web page live confirms the fix instantly without even a browser reload. Submitting `-1`, `0`, or `-123` returns an explicit `invalid input`; submitting `3`, `4`, and `13` returns correct results; and submitting a massive string of nines completes as fast as lightning. Reflecting on the ease of observing, diagnosing, and hot-patching an active system, Jurić argues:
> “As a writer of systems and ultimately one of people who is responsible for managing those systems in production, this is what I want from my tool.”
He notes that he knows of no other runtime technology that comes remotely close to the operational service BEAM provides.
---
[29:51](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1791s) BEAM Clustering and Distributed Primitives
Jurić turns to the BEAM's native support for distributed computing. While developers typically run one BEAM instance per physical or virtual machine, multiple instances can easily run on a single local host for testing. Connecting two separate BEAM instances over a standard TCP connection requires only a single function invocation. Once linked, the instances form a single BEAM cluster, and each instance within that cluster is referred to as a node.
Within a cluster, Erlang and Elixir expose low-level distributed primitives directly to the application layer. The programmer can enumerate all active nodes in the cluster, subscribe to cluster membership events (nodes joining or leaving), spawn a new process on any arbitrary remote node, and seamlessly send or receive messages across machine boundaries. The syntax for inter-process communication remains identical whether the target process lives on the local core or across the network, illustrating the BEAM philosophy that concurrency and distribution are two facets of the exact same model.
- A BEAM cluster is established by opening a TCP connection between nodes via simple API calls. - Process spawning (`spawn/4`), message sending (`send` or `!` operator), and message receiving work identically regardless of physical node location. - Nodes can monitor remote processes and listen for cluster topology changes using built-in system monitors.
---
[30:53](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=1853s) Live Demo: Swarm, Dynamic Service Placement, and Node Failover
To showcase distributed mechanics in practice, Jurić opens an administrative dashboard designed to manage named services across the cluster. In BEAM terminology, a "service" is simply a stateful concurrent process registered under a global, cluster-wide identifier. The interface initially shows only a single active node, so he launches a second node (`node2`) in the foreground from his terminal; the dashboard detects it immediately and displays both members.
Jurić registers a new service named `foo`. The cluster automatically chooses a host node, spawns the process, and registers its name globally. Clicking `foo` triggers a lookup: the web frontend locates the process by its global identifier anywhere in the cluster, sends it a message, receives a reply, and renders the payload. The output reveals that the process maintains state—retaining its own assigned name—and knows the identity of the specific node it is running on. When he registers a second service named `bar`, the system places it on `node2`, balancing the load across instances while providing an identical programming interface.
Next, Jurić simulates hardware failure by abruptly terminating `node2`. The cluster detects the loss, triggers automatic failover, and resurrects `bar` on `node1`. Clicking `bar` again shows that it now responds from `node1`, proving that application-level fault tolerance can withstand the sudden death of individual machines.
However, Jurić offers an honest assessment of standard distributed Erlang:
> “Full disclosure, complete honesty, and this all looks well and fine, but I would say that BEAM distribution is sort of plagued with a bunch of issues—serious issues.”
Because of these limitations—spanning from fundamental runtime networking assumptions down to the libraries built on top—many experienced practitioners advise against using native BEAM distribution in certain mission-critical environments. Nevertheless, Jurić emphasizes that these issues are purely mechanical implementation details rather than design flaws; they can be addressed and improved over time without breaking the high-level programmer-facing API contract.
---
[33:20](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2000s) Embedding Distributed Orchestration via Elixir Dependencies
Walking through the source code behind the distributed demo, Jurić highlights its architectural simplicity. The underlying worker module is implemented as standard, concurrent Elixir code that remains completely agnostic of the network layer. It contains plain functions and process loops, functioning identically whether running on a standalone single node or across an interconnected multi-node cluster.
Transforming that standalone local worker into a resilient, cluster-aware service required only three concrete steps. The first step was adding a single line to the dependencies block of the project’s `mix.exs` build configuration file: bringing in the `swarm` library. Swarm acts as a high-level distributed service orchestrator that coordinates cluster-wide process registration, placement, and automatic failover.
Jurić emphasizes a fundamental difference between BEAM libraries and dependencies in other programming ecosystems: in most languages, pulling in an external package merely imports passive code (modules and functions) waiting to be called. In the BEAM world, bringing in a dependency can also introduce running processes. When the application boots, Swarm automatically launches its own internal supervision tree and background processes. These active processes monitor node-up and node-down events, maintain the distributed registration table, and continuously manage the state mapping process identifiers across the network—all triggered by adding that single line in `mix.exs`. Because the orchestrator runs directly inside the BEAM runtime as an integrated component of the running application, developers avoid the need for separate external coordination daemons.
[35:13](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2113s) Clustering with Swarm and True Location Transparency
Saša Jurić demonstrates how easy it is to coordinate services across an entire cluster using an Elixir library called Swarm. Once Swarm is running inside the cluster, starting a globally named service boils down to invoking a single Swarm function. Swarm automatically selects a node, starts the process on that node, and tracks the association between the chosen name and the process identifier (`PID`). When a client process needs to find and invoke that service later, it calls another Swarm function with the service's name. Swarm returns a `PID`, which uniquely identifies the process across the entire cluster—not just on the local node.
Because `PID`s in Erlang and Elixir are location-transparent, they function identically whether pointing to a local process on the same machine or a remote process on another node in the cluster. Most BEAM-agnostic functions that take a `PID` do not care where the underlying process actually resides; you pass the `PID` to the function and it simply works. Jurić highlights that it took only three lines of code to turn the system into a distributed cluster. There are no sidecar processes running, no auxiliary operating system processes, no Docker containers, no virtual machines, and no external cloud services employed. The entire demo operates while completely disconnected from the network, and even the frontend user interface is driven straight from Elixir code without writing any JavaScript.
```mermaid flowchart LR accTitle: Distributed Service Discovery with Swarm accDescr: Swarm starts a named process on a chosen node, returns a location-transparent PID, and allows any node to communicate with it directly. subgraph Cluster["BEAM Cluster"] direction TB NodeA["Node A (Local)"] NodeB["Node B (Remote)"] SwarmReg["Swarm Registry"] ProcB["Service Process (PID)"] end
NodeA -->|"1. swarm:register_name / whereis_name"| SwarmReg SwarmReg -->|"2. Spawns / Resolves"| ProcB SwarmReg -->|"3. Returns cluster-wide PID"| NodeA NodeA -->|"4. Message passing via PID"| ProcB ```
- Starting, discovering, and communicating with a distributed service required just three lines of code. - A `PID` is completely location-transparent, seamlessly referencing either local or remote processes across nodes. - No third-party infrastructure (like service meshes or container orchestrators) was required; the entire stack ran self-contained and completely offline.
---
[36:30](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2190s) Technical Uniformity: System A vs. System B
Jurić shares a comparison table originally published in his book *Elixir in Action*, reflecting on two real-world systems he worked on side-by-side around the start of the decade. System A started with a foundational technology that lacked native primitives for concurrent, distributed, fault-tolerant software systems. When faced with inherent operational requirements—such as background jobs, caching, task queuing, or real-time communication—the engineering team had no choice but to fall back to the operating system level. They spun up multiple OS processes, brought in different programming languages where concurrency was needed, and layered on a patchwork of third-party products. While this cobbled-together stack technically satisfied their feature requirements, the resulting complexity barrier was massive. A junior developer joining the project faced an intimidating mountain of disconnected tools and paradigms before they could safely touch production code.
In contrast, System B was built on Erlang and the BEAM runtime. Because the BEAM effectively acts as an operating system specifically tailored for running code, the language itself serves as the interface to concurrency and fault isolation. The team fulfilled every operational requirement—distributed state, background processing, monitoring, and orchestration—directly inside Erlang, in a single project running within a single OS process per machine. While native Erlang implementations might lack every specialized bell and whistle found in dedicated third-party enterprise tools, they were more than sufficient for the application's actual needs. In exchange, the project achieved complete technical uniformity. As Jurić observes:
> “It is in fact sometimes said that BEAM is an operating system for our code, and if I have the operating system directly in my runtime and I have the language as the main interface, then I don't need to fall back to the OS level.”
| System Attribute | System A (Fragmented Architecture) | System B (Erlang / BEAM) | |:--- |:--- |:--- | | **Foundation** | Tech lacking native concurrency/distribution | Erlang on top of the BEAM runtime | | **Concurrency Strategy** | OS processes, multi-language bridging | Native BEAM lightweight processes | | **Auxiliary Needs (Cache, Queues)** | External third-party tools & daemons | Handled in-language within the project | | **Runtime Footprint** | Many disparate processes, containers, configs | Single OS process per machine | | **Complexity Barrier** | Extremely high onboarding learning curve | Low; single language and operational model |
---
[38:20](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2300s) The Compounding Value of Lowering the Technical Bar
Uniformity should never be pursued merely for its own sake, but Jurić argues that having an entire stack built in a single technology—one language, one project repository, and one OS process per physical machine—transforms every phase of the software engineering lifecycle. Setting up a local development environment becomes trivial, writing tests and configuring continuous integration (CI) pipelines is straightforward, and deployment shifts from coordinating five to ten fragile moving parts to deploying a single unified release. In production, operations engineers have a single target to hook into, monitor, log, and remotely debug.
Crucially, the greatest dividend of technical uniformity is how it empowers teams. When the technical bar is drastically lowered, developers do not need to fracture into rigid specialty silos. Organizations no longer need a dedicated Redis specialist, an NGINX specialist, a ZooKeeper specialist, a Kafka specialist, or a Kubernetes wrangler. Instead, onboarding engineers learn Elixir and can immediately build features across the entire stack. Teams are freed from wrestling with fragile configuration files:
> “And we don't need to wrestle with all these YAMLs and INIs and JSONs and TOMLs and XMLs and other kind of ad-hoc DSLs and their quite substandard tooling.”
Rather than relying on brittle markup formats and separate daemons, the entire system can be expressed in a single Turing-complete language that integrates cleanly with the host OS, offering full compile-time and runtime flexibility.
---
[39:58](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2398s) Unifying Fast Starts and Long-Term Scalability
Looking at the broader software industry, Jurić identifies what he considers the greatest, yet still unrealized, potential of BEAM languages. In typical development, engineering teams face a painful trade-off: choose a "quick start" framework that gets a minimum viable product out the door rapidly but must be completely rewritten once traffic scales, or choose an enterprise-grade distributed stack upfront that incurs massive operational overhead from day one.
BEAM dismantles this artificial dichotomy. Jurić paints an ideal development scenario: an engineer needs to build a small-to-medium web-facing CRUD application, initializes a new Elixir project, adds a couple of libraries, and within five minutes possesses native support for distribution, clustering, and replicated SQL-based persistence running directly inside the application. Because the application runs on a Turing-complete language backed by a runtime refined over three decades of battle-testing across massive telecommunications systems, developers can rest confident that they will not hit a wall. When complex architectural challenges and massive scaling pressures arrive down the road, the system can evolve without requiring a total ground-up rewrite:
> “The artificial trade-off of having to choose between a quick start technology and the long-run one disappears, flies out the window, and I get both in the same piece of technology.”
Jurić concludes that this rare harmony between initial simplicity and infinite operational headroom is enabled entirely by the BEAM, making it the true soul of Erlang and Elixir.
---
[41:20](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2480s) Concluding Remarks and Book Giveaway
Jurić wraps up his talk by offering a few parting gifts to the audience. He shares discount codes on screen for his book *Elixir in Action*, offering 50% off for the day of the conference and 40% off for subsequent days. He also announces that he brought along two physical hardcover copies and several digital eBook versions of *Elixir in Action* to give away to attendees. In a moment of dry humor, he invites anyone interested to come up and grab one directly:
> “And if you wanna grab that, just approach me after the talk and ask for it; you don't have to pretend that you're interested in anything else.”
He asks the audience to rate the session, provide feedback on what worked and what didn't, switches the slide back to the discount coupon codes for the room, and bids the audience thank you to closing applause.
## Notable quotes > [39:58](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2398s) The artificial trade-off of having to choose between a quick start technology and the long-run one disappears, flies out the window, and I get both in the same piece of technology. — Saša Jurić
> [36:30](https://www.youtube.com/watch?v=JvBT4XBdoUE&t=2190s) It is in fact sometimes said that BEAM is an operating system for our code, and if I have the operating system directly in my runtime and I have the language as the main interface, then I don't need to fall back to the OS level. — Saša Jurić
## Mentioned Joe Armstrong, Saša Jurić, Erlang, Elixir, BEAM, Swarm, Elixir in Action, Programming Phoenix 1.4, Chris McCord, Bruce Tate, José Valim, GOTO Chicago 2019, Docker, Kubernetes, Redis, NGINX, Apache ZooKeeper, Apache Kafka
## Video description This presentation was recorded at GOTO Chicago 2019. #gotocon #gotochgo http://gotochgo.com
Saša Jurić - Author of "Elixir in Action” @sasajuric5850
ABSTRACT Whether you're developing a small web site or a large-scale distributed system, Erlang and Elixir are great tools for building server-side systems. Both languages can help you start quickly and deal with highly complex problems at a large scale. This talk aims to explain what makes Erlang and Elixir suitable for such tasks. We'll look past the syntax and the ecosystem, focusing instead on the concurrency model of these languages. Combining a bit of high-level theory and a couple of demos, we'll examine some important properties of Erlang/Elixir concurrency, with a special emphasis on why these properties are very relevant when building highly available systems. The talk targets experienced backend developers, but no knowledge of Erlang or Elixir is required. You will walk away from [...]
Download slides and read the full abstract here: https://gotochgo.com/2019/sessions/712
RECOMMENDED BOOKS Saša Jurić • Elixir in Action • https://amzn.to/2RZh5eN McCord, Tate & Valim • Programming Phoenix 1.4 • https://amzn.to/3zcUqj4
https://twitter.com/GOTOcon https://www.linkedin.com/company/goto- https://www.instagram.com/goto_con https://www.facebook.com/GOTOConferences #erlang #elixir #programming #DistributedSystems #FunctionalProgramming #SasaJuric
CHANNEL MEMBERSHIP BONUS Join this channel to get early access to videos & other perks: https://www.youtube.com/channel/UCs_tLP3AiwYKwdUHpltJPuA/join
Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech Sign up for updates and specials at https://gotopia.tech/newsletter
SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1
[Source](https://www.youtube.com/watch?v=JvBT4XBdoUE)