HRT Highlights 2025 Software Engineering Intern Infrastructure and Trading Projects

HRT Highlights 2025 Software Engineering Intern Infrastructure and Trading Projects Hudson River Trading featured three summer intern projects tackling core trading and infrastructure challenges: tabular pub-sub compression, Parquet read optimization via Z-ordering, and out-of-process latency stall monitoring. SmallGrid added compressed columnar formats and variable-length packing with null bitmasks, reducing CPU overhead by 57% and total message count by 34%. A custom Z-ordering implementation for HRT's internal Apache Iceberg ingestion pipeline improved average query times on Parquet files by 20% to 25%. An out-of-process stall monitor tracks trading process event timestamps via shared memory and non-temporal stores, adding roughly 15 microseconds of overhead to copy stack traces during 100-microsecond stalls. Optimizing Compression in SmallGrid Tabular Pub-Sub HRT's SmallGrid pub-sub system transports large volumes of tabular data using the DenseColumnarTable format. To balance bandwidth reduction with CPU latency, the system added support for LZ4, LZ4Small, and Deflate compression, selecting LZ4 primarily because skipping entropy encoding produces consistent, predictable throughput. While variable-sized columns serialized cell-wise initially consumed 68% more CPU due to redundant update-key metadata, transitioning to column-wise packing using minimal length tags and explicit null bitmasks resolved this bottleneck. The column-packing layout eliminated per-cell overhead, produced smaller uncompressed payloads than earlier compressed versions, decreased CPU overhead by 57%, and cut total message counts by 34%. Balancing Multi-Column Read Performance with Z-Ordering HRT stores petabyte-scale datasets in Apache Iceberg tables backed by Parquet files, querying them with engines like Trino and DuckDB that use row group min/max statistics to prune reads. While traditional multi-column sorting optimizes lookups for primary sort columns, it scatters data across lower-priority columns. Because HRT utilizes an in-house write and ingestion pipeline, an intern implemented custom Parquet rewriting using Z-ordering, an algorithm that interleaves lexicographically sortable bits to preserve multi-dimensional data locality. When applied to commonly filtered columns like date, asset class, team, and symbol, Z-ordering yielded a 20% to 25% improvement in average query execution time. Out-of-Process Stall Detection for Trading Systems Event-driven automated trading systems can suffer latency spikes when unexpected blocking tasks, such as memory allocations or configuration reads, occur on the critical event path. To identify root causes without relying on coarse post-event logs, a stall monitoring server was developed to run independently on each host. Client applications use shared memory and cache-bypassing non-temporal stores to write and clear timestamps around event handlers. If the monitor detects an expired timestamp, it attaches to the stalled process to collect diagnostics, adding only around 15 microseconds of overhead to unwind stack traces during a 100-microsecond stall.

HRT盘点2025届软件工程实习生三大核心基建项目 Hudson River Trading（HRT）回顾了2025届软件工程实习生的三项代表性项目。这些项目涉及表格发布订阅传输优化、PB级Iceberg数据集的Z-ordering排序优化，以及低延迟交易系统的共享内存停顿监控机制，三位实习生均已转为全职员工。 SmallGrid表格发布订阅系统通过按列打包变长数据与结合LZ4压缩，降低了57%的CPU开销并减少了34%的消息数量。 针对内部自研Iceberg/Parquet写入路径实现Z-ordering算法，多过滤条件下使平均查询时间缩短20%至25%。 基于独立进程与共享内存构建轻量级Stall Monitor，在交易进程发生停顿时代价仅增加约15微秒即可捕获栈追踪等排查信息。 SmallGrid表格压缩与列式打包优化 SmallGrid是HRT用于进程通信和系统自省的表格发布订阅系统，大多表采用DenseColumnarTable格式。为在带宽占用与CPU开销之间取得平衡，项目引入了增量打包与批量解压机制，并支持LZ4与Deflate压缩。针对按单元格序列化变长列带来冗余元数据和高CPU开销的问题，团队实现了按列打包格式，使用最小长度标签标识边界并配合空值位图。优化后，未压缩的列式更新体积甚至小于压缩后的单元格更新，CPU开销下降57%，消息总量减少34%，且LZ4凭借一致的处理吞吐获得了接近DEFLATE的压缩率。 PB级Iceberg与Parquet数据的Z-ordering多列优化 HRT使用Apache Iceberg管理PB级数据集，底层基于Parquet格式并支持Trino与DuckDB查询引擎。传统按单一或多列首位排序的方式仅能加速高优先级列过滤，导致次要列过滤仍需扫描大量Row Group。由于HRT采用自研写入管道，实习生自行实现了Parquet文件的Z-ordering重写算法，通过将多列值转换为字典序格式并交织各bit位，在多维空间中保持局部性。在针对日期、资产类别、团队与代码等多字段查询中，该算法使平均查询耗时降低了20%至25%。 基于共享内存的交易系统停顿监控器（Stall Monitor） 在事件驱动的交易系统中，DNS解析、大内存分配或NFS配置读取等突发昂贵操作常引发停顿（stall）。为解决事后报警缺乏诊断上下文的问题，项目设计了独立的停顿监控守护进程。交易进程通过轻量客户端在共享内存中记录事件处理前后的时间戳，并使用非临时写入（non-temporal store）绕过缓存。一旦监控服务发现超时时间戳，便会挂起目标进程并提取栈信息。基准测试显示，在生产机上发生100微秒停顿时，抓取栈帧仅引入约15微秒额外延迟，以极小影响实现了对生产延迟峰值的精准追溯。

As another internship season comes to a close, it’s time for our annual look under the hood at what our Software Engineering interns were up to last summer. Every year, our interns take on serious, independent projects, building systems and solutions that tackle some of our hardest trading and infrastructure problems. For this spotlight series, we’re diving into three standout projects from our 2025 Software Engineering class, all of whom recently joined HRT full-time. And if you want to join us for the next cohort, [you can apply for our upcoming summer internships here](https://www.hudsonrivertrading.com/student-opportunities/). SmallGrid is a tabular pub-sub system at HRT, which started out as a summer internship project in 2021 \[[read more here](https://www.hudsonrivertrading.com/hrtbeat/an-intro-to-smallgrid/)\]! It’s widely used to communicate between processes, populate GUIs, and introspect components of our trading infrastructure. Most tables today are stored as our DenseColumnarTable type, which stores columns in Arrow buffers for improved performance when most entries are non-null. Because of the large volume of data transmitted via SmallGrid each day, it’s important to support compression in order to reduce bandwidth usage. However, this comes at the cost of additional CPU overhead, which oftentimes isn’t tolerable for teams with strict latency requirements. The focus of my project is to support compression for DenseColumnarTable while optimizing this trade-off between bandwidth and CPU time. We added compressed versions of our existing DenseColumnarTable message formats, which already cover full-state and delta updates. On the publisher’s side, we pack buffers incrementally and flush when necessary to emit compressed messages. On the subscriber’s side, partial buffers are accumulated until a full batch is available to decompress altogether. After benchmarking on our common streaming data distributions, we chose to support an array of compression algorithms: LZ4, “LZ4Small” (different acceleration level), and Deflate. These provide a range of compression ratios (30-70%) with corresponding CPU time overheads (5-250% increases). We chose LZ4 in addition to the standard ZLib API specifically because LZ4 skips the traditional entropy encoding stage (such as Huffman coding). In entropy-heavy algorithms, variable-bit outputs and complex frequency tables make performance fluctuate wildly depending on how structured or random the input data is. By omitting this stage, LZ4 maintains consistent throughput and predictable execution times regardless of data distribution, which is particularly useful in lightweight, real-time compression settings. At this point though, we observed variable-sized columns (serialized cell-wise) produced larger baselines for both uncompressed and compressed sizes compared to fixed-size columns (serialized column-wise), consuming 68% more CPU time on average. We hypothesized that this was due to compression algorithms working intensely on cell-update keys rather than table values themselves. Encoding (row, column, size) tuples creates redundant key patterns that unnecessarily increase LZ4 and DEFLATE processing overhead through extra LZ77 matches, back-references, and token parsing. And so to better support variable-size columns, we extended column-wise packing with a new pseudo-message format that uses minimal length tags to mark boundaries between variable-length elements. We purposefully reused ChunkUpdateHeader across both fixed and variable column types so they can share underlying memory buffers. Additionally, instead of forcing consumers to check for and parse length-zero tags for missing data, we encode null locations using explicit null bitmasks. This keeps zero-width values out of the payload stream and simplifies consumer-side decoding. Ultimately, uncompressed variable-size column packing resulted in smaller overall payloads than even compressed cell-wise updates, as grouping values by column eliminated per-cell metadata overhead. Column packing also yielded significantly higher compression ratios. Interestingly, LZ4 performed nearly as well as DEFLATE; because columnar alignment clusters homogeneous data together, repeated values (such as ticker symbols) and uniform string lengths generated long, repeating sequences of VarUpdateHeaderTags that LZ4’s simple dictionary matches easily captured. Finally, CPU overhead decreased by 57% because column-wise parsing is computationally simpler than general-purpose decompression, and total message count fell by 34% because buffer sharing aggregated multiple column updates into fewer total payloads. At HRT, we use [Apache Iceberg](https://iceberg.apache.org/) as our table format of choice to organize large, petabyte-scale datasets. These tables are used in many domains, from monitoring the health of our research infrastructure to post-trade analysis. We like Iceberg because it has little vendor lock in, allowing us to substitute different query engines and customize the ingestion path. Our entire write path, in particular, is built internally to better leverage our HRT-specific distributed compute infrastructure. This means we have in-house tools to optimize datasets. Optimizations may include resorting, resharding, or adjusting row group sizes of Parquet files. At HRT, we use [Parquet](https://parquet.apache.org/) as our data storage format of choice for tabular datasets. We like Parquet files for the same reasons as Iceberg: it has little vendor lock-in, high performance, and designed for storing large, complex datasets. Furthermore, Parquet is the de-facto standard for Iceberg tables. For the scope of this post, a simple way to view a Parquet file is a list of *row groups*, with each group containing a batch of data: The footer of a Parquet file contains a variety of statistics about each of these row groups; most notably, it stores the minimum and maximum of each column. These statistics are invaluable for query engines, as they can make inferences about whether a row group can be skipped, saving compute. To query this data at HRT, we support two query engines: [Trino](https://trino.io/) and [DuckDB](https://duckdb.org/). As alluded to above, these query engines are capable of inspecting column statistics within Parquet row groups and Iceberg’s manifest files, enabling them to prune unnecessary reads. It’s common for tables to contain data pertaining to a large range of asset classes and symbols. This means that researchers pass aggressive filters to request just a subset of the table that they care about. Two trivial queries might look like this, which we will revisit throughout this blog post: ``` > SELECT * FROM table WHERE symbol = ‘AAPL’; > SELECT * FROM table WHERE team = ‘Team 1’; ``` If the data is not optimally ordered, then the rows where `symbol = ‘AAPL’` may be in numerous row groups. This means the query engines must check all those row groups. One simple optimization involves sorting the Parquet file by `(symbol, team).` This significantly reduces the number of row groups the query engines need to inspect. Although the optimization above performs strongly on symbol filters, it falls extremely short on team filters. Despite the multi-column sort, the layout above would be ill-suited for a WHERE team = ‘Team 1’ query, as lower-priority sort columns are still scattered throughout the parquet files: Thus, I needed to build an algorithm that could *balance query performance across many columns*. We had the idea to use [Z-ordering](https://en.wikipedia.org/wiki/Z-order_curve), which would have more evenly distributed “sortedness” across multiple columns. Z-ordering is a known algorithm that some table engines provide out-of-box, such as [Dremio](https://www.dremio.com/blog/how-z-ordering-in-apache-iceberg-helps-improve-performance/). However, since HRT has a bespoke data ingestion pipeline for Iceberg tables, I had to write my own implementation that rewrites a parquet file to be Z-ordered. To calculate the Z-order of a row, you take the values, convert them into a lexicographically sortable format, and then interleave the bits of each value. The result is a single sortable value that approximately preserves locality across all of the columns, so rows with similar combinations of values tend to end up near one another. In fact, the name becomes much clearer when one visualizes a Z-ordering across two continuous integer columns: Going back to the more concrete example, implementing a Z-ordering algorithm provides read performance improvements for both symbol and team filtering: Although we walked through a toy example, Z-ordering really shines as one scales up the size of the datasets. At HRT, we regularly build petabyte-scale Iceberg datasets, where read performance can greatly increase productivity and the user experience for our researchers. A single query on a small subset of (date, asset\_class, team, symbol) columns of some of our tables might require opening hundreds of gigabytes of parquet. Using Z-ordering on the most commonly filtered columns, we saw a 20-25% improvement in *average* query time. A typical automated trading system is driven by market events, such as an order placed on an exchange or a trade occurring. These event-driven systems encounter production issues due to “stalls,” where the program takes too much time to process an event and can’t perform its other duties. To detect these stalls, we determine how long it takes to handle each event: ``` auto start = Time::now(); handleEvent(); auto duration = Time::now() - start; if (duration > WarnThreshold) { log.warn("handling event took %s", duration); } ``` Stalls typically happen when an unexpected expensive operation takes place on the critical path to process an event. There are many possible such operations: DNS resolutions, large heap allocations, or even reading config files from NFS. But the current approach offers little visibility into what actually caused a particular stall. We’d like to: Our solution is a stall monitor server that runs on each host. Latency-sensitive processes on that host can register with the stall monitor to be given a slot in shared memory. They write the current time to their slot before handling each event and clear it after they finish. The server repeatedly checks the slots for stale timestamps, which indicate a stall. This offloads the work of detecting stalls to the server, which runs in its own process. Once a stall is detected, the server can attach to the stalled process to collect information about the stall. We provide a lightweight Client class that each client process instantiates to register with the stall monitor. Apart from a one-time handshake with the server during initialization, the Client class does only two things: The non-temporal stores bypass the cache to avoid polluting it. In benchmarks, the slowdown from these two operations was negligible relative to typical event processing latency. When the server detects a stall, it can: To verify the reliability of the stall monitor, we can set up parallel test clients that stall for randomized intervals and durations, and observe the server’s output. To profile the overhead of the server stopping the client process to collect information about the stall, we can measure the duration of the stall in the client with and without the stall monitor. For 100 microsecond stalls on a production trading machine, we observe the stall is \~15 microseconds longer when the server copies the client’s stack to unwind into a stack trace. By offloading stall detection to a separate process and using efficient communication via shared memory, we can now collect detailed traces of latency spikes with minimal impact on latency during production trading.