The Edge LLM Memory Tax: ExecuTorch vs ONNX Runtime QNN EP on Snapdragon X Elite

By Rizowan Ahmed (@riz1raj)
Senior Technology Analyst | Covering Enterprise IT, Hardware & Emerging Trends

The marketing departments of silicon vendors have spent the last few years conditioning us to worship a single, highly misleading metric: NPU TOPS. We are told that 45 TOPS on the Qualcomm Snapdragon X Elite (X1E-84-100) is the golden ticket to running local Large Language Models (LLMs) at lightning speed. But in the engineering trenches of edge AI, we know the brutal truth: compute is cheap; memory is the bottleneck. If your runtime environment consumes half of your available LPDDR5X-8448 system memory just to initialize its graph and allocate its execution context, your highly optimized INT4 quantized LLM is going to choke, page to disk, or trigger the Out-Of-Memory (OOM) killer before it generates its first token.

When deploying local LLMs like Llama-3-8B or Mistral-7B on the Oryon CPU and Hexagon NPU architecture, developers are faced with a critical architectural decision. Do you go with Meta’s lean, PyTorch-native ExecuTorch runtime, or do you rely on Microsoft’s enterprise-grade, highly mature ONNX Runtime (ORT) with the Qualcomm Neural Network (QNN) Execution Provider (EP)? This article provides a deep-dive, low-level technical comparison of the memory overhead profiles of these two runtimes, analyzing how they manage memory allocation, graph compilation, and KV caching on the Snapdragon X Elite platform.

Architectural Paradigms: How ExecuTorch and ORT Interface with the Hexagon NPU

To understand why their memory footprints differ so drastically, we must first look at how both frameworks interface with the underlying Qualcomm QNN SDK and the Hexagon v73 HTP (Hexagon Tensor Processor).

ExecuTorch: The Ahead-of-Time (AOT) Minimalist

ExecuTorch was designed from the ground up to strip away the heavy runtime overhead of standard PyTorch. It operates on a strict Ahead-of-Time (AOT) compilation philosophy. The compilation pipeline looks like this:

• PyTorch Export: The model is traced using torch.export into a stable, strongly-typed graph representation.
• Quantization and Lowering: The graph is quantized (typically to INT4 or INT8 weights with INT16 or FP16 activations) and lowered to the ExecuTorch dialect.
• QNN Backend Delegation: The ExecuTorch QNN compiler delegate maps the operators directly to Qualcomm QNN APIs, producing a serialized .pte flatbuffer binary containing the compiled QNN context.

At runtime, the ExecuTorch engine acts as a lightweight, zero-allocation-by-default runner. It does not perform graph optimization, shape inference, or memory planning at runtime. Everything is pre-calculated during the offline compilation phase.

ONNX Runtime with QNN EP: The Just-in-Time (JIT) Optimizer

ONNX Runtime takes a fundamentally different approach. While it supports pre-compiled QNN Context Binaries, it is architecturally built as a dynamic, highly flexible execution engine. The standard pipeline involves exporting a PyTorch model to an .onnx file, loading it into ORT, and letting the QNN Execution Provider compile the graph during the session initialization phase (or loading a pre-compiled QNN context block embedded in the ONNX model).

Because ORT is designed to support heterogeneous execution (falling back to CPU or GPU if the NPU lacks operator support), it maintains a heavy internal infrastructure for tensor allocation, shape engine tracking, and fallback execution kernels. This flexibility comes at a severe cost in memory overhead.

ExecuTorch vs ONNX Runtime QNN EP memory overhead comparison on Snapdragon X Elite

When evaluating the trade-offs of ExecuTorch vs ONNX Runtime QNN Execution Provider: Compiling Local LLMs for Qualcomm Hexagon NPU Acceleration, developers must look past raw execution speed and analyze the silent killer of edge AI: memory fragmentation. Let us break down the memory overhead into three distinct vectors: Static Runtime Footprint, Graph Compilation/Loading Memory, and Dynamic Activation/KV Cache Management.

1. Static Runtime Footprint (The 'Tax' of Existing)

The static runtime footprint refers to the memory consumed by the runtime engine libraries, metadata, and basic allocator structures before any model weights or activation buffers are loaded into RAM.

• ExecuTorch: The core C++ ExecuTorch runtime library is incredibly compact, often compiling down to a highly compact binary size. It uses no global heap allocators and does not initialize complex internal telemetry, logging, or optimization subsystems. Its static memory overhead is virtually negligible, requiring minimal heap memory.
• ONNX Runtime + QNN EP: ORT is a massive C++ codebase. Even when compiled with minimal options, the ORT shared libraries, coupled with the QNN EP wrapper and the core QNN SDK runtimes (libQnnSystem.so, libQnnHtp.so), consume substantial disk and memory space. The static memory overhead of initializing an ORT session with QNN EP enabled is typically much higher due to these additional components. This is because ORT initializes its own internal thread pools, memory arenas (like the Arena Allocator), and extensive metadata structures to track graph nodes.

2. Graph Compilation and Loading Overhead

This is where the difference between AOT and JIT paradigms becomes painfully obvious. When loading an LLM, the runtime must load the weights and prepare the execution graph for the Hexagon NPU.

On the Snapdragon X Elite, the Hexagon NPU cannot execute raw ONNX or PyTorch operators directly; it requires a compiled QNN HTP graph. This graph can either be compiled on-device (JIT) or pre-compiled offline (AOT).

If you choose on-device JIT compilation with ONNX Runtime QNN EP, the memory overhead spikes significantly. Compiling an INT4 quantized Llama-3-8B model on-device can consume a significant amount of peak transient RAM during the compilation phase. This is because the QNN compiler backend must build the execution topology, optimize weights, and allocate massive intermediate buffers in system memory. On a 16 GB Snapdragon X Elite laptop, this spike can push the system into swap space, causing system stutter.

Even when using pre-compiled QNN Context Binaries, ONNX Runtime still incurs a loading penalty. It must parse the ONNX wrapper, map the inputs and outputs, and instantiate the QNN EP session. ExecuTorch, by contrast, bypasses this entirely. The .pte file maps directly to the pre-compiled QNN binary. ExecuTorch uses memory-mapped files (mmap) to load the model weights and the QNN context directly into physical memory pages, eliminating duplicate copies and keeping the loading memory spike to near zero.

3. Dynamic Activation and Memory Arena Planning

During inference, the runtime must allocate memory for intermediate activations (the outputs of each neural network layer) and the Key-Value (KV) cache. How each framework manages this memory dictates whether your application can run concurrently with other system processes.

ExecuTorch leverages a highly deterministic Static Memory Planning compiler pass. During the offline compilation phase, ExecuTorch analyzes the lifetime of every single tensor in the graph. It then calculates a non-overlapping memory layout, packing all intermediate activations into a single, contiguous memory buffer called the 'Memory Arena.' At runtime, ExecuTorch allocates this single buffer once. There are zero dynamic malloc or free calls during the execution loop. This completely eliminates runtime memory fragmentation and ensures that the peak memory usage is known down to the exact byte before the application even runs.

ONNX Runtime QNN EP relies on a dynamic memory allocator. While the QNN SDK itself manages the internal HTP hardware buffers, ORT must still manage the boundary tensors (the inputs and outputs of the QNN subgraph). ORT uses its OrtAllocator to dynamically allocate and free buffers as tensors flow through the graph. On the unified memory architecture (UMA) of the Snapdragon X Elite, where the CPU and NPU share the same physical LPDDR5X channels, dynamic allocation causes memory fragmentation. Over long chat sessions, this fragmentation can cause the virtual memory footprint of the ORT process to steadily drift upward, a phenomenon often mistaken for a memory leak but which is actually just allocator overhead.

The KV Cache Conundrum: Static vs. Dynamic Allocation

For LLMs, the KV cache is the single largest consumer of dynamic memory. For a Llama-3-8B model with a 2048-token context window, the KV cache can consume hundreds of megabytes of RAM. Managing this cache efficiently is paramount.

In ONNX Runtime QNN EP, the KV cache is typically handled by passing 'past' and 'present' state tensors back and forth between the model and the runtime. If not configured with extreme care using IO Binding, ORT will copy these KV cache tensors across the CPU-NPU boundary on every single token generation step. This not only impacts performance but also increases the memory overhead of the KV cache, as duplicate copies exist in CPU-accessible RAM and NPU-secure memory domains.

ExecuTorch handles the KV cache by treating it as a static, pre-allocated tensor subclass that resides directly within the QNN device memory. By utilizing PyTorch’s native tensor subclassing and ExecuTorch’s custom operator registry, developers can pin the KV cache in the Hexagon NPU’s local memory space. This bypasses the host-side memory allocation overhead and ensures zero-copy execution throughout the entire autoregressive generation loop.

Summary Comparison Matrix

To synthesize these architectural differences, let us look at how they compare across key memory metrics when running an INT4 quantized 8B parameter LLM on the Snapdragon X Elite (X1E-84-100) under Windows on ARM or Linux:

Memory Metric	ExecuTorch (QNN Delegate)	ONNX Runtime (QNN EP)
Static Engine Footprint	Extremely Low	Moderate to High
Peak Loading Memory Spike	Near Zero (uses mmap directly to QNN binary)	High (due to graph parsing and session initialization)
On-Device JIT Compilation Memory	Not Supported (AOT only, preventing runtime spikes)	Extremely High (transient spike if compiling on-device)
Memory Allocation Strategy	Static Memory Planning (zero dynamic runtime allocations)	Dynamic Arena Allocator (susceptible to fragmentation)
KV Cache Handling	Pinned, zero-copy static tensors within NPU domain	Dynamic boundary tensors (requires complex IO Binding to avoid copying)
Memory Fragmentation Risk	Zero (fully deterministic)	Moderate (increases over long context windows)

The Verdict and Future Outlook

The choice between ExecuTorch and ONNX Runtime QNN EP on the Snapdragon X Elite is not a matter of which framework is 'better,' but rather a strict engineering trade-off between flexibility and predictability.

If you are building an enterprise application that must run across a heterogeneous fleet of devices (some with Intel Core Ultra NPUs, some with AMD Ryzen AI, and some with Qualcomm Snapdragon X Elite), ONNX Runtime QNN EP remains the pragmatic choice. Its unified API and robust fallback mechanics shield developers from the fragmentation of the edge hardware landscape. However, you must pay the 'ORT Tax' in the form of higher static memory overhead, dynamic allocation fragmentation, and complex IO Binding configurations to keep the KV cache from impacting performance.

If, however, you are targeting the Snapdragon X Elite specifically and demand absolute maximum efficiency—such as embedding a local LLM into a background system service or a mobile-style application where every megabyte of RAM counts—ExecuTorch is the clear victor. Its static memory planning, negligible runtime footprint, and native PyTorch compilation path represent the gold standard for deterministic edge AI deployment.

Looking ahead, as local LLMs scale down to highly efficient smaller parameter architectures designed to run constantly in the background of Copilot+ PCs, the runtime memory overhead becomes even more critical. A larger runtime overhead might be negligible for a cloud server, but for an edge device running a smaller model, that overhead represents a significant percentage of the model’s total footprint. The industry is increasingly looking toward AOT-compiled, zero-allocation runtimes like ExecuTorch to make ambient, always-on local AI a practical reality on consumer hardware.

---