Application Sandboxing

Introduction

Application sandboxing provides a tightly controlled environment in which software can execute with restricted access to system resources. Unlike traditional isolation techniques that focus primarily on separating groups of processes, sandboxing enables fine-grained control over what an individual application can do, including non-privileged actions like reading specific files or accessing the network.

This approach takes containment to another level. While mechanisms like containers, jails, and the underlying technologies—such as namespaces, control groups, and capabilities—are effective at isolating and limiting where a process runs and what it can see, they do not provide adequate controls over what the process can do, especially when it comes to ordinary system calls made by non-root processes.

Sandboxing complements other containment mechanisms by enabling behavioral restrictions at the system call level. It forms a critical layer of defense, especially for running untrusted code, performing dynamic malware analysis, or enforcing least-privilege principles in complex applications.

Sandboxing is particularly important for protecting users from their own applications. It allows users to run untrusted or potentially risky programs while enforcing policies that restrict those programs to specific behaviors. For example, a sandbox might allow an application to read only .txt files within the user’s home directory, but not write to them—or any other file. It might allow only TCP networking, blocking access to raw sockets or other protocols.

Traditional tools like chroot or filesystem namespaces can restrict access to parts of the file system, but they do so by changing the process’s view of the entire directory structure—requiring elaborate setup to ensure the application can still access libraries and shared resources. These tools also lack the ability to enforce fine-grained policies like “allow read-only access to files ending in .txt.”

Similarly, namespaces cannot control which files are opened or with what permissions, nor can they restrict access to specific types of network traffic. Linux capabilities allow us to limit privileged operations (e.g., mounting filesystems or changing user IDs), but they do not address the more common and potentially risky behaviors of regular applications, such as reading sensitive files or opening network connections.

What’s missing is the ability to define rule-based policies that restrict which system calls an application may invoke—down to the specific parameters passed to those calls. Sandboxing frameworks like seccomp-bpf, AppArmor, and SELinux provide this level of control by allowing policies that specify exactly what an application is allowed to do, regardless of whether it runs with elevated privileges.

Configuring sandboxes

Instead of building a jail (a container), we will add an extra layer of access control. An application will have the same view of the operating system as any other application but will be restricted in what it can do.

Applications interact with their environment by making system calls to the operating system. Any operation that an application needs to do aside from computation must be done through system calls: accessing files or devices, changing permissions, accessing the network, talking with other processes, etc.

An application sandbox will allow us to create policies that define which system calls are permissible to the application and in what way they can be used.

Sandboxing is currently supported on a wide variety of platforms at either the kernel or application level. We will examine four techniques of designing application sandboxes:

1. User-level validation
2. OS support
3. The Java sandbox
4. Browser-based application sandboxing

Note that there are many other sandbox implementations. This is just a sampling of how they can be built at different layers.

1. Application sandboxing via system call interposition & user-level validation

If the operating system does not provide us with sandboxing support and we do not have the ability to recompile an application to force it to use alternate system call libraries that can force the use of rule-based filters, we can use system call interposition to construct a sandbox. System call interposition is the process of intercepting an app’s system calls and performing additional operations. The technique is also called hooking. In the case of a sandbox, it will intercept a system call, inspect its parameters, and decide whether to allow the system call to take place or return an error. A hook is simply a mechanism that redirects an API request somewhere else and allows it to return for normal processing. For example, a function can be hooked to simply log the fact that it has been called.

Example: Janus

One example of validating at the user level is the Janus sandboxing system, developed at UC Berkeley, originally for SunOS but later ported to Linux. Janus uses a loadable, lightweight kernel module called mod_janus. The module initializes itself by setting up hooks to redirect system call requests to itself. The Janus kernel module copies the system call table to redirect the vector of calls to the mod_janus.

A user-configured policy file defines the allowable files and network operations for each sandboxed application. Users run applications through a Janus launcher/monitor program, which places the application in the sandbox. The monitor parses the policy file and spawns a child process for the user-specified program. The child process executes the actual application. The parent Janus process serves as the monitor, running a policy engine that receives system call notifications and decides whether to allow or disallow the system call.

Whenever a sandboxed application makes a system call, the call is redirected by the hook in the kernel to the Janus kernel module. The module blocks the thread (it is still waiting for the return from the system call) and signals the user-level Janus process that a system call has been requested. The user-level Janus process' policy engine then requests all the necessary information about the call (calling process, type of system call, parameters). The policy engine makes a policy decision to determine whether, based on the policy, the process should be permitted to make the system call. If so, the system call is directed back to the operating system. If not, an error code is returned to the application.

Challenges of user-level validation

The biggest challenge with implementing Janus is that the user-level monitor must mirror the state of the operating system. If the child process forks a new process, the Janus monitor must also fork. It needs to keep track of not just network operations but the proper sequencing of the steps in the protocol to ensure that no improper actions are attempted on the network. This is a sequence of socket, bind, connect, read/write, and shutdown system calls. If one fails, chances are that the others should not be allowed to take place. However, the Janus monitor does not have the knowledge of whether a particular system call succeeded or not; approved calls are simply forwarded from the module to the kernel for processing. Failure to handle this correctly may enable attack vectors such as trying to send data on an unconnected socket.

The same applies to file operations. If a file failed to open, then read and write operations should not be allowed. Keeping track of state also gets tricky if file descriptors are duplicated (e.g., via the dup2 system call); it is not clear whether any requested file descriptor is a valid one or not.

Pathname parsing of file names has to be handled entirely by the monitor. We earlier examined the complexities of processing "../" sequences in pathnames. Janus has to do this to validate any policies on permissible file names or directories. It also has to keep track of relative filenames since the application may change the current directory at any time via the chdir system call. This means Janus needs to intercept chdir requests and process new pathnames within the proper context. Moreover, the application may change its entire namespace if the process calls chroot.

File descriptors can cause additional problems. A process can pass an open file descriptor to another process via UNIX domain sockets, which can then use that file descriptor (via a sendfd and recv_fd set of calls). Janus would be hard-pressed to know that this happened since that would require understanding the intent of the underlying sendmsg system calls and cmsg directives.

In addition to these difficulties, user-level validation suffers from possible TOCTTOU (time-of-check-to-time-of-use) race conditions. The environment present when Janus validates a request may change by the time the request is processed.

2. Application sandboxing with integrated OS support

The better alternative to having a user-level process decide on whether to permit system calls is to incorporate policy validation in the kernel. Some operating systems provide kernel support for sandboxing. These include the Android Application Sandbox, the iOS App Sandbox, the macOS sandbox, and AppArmor on Linux. Microsoft introduced the Windows Sandbox in December 2018, but this functions far more like a container than a traditional application sandbox, giving the process an isolated execution environment.

Seccomp-BPF kernel sandboxing

Seccomp-BPF (short for Secure Computing with Berkeley Packet Filters) is a Linux kernel feature that allows processes to install filters controlling which system calls they are allowed to make. It provides a powerful mechanism for restricting application behavior at the system call level, making it an essential tool for building secure sandboxes.

Once a process installs a seccomp-BPF filter, the kernel enforces that filter for the process and all of its descendants. The filter can define exactly which system calls are allowed or denied, and can even examine the arguments to those calls to make more nuanced decisions. For example, a filter could permit open() calls only when used with read-only flags or restrict socket() calls to TCP only.

A key limitation of seccomp-BPF is that it cannot dereference pointers in system call arguments. This means it cannot inspect memory contents or compare strings (e.g., file paths). It can only examine directly accessible values like integers, flags, and system call numbers.

Seccomp has been a core part of Android’s security architecture since Android 8.0 (Oreo), released in August 2017, and is widely used in container runtimes (e.g., Docker) and security-focused applications (e.g., web browsers like Chrome and Firefox).

How It Works

Seccomp-BPF uses the Berkeley Packet Filter (BPF) framework, which was originally designed for network packet filtering. In its original form, BPF filters were used to allow or block network packets based on their contents.

In the context of seccomp, the kernel treats system calls as “packets” that can be inspected by the BPF engine. The developer writes a filter—a small BPF program—that is executed every time the process makes a system call. The filter can examine the system call number and arguments, then decide what to do based on user-defined rules.

Possible actions include:

• Allow the system call to proceed
• Deny the system call and return a specific error code
• Send a signal to the process (usually SIGSYS)
• Terminate the process immediately

These actions enable precise enforcement of security policies.

Limitations and Challenges

Seccomp-BPF is not a complete sandbox on its own. It only controls system call behavior; it does not isolate file systems, users, or network interfaces. Therefore, it is typically used in conjunction with other Linux isolation mechanisms—such as namespaces, capabilities, and control groups—to build comprehensive sandbox environments.

Another challenge is complexity. BPF is a low-level, stack-based virtual machine that supports arithmetic operations, conditional branches, scratch memory, and register manipulation. Policies are compiled into BPF bytecode and loaded into the kernel, where they are interpreted at runtime.

Because BPF programs reference system calls by numeric ID, filters must account for architecture-specific differences in system call numbering. For example, the system call number for open() differs between x86_64 and ARM.

The real difficulty is crafting a policy that enforces the principle of least privilege: allowing only the system calls that are strictly necessary for the application to function correctly. This includes not only the application’s main functionality but also supporting actions like error reporting, logging, and dynamic library loading—which can easily be overlooked during policy development.

Seccomp-BPF is a foundational building block for application sandboxing on Linux. While powerful, its effective use requires careful analysis of application behavior and low-level system knowledge. When used properly, it can significantly reduce the attack surface of untrusted or exposed processes.

AppArmor kernel sandboxing

AppArmor (Application Armor) is a Linux Security Module (LSM) that provides mandatory access control (MAC) through path-based policies. Its purpose is to confine programs by defining what files and capabilities they are allowed to access, regardless of the user’s identity or traditional Unix permissions.

AppArmor is designed around human-readable policy profiles that specify the precise actions an application can perform. These profiles are based on filesystem paths and can control:

• File read/write/execute access
• Network usage
• Use of POSIX capabilities
• Execution of other programs
• Access to specific kernel interfaces (e.g., ptrace, /proc)

When a program runs under an AppArmor profile, the kernel enforces the defined restrictions even if the program runs as root. If the program tries to access a file or resource not permitted by the profile, the action is denied and optionally logged.

While seccomp-BPF filters system calls directly (at the syscall dispatch level), AppArmor operates through the LSM (Linux Security Module) hook framework in the Linux kernel. These hooks are placed in strategic points inside the kernel’s internal logic — not at the raw syscall entry point.

While AppArmor can restrict access to certain high-level kernel interfaces, such as ptrace, mount, or access to /proc, through its policy language. These controls are expressed in terms of actions or paths, not raw system call numbers. AppArmor does not operate at the system call level. It cannot directly filter syscalls based on arguments or syscall numbers (e.g., deny open() unless the flags include O_RDONLY). That kind of filtering is what seccomp-BPF does best.

How AppArmor Works with seccomp-BPF

AppArmor and seccomp-BPF are complementary:

• seccomp-BPF filters system calls and their arguments. It is precise and fast but operates at a low level and cannot see file paths or context.
• AppArmor operates at a higher semantic level, enforcing path-based policies and access control rules before a syscall is executed.

Used together, they provide defense-in-depth:

• AppArmor can define what resources an application can use.
• seccomp-BPF can define how the application can interact with the kernel at the syscall level.

For example, AppArmor can restrict a process to read-only access to /home/user/*.txt, while seccomp-BPF can simultaneously block dangerous system calls like ptrace or clone.

AppArmor is particularly well-integrated into Ubuntu and other Debian-based systems, and it’s widely used to confine system services, network daemons, and desktop applications. It provides an approachable policy model for administrators who want sandboxing without writing low-level seccomp rules.

The Apple Sandbox

Conceptually, Apple’s sandbox is similar to seccomp in that it is a kernel-level sandbox, although it does not use the Berkeley Packet Filter. The sandbox comprises:

• User-level library functions for initializing and configuring the sandbox for a process
• A server process for handling logging from the kernel
• A kernel extension that uses the TrustedBSD API to enforce sandbox policies
• A kernel extension that provides support for regular expression pattern matching to enforce the defined policies

An application initializes the sandbox by calling sandbox_init. This function reads a human-friendly policy definition file and converts it into a binary format that is then passed to the kernel. Now the sandbox is initialized. Any function calls that are hooked by the TrustedBSD layer will be passed to the sandbox kernel extension for enforcement. Note that, unlike Janus, all enforcement takes place in the kernel. Enforcement means consulting a list of sandbox rules for the process that made the system call (the policy that was sent to the kernel by sandbox_init). In some cases, the rules may involve regular expression pattern matching, such as those that define filename patterns).

The Apple sandbox helps avoid comprehension errors by providing predefined sandbox profiles (entitlements). Certain resources are restricted by default, and a sandboxed app must explicitly ask the user for permission. This includes accessing:

• the system hardware (camera, microphone, USB)
• network connections, data from other apps (calendar, contacts)
• location data, and user files (photos, movies, music, user-specified files)
• iCloud services

For mobile devices, there are also entitlements for push notifications and Apple Pay/Wallet access.

Once permission is granted, the sandbox policy can be modified for that application. Some basic categories of entitlements include:

• Restrict file system access: stay within an app container, a group container, any file in the system, or temporary/global places
• Deny file writing
• Deny networking
• Deny process execution

Here’s a cleaned-up and corrected version of your description of Apple’s sandboxing system, with improved structure, clarity, and technical accuracy:

Apple Sandbox

Conceptually, Apple’s sandbox is similar to Linux’s seccomp in that it operates at the kernel level to restrict process behavior. However, unlike seccomp, it does not use the Berkeley Packet Filter (BPF). Instead, Apple’s sandboxing system is built around a different architecture that includes:

• User-space APIs for initializing and configuring sandbox rules for a process
• A sandbox daemon for handling logging and communication with the kernel
• A kernel extension that uses the TrustedBSD API to enforce access control policies
• An additional kernel extension that supports regular expression pattern matching for rules, such as those involving file path restrictions

Applications enter the sandbox by calling the sandbox_init function, which loads a human-readable sandbox policy definition (written in a custom profile language), compiles it into a binary format, and passes it to the kernel. Once initialized, the kernel enforces these rules using the TrustedBSD framework.

Any system call that is intercepted by the TrustedBSD hooks is passed to the sandbox kernel extension for evaluation against the calling process’s policy. This enforcement occurs entirely in the kernel, unlike earlier sandboxing tools like Janus, which performed userspace mediation. For file-related operations, policies may include regular expression rules to match or exclude specific filename patterns.

Entitlements and Predefined Profiles

To reduce configuration errors and simplify the security model for developers, Apple provides predefined sandbox profiles, often referred to as entitlements. These profiles define default restrictions and require apps to explicitly declare the permissions they need.

By default, sandboxed applications cannot access sensitive resources or perform certain operations without user consent. These include:

• Hardware access: camera, microphone, USB devices
• Private data: contacts, calendar, photos, music, location data
• Network access: both inbound and outbound connections
• iCloud services: syncing, document storage, keychain access
• App-specific resources: push notifications, Apple Pay, Wallet access

Once permission is granted—typically through a user prompt or provisioning profile—the sandbox policy is updated to reflect the granted entitlements.

Some common restrictions managed through entitlements include:

• Filesystem access: limit access to within the app container, group containers, or temporary/global directories
• Deny file writing: restrict write access even within allowed directories
• Deny networking: prohibit use of sockets and networking APIs
• Deny process execution: prevent the app from launching child processes

Apple’s sandboxing system enforces the principle of least privilege and provides strong, fine-grained control over application behavior. By combining kernel-level enforcement with user-consented entitlements and predefined profiles, it reduces the risk of both over-permissioning and misconfiguration, offering a secure default stance with flexibility for legitimate app needs.

3. Process Virtual Machine Sandboxes: Java

A distinct type of sandboxing model is provided by the Java Virtual Machine (JVM)—an example of a process virtual machine sandbox. The Java language was originally designed for running untrusted code safely within a web browser, particularly in the form of applets: small, compiled Java programs that could be downloaded and executed dynamically upon visiting a web page.

Because applets were expected to run on a wide range of platforms, Java programs were compiled not to native machine code, but to bytecode, a platform-independent instruction set targeting a hypothetical processor called the Java Virtual Machine (JVM). Each host system would run a JVM implementation that interpreted or compiled this bytecode at runtime. Crucially, the JVM was designed to provide strong isolation guarantees—enforcing what the untrusted code could and could not do.

The Java sandbox traditionally consists of three key components:

1. Bytecode Verifier

The bytecode verifier checks compiled Java bytecode before it is executed by the JVM. It ensures that the code conforms to the safety rules of the Java language, preventing operations such as:

• Violating access control (e.g., accessing private fields)
• Performing illegal type casts or data conversions
• Forging object references or pointers
• Accessing array elements out of bounds

This step ensures that bytecode cannot subvert the memory safety guarantees expected from Java code.

2. Class Loader

The class loader governs how Java classes are located and loaded at runtime. It enforces separation between trusted and untrusted code, ensuring that untrusted applications cannot override or tamper with core system classes (e.g., those in the Java standard library). It also controls whether an application can load external or dynamically generated classes.

Although the class loader does not directly implement Address Space Layout Randomization (ASLR) in the traditional OS sense, it contributes to unpredictability by managing the runtime layout of loaded classes and data structures, which can provide modest obfuscation benefits against certain attacks.

3. Security Manager

The security manager enforces the sandbox policy at runtime. It acts as a gatekeeper for sensitive operations by intercepting method calls that access system resources such as:

• File I/O
• Network connections
• Process execution
• System properties

Before allowing these actions, the security manager consults a user-defined policy file that specifies what permissions a particular application has. If an operation is not explicitly permitted, the JVM throws a SecurityException, preventing the action. This component is critical for enforcing the principle of least privilege: a well-written policy can restrict a program to just the resources it needs, and nothing more.

Limitations and Security Challenges

While conceptually elegant, the Java sandbox has proven to be difficult to implement securely in practice. Despite the Java language offering memory safety, automatic garbage collection, and built-in array bounds checking, numerous vulnerabilities have been discovered over the years—often in the underlying native code written in C that supports the JVM.

Challenges include:

Native Methods:

Java supports the use of native methods (via JNI, the Java Native Interface), allowing Java programs to call into libraries written in C or C++. These libraries operate outside the JVM’s control and can bypass sandbox restrictions entirely, accessing the file system, memory, and OS services directly.

Implementation Variability:

The security of a Java program depends not only on the correctness of the Java code, but also on the specific JVM implementation and platform. Bugs or differences in behavior between JVMs can lead to inconsistent enforcement of sandbox policies.

Complexity and Legacy

After decades of development, the sandbox codebase has become complex, and historically, many critical bugs have been discovered in its various components. Although most severe vulnerabilities have likely been addressed, maintaining confidence in such a large, multifaceted system remains a challenge.

The Java sandbox is a pioneering example of a process-level sandbox built into a language runtime. While it offered important early lessons in application containment, it also illustrates how difficult it is to create and maintain a secure and consistent sandbox across diverse platforms and implementations.

4. Browser-Based Sandboxing: Google Chrome

Certainly! Here’s a detailed lecture-style write-up on Google Chrome as a modern example of browser-based sandboxing, written in the same style and tone as your other sandboxing sections:

4. Browser-Based Sandboxing: Google Chrome

Web browsers today are a primary vector for software attacks. They routinely fetch and execute untrusted content—HTML, JavaScript, WebAssembly, images, videos, plugins—from remote servers, often without user interaction. This makes them an ideal target for attackers attempting to steal credentials, exfiltrate data, or exploit memory safety vulnerabilities.

Google Chrome (and the open-source Chromium project it’s based on) was the first widely adopted browser to be architected with sandboxing as a foundational design principle. Rather than relying on monolithic process models like early versions of Firefox or Internet Explorer, Chrome uses a multi-process architecture to separate privileges and isolate untrusted content. This approach is now standard across all major browsers.

Motivation and Threat Model

The central idea is to minimize the privileges of web content and enforce a strict separation between:

• Trusted components (e.g., the browser UI, password manager)
• Untrusted components (e.g., code from arbitrary websites)

The browser must protect against:

• Malicious websites exploiting rendering or JavaScript engine bugs
• Cross-site attacks (e.g., stealing data from one origin by attacking another)
• Compromised processes attempting to escalate privilege or escape to the host OS

Multi-Process Architecture

At the heart of Chrome’s sandbox is a multi-process model:

• Each renderer process is responsible for parsing and rendering HTML, executing JavaScript, and handling layout and styling for a specific web origin or group of origins.
• Renderer processes are unprivileged and highly restricted.
• Other tasks (e.g., disk access, network communication, GPU acceleration) are delegated to separate utility processes that have only the minimal privileges required.

This division enforces privilege separation and fault isolation. A crash or compromise in one renderer cannot affect other tabs or steal sensitive data from unrelated sites.

Chrome uses a layered defense-in-depth strategy combining multiple sandboxing mechanisms:

1. Operating System Sandboxes

Each renderer process runs inside a sandbox enforced by the host operating system:

• Linux: Uses seccomp-BPF to block dangerous system calls, Linux namespaces to isolate system views (e.g., mount, PID), and setuid sandboxing to drop privileges.
• Windows: Uses Job Objects, Integrity Levels, and AppContainers to restrict access to OS resources.
• macOS: Uses the native sandbox, a mandatory access control (MAC) system based on the TrustedBSD API.

These mechanisms prevent renderer processes from reading files, opening raw sockets, accessing device drivers, or spawning new processes.

2. Broker Processes and IPC

Renderer processes cannot access system resources directly. Instead, they communicate via inter-process communication (IPC) with more privileged broker processes that mediate access to the file system, network, and other services.

The broker enforces policies on each request. For example, a renderer process might request access to a file, but the broker can deny it unless explicitly allowed by the user’s actions (e.g., through a file picker dialog).

3. Site Isolation

Chrome’s site isolation model strengthens sandboxing by ensuring that content from different origins is rendered in separate processes. This protects against cross-site data leaks, such as those enabled by speculative execution attacks like Spectre.

Each renderer process is now bound to a single origin or site. DOM elements, JavaScript, and cookies from one origin are never processed in the same memory space as another.

This prevents attackers from exploiting a renderer bug on a malicious site (e.g., evil.com) to access data belonging to a different site (e.g., bank.com).

4. Fine-Grained Permissions

Sandboxed processes are granted only the minimal permissions necessary:

• No filesystem access by default
• Limited network access (subject to origin policies)
• Clipboard, microphone, camera, and location access require explicit user consent
• Access to local devices (e.g., USB) is controlled by permission prompts and restricted APIs

This enforcement of the principle of least privilege reduces the damage even if a sandbox escape occurs.

Benefits and Limitations

Benefits

• Isolation by default: Web content is automatically untrusted and run in separate, low-privilege processes.
• Resilience against bugs: Even if a vulnerability is found in the rendering engine, an attacker still has to escape the sandbox.
• Reduced cross-site leakage: Site isolation prevents compromised tabs from stealing data from others.
• Modular hardening: Different components (GPU, PDF viewer, audio service) are sandboxed independently.

Limitations

• Performance overhead: Process isolation increases memory usage and adds IPC latency, although mitigated by optimizations like process reuse.
• Sandbox escape bugs: Kernel or OS-level vulnerabilities (e.g., privilege escalation) can allow attackers to break out of the sandbox if not patched.
• Complexity: The security architecture is complex and requires precise policy design and thorough testing.

The Chrome browser sandbox is one of the most mature and robust examples of a real-world, large-scale sandboxing implementation. It combines:

• Operating system-level sandboxing
• Process isolation and privilege separation
• Brokered access to system resources
• Site isolation to defend against cross-origin attacks

Together, these mechanisms enforce strict boundaries around untrusted content while still allowing powerful and interactive web applications to run.

Chrome’s sandboxing architecture shows how modern software applies defense-in-depth to contain threats, minimize attack surfaces, and enforce user intent in complex, untrusted environments.