pk.org: Computer Security/Lecture Notes

Malware and Defenses

Study Guide

Paul Krzyzanowski – 2025-11-08

Malware is software intentionally designed to perform unwanted, unexpected, or harmful actions on a target system. Three requirements: intentional (bugs don't count), unwanted by the legitimate owner, and causes harm or performs unauthorized actions.

Zero-Day and N-Day Exploits

Malware often relies on software vulnerabilities to gain access or escalate privileges. Two common terms describe how attackers exploit flaws based on when they become known.

Zero-day attacks reflect gaps in vendor and researcher awareness; N-day attacks expose weaknesses in operational security and patch management. Both remain central to modern malware campaigns.

Malware Classification by Function

Self-Propagating Malware

The critical distinction is agency -- whether human action is required.

Virus: Attaches to host files (executables or documents with macros). Requires user action to spread—running infected programs or opening infected documents. When activated, seeks other files to infect.

Worm: A self-contained program that spreads autonomously across networks without user intervention. It scans for vulnerable systems and automatically attempts to infect them.

Key difference: Viruses need users to help them spread; worms spread on their own.

Stealth and Unauthorized Access

Trojan Horse: Appears to be legitimate software that users willingly install. it combines an overt purpose (cache cleaning, system optimization) with covert malicious actions (installing backdoors, spyware, ransomware).

Backdoor: Provides remote access bypassing normal authentication. It allows attackers to return to compromised systems at will.

Rootkit: Operates at the kernel or system level to evade detection. It intercepts system calls and lies to security tools, concealing files, processes, network connections, and registry entries.

Financial Malware

Ransomware: Encrypts files or locks systems, demanding payment for restoration. Some variants use double extortion—encrypting data while also exfiltrating it and threatening to publish stolen data.

Cryptojacking: Secretly uses the victim's computing resources to mine cryptocurrency. This causes degraded performance and increased power consumption.

Adware: Displays unwanted advertisements for revenue, often bundled with free software.

Data Theft

Spyware: Monitors user activity without consent. This includes operations like keylogging, screen capture, and browser monitoring.

Keylogger: Records every keystroke to capture passwords, credit card numbers, and private messages.

Information Stealer: Targets stored credentials, browser data, cryptocurrency wallets, and other valuable information.

Remote Control

Bot/Botnet: Infected computers (bots, also known as zombies) controlled remotely and organized into networks (botnets) for coordinated attacks. Used for DDoS, spam distribution, and credential stuffing.

Remote Access Trojan (RAT): Provides the attacker with comprehensive remote control: file access, screen viewing, webcam/microphone activation, and command execution.

Destructive Malware

Logic Bomb: Remains dormant until specific conditions trigger it (date, event, command).

Wiper: Destroys data and systems without a financial motive, often used in geopolitical conflicts.

Nuisance Malware

Scareware: Falsely claims the system is infected to push fake security software purchases.

Browser Hijacker: Modifies browser settings to redirect users and track browsing activity.

How Malware Spreads

Malware employs three broad categories:

  1. Network-based attacks exploiting technical vulnerabilities

  2. User-assisted methods relying on human interaction

  3. Supply chain compromises poisoning trusted distribution channels.

1. Network-Based Propagation

2. User-Assisted Propagation

Attackers also use domain deception to mislead users into visiting malicious sites.

3. USB-Based Attacks

4. Supply Chain Attacks

Social Engineering

Social engineering manipulates human psychology rather than exploiting technical vulnerabilities. This is often the weakest link in security.

Psychological Manipulation Tactics

Common Attack Vectors

The Malware Lifecycle

Modern malware operates through a six-stage lifecycle rather than as a single program.

Stage 1: Infection and Delivery

Getting malware onto the target system through exploiting vulnerabilities, social engineering, physical access, supply chain compromise, or drive-by downloads. The initial payload is often just a small first stage that downloads the real malware.

Stage 2: Dropper and Loader

Dropper (or downloader): Downloads and installs the main malware from a remote server. Small and obfuscated, it performs environment checks (VM detection, antivirus status, OS version) before proceeding.

Loader: Includes payload embedded within itself, encrypted or compressed. Unpacks and executes the hidden payload.

Advantages: Smaller initial payloads are easier to deliver, malware can be updated easily (it's a separate component), environment detection avoids sandboxes, and flexible payload delivery.

Stage 3: Persistence Mechanisms

Ensures malware survives reboots. Malware often establishes multiple mechanisms for redundancy.

Windows:

Linux/macOS:

Cross-platform:

Stage 4: Trigger Conditions

Immediate execution: Runs as soon as installed (common for ransomware).

Time-based triggers:

Event-based triggers: Banking website visits, accessing specific files, system idle, reboot count, and presence of analysis tools.

Manual triggers: Waits for commands from C2 server; operators decide when to activate based on reconnaissance.

Delayed activation evades time-limited sandbox analysis and enables synchronized attacks.

Stage 5: Payload Execution

What malware actually does—its core functionality:

Stage 6: Propagation

Viruses: Infect other files on the same system. Spreading requires user action—sharing and opening infected files.

Worms: Spread autonomously. Typical cycle: infect, scan network, test vulnerabilities, exploit, copy and execute, repeat. Creates exponential growth.

Common methods: Email (address book contacts), network shares, removable media, network exploits, peer-to-peer networks, social media, APIs with stolen credentials.

Command and Control (C2 or C&C) Mechanisms

Attackers need communication channels to send commands, receive stolen data, and update malware. This is essential for bots, which often sit idle until instructed to launch attacks or download new payloads. The attacker’s challenge is to maintain communication without being noticed or blocked.

C2 Communication Methods

Malware uses a variety of mechanisms to exchange data with its controllers. These channels must work across firewalls and blend into normal traffic. Common communication strategies include:

C2 Evasion Techniques

These techniques help hide the location of C2 servers, disguise traffic patterns, and avoid simple blocking rules:

Evading Detection and Analysis

Code Obfuscation Techniques

Crypters: Encrypt the malware so that only encrypted data appears on disk; it's decrypted only at runtime. Security tools scanning files see only encrypted content.

Packers: Tools that compress executables and add a small unpacking stub. The real code appears only after the program runs.

Polymorphism: The malware mutates its wrapper code (decryptor or unpacking stub) using techniques such as code reordering, instruction substitution, and junk-code insertion. The payload stays the same. Each copy looks different enough to evade signature-based detection.

Metamorphism: The malware the malware’s entire code body using the same types of transformations (reordering, substitution, junk instructions) so each copy has a different internal structure. There is no constant core to match, which makes signature-based detection significantly harder.

Anti-Analysis Techniques

Virtual Machine Detection: Checks for VM artifacts (hardware IDs, VM-related files/processes, timing inconsistencies). Refuses to execute in detected VMs.

Sandbox Detection: Detects analysis environments through limited user activity, small number of files, short uptime. Remains dormant or behaves benignly.

Debugger Detection: Identifies debugging tools through API calls or debugging flags. Alters behavior or terminates when detected.

Time-based Evasion: Delays malicious activity using sleep functions or waits for specific dates/events. Evades automated analysis with time limits.

Side-Channel Attacks

Side-channel attacks exploit unintended signals rather than software flaws. Malware can leak data or receive commands by manipulating physical or observable system behavior, such as timing, power usage, or device LEDs.

An example is using the keyboard Caps Lock LED to blink encoded data (Bad Bunny), allowing a nearby camera or sensor to capture exfiltrated information. These channels bypass normal network defenses because they rely on observable side effects rather than network traffic.

Fileless Malware

Fileless malware operates entirely in memory without writing files to disk, making it significantly harder to detect since traditional antivirus scans files on disk.

PowerShell-based: Uses Windows PowerShell to download and execute code directly in memory.

Registry-based: Stores code in registry values rather than files.

Living off the land: Uses legitimate system tools (PowerShell, Windows Script Host) for malicious purposes.

Privilege Escalation

Once malware gains initial access, it often needs to escalate privileges to gain full system control and bypass security restrictions.

Kernel exploits: Exploits OS vulnerabilities for system-level access.

Privilege prompt bypasses: Avoid or subvert mechanisms that require user approval for elevated actions, such as Windows UAC or macOS authorization dialogs.

Ken Thompson's "Reflections on Trusting Trust" described perhaps the most insidious backdoor: one that evades even source-code inspection by hiding in the compiler itself, demonstrating that perfect security is impossible and trust is unavoidable at some level.

Defending Against Malware

No single defense is sufficient; each has gaps. Effective defense requires layered approaches.

Anti-Malware Software: Signature-Based Detection

Uses databases of known malware signatures: unique byte patterns identifying specific malware.

Strengths: Fast, accurate for known threats, low false positive rates.

Limitations: Requires updates for new threats; easily evaded through polymorphism and encryption.

Anti-Malware Software: Heuristic and Behavioral Analysis

Examines behavior and characteristics rather than exact signatures.

Static heuristics: Analyzes file structure without execution (suspicious API calls, unusual code patterns, obfuscation indicators).

Dynamic heuristics: Observes program behavior during execution (file modifications, registry changes, network connections, process creation).

Machine learning is often used to train models on malware/benign samples to identify suspicious characteristics. Heuristics are better at detecting new threats but have higher false positive rates and are more resource-intensive.

Sandboxing

Executes suspicious files in isolated environments to safely observe behavior.

Types: Virtual machines (complete OS isolation), application sandboxes (restrict program capabilities).

Benefits: Safe observation, detects unknown threats.

Application sandboxes are the dominant means of protecting mobile devices and are increasingly being adopted across other systems (but slowly). Virtual machines provide an environment for anti-malware software writers to test for threats.

Limitations: Sophisticated malware can detect and evade sandboxes ... or use techniques such as waiting several days or a certain number of reboots before activation.

Honeypots

Honeypots are isolated decoy systems that appear vulnerable or valuable but contain no real data. They attract attackers and record their activity, providing early warning of intrusions and insights into tools and techniques without risking production systems.

Access Control and Privilege Management

Restricting the ability of users to access files and other system resources restricts the ability of malware to do the same. It limits the damage from successful infections:

Email Security

Email has long been a popular channel for social engineering, often delivering malicious attachments or links behind seemingly trustworthy messages. Most email-based attacks depend on deception: impersonating legitimate senders.

Several mechanisms help receiving systems validate the origin and integrity of email:

Additional security measures focus on message content:

Patch Management

Regular software updates address known vulnerabilities and help avoid N-day attacks. Challenges include compatibility issues, testing requirements, and zero-day vulnerabilities (unknown to the vendor).

Terms you should know