CS 487 — Secure Computer Systems

Foundations of secure computer systems: software vulnerabilities and exploitation, memory-corruption defenses, cryptography (symmetric/asymmetric, hashing, PKI), authentication and access control, OS and side-channel security, and network defense.

1. Linux Permissions, Set-UID & Environment Variables

   Linux enforces access control through permission bits and user/group ownership — but the Set-UID mechanism deliberately grants temporary elevated privilege, making Set-UID root programs a prime target for privilege escalation via environment variable manipulation.

2. Control-Flow Hijacking

   How a stack buffer overflow lets attacker-supplied input redirect a program's execution — overwriting local data, function pointers, and the return address, up to injecting shellcode for a privileged shell.

3. Shellcode

   When there is no convenient function to redirect execution into, attackers inject their own machine code. This module covers the Linux system-call ABI, building a NULL-free execve("/bin/sh") payload using the stack for position independence, and deploying it with a NOP sled — plus the defenses (NX stack, ASLR, stack canaries) that motivated later techniques like ROP.

4. Code-Reuse Attacks: ret2libc

   When a Non-Executable Stack (DEP/NX) blocks injected shellcode, attackers pivot to code that is already loaded — redirecting the return address straight into libc functions like system("/bin/sh"). This module builds the ret2libc technique from first principles and shows why it sets the stage for ROP.

5. Return-Oriented Programming (ROP)

   DEP/NX blocks injected shellcode, but it cannot stop an attacker who reuses the program's own instructions — ROP chains short existing code sequences called gadgets, each ending in `ret`, to perform arbitrary computation without a single byte of injected code.

6. Control-Flow Integrity (CFI)

   ROP and ret2libc attacks succeed by bending indirect control-flow transfers to arbitrary targets; CFI defeats them by enforcing at runtime that every indirect jump, call, and return lands only on destinations permitted by a statically computed control-flow graph.

7. Software Testing, Fuzzing & Symbolic Execution

   Covers automated vulnerability discovery through the lens of software testing theory (black-box vs. white-box, code coverage), coverage-guided fuzzing with AFL and libFuzzer, sanitizers like AddressSanitizer, and symbolic/concolic execution with SMT solvers.

8. Format-String Vulnerabilities

   A format-string bug arises when attacker-controlled data is passed directly as the format argument to printf-family functions, enabling arbitrary stack reads (%x/%s), arbitrary memory writes (%n), and ultimately control-flow hijacking.

9. Reverse Shells

   Understand how a reverse shell flips the connection direction to defeat inbound-blocking firewalls, and trace the mechanism from file descriptors and dup2 redirection to the one-liners attackers drop as exploitation payloads.

10. Integer Overflow

    Fixed-width integers silently wrap, truncate, or change sign in C — and when those integers control buffer sizes or security checks, the result is exploitable. This module covers the three flavors of integer overflow, how attackers leverage them, and compiler/code-level defenses.

11. Heap Exploitation

    Explores how dynamic memory managed by glibc malloc works — chunks, headers, and free lists — and how attackers exploit that structure through heap buffer overflows, use-after-free bugs, and metadata corruption to achieve arbitrary writes and control-flow hijacking.

12. Memory-Corruption Defenses: Canaries, DEP/NX, ASLR & RELRO

    Surveys the four standard OS- and compiler-level mitigations — stack canaries, DEP/NX, ASLR/PIE, and RELRO — explaining what each stops, how they interact, and how the attack modules progressively sidestep them.

13. Malware

    A survey of malicious software: taxonomy (viruses, worms, trojans, ransomware, rootkits, botnets), how malware gets onto a system, evasion techniques (encrypted, polymorphic, and metamorphic viruses), and detection/defense approaches (signatures, heuristics, IDS/IPS).

14. Symmetric-Key Cryptography

    Covers the foundational goals of cryptography, Kerckhoffs's principle, and the symmetric encryption model — including classical ciphers and their failure modes, stream vs. block ciphers, DES/3DES history, AES, block-cipher modes (ECB/CBC), and the key-distribution problem that motivates public-key crypto.

15. AES, Encryption Modes & Hash Functions

    Covers the AES block cipher and its four internal transformation rounds, the encryption modes (ECB, CBC, CFB, CTR) that determine how AES handles multi-block messages, PKCS#7 padding, IV requirements, and cryptographic hash functions including the SHA family, broken algorithms, and practical applications such as password storage with salts.

16. Public-Key Cryptography

    Introduces asymmetric (public-key) cryptography as the solution to the symmetric-key distribution problem, covering the keypair model, Diffie-Hellman key exchange, RSA fundamentals, hybrid encryption, digital signatures, and the man-in-the-middle threat that motivates PKI.

17. RSA

    RSA is the most widely deployed public-key cryptosystem, securing SSH, TLS, and digital signatures. This module covers key generation, encryption and decryption, correctness via Euler's theorem, digital signing, and why textbook RSA must never be used without padding.

18. PKI & Authentication

    Explains the key-authenticity problem and how Public Key Infrastructure solves it with X.509 certificates, certificate authorities, and chain-of-trust verification. Also covers entity authentication factors — passwords, tokens, biometrics — together with secure password storage, multi-factor authentication, and common attacks such as MITM and password cracking.

19. Access Control

    Access control is the mechanism that decides which authenticated subjects can perform which operations on which objects. This module covers the access control matrix and its two practical representations (ACLs and capabilities), the four major design policies (DAC, MAC, RBAC, ABAC), formal confidentiality and integrity models, and the principle of least privilege.

20. Side Channels, CPU Bugs & OS Security

    Explores how attackers extract secrets through physical and microarchitectural side channels — including cache timing, Meltdown, and Spectre — and how operating systems enforce the kernel/user boundary to limit damage.

21. Intrusion Detection & Firewalls

    Covers how intrusion detection systems (IDS) identify malicious activity through signature-based and anomaly-based analysis, and how firewalls enforce network policy via packet-filter, stateful, and application-proxy mechanisms.

22. Network Security: TCP/IP, Sniff/Spoof & IPsec

    Classic TCP/IP was designed for connectivity, not security — no source-address authentication and no confidentiality. This module covers how attackers exploit those gaps through packet sniffing and spoofing, and how IPsec's AH and ESP protocols retrofit security at the network layer.