Memory-Corruption Defenses: Canaries, DEP/NX, ASLR & RELRO

Every attack module in this course — stack overflows, shellcode, ret2libc, ROP, CFI bypass — runs head-first into a defense. These defenses are not independent options; the OS and compiler stack them deliberately. Understanding why each mitigation exists, what it stops, and how attackers route around it ties the entire exploitation unit together.

The honest answer is that no single mitigation is sufficient. Each one has a known bypass, which is exactly why they are layered. This module builds that mental model.

Defense overview

Stack canaries

What it is

A stack canary (introduced by StackGuard, now gcc -fstack-protector) is a randomly chosen value placed on the stack between the local buffers and the saved %ebp/return address at function entry. Before the function executes ret, the compiler inserts a check: re-read the canary slot and compare it to the stored original. A mismatch triggers an immediate program abort.

Stack frame layout with canary:

[ buffer              ]   ← overflow starts here
[ stack canary value  ]   ← written at entry, checked before ret
[ saved %ebp          ]
[ return address      ]

The canary value is generated fresh per-process (usually from /dev/urandom) and cached in a thread-local region (%gs:0x14 on 32-bit Linux), so the attacker cannot predict it without a read primitive.

What it stops

A classic linear buffer overflow: the payload must march through the canary slot on the way to the return address, corrupting it. The check catches this before ret fires.

How attackers bypass it

1. Info leak. If the program has a separate read-beyond-bounds bug (a printf format string, an out-of-bounds array read), an attacker can read the canary's current value and include it verbatim in the overflow payload, passing the check.
2. Non-contiguous write. Some vulnerabilities allow overwriting an arbitrary memory location without touching the bytes in between — for example, an integer-overflow index or a heap write. These skip over the canary entirely.
3. Brute force (32-bit only). On 32-bit systems with forking servers, the canary might be guessable byte-by-byte (256 attempts per byte × 4 bytes = 1024 tries in the best case), because fork() preserves the parent's canary.

gcc -fstack-protector-all applies canaries to every function, not just those with large local arrays. -fstack-protector-strong is the production compromise: it covers functions with arrays or address-taken locals.

DEP / NX / W^X

What it is

Data Execution Prevention (DEP) (Microsoft's name), NX (No-eXecute, Intel's hardware bit), and W^X (Write XOR Execute, OpenBSD's policy) are three names for the same idea: a memory page cannot be both writable and executable at the same time.

• Stack and heap pages: writable, not executable.
• Code (.text) pages: executable, not writable.

The CPU enforces this per-page using the NX bit in the page-table entry. On 32-bit x86 you need PAE enabled; on x86-64 it is always available. Linux calls the combined feature W^X or simply non-executable stack/heap.

What it stops

Injected shellcode. The attacker can still overflow the buffer and write their payload to the stack, but when the return address redirects %eip into that region, the CPU raises a fault instead of executing the bytes.

How attackers bypass it

DEP/NX does not protect against reusing already-executable code. The .text segment and all loaded libraries remain executable by design. This gave rise to:

• ret2libc: overwrite the return address with system() in libc; place "/bin/sh" as its argument.
• Return-Oriented Programming (ROP): chain short gadgets — instruction sequences ending in ret — already present in the binary or libraries. No injected bytes are needed. (See the ROP module for the full treatment.)

DEP/NX is a critical mitigation that permanently raises the bar, but it shifts the attack rather than eliminating it.

ASLR and PIE

What it is

Address Space Layout Randomization (ASLR) randomizes the base addresses of the stack, heap, and shared libraries each time a process starts. On Linux it is enabled system-wide via /proc/sys/kernel/randomize_va_space:

Position-Independent Executable (PIE) extends randomization to the executable itself (the .text, .data, .bss segments). A program must be compiled with gcc -fPIE -pie to benefit from PIE; without it, the main binary loads at a fixed address even when ASLR is on.

Together, ASLR + PIE means every segment has a fresh random base on each run. Hardcoded addresses — the attacker's ret2libc or ROP gadget addresses — are wrong on every run.

What it stops

Any exploit that relies on knowing the absolute address of a function, a gadget, a buffer, or a string at exploit-write time. With full ASLR + PIE, all such addresses change per execution.

How attackers bypass it

1. Info leak (most reliable). A format-string bug, use-after-free read, or any primitive that lets the attacker read a pointer from the process gives away the current base address. One leaked library pointer reveals the entire libc layout (because the library's internal layout is fixed — only the base is randomized).
2. Partial overwrite. ASLR randomizes the upper bits of an address, but the lower 12 bits (one page) are always zero-aligned, and even more bits may be stable depending on entropy. On 32-bit Linux, ASLR provides only ~8–16 bits of entropy, making brute force feasible over a respawning service.
3. No-PIE binary. If the executable was not compiled with -fPIE, its .text and GOT are at fixed addresses. An attacker can jump to code inside the main binary without needing to leak anything — this is why many modern Linux distributions build all packages with PIE.
4. Heap spray / JIT spray. Flood the address space with copies of the payload so that a random jump has a good chance of landing in a valid copy (more relevant in browser/JIT contexts than in CTF-style binaries).

RELRO

What it is

RELRO (RELocation Read-Only) is a linker/loader hardening feature that makes certain ELF data sections read-only after the dynamic linker finishes using them.

Full RELRO forces the dynamic linker to resolve all symbols eagerly at load time (the -z now part), so the GOT can be locked immediately. Partial RELRO leaves the GOT writable because lazy binding still needs to update it.

What it stops

GOT-overwrite attacks. The Global Offset Table holds pointers to libc functions (e.g., the runtime address of printf, exit, __stack_chk_fail). Without RELRO, an attacker who can write an arbitrary memory address can redirect a GOT entry to point to shellcode or a ROP gadget. Full RELRO makes the GOT read-only, so any attempted write faults immediately.

How attackers bypass it

RELRO only hardens the GOT and a few other ELF metadata sections. It does not protect:

• Heap metadata (function pointers stored in malloc arena headers).
• BSS or data-segment function pointers (e.g., a global void (*handler)(void) variable).
• Stack data (the return address and any function pointer stored on the stack).

If a writable function pointer exists outside the GOT, full RELRO does not help.

CFI and shadow stacks: the next layer

Stack canaries detect overflows after the fact; ASLR hides addresses; RELRO protects the GOT. Control-Flow Integrity (CFI) takes a different approach: it validates every indirect transfer (indirect calls, indirect jumps, and ret) against a policy derived at compile time or enforced in hardware.

Shadow stacks (e.g., Intel CET's shadow stack, glibc's pointer-mangling) maintain a separate, attacker-inaccessible copy of return addresses. When ret executes, the CPU compares the stack's return address against the shadow copy; a mismatch raises a fault. This directly defeats ROP chains that pivot on ret, independent of whether ASLR is in play.

See the CFI module for the full treatment of policy types, forward-edge vs. back-edge CFI, and hardware enforcement.

Defense-in-depth: no single mitigation is enough

The reason all four mitigations are deployed together is that each has a known bypass:

• A canary leak lets you bypass the canary.
• DEP/NX forces reuse attacks (ret2libc/ROP) rather than blocking them.
• An info leak collapses ASLR.
• RELRO does not cover every writable function pointer.

The attacker must defeat all active layers simultaneously — which is why real-world exploits against hardened targets require chaining a memory read (to leak an address), a write (to corrupt a control pointer), and precise knowledge of the binary layout. Each additional mitigation raises the cost and reduces the attacker's error budget. Defense-in-depth does not make exploitation impossible; it makes it expensive and brittle.

Key takeaways

• Stack canaries sit between local buffers and the return address; they detect contiguous overflows but are defeated by info leaks or non-contiguous writes.
• DEP / NX permanently eliminates shellcode injection as an attack class; it motivated ret2libc and ROP as the standard successors.
• ASLR + PIE randomizes all base addresses; without PIE the main binary is still at a fixed location, leaving gadgets and the GOT accessible without leaking.
• Full RELRO locks the GOT after load-time symbol resolution, blocking GOT-overwrite attacks; partial RELRO leaves the GOT writable.
• CFI and shadow stacks validate control-flow transfers at a finer grain and are the state-of-the-art defense against ROP.
• These mitigations are designed to be layered: real exploitation requires bypassing multiple layers in sequence, which is why hardened targets are so much harder to exploit than unprotected CTF binaries.