Integer Overflow

Buffer overflows get all the headlines, but they often have a quieter accomplice: integer overflow. An attacker who controls a number that is used to size a buffer allocation or guard a copy does not need to smash the stack directly — they just need to make that number lie. A size computation that wraps to zero, a negative length that becomes enormous when treated as unsigned, or a 32-bit value silently chopped to 16 bits: any of these can trick a program into allocating far too little memory, then cheerfully copying far too much into it. The result is a buffer overflow triggered by bad arithmetic, not bad string handling. Integer overflow appears on the CWE Top 25 precisely because it is everywhere in C code that deals with untrusted sizes and counts.

The lecture distinguishes three distinct flavors.

1. Truncation

Truncation happens when a value is assigned to a narrower type and the upper bits are silently discarded.

int i = 0x12345678;   // 32-bit: 305,419,896
short s = i;          // 16-bit: keeps only 0x5678 → 22,136
char  c = i;          //  8-bit: keeps only 0x78   → 120

Running that code prints:

i=0x12345678(305419896), s=0x5678(22136), c=0x78(120)

The compiler does not warn by default and the machine code is a plain mov — the hardware just stores fewer bytes. A security-sensitive pattern is reading a user-supplied length into an int, performing a bounds check, then passing the value to a function that expects a short or a char. The check passes against the full 32-bit value, but the narrowed copy that actually drives the write can be far smaller — causing an over-read — or, if the high bits set a carry, far larger than expected.

2. Signedness Bugs

Signedness bugs arise when the same bit pattern is interpreted as signed in one context and unsigned in another.

Signed vs. unsigned comparison

sizeof returns size_t, which is an unsigned type. Comparing a user-supplied int against it triggers C's usual arithmetic conversions: the signed value is reinterpreted as unsigned.

int size;
scanf("%d", &size);
if (size < sizeof(buffer))    // BUG: if size == -1, this becomes huge unsigned
    memcpy(dst, src, size);   // size cast to size_t: 0xFFFFFFFF bytes copied

A negative size like -1 is 0xFFFFFFFF when viewed as unsigned — far larger than sizeof(buffer) — so the check passes, and memcpy is called with a gigantic count. The same issue arises with malloc(size) when size < 0: malloc treats its argument as size_t, so it attempts a ~4 GB allocation.

Assuming nonnegativity in signed comparisons

if (x < 100)
    do_something();

If x is int, negative values satisfy the check too. Code written assuming x >= 0 may use x as an index or a length downstream without further validation, opening an out-of-bounds access.

The INT_MIN negation trap

A subtler variant involves signed overflow through negation. Consider an absolute() function:

int absolute(int i) {
    if (i < 0) return -i;
    else        return  i;
}

And a caller that checks the result:

if (absolute(passwd) < 0) { printf("Password OK\n"); }

This looks impossible — how can absolute return negative? The answer is INT_MIN (0x80000000 = −2,147,483,648). In two's-complement arithmetic, negating INT_MIN overflows: -INT_MIN wraps back to INT_MIN, which is still negative. At the machine level, neg %eax on 0x80000000 yields 0x80000000 again, so absolute(INT_MIN) returns a negative value and the "Password OK" branch is taken.

The two's complement formula for an N-bit integer makes the most-significant bit carry weight −2^(N−1), which is why INT_MIN has no positive counterpart.

3. Arithmetic Overflow

Arithmetic overflow occurs when the mathematical result of an operation exceeds the range of the destination type. For unsigned types in C, the result wraps modulo 2^N — this is defined behavior. For signed types, overflow is undefined behavior in the C standard, which lets optimizing compilers assume it never happens and eliminate overflow checks you wrote yourself.

unsigned int a = 0xEF345678;
unsigned int b = 0x12345678;
unsigned int sum = a + b;    // 0x10168ACF0 → wraps to 0x0168ACF0

The x86 add instruction sets the carry flag when unsigned overflow occurs, but C does not expose that flag — the result in the register is just the truncated low bits.

Integer-overflow-to-buffer-overflow: the OpenSSH example

A real-world instance from OpenSSH:

nresp = packet_get_int();          // attacker-controlled
if (nresp > 0) {
    response = xmalloc(nresp * sizeof(char *));  // sizeof(char*) == 4
    for (i = 0; i < nresp; i++)
        response[i] = packet_get_string(NULL);
}

If an attacker sets nresp = 0x40000000 (1,073,741,824), then:

nresp * sizeof(char *) = 0x40000000 * 4 = 0x100000000

On a 32-bit system this overflows to 0 — xmalloc(0) returns a tiny (or zero-size) buffer. The subsequent loop then writes 1,073,741,824 pointer-sized values into that buffer, producing a massive heap overflow. The attacker turned a numeric calculation into remote code execution.

The general pattern: attacker controls a count → multiply overflows → allocation is too small → subsequent write overflows the allocation → memory corruption.

Why Checking Is Hard at the Machine Level

Low-level machine code has no concept of types. The C add instruction is the same whether you write int a + int b or unsigned int a + unsigned int b — just raw bits in registers. The carry and overflow flags are set in hardware but C gives no portable way to read them after an operation. This means runtime detection requires either compiler instrumentation or explicit pre-operation checks in source code.

Detection and Defenses

Use the right types

Using size_t for sizes eliminates signed/unsigned comparison mismatches; using fixed-width types (uint32_t etc.) makes bit-width explicit and portable.

Check before you compute

For an unsigned multiply, validate before performing it:

// Safe multiply: check that a * b won't overflow size_t
if (b != 0 && a > SIZE_MAX / b) {
    /* handle overflow */
}
size_t total = a * b;

For addition: if (a > UINT_MAX - b) { /* overflow */ }.

Compiler flags

Example:

gcc -ftrapv int.c -o safe-int

UBSan (-fsanitize=undefined) is the most thorough option during development and testing; -fwrapv is useful when you want predictable wrap-around semantics in production code.

Key takeaways

• Integer overflow comes in three flavors: truncation (narrowing cast drops high bits), signedness bugs (same bits reinterpreted as signed vs. unsigned), and arithmetic overflow (result exceeds type range and wraps or causes UB).
• Unsigned overflow wraps modulo 2^N (defined); signed overflow is undefined behavior in C — the compiler may eliminate checks that assume it.
• INT_MIN is the classic signedness trap: negating it wraps back to itself, so an "always non-negative" check can fail.
• The dangerous pattern is integer-overflow-to-buffer-overflow: an overflowed size drives an under-allocation, and the subsequent write overflows the heap or stack buffer.
• Defenses: use size_t / fixed-width types, validate before multiplying or adding, and instrument with -fsanitize=undefined during development.