NexusCTF writeup

This challenge was first-blooded and only solved by me during the entire competition , if you’re a reverse engineering enthusiast, this blog is for you. Let’s dive in and break down the reversing process step by step , have fun, See you next year at NexHuntCTF

📎 Attachments

eleonoras

1) Overview

This challenge is a high-quality VM-based reverse-engineering problem combining:

• A custom bytecode virtual machine
• Heavy static linking and stripping
• Intentional decoy cryptography
• Multi-segment encrypted flag reconstruction

No patching, no brute force, no hooking — the solution requires pure understanding of execution semantics.

2) Binary Properties

• Architecture: x86_64
• Type: ELF
• Statically linked
• Stripped (no symbols)
• No libc imports
• Custom syscalls only

This immediately rules out:

• LD_PRELOAD tricks
• Easy dynamic analysis
• Simple string extraction

4) Entry Point & Runtime Noise

Execution begins in _start and flows through a large amount of boilerplate:

• TLS initialization
• arch_prctl
• signal masking
• stack guards
• relocation fixups

This stage exists purely to hide real logic and slow down analysis.

Eventually execution reaches: sub_402620(...) which calls a function pointer resolved at runtime. That pointer leads to:

This is the core of the challenge.

5) Identifying the Virtual Machine

Function: sub_401DE0

This function contains:

• An infinite loop
• A bytecode pointer
• A program counter
• A massive switch on a single byte

This is a classic Fetch-Decode-Execute VM loop: FETCH → opcode = bytecode[pc] DECODE → switch(opcode) EXEC → handler

VM State

6) Opcode Analysis

The dispatcher contains 30+ opcode cases.

Most opcodes:

• Modify VM memory
• Adjust the PC
• Exist purely as noise

Only one opcode is relevant.

7) Opcode 16 — Segment Controller

Opcode 16 (0x10) is the only meaningful instruction.

Important observation:

Opcode 16 does not decrypt the flag directly.

Instead, it:

1. Selects a segment index (0–4)
2. Chooses a key address
3. Chooses a ciphertext address
4. Calls a shared worker function

Segment Routing

Each segment is processed independently.

8) The Decoy Trap

The function called by opcode 16 (sub_401C50) appears to implement encryption logic using:

• XOR chains
• SIMD instructions
• Magic constants
• Rolling math

Running it as written produces a fake flag.

This logic is intentionally misleading.

The real cryptography must be reconstructed semantically, not by copying the C code.

9) Real Cryptographic Logic (Recovered)

Each flag segment is decrypted independently.

Step 1 — Polynomial Unfolding

For byte index i:

poly(i) = i² + 7i + 13

Apply:

Step 2 — Keystream Generation

Generate a keystream using:

1
HMAC-SHA512(
2
key = segment_key,
3
message = segment_key
4
)

Repeat or truncate to segment length.

Step 3 — Dual Keystream XOR

Apply keystream twice:

1
stage2[i] = stage1[i] XOR keystream[len-1-i]
2
plaintext[i] = stage2[i] XOR keystream[i]

10) Extracting Keys and Ciphertexts from .rodata

After identifying the VM dispatcher (sub_401DE0) and isolating opcode 16, the next critical step was extracting the static cryptographic material embedded in the binary.

Opcode 16 does not perform decryption itself. Instead, it acts as a dispatcher that selects:

which flag segment to process,

which key to use,

and which ciphertext buffer to decrypt.

The actual decryption logic is delegated to a shared worker function.

11) Opcode 16 — Segment Dispatcher

Inside ‘sub_401DE0’, opcode 16 contains a nested switch that selects the segment index (0 → 4) and maps it to a key pointer and a ciphertext pointer.

Conceptually, the logic looks like this:

1
case 16:
2
    switch (segment_id) {
3
        case 0: key = 0x4A4080; cipher = 0x4A4100; break;
4
        case 1: key = 0x4A4040; cipher = 0x4A40F0; break;
5
        case 2: key = 0x4A4000; cipher = 0x4A40E0; break;
6
        case 3: key = 0x4A3FC0; cipher = 0x4A40D0; break;
7
        case 4: key = 0x4A3F80; cipher = 0x4A40C0; break;
8
    }

12) Key Extraction

Each segment key is 64 bytes long, stored contiguously in .rodata.

Example — Segment 0 key:

Address : 0x4A4080

Length : 64 bytes

Hex dump:

c0e97a6b15beb64963e3d94e3cd05c3f

7f030d57081f474d39417b0857216480

e74a79069dadc385b8ca87cf46e8269a

6c5f10b5195f6007e8d8821e18bedbf1

13) Ciphertext Extraction

Each ciphertext is a short fixed buffer (10–11 bytes), also stored in .rodata.

Example — Segment 0 ciphertext:

Address : 0x4A4100

Bytes : 7c 02 d6 e9 77 16 7e aa 63 c0 d5

Length : 11 bytes

14) Final Mapping

1
KEYS = [
2
    KEY_0,  # @ 0x4A4080
3
    KEY_1,  # @ 0x4A4040
4
    KEY_2,  # @ 0x4A4000
5
    KEY_3,  # @ 0x4A3FC0
6
    KEY_4,  # @ 0x4A3F80
7
]
8

9
CIPHERS = [
10
    CIPHER_0,  # @ 0x4A4100
11
    CIPHER_1,  # @ 0x4A40F0
12
    CIPHER_2,  # @ 0x4A40E0
13
    CIPHER_3,  # @ 0x4A40D0
14
    CIPHER_4,  # @ 0x4A40C0
15
]

At this stage:

No patching was required

No brute-forcing was involved

All secrets were recovered statically

15) Final Trick — Deck-of-Cards Reassembly

The flag is not concatenated.

It is reconstructed using modulo-5 interleaving:

1
seg0[0], seg1[0], seg2[0], seg3[0], seg4[0],
2
seg0[1], seg1[1], ...

1
import hmac, hashlib, struct
2

3
KEYS = [
4
    bytes.fromhex(
5
        "c0e97a6b15beb64963e3d94e3cd05c3f"
6
        "7f030d57081f474d39417b0857216480"
7
        "e74a79069dadc385b8ca87cf46e8269a"
8
        "6c5f10b5195f6007e8d8821e18bedbf1"
9
    ),
10
    bytes.fromhex(
11
        "bd849882bd87b854aa80dfd3c01a95d4"
12
        "14bd1c2b9057c67e966a1c90ad4d1a86"
13
        "89cbd5fd65fe8b739a4c131032b0ceb6"
14
        "b501130de0455d440ebd5ff2dd441090"
15
    ),
16
    bytes.fromhex(
17
        "514d7b0a3a05795b34a847346fe35674"
18
        "dd6dc53cdac7efb3b493b06bdb06a092"
19
        "4cb1e1a532bc1d4efb14b3a73db32c88"
20
        "e990733745afa0488e92fde58b79da4f"
21
    ),
22
    bytes.fromhex(
23
        "5652b6fee939b5ebae2ed5b29ee842d0"
24
        "d9408339a27a0ecd9b20c202ae0d6f96"
25
        "a014e94f539d03c627514209ef0a783c"
26
        "234e0cc680e1b96971947e91905871ff"
27
    ),
28
    bytes.fromhex(
29
        "9429d0e16d31e060849f7e7e1e086953"
30
        "9dde5d5e3dcc5d80d0da883e39a2bf06"
31
        "29bfd69fe102536a6455b0f9091b3b7a"
32
        "62984c217e7884ee290798e8b99f50ee"
33
    ),
34
]
35

36
CIPHERS = [
37
    bytes.fromhex("7c02d6e977167eaa63c0d5"),
38
    bytes.fromhex("e0451259cdba4262fe24"),
39
    bytes.fromhex("e7d180dba2d5f1a11661"),
40
    bytes.fromhex("de7e988ed4adfeb9f75c"),
41
    bytes.fromhex("ada457971978ce237e29"),
42
]
43

44
def poly(i):
45
    return (i*i + 7*i + 13) & 0xFF
46

47
def keystream(key, n, endian):
48
    out = b""
49
    c = 0
50
    while len(out) < n:
51
        if endian == "little":
52
            msg = c.to_bytes(8, "little")
53
        else:
54
            msg = struct.pack(">Q", c)
55
        out += hmac.new(key, msg, hashlib.sha512).digest()
56
        c += 1
57
    return out[:n]
58

59
def decrypt_one(key, ct, endian):
60
    n = len(ct)
61
    ks = keystream(key, n, endian)
62
    a = bytearray(n)
63
    for i in range(n):
64
        a[i] = ct[i] ^ poly(i)
65
    b = bytearray(n)
66
    for i in range(n):
67
        b[i] = a[i] ^ ks[n-1-i]
68
    c = bytearray(n)
69
    for i in range(n):
70
        c[i] = b[i] ^ ks[i]
71
    return bytes(c)
72

73
def assemble_mod5(segments):
74
    out = bytearray()
75
    m = max(len(s) for s in segments)
76
    for i in range(m):
77
        for s in segments:
78
            if i < len(s):
79
                out.append(s[i])
80
    return bytes(out)
81

82
def run(endian):
83
    segs = [decrypt_one(KEYS[i], CIPHERS[i], endian) for i in range(5)]
84
    raw = b"".join(segs)
85
    deck = assemble_mod5(segs)
86
    for cand in (raw, deck):
87
        if b"nexus{" in cand and cand.count(b"{") == 1 and cand.count(b"}") == 1:
88
            try:
89
                print(cand.decode())
90
                return True
91
            except:
92
                pass
93
    return False
94

95
if not run("little"):
96
    run("big")

flag : nexus{t4k3_taa4t_4dv444nt_33ge_4nd_st000p_cry11ngg}

would like to extend my sincere gratitude to N!L for designing this exceptional challenge. I look forward to participating again next year