This is my solution for the Windows RE Extreme challenge from the 2023 USCG Combine. It was one of the first reverse engineering challenges I’d ever completed.

Challenge Description

For the extreme-level challenge, submit your walkthrough (or link to it) as well as the solution. The judges will review submissions at least once a day. Good luck!

Summary

In this challenge, the objective is to reverse-engineer a binary to extract an encrypted flag. The program employs RC4 encryption, multithreading, and various anti-debugging techniques. The primary goals are to patch the win function to decrypt the encrypted flag — instead of encrypting the input — and to modify the encrypt function to bypass anti-debugging checks. After applying these patches and debugging the program, the decrypted flag becomes visible on the stack.

Understand the Program

Note: Some function and variable names have been changed to enhance the reader’s understanding of the program flow.

Important details for main:

• Lines 6 and 7 enable multithreading in the program
• Line 8 and 9 are anti-debugging techniques
• The function requires a command line argument

Important details for win:

• The encrypted flag is stored in enc_flag
• The for loop on line 69 reads the values from argv[1], and stores each byte of the flag into the local variable v2 (line 70), and each contiguous variable in memory
• The flag is encrypted with the encrypt function

Important details for encrypt:

• On line 39, the RC4 Key-Scheduling algorithm is ran.
• During the for loop starting on line 40, the RC4 Pseudo-Random Generation Algorithm is ran 8192 times.
  • Note that there is a sleep(0); function call each loop iteration, this is another anti-debugging feature.
  • Line 42 checks if the result of the getTickCount(); call equals 1 (another anti-debugging technique)
• On the for loop starting on line 48, the flag is XOR’d, resulting in the encryption of the flag.

Patch the win Function

After doing some research on RC4, I discovered that encrypting a RC4 encrypted string is the equivalent of decrypting it, due to the nature of the xor function. We need to modify the win function to use the enc_flag array, so the program decrypts the flag, instead of encrypting our input.

Here is the corresponding assembly code: Modifying the loop in main to use enc_flag can be accomplished by patching the bytes with IDA’s Assemble feature.

First, click inp_flag in the assembly view. Then, go to Edit—>Patch Program—>Assemble (Note, you must use a lea instruction, as you need to reference the memory address of a local variable, not the actual value stored in [ebp+enc_flag]).

The disassembly should look like this:

Patch the encrypt Function

This loop in the encrypt function is causing us some issues: As a reminder, the result of the getTickCount() function is put into tick_count_1. Since debugging a program usually results in this value being much larger than if it ran naturally, we need to patch the for loop to skip checking if dword_4244C4 is 0.

This can be accomplished by modifying the highlighted jz instruction. Change it to unconditionally jump after the cmp instruction is ran:

Edit—>Patch Program—>Assemble

Your disassembly should look like this:

Finally, apply the patches we made.

Edit —> Patch Program —> Apply Patches to Input File

Debug the program

The main function requires a command line argument (argv[1]):

  if ( argc == 2 )
  {
    if ( win(argv[1]) )
    {
      printf("[+] You achieved level 12!\r\n");
      uExitCode = 0;
    }

Set the argument by going to Debugger—>Process Options:

Set a breakpoint on line 72 in the win function (right after decryption has occurred):

Hit F9 to start the debugging process

Looking at the General Registers, it appears that ESP is pointing to the flag on the stack:

Hit Shift+F12 to open strings to view the full flag:

Flag: flag_{8079b1c9a4844ecd9e5d33dc6642d779}