• Dana Epp

How to approach reverse engineering ELF x64 buffer overflows these days

So this weekend TryHackMe released a new challenge room called DearQA. It's marked as an Easy room, and it didn't take much time at all for me to complete. However, I've seen a bunch of posts on the forum and on Discord of people struggling. That surprised me, and got me thinking...

The Internet is a treasure trove of information when it comes to buffer overflows and reverse engineering. However, so much information ages pretty quickly. YouTube has tons of great videos. Medium has tons of great posts. But on the surface, so much of it is based on 32 bit overflows.

Worse yet, few places show the art of reverse engineering using tooling like IDA that really can simplify things. So this post is my writeup on how I approached this room using freely available tools to figure out the challenge. In another post, I will show a manual way to approach the room for those who don't like using tools like IDA.

So here goes.

Step 1: Recon

So we join the room and are confronted with the task of downloading the binary, called DearQA.DearQA. The challenge also offers us a VM to spin up, and we are told that port 5700 is accessible. Let's worry about that later. Let's first reverse engineer the binary we got.

Let's start by figuring out what sort of file this is. Running file is always a good start...

So we have a 64bit ELF binary that hasn't had their symbols stripped. Let's check the security properties of the file...

Oh wow. OK, this isn't normal these days. This binary was built with all modern security properties/features turned off. You don't see this much in the real world anymore because the default compiler options enable things like stack canaries, position independent execution (PIE) and non-executable (NX) stack pages. This will make exploitation of this binary pretty simple if we can find any vulnerabilities.

Now that we have a good idea that this is a 64 bit ELF binary with virtually no security features enabled, lets figure out what it does.

Some people are brave and will just run it; that seems foolish to me as you don't know what you could be possibly detonating. If you want to do that, then use a throw away virtual machine that is air-gapped and being monitored for all activity. For me though, I'd rather disassemble it first and see what it does.

Enter IDA, a binary code analysis tool that does a decent job of disassembling, decompiling and debugging things. As I know many users on THM don't have IDA Pro, I will demonstrate everything in this article using the community edition of IDA Freeware. If you get deeper into reverse engineering though, I highly recommend you check out IDA Pro. It's a huge time saver.

Step 2: Disassembly

Start up IDA, and when prompted click "New" and select to disassemble a new file. Find where you saved the binary and load it up. When prompted, accept the defaults to load an ELF64 file...

Press OK and let IDA do all the work. You should come up to an IDA View with an assembly representation of the binary. In your main pane, you will see function main() broken out....

Let's take a closer look...

So we can immediately see some interesting bits here.

  1. There is a single variable called var_20 which appears to be allocating 32 bytes of memory on the stack. (20h == 32d for the non-programmers out there)

  2. There is call to scanf, a known unsafe C-library function that does no bounds checking. And sure enough, its dropping untrusted input given by the user right into that buffer where var_20 is pointing to. There's our buffer overflow attack vector.

I'll be honest. This is easy to see because the program is so simple. Normally I don't even bother to try to read the assembler right away if it's too complex. I like to go straight to the pseudo-code that IDA can produce through decompiling. Reading somewhat legible C code is much easier than trying to follow stuff in ASM. So hit F5 and decompile this thing...

Sure enough.... what we determined through the assembly code is clear as day in C code. A simple program that prompts for a name, stores the results in a 32 byte array and then echoes it back out. And is vulnerable to a buffer overflow attack.

TIP: One of the cool features in IDA is being able to rename variables and functions as you are doing your analysis. It will save it in an internal database, never altering the actual binary. Since we can see the array is storing a username, let's rename it so all the C and ASM code will now reference that, making our lives easier as we review things. Simply right click on the variable and select "Rename lvar" and rename it to "username"...

Much better.

OK. So we know we have a main() function with a buffer overflow... with a VERY limited size stack. Not that easy to play with. But fear not. If you look over on the left side of the console, you will notice a list of all functions in the binary...

Notice that above main() is another function called vuln(). Let's decompile that by double clicking on that function...

Boom! OK. This is pretty clear. The kill chain is simple.... input more than 32 bytes of data when prompted to cause a buffer overflow and manipulate the program to jump to this vuln() function so we can launch a bash shell.

Remember when we checked the security properties of the binary, and nothing was turned on? That will serve us well, because that means that defenses like PIE and RELRO aren't enabled, and that the addresses of the functions are statically defined. In other words, we can get the address we need to jump to by simply looking at the Exports tab of the binary in IDA to get the address of the vuln() function...

So the address of vuln() is at 0x0000000000400686. (Remember we need to be looking at 64bit addresses here).

We have everything we need. Just one last thing to work out.

If you recall from the original recon checksec reported we were on a CPU architecture of amd64-64-little. That means we have a little-endian processor. So when we write out the return address we want, it has to be reversed when we write out the bytes. In python, that will be represented as '\x86\x06\x40\x00\x00\x00\x00\x00'.

So let's now craft a payload in python to feed our binary with data to overflow the buffer, and manipulate the program to jump to vuln():


Unless you only have python3 installed. Booo. Hiss. I'll explain why in a moment. But first, let's step through the payload.

  1. Write out 32 'A' characters, filling up the buffer to its max capacity.

  2. Write out 8 'B' characters, smashing past the stack to get us to the location to overwrite the RIP register (64bit instruction pointer) with our desired return address where we want the program to jmp to.

  3. Write out the return address of the vuln() function, written in little-endian bytes.

OK, now back to why this didn't work for you if you were using python3.

Consider this output of our payload generation with python2:

Now check out that same payload generated with python3:

See the difference? Did you notice the C2 byte generated with python3? As of python3, the default encoding is UTF8, and it will not properly serialize the raw bytes out when you use print().

Ahhh the ugliness of Unicode.

So if you want to ensure the payload works correctly in python3, you should use sys.stdout.buffer.write() instead to ensure raw bytes are written as expected:

And with that, we have a working exploit for our binary. Now to throw it at the server.

Step 3: Exploit the remote server

Alright, with a working payload in hand, let's fire it to the server using netcat:

Success.... kinda. We did overflow and execute a bash shell, but it just hangs when we enter anything.

What gives?

When we instruct python to pipe the payload into netcat it does what we expect, and then ends. When it does, stdin and stdout and closed, leaving us in a hung state.

There is a work around.

If we call the cat command after the python command inside of brackets, it is treated as a subshell and will keep running. That means its stdout will stay open allowing the shell to continue to communicate back to us. So.... a small change and we have an interactive shell...

Pwnage! We have full shell, and we can grab our flag.


Binary exploitation (Binex) can be fun. It can definitely be a headache... but with binary code analysis tools like IDA, its becomes a lot easier to explore how a binary works when you can both disassemble and decompile the instructions to see how things work.

I hope you find this writeup useful the next time you have to look at a binary.