Learning how to reverse engineer software has enormous benefits. Whether you want to develop ROM hacks or unlock programming secrets when no source code is available, being able to effectively study and infer functionality from compiled binaries is an extremely useful skill. In cybersecurity, reverse engineering can be used to understand and uncover malware. Software developers can use it to become better programmers by learning reverse engineering to bridge the gap between their high level coding experience and what actually gets executed on the machine. I can’t speak highly enough about its usefulness, and it’s no wonder government entities like the NSA in the United States have developed their own reverse engineering tools.
You may have heard of Ida and Ghidra. Both are powerful reverse engineering software packages with helpful user interfaces and advanced features. These tools are great, and you are likely to bust them out for advanced usage, but they aren't fundamentally required for reverse engineering. You can get a lot done with a handful of command line tools, especially when working with smaller programs. Regardless of the tools you use, you will need to know how to disassemble binaries, find key strings, and use debuggers.
In this tutorial, we will be using command line tools to reverse engineers simple “crackmes”. The website crackmes.one is one of several dedicated websites where users upload executables for others to attempt to crack in order to help them learn how to reverse engineer software.
Many of the simple crackmes involve trying to determine a hidden password or key phrase that the program asks you for when you run it. Upon successfully entering the correct password or phrase, you will typically get a success message indicating that you cracked the code. These are incredibly useful when learning reverse engineering and are also just a lot of fun. We will be cracking Unix executables typically written in C/C++ or x86-64 assembly that run on x86-64 Intel machines. Naturally, you’ll need a computer with x86-64 (AMD64) architecture. We will be using a Linux Virtual Machine to contain our reverse engineering environment so it won't matter whether you are running Windows, MacOS or Linux on your host machine. We'll be creating our Virtual Machine (VM) with VirtualBox.
This tutorial is intended for relative new comers when it comes to reverse engineering. Hopefully you have some programming experience already in at least one language, but the more the better. It will be especially helpful if you know some basic C/C++ or a from of assembly language. Assembly language in particular is extremely useful for reverse engineering. You do not need to be a master of writing assembly code, but you should know what it is, and some basic commands. If you have never worked with assembly code, I recommend you check out our x86-64 MacOS Hello World assembly tutorial or watch Low Level Learning's excellent video for a beginner friendly 10-minute intro to x86-64 assembly on Linux. I would make sure you can first write, assemble and run a “Hello, World!” program in assembly for x86-64 before you continue with this tutorial.
The fact that we are downloading executables from the internet and running them as part of our reverse engineering efforts should make you at least a little paranoid. While the programs submitted to crackmes.one are reviewed by the owners of the site, you can never be too careful. And of course, outside of crackmes.one you may download software you want to reverse engineer without putting yourself at major risk. We’ll take the following precautions to get in the habit of good practice while reverse engineering:
First you’ll want to make sure you can create a new Virtual Machine. We will be using an Ubuntu 22.04 VM. Download and install VirtualBox if you don’t already have it. Also make sure you have an Ubuntu 22.04 ISO to use with VirtualBox. Then,
Here is a brief overview of the types of analysis and tools we’ll be using and that you can use going forward for reverse engineering on the command line.
Static Analysis:
Static analysis involves studying the compiled binary when it is not running. You will simply look at the bytes from the compiled binary and use various tools to extract information. The tools we’ll be using for this are:
Let’s start with the the first program by nutcake with the executable name “crack”. Run the program to get a feel for it: