The use of XOR in assembly and shellcode


If we XOR a byte (or bit) with itself, we will get a null byte (or zero bit).

We will explore all cases:

0 XOR 0 = 0
1 XOR 1 = 0

This is a way to create null bytes in shellcode, without actually including the null values.

XOR EAX, EAX ( which has the code bytes 33 C0)
This command XORs EAX with itself. Whatever the contents of EAX are, in the end EAX will be 0 (all 4 bytes – or 32 bits – will be zero).

The previous command is better than
MOV EAX, 0 (which has the code bytes B8 00 00 00 00)
because the latter command contains a null byte (actually four null bytes, but even one would be disastrous), and we avoid placing null bytes in shellcode, because usually shellcode fills strings. Strings are terminated by null bytes, and we do not want our shellcode to end prematurely.

Also, the former command is smaller by two bytes, resulting in smaller shellcode.


If we XOR a byte (or bit) with a constant byte (or bit), and we XOR the result with the same constant byte (or bit), we will get back the original byte (or bit).

We will explore all cases:

Data   Constant   First XOR (Encoding XOR)      Second XOR (Decoding XOR)

                  Data XOR Constant = Result    Result XOR Constant = Data

 0        0            0 XOR 0      =   0            0 XOR 0        =  0
 0        1            0 XOR 1      =   1            1 XOR 1        =  0
 1        0            1 XOR 0      =   1            1 XOR 0        =  1
 1        1            1 XOR 1      =   0            0 XOR 1        =  1

In shellcode, we choose to XOR-encode bytes with a byte that does not appear in the bytes we will XOR-encode. This is because, although the XOR-decoding will work fine, the XOR-encoding will produce a null byte when XOR-ing with the same byte as the XOR constant. Since we do not want null bytes in shellcode, we avoid using a XOR constant that appears in the bytes we want to XOR-encode.


About Dimitrios Kalemis

I am a systems engineer specializing in Microsoft products and technologies. I am also an author. Please visit my blog to see the blog posts I have written, the books I have written and the applications I have created. I definitely recommend my blog posts under the category "Management", all my books and all my applications. I believe that you will find them interesting and useful. I am in the process of writing more blog posts and books, so please visit my blog from time to time to see what I come up with next. I am also active on other sites; links to those you can find in the "About me" page of my blog.
This entry was posted in Security. Bookmark the permalink.