x86 assembly on OSX

During last terms compiler design course I learned x86 assembly "the hard way"... Here's some stuff that I "found out" while developing x86 assembly on OSX 10.10.

Read this first

I'm writing this post a couple of months after the course. I did not lookup the instructions and I'm justing writing what I can remember. It's therefore possible that some instructions below are incorrect. Better look them up yourself instead of copying from here...

Tools

I used gcc (compiler/assembler) and gdb (debugger). Place your assembly code in a .s file and assemble it with gcc -m32 -ggdb yoursourcefile.s -o programname. gcc is useful to have a look at generated assembly code too. Just write a c programm and then compile it with the -S flag to generate assembly output.

Stack alignment on OSX

There's this sentence

The stack is 16-byte aligned at the point of function calls

in the IA-32 Function Calling Conventions that gave us in the beginning so much pain. What does it mean? Before calling anything make sure that the stack is aligned to 16 bytes. The annoying thing is, that inside each function there are already four bytes on the stack (the return-address). After the normal function prologue (enter / push %ebp) there will be 8 bytes on the stack.

Example

.globl _main

_main:
    enter $8, $0                     
    # stack is now 16 byte aligned. 
    # 4 + 4 + 8
    # return-address + old %ebp value + 8 bytes because of "enter $8.."

    leave ret

So, make sure to always increase the stack size by a multiple of 16. The hard part was, that the alignment is only enforced when using systemcalls. In the beginning our programs jumped around a couple of times with a misaligned stack without any problems and then just blew up when printing or reading something. Quite hard to debug but very educational... (learn the hard way indeed).

Some tricks

Below are just some tricks I learned. If you've programmed assembly before you'll be bored.

; inside your assembly func you should backup the ebp
; either do
push %ebp 
movl %esp, %ebp

; or use the enter instruction:
enter $0, $0

; after the steps above there will be 8 bytes on the stack.
; align it with
subl $8, %esp
; or use enter $8,$0 

; because we set %ebp as the first step, we can now use it to 
; address stuff in our stack frame. 
; some examples:
movl $4, (%ebp)    ; write 4 into the stack address where %ebp points to 
movl $5, -4(%ebp)  ; write 5 into the address four bytes below
movl 8(%ebp), -8(%ebp) ; move the value of a parameter (higher stack address) into a local variable (lower stack address)

; to create local variables, just make space by reducing the esp
subl $4, %esp     ; create a 4 byte "variable" on the stack 
movl $42, 4(%esp) ; write something to this variable
addl $4, %esp     ; free/forget the variable

; instead of subl %esp you can also: 
pushl $42         ; decrease %esp by 4
popl %eax         ; read the previously pushed $42 to %eax and increase the stack pointer again

; set eax to zero (eax is used as the return value when exiting the program)
xor %eax, %eax

; no example but a tipp: checkout the leal instruction! it is useful for example when using scanf and writing to the stack

; there are instructions for leaving your function
leave
ret