I have read the AngularJS documentation doc page but I am still not clear on what exactly happens. How does the compiler run? I mean is the compiler a piece of Javascript that is triggered on page load to run and inspect the DOM. If that is the case then is there much overhead every time the page is loaded?
I also have read that you should never change the DOM inside your controller. Why is that and could someone give me a simple example of what I should not do.
You should do a walkthrough of the mobile phone tutorial on the site. The directions are clear and it will show you how to set up your app and where the proper code should be. As Arun said, DOM manipulation should be handled mainly in your directives. Controllers handle the logic, the template handles data binding and incorporates directives to accomplish DOM manipulation. As you work through the tutorial, you will start to see Angular as a different way of thinking.
As far as compilation goes, the index.html page is rendered, the scripts are then loaded, and then Angular gets to work looking for the attributes to include the view template based on the routing and controllers. The template is then parsed with variables bound and watched, and then displayed to the user. Of course, there is a slight delay, as you can see on http://builtwith.angularjs.org/ . On the top right, you see "75 neat things built with AngularJS". If you refresh the page, you notice 75 is replaced with a ? until the page loads (less than a second later). Honestly, unless your controllers and views are incredibly complex, rendering time will never be very long at all. Personal example, I am generating a reports page with 12 columns of data 144 rows long, by parsing and looping through a JSON object multiple times and running calculations and creating a new object, all in the controller when the template is called. The page appears blank for about a quarter of a second before the data appears, templated, formatted, and with the appropriate callbacks.
Again, try it out, see for yourself.