AngularJS ng-include does not include view unless passed in $scope

Is it wrong to assume that ngInclude can take a raw path? I keep trying to set my ngInclude as follows:

<div ng-include src="views/header.html"></div>

This does not work but if I do something like this it does work.

// HeaderController
app.controller('HeaderCtrl', function($scope){
   $scope.templates = {[
     template: { url: 'views/header.html' }
   ]};

   $scope.template = $scope.templates[0].template;
});

In my index.html

<div ng-controller="HeaderCtrl">
  <div ng-include src="template.url"></div>
</div>

Does ngInclude only except values off of the scope? If so why is it this way and not a straight include of the html partial.

Thanks