Angular directive and dynamic templates: where does the controller logic go?

Here's the situation: I have entities that have many different properties: scalars, arrays, references to other entities, etc. Each property type can be displayed differently depending on the action (create, edit property definition, edit property value, view). Each property is treated equally by the entity and is usually rendered by iterating array of properties and creating a generic property directive.

The property directive checks property type and desired action in linking function, fetches the appriopriate template and compiles it. Here's some pseudo-code:

myModule.directives('property', function() {
  return function(scope, element, attributes) {
    var template = fetchTemplate(scope.propertyType, attributes.action);
    element.html($compile(template)(scope));
  }
}

Where should I put the controller logic? Currently I define a dedicated controller for each property in the template via ng-controller and put that controller to the application's controllers file. Yet I feel the logic should go into the directive itself. Here's the catch: I cannot use directive's controller function as the template is not ready by then. Should I handle all the logic for different property types in the same link function that fetches template? Or should I define directives for each property and then declare those directives in property templates? (This one seems doesn't seem right at all because a directive loads a template that declares another directive to do controller stuff).

What is the angular way of achieving this with dynamic templates?