Is it possible to create a directive for a dynamic template?

I would like to do something like this:

<ul>
    <div partial="note"></div>
    <div partial="controls"></div>
</ul>

Where the partial attribute specifies which template to use for the element. The directive allowing this to be possible would be like:

.directive('partial', function() {
    return {
        restrict: 'E',
        templateUrl: '/partials/' + attrs.partial + '.html'
    }
})

The only thing is, I don't know how to get that attrs object there where it needs to be. I tried injecting it in various ways, but couldn't get it to work.