angular ng-if or ng-show responds slow (2second delay?)

I'm trying to show or hide a loading indicator on a button when a request is busy. I do that with angular by changing the $scope.loading variable when a request is loading or when it's done loading.

 $scope.login = function(){
     $scope.loading = true;
    apiFactory.getToken()
        .success(function(data){

        })
        .error(function(error){

        })
         .finally(function(){
               $timeout(function() {
                 $scope.loading = false;
               }, 0);
         });
 };

In the frontend:

<button ng-disabled="loading" class="button button-outline button-positive" type="submit">
Log in 
<span ng-if="loading" class="ion-refreshing"></span>
</button>

This works fine, but the loading icon (ion-refreshing) is shown for about 2 seconds, while the $scope variable is updated immediately. I tried $scope.$apply but that doesn't seem to be what's wrong here, the scope is updated just fine and immediately after the request. It's just the icon that isn't responding quickly enough.

Thanks for helping me understand this!