Remove app from listing on Heroku

I was added as a collaborator to an app on Heroku. Now that the job is done, I wanted to remove it from my dashboard so that it doesn't litter it. So I went to app collaborators tab and removed myself.

The problem is, the app is still visible in my dashboard, and when I try to access it, I get a big flashing error "You do not have access to the app". How do I remove it from my listing?

Edit

Also, I don't see sleep indicators in the list. That hurts.