I'll keep looking if there's a cleaner way of doing it, but what I do to unprioritize is to delete any prioritized game from my collection, and then add it back in. When it's re-added, it's no longer prioritized.
Mind you, this can be a little bothersome if you've prioritized three dozen titles...