Failure of a console is largely a subjective term ...
If you mean whether a console will have a disasterous end (say like the Dreamcast) the signs would be that the console manufacturer is losing a lot of money, third party support is fleeing the platform, and sales are amazingly low (probably under 15 Million world wide after 2 years). The canary in the mine would be the Madden Franchise ... We really don't know what the criteria is but EA is under contractual obligation to release Madden to every living platform (this is one of the requirements to getting the NFL licence exclusively); if a console didn't have a Madden game in the year that would imply that it was dead.
Beyond that, if you mean failure as in the late years of the N64 and Gamecube the signs would be slow sales in the begining that lead to third party support being (almost) entirely multiplatform games; this in turn will lead to slower sales and multiplatform games will begin to leave the platform.
