How to see if you're going unstable with overclocking.

First of all, you have to see if your entire computer is unstable as a result of overclocking or if your video card is clocked too high?

If the video card is clocked too high, it shouldnt have too much of an effect on the overall stability. You will just start seeing artifacts and what not.

To test the stability of the entire system run SuperPi at 32M, Prime 95 for as long as you can, or OCCT as long as you can. The longer you can run these, the more stable your system is.