These kind of problems can be generated by several systems. First thing I would check is the power supply, especially if it's a 20 pin instead of 24 (or possible fluctuations if you have power fluctuations in your neighborhood).

Does this have a DVI to analog converter? I haven't seen one cause any problems like this, but it's easy enough to change.
Also you can run some diagnostics on your memory an cpu easy enough from a boot CD like this.
http://www.syschat.com/download108.html
You could double check your BIOS settings, make sure they are not set to overclocking, and check the video settings. When you're in the BIOS see that the CPU or system is not overheating.