Not powering the card correctly could definately cause that issue.
Speaking of power, the 5770 is one of the lowest power usage cards on the market, so if your having power problems with it,... not much else is going to work.
I actually destroyed the power supply on our xps 8100 by hooking up a gts450 to it and modding adobe premiere CS5 to use cuda acceleration on its mercury playback engine. GPU + CPU + max power draw = poof. It was making all sorts of weird noises and the capacitors were vibrating so we replaced it. Took out the power supply and the network jack. Had to replace the motherboard and the ps -.-.
We replaced it with :
Newegg.com - SeaSonic M12II 520 Bronze 520W ATX12V V2.3 / EPS 12V V2.91 SLI Ready 80 PLUS BRONZE Certified Modular Active PFC Power Supply
that guy.
Personally with that machine I would be wary about going too big anyhow, just becuase of the poor thermals of the machine. Our i7 860 was running way too hot when we were encoding video.