I agree with 3dender. You sound like you are a pretty good student anyway, so your overall G.P.A would be good anyway, even if it is not as good as it would be if you did just psychology. To me, college is all about exploring and discovering interests and learning in detail about things which interest me. Sure, grad schools do like students with high G.P.A's. But, they are also interested in what motivates you, and how dedicated you are to pursuing your interests. As 3dender said, you can always get field experience to boost your phD app., but not using this opportunity will probably just screw with your head later on.