I was reading something on this written by a professor at U of Northern Iowa (here it is: http://www.uni.edu/~gotera/gradapp/results.htm). He says that, no, you shouldn't turn it down. But then he quotes a professor who says basically what the above posters have said.
I tend to think that, yeah, this is your life and it's a huge committment. You want to make the best decision. Also, (as the article mentions) the school has a whole line of eager and qualified applicants below you. So it's not like you're really leaving the program or grad director in a bind, especially because it's not April 15 yet. If you were to wait until after April 15 to turn them down, then it might be a problem. So I'd say just do it as soon as possible, so that the school still has time to extend the offer to waitlisted applicants and all that.
Anyway, congrats on the new offer!