I know this feeling! I was wondering if I was overreacting so it is good to hear that someone else feels the same. I heard back from a school which accepted me into their M.S. program (I applied for Ph.D). While I am thrilled and totally grateful, the project/lab group just wasn't right for me. I feel terrible because this prof went through the trouble to email me, host me, etc but I know I wouldn't be happy there and there is no point in making myself unhappy to spare their feelings (they are in academia they should be used to rejection! haha). I mean I am just a potential graduate student, not taking me on isn't a career ender by any means.
I have come this far and I don't believe I should have to settle for anything less than a schools that is the best fit for me. Now I am drafting this "it isn't you, it is me" type email and I just can't find the words