I agree. I submitted in another post that I'm willing to leave my SO of five years for grad school and I think 4 or 5 people after me said that I'm the biggest loser for doing so, and I am a disgusting person for making that choice...
I totally understand you on this. It's not like I would walk away and say peace, I'm out. Of course we would try to make it work but I'm not naive enough to think that it would work out for sure, I'm sure some long-distance relationships end up working out, but the majority of them don't. Just think about this, you never know what could happen, and if a couple years down the road you break up, you will end up resenting him for it, and yourself for choosing not to go to your dream school. Think of all the reasons why you applied to grad school in the first place...
Good luck and I wish you happiness in whichever path you choose.