I've lived in some pretty dismal places, when I was married to someone in the military.
Twentynine Palms, California. It's literally one road, a few miles long, with a few fast food places and tattoo parlors. And the very tiny base. And a lot of sand. I ended up working for the Red Cross, making some lifelong friends, and dancing with a civic ballet company. It was awesome.
I guess what I'm saying is that sometimes you live somewhere that feels like the wrong place for you. Chances are, as an academic, this will not be the last time you're faced with this sort of situation. So my advice is that if you can't be happy, be as happy as you can. Sometimes going into a situation prepared for it to utterly suck is a blessing in disguise, because you start to find yourself pleasantly surprised at every turn.
If you don't mind saying, what are your geographic options?