It all depends what you're looking for in a town, I suppose. In many ways, Charlottesville is well deserving of the myriad high rankings and awards bestowed upon it each year. The downside to this, of course, is that a lot of boomers are fleeing Northern VA for C'ville, which leads to an expansion in the suburban/planned communities at its edges. But the nucleus of the town--UVA and the "Corner" (the downtown area right next to campus) and the Downtown Mall area--are wonderful stretches of great local food, cafes, bookstores, bars, and the like. Charlottesville, like many similar towns, has a nice blend of affluence and hippy culture to give it a rich intellectual culture and liberal vibe, yet enough Appalachian culture to give it a folkiness. It's also nestled into the Shenandoahs, so it's beautiful year round. And while Route 29 (the major highway from DC through C'ville) is lined with strip malls, it's quite easy to live in Charlottesville and never use 29--you can basically make your life exist near the University and Downtown areas. Generally, if you're into a more "suburban," box-store lifestyle, you can have that. But if you like all things local--especially your food--C'ville is an excellent place to live.
The downsides probably depend on where you're coming from. I currently live in Cambridge, MA, so housing in C'ville is comparatively dirt cheap. The History stipend--compared to English--at UVA is measly, but I'd actually be making out better than I do at present because housing is that much cheaper. [if you live off-campus, it looks like www.brac.com is one of the better places to find housing. It is a bit better than Craigslist there because Craigslist is dominated by condo realtors in the area] Also, if you're used to living in a city and without a car--and if that's something you enjoy--that becomes more difficult in Charlottesville. It has a strong bus system, but if you bike, be prepared for mammoth hills throughout town. I suppose if your life could take place in a small area, you could do without a car, but unless you live very close to or on campus, you may need a car. It'd actually be a good town for a scooter.
Let me know if I can provide other specifics.