Hi -
I'm a Baltimore resident and live in nearby Mt. Vernon. I love Baltimore and would encourage you to not be intimidated. The key is to have common sense (don't walk alone at night etc.). There is a lot to enjoy about the city and I wouldn't let fear determine your fate. Also, here's a blog that you may be interested in from a Hopkins student that had the same feelings as you on the outset but has grown to the love the area.
http://hopkins.typepad.com/kate/2008/12 ... pkins.html
Hope this helps.