I was there for a year last year. The Yale Campus itself is fantastic and very safe. But once you go too far North for example, past the Ezra Stiles and Payne Whitney it gets worse very quickly. The area around East Rock is generally pretty safe and a nice neighbourhood.
As for being trapped in a Bubble, well there's not much to do outside the Yale Campus and the New Haven dowtown area. All the restaurants/bars are there. So even if the other areas were safer, it's not like you'd be routinely going there.
On the bright side, if you get really bored you can just take the Metro North to New York!
Edit: Also most of the grad students I knew didn't stay in HGS. The facilities aren't great and if you have to get the meal plan it's really not worth it.
