I've had a good amount of colleagues who have lived there. I think it's very much relative to your expectations... though that may go without saying. The lower rent does match the fact that the student family apartments are very small. At the same time, you could end up paying twice as much for something just as small and poorly maintained trying to find affordable housing elsewhere.
It's kind of a mixed bag, in my opinion. I've had friends complain about late night street noise (Yonge St.), bug problems on the bottom four floors, heat problems on the higher up floors, and sun problems (in its total lack if you have a north-facing apartment). Maybe my friends are just complainers. Haha. There's also easy access to the subway, a lot of great places to visit and explore in the immediate area, a daycare specifically for Charles Street Families (the value of which should not be underestimated! Daycare has years long waiting lists here sometimes!), and a community of other grad students and their families going through the same thing you are.
If you can easily afford to live elsewhere, I would do that. If you're anywhere on this side of "financially comfortable", Charles Street is a very reasonable option.