It is a tough trade-off... Personally, I would hate to be in a position where I go to a relatively less prestigious department dependent on a handful of stars; if I do not get along with them, then I would be in a tough spot. Of course, if you end up striking a productive relationship, then it could be wonderful in helping your placements. I guess the campus visit would be crucial in gauging the kind of environment fostered by the department and knowing more about the individual faculty members.