If you're coming equipped with references from Armstrong/Joselit/Wood/etc & a senior paper/Yale graduate seminar paper as a writing sample you should be more than fine. GPA is fairly negligible portion of the application, so I wouldn't worry.
In addition, it's not so much the brand name of the school as it is the professors within that school's department. As you've more or less insinuated, the field is fairly small, especially, for example, if you're looking at Modern/Contemporary. The reality is that the vast majority of scholars at top schools are going from have some sort of link to either the October crowd or T.J. Clark/Anne Wagner (throw in W.J.T. Mitchell, Crow, Nochlin, etc as you wish). I'm not as familiar with Early Modern/Medieval/Classical/Asian scholarship, but I'd imagine that the same applies to some degree. Yes, it seems somewhat incestuous, but since you've presumably worked with professors from those circles you have a head start.