I wouldn't worry about that for a couple of reasons:
First, Yale is social sciences and humanities-heavy at the graduate level. Its very top-ranked programs are English, history, psychology, economics, political science, etc. The only similarly ranked programs are professional, such as law and art. Most people, whether of the social sciences, humanities, hard sciences, or professional variety, know this.
Second, there are some natural divisions between the social sciences and the humanities and the sciences (schools/department/lab locations are a big factor in this, for example), so there's less chance of that happening in the first place.
Third, Yale is a very relaxed place overall. The snottiness that one would automatically associate with the university just isn't there when people from different departments do interact with one another.
And lastly, depending on your subfield, Yale's sociology department is great. It has some of the best memory studies folk in the world, for example. It's small, yes, (ironically, mostly because one of our former Presidents who tried to gut the department - and others in the humanities and social sciences - to free up funding, space, etc. for the hard sciences), but it's very solid.