As someone who's spent the last few years in a department with very few women, and also even fewer people of color, who didn't initially think that these considerations would so heavily influence the norms of the department, I'm going to have to chime in here and simply say: diversity matters a lot! It's cool that the lower-ranked school is courting you so much, but (like in most relationships!) the initial courtship is not necessarily an indication of what's to come.