It sounds like some of the posts on race/gender factor are somewhat exaggerated. Yes, it is certainly a factor, but this is not as deterministic as some comments here imply. This is a bit ridiculous hypothetical example, but just because an applicant is a female, low SES, and from a minority group, it doesn't mean that she will get into Harvard.There is no department that will admit an "unqualified" applicant just because of race and gender. There is no department who will "lower the bar" for those who are from a minority group. No department will ever, "give privileges" to those applicants based on race and gender.
You need to have strong credentials and relevant experience to apply to grad schools in first place. We will never know what really goes on inside the admissions committee, but generally speaking, the factor about your background never comes up in early stage of the process. That is, admissions committees usually look at GPA and GRE scores first, and reject those who do not pass their minimum standard, whatever that may be. Then they will probably look at LoR's, and make sure all three of your letters are strong. I've come across a comment that even one average-letter may put your application in a reject-pile because all applicants these days have strong letters. Then it will probably the statement of purpose and writing sample to determine who will be more fit to the program, and to discern whether your research experience can contribute to the department and if department has the right tools available for you to succeed.
At the last stage, this will be probably the stage where the factors on gender, race, SES, and personal background will come into play. But at this stage, though, there are TONS of other factors being considered, not just race and gender. And who knows which factor will top another? Besides race and gender, they may look at whether you have family members working at the institution (come on, applications ask us if we have any family/relatives working at the school, and they are asking for a reason), your outside experience, internship/volunteer/work experience even though they may not be relevant to the field, the school you went to, etc... This will be rather a rare case, but if your family was rich and donated $$ to the department or school in the past, that might be a factor, too. All these things are pretty much equally considered at the later stage, once they believe all the applicants who passed up to the last stage are strong applicants with good fit.
Are these fair? Hard to tell. But by the last stage of their admissions process, all applicants will have strong credentials, good amount of research experience, and a good fit. They just need to consider other factors to finalize their decision. Personally, I don't think these factors by themselves are good indicators of whether the applicant will be successful. But given that the race/gender is just one of the many factors, I don't understand the frustration particularly with race/gender, when the rest of other factors seem equally arbitrary to me.