I think location "should" matter as much as you want it to matter. Don't let other people tell you how to prioritize your life. Many academics, especially those who are already in nice established positions, tend to recommend to students to put their studies/career first instead of worrying about location. But I don't think this would be very good advice for me. Personally, I have to live in a big enough city to support a diverse population, especially for the availability of ethnic grocery stores and not feeling like an outsider. I also dislike living in college towns. I learned a lot of what I liked from my experience living in a place I did not like for 2 years during my Masters.
In making my PhD decision, I weighed all non-academic reasons (e.g. location) and academic reasons (e.g. research fit) equally. But that's just how my priorities go. I think too many students unnecessarily feel "guilty" for considering any reason other than academic ones and I think this should not happen!
