Great idea to compile all the lessons we've learned! Here's hoping it'll help next year's round of applicants. I was fortunate enough to receive interviews from all 5 programs I applied for, and offers from 4 (one waitlist). It's been a whirlwind, but an incredible experience! As a bit of context, I applied for Psychology PhD programs, specifically for Cognitive Neuroscience (memory). Below are my tips.
Applications
Choosing programs
As a starting point, speak to your current mentor (if you have one) about where to begin your search. Post docs and PIs have excellent advice about big names in the field, up-and-coming researchers, mentorship styles, and the inside scoop about the atmosphere of different institutions.
Only apply to programs that you would seriously consider attending. If you know that you would definitely not want to take an offer from a backup school, don't bother in the first place. I have friends who submitted a dozen applications, multiple to places that they weren't very excited about, just in the hopes that something would stick. When it came time for interviews, they realized that it was impossible to proceed with multiple conflicts, and they had to triage.
Application components
Letters of reference are SO important. I was lucky to have developed really fantastic relationships with PIs at my undergrad institution, and they apparently provided such high praise that it was repeatedly mentioned when I attended interviews.
My current mentor gave me excellent advice about writing my personal statement. Of course you should convey that you are passionate about your chosen research topic, but bear in mind that when everyone says the same sort of thing, claims of passion fall hollow. Instead, focus on highlighting your own unique accomplishments right off the bat, in the first paragraph. What are the key points of your past experiences that demonstrate that you have the necessary skills and enthusiasm?
I fretted over my GRE scores because the quant component was lower than I had been getting on practice tests. I considered re-taking, but ultimately decided not to, and I am very glad that I did not waste my time and money on another shot. It didn't pose a problem in admissions, and nobody commented on test scores at all in any of my interviews.
One program that I applied to "strongly recommended" the Psych subject test of the GRE, but I decided that I didn't want to bother taking it (again, time and money). Omitting the subject test didn't seem to hurt my application at all.
Interview Trips
Travelling
Always dress for your travel day in an outfit that you would be okay with wearing to the first interview event. At 2 of my 5 interviews, I experienced massive travel delays. I lost all the buffer time in my schedule and ended up going directly to the first event from the airport.
If the travel agent sends you an itinerary that you don't like (e.g., leaving very early in the morning after the end-of-interview party), you can just ask for a different one.
Travel-sized dry shampoo will save your life.
Would recommend bringing some melatonin pills. Personally, I am a night owl, and interviews required shifting my sleep-wake cycle by quite a few hours. Combine that with nerves, and it can be very hard to sleep without some help.
Interviews
There will be many profs interviewing you, not just those you mentioned in your application. Many of them will conduct research that is completely irrelevant to your interests, but that's okay. You do not have to feign interest in joining their lab. I suggest looking up their lab website blurb and the abstracts of a few recent publications. Try to do some theory of mind and imagine what sort of angle they would take when hearing about your own research projects. You can change the way you frame your research to match the sorts of questions and methods that they clearly favor.
The best possible preparation for knowing what to say in an interview is to have presented a poster on your research projects in the past. If you have that experience, then you have several versions (varying in brevity) of a walk-through speech about each of your projects. You'll also have learned how to deal with interruptions and questions. If you haven't presented a poster on your research, I would strongly suggest practicing telling that scientific narrative.
When you meet with graduate students, remain on guard! Remember that they are there because they care about what they do, and they want a peer who will be like-minded. At my home institution, I overheard graduate students in my lab discussing how an interviewee had made a poor impression by asking "fun" questions instead of sensible ones (e.g., "What are your favorite pizza toppings? What would your lab mascot be?"). The "middle-school sleepover" vibes made them think that she did not take the research seriously.
You may think that you have a "ranking" of your program choices in your head, but keep an open mind! I went into the interview process agonized about not knowing which school I wanted to attend. After my second interview, I was completely convinced that I had found the right place for me. I was wrong! Interview #3 changed my mind, but not until I had taken several days to process everything. I strongly encourage you to attend all interviews and treat each one as a serious mission to get all the information you might need, even if you think that you don't want to attend the school.
Other applicants will have impressive backgrounds, but try not to succumb to imposter syndrome. If they invited you to interview, you earned that spot. As I am still finishing up my undergrad, I was a little intimidated to find that the vast majority of other prospective students were around 4-5 years my senior. We had different experiences, but each one of us deserved to be there.
Making a Decision
If a program wants you, you will know. They will make that very clear with strong and swift communication, friendly follow-ups from POIs, messages from current graduate students, offers to answer additional questions, etc. Personally, I thought that such positive and prompt responses seemed to reflect a department that values its students and has a solid organizational infrastructure in place, both important things to consider.
You are in data collection mode until the end game. Don't worry if you're halfway through the process and you still have no idea where you want to go. Just keep on doing your best to learn more about each program!
You can get a dozen opinions and attack the problem with rationality (I initially tried making a giant spreadsheet with 20 criteria weighted by importance...), but ultimately, you should choose a program that feels right. That comes down to research fit, interpersonal dynamic with the POI, and potential future peers (Could you see yourself being friends, or are people competitive? Do you care?). Personally, I was looking for a program that was interdisciplinary and collaborative, PIs who were approachable and responsive, and peers who would be friendly and genuinely passionate about their research.