My undergrad institute has a joint-program with Columbia so I started as a pre-engineering freshman. In my sophomore year, I took a conflict class and it literally blew my mind. I thought the topic was so fascinating and fortunately, I got a RA job at Northwestern the following summer. That was the time when I became seriously interested in studying and doing political science. So I told my school that I was about to quit the engineering program -- but since I took so many classes as a pre-engineering student, I was able to meet all the graduation requirements and the registrar told me that I could in fact graduate one year early if I was no longer doing the joint program with Columbia. So I decided to graduate early that summer. And during the same summer, I took my GRE, and wrote 12 SoPs with the hope that I could still enjoy my life as a junior applicant (which did not turn out to be true. Last seater was the most hectic one in my undergrad career). In this sense, I started this poli sci Ph.D. game pretty late, but luckily everything worked out and I really appreciate what I've got today!
And to the physics student in a previous post -- it's never too late to change your track once you find your passion! Sometimes, people would even view your non-traditional experience as a strength.