Hi toad11, I'm an international student, so I'm not even sure what I'm saying, but conversations I've had with current PhD students iin the US always led me to believe rankings were not so important. That the most important part of choosing a PhD program is the relationship with the advisor...
What do you feel that is happening in your current program? Is there a lack of funding due to raking levels falling? Or are you worried you won't get a good job after you graduate due to the program's reputation?
Also, I've never heard of anyone changing programs in their second year, but you should prioritize your well-being. If you feel you won't get your reputation 'marked' for this, go for it!
My two cents is that if I made this switch in my country, dropping off one program and trying to apply to others, and if my advisor got offended, I would probably be frowned upon in my field because professors talk... But I would also never be able to continue in a program where I am uncomfortable and anxious/stuck. So even if I were to be frowned upon, I would reapply and change.