Even if you switch to theory, you still need to understand basic statistical concepts to have a decent conversation with your colleagues. They are not that hard. Many people in polisci taught themselves stats. Nowadays you can ask AI to teach you everything from how to calculate the mean to DiD.
People switch subfields all the time, but not doing well in Quant classes should not be the reason for switching to theory. Do you have a political theory research question? Do you have a good grasp of the canon? Do you read theory journals?
Are they supportive? Do you understand what they are working on? You should talk to them about your interests and see if you can work with them. This should be the decisive factor, not your performance in quant classes.