I think it does matter a lot how well connected your teachers/advisors are. This would definitely
affect your academic career. To some extent, reputation is very important. If you decide not to
stay in academia after you finish your doctoral studies, then the reputation of the school would
play a big role in your job hunting If you decide to become a full time academic, then the
reputation of your advisors and school in general still matters.
If you change your mind and want to go to the other school, you can totally do so. Just let them
know why you make this decision. Be polite and sincere, it should be fine I think.