Thank you for this thread! I have been (over)thinking about the similar dilemma -- to visit or not to visit. I do think it is kind of rude to not visit; it looks like you're not even considering them, especially in my case I have also been in contact with very nice professors and their students. On the other hand I also feel guilty that they are paying (partially) for the trip.
So I talked to my current advisor about this, and he said I should still go visit: to avoid seeming rude AND to take it as a networking experience. The people you meet/contact at those schools are probably going to be attending the same conferences, working on similar projects as you etc. so it would be preferable to not "burn bridges". I also think the professors understand that this is a professional decision, so what I am planning to do is just make sure I send everyone I contacted a personal email when I have officially made a decision -- thanking them, saying hope we cross paths etc in the most sincere way possible.