Just one person's opinion, but I really think it would be best to still go. I completely understand it is not somewhere you currently see yourself attending, and that you would rather give the spot to someone else. However, it is too late for them to replace your spot should you now decline. Also, it may be seen as disrespectful to the people and program who put so much time and effort into making interview weekend happen, especially the graduate student who offered to host you for the weekend. I would say it's best in this case to stick to the commitment you made and try to go and keep an open mind, as it is only fair to the program who kindly invited you among all the others they could have chosen to attend interview weekend.