I followed the advises given here, and wrote one paragraph, thanking the potential advisors for their time and for the hosting, and explaining I chose to go to Awesomeness University instead of theirs since it better suits my research interest. While one of the potential advisors was kind and answered nicely, the other potential advisor was quite unpleasant. He wrote: "It's plain to me that either I'm missing something big about what you plan to do, or that someone who's advising you in Where-I-Am-Right-Now has a shockingly different opinion of His-Department and/or Awesomeness Department than I have, but in either case I wish you well".
It was two days ago and I still feel crappy. It made me feel so bad. I considered his university mostly because of him, and had really good time during the visit... My advisor says I should take it as a compliment, reflecting how much he wanted to have me as a student, I hope that this is indeed the case and that it will not affect our possible relationship in the future, I would really like to work with him in the future if it would be possible.
Sorry for the long post, I had to share :-(