That's actually really shitty on that professors part. You have a really hard decision and visiting schools is an important part of the decision. That response would have riled me up. I would respond back with something along the lines of asking "If I say yes to your visit, am I your top applicant?". If he says yes, then maybe ask the other school for an alternate date (assuming this professors school is higher on your list). If he doesn't say you are his top choice, then I would go to the other school for a visit.
This is also assuming you think you have a good shot at both schools (be realistic in your thinking). If one is a stretch, and the other is more realistic, def go to the more realistic one (imo of course)
School:Kent State
Track and/or specialization within psychology program: Developmental
Interview type:Skype; then invited to in person
Date:Feb 9th- 10th
School:West Virginia University
Track and/or specialization within psychology program: Life Span/Developmental
Interview type:In Person
Date:Jan 26-27th
Have not heard back from Mizzou, Univ. of Utah, North Dakota State University, Alabama or Virginia tech but I wont start worrying until the end of January.