If my undergrad years there are any indication, it really depends on the subfield and the professors. Some of the nicest and warmest professors are there along with the famous and hands-off faculty.
And most grad students I've spoken to seem to be quite content with the department itself. I suspect that the overall stress of Columbia (apparently ranked #1 stressful college) and NYC in general tend to create most of the problems.
I am in a similar boat. Both of the POIs at both schools are very welI matched in terms of research, which is amazing since it's a very narrow field and I've been thinking of ways to differentiate between the two. I had a long conversation with one POI over the weekend. What I found really helpful was asking her questions about funding international travel, asking how she advises her grad students (how often, what sort of schedules, etc), where recent grads of the field are and general questions about interdisciplinary work and working with other faculty members (I was very pleased to hear about co-advising).