there's research on how your advisor is THE most important determinant of your success in grad school. If the school is very collaborative and you have interests in common with multiple faculty, then even with a bad POI, you can maybe work with other professors and make up for the bad POI. But if he's the only one whose research aligns with yours and he is really as terrible as the students tell him to be, DON'T (especially if you have other offers where your POI and others are more friendly and collegial).
Also, by terrible, what do they mean? If they simply mean he's hands off and if you are confident in pursuing your research ideas independently without your POI's guidance, then that's fine. But if by terrible, they mean he demoralizes, demeans, or sabotages his students' careers, heck no!
Another thing to look into would be how his ex-students are fairing. Are there students who have successful jobs (Academic or otherwise)? If yes, reach out and see if they have a different perspective on this POI. I say this because people sometimes love to blame everyone/everything (except themselves) for their failure but the experience could be very different for someone who's more self-driven and motivated.
ps2020 got a reaction from Aghhhhh in Interview Tips!