That sounds exhausting.
I had a project like that, where the project advisor was, in my opinion, really bad at understanding why our relationship was not working. It was for very different reasons than your situation, I think, but it was very unhealthy and eventually, I had to explain via email and then in person why this was frustrating for me, and I ended up dropping the project. Given, he was not my main advisor, so I could afford to do that, but it's definitely tough.
Perhaps it would work for you to outline, for yourself, what you think a reasonable workload would be outside of personality problems. And then within that, sit with a good cup of tea and outline for yourself what kinds of current advisor behaviors would make that kind of workload unreasonable anyway. I think discussing the first part, if you think there is a workload issue, should be fairly simple and I think is easier to defend. One thing I do with my advisor, which I think helps with this understanding, is that we've kind of outlined how much time per week I should/will be working on different things. Then if he asks for me to do extra work, I can explain to him right then, in a matter-of-fact kind of way, what will be sacrificed.
The second part is more difficult. Do you know if other advisees are having the same issues?
