"But what really does deter me is that if I cannot get LOR from anyone on my committee, then the chances of my PhD application being successful is low".
First, how would anyone reading your LOR know that you could not get an LOR from any of your committee members? Secondly, how important are LORs anyway. Probably, not important at all. Do you think an admission committee member has ever read a bad LOR in an application package? No never, because LOR are all the same. Everyone writes and submits the same letter... John Smith is the best candidate for admission that ever existed. LOR are all BS..
Life is too short to let someone's opinion of your worth/value/intelligence deter you from going after your dreams. You seem like you are buying into the professor's assessment of your abilities. Stop. Do what you want and worry less. Just Do It.