What makes you want to do cancer immunotherapy? I did my undergrad thesis in cancer immunology, and I would say there are really only a handful of people at a handful of places doing purely translational work. Let's put it this way: do you want to do your PhD working on a topic like optimizing CAR T cells so they are 10% cheaper for the clinic, or do you want to do your PhD working on a topic like understanding why certain cancers elicit an immune response and why others don't. Don't get me wrong-- I agree that the field is hot, but because it's quite interdisciplinary, for young researchers it can be quite challenging. I'm one of those people who think that the purpose of graduate school is to not necessarily find the next Nobel prize-winning discovery, but to learn how to do good science.
It's unfortunate that your letters are not great (if they are what you believe them to be). You're right in that you can't change them, but you should also realize that they are the number one or number two most important part of your graduate school application. If you can take a year off to work in the lab and have your current PI write a glowing letter (specifically addressing how you have found your motivation and are a changed person), then it would be worthwhile to retake the GRE and reapply next cycle. Otherwise, I think IU is a good school to pursue your PHD-- there are great, reputable PI's there, and while there may not be a cancer immunotherapy center like one you see at Sloan Kettering or MD Anderson, you will find a lab that will teach you the fundamentals on how to think like any cancer immunologist. If you read up a bit on the history, you'll realize that most of the "giants" were trained as fundamental immunologists or molecular biologists and slowly migrated into the field after they had already established their own labs.
tl;dr You don't need to train as a cancer immunologist to become a great cancer immunologist, and the work you do as a PhD student may not represent your greatest contribution to science