Hi!
I have some opinions - although not sure if they will be at all helpful.
I also applied for physics PhD programs and was accepted into Yale as well as another school not as highly ranked. To me the decision is a real consideration. The name recognition of Yale is certainly a positive factor - but honestly lower on my list of reasons for attending. I suppose I'm thinking of certain aspects as being weighted more than others. For me - the most weight is upon the research experiences I will be afforded and next my quality of life. There is a PI at each of these schools that I would love to work with but Yale has more physicists in that field (not AMO). So I think it would be better for me to go to the place where there is a greater chance to continue the research I love.
I interned at a national lab and the consensus amongst the physicists there was that it was better to have a PI that was newer, perhaps an associate professor over someone more established. The idea was that they would have more time for you, more direct contact (instead of dealing with post docs), you would learn and accomplish more and most likely would be publishing more papers. I think this is an important consideration when making your decision. If you are continuing with physics, I imagine it's better to be established and well regarded in your field than come out of a high ranking school. As I'm in the same boat as you (just starting grad school), I don't know for sure if this is true but I do believe the advice I was given by senior physicists.
I would say who you work with and your research experience trumps every other factor especially if you will be comfortable with the atmosphere of each of these respective schools. I am fairly certain I will be choosing Yale because (heavily weighted 1) it will be good for the kind physics I would like to do, (2) I think I will be more comfortable on their campus due to the diversity (physical and intellectual) of the students that attend, (3) they offer a better stipend and health insurance which is perhaps a bigger factor for me as I have a child, and finally (less weight) - yeah it's Yale. I guess my opinion take away is go to the place with the best research fit where you will also physically be happy. Mental health is important. It's a really personal decision and probably your social circle says to go to Yale because of the strength of its name. But it seems wiser to seek the advice of physicists in your field (maybe your recommenders). Honestly, you are VERY lucky! You have choices! Although I know these decisions can be agonizing.
Lastly, you will do great anywhere. I am confident that you will be able to survive the graduate expectations at Yale or UMD - the schools wouldn't have admitted you if they didn't think so. I wouldn't worry about that part.
Good luck!