snowbound

hmm, that is a tough decision then... 
I studied environmental studies and international relations in undergrad. I always felt that I knew a little about a lot, but ultimately lacked any tangible, technical skills to get a substantive job in the field beyond an being an assistant. Hence why I've returned to school for more specific professional skills. 
I've tended to stay away from energy since my brain just can't comprehend it as well, so take my opinion with a grain of salt. I find that Duke's energy focus is more geared towards energy sector jobs and the business/economics behind it. So, completely different skills than what you would've gotten in undergrad. On the other hand, Yale to me seems like while you would still get the economic/business/tech side of energy, you're more likely to end up on the policy end. I feel like Yale looks more at the impact of energy on society, whereas Duke would make you a successful energy sector player (like working for GE). If you're thinking energy in a green building sort of way, I don't think you can go wrong with either program. All of that being said, this is only my opinion on a focus that I'm not at all involved in beyond a short stint in sustainable building. Probably not very helpful at all. 
I know some Yale grads that are very successful in international climate policy, and others that went into sustainable business and green building. The only Duke folks I know follow more along the enviro law lines, but that's a small sample. 
Lastly, HUGE congrats on getting into both Yale and Duke... that's an achievement you should be so proud of!! Don't stress. You've hit success, now pick what'll make you happiest. 

What area are you hoping to focus in? What undergrad experience do you already have? Both schools offer similar specializations (to some extent), but certain topics may be stronger at one school versus the other. Either place you'll be proud to say "I went here" ... just make sure you get the skills you're paying for. 
I'm someone who has moved from major cities to rural towns (with a population of under 1,000) and everything in between. Durham may seem small (it's really not!!), but it has the amenities. Also, it's warm there. Yes, New Haven is a city... so, it some perks. If this week is going to be your first time in New Haven, definitely drive around and get a feel for the neighborhoods. It probably won't give you the warm fuzzies compared to Durham.  (US News ranks Raleigh/Durham as #7 Best 100 Places to Live, and New Haven comes in at #81). Don't be afraid to try a new atmosphere and new experiences! And definitely do your research on what neighborhoods you can afford, what those apartments might look like on Craigslist, street view it, and see if you're comfortable walking at night (thanks, mom). 
I'm interested to hear how you feel about "fit" with the students at Yale vs Duke. I used to not think "fit" was a big deal until I didn't "fit." It really does keep you going.