Weather is about the same for both locales, with San Diego being slightly warmer on average. Both are located in expensive neighborhoods (UCLA - Westwood, Bel Air, UCSD - La Jolla), though UCSD is basically on the beach.
Both suffer from the typical California transportation problem, i.e. you need a car to get anywhere efficiently. But there are buses if you really don't want to deal maintaining a car. LA is a much bigger city, so naturally there is a lot more to do, concerts, festivals, etc. But San Diego is far from boing (Sea World, Legoland, Comic Con).
I was a UCLA undergrad in physics. It's a very pretty campus, just google it and you'll get shots of Royce Hall and Powell Library. The Physics department has a new building that's pretty slick. There's a lot of Jazz/World Music concerts on campus if that's your scene. If not, you're still in LA, which always attracts big name artists. As to grad culture I can't say much other than they seemed pretty typical of most departments. Most of grad students were chill, with only a few bad apples.
I'm also looking at UCSD for PhD. I have yet to visit so I won't say anything about the campus or the department.