Honestly, I think that prestige is something that applies more to groupings of schools rather than individually, meaning that if two schools are less than a certain "distance" apart in terms of rankings then it won't matter very much. I think Sinai and Weill are close enough together that the it won't make an enormous difference for your career prospects, and I have no doubt that you'll be able to find interesting cancer research being done in both places.
With that said, I think that the circumstances beyond academics are most important for you. If you believe that Sinai will make family life better, that's important. You mentioned the no-children housing at Weill, which makes me think that you want to keep the possibility of having a kid open in the future. If you think that will be easier at Sinai, I think you won't regret going there.