I hope I can help you with this.
The F1 visa is a student visa and allows you to work ON CAMPUS for 20h/week and 40h/week during breaks and recesses. You're not allowed to work off-campus.
The J1 visa is an exchange visitor visa and can have different categories. But, since you're going to be a grad student, you'll most probably get the student J1 visa, which, as the F1 visa, only allows you to work on campus, not off campus. Moreover, many times if you have the J1 visa you'll be subject to the 212 rule (AKA two years residence rule), that obliges you to live in your home country for two years after the end of your program before applying for some US visas (I'm not sure which, though). Sometimes, especially if you don't receive any money from the US government or your own government, you're waived that rule, but you should make sure you won't be subject to that rule before getting the visa.
I think that's how it works, but if somebody knows more details about it, all comments will be welcomed!
By the way, I've been here with a J1 visa (not subject to the rule) and I'm going to get the F1 visa, to avoid all the possible restrictions.