I think others have definitely covered what I think about grad/ug relationships (not being in a role of authority over, fit within norms of department/school, doesn't violate any rules or statutes, etc)- my husband and I met while he was in grad school and I was in undergrad at the same university (may have talked about this on another thread).
If there was a chance he could be a TA for me, I probably wouldn't have gone for it because of the potential conflict of interest and how it would look to professors/other students. Even with him being in a totally different school within the university (engineering/compsci v. arts and sciences) some people still threw a hissy fit because there is still a hinky sort of perception. To them, it didn't matter that we were less than 4 years apart and the only classes we ever had in common in ug and grad school were the university required classes that everyone going through the ug had to take- and we took them several years apart (he did ug at the same school). Rather, they saw a more advanced student in a superior position potentially taking advantage of a younger, inexperienced student.
Sure you could say that you don't care what other people think, but I don't think that is a wise position to take, especially in academia. Some concerns can be soothed by proving the contrary, but you still need to be careful in this sort of environment.