I err on the side of "no jeans" even when casual/comfortable attire was suggested. People are going to overdress for these occasions, and you don't want to look sloppy compared to them.
I'm not sure whether visit days are different than interview days, but all 8 of us prospectives were dressed formally for a humanities interview. Hope this helps