Hi there, I am also only 20 years old. I'm an undergrad senior graduating a year early and am currently in the middle of my application season. I feel that professors tell students to take time off because either the student is not focused enough on his/her academic goals or the student needs to work more on his prerequisites (languages, tests, grades). How do your own professors react to you telling them about your gradschool plans? Do you already have programs in mind?
As to what grad programs think about younger students, if your application is is a strong one, they will definitely take you seriously as a young but ambitious applicant. I made a good impression on my interviews and no questions were ever asked about my age. My own professors had much respect for my motivation to apply.