I pretty much feel the same way. I graduated with about 36k in undergrad debt about 3 years ago, and it's down to 29k now. I worked overseas for a few years, and saved roughly 20k while making minimum payments. I could have thrown more money at the loans, but I'd rather have that money for a) loan payments after grad school (they shouldn't be that high with IBR and the like, but just in case, I can absorb a few months of high payments) b ) potential relocation costs (I wouldn't be surprised if I ended up working outside of the country, and the last thing I want is to scramble for money or borrow from parents to get the tickets/apartment/etc.) c) emergencies.
I was fortunate enough to obtain some a not-insignificant fellowship, but I'm still looking at adding 60k~ (providing there are no TA/RA/additional fellowships) over the course of the next few years. Is it ideal? Of course not. However, it's just extra motivation to hustle for any TA or research positions that become available.