Drinking and watching sports are not your only options. If you enjoy the outdoors, there are a lot of great opportunities to hike, bike, camp, etc. in and near by Athens. In first few months after I moved here, it seemed like every weekend I was going someplace new.
Non-exhaustive list of possible places to explore:
In Athens: Sandy Creek Park, Sandy Creek Nature Center, Memorial Park, Oconee Forest Park, Georgia State Botanical Gardens
Within a 30-40 min drive from Athens: Watson Mill Bridge State Park, Ft Yargo State Park, Hard Labor Creek State Park, Victoria Bryant State Park, Oconee National Forest
And then you have the Chattahoochee National Forest to the north with tons of other opportunities...