There is a fair amount of theory in CS and I think it would be difficult to jump into graduate level courses from an unrelated degree based on what I've seen and know.
I'm wrapping up my undergrad in CS this semester. The degree is often times about 45-55 credit hours at most schools, that's a good chunk of material. Yes, CS has a close relationship with math, but its mostly discrete based. It wouldn't be hard for a mathematics major with programming experience to jump into graduate CS (in some areas they may even have an easier time than CS majors).
Anyway, you seem like a very bright and motivated person so maybe these issues ultimately won't matter. I just think its good to have a healthy notion of the amount of foundational material there is. Understanding object oreintated programming, data structures, discrete math and algorithms with get you a long way. Maybe the CS GRE is a good way of prooving your ability?