I went straight from undergrad into a PhD program (I'm actually at USC Annenberg, so if you'd like to PM me I'd be happy to talk to you about it more). There were parts of it that weren't as difficult as I expected it to be and then there were things that ended up being challenges that came out of nowhere.
Part of the problem you have starting a PhD program without an MA (although I think this would be less of a problem if you're doing a combined MA/PhD) is that you just aren't anywhere near as familiar with the way academia works, especially at that level. There's also a huge adjustment in the amount of work and reading you're doing and how much of the work is self-directed that's much greater if you're coming straight from undergrad. It's totally do-able though, so if you know a PhD is something you want to do, I wouldn't let the fact that you don't have an MA scare you off from applying straight to PhD programs. What I might do, however, is apply to a mixture of MA, MA/PhD, and PhD programs, so that you've got options.