I was in a very similar situation as you are now. In the end, I decided to take a year off between Masters and PhD. I have been working for the last year, and will now be starting a PhD this fall.
My main reason for taking a year off was that I wanted to work and pay off some student loans. That worked out well, and I now feel much more financially secure going into a PhD. But, even if finances weren't an issue, I think taking the year off was the right thing for me. It just gave me some breathing room after a very demanding Masters program, and a chance to clear my head, refresh, re-charge, etc. Stepping away from it for a year convinced me that academia was what I truly wanted to do. I think I might have burned out if I had gone straight through.
In terms of downsides, I would say that it's challenging to apply for a PhD when you are not a student. As a student, you are in close proximity to libraries, professors, campus resources, etc that can all be very helpful when applying. Not only that, but as a student your schedule is more or less flexible if you need to study for the GREs or prepare applications. This is not necessarily the case if you're working a 9-5 job in a non-academic environment. In my case, even something as simple as arranging transcripts or calling references became a major pain because I had to leave work early so I could get it done during business hours. Or, if I needed to look up a book or a journal article for my application, it became a major chore as I was no longer close to a university library.
But, these annoyances are in no way insurmountable. And, personally, I would still take the year off if I could choose again.