First of all congratulations on having the maturity to make a such a decision. There are many many people who have run off to top 5 or top 10 schools simply because of their name or ranking and have come to rue it later. The most important thing here is to make sure that you made the right decision for you.
As the previous poster so astutley pointed out, rankings not all they are craked up to be. Rankings are simply an easily codified way to compare the enormous specturm of programs that exist for any given course of study. But most of the ranking algorithms used by agencies like US News are based on statistical analysis, often comparing apples to oranges and failing to take into account what really matters to students.
However it sounds more like your concern is how your choice of school will affect your career. I think what your really worried about is the lack of WOW factor and it's a trap many people fall into. Yes if you go to a top 5, household name school, people's gut reaction will be to be impressed and think you're a genius. But that is as far as it goes, especially when it comes to finding jobs. As they say, the name on the diploma may get you in the door but it is what YOU can do that keeps you there. Employers want people with demonstrably marketable skills and real experience they can build upon. It is far better to be a stand out at a top 20 school with leadership relatable work experience than to be one of the crowd at a top 5 institution. Employers in your field will know the strengths of different programs and will know what kind of schools produce graduates who are trained in what they are looking for. This is what will matter when they hire you, what YOU did at school, not the name on your diploma.
If you picked a school that has a curriculum, alumni network, and career development services that are all geared toward what you want, where you can really show your stuff, you made the right decision. And hey a top 20 school is nothing to sneeze at!
Here is a link to how US News and World Report actually determines their rankings, which you may find enlightening.
http://www.usnews.com/education/best-colleges/articles/2012/09/11/how-us-news-calculates-its-best-colleges-rankings