If you think there is still a decent chance that you could change your mind and pick this school, then I would go. One of my "safety" schools is still in consideration just because they do offer some things that some of the larger (more impressive) schools do not. Like DarwinAG said, it is possible that you could change your mind about your top choice after visiting this school. Not having an offer finalized would make me a little worried too, so if you want to use this as a back-up, just in case something goes wrong with your offer, then that sounds logical to me.
However, if you are only planning on attending for the sole purpose of networking, then do not go. There are plenty of opportunities to network, and this is not one of them. If this is the case, then I would politely decline and try to keep in contact with that POI (as much as they would let me) for future opportunities.
Do you need to make a decision right away? Is the interview far off enough that you could wait a few days to see if your other offer is finalized before you give this school an answer?