Hey man, I must say that this is my third year in graduate school and I still feel inadequate every day. And I hate it. I went into Meteorology for graduate school even though my bachelor's was in mathematics. I had no background in the field at all except a few research projects I did during summer internships (which have absolutely nothing to do with class material) so it didn't help me at all.
My first semester of grad school was terrible. I had no idea of what anyone was talking about and I was always intimidated and afraid in class.'Classes had no structure at all and we had three textbooks but none of the Professors followed the order at all, so I couldn't prep beforehand (I don't think that would have helped me much though since the material is so complex when you're seeing it for the first time). I became very depressed as I saw that no matter how hard I worked I was not doing well in class because I just couldn't even understand the questions that were being asked during exams or the general lingo during seminar courses. Like you, I always disappeared from the building as soon as class was over because I just hated everything in general.
Everyone else in my classes had a meteorology background so they didn't seemed phased at all about anything, which was seriously irritating. Like you, I felt like I had the obvious disadvantage. I was always used to being the hard working top student in class during all my life up to grad school. It was incredibly frustrating to feel like the dumbest person on earth despite the fact that I know I worked 10 times as hard as anyone else. Professors didn't seem to understand the fact that it was hard for me because I had no background either. I was be afraid of asking the Professors anything because I just didn't want to be embarrassed by asking them dumb questions.
Now on my third year, I still feel inadequate to be honest. I feel that I have tons of knowledge gaps because the classes and instructors here just suck and don't care about teaching. They just care about their research and that's it. (that's top school Professors for ya). I originally wanted a PhD but not anymore. I'm tired of busting my hump for no purpose because no one besides my advisor appreciates my work. Grad school has pretty much made me a bitter person that just hates the field. So I really don't know if the "impostor syndrome" will go away at all because in academia you're always surrounded by people (Professors and students) that think they know everything and always need to open their mouth to prove their brilliant-ness. I really wish you good luck. At least you've been in the field for a Bachelors and can at least make some sense of what people are talking about. I think professors would be more understanding and willing to help you than in my situation, where I felt that the only questions I could ask would be comparable to what a freshman undergrad with no experience in the field would ask. You can at least ask your classmates since they were probably in your shoes once when they started their Masters. My classmates would have NEVER understood my situation because they have not switched fields.