I personally do not think average grades in multivariable calculus and linear algebra are so dramatic.
For what it is worth I got B+ in multi because I couldn't do the tricky double/triple integrals fast enough during the midterms. This did not affect my phd admissions but granted I had grad level math courses with As to make up for it.
Linear for some is also a tricky course the first time you see it. If I hadn't struggled with it before in high school I would've also gotten a B in my first semester linear class. I would say it is perfectly understandable to mess up the math classes in your first year if you didn't have exposure before.
" My grades in stats, machine learning and CS classes have all been A/A-'s and one or two B+'s. " <- that is very good.
The nice thing I notice about the United States is the culture of "second chances". The postdoc who helped me immensely to develop beautiful undergrad research also messed up his linear/multi and other math classes in undergrad. He said he had mostly bad grades during his math BS at UW. He stayed for a masters to fix his background, then did very well in his math phd and was able to secure a postdoc at duke with one of the best probabilists in the world. There are tons of similar examples from stat, including a stat professor I had at duke who climbed her way up from not prestigious undergrad and masters to phd at UFlorida, then postdoc at CMU, then tenure track at Duke. There is hope @L2norm! Many established and strong researchers were able to not let their "humble" beginnings define their future. The issue is that all that I know of are American citizens...so i can't really comment about internationals.
The question is: since your gpa is rather low but your math/stat/cs course grades are pretty good since you only have 2 Bs, it means you messed up most of your humanities/social sciences classes. Do stat programs really care about this?