It seems that we are on the exactly same boat! Also admitted to Harvard EALC PhD program, my first choice, but don’t feel like attending the Admit Day. (Btw, did Harvard send everything including funding information by postal mail rather than email?)
As regard the cost, our department did indicate the possibility of travel funding, though not sure in full or in part.
I think the primary advantage of not going is that it saves a lot of time (and probably money). On the other side, you would miss the opportunity of talking to people there to get first-hand experience. I did visit another university because I happened to be in that city on a non-academic trip when I received its admission letter, and the visit was very helpful in forming a vivid idea of how life there would be.
I don’t think the decision of not attending will annoy the faculty or cause anything as serious, but I am also concerned how to decline the invitation politely. Also wanted to know how much Admit Day is valued in the US.