It suddenly came up last week that I was short a reference on two letters, so I went down to the next professor on my list. She had previously offered to write me a letter, but I had declined since I already had enough references (or at least I thought I did) and she was comp sci and I was applying to math programs. Anyway, I sent her an email asking if she would still be willing to write me a letter and if she thought she would have enough to say about me in the context of the programs I was applying to. I waited a week with no response (I did try to see her in person in the meantime, but she was not in during her posted office hours) and then tried emailing her again. This time she replied with a very terse email telling me to look elsewhere for a reference. Apparently she took my suggestion that she might not have enough to say about my mathematical aptitude as some kind of personal slight against her letter-writing ability and was deeply offended by it. I doubt she would have bothered to reply at all if I had not followed up.
So that's my story, which I was going to give as advice to be careful what you say to your references. In reality, I guess it turned out to just be a vent, but it was a much needed one. In hindsight, if she didn't think that the mismatch in departments was a legitimate concern, she probably wouldn't have written me a very good letter anyway. Hopefully I can find someone better to take her place.