It's normal, though I'm a little surprised by one step of it.
When you switch from B to A, git sees that the file must be updated to match A's version, and does so silently because you've ignored it, saying that as far as branch B is concerned, the file doesn't matter. It has to do this - the only alternative is to refuse to check out branch A. (It would refuse if the file weren't ignored, saying "Untracked working tree file '' would be overwritten by merge". I'm actually surprised it doesn't do that in this case too. Anyone have any insight as to whether that's feature or bug?)
When you switch back to, git sees that branch B doesn't have that file, and removes it. Again, it has to do this. It has no way to read your mind and realize that that ignored file that was there a minute ago is one you want back - it simply gives you the contents of branch B, which state that the file doesn't exist.
As ewall suggests in the comments, if you want the file to survive these transitions, it needs to be tracked by all branches, or ignored in all branches.