When you say you're using it appropriately, what do you mean? I would imagine unless there's a good reason (maybe a common use of the ampersand like "P&P") it would be changed to 'and' in editing. That's if you go traditional. If you self-publish, do whatever you want. It's likely to be distracting if used frequently, though.
I'm not sure I could think of very many times to use it. It usually means a very strong "and." I use it only when I refer to multiple items as a single entity. "Alice went to sell girl scout cookies to Mr. and Mrs. Johnson." "Alice went to sell girl scout cookies to Mr. & Mrs. Johnson." To me, these two sentences have very slightly different meanings. In the first, I imagine Alice going to each member of the couple separately, in the second, I imagine her speaking to them as a single entity.
I'd say no, except for the sort of thing that @Tenderiser mentioned - an expression like B&B for bed and breakfast.
I agree that it shouldn't be used in a book unless it's specifically part of a name. & means and, just like + means plus and = means equals. You wouldn't use those in a book any more than you'd use lol. If you plan on self-publishing to kindle or some other means of text-to-speech, it's going to be spoken as "ampersand," which is very irritating to listen to. I've stopped listening to books just because they've had too many symbols. I even once started reading a book where the author used ='d for 'his strength equaled that of three men.' I demanded my money back. Just my humble opinion as a reader, but it would make me think the author was lazy and couldn't be bothered to type out the whole word.
You can do whatever you want to do however, I'd suggest typing out a chapter, printing it off, and reading it. If nothing else, have a friend read it and give his/her opinion about readability. Personally, I do not like seeing symbols in books in place of words. They're okay being used as @Tenderiser mentioned, but other than that, I really don't see where/how symbols in place of words is appropriate. Just MPO (my personal opinion).
