Geniuses are allowed to break their own rules. I agree w/ most everyone. Don't use them in dialogue. Use sparingly if at all. I used to love them and when I started to eliminate them, my writing improved.
I think you have that as the opposite of what you intended. It's in narrative that you need to be particularly careful with them. In dialogue they show you a lot about the character, and it would be entirely in order to have a characters speech heavily loaded with -ly adverbs, if they're the sort of character who would speak like that. They're likely to be the effusive type. "Oh, lovely, lovely lovely! I'm sure we'll get on divinely!" I bet you can already imagine the character.