Except there are first person novels that do just that - switch to third person POVs at times. Even those where the original first-person POV narrator dies and the narration switches over. Some of them are pretty popular books, too, so it seems to me a significant number of readers aren't bothered by this so long as they author does a good job of it.