Yes, I know before I even post what the answer is, but as many of you have struggled with the same demon, and as I sometimes find that writing out a problem helps me settle upon a solution, I thought I would post this little bit of internal struggle. As I have mentioned elsewhere, I am currently involved in a project for which I hold out much hope for success, unlike any other project I have undertaken before. Because it is a historical, my writing is periodically interrupted with additional research, as points in the story require that previously unanticipated historical bits be fleshed out for accurracy. So, I have grown accustomed to stoppages in the creative outflow process. And then she just popped in there - a new character, completely unrelated to the historical, just sprouted out of my imagination without me even looking for her. Not only that, but she came with a difficult personal problem and chained to a mystery. Okay, I thought. Good idea, I thought. Jot down a few notes and put in the "future projects" file, then get back to the historical. So, I did. I jotted down some notes. Twenty two pages of notes. Well, it wound up being more like a chapter. And the start of another. Stop. I told myself to stop. Forced myself back to my research material. I'm making progress, but she keeps popping in there, slowing me down. I know. Must be disciplined. Exercise impulse control. "Impulse control is damaged, Captain. Could take days before we are underway again." Ack!! All right. Here's what I'll do. I will allow no more than four days to compile an outline of a future project. Got that, guys?? F-U-T-U-R-E P-R-O-J-E-C-T. Four days to outline it and then it gets locked away. Thanks, I feel better now.
You're lucky you have such self control. I obviously don't. My mammoth story, first draft had this character Hugo - mentioned as a name drop from the past who died from a brick dropped on his head. ( bizarre yes, but I was fourteen or so when I wrote the first draft and I wanted a strange sounding death, that could be summed up in sentence, yet still provoke questions. ) Three, on again, off again drafts later the entire book shifted, he is now the main character!
This kind of thing happens to me a lot, but usually with short stories. I get a new idea or a character and it interrupts the story I'm currently working on. As I've said before, I don't outline, so I make a note or two about the idea or character and pretty much force myself back to the original story. When that story is done, I revisit the notes I've made to see if the idea is as strong as it seemed to be when it first occurred to me. Sometimes (rarely) it is, and in its original form. Other times (also rarely) it isn't. Mostly, though, the time that's gone by since I first had the idea has caused it to change pretty drastically, and I see it through a new lens. It's almost like a new idea at that point and I have to judge it as it is at that moment. It's a sad thing, but I bet it happens to every writer, especially those who write long novels, because that's such a huge commitment of time. Who knows how many great novels have been lost because a great writer only had enough years in his life to write twenty books, but he had 200 good ideas? (Conversely, of course, there are writers who write 200 books, but we kind of wish they'd only written twenty, but that's another thread ...)
I love it and hate it when that happens. Worse yet is when while writing a story from one character's pov, another supporting character starts taking over. The need/impulse to switch back and forth gets more and more urgent until I find myself writing completely from the supporting character's view and they are no longer supporting; they are center stage and have stolen the show. That's probably why I have ten or eleven half-finished stories. Getting sidetracked is fun, but detrimental to my story lines.