Put another way, at what point does a story that is "based on a true story" become fiction? And as far as publishing is concerned, are there any vetting procedures in nonfiction to confirm the legitimacy of the stories? Could someone write a realistic and believable fiction but pass it off as nonfiction?
I've forgotten the name of the author and the memoir, but I remember a scandal when a "memoir" was lauded by Oprah on her talk show, and unbeknownst to her and her staff the book was fiction. I think she had chosen it for Oprah's Book Club as either the very first memoir of the book club or the very first non-fiction book of the club, but my memory could be hazy on that. I do remember that it was a memoir about addiction and rehab and coming back from all that, so addicts in recovery were hurt by it as well. After it was exposed as a lie, Oprah had the guy on to try explain himself to her and she was furious with the guy, and he was a smug asshole who just seemed pleased to have made money off of it and had become famous from it. He kept saying "It's a memoir, not an autobiography" as an excuse, which Oprah wasn't buying. So, that doesn't answer the first part of your question, but yes, it has happened.
The same happens with movies all the time. The "based on a true story" really lost it's meaning over the years thanks to Hollywood. About the legitimacy: I'm not sure if publishers take care of that. I remember hearing from a friend that the author places a footnote in the book for sources and the reader can go check if any wrong interpretation is on the book. He also said they cite the real people that inspired the story in a part of the book, so you can search for it yourself. But I don't know if what my friend said was true, so don't assume this is 100% correct.
Pretty sure you're thinking of James Frey. His memoir might have been meant as fiction, but he neglected to tell his publisher that, so the scandal was pretty well-deserved....
Yes! That was him! Thank you! And yes, the scandal was well-deserved indeed. I couldn't remember his name, only that I despised the asshole and his lack of consideration for anyone but himself. Given the subject matter, his readers were especially devastated by it.