LoTR is pretty much the only fantasy series in my collection that has elves or orcs in it. It's common to characterize fantasy that way but the best stuff is pretty unique. How do we give fantasy the additional label of being literary? Perhaps it's just the books that have received the most acclaim over the decades. It may be mischaractarizing modern fantasy to conflate it in with YA or LoTR derivatives. If you aren't familiar with them yet, here are some modern fantasy series that certainly aren't YA or overly derivative: The Stormlight Archive, The First Law Trilogy, Farseer Trilogy, Discworld, The Lies of Locke Lamora, The Prince of Nothing.
I think one has to be careful of treating readers with contempt. If you're of the view "they'll never understand my writing", you're boxing yourself in. Write what you want to write, and if it's any good, there will be an audience.
