It's supposed to be quite small but there also suggestions it's influence has been underestimated, particularly in the transition to Middle English.
I think I'd find Latin terms popping up in the middle of a book quite distracting. There are tools like Awkwords that are designed to take the tedium out of creating an entire language. Pick your vowels, consonants, nasals, and a basic pattern of them, and it spits out however many words you choose. Hell, Vulgarlang will even create grammar rules and irregularities for you.
I know this has been answered already, (so don't bother reading) but I'll let myself indulge for a moment- I made up a few languages, with semi-working grammar structures and lots of vocabulary, something between a naming language and a real conlang. Honestly, the only person who'll care is you, since it takes so much work. There are benefits to your own languages beyond just worldbuilding. If you build a culture around your languages, and write in the attitude of said cultures... Well, you might have some fairly good worldbuilding on that alone. But it's up to you. Writing won't get worse if you don't add a language.
