Are you sure it has to be an auxiliary verb? I agree that "I've" doesn't work if it's the only verb in the sentence, but what you've said would mean that words such as "should've" were always incorrect, but it appears in several dictionaries. My take is that "shouldn't've" is perfectly good, although very informal, English. It's simply the result of applying two contractions. Having said that, it's unusual enough that it's difficult to read. So I'd avoid it unless you were really trying to draw attention to the dialogue.
Why is "should've" incorrect? "Should" (as well as "could" and "would") is what's known as a modal verb, and any modal verb + "have" can be made into a contraction.
Sorry , I've must have misunderstood your original post. I thought that you were saying that "have" could only be contracted if it appeared as an auxiliary verb, and that it wasn't auxiliary in "shouldn't've". Could you clarify what you did mean, please? It seems to me that it has the same role in both "should've" and "shouldn't've".
As an example, consider "he's not" vs. "he isn't." The former is an example of a contracted auxiliary, whereas the second one is not. Truth be told, I'm not 100% sure why this is the case. Maybe drawing a sentence diagram will make things clear. I don't know. When I get more time, I'll see if I can find any more info about this.