I just had a thought about how to make a compelling villain protagonist (or, even antagonist, now that I think about it a little more) You want them to succeed, just not at the thing they're doing. Does that make any sense or am I just rambling?
You get to explain how they got there. I'm a sucker about stories that have a main character that I hate. He or she is the scumiest of them all, but either early in the plot or at least later on, we get to see what were the circumstances that led them there. It's like you get conflicted about the result you want them to have. On one hand you don't want them to get what they want because they are evil, but on the other, you feel about their pitifulness and you hope that at least they won't get even more hurt upon their wrongdoings. You want redemption. You want them to get what they deserve but hopefully as a inevitable lesson and not as a cruel, unjust punishment. Furthermore, sometimes it's the bad guys that liberate you in a story. The instincts and mannerisms that you have drowned in order to have it more easily, that they are not afraid of throwing it in your face because well... they are the rebels, in one sense.
I really like this. I've subconsciously followed the same theme. My character actively fights (and references) pride, envy, gluttony and greed in her quest for moral righteousness, and this quest of trying to be morally supreme is actually her biggest flaw and falls back into pride. Also, in the same vein of having my 'good' characters given bad traits, I've given my bad characters good traits. They're evil scum, but you kinda feel sorry for them.
end of the day people are complicated - hitler liked dogs - no one person is all good or all bad, so believable characters generally have a bit of both , if your hero is so good and brave and strong that he has no flaws whatsoever then most readers won't associate with him , if your bad guy is evil incarnate you risk him becoming a parody of himself
It's kind of a broad question to ask. I guess for me good intentions is the key to making characters likeable. If the character is working towards or internally desire to achieve a noble goal but their flaws get in their way (without causing irreparable harm to others) , then I think they'll generally come off as likeable.
One possibility is to create a sympathetic contradiction within the character. For example, John Smith is livid with white hot anger at his brother for being a paraplegic. John is an asshole for being unjustly angry, but he's a lovable asshole because, at the end of the day, he personally cares for his brother when he could easily afford a private nurse or a room in a healthcare facility. In other words, create a central internal conflict that drives the character's actions.
Insecurities, doubts, regrets, these encourage us to grow. Perfection could also be taken as a goal achieved. After perfection what does one do? Start the frig over! Lol.
I do not feel that a character like House on a television program can really be compared to a character in a book. Watching television is like watching clowns in the circus. The audience is not close to the character. There is no feeling of 'could I live with this guy', the reaction is more from a distance, as an observer, as one being passively entertained, and it is perfectly fine for a television character to be a really bad person, because he's purely entertainment. A reader is more vested, and more emotionally involved (if it is a good book). But the idea that a character has to be 'likeable' in a book or story - where in the world did that idea come from? That's just not true at all. To cling to such rules is a reliable guarantee of mediocrity. There is a story about a criminal who escapes from jail. He shoots a whole family of people. NOT ONE of the people in the story is at all likeable, and yet the story is incredibly mesmerizing.
Main characters do not have to be likable. Just look at Humbert Humbert. Or Ballas, from Ian Graham's Monument, who is about as thoroughly unlikeable as you can get. I tend to say that you should make them at least interesting, but I'm not sure Ballas is that interesting in and of himself - he's an unintelligent, belligerent drunk and all around jackass. But the book is good, so readers go along with him. It's all down to what you can pull off.
Oh, I dunno. Back in the day I used to wish that Thomas Magnum from Magnum, P.I., were a real person and I'd make up scenarios about how I would meet him, and . . .