Where can I draw the line between "flawed protagonist" and "unsympathetic jerk"?

In one of my projects (which I most likely will never release) I have a character who is a humanity-hating mass murderer who had a horrible childhood in which he was abused, bullied, sent to mental hospitals and arrested repeatedly for things he did not do. He suffers from Asperger syndrome, PTSD, major clinical depression and a myriad of other undiagnosed mental conditions. He vomits these long-winded rants to the reader in his narratives about why humanity deserves to be exterminated, and they are not reasons you would expect a typical misanthrope to cite. I'll admit that I'm an extremely fucked-up person but I'd like to think that he's a sympathetic character, as he had a heart of gold that was ripped out and stomped on too many times.

I guess my point is - and others may disagree, which I'm fine with - what determines whether the character is sympathetic or jerkish is where the emphasis resides - on their thoughts and feelings, or what they do? There is indeed a cause-and-effect relationship.

ADDENDUM: furthermore, the specific motives of the character for his atrocities are huge. The misanthropic protagonist in my novel disowns other misanthropes as subhuman for having selfish and defamatory ideologies behind their hatred of humanity - whereas his are based in a morality which sympathizes with those who've been shat on by society (himself in particular). I suggest that you develop how the character in question processes the people with whom he associates in relation to his own belief. You can have a character who has all the asshole deeds or miserable thoughts and feelings to justify them, but in my opinion you need the latter to justify the former.

 

The key to this, I think:

1) The reader needs to know a little bit why the character is a jerk. The main character in High Fidelity is a selfish tool. But he was just recently dumped and he's in his mid-30s with no direction at all. He's miserable. (And being miserable also helped him end up in the position he's in.) But I understand why he's this way.

2) The main character also knows change will come at some point. Obviously, you build to the "lesson learned" part. But along the way there's an interaction or a thought or something where it gets in the character's head that something's not right. Like a look in the mirror or someone expressing disappointment.

 

It is highly subjective. There is a series called the Annals of the Chosen. There is very much pondering, conversing and ruminating throughout, which for me is highly entertaining and interesting.
Some persons consider it repetitive, or even consider the main character to be an idiot because of this.

 

We like flawed characters for several reasons, here are a few that I personally have noticed:

1. We can relate to them. This one's fairly simple - the character makes mistakes we already make. Maybe draw from your own life for that, that's the easiest way to make it realistic.

2. We're rooting for them for other reasons. (Warning, Harry Potter example incoming, and it won't be the last one. Hopefully I'll explain it well enough that you can understand it even if you haven't read/watched the series.) Ron Weasley and Draco Malfoy are both racist (Draco against muggle-borns and others, Ron only against non-humans such as werewolves and house-elves). But Ron is a loyal friend, eventually learns to overcome that racism for the sake of the people he cares about, and is brave and goofy and fun. A character can have one REALLY BAD flaw, or a few, but if we've grown to love them before that, chances are we'll continue to love them (as long as the traits we love aren't completely obliterated, or if they are, they're obliterated in a way that makes us sorry for them).

3. They have good intentions. Maybe a character is terrible at reading others' emotions - they say "yes" when someone asks them if this dress makes them look fat to save them the embarrassment later, things like that. Maybe they think that the best way to help someone is through "tough love" that ends up doing more harm than good, or the best way to protect someone is to force them to protect themselves. Whatever it is, their flaw is in doing harm when they try to do good. (Note that doing any of the above IS NOT NECESSARILY a flaw unless it actually causes problems, and even then, it could just be an accident. It's also possible to make a character have completely delusional ideas about what "good intentions" are, though I can't think of any examples right now.)

4. They're conflicted. Think about the fandom that's built up around Severus Snape, for example. He did awful things to Harry, some overtly abusive, but in the end he still loved Lily and wanted to protect her son, even if he did bear a terrible resentment against him.

5. They're desperate. This works for overemotional characters - characters who kill to avenge a loved one, characters who hurt others when they're hurting. They'll do anything to protect their child, even if what they do is worse than what would have happened to the child. It has to be used carefully; this is one of the deepest flaws a character can have. Those are just a few examples. We should dislike this character and we should see that they're wrong. Playing this character as though they're justified can make a quick cliché.

I'm no expert, obviously, and that's all just off the top of my head, but I hope it's something. Some well-made and well-known flawed characters worth studying, if you're interested: Albus Dumbledore, Anakin Skywalker, Gulliver, Garnet (if you're a Steven Universe fan), Tom Joad, George Bailey, Sherlock Holmes, Dr. House, Morpheus, Dallas Winston, Darrel Curis Jr., Korra. (I apologize for this list being so predominantly white, if you think of any worth adding on please do!)

Sorry if this is a bit of an info dump! Obviously I don't mean for you to go investigate every single person or idea here, I just want to leave you with some ideas to look at.

 

You could add that he's had his eye on his girlfriends increasingly hot younger sister, whom he fantasizes about "comforting" at her funeral as she's dying, just to reinstate his 'asshole status'!

 

While the book get panned a lot because it boils down creative writing to a very rigid formula, "Save the Cat" is named for exactly the type of character you are writing about.

The concept of "Save the Cat" is that when writing a flawed protagonist you need to have a event early on that makes this otherwise unlikeable character sympathetic. Like saving a kitten.

-John Wick is a killing machine, but he adopted and loved that puppy. After we see their relationship we suddenly have a soft spot for him even though everything he does the rest of the story is pure violent mayhem.

-Wolverine is an a-hole drifter, but he meets and becomes very protective of rouge. That relationship is why we like him.

- Jamie Lannister starts off as a total bad guy, but when GRR Martin wanted to change the readers perception he had Jamie comeback and save Brienne.

How you pull this off for your character is up to you, but early on give the reader a peek at a softer side. They do this in shows like House constantly. He is an arrogant prick 90% of the time, but he is always having moments of humanity which is why people like him vs. hate him.

Your character can do a lot of terrible things, but if you show us that he lets a guy live because he see that guy has a child. We suddenly see that he has a side that we can connect with. He becomes a flawed good guy, rather than a psychotic bad guy.