I try to read at least a couple chapters before deciding to quit. I can usually tell by then if the writing is good and if the book is going to hold my interest. On a related note, there are quite a few books that I stopped reading and picked up again when I was older, so if you don't like it now, you can always come back to it later to see if your opinion has changed.
Three that I never finished were War and Peace, Gone with the Wind, and Revelation Space. Stopping points varied but I just could not get into those 3. Only slightly different was Game of Thrones. I finished the first but will not read another.
Consider how many people went to see the Transformers movies and whether you want let these people's opinions influence what you do or don't read. Personally I just give up when I get bored, which can take anything from two pages to a hundred.
It depends on whether I have put down any money on the book. If I have paid full retail price, I will often push myself to like it a lot more than if it is a library book. Often I will get a whole lot of ‘good looking’ books from the library, take them home, and read just a few pages and then return them to the library. I suppose I could just sit at the library and do the same thing...
I can think of only a couple books I didn't finish - one was Leon Uris's Trinity. It's not something I'd generally choose, but my father-in-law thought I'd like it, so I tried. I got too impatient with the dialogue. The other one was the Phantom Tollbooth - the kids book. I somehow missed that one as a child, but picked it up when my kids were reading it. It's very good and very clever, but I once I got the shtick I got kind of bored. I can't think of anything else. I think the "not something I'd generally choose" is key to me not finishing. I rarely pick up something randomly - I have too many "must-reads" on my list and will see them through to the end. Even if it's painful.
I used to power through every book I started. Even if I disliked it, I'd be curious to see it through to the end. And then, I was in the eighth grade. Two of my friends, whose tastes I usually trusted, were raving about this brand new book that had just come out. I couldn't not check it out. So I borrowed my friend's copy, eager to see what the fuss was about. Stephanie Meyer's Twilight was the first book I ever gave up on. It taught me the valuable lesson that there are only so many hours in the day, and you don't have to waste your time on books that just don't do it for you. (I did, however, end up seeing all the movies. With friends on weekday nights, so we had the theatre to ourselves and could poke fun at it without disturbing anyone else. A bad movie is fun because you can enjoy it with others - books, not so much.)