Generally, it's whether something to follow on was intended. Lord of the Rings was intended to be one gargantuan book originally but then Tolkein was convinced to split it into three. This is where you get the different between a sequel and a series. Obviously some sequels do work. Battlestar Galactica was remade for after the wars and worked much better than the first. Though this is probably more to do with the new writing team than it working as a sequel. Trudi Canavan's Traitor Spy Trilogy seems to be working though but it's obvious she left some mysteries open in case the opportunity for a second trilogy in that world to be written. But sequels where there wasn't room for one rarely work. There needs to be something to work from.
