if I get a short story published in a magazine do they own my characters and could I still use them in a book I'm writing
If you get it published, they should send you an easy-to-read legal document. Most likely, they won't even own your short story; they'll only have temporary publishing rights to it. You can have it published in several magazines, as long as none of them insist on "first rights".
Yes they do own it if they are paying for it. If you want them to give temporary rights, you need to discuss this prior to publishing.
Paying for it doesn't transfer copyright ownership. They won't own the characters or the copyright in the story unless there is a written agreement that says they do. If there is, pick another market because that's not normal.
Steerpike is correct. Simply publishing a story does not give the publisher rights TO the story. If there is no agreement to the contrary, then the rights are held by the author.
