I am a short story writer, but I've only published once before. I have a story that I sent out to several lit mags and contests in the Fall. In November, I learned that the story was a runner-up for a contest run by a small lit mag, who wanted to publish it (without paying me). In late November, after hemming and hawing, I accepted. I withdrew my story from all the other places, save two: the two biggest and best known, mostly because it was a hassle. Bad move! Today, I got an email from one of the biggest lit mags in the business telling me that my story is on a short list for possible inclusion in the Fall 2013 issue. I have no idea what my obligation to the small mag is, legally speaking, since no money changed hands. I have no idea what their publication calendar looks like. I have no idea if I could even withdraw this story if I wanted to. I feel badly about this, because I screwed up. I don't want to behave badly, but the small mag has a circulation under 800 and the other would be a HUGE thing for my career. HUGE. I am pretty sure you will all tell me that I need to just suck it up and kiss the big guy goodbye, but I wanted to check for some advice.
The first places to look are the contest rules for the contest and the submission guidelines for the large magazine. Frequently small magazines state that you keep all publication rights, but you should check because it is possible that they required exclusivity, maybe for a certain time period. But the other issue is for the large lit magazine -- you need to check to see whether it's okay to have had the story published anywhere else. Sometimes it is, sometimes it's not. Best case scenario, the small mag allows you to retain all publication rights and the large magazine is okay with previous publication. But you'll need to investigate and then decide what to do.
One thing to consider is that the big market has only shortlisted your story. It may or may not get published. Also keep in mind that a lot of the big magazines don't publish stories that have already been published (not counting translations of famous, established writers). By having it published in the small magazine, you're using up your first publication rights. If your story is good enough to be a runner-up in a contest and be shortlisted for possible publication in a big magazine, I would withdraw the story from the non-paying market (if possible) and aim for a paying market.
I guess I'd agree that you could check to see if you can withdraw from the first magazine, but it leaves a bad taste in my mouth, quite honestly. You did accept their offer of publication, after all, and they may have already gotten the magazine set up with space for it. And, as mentioned, there's no guarantee the second magazine will actually publish your story, so you could end up with no publication and a bit of a smudge on your reputation. Then again, maybe this is SOP with magazines...