A country in Europe, time 1890s. Just massacred ~2 millions in Balkan and Palestine, Ottoman cannot protect them anymore. Government estimate hundreds of thousands still in Europe. Will be killed when found. Ignoring international reputation damage for conducting genocide. Most Balkan country support it, and would actively help. How to find those that hides? How to negotiate for extradition with other non-supporting country? Also, how should those Muslim hide and escape? Constantinople and Jerusalem had been wiped out of Muslim and currently under European control. Plot only, not personally supporting genocide.
You hunt people the same way regardless of the end purpose. They had normal lives before the purge began, which is where you start. There are records in their home town city hall (best in the 1890s when mobility was much less), their place of worship, where they were employed. People flee to the familiar: friends, family. Beyond that, to the like minded. For those whom have fled the country, ascents, the need to support themselves using the only work skills they have. Criminal organizations will be glad to sell information on them as they will track refugees and 'out of place' communities; local police likewise. The Germans rounded up 95% of Polish Jews between the winter of 1939 and spring 1944. In less than sixty months they collected or caused to flee a population that was as old as the nation. The key to hunting people is that generally you do not catch them; others turn them in, or they weary of the stress of being a fugitive and simply give themselves up.
Muslims practice circumcision. If the group oppressing them doesn't, that could be a tool to identify male Muslims, as it was with male Jews in the Holocaust.
Just so you know, the proportion of Jews killed in the Holocaust correlates strongly with how thorough the censes were in that particular country.
In Shogun, James Clavell had his samurai screened for Christians by requiring that they all (IIRC) spit and stomp upon either a Bible or a cross, don't remember which. A similar test might work with a Quran, but I won't be the one to write it.
It might work for some, but both major sects of Islam permit believers to lie under duress about religious belief. https://en.m.wikipedia.org/wiki/Taqiya That presents another facet to OP's question, because those threatened could simply pretend to convert away from Islam or say they'd never been Muslim. I don't recall if the government in question in OP's work had anything against Jews, and I'm not sure about rate of circumcision in Europe in this time frame, but if it doesn't have an issue with Jews, Muslims could pretend to be Jews. That would provide solid cover for circumcisions. If the government in question is deeply invested in stamping out Islam in its territory, but not going after legitimate apostates from it, something equivalent to the old Inquisitions would be required to separare those practicing in secret from true apostates.
1/ Zal Batmanglij Batmanglij at Wondercon 2012 Born 1980/1981 (age 36–37)[1] France That's how good records are TODAY...somebody born in France doesn't even have an accurately recorded year of birth. How good do you think record-keeping was in the Balkans in the 1890s? 2/ "I'm going to get killed if I turn myself in, but that's so much better than all this running and hiding and staying alive." Members of DAESH are fleeing Mosul by shaving off their beards and passing themselves off as ordinary guys.
Setting fire to a Quran. It's a sin to burn a Quran. It would work. (Side note-- if we put Qurans in our military vehicles, wouldn't that mean ISIS can't blow them up?)
There's bound to be racism involved too. People, even today, associate Arabs with Islam. Anyone who looks Arabic will become a target and required to prove their non-Muslim status. Conversely, non-Arab Muslims, especially white Muslims, might never be suspected unless they are caught harboring other Muslims. Even then they might be mistaken for allies only-- though I imagine the punishment would be little different.
You would be surprised. I've done some ancestry research, and used church records from the 1630s (digital photos of same). Dates of birth, addresses, parent, grandparents, occupations...just because France was inaccurate with one guy does not prove anything. Pretty fair, IME. As I noted above. Bureaucracy is nothing new. I've spent thirty years hunting people for whom getting caught meant prison, sometimes for the rest of their lives. They do turn themselves in, simply because the stress of running grows too heavy. Likewise, many Jews during the Final Solution turned themselves or killed themselves in simply because they were sick of hiding. When hope fails there is nothing left. I've hunted people with nothing more than a nickname and a vague description (white male, early-mid 20s) and found them. There is a reason the Germans used ordinary street police for much of their hunt for Jewish populations, and the Imperial Russians and Soviets did the same for their pogroms: because they are used to finding people. There are countless books written on the subject, starting with works commissioned by Sir Peel two hundred years ago.
A couple points, that would make something like this ring more true for me: 1. There's no foolproof test. Any such round up is going to catch a bunch of false positives (thinking someone is a Muslim when they're not), and is also going to miss a bunch of actual Muslims. I think any sort of government carrying out something on this scale wouldn't worry too much about this, though. 2. There's going to be a lot of local variation. In some places non-Muslim populations will be more sympathetic and help their neighbors to hide/escape, in other places the local populations will willingly help the authorities. But everywhere there will be a range of reactions present among non-Muslims, just some people will be able to act on their reactions while others will have to be more quiet about it. 3. There will also be a range of reactions among Muslim populations. Some populations would let themselves be corralled more easily, while others might take to the woods as partisans, and still others might try and organize mass escapes. Yeah, I guess really there's a lot of ways you could go with this, depending on what sort of story you want to tell.
Eh, I'm pretty sure there won't be an issue with racism as the modern world knows it. The idea of who is white evolved with time; Italians and Poles would be dealing with racial prejudice in the US in this time frame, for example. If OP is still on the project I recall, and it sounds like he is, the government in question is basically an Austria dominated (rather than Prussia dominated) German Empire. They would be well aware of the ethnic and religious diversity of the Balkans, and well aware of "white" ethnic groups with large Muslim populations.
The way the Serbs did it in bosnia (and later kosovo) was that the local communities largely know who their muslim members are, so when the population is factionalised by hate and agit prop they can easily identify them (and put a circled M on their doors)....
That is the key issue: no one lives in a vacuum. You're not looking for members of a secret cabal, at one time they openly practiced their faith.