The paleness (even though aged and with patina) and crispness of line says limestone to me. Where is this? That might actually help.
When I see something as smooth as the one on the left I think marble, but it would probably depend where your story takes place - it's pretty expensive to move stone long distances, so it's usually sourced fairly locally. Where I live, limestone's the most common building stone - won't last as long as some, but plentiful and easy to work with. I'd say either of those buildings could be limestone if it were cut properly... see Chateau Laurier for a grey version, at http://30dayadventures.ca/30daysinottawa-sleeping-in-a-castle-fairmontlaurier/ We also use sandstone if we're feeling fancy. You could check http://www.gly.uga.edu/railsback/BS-Main.html for a catalogue of images that might twig something for you.
Looks like limestone to me. There's a ton of it where I live. Poking around it seems like a lot of the buildings in the area are either limestone or sandstone.
I bet you could contact these guys - https://www.facebook.com/InvernessLocalHistory/ - and they'd be able to help you out. Google maps makes it look like most of downtown is built of that lovely taupe stone from the right-hand building.
Looking for building stone in Scotland I found: Geology and architecture in South Scotland. Should be similar in the north. And this: Building with Scottish Stone , which also looks very promising.
It's fascinating to me the different approaches people have for searching out information like this. I never think about historical groups or looking for those groups on social media. You've expanded my horizons.
Ditto on the limestone from me. And I'm a licensed architect; I'm supposed to know these things. Dressed ashlar, most of it. The building on the left features quoins at the corner, and the one on the right has blocks worked with margins on the ground floor where it's supporting those engaged columns on the first storey. BTW, the upper storeys there appear to be brownstone, which is a particular kind of limestone. It's lovely work. I never made it to Inverness when I toured Scotland. Too bad for me.
