Firstly: 

If it was slightly off, then it does not matter. 

Secondly: 

If a person tries his best to find out the qiblah, whether by asking local people who appear to him to be sincere, or otherwise, then he has done what is required of him. If he finds out later on that it was wrong, his prayer is still valid and he does not have to repeat it.

 It says in al-Mughni by Ibn Qudaamah (may Allaah have mercy on him): 

Whoever prays facing a certain direction after striving to work it out, then he finds out that he got the qiblah wrong, does not have to repeat it. In other words, if a person does his best to work out the direction and prays, then he finds out for certain that he prayed in a direction other than the direction of the Ka’bah, he does not have to repeat it, and the same applies to the one who follows him. This is the view of Maalik, Abu Haneefah and al-Shaafa’i, according to one of his two opinions.