Firstly: 

The call in the Holy Qur'aan, “O mankind” is mentioned in twenty places in the Book: two in Soorat al-Baqarah, four in al-Nisa’, one in al-A’raaf, four in Yoonus, four in al-Hajj, one in Luqmaan, three in Faatir and one in al-Hujuraat. 

The one who ponders all of these verses will find that they are addressed to all of mankind, believers and disbelievers, righteous and evildoers, calling them to think about that which will benefit them in the Hereafter and reminding them of the Lordship of Allah over them and their need for Him, so that this will motivate them to worship Him alone with no partner or associate, and to devote their religious commitment sincerely to Him alone. Because this is addressed to all of mankind, and is not limited to any specific group, it is appropriate that it should say “O mankind” rather than “O you who believe.” 

That has nothing to do with mention of the Ka’bah, Makkah or Hajj in particular; rather most of the soorahs mentioned above in which the call, “O mankind” appears, do not mention anything about the Ka’bah, Makkah or Hajj at all. We do not think that this call comes only when there is mention of the Ka’bah or Makkah, to the exclusion of other Islamic duties. 

Some scholars have mentioned, when discussing the difference between Makkan and Madinan soorahs, that the words “O mankind” appear in the Makkan soorahs and “O you who believe” appear in the Madinan soorahs. This is what is usually the case. In Madinan soorahs such as al-Baqarah and al-Nisa’, it says “O mankind” and in some of the Makkan soorahs such as al-Hajj, it says “O you who believe.” 

Secondly: 

With regard to places in the Holy Qur'aan in which words addressed in general to “mankind” appear in the context of Hajj and its rituals, perhaps the reason for that is that Hajj was first enjoined with the call of Ibraaheem (peace be upon him) to all the people of Earth, calling them to visit the sacred House of Allah and to perform the Hajj rituals there. That appears in the verse in which Allah says (interpretation of the meaning):

“And proclaim to mankind the Hajj (pilgrimage). They will come to you on foot and on every lean camel, they will come from every deep and distant (wide) mountain highway (to perform Hajj)”

[al-Hajj 22:27]. 

Al-Haafiz Ibn Katheer (may Allah have mercy on him) said: 

Meaning: give the call to mankind, calling them to come on the pilgrimage to this House which We have commanded you to build. 

It was narrated that he said: O Lord, how can I call mankind when my voice will not reach them? It was said: Call, and it is for Us to convey. So he stood on his Maqaam (station), or it was said on the Hijr, or on al-Safa, or on Abu Qubays [a mountain beside the Ka’bah], and said: O mankind, your Lord has established a House, so come on pilgrimage to it. It was said that the mountains lowered themselves so that his voice could reach all the corners of the earth, and those who were still in their mothers’ wombs and fathers’ loins heard it, and everything that heard him of rocks, plains and trees responded, as did those who Allah decreed would perform Hajj until the Day of Resurrection, saying: Labbayk Allahumma labbayk. This is a summary of what is narrated from Ibn ‘Abbaas, Mujaahid, ‘Ikrimah, Sa’eed ibn Jubayr and others of the salaf. And Allaah knows best. 

It was narrated by Ibn Jareer and Ibn Abi Haatim at length. End quote. 

Tafseer al-Qur’aan al-‘Azeem, 5/414 

And Allah knows best.