Firstly: 

A brother should help his sister to obey Allaah, which includes advising her to wear hijab which is enjoined by Allaah, and using wise methods in doing so, exhorting and calling her in the way that is best. He should avoid being harsh and cruel, for there is no kindness in a thing but it adorns it, and it is not taken away from a thing but it makes it defective. Allaah says (interpretation of the meaning): 

“Invite (mankind, O Muhammad صلىالله عليه وسلم) to the way of your Lord (i.e. Islam) with wisdom (i.e. with the Divine Revelation and the Qur’aan) and fair preaching, and argue with them in a way that is better. Truly, your Lord knows best who has gone astray from His path, and He is the Best Aware of those who are guided”

[al-Nahl 16:125]

“And by the Mercy of Allaah, you dealt with them gently. And had you been severe and harsh‑hearted, they would have broken away from about you”

[Aal ‘Imraan 3:159] 

“And say to My slaves (i.e. the true believers of Islamic Monotheism) that they should (only) say those words that are the best. (Because) Shaytaan (Satan) verily, sows a state of conflict and disagreements among them. Surely, Shaytaan (Satan) is to man a plain enemy”

[al-Isra’ 17:53]

The Prophet (peace and blessings of Allaah be upon him) said: “Allaah is Kind and loves kindness, and He rewards for kindness in a way that He does not reward for harshness or for anything else.” Narrated by Muslim (2593). 

And he (peace and blessings of Allaah be upon him) said: “Whoever is deprived of kindness is deprived of goodness.” Narrated by Muslim (2592). 

This kindness is required even more so of the daa’iyah if his words are addressed to his family and relatives, because of the rights they have of kinship, kindness and respect.  

Secondly: 

Shunning those who follow innovation and commit sin is prescribed in Islam, if one thinks it most likely that it will be of benefit and have an effect, as the Prophet (peace and blessings of Allaah be upon him) shunned the three who stayed behind from the campaign to Tabook, until Allaah accepted their repentance. 

But shunning may make a person more rebellious and stubborn, and prevent further opportunities to advise and call him; in that case it should not be done. 

Shunning is like medicine; it may be used when needed, if it is thought most likely that it will be of benefit. 

Shaykh al-Islam Ibn Taymiyah (may Allaah have mercy on him) said: Shunning varies according to the strength or weakness of those who are doing it, and whether they are few or many. The purpose of it is to rebuke and discipline the one who is being shunned, and to put others off from being like him. If that is likely to serve an interest, so that shunning him will weaken and reduce the evil, then it is prescribed, But if neither the one who is being shunned nor anyone else will be deterred by that, rather it will make things worse, and the one who wants to shun him is weak, so that it will do more harm than good, then it is not prescribed to shun, rather softening people’s hearts may be more beneficial in some cases than shunning. 

And shunning is more beneficial in some cases then softening hearts. Hence the Prophet (peace and blessings of Allaah be upon him) softened the hearts of some people and shunned others. End quote from Majmoo’ al-Fataawa, 28/206. 

Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on him) was asked: Some young men – may Allaah guide us and them – are very negligent about prayer, to the extent of neglecting it altogether. These young men may be our relatives such as brothers and the like, and some of them are friends. How should we deal with them in your opinion? Should we shun them as prescribed in Islam or what? 

He replied: undoubtedly neglecting prayer is a cause of doom, because if prayer is done properly all one’s deeds will be in order, but if it is not in order, all of one’s deeds will be spoiled. It is like the heart of one's deeds. Hence when the Reckoning comes on the Day of Resurrection, the first thing that will be examined will be one's prayer. If a person neglected it then he is more likely to have neglected other duties, but if he prayed regularly then the rest of his deeds will be examined. And prayer is the pillar (or foundation) of faith; if it falls, the entire structure will collapse. Allaah says (interpretation of the meaning): 

“Then, there has succeeded them a posterity who have given up As‑Salaah (the prayers) [i.e. made their Salaah (prayers) to be lost, either by not offering them or by not offering them perfectly or by not offering them in their proper fixed times] and have followed lusts. So they will be thrown in Hell.

60. Except those who repent and believe (in the Oneness of Allaah and His Messenger Muhammad صلى الله عليه وسلم), and work righteousness. Such will enter Paradise and they will not be wronged in aught”

[Maryam 19:59-60].

If people take the matter of prayer lightly, whether they are fathers, sons, brothers, uncles, aunts, other relatives, friends or neighbours, we must advise them, warn them and explain to them what goodness, great reward and good effects there are in prayer, and what evil and doom there is in neglecting it. If they pay heed, this is what we want. If they do not pay heed, then we should look at the matter further. Will shunning them and keeping away from them serve any interest such as making them feel ashamed so that they will repent, or will it only make matters worse and put them off more? If it is the latter, then we should not forsake them. If it is the former, then we should shun them, i.e., if our shunning them will make them feel ashamed and mend their ways, then we should shun them until they mend their ways. But if shunning will not achieve anything, rather it will only make matters worse, then we should not shun them, because shunning is a medicine, and when is medicine used? It is used when needed and when it is thought it will be of benefit. If it is thought that the medicine will not be of benefit, then it should not be used. End quote from Liqa’ al-Baab il-Maftooh (5/209). 

Secondly: 

You have no right to resort to beating your sister if she insists on not wearing hijab, because you have no authority over her in this regard. It is sufficient for you to advise and explain, and to seek the help of your parents, and remind them of the responsibility that Allaah has given them to look after those who are under their care, as Allaah says (interpretation of the meaning): 

“O you who believe! Ward off yourselves and your families against a Fire (Hell) whose fuel is men and stones, over which are (appointed) angels stern (and) severe, who disobey not, (from executing) the Commands they receive from Allaah, but do that which they are commanded”

[al-Tahreem 66:6]

And the Prophet (peace and blessings of Allaah be upon him) said: “Each of you is a shepherd and each of you is responsible for his flock. The ruler of the people is a shepherd and is responsible for his flock. A man is the shepherd of his household and is responsible for his flock. A woman is the shepherd of her husband’s house and children and is responsible for her flock. The slave is the shepherd of his master’s wealth and is responsible for it. Each of you is a shepherd and each of you is responsible for his flock.” Narrated by al-Bukhaari (893) and Muslim (1829). 

We ask Allaah to guide your sister. 

And Allaah knows best.