Firstly: It is not permissible for a man to swear against himself and invoke the curse or wrath of Allaah upon himself, or swear an oath by saying that he will be a kaafir or beyond the pale of Islam or he will be an apostate and so on. The evidence for that is as follows: 

1 – The verse in which Allaah says (interpretation of the meaning): 

“And were Allaah to hasten for mankind the evil (they invoke for themselves and for their children, while in a state of anger) as He hastens for them the good (they invoke) then they would have been ruined”

[Yoonus 10:11]

al-Qurtubi said: 

Mujaahid said: This was revealed concerning a man who prayed against himself, his wealth or his children when he got angry, saying, “O Allaah, destroy it, O Allaah, do not bless it and curse it” etc. If that were to be responded to as prayers for good were responded to, then they would have been ruined. So this verse was revealed to criticize this bad characteristic that exists in some people who pray for good and want a quick response, then sometimes their bad attitude makes them pray for bad things, but if those prayers were to be answered quickly they would be ruined. 

Tafseer al-Qurtubi, 8/315. 

2 – It was narrated from Thaabit al-Dahhaak (may Allaah be pleased with him) that the Messenger of Allaah (peace and blessings of Allaah be upon him) said: “Whoever swears an oath by saying that he will belong to a religion other than Islam, then he is as he said.”  Narrated by al-Bukhaari, 5700.

3 – The Prophet (peace and blessings of Allaah be upon him) heard a man cursing his camel when it walked too slowly. The Messenger of Allaah (peace and blessings of Allaah be upon him) said: “Who is cursing his camel?” He said: “It is me, O Messenger of Allaah.” He said: “Get down from it, for we do not want to be accompanied by something that is cursed. And do not pray against yourselves or against your children, and do not pray against your wealth, lest you say such a thing and it coincides with a time when Allaah answers prayers and responds to what you ask for.” Narrated by Muslim, 3014. 

Secondly:  With regard to what the one who has said such a thing should do, there is a difference of opinion among the scholars. Some of them are of the view that he has to offer expiation, and others are of the view that he does not have to offer expiation.

In al-Musannaf, ‘Abd al-Razzaaq narrated some reports that were transmitted from the salaf concerning that: 

It was narrated from Ibn ‘Abbaas concerning a man who swore that he would be a Jew or a Christian or a Magian or out of Islam or that the curse of Allaah would be upon him or that he would be obliged to do something, that this was a binding oath. (15974). 

It was narrated that Tawoos said: “Whoever says ‘I am a kaafir’ or ‘I am a Jew or a Christian or a Magian’ or ‘May Allaah humiliate me’ and the like (if I do such and such), this is an oath (yameen) for which he must offer expiation.” (10975). 

It was narrated that Ibn Jurayj said: I heard a person say to ‘Ata’: “If a man says, May the wrath of Allaah be upon me, or May Allaah humiliate me or I pray to Allaah against myself or …, will that put him beyond the pale of Islam?” He said: “I would rather that he carried out what he resolved to do. If he did not carry out what he resolved to do, then he does not have to do anything because it is not an oath.” (15977). 

It was narrated that Ibn Jurayj said: I heard ‘Ata’ being asked about a man who says: “I give my promise and covenant to Allaah,” then he broke his word; was that an oath (yameen)? He said: “No, unless he intended it as an oath or he said, May Allaah humiliate me, or may the curse of Allaah be upon me, or he said I will associate someone else with Allaah or I will disbelieve in Allaah and so on. He said: No (meaning, it is not an oath), unless he swore by Allaah.” (15978). 

Ibn Qudaamah (may Allaah have mercy on him) said that it is more likely that he does not have to offer expiation because there is no evidence to suggest that offering expiation in this case is obligatory.  See al-Mughni, 13/465. 

Shaykh ‘Abd al-Rahmaan ibn Jibreen (may Allaah preserve him) was asked about this matter and he said: This is not an oath (yameen) and no expiation is required, but he has to repent to Allaah from that. 

If the Muslim wants to be more on the safe side and offer kafaarat yameen (expiation for breaking an oath), that is fine. 

And Allaah knows best.