Firstly: 

“At the end of the prayer” may mean after it, following it, or in the last part of it. 

There are several hadeeths which encourage dhikr and du‘aa’ at the end of the prayers, such as the following: 

1.

The report narrated by al-Bukhaari (6330) and Muslim (594) from al-Mugheerah ibn Shu‘bah (may Allah be pleased with him), according to which the Messenger of Allah (blessings and peace of Allah be upon him) used to say at the end of every prayer after saying the salaam: Laa ilaaha ill-Allaah wahdahu laa shareeka lah, lahu’l-mulk wa lahu’l-hamd wa huwa ‘ala kulli shay’in qadeer. Allaahumma laa maani‘ lima a‘tayta wa laa mu‘ti lima mana‘ta wa la yanfa‘ dhaa’l-jadd minka al-jadd  (There is no god but Allaah Alone, with no partner or associate, His is the sovereignty and to Him be praise, and He is Able to do all things. O Allaah, none can withhold what You give and none can give what You withhold, and no wealth or majesty can benefit anyone for from You is all wealth and majesty). 

2.

The report narrated by al-Bukhaari (6329) from Abu Hurayrah (may Allah be pleased with him): They said: O Messenger of Allah, the wealthy have attained exclusively the high ranks and eternal blessing (in Paradise). He said: How is that? They said: They pray as we pray, and they strive in jihad as we strive, and they spend from the surplus of their wealth, but we have no wealth. He said: “Shall I not tell you of something by means of which you will catch up with those who have gone before you and will go ahead of those who come after you, and no one could achieve what you achieve except one who does what you do. At the end of every prayer, glorify Allah (by saying Subhaan Allah) ten times, and praise Him (by saying al-hamdu Lillah) ten times, and magnify Him (by saying Allahu akbar) ten times. 

Al-Bukhaari (843) narrated: “Glorify Allah (by saying Subhaan Allah), praise Him (by saying al-hamdu Lillah), and magnify Him (by saying Allahu akbar) thirty-three times after each prayer.” 

Muslim (595) narrated: “Glorify Allah (by saying Subhaan Allah), praise Him (by saying al-hamdu Lillah), and magnify Him (by saying Allahu akbar) thirty-three times at the end of each prayer.” 

3.

Muslim (596) narrated from Ka‘b ibn ‘Ujrah (may Allah be pleased with him) that the Messenger of Allah (blessings and peace of Allah be upon him) said: “Dhikrs at the end of the prayer, the one who says them or does them at the end of each obligatory prayer will not be disappointed: thirty-three tasbeehahs, thirty-three tahmeedahs and thirty-four takbeerahs.” 

What is meant by “at the end of the prayer” in these hadeeths is immediately after the prayer, that is, after the salaam, as is stated clearly in some reports. The same is mentioned in the reports which speak of reciting Aayat al-Kursiy and the Mu‘awwidhaat at the end of the prayer. What is meant is after the salaam. 

4.

Abu Dawood (1522) narrated from Mu‘aadh ibn Jabal (may Allah be pleased with him) that the Messenger of Allah (blessings and peace of Allah be upon him) took him by the hand and said: “O Mu‘aadh, by Allah verily I love you, by Allah verily I love you.” Then he said: “I urge you, O Mu‘aadh, never to stop saying at the end of every prayer: Allahumma a‘inni ‘ala dhikrika wa shukrika wa husni ‘ibaadatika (O Allah, help me to remember You, give thank to You and worship You properly).”

Classed as saheeh by al-Albaani in Saheeh Abi Dawood

An-Nasaa’i (1303) narrated it as follows: “Do not neglect to say in every prayer, Rabbiy a‘inni ‘ala dhikrika wa shukrika wa husni ‘ibaadatika (My Lord, help me to remember You, give thank to You and worship You properly).

Classed as saheeh by al-Albaani in Saheeh an-Nasaa’i.

What is meant by “at the end of the prayer (dibr as-salaah)” here is in the last part of the prayer before the salaam, because dibr ash-shay’ (lit. the end of a thing) is part of it. This is confirmed by the words in the report of an-Nasaa’i: “in every prayer”. 

Ibn al-Qayyim (may Allah have mercy on him) said in Zaad al-Ma‘aad (1/294): “At the end of the prayer” may be understood as meaning before the salaam or after it. Our shaykh [i.e., Ibn Taymiyah] regarded it as more likely that it is before the salaam. I asked him  about that and he said: Dibr kulli shay’ (the end of everything) is part of it, like the dibr (rear end) of an animal. End quote. 

5.

at-Tirmidhi (3499) narrated that Abu Umaamah (may Allah be pleased with him) said: It was said: O Messenger of Allah, what du‘aa’ (supplication) is most likely to be heard (and responded to)? He said: “(That which is offered) in the last part of the night and at the end of the prescribed prayers.”

This hadeeth was classed as hasan by at-Tirmidhi and by al-Albaani in Saheeh at-Tirmidhi. 

What appears to be the case is that what is meant by “at the end of the prayer” here means before the salaam. 

Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) said: What appears to be the case is that what is meant by “at the end of the prescribed prayers” in the hadeeth of Abu Umaamah – if it is saheeh –is: in the last part of the prayer. End quote. 

Majmoo‘ Fataawa Ibn ‘Uthaymeen, 13/268 

The scholars noted concerning this that in the texts which mention the end of the prayer, if it is dhikr (such as saying Subhaan Allah, al-hamdu Lillah, Allahu akbar, or reciting Aayat al-Kursiy and the Mu‘awwidhaat), then what is meant by the end of the prayer in this case is after the prayer; if it is du‘aa’ (supplication), then what is meant by at the end of the prayer is in the last part of it, i.e., before the salaam. 

But if there is a report to indicate that a particular du‘aa’ should be said after the salaam, such as when the Prophet (blessings and peace of Allah be upon him) said: “Ask for forgiveness (by saying Astaghfirullah) three times,” this is a du‘aa’, but the Sunnah indicates that it should be said after the salaam. 

Shaykh Ibn Baaz (may Allah have mercy on him) was asked: 

What is meant by “at the end of the prayer” in the hadeeths which encourage saying du‘aa’ or dhikr at the end of each prayer? Is it the last part of the prayer or after the salaam? 

He replied: 

The phrase “the end of the prayer” may be applied to the last part of it before the salaam, or it may be applied to what comes immediately after the salaam. There are saheeh hadeeths that mention that, most of which indicate that what is meant is the last part of the prayer before the salaam if it has to do with du‘aa’, such as the hadeeth of Ibn Mas‘ood, in which the Messenger (blessings and peace of Allah be upon him) taught him the tashahhud, then he said: “Then let him choose whatever du‘aa’ he likes and say it.” And according to another version he said, “Let him choose after asking whatever he wants.” (saheeh – agreed upon). 

Another example is the hadeeth of Mu‘aadh in which the Prophet (blessings and peace of Allah be upon him) said to him: “Do not neglect to say at the end of every prayer: Allahumma a‘inni ‘ala dhikrika wa shukrika wa husni ‘ibaadatika (O Allah, help me to remember You, give thank to You and worship You properly).”

Narrated by Abu Dawood, at-Tirmidhi and an-Nasaa’i with a saheeh isnaad. 

Another example is the hadeeth narrated by al-Bukhaari (may Allah have mercy on him) from Sa‘d ibn Abi Waqqaas (may Allah be pleased with him) who said: The Prophet (blessings and peace of Allah be upon him) used to say at the end of every prayer: “Allaahumma inni a‘oodhu bika min al-bukhl, wa a‘oodhu bika min al-jubn, wa a‘oodhu bika min an uradda ila ardhal il-‘umr, wa a‘oodhu bika min fitnat ad-dunya wa min ‘adhaab al-qabr (O Allah, I seek refuge in You from miserliness, and I seek refuge in You from cowardice, and I seek refuge in You from reaching feeble old age, and I seek refuge in You from the trials of this world, and I seek refuge in You from the torment of the grave. 

With regard to the adhkaar that have been narrated, the saheeh hadeeths indicate that they are to be recited at the end of the prayer after the salaam. An example of that is to say after saying the salaam: “Astaghfirullah, astaghfirullah, astaghfirullah. Allaahumma anta al-salaam wa minka al-salaam tabaarakta ya dhaa’l-jalaali wa’l-ikraam (I ask Allaah for forgiveness, I ask Allaah for forgiveness, I ask Allaah for forgiveness. O Allaah, You are the One Who is free from all defects and deficiencies and from You is all peace, blessed are You, O Possessor of majesty and honour), whether he was the imam or praying behind the imam or praying on his own. Then after that the imam should turn to face the congregation, and the imam, the one who prayed behind the imam and the one who prayed on his own should say, after that dhikr and prayer for forgiveness: Laa ilaaha ill-Allaah wahdahu laa shareeka lah, lahu’l-mulk wa lahu’l-hamd wa huwa ‘ala kulli shay’in qadeer. Laa hawla wa laa quwwata illa Billaah wa laa na‘budu illaa iyyaah. Lahu’l-ni‘mah wa lahu’l-fadl wa lahu’l-thanaa’ al-hasan. Laa ilaaha ill-Allaah mukhliseena lahu’l-deena wa law kariha’l-kaafiroon. Allaahumma laa maani‘ lima a‘tayta wa laa mu‘ti lima mana‘ta wa la yanfa‘ dhaa’l-jadd minka al-jadd  (There is no god but Allaah Alone, with no partner or associate His is the sovereignty and to Him be praise, and He is Able to do all things. There is no power and no strength except with Allaah, and we worship none but Him.  From Him (alone) come all blessings and favours, and all good praise is due to Him. There is no god but Allaah and we make our worship purely for Him (alone) however much the disbelievers may hate that. O Allaah, none can withhold what You give and none can give what You withhold, and no wealth or majesty can benefit anyone for from You is all wealth and majesty). 

It is mustahabb for the Muslim, male or female, to recite this dhikr after each of the five daily prayers, then to glorify Allah (by saying Subhaan Allah), praise Him (by saying al-hamdu Lillah), and magnify Him (by saying Allahu akbar) thirty-three times, then to complete one hundred by saying: Laa ilaaha ill-Allaah wahdahu laa shareeka lah, lahu’l-mulk wa lahu’l-hamd wa huwa ‘ala kulli shay’in qadeer (There is no god but Allaah Alone, with no partner or associate His is the sovereignty and to Him be praise, and He is Able to do all things).

All of that is proven in hadeeths from the Messenger of Allah (blessings and peace of Allah be upon him). After that it is mustahabb to recite Aayat al-Kursiy once, quietly, and to recite Qul Huwa Allahu Ahad (Soorat al-Ikhlaas) and al-Mu‘awwidhatayn once, quietly, except in the case of Maghrib and Fajr, when it is mustahabb to repeat the recitation of the three soorahs mentioned, three times. It is also mustahabb for the Muslim, male or female, to say after praying Maghrib and Fajr: Laa ilaaha ill-Allaah wahdahu laa shareeka lah, lahu’l-mulk wa lahu’l-hamd yuhyi wa yumeet wa huwa ‘ala kulli shay’in qadeer (there is no God but Allaah alone, with no partner or associate, His is the Dominion and to Him be praise, He gives life and gives death, and He has power over all things) ten times in addition to what is mentioned above, before reciting Aayat al-Kursiy and before reciting the three soorahs mentioned above, in accordance with the saheeh hadeeths that have been narrated concerning that. End quote. 

Majmoo‘ Fataawa Ibn Baaz, 11/194 

Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) said: 

If one ponders this matter it will become clear that what is connected to the end of the prayer, if it is dhikr it comes after the prayer and if it is du‘aa’ it comes in the last part of the prayer. 

With regard to the former, Allah, may He be exalted, has made the time after the prayer a time for dhikr, as He says (interpretation of the meaning): “When you have finished As-Salat (the prayer - congregational), remember Allah standing, sitting down, and lying down on your sides” [an-Nisa’ 4:103]. And the Sunnah explains what is mentioned in general terms in this verse of dhikr, such as when the Prophet (blessings and peace of Allah be upon him) said: “The one who glorifies Allah (by saying Subhaan Allah) at the end of every prayer thirty-three times…” Each text that mentions dhikr at the end of the prayer is to be understood as referring to after the prayer, in accordance with this verse. 

With regard to the latter, the Prophet (blessings and peace of Allah be upon him) regarded the time after the final tashahhud as a time for du‘aa’. So every text that mentions du‘aa’ at the end of the prayer is to be understood as referring to the last part of it, so that the du‘aa’ will be at the point where the Prophet (blessings and peace of Allah be upon him) taught us to offer du‘aa’, unless interpreting the text in this manner is impossible or unlikely, according to the context, in which case it is to be understood as indicated by the context. End quote. 

Majmoo‘ Fataawa Ibn ‘Uthaymeen, 13/268 

Secondly: 

It is not prescribed to raise the hands when saying du‘aa’ after the prayer, because that was not narrated from the Prophet (blessings and peace of Allah be upon him). It says in Fataawa al-Lajnah ad-Daa’imah (7/103): When saying du‘aa’ after the obligatory prayer it is not Sunnah to raise the hands, whether that is done by the imam on his own or the one who prayed behind the imam on his own, or by both of them. Rather that is an innovation, because it was not narrated from the Prophet (blessings and peace of Allah be upon him) or from his Companions (may Allah be pleased with them). As for saying du‘aa’ without doing that (raising the hands), there is nothing wrong with it, because there are some hadeeths that mention that. End quote. 

See also the answer to question no. 21976 and 7886 

And Allah knows best.