Firstly: 

With regard to when the prostration of forgetfulness should be done, before or after the salaam, there is a great difference of opinion among the scholars. The most correct view is that if a person does something extra in the prayer by mistake, the prostration must be done after the salaam, and if he omits something then he should do the prostration before the salaam. If he is not sure then it is subject to further discussion. If one thing seems more likely to him than the other then he should do the prostration after the salaam, and if neither seems more likely then he should do the prostration before the salaam. This has been discussed in the answer to question no. 12527. 

Secondly: 

It says in Fataawa al-Lajnah al-Daa’imah (8/7):  

The first tashahhud in the prayer is one of the obligatory parts of prayer, according to the more correct of the two scholarly views, because the Prophet (peace and blessings of Allaah be upon him) used to do it and he said: “Pray as you have seen me praying.” If he omitted it by mistake, he should prostrate because of forgetting, and if he omitted it deliberately, then his prayer is invalid. The one who omitted it by mistake can make up for it by doing the prostration of forgetfulness before the salaam. End quote. 

Thirdly: 

It is not prescribed to repeat the tashahhud after the prostration of forgetfulness, whether it comes before or after the salaam. This has been discussed in the answer to question no. 7895. 

Fourthly: 

The prostration of forgetfulness is to be done like the prostration in prayer, so one should prostrate on seven bones as in the prostration of prayer, and remember Allaah by reciting the well-known dhikr (Subhaana Rabbiyal-A’la (Glory be to my Lord most High)) and saying between the two prostrations Rabb ighfir li, Rabb ighfir li  (Lord forgive me, Lord forgive me). There is no special dhikr for the prostration of forgetfulness. This is what has been stated by the scholars. 

Al-Mardaawi said in al-Insaaf (2/159): 

The prostration of forgetfulness, what is said during it and after rising from it are the same as the prostration in prayer. End quote. 

Al-Ramli said in Nihaayat al-Muhtaaj (2/88): 

The manner in which it (i.e., the prostration of forgetfulness) is done is the same as the prostration in prayer, with regard to its obligatory duties and recommended parts, such as placing the forehead on the ground, being at ease in the posture, and iftiraash (sitting on the left thigh with the right foot upwards and its toes pointed towards the qiblah) when sitting in between them. End quote. 

Some fuqaha’ regard it as mustahabb to say Subhaana man la yas-hu wa laa yanaam (Glory be to the One Who does not forget or sleep) during the prostration of forgetfulness but there is not evidence for this. What is prescribed is to limit oneself to the dhikr that is said when prostrating during prayer, and not to recite any other dhikr. 

Other scholarly comments are quoted in the answer to question no. 39399. 

And Allaah knows best.