It is derived from cursing because each one of the married couple curses himself in the fifth if he or she is a liar. Some people said: It was called as such because one of the married couple should be lying, so he is cursed.
In the Shari`ah: Li`an is a set of testimonies confirmed by oaths from both sides correlated by cursing and anger; in this case either the prescribed penalty of false accusation shall be applicable on him or the prescribed penalty of adultery shall be applicable on her. Sahl ibn Sa`d As-Sa`idy reported that `Uwaymir Al `Ajlany came to `Asim ibn `Ady Al Ansary and said to him. Tell me about a person who finds a man with his wife; should he kill him, and be killed in retaliation; or how should he act? O `Asim, ask the Prophet (peace be upon him) about that for me. So, `Asim asked Allah's Messenger (peace be upon him) and he did not like this question and he disapproved of it so much that `Asim felt aggrieved at what he had heard from Allah's Messenger (peace be upon him). When `Asim came back to his family, `Uwaymir came to him and said: O `Asim, what did Allah's Messenger (peace be upon him) say to you? `Asim said to `Uwaymir: You did not bring something good. Allah's Messenger (peace be upon him) did not like this religious verdict that I sought from him. `Uwaymir said: By Allah, I will not rest until I ask him about it. `Uwaymir proceeded until he came to Allah's Messenger (peace be upon him) as he was sitting amidst people, and said: O Messenger of Allah, tell me about a person who finds a man with his wife. Should he kill him, and then you would kill him, or how should he act? Thereupon, Allah's Messenger (peace be upon him) said: Ayahs have been revealed concerning you and your wife; so go and bring her. Sahl said that they both invoked curses (and further said): I was along with people in the company of Allah's Messenger (peace be upon him). And when they had finished, `Uwaymir said: Allah's Messenger, I shall have told a lie against her if I keep her (now). So, he divorced her with three pronouncements before Allah's Messenger (peace be upon him) had commanded him. Ibn Shihab said: Subsequently that was the practice of invokers of curses (Al Mutala`inain). Sahl said: She was pregnant and after birth, the child was ascribed to his mother, then it has become a Sunnah that he inherits from her and she inherts from him the prescribed share of inheritance. (2)
(1) See: Bada'i` As-Sana'i` 3/242, Mughny Al Muhtaj 3/367, Al Mubdi` Sharh Al Muqni` 8/65, `Umdat Al Fiqh 1/111, Hashiyat Ar-Rawd Al Murbi` 7/29, and Al Mughny 9/3.
(2) Reported by Al Bukhari in the chapter on cursing one another in the masjid, Hadith No. 5003 - 5/2033, and Muslim in the chapter on "Yahya ibn Yahya told us," Hadith No. 3816 - 4/205 with his own wordings.