He (Kasyapa Muni) could have refused her (Diti) with strong words expressing impossibility, but he was not as spiritually strong as Vidura
SB Canto 3
O hero (Vidura), Diti, being thus afflicted by the contamination of lust, and therefore poor and talkative, was pacified by the son of Marīci in suitable words.
When a man or woman is afflicted by the lust of sex desire, it is to be understood as sinful contamination. Kaśyapa was engaged in his spiritual activities, but he did not have sufficient strength to refuse his wife, who was thus afflicted. He could have refused her with strong words expressing impossibility, but he was not as spiritually strong as Vidura. Vidura is addressed here as a hero because no one is stronger in self-control than a devotee of the Lord. It appears that Kaśyapa was already inclined to have sexual enjoyment with his wife, and because he was not a strong man he tried to dissuade her only with pacifying words.