Nilambara Cakravarti, the grandfather of Sri Caitanya Mahaprabhu, was very intimately related to Raghunatha dasa's father and uncle. Nilambara Cakravarti used to call them his younger brothers
The Lord continued, "Your father and his elder brother are both related as brothers to My grandfather, Nīlāmbara Cakravartī. Therefore I consider them My grandfathers."
Nīlāmbara Cakravartī, the grandfather of Śrī Caitanya Mahāprabhu, was very intimately related to Raghunātha dāsa's father and uncle. Nīlāmbara Cakravartī used to call them his younger brothers because both of them were very much devoted to the brāhmaṇas and were very respectable gentlemen. Similarly, they used to call him Dādā Cakravartī, addressing him as an elder brother brāhmaṇa. Raghunātha dāsa, however, was almost the same age as Lord Śrī Caitanya Mahāprabhu. Generally a grandchild may joke about his grandfather. Therefore Śrī Caitanya Mahāprabhu took advantage of the relationship between His grandfather and Raghunātha dāsa's father and uncle to speak in a joking way.