Rashi on Bereishis 46:3, citing Pirkei de-Rabbi Eliezer 39.
Sefaria — Rashi on Genesis 46:3
אַל תִּירָא מֵרְדָה מִצְרַיְמָה — לְפִי שֶׁהָיָה מֵצֵר עַל שֶׁנִּזְקָק לָצֵאת לְחוּצָה לָאָרֶץ.
English: “Do not be afraid to go down to Egypt” — because he was distressed that he was compelled to leave the Land of Israel.
What Is Learned: Rashi explicitly attributes Yaakov’s fear to the pain of leaving the Land, and names Pirkei de-Rabbi Eliezer 39 as his source.