Parashas Bamidbar · Passuk 1:02
Take a census of the whole Israelite communityawhole Israelite community Or those eligible to be fighters on the community’s behalf. by the clans of its ancestral houses,bits ancestral houses I.e., its tribes. listing the names, every male, head by head.
Chabad.org — English translation + Rashi
Source: Yevamos 54b.
English: “By their families, by their fathers’ houses” — from here [we learn] that the family of the father is called a family; the family of the mother is not called a family.
In context: Paternal lineage defines tribal and family identity for the census and for inheritance. The midrash states the principle succinctly.
Custom
Chazal derive that “family” for these purposes follows the father’s house, not the mother’s.