Opposing effects of Interferon-induced transmembrane proteins (IFITMs 1, 2 and 3) on SARS-CoV-2 infection have been reported. The reasons for this are unclear and the role of IFITMs in infection of other human coronaviruses (hCoVs) remains poorly understood. Here, we show that endogenous expression of IFITM2 and/or IFITM3 is critical for efficient replication of SARS-CoV-1, SARS-CoV-2 and hCoV-OC43 but has little effect on MERS-, NL63- and 229E-CoVs. In contrast, overexpression of IFITMs inhibits all these hCoVs, as well as the corresponding Spike-containing pseudo-particles, except OC43, which is enhanced by IFITM3. We further demonstrate that overexpression of IFITMs impairs cell surface expression of ACE2 representing the entry receptor of SARS-CoVs and hCoV-NL63 but not hCoV-OC43. Our results explain the inhibitory effects of artificial IFITM overexpression on ACE2-tropic SARS-CoVs and show that three hCoVs, including major causative agents of severe respiratory disease, hijack IFITMs for efficient infection of human cells.