Cross-reactive T cell immunity between common cold coronaviruses and SARS-CoV-2 may influence COVID-19 susceptibility. To identify cross-reactive CD8 T cell epitopes, we analysed responses to 21 homologous SARS-CoV-2 replicase peptides in 177 PLWH on antiretroviral therapy, of which 133 did not have prior SARS-CoV-2 infection. Replicase peptides induced IFN-γ responses in 63% of the SARS-CoV-2-naïve individuals and in 73% of individuals with prior SARS-CoV-2-infection. We could define several cross-reactive epitopes, including the HLA-B*35:03 restricted CoV-YL8, and characterized a CoV-YL8-specific T-cell receptor cloned from a SARS-CoV-2 seronegative individual. Analysis of the association between HLA-I alleles and SARS-CoV-2 infections over a 16-months period revealed that in a cohort of 452 PLWH, HLA-B*35:03 and C*07 were underrepresented in the 55 persons with a history of SARS-CoV-2 infection while HLA-B*35:01 and HLA-C*04 were associated with a higher infection rate. Taken together, our study suggests an HLA-I-mediated effect of common cold coronaviruses on SARS-CoV-2 immunity.