The resolution of SARS-CoV-2 replication hinges on cell-mediated immunity, wherein CD8+ T cells play a vital role. Nonetheless, the characterization of the specificity and TCR composition of CD8+ T cells targeting non-spike protein of SARS-CoV-2 before and after infection remains incomplete. Here, we analyzed CD8+ T cells recognizing six epitopes from the SARS-CoV-2 nucleocapsid (N) protein and found that SARS-CoV-2 infection slightly increased the frequencies of N-recognizing CD8+ T cells but significantly enhanced activation-induced proliferation compared to that of the uninfected donors. The frequencies of N-specific CD8+ T cells and their proliferative response to stimulation did not decrease over one year. We identified the N222-230 peptide (LLLDRLNQL, referred to as LLL thereafter) as a dominant epitope that elicited the greatest proliferative response from both convalescent and uninfected donors. Single-cell sequencing of T cell receptors (TCR) from LLL-specific CD8+ T cells revealed highly restricted Vα gene usage (TRAV12-2) with limited CDR3α motifs, supported by structural characterization of the TCR-LLL-HLA-A2 complex. Lastly, transcriptome analysis of LLL-specific CD8+ T cells from donors who had expansion (expanders) or no expansion (non-expanders) after in vitro stimulation identified increased chromatin modification and innate immune functions of CD8+ T cells in non-expanders. These results suggests that SARS-CoV-2 infection induces LLL-specific CD8+ T cell responses with a restricted TCR repertoire.