During the COVID-19 pandemic, SARS-CoV-2 infected millions of people and claimed hundreds of thousands of lives. Virus entry into cells depends on the receptor binding domain (RBD) of the SARS-CoV-2 spike protein (S). Although there is no vaccine, it is likely that antibodies will be essential for protection. However, little is known about the human antibody response to SARS-CoV-21-5. Here we report on 149 COVID-19 convalescent individuals. Plasmas collected an average of 39 days after the onset of symptoms had variable half-maximal neutralizing titers ranging from undetectable in 33% to below 1:1000 in 79%, while only 1% showed titers >1:5000. Antibody cloning revealed expanded clones of RBD-specific memory B cells expressing closely related antibodies in different individuals. Despite low plasma titers, antibodies to three distinct epitopes on RBD neutralized at half-maximal inhibitory concentrations (IC50s) as low as single digit ng/mL. Thus, most convalescent plasmas obtained from individuals who recover from COVID-19 do not contain high levels of neutralizing activity. Nevertheless, rare but recurring RBD-specific antibodies with potent antiviral activity were found in all individuals tested, suggesting that a vaccine designed to elicit such antibodies could be broadly effective.