Among fundamental questions in sexual reproduction, two stand out: what are the inheritance modes of fitness, the ultimate evolutionary trait, from parents to offspring; and do parents exhibit differential mating affinities? Tools of quantitative genetics are required to address these questions, and they demand large collections of diverse phenotyped-genotyped strains from the same species. Here we leverage sexual mating among nearly 100 natural isolates of the yeast S. cerevisiae, capable of generating thousands of offspring combinations, in several growth conditions. Using genomic barcoding and a barcode recombination technique to track mating events, we measure fitness for all parents and offspring across different conditions. We then quantify the inheritance of fitness upon sexual reproduction in each growth condition. We find that offspring fitness in a fermentable carbon source (glucose) correlates positively, yet modestly, with parental fitness, while on non-fermentable carbon (glycerol), offspring fitness is not detectably correlated with parental fitness. Instead, in the non-fermentable condition, the fitness of offspring increases sharply with genetic distance between their parents, suggesting that outbreeding maximizes offspring fitness irrespective of parental fitness in that condition. Additionally, the number of minor alleles in the genome negatively correlates with offspring fitness in both conditions. Fitness inheritance can be explained by simple dominance or co-dominance modes of inheritance, in the non-fermentable and fermentable conditions respectively. Finally, by quantifying mate statistics between all parental pairs, we show that mate affinity is an active factor in yeast, with certain parental pairs exhibiting heightened affinity for each other, whereas certain strain combinations are avoided. Our study provides new resources for studying quantitative genetics, unprecedented in their extent, and reveals factors that mediate fitness inheritance.