Arthropod-borne viruses are a major cause of global viral infections, displaying evolutionary dynamics that differ significantly from vertebrate-specific pathogens. To investigate evolutionary constraints across host environments, we conducted deep mutational scanning (DMS) on the Envelope (E) protein of West Nile virus (WNV), a model arbovirus. Our findings reveal a high degree of constraint in WNV E. While overall tolerance to substitutions is higher in mosquito cells, fitness between mosquito, avian, and human cells is highly correlated. We use our data to explore the extent of antagonistic pleiotropy, or fitness tradeoffs, between the three host environments. We highlight distinct hotspots where tolerance to amino acid substitutions differs, finding that avian cells exhibit particularly distinct mutational tolerance. By examining the distribution of mutational fitness effects of naturally occurring amino acid substitutions, we gain insight into which substitutions can theoretically persist across both mosquito and vertebrate environments. This understanding is crucial for connecting DMS results to the evolutionary potential of arboviruses. Finally, we demonstrate the utility of WNV E DMS in explaining the natural diversity of highly dissimilar flavivirus pathogens. We identify specific sites in each clade of flaviviruses that are not well predicted by our mutational data, due to functional and genetic diversification of E proteins over long-term evolution. This work bridges a key gap in understanding the molecular constraints maintaining infection in distinct host environments. Further, we highlight specific sites in the E protein that require further study to understand the dynamic fitness landscape navigated by the genus Flavivirus.