Host restriction factors constitute a formidable barrier for viral replication to which many viruses have evolved counter-measures. Human SAMD9, a tumor suppressor and a restriction factor for poxviruses in cell lines, is antagonized by two classes of poxvirus proteins, represented by vaccinia virus (VACV) K1 and C7. A paralog of SAMD9, SAMD9L, is also encoded by some mammals, while only one of two paralogs is retained by others. Here, we show that SAMD9L functions similarly to SAMD9 as a restriction factor and that the two paralogs form a critical host barrier that poxviruses must overcome to establish infection. In mice, which naturally lack SAMD9, overcoming SAMD9L restriction with viral inhibitors is essential for poxvirus replication and pathogenesis. While a VACV deleted of both K1 and C7 (vK1L^-C7L^-) was restricted by mouse cells and highly attenuated in mice, its replication and virulence were completely restored in SAMD9L^-/- mice. In humans, both SAMD9 and SAMD9L are poxvirus restriction factors, although the latter requires interferon induction in many cell types. While knockout of SAMD9 with Crispr-Cas9 was sufficient for abolishing the restriction for vK1L^-C7L^- in many human cells, knockout of both paralogs was required for abolishing the restriction in interferon-treated cells. Both paralogs are antagonized by VACV K1, C7 and C7 homologs from diverse mammalian poxviruses, but mouse SAMD9L is resistant to the C7 homolog encoded by a group of poxviruses with a narrow host range in ruminants, indicating that host species-specific difference in SAMD9/SAMD9L genes serves as a barrier for cross-species poxvirus transmission.