Program death receptor-1 (PD-1) and program death receptor-1 ligand (PD-L1) inhibitors are increasingly being used in the clinic to treat a growing number of malignancies, including many genitourinary (GU) malignancies. These immune-based therapies have demonstrated a distinct toxicity profile compared to traditional chemotherapy and the targeted therapies directed at the vascular endothelial growth factor pathway or the mammalian target of rapamycin pathway. Autoimmune toxicity targeting the skin, gastrointestinal tract, or the endocrine organs are some of the more common adverse events (AEs) noted with these therapies. Here in, we report the results of a systematic review of the incidence of toxicities in GU cancers reported in the phase II or phase III clinical trials using single-agent PD-1 or PD-L1 inhibitors. Overall, the rate of serious (grades 3–4) AEs was noted in approximately 15% of patients. The AEs noted were similar between all the agents tested, highlighting the overall class effect of these therapies. The incidence in GU cancers is similar to those seen in other malignancies. Given the widespread and high volume real-world use of these agents, it is important for oncologists to be familiar with these side effects to minimize the risks for patients while undergoing therapy.