Hormone Therapy and Ipilimumab in Treating Patients With Advanced Prostate Cancer

Inclusion Criteria: * NOTE: All values must be obtained =\< 14 prior to study entry * Histologically confirmed adenocarcinoma of the prostate staged within 180 days of study enrollment, \>cT2cN0/M0 stage with or without metastatic disease, with the exclusion of central nervous system (CNS) metastases; includes post radical prostatectomy patients with a rising PSA * An initial PSA \>= 4.0 ng/mL (Hybritech Assay) * For those patients who have received hormone therapy =\< 21 days, a documented PSA of \>= 4.0 prior to initiation of hormone therapy is acceptable. * For patients who are post radical prostatectomy, a rising PSA is acceptable. * Adequate organ function defined as: WBC \>= 3,000/uL; platelets \>= 75,000/uL; total bilirubin =\< 1.5 mg/dL; transaminases =\< 2.5 x upper limit of normal (ULN); serum creatine =\< 2.0 mg/dL or calculated creatinine clearance \>= 60 mL/min * ECOG performance status of 0-2 * Able to understand and sign informed consent Exclusion Criteria: * Underlying other serious medical condition which, in the opinion of the investigator precludes study participation; this includes immune-suppressive disease such as AIDS or autoimmune disorders such as multiple sclerosis, lupus, or myasthenia gravis * Patients not recovered from major infections and/or surgical procedures * Prior hormonal therapy \> 21 days prior to enrollment, including estrogens, LH/RH agonists, or antiandrogens * Recent (=\< 3 months of informed consent) usage of immune-suppressive medication including steroids, Immuran, Cyclosporin; topical or inhalational steroid use is permissible * Prior systemic chemotherapy * Prior radiation therapy to the prostate * Prior malignancy, unless the patient has been cancer-free for five years or more * Uncontrolled underlying medical or psychiatric illness, or serious active infections * Patient unwilling to complete all required follow-up visits * History of motor neuropathy considered of the autoimmune origin (e.g. Guillian-Barre Syndrome) * Concurrent malignancy, except for adequately treated basal cell or squamous cell skin cancer * For patients who elect to undergo the baseline transrectal needle biopsy of the prostate, current usage of systemic anticoagulation therapy, i.e. heparin or Coumadin or inability to discontinue aspirin, aspirin-containing products or ibuprofen for seven days prior to the prostate biopsies required for this study * No other investigational drugs will be allowed during the study * Other chemotherapy, radiation therapy, immunotherapy, hormonal therapy, or biologic therapy may not be used while the patient is on study