Turns out RefCOCO had a bunch of bad data inside of it. Not just with the masks, but also with the prompts like "slut on phone" and "bitch kid". RefCOCO-M cleans it up.