There's a higher correlation of being black and poverty than some other races in the US. That higher poverty rate combined with some other social issues leads to lower levels of academic preparedness overall.
This is evident by the weights that universities used for affirmative action. Using white students as a basis for comparison, Asian students needed to get a higher SAT score, while black students had equal odds of admission with a lower score.
See also the number of k-12 schools with literally zero students performing at grade level, which tend to be concentrated in areas of higher poverty.
Assuming, as OP pointed out, that you want to bring in higher numbers of black students, there are two possibilities: for some reason, academically prepared students aren't going to college (less likely) and / or you need to lower the bar (more likely, and one I have personally witnessed).
It sounds racist because they phrased it very bluntly, but that aren't saying that black people can't do math. One of the reasons why Americans are bad at math is because our expectations of math are abysmal. Raising these expectations will reveal just how far behind black students are in math, which will get you accused of racism, so schools are incentivized to keep them low. One can argue that this is actually a form of systemic racism because the schools that are failing to properly educate black students are shielded from scrutiny.
