[qqokyeyk] Separate but equal

One curious precursor to Brown vs Board was that even though Plessy vs Ferguson mandated "separate but equal", de facto conditions were very clearly not equal.  Why was this legally tolerated?  If an activist judge wanted to make a point, whether at the Supreme Court or any lower level, it seems it should have been easy to strike down whatever contorted legal definition of "equal" permitted such obvious inequality.

Is there a fundamental flaw in the layers of the court system that such a ruling could not occur?  Perhaps something like appellate courts cannot question trial court findings of fact about equality?  Was the only way the Supreme Court could rule for more equality to overturn Plessy?  Could they instead have ruled that Plessy remain law, but has been incorrectly applied in this case, that the conditions were clearly not "equal"?