I think the assumption to work with is that Uther's Bastards (a knightly order if ever there was one, right?) are all unacknowledged, probably cared-for through his largesse, and technically lowborn. His swiving is a matter of widespread rumor, but done quietly both I think because of fear of his temper but also because he's the King (capital K) and that's just what's done when you're an unmarried king. The Church would disagree, but probably not to Uther and especially not after he's broken Gorlois. So all those bastards? Not enough to stake a claim to protecting the throne of Logres.
Now a high-born lady, even a queen like Ygraine, brings a measure of legitimacy through association and is a more reasonable target.
If there were legitimate, acknowledged issue from Uther or Madoc, then they would be in a higher category of claimant, at least on par with some of the extended family/powerful lord members of the Anarchy. Fun times.
Keep in mind as well that secret marriages are a thing. It entirely possible for the PK to claim--especially after Madoc's death--that their well-known affair was actually a private marriage and that was the reason Uther didn't mess with them. A few bribes to churchmen and maybe some winking at Uther and you're set.
Personally, because I'm using some related ideas to this thread, I like the "secret marriage to Madoc just outside of Terrabil" plot. The next morning is a hangover, of course.
But then I'm mean.
--Khanwulf
PS. Edit for Atgxtg: I don't think it's fair to make Uther chase anything in a skirt. He just doesn't find someone worthy of tying the knot with until Ygraine comes along, in the balance of beauty and station. But he does insist on having a bedwarmer. A pretty one. All the time. Deprived a few days and he'll find a replacement in short order; which is what happens when Ygraine rejects him.
Frankly I think Ygraine lost her charm immediately. I mean the chasing is better than the having, and she was all weepy, then all pregnant, and then bitchy (in GPC) about losing Arthur before she even named him. She was not super-young either and I suspect Uther had some serious buyer's remorse at having chained himself to such a strong woman (Valorous 19!?!). Her departure to the nunnery sounds like the end of an argument or few that, were she male, would have resulted in the king doing some stabbing.