# Machine Learning: Calibrating SubGroups of Probability Predictions inside a Dataset which should sum to 100%

I am working on an interesting type of problem where I want to make predicitons for individual elements within subgroups- with the knowledge that the sum of the probabilities within a subgroup should sum to 1. The initial response will probably be to use a softmax system but this is subtley different (unless I have misunderstood something).

I will use an analogy to keep it straighforward:

Effectively in a flowerpot there are 10 flowers- only 1 of which is certainly is a daffodil and the other 9 are certainly not- some are quite similar to daffodils while others are more similar to flavorsome herbs or thorny bushes.

I have a dataset of 100,000 flowers (which are in 10,000 flagged lower pots/subgroups). I know characteristics for each of the flowers such as stem height, petal colour, odour pleasure, diameter etc. as well it being flagged whether or not it is indeed a delightful daffodil.

I am able to generate a logistic ML model quite easily which informs me of the probability that a certain flower is a daffodil based on the data from the other flowers. But I am unable to get it to work so the sums of probabilities within sub groups equal one- generally they are close but often as low as 65% or as high as 130%.

My interim solution is to carry out a post prediction clean, which determines the total probability assigned to the group and normalises predictions so they sum to one. Not awful but not great.

Am I a moron? Is this a softmax problem? Or something that a ML algo in existence can handle?

Thank you, a daffodil to those who help :)

Let $$p_i=\Pr[F_i]$$ be the probability that flower $$i$$ is a daffodil, according to your model (without taking into account the group structure).

Now you are given a group of 10 flowers, with probabilities $$p_1,\dots,p_{10}$$, and you're given the promise that exactly one of them is a daffodil. To update your probabilities based on this additional promise, you can use Bayes rule. Let $$E$$ be the event that exactly one in the group is a daffodil. Then

\begin{align*} \Pr[F_i|E] &= {\Pr[F_i \land E] \over \Pr[E]}\\ &= {\Pr[F_i \land E] \over \sum_j \Pr[F_j \land E]}. \end{align*}

Note that

$$\Pr[F_i \land E] = \Pr[\neg F_1 \land \cdots \land \neg F_{i-1} \land F_i \land \neg F_{i+1} \land \cdots] = (1-p_1) (1-p_2) \cdots p_i \cdots (1-p_{10}) = {p_i \over 1-p_i} \prod_j (1-p_j).$$

Therefore,

$$\Pr[F_i|E] = {{p_i \over 1-p_i} \over \sum_j {p_j \over 1-p_j}}.$$

This is the probability you wanted to compute; so, use that.

• Interesting- the result is slightly different to dividing each initial sub group probability by the total initial sub group probability. I will do some testing to see if this cleaning method proves more predictive. A daffodil for you my dearest and longest friend :) Commented Dec 3, 2020 at 1:01