This is not an answer. It is a simple but long observation. I hope it will be useful.
The decision version of your problem is: Does $\cal X$ contain a subset of $A$?
This problem is related to the problem of evaluating monotone boolean functions of $n$ variables. A subset of $\{1,\ldots,n\}$ is equivalent to an $n$-bitstring, so the family $\cal X$ is equivalent to a boolean function $f$ of $n$ variables. Given a function $f$, one can define the least monotone function that is not bigger than $f$, namely $g(y)=(\exists x\subseteq y,\,f(x))$. The original problem is then reduced to evaluating $g(A)$. Conversely, the problem of evaluating a monotone boolean function can be reduced to the original problem, either naively by taking $f=g$ or by choosing an $f$ that makes $\cal X$ smaller.
In practice BDDs tend to work well. So one possible approach is to build the BDD for $f$, derive from it the BDD for $g$, and then evaluate $g$. The average size of the BDD for $g$ must be $\Omega\left(\binom{n}{n/2}\right)$, because there are many monotone boolean functions. Hence, in theory this is a bad solution.
But (1) a better analysis might be possible and (2) there might be tweaks to this approach that make it better. For example, I didn't use in any way the correlation between the size of $\cal X$ and the size of $g$'s BDD. (There must be a correlation, but I don't know if it is simple or usable here.)
For completeness, a simple algorithm for computing the BDD for $g$ from the BDD for $f$ is the following.
$$m(x?f_1:f_0)=x?(m(f_0)\lor m(f_1)):m(f_0)$$
Here $\lor$ is the standard or-operation on BDDs.