Short Version
Suppose that you want to consider a model of quantum computation in which the gates used in the circuits may depend on the input size. Are there pitfalls to avoid when defining the circuits, to avoid obtaining unreasonable computational power from combining the coefficients themselves? Conversely: what limitations (aside from the conservative one of having a gate set of size $O(1)$ for the entire circuit family) yield a model whose complexity is still reasonable, and in particular does not disrupt too many of the things which we know hold for gate-sets of size $O(1)$?
Motivation and Details
The class $\def\EQP{\mathsf{EQP}}\EQP$ is the set of languages which can be decided with certainty by a polynomial-size uniform circuit family. Except — because of the fact that there is no exactly universal gate-set for quantum computation, the gates allowed may change from problem to problem. And it would seem that people remain unsure of whether to allow infinite gate-sets, so that the gates which are used in each circuit for a problem may depend on the input size.
It seems reasonable to me, in principle, to allow a unitary circuit family to use an infinite gate-set: for instance, a set in which the different gates used by the circuit on inputs of length $n$, grow as $O(n)$ rather than the scenario of $O(1)$ which is possible with (approximately) universal gate-sets. Gate-sets which scale as $O(n)$ are part of classical circuit complexity, after all. Furthermore, it's not clear to me that it should be necessary to allow a gate set of size larger than $O(n^2)$ — which would allow for a constant number of gate-sets having arity ranging from constant to the entire system. For many purposes, it may even be enough to have a gate-set of size $O(1)$ for each circuit size, but a different such set for each $n$ — so that one may describe the gate set used by the family as being $O(n)$ for the first $n$ circuit sizes.
However, allowing even a single distinct gate for each input size allows you to do strange things, even if you require that the coefficients of the gates be nicely behaved. For instance, you could allow as a gate a controlled-QFT gate $\Lambda\mathrm{QFT}$, acting on $2n$ qubits, which is defined on $\def\ket#1{|#1\rangle}\ket{m}\ket{x}$ for $m,x \in \{0,1\}^n$ by $$\begin{aligned} \Lambda\mathrm{QFT} \ket{m}\ket{x} &= \ket{m} \otimes \mathrm{QFT}_m\ket{x} \\[1ex]&= \begin{cases} \tfrac{1}{\sqrt m} \sum\limits_{k=0}^{m-1} \mathrm{e}^{2\pi i kx/m} \ket{m}\ket{k}, & \text{if $0 \leqslant x < m$}; \\[2ex] \qquad\ket{m}\ket{x}, &\text{otherwise}.\end{cases} \end{aligned}$$ This effectively allows you to simulate gates which not only depend on the input size, but on the input itself. In addition to being strange in itself, and making it potentially difficult to obtain a meaningful notion of complexity (how much work is being absorbed into the machine which describes the gates?), it would undo classic results in quantum computational complexity such as $\mathsf{EQP \subseteq LWPP}$ [Fortnow+Rogers-1999]. Admittedly, that result relies on a definition of $\EQP$ in terms of quantum Turing machines rather than circuits, so inherently depends on the gate-set of a circuit being finite; it would not be shocking to negate it, but negating it is a warning sign that we may be doing something unreasonable. It would still be nice to find a way to keep it intact.
There is a potentially more awkward problem, however, which I am less well-equipped to consider myself.
If we study $\EQP$ out of interest in theory of computation (as opposed to practical application, where $\mathsf{BQP}$ will be more relevant for the foreseeable future), it doesn't make sense to consider coefficients which are themselves only computable to finite precision. Instead we should probably consider coefficients which are expressed symbolically as algebraic numbers: that is, where each individual gate has coefficients drawn from some finite extension of $\mathbb Q$. This picture of quantum computation is preicsely what the classic results $\mathsf{BQP \subseteq PP}$, $\mathsf{EQP \subseteq LWPP}$, and $\mathsf{NQP = coC_=P}$ rely on, so I'm quite comfortable allowing the coefficients be specified abstractly and "symbolically", so that the only precision involved is the length of the representation of a polynomial of which a given coefficient is a root, or something similar.
The problem is this: completely aside from whether or not you can hide non-trivial polynomial amounts of computation in the computation of the coefficients, I worry that the linear combinations of these algebraic coefficients, arising from the matrix multiplication itself, might itself hide non-trivial amounts of work by simulating a Blum-Shub-Smale machine. I'm not very familiar with the issues involved in that, so I wonder whether allowing infinite gate-sets might be a bit of a minefield.
Question #1. What theoretical problems (as opposed to practical problems, i.e. let us not concern ourselves any more with experimental implementation than we would with the polynomial hierarchy) does one have to worry about — for bounded error as well as for exact quantum computation — if one considers infinite gate-sets allowing the simulation of complicated arithmetic over the reals?
Question #2. What are the sufficient conditions for an infinite gate set to be "benign", from a standpoint of computational complexity?