Bounded nondeterminism associates a function $g(n)$ with a class $C$ of languages accepted by resource-bounded deterministic Turing machines, to form a new class $g$-$C$. This class consists of those languages that are accepted by some nondeterministic Turing machine $M$ obeying the same resource bounds as are used to define $C$, but where $M$ is allowed to make at most $g(n)$ nondeterministic moves. (I am using the notation of Goldsmith, Levy and Mundhenk, instead of the original by Kintala and Fischer, and $n$ is the size of the input.)
My question:
Is there a constant $c\ge0$ such that GRAPH ISOMORPHISM is in $c\sqrt{n}$-$\mathsf{PTIME}$?
(Edit: Joshua Grochow pointed out that a positive answer to this question would imply an algorithm for GI that has better asymptotic runtime bounds than are currently known. I would therefore be happy to relax the bound, allowing $o(\sqrt{n}\log n)$ nondeterministic moves.)
Background
For every fixed constant $c \ge 0$, $\mathsf{PTIME} = {c\log n}$-$\mathsf{PTIME}$, as $c\log n$ nondeterministic moves create at most a polynomial number of configurations to explore deterministically. Moreover $\mathsf{NP} = \cup_c n^c\text{-}\mathsf{PTIME}$, and by means of padding one can exhibit NP-complete languages in $n^\varepsilon$-$\mathsf{P}$ for every $\varepsilon > 0$.
Kintala and Fischer observed that deciding if an input graph with vertices $V$ has a $(|V|/3)$-clique is $\mathsf{NP}$-complete, but is in $O(\sqrt{n})$-$\mathsf{PTIME}$. To see this, discard the vertices which have at most $|V|/3 - 2$ neighbours. If there are too few remaining vertices, then reject. Otherwise the remaining vertices form a graph of size $\Omega(|V|^2)$. Then guess a $|V|/3$-subset of vertices using $|V| = O(\sqrt{n})$ nondeterministic steps and verify that they form a clique in polynomial time.
Some other languages $L$ of dense graphs in $\mathsf{NP}$ are also in $O(\sqrt{n})$-$\mathsf{PTIME}$. This is the case for any problem where a subset of the vertices serves as a certificate, and the size of the input graph is $\Omega(|V|^2)$. Examples are the promise versions of Induced Path or 3-Colouring for the case of dense graphs. Other problems seem to require larger certificates, for instance a list of vertices defining a Hamiltonian circuit seems to require $\Omega(|V| \log |V|)$ bits. It is not clear to me whether one could use an amount of nondeterminism that is too small to guess the certificate to decide such problems.
Given that $n^\varepsilon$-$\mathsf{P}$ can contain NP-complete languages, it then seems interesting to ask where in the bounded nondeterminism hierarchy potentially easier languages fall. One might expect GI, as a language that does not seem to be NP-complete, to be in the hierarchy closer to $\log n$-$\mathsf{P}$ than to $n$-$\mathsf{P}$. However, the obvious certificate for GI specifies the map using $|V|\log |V|$ bits, which is $\omega(\sqrt{n})$.
Another way to think about this question: is specifying a map between the sets of vertices a shortest possible certificate for GI?
Edit: Some further (corrected) remarks follow, to address the comments of Joshua Grochow.
If a certificate uses $f(n) = \Omega(\log n)$ bits and can be checked in polynomial time, then brute force gives an algorithm for GI taking $poly(n)2^{O(f(n))} = 2^{O(f(n))}$ time. With a certificate of size $O(\sqrt{n})$, brute force gives an algorithm taking $2^{O(\sqrt{n})}$ time, while a certificate of size $O(\sqrt{n}\log n)$ yields a brute force approach taking $2^{O(\sqrt{n}\log n)}$ time. The long-standing upper bound of Luks is $2^{O(\sqrt{n\log n})}$ time, which is between these two bounds up to constant exponents.
These considerations suggest that there might be an alternative approach to GI. The approach of Luks seems to rely at its core on identifying a subset of generators of an associated group. A nondeterministic machine might therefore guess a subset of the group. These subsets could then be checked exhaustively to yield a deterministic algorithm. If the list of elements can be specified succinctly, either because the associated group is never much larger than the size of the graph, or because the number of generators required is always small, and checking each candidate subset does not take too long, then this might yield an alternative approach to GI.
- Chandra M. R. Kintala and Patrick C. Fischer, Refining Nondeterminism in Relativized Polynomial-Time Bounded Computation, SIAM Journal on Computing 9(1), 46–53, 1980. doi:10.1137/0209003
- Judy Goldsmith, Matthew A. Levy, Martin Mundhenk, Limited nondeterminism, SIGACT News 27(2), 20–29, 1996. doi:10.1145/235767.235769
- László Babai and Eugene M. Luks, Canonical Labeling of Graphs, STOC 1983, 171–183. doi:10.1145/800061.808746