Techniques for showing that problem is in hardness "limbo"

Question

Given a new problem in $\mathsf{NP}$ whose true complexity is somewhere between $\mathsf{P}$ and being NP-complete, there are two methods that I know of that might be used to prove that resolving this is difficult:

1. Show that the problem is GI-complete (GI = Graph Isomorphism)
2. Show that the problem is in $\mathsf{co-AM}$. By known results, such a result implies that if the problem is NP-complete, then PH collapses to the second level. For example, the famous protocol for Graph Nonisomorphism does exactly this.

Are there any other methods (maybe with different "strengths of belief") that have been used ? For any answer, an example of where it has actually been used is required: obviously there are many ways one might try to show this, but examples make the argument more convincing.

Answer 1

Showing that your problem is in coAM (or SZK) is indeed one of the main ways to adduce evidence for "hardness limbo." But besides that, there are several others:

• Show that your problem is in NP ∩ coNP. (Example: Factoring.)
• Show that your problem is solvable in quasipolynomial time. (Examples: VC dimension, approximating free games.)
• Show that your problem is no harder than inverting one-way functions or solving NP on average. (Examples: Lots of problems in cryptography.)
• Show that your problem reduces to (e.g.) Unique Games or Small-Set Expansion.
• Show that your problem is in BQP. (Example: Factoring, though of course that's also in NP ∩ coNP.)
• Rule out large classes of NP-completeness reductions. (Example: The Circuit Minimization Problem, studied by Kabanets and Cai.)

I'm sure there are others that I'm forgetting.

Answer 2

From the comment above: if a problem seems hard enough, but you are not able to prove that it is NP-complete, a quick check is to count the number of strings of length $n$ in the language: if the set is sparse it is unlikely to be NPC, otherwise P=NP by Mahaney's theorem ... so it's better to direct efforts towards proving that it is in P :-) :-)

An example is the problem of partitioning numbers into k-boxes (from Fortnow & Gasarch's blog, original source: Doctor Ecco's Cyberpuzzles):

$\{ (n,k) \mid \text{ there exists a way to partition }$ $\{1,...,n\} \text{ into at most k boxes so that no box has } x,y,z \text{ with } x+y=z \}$

Answer 3

Here are three additions to Scott's list:

• Show your problem is in fewP. This means that the number of solutions is bounded by some polynomial. (Example: Turnpike problem). No NP-complete problem is known to be in fewP. (impossible unless fewP=NP).
• Show your problem is in $LOGNP$ or in $NP[log^2n]$ (Can be solved using limited number of nondeterministic bits, Example Dominating set problem in tournaments)
• Show that your problem has sub-exponential density (H. Buhrman and J. M. Hitchcock proved density lower bound ($2^{n^{\epsilon}}$), unless the polynomial hierarchy collapses. Therefore, any $NP$-complete set must have some $\epsilon \gt 0$ such that for infinitely-many integers $n\ge 0$, the set contains at least $2^{n^{\epsilon}}$ strings of length $n$. ). This is a much stronger than proving just sparsity (as stated in Marzio's answer).

H. Buhrman and J. M. Hitchcock, NP-Hard Sets are Exponentially Dense Unless $coNP ⊆ NP/poly$, In IEEE Conference on Computational Complexity, pages 1–7, 2008

Answer 4

Show your problem is in $RP$ ( hence unlikely to be $NP$-hard)

Example:

Exact Perfect Matching problem

Input: A graph G, with each edge colored red or blue, and integer k.

Problem: Decide whether there exists a perfect matching M in G with exactly k red edges