Imagine we have two size $m$ sets of points $X,Y\subset \mathbb{R}^n$. What is (time) complexity of testing if they differ only by rotation?: there exists rotation matrix $OO^T=O^TO=I$ such that $X=OY$?
There is an issue of representing real values here - for simplicity assume that there is (a short) algebraic formula for each coordinate, such that cost of basic arithmetic operations can be assumed as O(1).
The basic question is if this problem is in P?
While at first view this problem might seem simple - usually it is sufficient to test norms of the points and local relations like angles, there are nasty examples where it is e.g. equivalent to graph isomorphism problem.
Specifically, looking at eigenspaces of adjacency matrix of strongly regular graphs (SRG), we can give it geometrical interpretation. Below is the simplest example - two 16 vertex SRGs, which locally look identical, but are not isomorphic:
Adjacency matrix of SRGs has always only three eigenvalues (of known formulas) - looking at eigenspace for eigenvalue 2 above (kernel of $A-2I$), it has dimension 6 - of basis written above. Orthonormalizing it (Gram-Schmidt), we get large space of possible orthonormal bases - differing by $O(6)$ rotation, which rotates "vertical vectors": 16 of length 6. Define such set of vectors as $X\subset \mathbb{R}^6$, $|X|=16$ here, and $Y$ correspondingly for the second graph - converting graph isomorphism question into question if $X$ and $Y$ differ only by rotation.
The difficulty is that all these points are in a sphere and recreate original relations: all neighbors (6 here) are in fixed angle <90 degrees, all non-neighbors (9 here) in another fixed angle >90 degrees, like in the schematic picture above.
So testing based on norm and local angles takes as back to graph isomorphism problem ... but geometric interpretation allows to work on global properties like rotation invariants.
Generally, a natural "global" approach is trying to describe both sets "modulo rotation" (which contains $n(n-1)/2$ degrees of freedom), and then just check if both descriptions are identical.
We can usually define rotation invariants - the question is constructing a complete set of rotation invaraints: completely determining a set modulo rotation.
While I couldn't find a way for practical rotation invariants directly working on points (?), it can be done for polynomials (stack). For degree 2 polynomial $x^T A x$, a complete basis of rotation invariants is e.g. $Tr(A^k)$ for $k=1,\ldots,n$. Diagrammatically they can be represented as length $k$ cycle, and we can analogously construct rotation invariants for higher order polynomials (the remaining question is their independence), e.g. each graph below corresponds to a single rotation invariant of degree 1,2,3,4 polynomial:
The question is how to describe a set of points with a polynomial - generally we need high degree polynomials, e.g. $p(z)=\prod_{x\in X} (x\cdot (z-x))$, but sets for SRGs are quite regular - can be described with just degree 6 polynomial:
$$p(z)=\sum_{x\in X} (x\cdot z -a)^2 (x\cdot z -b)^2 (x\cdot z -c)^2 $$ where $a,b,c$ describe norm and angles in sets obtained for given SRG (are known).
So can we test if two degree 6 polynomials differ only by rotation in polynomial time? If so, graph isomorphism for SRGs is in P.
Are there tougher examples (for testing if two sets differ only by rotation) than from SRGs? I doubt it, allowing for quasi-polynomial upper bound thanks to Babai (?)
Update: I was pointed similarity with (solved) orthogonal Procrustes problem:
$$\min_{O:O^TO=I} \|OA-B\|_F \quad\textrm{achieved for}\quad O=UV^T,\quad\textrm{ where}\quad BA^T=UDV^T$$
from singular value decomposition. We could construct these matrices from our points, however, it would require knowing the order - which we don't know and there are $m!$ possibilities.
We could try e.g. Monte-Carlo or genetic algorithm: switching some points and testing distance improvement using above formula, however, I suspect that such heuristic algorithm might have exponential number of local minima (?)