Let me provide you with an algorithm for recursively constructing an infinite state machine to decide any language $L \subseteq \{0,1\}^\ast$ that you like.
- Make the initial state accept if the empty string is in the language.
- Create two states for the strings 0 and 1, which the initial state branches to depending on whether the first symbol is 0 or 1. Label these $q_0$ and $q_1$. If either of the strings 0 or 1 are in $L$, make the corresponding state accept.
- For a given $n \geqslant 1$, and for each state $q_x$ for $x \in \{0,1\}^n$, create two states $q_{x0}$ and $q_{x1}$, to which the state $q_x$ branches depending on whether the next symbol is 0 or 1. If either $x0 \in L$ or $x1 \in L$ is in the language, make the corresponding state accept. Having performed this for all $x \in \{0,1\}^n$, proceed to the next value of $n$, ad infinitum.
This produces an infinite state machine which decides the language $L$. In particular, it decides $L$ regardless of whether $L$ is even computable, and even regardless of its position in the arithmetic hierarchy. So not only can you have infinite state machines which have power intermediate between FSMs and Turing machines, you can make it have as much or as little power as you like.
As to algorithms, note that the ISMs constructed this way have the same structure for any language: an infinite balanced binary tree. It in essence also represents the language directly, in that it effectively is equivalent to an infinite boolean string representing the characteristic function of the language $L$. As such, the subset of these ISMs which represent regular languages is not recursively enumerable: there is no way to conclude after a finite number of steps that the language recognised is regular, because for any candidate language that the ISM recognises, there could always be some exception — but the ISM might still represent some more complicated regular expression, so neither can one conclude in a finite number of steps that the language it decides is not a regular language. The complete lack of compression in the description of the language (which is necessary to be able to represent arbitrary languages using a common structure) makes it effectively impossible to decide such problems.
