I essentially agree with Martin's comment, I can elaborate on that to make a tentative answer, knowing that there is no general formal definition of calculus or abstract machine and that what I am going to describe cannot possibly cover the meaning of all instances of these two words found in the literature.
In brief: a calculus usually gives you the abstract spefication of the meaning of programs, whereas an abstract machine usually implements that specification. Such an implementation is likely to be still high-level (i.e., many low-level details are not specified), hence the adjective "abstract", but it gets closer to what a phyisical machine would do to execute programs (according to the spefication given by the calculus).
More in detail: a calculus usually comes with an operational semantics, which gives you the meaning of programs in terms of the result they denote (if any). For this, one often uses the notation
$t\Downarrow v$,
which means that "the value (i.e., the final result) of the expression $t$ is $v$". Now, such a "big step" operational semantics (as it is sometimes referred to) is usually given in terms of a derivation system (i.e., a system in which you can prove judgments of the form $t\Downarrow v$), which is useful to reason abstractly about programs but is a bit far from describing how the execution of $t$ would look like on a concrete machine.
One may then move to a "small step" operational semantics, which is a set of rewriting rules on expressions (written $t\rightarrow t'$) describing how one can, step by step, compute the value of an expression (if any):
$t\Downarrow v\quad$ iff $\quad t\rightarrow^\ast v$.
Now, even the small-step semantics may be too coarse: typically, some rewriting rules may duplicate arbitrary big sub-expressions, and a chain of duplications may cause an exponential blow-up of the size of expressions in only a linear number of steps, which means that these steps are not so "small" after all... In particular, one may want to give a more realistic description of the execution of programs, getting even closer to the machine level. This is where abstract machines come into the picture.
It is impossible to give a general description of an abstract machine, since they may differ greatly. However, these usually include low-level components (e.g. a pointer to a "code memory" where the executed program is stored, a stack, a memory for storing the values of variables, etc.) which make them look much more like physical machines executing the programs of the language underlying the original calculus.
About where to "draw the line" between calculi and machines: this is actually a tricky question. There are calculi whose small-step semantics is more or less the same thing as an abstract machine. Lambda-calculi with explicit substitutions are examples of this: in such calculi, expressions contain constructs (the explicit substitutions) which make it possible for the small-step semantics to operate at a lower level, much closer to that of the machine.
About your reasoning on how the levels of expression of an algorithm are "layered", I am not sure I follow it, so I cannot say much. But I hope that at least I gave you some elements of answer to your last two questions.
