What Big-O Means
Big-O notation, or Landau Symbols, are not particular to any model of computation, or even to computer science. When used to illustrate the asymptotic complexity of an algorithm, they are always given relative to some model of an abstract machine, even if that abstraction isn't made explicit.
In texts on algorithms the model used is often a RAM machine, and usually is used without explicit acknowledgement (see for example the unqualified use of big-O notation on the wikipedia pages for Quicksort or Dijkstra's Algorithm, or the examples in the Table of Common Time Complexities).
On a different machine model, or on a real machine, the scaling properties will be different. With a suitable model of computation, one could derive the asymptotic performance and state it in big-O notation, and the result may be different to that for the RAM machine and therefore to commonly published big-O values.
For example: on a machine with sequential memory (a Turing Machine with a single tape, say), binary search no longer has a running time O(log n) - the commonly cited result for a RAM machine - because traversing the tape requires O(n) time.
As a second example, this chapter shows an explicit computational model described, and used to derive the asymptotic performance of various algorithms.
Realistic Performance Envelopes
Analysing the asymptotic complexity of an algorithm makes the assumption that the algorithm would still be viable regardless of the problem size. But ultimately, no real machine can cope with infinite size problems. So unless the algorithm is O(1) in time and space, there will always come a point beyond which the performance of the algorithm is limited by the capacity of the machine, rather than the theoretical complexity bound.
It would be perfectly possible to analyse the algorithm (using a realistic model of a resource-limited machine) to determine the maximum solvable problem size. And if the real machine went through different modes of computation at different problem sizes (for example, it stops being able to hold data in cache at a particular size, then stops being able to hold it in RAM, then runs out of disk space, etc), one could work out where these transitions would occur, and give a more detailed characterization of its performance envelope. These kinds of analysis might be important if you need to make guarantees about real-world performance, but are completely unrelated to asymptotic performance analysis.
There isn't a special big-O like notation for those results, you would just give the equations for the performance explicitly. Given that you are now dealing with explicit upper bounds, big-O notation would be inappropriate.