Generally spealing, the definition of a cryptographic protocol consists of two different parts: syntax and security.
The syntax specifies the functionality of the protocol under legitimate use (dealing with issues such as key generation, correct decryption, valid signature verification, or more generally some desired output). A protocol can be totally "insecure" and still satisfy the syntax of the cryptographic task at hand.
The security part deals with guaranteeing that the protocol can be used safely in a cryptographic context. It specifies the access that the adversary has to the protocol, as well as what it means for the adversary to break it.
It is desirable that a security definition carries some "semantics" with it, in the sense that if a protocol satisfies this definition then the user is convinced that the security guarantee is meaningful (e.g. having a security definition that allows any protocol is certainly "legitimate" but it clearly doesn't guarantee any security).
The biggest conceptual contribution of modern cryptography is to develop a methodology for coming up with security definitions that are extremely meaningful and at the same time realizable (see Goldwasser Micali, Goldreich, Goldwasser, Micali and Goldwasser, Micali, Rivest for prime examples of this methodology).
Following the works mentioned above it has become common (some would say mandatory) practice to define both syntax and security and to prove that a given protocol satisfies the given definitions (usually under some widely accepted intractability assumptions). The precise definitions to be satisfied depend on the cryptographic task at hand, and are evaluated in light of the intended application.
As Sadeq points out in his answer, the general syntax of protocols is defined via interactive Turing Machines (by Goldwasser Micali Rackoff). This definition allows to model players that "keep state" between messages that are sent and received. The GMR paper is also the first to rigorously define security for interactive protocols, and in particular what it means for a protocol to be zero-knowledge. More general security requirements are given in later papers on secure two and multi-party computation. For references to these see Sadeq's answer.