Deterministic context-free languages (DCFL) and visibly pushdown languages (VPL) are both sets of formal languages between context-free languages (CFL) and regular languages (REG). Is there a readable notation that can be expressed in plain ASCII like Backus-Naur-Form for CFL and regular expressions for REG?
Regarding DCFLs, I do not see a better notation than the transition function of the deterministic pushdown automaton, i.e. writing explicitly rules of form $q,z,a\to q',\gamma$ with $q,q'$ states in $Q$, $z$ a stack symbol, $\gamma$ a sequence of stack symbols, and $a$ an input symbol or the empty string. The notation itself does not enforce determinism, but it is easily checked. Using a context-free grammar kind of notation (as BNF), you are going to run into problems since DCFLs are a proper subclass of CFLs, and as noted by DaniCL, you cannot decide in general given a CFG whether its language is deterministic.
Regarding VPLs, a bracketed/parenthesis-style for CFGs would be good enough, with rules of form $A\to a\alpha b$ where $A$ is a nonterminal, $a$ a call symbol, $b$ a return symbol, and $\alpha$ a
sequence of regular expression over mixed internal symbols and nonterminals. Since any VPL is also a (D)CFL you could also reuse the above notation for pushdown automata and check that the stack operations match the calls and returns, or write down the transition relation of a nested word automaton (which would be less redundant).
Edit: come to think of it, a notation for XML schemas, like the compact syntax of RelaxNG---which is an ASCII notation---, could easily be used for VPLs. You'd just need to enforce some naming conventions for tags, e.g. an opening tag "<ab>" for a call symbol $a$ matching a closing tag "</ab>" for a return symbol $b$.
In order to find a canonical representation, consider the following: the class of DCFL is equivalent to the class of languages generated by LR(k) grammars, which again is equivalent to LR(1). This means that you can find an LR(1) grammar for every DCFL. Of course, a LR(1) grammar is still a context-free grammar, but with a special property: from LR(1) grammars, we can easily build parse tables to guide a deterministic parser (with lookahead of 1 symbol, hence LR(1)). These parse tables would be another representation, albeit somewhat less readable.
By the way, mind that it is undecidable whether a given context-free language is deterministic (Greibach's Theorem).
I must admit I have never heard of VPL.