First of all, in order to avoid confusion with the parenthesis found in a regular expression, I will use the characters $[$ and $]$ instead of parenthesis in my strings.
I can't tell from the question whether you want the strings in $L$ to be validly parenthesized.
One way I an interpret your question is that you want the strings of your language to be validly parenthesized. You can then use the following regular expression:
(a | b | [(a | b)*])*
The other way I can interpret your question is that the strings of your language do not have to be validly parenthesized; for example, $aba[b \in L$. Under this interpretation, the reason that the string $[a[a$ is not in $L$ is that it contains two open parenthesis in a row (ignoring $a$s and $b$s). The rule for $L$ with this interpretation is that a string is in $L$ if and only if removing all the $a$s and $b$s results in an element of
(ε | ])()*(ε | [).
In this case, simply inserting $a$s and $b$s everywhere gives us the regular expression
(a | b)*(ε | ])((a | b)*[(a | b)*])*(a | b)*(ε | [)(a | b)*.