[kpypiruu] Parentheses minimization with square brackets

Consider a tree structure written out as parentheses.  Replace some pairs of parentheses with square brackets.  For every closing square bracket, delete consecutive previous closing parenthesis: a square bracket is a "power" close paren that automatically closes all parenthesis up to the nearest open square bracket.

[((( () ((] is, for example, valid and is equivalent to (((( () (()) )))).

For a given tree, what is the optimal placement of square brackets to minimize the length of the string?

Is there an online algorithm to choose whether to use a opening square or round parenthesis, if you don't know the rest of the input?  (I'm guessing not.)

There is a Lisp IDE which implements a simpler variation: the key right-square-bracket inserts enough parentheses to close all parentheses up to the top level.