Reckon it might need splitting into two separate code blocks, one to process midi inputs, one to process the OSC feedback from Reaper with a single flag UIopen that tracks the GUI state.
Loosely in pseudo code something like
Code:
@midimsg
if button is pressed
if UIopen
close VSTi GUI
else
open VSTi GUI
@oscmsg
if /fxinst/openui
if oscparm(0,f) == 0
UIopen=false // it is closed
if oscparm(0,f) == 1
UIopen=true // it is open