When the toolbar action is run by double click or via the Run button.
It only gets updated after toggling another toolbar or after closing and reopening the Actions list.
Neither manual closing or opening a toolbar in Arrange makes its toggle state update in the open Actions list.
It's also annoying that at click on the action the list auto-scrolls up.