Does the offset appear to change? ie. You do a loopback test, set the offset, run another pass to confirm it. But later it appears changed again? Something like that going on? Did you try a manual correction only with the box for 'use driver reported offset' unticked?
You may need to find different offsets depending on what sample rate you are running at. Putting multiple interfaces in aggregate device config can add to the baseline latency of the devices too. Some of the drum machine, etc devices connect like a USB audio interface. That leads to vetting their 'alignment' and especially if you are making aggregate device configs to keep your monitor system interface in the mix.
Or... If you were only relying on the 'use driver reported offset' and looking no further. Now you know what the manual offset is for!