Are you saying that the lowest gain knob position for those xlr inputs is still higher than unity gain? There's no way to connect a balanced line level input to that xlr without attenuating it first?
XLR inputs will often have the lowest level = line level but not every piece of gear does that. Some mic inputs are already boosted at the lowest input level - being that they were made to connect microphones to and all.
Getting a whole additional audio interface or external ADC unit to do your AD conversion and output to SPDIF so you can continue to use the Tascam? Shit, I'd just take this as an excuse to upgrade that interface altogether and get something with better AD converters and proper line level balanced inputs!
A used Apogee Duet (1st gen) has Apogee ADC's and sells for as low as $50 on Ebay, for one example. You'd notice the AD stage improvement over that Tascam. You're not going to find a stand-alone ADC with SPDIF with that good of AD converters for that. Oh, and you'd finally have a pair of balanced outputs too.
PS. There's no latency difference between a hardware send on a track used as a source track vs a track used as a hardware output bus. Your latency is determined by the block size and the baseline latency of the hardware audio interface you connect.