Getting Rid Off Mic Delay

[radio42]

As you correctly mentioned, latency is key here! See here for a detailed explanation: https://en.wikipedia.org/wiki/Latency_(audio)
A latency of around 10ms or below is typically not noticeable.

The most important part is the audio driver, a good audio card typically comes with an ASIO driver, which ensures a really good latency around 10ms or below, for example as provided by professional equipment.

When using the standard WDM or WASAPI driver model (esp. in shared mode) you are typically at around 30ms or above; when adding additional things like the voicemeter driver in between might add more milliseconds on top.
You recognize this by the noticeable delay of your MIC in your headphone.

A solution, if your equipment doesn't come with its own ASIO driver might be https://www.asio4all.org

Even if the WASAPI driver in exclusive mode might reduce the latency, but I rarely found a system performing as good as ASIO on a PC - but give it a try, as in WASAPI mode you can configure the exclusive mode and play with the Buffer size and update Period.
Note, that WDM isn't even supported anymore on Windows 10, but is emulated by WASAPI here in shared mode, which typically results in around 50ms.

[radio42]

I assume, that this is just by coincident and the way you are testing resp. your mixer was setup initially.
Talkover and Talkuser doesnt touch any buffer or recording/inout processing!
Instead, the Input mixer channel is simply unmuted resp. muted.

Over time the input and the output buffer will get filled up during digital processing.

As said, please use ASIO for low latency results and even try without any additional virtual sound device, like Voicemeeter!