Here is for the benefit of others with the same issue:
I kept at it, as Mitch and Ian said, and their advice certainly helped. But I also noticed after a while that, without knowing it initially, I am/was drawing harder than I was blowing. So if you also slowly fill up with air, maybe you are doing the same.
I notice that when I pay close attention, I can blow more air without really increasing the volume. And that helps balance my intake and output.