You’re not doing anything wrong there. It’s just a matter of those holes having larger reeds, so in essence it’s a larger channel and more air will pass through the harmonica in a quicker fashion, with the same amount of lung capacity.
So, you will always be able to play the higher holes for longer periods of time. Keep working on your breathing exercises, and chugs, and in time you will have those lower holes also up to around 10 -15 seconds, and more!! 🙂