Seems to me that if you don't gain control soon," annoying" will turn to " aggressive" before you realize it. He's apparently learned how to irritate you to get his own way, try pushing back, and I don't mean with the flat of your hand. Use something pointier, like a knee or elbow, so he notices.