Chad Solution : Fight with him, show them who is the boss, be gentle!
But the most important thing is to stand for yourself , self respect is very much important . I can understand that you are a quiet,shy kind of a guy but always remember one thing that if you can't take a stand for yourself then why do you think others will take a stand for you. Directly confront him , not forcing you but you should think about it . Learn to value yourself , as i said earlier that i dont have any dating experience and it is simply not needed i used my experience of life!
Tell him "I'm sure you could handle another girl"