How come I'm 5'7" and 19, and my brother is 16 and 5'11?

On my mom's side of the family the average for males is pretty short, around 5'6", and on my dad's side the average ranges from 5'9"-6'0. I am the more athletic brother, and even played basketball on a daily basis, while my brother spends most of his time on the computer...Yet he has grown considerably taller than I. My father is 5'9" and my mother is 5'3". Both of my grandfathers were fairly short in stature at around 5'6". Is it just the fact that my brother might have inherited the "taller genes" from my fathers side, and that I inherited the "short" gene? Is there anything I could do about it, it really bugs me and people bring it up ALL the time, and I mean EVERY SINGLE TIME. And I'm just sick of it. I weigh about 132 pounds, and my brother about 127. He's skinny and lanky which gives him the illusion of looking EVEN taller. I am lean, but have packed on considerable muscle, which I feel at least eases some stress caused by the height issue. Is there anything I can do about it to grow, or is it all blatantly summed up to "genetics"?