test it early morning when your neighbor peeps are sleeping (or bulk traffic torrenting - QoS usually knocks them back down pretty quickly). at the least you are looking for rock solid ping times. if pings are wild your link or the community bandwidth is possibility saturated.
if you really want a deep dive (cuz your post suggests that), fire up wireshark on a speed test PC and capture the traffic while you test. look for out-of-order, missing and corrupted packets. you will also get awesome stats on the traffic - wireshark is your best friend. be warned, this is the entrance to the rabbit hole.
edit: because at some point you are going to want to slide wireshark between the cable modem and your router - for general troubleshooting (and funsies!) then things get interesting as you figure out how to do that properly.