I would guess that your neighbor has a device that operates on 5 GHz that interferes with your WiFi. 5 GHz interference isn't too common even in RF congested areas unless your neighbor is using 5.8 GHz home phone for example.
I just tested mine and only got exactly 1 random spike to 44ms (probably router is busy as I have other 5 GHz devices streaming internet at the same time as I'm doing this test) so I don't consider that it's wireless card fault unless yours is a hardware fault
