I've only heard of 2-3 people having this problem. It's certainly not how the game should be. I'd suggest posting it as a bug in the Test Feedback channel. Include some basic info about your system, like CPU, GPU and especially what kind of mouse you use.