Chosen Solution

I have the same problem yes it stops when unplugging controller and it also doesn’t do it when I use the Xbox app

This might be a drift issue. Can you run a controller calibration test on it? I’m not sure about the Xbox One but you can hook up your controller to a PC and run a test here: https://gamepad-tester.com/ That will show you if the sticks are registering any input in the neutral position.