Is it a wide yoyo or undersized but wide? because these are the kind of yoyos that usually get this problem.
Try swapping the bearing, maybe put the axle in the opposite way. There isn’t much you can do unless you’re willing to do some risky modding.