Is there any way to make my metal drifter unresponsive?

I would clean the bearing very thoroughly, spray it out with compressed air, leave it dry, then remove one response pad.
If you take the response pad off neatly enough, you can put it on some wax paper or something and save it for later! Yes, I’m a super cheap yoyoer.