Step size is the whole game.
A large h measures the wrong neighborhood. A very small h eventually runs into floating-point subtraction. The interesting numerical question is not simply “make h smaller,” but how the approximation behaves as h changes.