The cmp() function used to compare btree keys was using the
trick of comparing unsigned values by coercing the result
of subtraction to a signed type. This is fine as long as
the range of differences never actually exceeds the limits
of the signed type. For instance, with int64, as long as
the magnitude of the difference is under 2^63 or so, it
works reasonably well.
Plain int, however, can be a 32-bit type, at which point
the magnitude of difference needed to break it is only
2^31 or so.
Subtraction and type conversion is enough cheaper than
branches that this is worth preserving, but it's worth
preserving by switching to an explicit int64 for the
operation and return type. This will not actually affect
performance except for people using the btree code on
32-bit machines, so it probably won't ever matter.
Which may also be true of the potential wrong answers,
but "might be slower" is a better risk than "might
crash".
