System call instrumentation on Linux/x86-64 using memory-indirect calls (in vain?), part one

The approach also behaves unpredictably if buggy code invokes a system call with a high value in %rax, whereas the kernel would fail cleanly (with ENOSYS).

This isn't exactly true. The value in rax is sign-extended from 32 bits in the kernel's syscall entry point, so rax=2**32+1 and rax=1 are equivalent. As far as I know, nobody uses this feature, though :)