OK, I've been thinking a bit more, and I've come up with some weaknesses. Not holes, exactly, but I could be doing a couple things a bit more securely.
1) Don't encrypt the nonce with a private key. Anyone with the corresponding public key can decrypt it then. Instead, encrypt it with the other machine's public key. Then, you need the corresponding private key to be able to figure out the value.
2) Instead of adding a fixed string in there to test the signature (because that's effectively all it was), I should be generating the signature, and then appending it to the encrypted value. This way, the other side can check the signature and decrypt the actual value using the other side's public key and its private key, respectively.
So I've changed the protocol to do this:
-----
On each call, get_nonce() will encrypt the newly-generated random nonce value with the caller's public key. Then it will sign the result with its own private key, and append the signature to the encrypted nonce.
When the encrypted, signed nonce comes back to the caller, it will check the signature first (using the callee's public key), then decrypt the actual nonce value (using its own private key). Then it will re-encrypt (using the callee's public key), and re-sign (using its own private key). This result will be transmitted in the actual XML-RPC call.
When the callee gets the first parameter, it'll check the signature and decrypt the nonce, then compare the nonce to its stored list to check for authentication.
-----
This uses the fact that encrypt-then-authenticate (EtA) is always at least as strong as the weaker of the encryption and authentication (signature) schemes. Since I'm using the same algorithm for both, that means that EtA is as strong as whatever algorithm I use.
The strength of AtE (authenticate, then encrypt) depends on the interaction between the encryption and authentication algorithms. It's not always as secure as EtA. And E&A (encrypt-and-authenticate, doing both in parallel) is the same -- sometimes it is as secure as EtA, sometimes it isn't.