The idea of PKI (Public-Key-Infrastructure), also SSH-keys is that each user can put the public key for their own private key onto the host they want to SSH into. The public key is then used by the SSH server to check whether the user that connects to it has the corresponding private key. The public key can be shown to anyone. So your colleague just sends you his public key and you add it to the list of allowed public keys on the server and then your coulleague can SSH into it using his private key that never leaves his own machine.
Ed.it: Public-Key-Infrastructure, not Private-Key-Infrastructure, lol…