After some searching, the man page of rsync finally offered a solution:

# rsync -avz -e “ssh -p $portNumber” user@remoteip:/path/to/files/ /local/path/

or

# rsync -avz -e “ssh -p $portNumber” /local/folder user@remoteip:/path/to/files

Passing the port parameter to ssh with the -e option worked like a charm. 

This is why Unix rocks.