After some searching, the man page of rsync finally offered a solution:
# rsync -avz -e “ssh -p $portNumber” user@remoteip:/path/to/files/ /local/path/
or
# rsync -avz -e “ssh -p $portNumber” /local/folder user@remoteip:/path/to/files
Passing the port parameter to ssh with the -e option worked like a charm.
This is why Unix rocks.