Being a user of Linux, I got used to change the port of SSH on my Mac with the following line:
sudo vi /etc/sshd_config
Then I could change the number and uncomment the following line:
#Port 22

Sadly I noticed that it makes my IPv6 SSH connection unavailable on the new port (neither on the previous port 22).

The real clean way of changing a Mac OS X Daemon is in the plist files

So now I just changed the port in the ssh.plist file and my SSH Daemon runs on IPv4 and IPv6, on the new port.
sudo vi /System/Library/LaunchDaemons/ssh.plist

And modify as following if you want the port to become 22222:

<key>Sockets</key>
    <dict>
            <key>Listeners</key>
            <dict>
                    <key>SockServiceName</key>
                    <string>22222</string>
                    <key>Bonjour</key>
                    <array>
                            <string>22222</string>
                            <string>sftp-ssh</string>
                    </array>
            </dict>
    </dict>

You can add 2 lines to make SSH available only via IPv4:

<key>Sockets</key>
    <dict>
            <key>Listeners</key>
            <dict>
                    <key>SockServiceName</key>
                    <string>22222</string>
                    <key>SockFamily</key>
                    <string>IPv4</string>
                    <key>Bonjour</key>
                    <array>
                            <string>22222</string>
                            <string>sftp-ssh</string>
                    </array>
            </dict>
    </dict>