If set to 'file', the file named in the 'Password file' will be used. The format is as follows: user:cryptedpass:uid:gid:Real Name:/home/dir:shell, or, in other words, it is the standard UNIX passwd-file format.
If set to 'ypcat', use the yp (often called NIS for copyright reasons) method to retrieve the password database. If you can do 'ypcat passwd' on your system, you know that this is the method you should use.
If set to 'niscat', use the NIS+ method. This is probably only supported by Solaris 2.X. If 'niscat passwd.org_dir' works, this is the method you should use.