One of the things you will want to get going as soon as possible is opening a bank account. In fact you need a bank account to get a tax number. See the thread on getting a tax number.
It is even possible to open a bank account before you even arrive in New Zealand, though you will not be able to withdraw funds until you present yourself to the bank with adequate proof of identity.
Banking is a very personal thing. One person may love a bank while somebody else will hate that bank. It's all dependent on the service that each individual gets from a bank. Unfortunately banks cannot guarantee the same experience for two different people.
The larger banks are