The Black Caps, losing finalists four years ago, achieved a seemingly modest target of 245 thanks mainly to Ross Taylor’s 82.
But they lost three wickets for 27 runs before Mitchell Santner (17 not out) and Lockie Ferguson (four not out) saw them to a victory achieved with 17 balls to spare.
Paceman Matt Henry grabbed four wickets as New Zealand bowled Bangladesh out for 244.
Henry finished with figures of 4-47 to follow his three wickets against Sri Lanka as Bangladesh — sent in to bat — lost their last six wickets for just 65 runs.
Bangladesh all-rounder Shakib Al Hasan hit a fighting 68-ball 64 with seven boundaries in his 200th one-day international but no other batsmen cashed in on good starts.
Once Shakib was caught by wicketkeeper Tom Latham off the bowling of Colin de Grandhomme in the 31st over, the innings fell apart.
Mohammad Mitun (26), Mahmudullah (20) and Mosadeek Hossain (11) all failed to build a significant innings.
Tailender Mohammad Saifuddin hit a 23-ball 29 with a six and three boundaries before Henry and Trent Boult (2-44) mopped up the tail.
Earlier, openers Tamim Iqbal, who made 24, and Soumya Sarkar, who scored 25, gave Bangladesh a solid base, putting on 45 runs for the first wicket.
But Henry made the breakthrough for New Zealand, bowling Sarkar in the ninth over.
That brought Shakib to the crease, much to the appreciation of a large contingent of Bangladeshi supporters.
Iqbal was out shortly afterwards, caught by Boult off the bowling of Lockie Ferguson, to leave Bangladesh 60-2.
Mushfiqur Rahim rebuilt the innings with Shakib, putting on exactly 50 for the third wicket before he was run out for 19.
Both New Zealand and Bangladesh won their first matches at the World Cup, taking place in England and Wales.
New Zealand crushed Sri Lanka by 10 wickets at Cardiff while Bangladesh beat South Africa.
Bangladesh: Mashrafe Mortaza (capt), Tamim Iqbal, Soumya Sarkar, Shakib Al Hasan, Mushfiqur Rahim, Mohammad Mithun, Mahmudullah, Mosaddek Hossain, Mehidy Hasan Miraz, Mohammad Saifuddin, Mustafizur Rahman
New Zealand: Kane Williamson (captain), Martin Guptill, Colin Munro, Ross Taylor, Tom Latham, James Neesham, Colin de Grandhomme, Mitchell Santner, Matt Henry, Lockie Ferguson, Trent Boult
Umpires: Paul Reiffel (AUS) and Bruce Oxenford (AUS)
TV umpire: Joel Wilson (WIS)
Match referee: David Boon