We’ve all heard about or been in the position where the only person who knew a critical server password has left the company. Or over time and through various change control, the password documented is no longer working and rebuilding the server from scratch for any reason is never appealing, let alone the core appliance central to your networking and firewall polices.

In this article we are going to walk through how to reset/recover your NSX-T root, admin or audit password. The process is involved, but I assure you the process outlined is straight forward with detailed images to help along the way. Let’s get started.

GRUB MUST BE CONFIGURED TO RECOVER YOUR ROOT PASSWORD

1. First, we will need to power our NSX-T manager down. 

2. Power the NSX-T manager up and quickly open a console screen and press the SHIFT key to stop the boot sequence and enter the GRUB loader.

3. Select the ‘Advanced options for Ubuntu‘ and press Enter.

4. You will be presented with a login prompt. This prompt is used to login using the predefined GRUB credentials. 

Enter username: root
Enter password: VMware1

5. On the next screen, highlight the ‘Ubuntu, with Linux‘ option without (upstart) at the end and press the e key to edit. 

6. Now that we have the editor open, we need to scroll to the bottom of the text box and locate the line with the word ‘linux‘. Follow that line to the end where you see root=UUID=<UUID>. Delete everything on that line after the root=UUID=<UUID>. 

                    if [ x$feature_platform_search_hint = xy ]; then
                      search --no-floppy --fs-uuid --set=root --hint-bios=hd\
0,msdos2 --hint-efi=hd0,msdos2 --hint-baremetal=ahci0,msdos2 4b76f7c8-7\
84c-4741-950e-c8d2dca486c5
                    else
                      search --no-floppy --fs-uuid --set=root 4b76fc8-784c-\
4741-950e-c8d2dca486c5
                     fi
                     echo         'Loading Linux 4.14.74-nn3-server ...'
                     linux         /vmlinuz-4.14.74-nn3-server root=UUID=4b76f\
7c8-784c-4741-950e-c8d2dca486c5 ro audit=1 quiet splash rootdelay=90 net\
.ifnames=0 biosdevname=0 $vt_handoff
                     echo         'Loading initial ramdisk ...'
                     initrd          /initrd.img-4.14.74-nn3-server

In this example we will be partially deleting line 11 and all of line 12 as shown here in red:

7. Now, we need to replace the text we deleted with our own to so we can load our bash shell to reset the NSX-T password.

We need to append after the root=UUID=<UUID> with the following:

rw single init=/bin/bash

The text box should now look similar to this with the addition in the red box:

                    if [ x$feature_platform_search_hint = xy ]; then
                      search --no-floppy --fs-uuid --set=root --hint-bios=hd\
0,msdos2 --hint-efi=hd0,msdos2 --hint-baremetal=ahci0,msdos2 4b76f7c8-7\
84c-4741-950e-c8d2dca486c5
                    else
                      search --no-floppy --fs-uuid --set=root 4b76fc8-784c-\
4741-950e-c8d2dca486c5
                     fi
                     echo         'Loading Linux 4.14.74-nn3-server ...'
                     linux         /vmlinuz-4.14.74-nn3-server root=UUID=4b76f\
7c8-784c-4741-950e-c8d2dca486c5 rw single init=/bin/bash
                     echo         'Loading initial ramdisk ...'
                     initrd          /initrd.img-4.14.74-nn3-server

8. After making the change press ‘Ctrl-x‘ or the ‘F10‘ key to save our changes and reboot the manager into single user mode.

Once the NSX-T manager reboots, it will boot into single user mode. If your console session disconnects, simply open a new console window.

Your screen will look something like this. You may have to press the Enter key to get the shell prompt.

**After Step #9, You Will Have 15 Minutes to Complete the Remaining Steps**

9. If you are resetting the password for the ‘root’ account, run the following command:

passwd

If you are resetting the password for the ‘admin’ or ‘audit’ accounts, run the following command:

passwd <admin or audit user>

**Note: You can run the passwd command multiple times**

After running the passwd command above, you will be prompted to enter a password, press enter and then enter the same password again. Press enter again to complete the password change.

10. After you have changed the password on the accounts you want, we will then synchronize the passwords by running:

sync

11. Once complete, shut the NSX-T manager down for the final phase of the password reset process.

12. Next, we need to download ubuntu-16.04.6-server-amd64.iso. If you have not already done so at the beginning of this article, you can download it here.

13. After the Ubuntu iso is on your desktop, we need to place the iso onto a datastore accessible to the ESXi host where the NSX-T manager virtual machine resides and attach the iso to the NSX manager virtual machine. To do this, we need to login to the vCenter server or ESXi web UI and import/upload the Ubuntu iso file onto the datastore.

Once the Ubuntu server iso has been uploaded to a datastore, we need to attach the iso to the NSX Manager virtual machine by right clicking on the NSX-T manager virtual machine and clicking ‘Edit Settings’. If needed, add a CD ROM drive if one does not already exist.

In the CD ROM Drive configuration, check the Connect at power on checkbox.

In CD/DVD Media, select Browse then select the Ubuntu image from the applicable datastore.

14. Next, we need to change the boot order from within the BIOS of the NSX manager virtual machine so the CD-ROM drive is used before the primary hard disk. To do this, we need to force the NSX manager to boot to its BIOS so we can make the change.

Click on the ‘VM Options‘ tab at the top of the ‘Edit Settings‘ page.

Select the radio button to ‘Force BIOS setup‘ on the next boot. Then click, OK.

15. Now we can boot the NSX-T manager into its BIOS so we can make the necessary boot order changes.

The NSX-T manager will now reboot into the Ubuntu Server CD we mounted earlier.

16. Once the NSX-T manager reboots, you will navigate through the Ubuntu setup pages.

1. Select Your Preferred Language
2. Select ‘Rescue a broken system‘
3. Select Your Preferred Language (again)
4. Select Your Location
5. Select Your Keyboard Configuration
6. Select ‘Do not configure the network at this time‘
7. Leave Host Name as ‘ubuntu‘ (not used for recovery)
8. Select Your Timezone
9. Select ‘Do not use a root file system‘
10. Select ‘Execute a shell in the installer environment‘
11. Select ‘<Continue>‘ to Execute a Shell

17. After you press the Enter key on <Continue>, a shell prompt will appear on the next screen. Here you will run the following commands to ensure the password change(s) made in the previous steps do not revert and are permanent.

The following command will mount the /dev/sda2 partition of the NSX-T manager to the /mnt directory.

mount /dev/sda2 /mnt

Next, we need to mount the /dev directory. However, we need to use the bind mount option instead so we take the existing directory tree of /dev and replicate it under /mnt. The directories and files in the bind mount are the same as the original.

mount --bind /dev /mnt/dev

Next, we need to change the root directory for our recovery purposes to /mnt. To do this, we will be using the chroot command. The chroot command will allow us to move back into the NSX-T partition from our ‘alternate’ root file system that we just loaded from the rescue CD.

chroot /mnt

Next, move to the section for your version of NSX-T. It is divided up into two sections, 2.4.x and 2.5.x.

Steps for NSX-T 2.4.x Versions

Now, we need to mount the /config directory and touch the ‘reset_cluster_credentials’ file to update its timestamp. Then we will un-mount the /config directory when done.

mount /config
touch /config/vmware/nsx-node-api/reset_cluster_credentials
umount /config

Steps for NSX-T 2.5.x Versions

touch /var/vmware/nsx/reset_cluster_credentials

Steps for NSX-T 3.x Versions

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.0/administration/GUID-8816B842-2EC4-40A8-A618-F68DB29FABD2.html

Now that we have completed making our password changes permanent, we need to exit our chroot instance and clean up.

exit
umount /mnt/dev
umount /mnt
sync
exit

Once we have performed all the above steps and exited the shell, we can reboot the system by selecting ‘Reboot the system’.