vmWare - Vcenter Server Appliance 5.5 - embedded DB problem

Problem seen: The embedded DB didnt start due to which vCenter configurations like Dswitch, hosts, etc were not available.

vSA-173:~ # service vmware-vpxd status
vmware-vpxd is stopped
tomcat is stopped
vSA-173:~ # service vmware-vpxd start
Waiting for the embedded database to start up: ............................failed
Verifying EULA acceptance: success
Executing pre-startup scripts...
Updating the vCenter endpoint in the Lookup Service.
Intializing registration provider...
Getting SSL certificates for https://135.249.34.173:7444/lookupservice/sdk
Service with name 'vpxd-vSA-173-81a29e14-9705-47e2-a165-3cacb809fd43' and ID 'local:20165866-25c5-41ac-8e5e-9fc277b7bc6b' was updated.
Return code is: Success
Starting ldap-server..done
Starting vmware-vpxd: success
Waiting for vpxd to initialize: ....................
.................................^C
vSA-173:~ # Traceback (most recent call last):
  File "/usr/lib/vmware-vpx/py/wait_for_vpxd.py", line 48, in <module>
    test_connect()
  File "/usr/lib/vmware-vpx/py/wait_for_vpxd.py", line 38, in test_connect
    time.sleep(delay)
KeyboardInterrupt
failed

vSA-173:~ #





vSA-173:~ # service vmware-vpxd stop
Stopping tomcat: success
Stopping vmware-vpxd: success
Shutting down ldap-server..done
vSA-173:~ # service vmware-vpxd stop
Stopping tomcat: success
Stopping vmware-vpxd: success
Shutting down ldap-server..done
vSA-173:~ # /usr/sbin/vpxd_odbcconfig
WARNING - Use of this command is not supported unless you are doing this
with the consultation of VMware Technical Support or a VMware published
technical article.

VMware vCenter Server Appliance ODBC Configuration

Choose a database to use with VMware vCenter Server
1) oracle
2) embedded
#? 2

Configuring ODBC for embedded
Error: Failed to connect to database.
Database  configuration has failed.
Would you like to try again or modify the settings manually?
 [A]gain/[m]anually : A

Choose a database to use with VMware vCenter Server
1) oracle
2) embedded
#? 2

Configuring ODBC for embedded
Error: Failed to connect to database.
Database  configuration has failed.
Would you like to try again or modify the settings manually?
 [A]gain/[m]anually : m

vSA-173:~ # service --status-all
Checking for httpd2:                                                                                                                                                                      unused
Checking for service arpd                                                                                                                                                                 unused
Checking for service atftp:                                                                                                                                                               unused
Checking for service auditd                                                                                                                                                               running
Usage: /etc/init.d/autoyast {start|stop}
Checking if boot.cgroup has run                                                                                                                                                           running
                                                                                                                                                                                          running
Current Hardware Clock: Tue Jul 21 14:26:29 2015  -1.042606 seconds                                                                                                                       running
                                                                                                                                                                                          running
 ...                                                                                                                                                                                      unused
 ...                                                                                                                                                                                      unused
                                                                                                                                                                                          running
                                                                                                                                                                                          unknown
                                                                                                                                                                                          unknown
checking if boot.ipconfig has been run                                                                                                                                                    running
                                                                                                                                                                                          running
                                                                                                                                                                                          running
Checking if boot.loadmodules has run                                                                                                                                                      running
Checking if boot.localfs has run                                                                                                                                                          running
                                                                                                                                                                                          running
                                                                                                                                                                                          unknown
                                                                                                                                                                                          running
MD RAID arrays: No arrays active                                                                                                                                                          unused
Checking multipath targets:                                                                                                                                                               unused
                                                                                                                                                                                          running
Checking if boot.rootfsck has run                                                                                                                                                         running
                                                                                                                                                                                          running
                                                                                                                                                                                          unknown
Checking for service sysstat:                                                                                                                                                             running
Checking for udevd:                                                                                                                                                                       running
                                                                                                                                                                                          running
Checking for Cron:                                                                                                                                                                        running
Checking for service D-Bus daemon                                                                                                                                                         running
                                                                                                                                                                                          unused
                                                                                                                                                                                          unused
Checking for ISC DHCPv4 4.x Server:                                                                                                                                                       unused
Checking for ISC DHCPv6 4.x Server:                                                                                                                                                       unused
Checking for service syslog:                                                                                                                                                              running
running (standalone: 2592)
Neither the variables MOUSEDEVICE and MOUSETYPE nor the variable GPM_PARAM
is set in /etc/sysconfig/mouse
Run 'yast mouse' to set up gpm
Checking for service HAL daemon                                                                                                                                                           running
Checking for haveged daemon                                                                                                                                                               running
Checking for service irqbalance                                                                                                                                                           running
Keymap /usr/share/kbd/keymaps/i386/qwerty/us.map.gz is loaded.
Checking for service ldap:                                                                                                                                                                unused
running (standalone: 3085)
Usage: /etc/init.d/lvm_wait_merge_snapshot {start}
running (standalone: 2702)
running (standalone: 1265)
                                                                                                                                                                                          running
Checking for service mdadmd                                                                                                                                                               unused
Checking for multipathd:                                                                                                                                                                  unused
running (standalone: 2746)
Checking optional network interfaces:
    eth0      device: VMware VMXNET3 Ethernet Controller
    eth0      IP address: 135.249.34.173/24
    eth0      is up                                                                                                                                                                       running
Checking mandatory network interfaces:
    lo
    lo        IP address: 127.0.0.1/8
    secondary lo IP address: 127.0.0.2/8
    lo        is up                                                                                                                                                                       running
Checking service network .  .  .  .  .  .  .  .  .  .  .                                                                                                                                  running
Checking mandatory network interfaces:
    lo
    lo        IP address: 127.0.0.1/8
    secondary lo IP address: 127.0.0.2/8
    lo        is up                                                                                                                                                                       running
Checking service (remotefs) network .  .  .  .  .  .  .  .  .  .  .                                                                                                                       running
Checking for mounted nfs shares (from /etc/fstab):idmapd not running
                                                                                                                                                                                          unused
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*172.21.176.208  LOCAL(0)        11 u   35   64  377    0.397  -38.761  19.439

Checking for network time protocol daemon (NTPD):                                                                                                                                         running
Checking for PC/SC smart card daemon                                                                                                                                                      unused
Checking for UPS monitoring service                                                                                                                                                       unused
Checking for random generator (always true)                                                                                                                                               running
                                                                                                                                                                                          dead
Checking for service rpasswd daemon:                                                                                                                                                      unused
Checking for service rpcbind                                                                                                                                                              running
                                                                                                                                                                                          unknown
Checking for service sendmail:                                                                                                                                                            unused
/dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A
/dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A                                                                                                                                                running
/usr/sbin/FOO not installed
                                                                                                                                                                                          unknown
Checking for service sshd                                                                                                                                                                 running
Checking for stunnel (SSL tunnel):                                                                                                                                                        unused
Checking for service syslog:                                                                                                                                                              running
Checking for syslog-collector service:                                                                                                                                                    running
Checking for syslog-collector SSL service:                                                                                                                                                running
Checking for service uuidd                                                                                                                                                                unused
Checking vami-lighttpd status: 2430 ?        00:00:00 vami-lighttpd
Checking vami-sfcbd status: 2440 ?        00:00:00 vami-sfcbd
 2442 ?        00:00:00 vami-sfcbd
 2447 ?        00:00:00 vami-sfcbd
 2448 ?        00:00:00 vami-sfcbd
 2450 ?        00:00:00 vami-sfcbd
 2451 ?        00:00:00 vami-sfcbd
 2455 ?        00:00:00 vami-sfcbd
 8389 ?        00:00:00 vami-sfcbd

running (standalone: 3112)
Warning: vmci status: unimplemented
running (standalone: 3150)
running (standalone: 3194)
VMware Inventory Service is running (PID:5671, Wrapper:STARTED, Java:STARTED)
VMware Log Browser is running
/usr/sbin/vmware-netdumper is running
Checking for /usr/bin/rbd-watchdog-linux:                                                                                                                                                 unused
VMware vSphere Profile-Driven Storage Service is not running.
Checking for VMware STS IDM Server ...                                                                                                                                                    running
Checking for service vmware-stsd                                                                                                                                                          running
vmtoolsd is running
could not change directory to "/root"
pg_ctl: no server running
vmware-vpxd is stopped
tomcat is stopped
Warning: vsock status: unimplemented
VMware vSphere Web Client is running: PID:12210, Wrapper:STARTED, Java:STARTED
Checking for service xinetd:                                                                                                                                                              unused
Checking for ypbind:                                                                                                                                                                      unused
vSA-173:~ #









Solution:
vSA-173:/storage/db/vpostgres # /etc/init.d/vmware-vpostgres restart
Restarting VMware vPostgres: pg_ctl: PID file "/storage/db/vpostgres/postmaster.pid" does not exist
Is server running?
waiting for server to start... done
server started
ok
vSA-173:/storage/db/vpostgres #
vSA-173:/storage/db/vpostgres #


vSA-173:/storage/db/vpostgres # service --status-all
Checking for httpd2:                                                                                                                                                                      unused
Checking for service arpd                                                                                                                                                                 unused
Checking for service atftp:                                                                                                                                                               unused
Checking for service auditd                                                                                                                                                               running
Usage: /etc/init.d/autoyast {start|stop}
Checking if boot.cgroup has run                                                                                                                                                           running
                                                                                                                                                                                          running
Current Hardware Clock: Tue Jul 21 14:34:59 2015  -1.043426 seconds                                                                                                                       running
                                                                                                                                                                                          running
 ...                                                                                                                                                                                      unused
 ...                                                                                                                                                                                      unused
                                                                                                                                                                                          running
                                                                                                                                                                                          unknown
                                                                                                                                                                                          unknown
checking if boot.ipconfig has been run                                                                                                                                                    running
                                                                                                                                                                                          running
                                                                                                                                                                                          running
Checking if boot.loadmodules has run                                                                                                                                                      running
Checking if boot.localfs has run                                                                                                                                                          running
                                                                                                                                                                                          running
                                                                                                                                                                                          unknown
                                                                                                                                                                                          running
MD RAID arrays: No arrays active                                                                                                                                                          unused
Checking multipath targets:                                                                                                                                                               unused
                                                                                                                                                                                          running
Checking if boot.rootfsck has run                                                                                                                                                         running
                                                                                                                                                                                          running
                                                                                                                                                                                          unknown
Checking for service sysstat:                                                                                                                                                             running
Checking for udevd:                                                                                                                                                                       running
                                                                                                                                                                                          running
Checking for Cron:                                                                                                                                                                        running
Checking for service D-Bus daemon                                                                                                                                                         running
                                                                                                                                                                                          unused
                                                                                                                                                                                          unused
Checking for ISC DHCPv4 4.x Server:                                                                                                                                                       unused
Checking for ISC DHCPv6 4.x Server:                                                                                                                                                       unused
Checking for service syslog:                                                                                                                                                              running
running (standalone: 2592)
Neither the variables MOUSEDEVICE and MOUSETYPE nor the variable GPM_PARAM
is set in /etc/sysconfig/mouse
Run 'yast mouse' to set up gpm
Checking for service HAL daemon                                                                                                                                                           running
Checking for haveged daemon                                                                                                                                                               running
Checking for service irqbalance                                                                                                                                                           running
Keymap /usr/share/kbd/keymaps/i386/qwerty/us.map.gz is loaded.
Checking for service ldap:                                                                                                                                                                running
running (standalone: 3085)
Usage: /etc/init.d/lvm_wait_merge_snapshot {start}
running (standalone: 2702)
running (standalone: 1265)
                                                                                                                                                                                          running
Checking for service mdadmd                                                                                                                                                               unused
Checking for multipathd:                                                                                                                                                                  unused
running (standalone: 2746)
Checking optional network interfaces:
    eth0      device: VMware VMXNET3 Ethernet Controller
    eth0      IP address: 135.249.34.173/24
    eth0      is up                                                                                                                                                                       running
Checking mandatory network interfaces:
    lo
    lo        IP address: 127.0.0.1/8
    secondary lo IP address: 127.0.0.2/8
    lo        is up                                                                                                                                                                       running
Checking service network .  .  .  .  .  .  .  .  .  .  .                                                                                                                                  running
Checking mandatory network interfaces:
    lo
    lo        IP address: 127.0.0.1/8
    secondary lo IP address: 127.0.0.2/8
    lo        is up                                                                                                                                                                       running
Checking service (remotefs) network .  .  .  .  .  .  .  .  .  .  .                                                                                                                       running
Checking for mounted nfs shares (from /etc/fstab):idmapd not running
                                                                                                                                                                                          unused
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*172.21.176.208  LOCAL(0)        11 u   26   64  377    0.511   19.842  20.538

Checking for network time protocol daemon (NTPD):                                                                                                                                         running
Checking for PC/SC smart card daemon                                                                                                                                                      unused
Checking for UPS monitoring service                                                                                                                                                       unused
Checking for random generator (always true)                                                                                                                                               running
                                                                                                                                                                                          dead
Checking for service rpasswd daemon:                                                                                                                                                      unused
Checking for service rpcbind                                                                                                                                                              running
                                                                                                                                                                                          unknown
Checking for service sendmail:                                                                                                                                                            unused
/dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A
/dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A                                                                                                                                                running
/usr/sbin/FOO not installed
                                                                                                                                                                                          unknown
Checking for service sshd                                                                                                                                                                 running
Checking for stunnel (SSL tunnel):                                                                                                                                                        unused
Checking for service syslog:                                                                                                                                                              running
Checking for syslog-collector service:                                                                                                                                                    running
Checking for syslog-collector SSL service:                                                                                                                                                running
Checking for service uuidd                                                                                                                                                                unused
Checking vami-lighttpd status: 2430 ?        00:00:00 vami-lighttpd
Checking vami-sfcbd status: 2440 ?        00:00:00 vami-sfcbd
 2442 ?        00:00:00 vami-sfcbd
 2447 ?        00:00:00 vami-sfcbd
 2448 ?        00:00:00 vami-sfcbd
 2450 ?        00:00:00 vami-sfcbd
 2451 ?        00:00:00 vami-sfcbd
 2455 ?        00:00:00 vami-sfcbd
 8389 ?        00:00:00 vami-sfcbd

running (standalone: 3112)
Warning: vmci status: unimplemented
running (standalone: 3150)
running (standalone: 3194)
VMware Inventory Service is running (PID:5671, Wrapper:STARTED, Java:STARTED)
VMware Log Browser is running
/usr/sbin/vmware-netdumper is running
Checking for /usr/bin/rbd-watchdog-linux:                                                                                                                                                 unused
VMware vSphere Profile-Driven Storage Service is not running.
Checking for VMware STS IDM Server ...                                                                                                                                                    running
Checking for service vmware-stsd                                                                                                                                                          running
vmtoolsd is running
pg_ctl: server is running (PID: 16913)
/opt/vmware/vpostgres/9.0/bin/postgres "-D" "/storage/db/vpostgres"
vmware-vpxd is running
tomcat is stopped
Warning: vsock status: unimplemented
VMware vSphere Web Client is running: PID:12210, Wrapper:STARTED, Java:STARTED
Checking for service xinetd:                                                                                                                                                              unused
Checking for ypbind:                                                                                                                                                                      unused
vSA-173:/storage/db/vpostgres # service vmware-vpxd restart
Stopping tomcat: success
Stopping vmware-vpxd: success
Shutting down ldap-server..done
Waiting for the embedded database to start up: success
Verifying EULA acceptance: success
Executing pre-startup scripts...
Updating the vCenter endpoint in the Lookup Service.
Intializing registration provider...
Getting SSL certificates for https://135.249.34.173:7444/lookupservice/sdk
Service with name 'vpxd-vSA-173-81a29e14-9705-47e2-a165-3cacb809fd43' and ID 'local:20165866-25c5-41ac-8e5e-9fc277b7bc6b' was updated.
Return code is: Success
Starting ldap-server..done
Starting vmware-vpxd: success
Waiting for vpxd to initialize: .success
Starting tomcat: success
Executing startup scripts...
Autodeploy service is disabled, skipping registration.
Starting VMware vSphere Profile-Driven Storage Service...Waiting for VMware vSphere Profile-Driven Storage Service......
VMware vSphere Profile-Driven Storage Service started.

vSA-173:/storage/db/vpostgres #

Error seen on a Virtual Machine

kernel:BUG: soft lockup - CPU#2 stuck for 61s! [java:31558]





Comments

Popular posts from this blog

Installing Pytest on Windows Platform

Shell Scripting