This post is about debugging the cloudstack router below issue.
Issue:
1. Unable to contact resource
2. Unable to apply *** on router
Reason:
These issues shows up due to following reason:
1. The cloudstack router is not reachable (not able to ssh on 3922 port) from the host to apply configuration on the router.
2. There is some thing going wrong with the script which is called by cloudstack to execute on the router.
How to debug:
Step1:
Check the management and agent (host) logs to see what is failing from the router.
log files: 1. management server log file
2.1 xenserver: /var/log/SMlog
2.2 kvm: /var/log/cloudstack/agent/agent.log
3.3 vmware: for agent log check management server log
Step2:
If you logs similar to Example logs shown below then agent log reveals the issue.
For debugging the script you can set 'set -x' in the router and run the script.
Identify the problem. If the host is not able to ssh to VR (in case of xen, kvm) for vmware MS ssh to VR.
There is could be issue with the key file.
Example: In the below example vm create failed because in router save_password_to_domr.sh script execute got failed.
MS log:
2014-02-05 15:52:08,172 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-2:job-1108 = [ f2c74a8d-e7fe-45ea-85ee-8e27a10abe34 ]) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [Pod:1] is unreachable: Unable to apply userdata and password entry on router
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3806)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyUserData(VirtualNetworkApplianceManagerImpl.java:2993)
at com.cloud.network.element.VirtualRouterElement.addPasswordAndUserdata(VirtualRouterElement.java:926)
at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2076)
at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2191)
at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2127)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:887)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-02-05 15:52:08,174 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-5:null) Seq 1-1025769488: No more commands found
Agent log: /var/log/SMlog
Feb 7 10:11:38 gcloudblr-sm1 SM: [24026] ['/bin/bash', '/opt/xensource/bin/dhcp_entry.sh', '-r', '169.254.2.207', '-v', '10.105.89.77', '-m', '06:9d:12:00:00:48', '-n', 'JA-Win7-Sreekanth', '-d', '10.105.88.1', '-N', '10.105.89.23']
Feb 7 10:11:39 gcloudblr-sm1 SM: [24026] pread SUCCESS
Feb 7 10:11:39 gcloudblr-sm1 SM: [24026] #### VMOPS exit saveDhcpEntry ####
Feb 7 10:11:39 gcloudblr-sm1 SM: [24047] #### VMOPS enter savePassword ####
Feb 7 10:11:39 gcloudblr-sm1 SM: [24047] ['/bin/bash', '/opt/xensource/bin/save_password_to_domr.sh', '-r', '169.254.2.207', '-v', '10.105.89.77', '-p', 'fnirq_cnffjbeq', 'JA-Win7-Sreekanth']
Feb 7 10:11:39 gcloudblr-sm1 SM: [24047] FAILED in util.pread: (rc 1) stdout: '', stderr: 'sed: couldn't flush /var/cache/cloud/sed0zl7BZ: No space left on device
Feb 7 10:11:39 gcloudblr-sm1 SM: [24047] bash: line 0: echo: write error: No space left on device
Feb 7 10:11:39 gcloudblr-sm1 SM: [24047] '
Feb 7 10:11:39 gcloudblr-sm1 SM: [24047] save password to domr failed
Feb 7 10:11:39 gcloudblr-sm1 SM: [24047] #### VMOPS exit savePassword ####
Feb 7 10:11:40 gcloudblr-sm1 SM: [24060] lock: acquired /var/lock/sm/337554c4-5228-f9f2-059e-482021b08a5c/sr
</SMlog>
No comments:
Post a Comment