Corporate Home Open Source Home
Syndicate content
Eucalyptus
Forums » Eucalyptus General » instance disappeared after some time
15 replies [Last post]
Tue, 09/07/2010 - 21:12
madhu_s
Offline
Joined: 04/19/2010

Hi
i am running eucalyptus 1.6.2 head , after starting instance it works fine for some time and after some time i see messages like this

59502:[Tue Sep 7 06:27:23 2010][021175][EUCAINFO ] started VM instance i-38CF07BF
......
.....
........

115149:[Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] vrun(): [rm -rf /opt/ext/instances/admin/i-38CF07BF]

115167:[Tue Sep 7 22:55:57 2010][010580][EUCAWARN ] WARNING: failed to recover Eucalyptus metadata of running domain i-38CF07BF, ignoring it
115189:[Tue Sep 7 22:56:10 2010][010654][EUCADEBUG ] scRecoverInstanceInfo: didn't find instance i-38CF07BF!

-----

---------

-----------------------------

now the question is why should it remove instance without user command

and xm list still shows the instance but public ip is revoked

[root@my_nc ~]# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 512 4 r----- 52140.7
i-38CF07BF 22 2048 2 -b---- 255.6
[root@my_nc ~]# losetup -a
/dev/loop0: [0811]:7831562 (/opt/ext/instances/admin/i-38CF07BF/swap)
/dev/loop1: [0811]:7831561 (/opt/ext/instances/admin/i-38CF07BF/root)
[root@my_nc ~]# ls -la (/opt/ext/instances/admin/
-bash: syntax error near unexpected token `('
[root@my_nc ~]# ls -la /opt/ext/instances/admin/
total 8
drwxr-xr-x 2 eucalyptus eucalyptus 4096 Sep 7 22:55 .
drwxr-xr-x 3 eucalyptus eucalyptus 4096 Sep 6 12:44 ..
[root@my_nc ~]#

this behavior is consistent , did not observe this behavior with 1.6.2 source tar

any ideas please
Thanks
Madhu

Tue, 09/07/2010 - 21:51
#1
neil
Offline
Joined: 04/28/2009
Did the NC receive a

Did the NC receive a terminate. Please check the rest of the nc.log. Also check for any terminates in cc.log and cloud-output.log. I don't think anything changed in the head that would cause your instances to suddenly terminate. The logs should reveal the underlying issue.

Tue, 09/07/2010 - 22:09
#2
madhu_s
Offline
Joined: 04/19/2010
here are the logs

cc.log
-------------------------------------

134769 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] monitor_thread(): running
134770 [Tue Sep 7 16:03:28 2010][008613][EUCAINFO ] refresh_resources(): called
134771 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] refresh_resources(): calling http://10.208.137.148:8775/axis2/services/EucalyptusNC
134772 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] refresh_resources(): time left for next op: 60
134773 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] refresh_resources(): node=10.208.137.148 mem=16155/16155 disk=1360/1360 cores=4/4
134774 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] refresh_resources(): done
134775 [Tue Sep 7 16:03:28 2010][008613][EUCAINFO ] refresh_instances(): called
134776 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] invalidate_instanceCache(): invalidating instance 'i-38CF07BF' (last seen 303 seconds ago)
134777 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] refresh_instances(): timeout(60/60) len
134778 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] refresh_instances(): node 10.208.137.148 idle since 1283900311: (297/300) seconds
134779 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] refresh_instances(): done
134780 [Tue Sep 7 16:03:28 2010][008613][EUCADEBUG ] monitor_thread(): done

138677 [Tue Sep 7 16:13:31 2010][008578][EUCAINFO ] TerminateInstances(): called
138678 [Tue Sep 7 16:13:31 2010][008578][EUCADEBUG ] TerminateInstances(): params: userId=UNSET, instIdsLen=1
138679 [Tue Sep 7 16:13:31 2010][008578][EUCAINFO ] TerminateInstances(): calling terminate instance (i-38CF07BF) on (10.208.137.148)
138680 [Tue Sep 7 16:13:31 2010][012849][EUCAERROR ] ERROR: TerminateInstance() could not be invoked (check NC host, port, and credentials)
138681 [Tue Sep 7 16:13:31 2010][008578][EUCADEBUG ] TerminateInstances(): call complete (pid/rc): 12849/1
138682 [Tue Sep 7 16:13:31 2010][008578][EUCADEBUG ] TerminateInstances(): done.

----------------------------------------------------------------

nc log

----------------------------------------------------------------

115126 [Tue Sep 7 22:55:45 2010][010556][EUCAINFO ] - adopted running domain i-38CF07BF from user admin
115127 [Tue Sep 7 22:55:45 2010][010556][EUCAINFO ] vnetInit(): VNET Configuration: eucahome=/opt/eucalyptus, path=/opt/eucalyptus/var/run/eucalyptus/net, dhcpdaemon=, dhcpuser=, pubInter face=eth0, privInterface=eth0, bridgedev=xenbr0, networkMode=MANAGED-NOVLAN
115128 [Tue Sep 7 22:55:45 2010][010556][EUCAINFO ] checking the integrity of instances directory (/opt/ext/instances)
115129 [Tue Sep 7 22:55:45 2010][010556][EUCAWARN ] warning: could not stat file /opt/ext/instances/admin/i-38CF07BF/root
115130 [Tue Sep 7 22:55:45 2010][010556][EUCAWARN ] warning: non-standard instance directory /opt/ext/instances/admin/i-38CF07BF
115131 [Tue Sep 7 22:55:45 2010][010556][EUCAINFO ] checking the integrity of the cache directory ((null))
115132 [Tue Sep 7 22:55:45 2010][010556][EUCAINFO ] no cache directory yet
115133 [Tue Sep 7 22:55:45 2010][010556][EUCAINFO ] Maximum disk available = 1356 (under /opt/ext/instances)
115134 [Tue Sep 7 22:55:45 2010][010556][EUCADEBUG ] doDescribeInstances() invoked
115135 [Tue Sep 7 22:55:45 2010][010556][EUCADEBUG ] Starting monitoring thread
115136 !
115137 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] NC is looking for configuration in /opt/eucalyptus/etc/eucalyptus/eucalyptus.conf//opt/eucalyptus/etc/eucalyptus/eucalyptus.local.conf
115138 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] SC is looking for configuration in files (/opt/eucalyptus/etc/eucalyptus/eucalyptus.conf,/opt/eucalyptus/etc/eucalyptus/eucalyptus.loca l.conf)
115139 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] euca_init_cert(): using file /opt/eucalyptus/var/lib/eucalyptus/keys/node-cert.pem
115140 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] euca_init_cert(): using file /opt/eucalyptus/var/lib/eucalyptus/keys/node-pk.pem
115141 [Tue Sep 7 22:55:51 2010][010563][EUCADEBUG ] doInitialized() invoked
115142 [Tue Sep 7 22:55:51 2010][010563][EUCADEBUG ] system_output(): [/opt/eucalyptus/usr/lib/eucalyptus/euca_rootwrap /opt/eucalyptus/usr/share/eucalyptus/get_xen_info]
115143 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] Using 4 cores
115144 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] Using 16155 memory
115145 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] looking for existing domains
115146 [Tue Sep 7 22:55:51 2010][010563][EUCAWARN ] WARNING: failed to get info on running domain #22, ignoring it
115147 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] vnetInit(): VNET Configuration: eucahome=/opt/eucalyptus, path=/opt/eucalyptus/var/run/eucalyptus/net, dhcpdaemon=, dhcpuser=, pubInter face=eth0, privInterface=eth0, bridgedev=xenbr0, networkMode=MANAGED-NOVLAN
115148 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] checking the integrity of instances directory (/opt/ext/instances)
115149 [Tue Sep 7 22:55:51 2010][010563][EUCAINFO ] vrun(): [rm -rf /opt/ext/instances/admin/i-38CF07BF]

----------------------------------------

cc is running in PDT time zone (current time Tue Sep 7 23:06:27 PDT 2010)
and NC is running in UTC time zone (current time Wed Sep 8 06:06:40 UTC 2010)

will this be a problem ?

i don't see any messages related to i-38CF07BF

Thanks

Tue, 09/07/2010 - 22:24
#3
neil
Offline
Joined: 04/28/2009
Did you install the current

Did you install the current head from scratch or did you update an existing installation? If the latter, did you terminate your instances before the update?

Tue, 09/07/2010 - 22:32
#4
madhu_s
Offline
Joined: 04/19/2010
It is fresh install from

It is fresh install from scratch ( old installation moved to eucalyptus_old) and also rebooted cc and nc machines

Thanks

Tue, 09/07/2010 - 22:39
#5
madhu_s
Offline
Joined: 04/19/2010
last_revison

cat last-revision
1236 neil@pall-20100901230636-wdq8oaw2324mmawn
[lroot@my_nc branch]# pwd
/opt/src/eucalyptus-1.6.2/.bzr/branch
[lroot@my_nc branch]#

both machines running same branch

Wed, 09/08/2010 - 21:56
#6
graziano
Offline
Joined: 01/14/2010
Hello, the 2 logs you sent

Hello,

the 2 logs you sent seems to indicate that a Terminate was received by the CC for the instance at 16:13 and the NC seems to have later on rebooted and finally cleared out the state around 22:55. The Terminate seems to have failed because the CC couldn't talk to the NC.

Could it be that the NC is having trouble talking with the hypervisor? Can you check all the logs (eucalyptus and systems) to check if the NC is indeed rebooting, and if the hypervisor log is showing any problem?

cheers
graziano

Wed, 09/08/2010 - 22:10
#7
madhu_s
Offline
Joined: 04/19/2010
Hi, it happened twice , i

Hi,

it happened twice , i also suspected NC machine (hypervisor/libvirt ) has some problem and rebooted machine keeping all services down on CC machine as well , after that did a clean start . now i don't see it again , running for almost 20 hours . will keep you posted if i see any problem. Thanks for looking into it.

-Madhu

Thu, 09/09/2010 - 02:11
#8
madhu_s
Offline
Joined: 04/19/2010
it happened again , but this

it happened again , but this time one instance is running and another disappeared in one hour , and after that i am not able to launch any more instances.

don't see any message related to instance launch in nc.log or cc.log , instance keeps in pending state for ever also assigned 0.0.0.0 address for both private and public

two addresses are available

[eucalyptus@my_cc ~]$ euca-describe-addresses
ADDRESS 10.208.137.150 i-399C0696 (eucalyptus)
ADDRESS 10.208.137.151 available (eucalyptus)
ADDRESS 10.208.137.152 available (eucalyptus)

[terminated instance was assigned 10.208.137.151]

i even restarted nc, but no change , after restarting nc i got this message in hybridfox

Not enough resources available: addresses (try --addressing private)

current availability
----------------------------
[eucalyptus@my_cc ~]$ euca-describe-availability-zones verbose
AVAILABILITYZONE eu_demo 10.208.137.146
AVAILABILITYZONE |- vm types free / max cpu ram disk
AVAILABILITYZONE |- m1.small 0002 / 0004 1 128 2
AVAILABILITYZONE |- c1.medium 0002 / 0004 1 256 5
AVAILABILITYZONE |- m1.large 0001 / 0002 2 2048 12
AVAILABILITYZONE |- m1.xlarge 0001 / 0002 2 2048 20
AVAILABILITYZONE |- c1.xlarge 0000 / 0001 4 2048 20

----------------------------------------------------------------------------------

i keep seeing these messages in cc.log though networking is working , do they signify anything
---------------------------------------------------------------------------------------------------------------------------------
[Thu Sep 9 02:51:57 2010][007274][EUCAERROR ] vnetAttachTunnels(): bad input params
[Thu Sep 9 02:51:57 2010][007274][EUCADEBUG ] maintainNetworkState(): failed to attach tunnels for vlan 10 during maintainNetworkState()
[Thu Sep 9 02:51:57 2010][007274][EUCAERROR ] shawn(): network state maintainance failed
[Thu Sep 9 02:51:57 2010][007524][EUCAERROR ] vnetAttachTunnels(): bad input params
[Thu Sep 9 02:51:57 2010][007524][EUCADEBUG ] maintainNetworkState(): failed to attach tunnels for vlan 10 during maintainNetworkState()
[Thu Sep 9 02:51:57 2010][007524][EUCAERROR ] shawn(): network state maintainance failed
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] monitor_thread(): running
[Thu Sep 9 02:51:58 2010][007770][EUCAINFO ] refresh_resources(): called
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] refresh_resources(): calling http://10.208.137.148:8775/axis2/services/EucalyptusNC
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] refresh_resources(): time left for next op: 60
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] refresh_resources(): node=10.208.137.148 mem=16155/14107 disk=1356/832 cores=4/2
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] refresh_resources(): done
[Thu Sep 9 02:51:58 2010][007770][EUCAINFO ] refresh_instances(): called
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] refresh_instances(): timeout(60/60) len
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] refresh_instances(): timeout(60/60) inst
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] refresh_instances(): describing instance i-399C0696, Extant, 0
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] find_instanceCache(): found instance in cache 'i-399C0696/10.208.137.150/10.10.1.2'
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] ccInstance_to_ncInstance(): setting volumesSize: 1
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] refresh_instances(): storing instance state: i-399C0696/Extant/10.208.137.150/10.10.1.2
[Thu Sep 9 02:51:58 2010][007770][EUCADEBUG ] refresh_instances(): done

-------------------------------------------------------------------------

networking config on cc
VNET_PUBINTERFACE="eth0"
VNET_PRIVINTERFACE="eth0"
#VNET_BRIDGE="xenbr0"

VNET_MODE="MANAGED-NOVLAN"
VNET_SUBNET="10.10.0.0"
VNET_NETMASK="255.255.0.0"
VNET_DNS="130.35.249.52"
VNET_ADDRSPERNET="32"
VNET_PUBLICIPS="10.208.137.150 10.208.137.151 10.208.137.152"

-------------------------------------

messages in cc.log related to instance CC machine (10.208.137.146)

159474 [Thu Sep 9 00:11:14 2010][007770][EUCAINFO ] refresh_resources(): called
159475 [Thu Sep 9 00:11:14 2010][007770][EUCADEBUG ] refresh_resources(): calling http://10.208.137.148:8775/axis2/services/EucalyptusNC
159476 [Thu Sep 9 00:11:14 2010][007770][EUCADEBUG ] refresh_resources(): time left for next op: 60
159477 [Thu Sep 9 00:11:14 2010][007770][EUCADEBUG ] refresh_resources(): node=10.208.137.148 mem=16155/14107 disk=3937/3413 cores=4/2
159478 [Thu Sep 9 00:11:14 2010][007770][EUCADEBUG ] refresh_resources(): done
159479 [Thu Sep 9 00:11:14 2010][007770][EUCAINFO ] refresh_instances(): called
159480 [Thu Sep 9 00:11:14 2010][007770][EUCADEBUG ] invalidate_instanceCache(): invalidating instance 'i-55B208F8' (last seen 303 seconds ago)
159481 [Thu Sep 9 00:11:14 2010][007770][EUCADEBUG ] refresh_instances(): timeout(60/60) len
159482 [Thu Sep 9 00:11:14 2010][007770][EUCADEBUG ] refresh_instances(): timeout(60/60) inst

........................
.......................
.......................

163843 [Thu Sep 9 00:21:15 2010][025418][EUCAINFO ] DescribeInstances(): called
163844 [Thu Sep 9 00:21:15 2010][025418][EUCADEBUG ] DescribeInstances(): params: userId=eucalyptus, instIdsLen=0
163845 [Thu Sep 9 00:21:15 2010][025418][EUCAWARN ] vnetInitTunnels(): in MANAGED-NOVLAN mode, priv interface 'eth0' must be a bridge, tunneling disabled
163846 [Thu Sep 9 00:21:15 2010][025418][EUCADEBUG ] DescribeInstances(): returning: instanceId=i-399C0696, state=Extant, publicIp=10.208.137.150, privateIp=10.10.1.2, volumesSize=1
163847 [Thu Sep 9 00:21:15 2010][025418][EUCADEBUG ] DescribeInstances(): done
163848 [Thu Sep 9 00:21:15 2010][007276][EUCADEBUG ] DescribeNetworks(): done
163849 [Thu Sep 9 00:21:15 2010][007276][EUCAERROR ] vnetAttachTunnels(): bad input params
163850 [Thu Sep 9 00:21:15 2010][007276][EUCADEBUG ] maintainNetworkState(): failed to attach tunnels for vlan 10 during maintainNetworkState()
163851 [Thu Sep 9 00:21:15 2010][007276][EUCAERROR ] shawn(): network state maintainance failed
163852 [Thu Sep 9 00:21:15 2010][025418][EUCAERROR ] vnetAttachTunnels(): bad input params
163853 [Thu Sep 9 00:21:15 2010][025418][EUCADEBUG ] maintainNetworkState(): failed to attach tunnels for vlan 10 during maintainNetworkState()
163854 [Thu Sep 9 00:21:15 2010][025418][EUCAERROR ] shawn(): network state maintainance failed
163855 [Thu Sep 9 00:21:15 2010][007275][EUCAERROR ] vnetAttachTunnels(): bad input params
163856 [Thu Sep 9 00:21:15 2010][007275][EUCADEBUG ] maintainNetworkState(): failed to attach tunnels for vlan 10 during maintainNetworkState()
163857 [Thu Sep 9 00:21:15 2010][007275][EUCAERROR ] shawn(): network state maintainance failed
163858 [Thu Sep 9 00:21:15 2010][007273][EUCAWARN ] vnetInitTunnels(): in MANAGED-NOVLAN mode, priv interface 'eth0' must be a bridge, tunneling disabled
163859 [Thu Sep 9 00:21:15 2010][007273][EUCAINFO ] TerminateInstances(): called
163860 [Thu Sep 9 00:21:15 2010][007273][EUCADEBUG ] TerminateInstances(): params: userId=UNSET, instIdsLen=1
163861 [Thu Sep 9 00:21:15 2010][007273][EUCAINFO ] TerminateInstances(): calling terminate instance (i-55B208F8) on (10.208.137.148)
163862 [Thu Sep 9 00:21:15 2010][007559][EUCAERROR ] ERROR: TerminateInstance() could not be invoked (check NC host, port, and credentials)
163863 [Thu Sep 9 00:21:15 2010][007273][EUCADEBUG ] TerminateInstances(): call complete (pid/rc): 7559/1
163864 [Thu Sep 9 00:21:15 2010][007273][EUCADEBUG ] TerminateInstances(): done.
163865 [Thu Sep 9 00:21:15 2010][007273][EUCAERROR ] vnetAttachTunnels(): bad input params
163866 [Thu Sep 9 00:21:15 2010][007273][EUCADEBUG ] maintainNetworkState(): failed to attach tunnels for vlan 10 during maintainNetworkState()
163867 [Thu Sep 9 00:21:15 2010][007273][EUCAERROR ] shawn(): network state maintainance failed
163868 [Thu Sep 9 00:21:16 2010][007274][EUCAWARN ] vnetInitTunnels(): in MANAGED-NOVLAN mode, priv interface 'eth0' must be a bridge, tunneling disabled
163869 [Thu Sep 9 00:21:16 2010][007274][EUCAINFO ] UnassignAddress(): called
163870 [Thu Sep 9 00:21:16 2010][007274][EUCADEBUG ] UnassignAddress(): params: userId=UNSET, src=10.208.137.151, dst=10.10.1.3
163871 [Thu Sep 9 00:21:16 2010][007274][EUCADEBUG ] vnetApplySingleTableRule(): applying single table (nat) rule (-D PREROUTING -d 10.208.137.151 -j DNAT --to-destination 10.10.1.3)
163872 [Thu Sep 9 00:21:16 2010][007274][EUCADEBUG ] vnetApplySingleTableRule(): applying single table (nat) rule (-D OUTPUT -d 10.208.137.151 -j DNAT --to-destination 10.10.1.3)
163873 [Thu Sep 9 00:21:16 2010][007274][EUCADEBUG ] vnetApplySingleTableRule(): applying single table (nat) rule (-D POSTROUTING -s 10.10.1.3 -d ! 10.10.0.0/16 -j SNAT --to-source 10.208.137.1 51)
163874 [Thu Sep 9 00:21:16 2010][007274][EUCADEBUG ] UnassignAddress(): running cmd '/opt/eucalyptus//usr/lib/eucalyptus/euca_rootwrap ip addr del 10.208.137.151/32 dev eth0'
163875 [Thu Sep 9 00:21:16 2010][007274][EUCADEBUG ] UnassignAddress(): done
163876 [Thu Sep 9 00:21:16 2010][007770][EUCADEBUG ] monitor_thread(): running
163877 [Thu Sep 9 00:21:16 2010][007770][EUCAINFO ] refresh_resources(): called
163878 [Thu Sep 9 00:21:16 2010][007770][EUCADEBUG ] refresh_resources(): calling http://10.208.137.148:8775/axis2/services/EucalyptusNC
163879 [Thu Sep 9 00:21:16 2010][007770][EUCADEBUG ] refresh_resources(): time left for next op: 60
163880 [Thu Sep 9 00:21:17 2010][007770][EUCADEBUG ] refresh_resources(): node=10.208.137.148 mem=16155/14107 disk=3937/3413 cores=4/2

-----------------------------------------------------

on NC (10.208.137.148) nc.log messages :

208298:[Thu Sep 9 07:05:21 2010][017873][EUCAINFO ] - running instance i-55B208F8 directory, size=1596082092
208321:[Thu Sep 9 07:05:27 2010][017894][EUCAINFO ] vrun(): [rm -rf /opt/ext/instances/admin/i-55B208F8]
208341:[Thu Sep 9 07:05:34 2010][017912][EUCADEBUG ] scRecoverInstanceInfo: didn't find instance i-55B208F8!
208342:[Thu Sep 9 07:05:34 2010][017912][EUCAWARN ] WARNING: failed to recover Eucalyptus metadata of running domain i-55B208F8, ignoring it
208364:[Thu Sep 9 07:05:40 2010][017945][EUCADEBUG ] scRecoverInstanceInfo: didn't find instance i-55B208F8!
208365:[Thu Sep 9 07:05:40 2010][017945][EUCAWARN ] WARNING: failed to recover Eucalyptus metadata of running domain i-55B208F8, ignoring it
208386:[Thu Sep 9 07:05:41 2010][017969][EUCADEBUG ] scRecoverInstanceInfo: didn't find instance i-55B208F8!
208387:[Thu Sep 9 07:05:41 2010][017969][EUCAWARN ] WARNING: failed to recover Eucalyptus metadata of running domain i-55B208F8, ignoring it
208423:[Thu Sep 9 07:06:30 2010][018147][EUCADEBUG ] scRecoverInstanceInfo: didn't find instance i-55B208F8!
208424:[Thu Sep 9 07:06:30 2010][018147][EUCAWARN ] WARNING: failed to recover Eucalyptus metadata of running domain i-55B208F8, ignoring it
208482:[Thu Sep 9 07:08:27 2010][018490][EUCADEBUG ] scRecoverInstanceInfo: didn't find instance i-55B208F8!
208483:[Thu Sep 9 07:08:27 2010][018490][EUCAWARN ] WARNING: failed to recover Eucalyptus metadata of running domain i-55B208F8, ignoring it
208504:[Thu Sep 9 07:08:33 2010][018491][EUCADEBUG ] scRecoverInstanceInfo: didn't find instance i-55B208F8!

Please let me know if you need more information

Thanks
Madhu

Thu, 09/09/2010 - 17:25
#9
madhu_s
Offline
Joined: 04/19/2010
i started both the instances

i started both the instances in same security group , will that be a problem?

Thanks

Fri, 09/10/2010 - 00:04
#10
madhu_s
Offline
Joined: 04/19/2010
consistently terminating ,

consistently terminating , now the other instance also terminated.

here are logs ..

nc.log

136314:[Wed Sep 8 09:50:54 2010][006982][EUCAINFO ] started VM instance i-399C0696
.....
.....
.....
.....

232955:[Thu Sep 9 14:12:18 2010][020126][EUCAINFO ] vrun(): [rm -rf /opt/ext/instances/admin/i-399C0696]
232984:[Thu Sep 9 14:13:01 2010][020211][EUCADEBUG ] scRecoverInstanceInfo: didn't find instance i-399C0696!
232985:[Thu Sep 9 14:13:01 2010][020211][EUCAWARN ] WARNING: failed to recover Eucalyptus metadata of running domain i-399C0696, ignoring it
233003:[Thu Sep 9 14:13:01 2010][020218][EUCADEBUG ] scRecoverInstanceInfo: didn't find instance i-399C0696!

cc.log

23745:[Thu Sep 9 07:18:51 2010][000487][EUCADEBUG ] DescribeInstances(): returning: instanceId=i-399C0696, state=Extant, publicIp=10.208.137.150, privateIp=10.10.1.2, volumesSize=1
23781:[Thu Sep 9 07:18:57 2010][000413][EUCADEBUG ] DescribeInstances(): returning: instanceId=i-399C0696, state=Extant, publicIp=10.208.137.150, privateIp=10.10.1.2, volumesSize=1
23821:[Thu Sep 9 07:19:03 2010][000487][EUCADEBUG ] DescribeInstances(): returning: instanceId=i-399C0696, state=Extant, publicIp=10.208.137.150, privateIp=10.10.1.2, volumesSize=1
23861:[Thu Sep 9 07:19:09 2010][000412][EUCADEBUG ] DescribeInstances(): returning: instanceId=i-399C0696, state=Extant, publicIp=10.208.137.150, privateIp=10.10.1.2, volumesSize=1
23905:[Thu Sep 9 07:19:15 2010][000539][EUCADEBUG ] DescribeInstances(): returning: instanceId=i-399C0696, state=Extant, publicIp=10.208.137.150, privateIp=10.10.1.2, volumesSize=1
23924:[Thu Sep 9 07:19:17 2010][000453][EUCADEBUG ] invalidate_instanceCache(): invalidating instance 'i-399C0696' (last seen 302 seconds ago)
27827:[Thu Sep 9 07:29:21 2010][000414][EUCAINFO ] TerminateInstances(): calling terminate instance (i-399C0696) on (10.208.137.148)
30867:[Thu Sep 9 07:39:27 2010][000412][EUCAINFO ] TerminateInstances(): calling terminate instance (i-399C0696) on (10.208.137.148)

also launch instance is not possible .

xm list still shows the instance in output. but root image is not seen on the disk

Thanks
Madhu

Mon, 09/13/2010 - 20:54
#11
madhu_s
Offline
Joined: 04/19/2010
I think i found cause for

I think i found cause for this, libvirt is returning NOSTATE (by doing some bitmask op) sometimes when it sees '------' as state from xen,normally all my domains has '-b----' as status (xm list shows like that) .

as a workaround for now i added code to wait for a sec and check domain status again in node/handler.c

that fixed the problem , i kept 3 domains running since 24 hours, they are all running fine.

Thanks

Mon, 09/13/2010 - 21:33
#12
neil
Offline
Joined: 04/28/2009
Hello, which distro are you

Hello, which distro are you using? What is your xen version and libvirt version?

thanks.

Mon, 09/13/2010 - 21:45
#13
madhu_s
Offline
Joined: 04/19/2010
it is Ovs

it is
Ovs summary
ovs-release: 2.2.0
ovs-date: Wed Sep 23 2009
xen-release: 3.4.0-0.0.20
xen-date: Mon May 18 2009
xen-changeset: 19607

OVS is oracle virtual server

libvirt 0.8.3

kernel : 2.6.18-128.2.1.4.9.el5xen
Thanks

Wed, 09/15/2010 - 07:21
#14
dmitrii
Offline
Joined: 04/28/2009
Thanks!

Thanks for the letting us know!

Thu, 09/16/2010 - 13:03
#15
dmitrii
Offline
Joined: 04/28/2009
Feel free to submit the code

We'd love to see your workaround and maybe use it in some form, if you'd like to contribute:

http://open.eucalyptus.com/participate/contribute

Thanks!

Navigation
User login
Active forum topics
more