Hi, we have this issue in all our envs. Whenever tunneling is enabled we get intermittent NAT issues. You can see from this ping log the issue where internal vm replies. This causes connections to hang and/or drop.
64 bytes from 172.16.100.10: icmp_seq=463 ttl=63 time=0.685 ms
64 bytes from 172.16.100.10: icmp_seq=464 ttl=63 time=0.616 ms
64 bytes from 172.16.100.10: icmp_seq=465 ttl=63 time=0.644 ms
64 bytes from 172.16.100.10: icmp_seq=466 ttl=63 time=0.605 ms
64 bytes from 10.10.8.2: icmp_seq=467 ttl=63 time=0.726 ms
64 bytes from 10.10.8.2: icmp_seq=468 ttl=63 time=0.746 ms
64 bytes from 10.10.8.2: icmp_seq=469 ttl=63 time=0.602 ms
64 bytes from 10.10.8.2: icmp_seq=470 ttl=63 time=0.622 ms
64 bytes from 10.10.8.2: icmp_seq=471 ttl=63 time=0.680 ms
64 bytes from 10.10.8.2: icmp_seq=472 ttl=63 time=0.633 ms
64 bytes from 10.10.8.2: icmp_seq=473 ttl=63 time=0.759 ms
64 bytes from 10.10.8.2: icmp_seq=474 ttl=63 time=0.727 ms
64 bytes from 10.10.8.2: icmp_seq=475 ttl=63 time=0.713 ms
64 bytes from 10.10.8.2: icmp_seq=476 ttl=63 time=0.872 ms
64 bytes from 10.10.8.2: icmp_seq=477 ttl=63 time=0.910 ms
64 bytes from 172.16.100.10: icmp_seq=478 ttl=63 time=0.585 ms
64 bytes from 172.16.100.10: icmp_seq=479 ttl=63 time=0.724 ms
64 bytes from 172.16.100.10: icmp_seq=480 ttl=63 time=0.580 ms
64 bytes from 172.16.100.10: icmp_seq=481 ttl=63 time=0.673 ms
64 bytes from 172.16.100.10: icmp_seq=482 ttl=63 time=0.595 ms
64 bytes from 172.16.100.10: icmp_seq=483 ttl=63 time=0.629 ms
64 bytes from 172.16.100.10: icmp_seq=484 ttl=63 time=0.619 ms
64 bytes from 172.16.100.10: icmp_seq=485 ttl=63 time=0.719 ms
64 bytes from 172.16.100.10: icmp_seq=486 ttl=63 time=0.578 ms
64 bytes from 172.16.100.10: icmp_seq=487 ttl=63 time=0.570 ms
64 bytes from 172.16.100.10: icmp_seq=488 ttl=63 time=0.609 ms
64 bytes from 172.16.100.10: icmp_seq=489 ttl=63 time=0.573 ms
64 bytes from 172.16.100.10: icmp_seq=490 ttl=63 time=0.594 ms
64 bytes from 172.16.100.10: icmp_seq=491 ttl=63 time=0.722 ms
64 bytes from 172.16.100.10: icmp_seq=492 ttl=63 time=0.539 ms
64 bytes from 172.16.100.10: icmp_seq=493 ttl=63 time=0.511 ms
64 bytes from 10.10.8.2: icmp_seq=494 ttl=63 time=0.715 ms
64 bytes from 10.10.8.2: icmp_seq=495 ttl=63 time=0.783 ms
64 bytes from 10.10.8.2: icmp_seq=496 ttl=63 time=0.786 ms
64 bytes from 10.10.8.2: icmp_seq=497 ttl=63 time=0.915 ms
64 bytes from 10.10.8.2: icmp_seq=498 ttl=63 time=0.808 ms
64 bytes from 10.10.8.2: icmp_seq=499 ttl=63 time=0.785 ms
64 bytes from 10.10.8.2: icmp_seq=500 ttl=63 time=0.846 ms
64 bytes from 10.10.8.2: icmp_seq=501 ttl=63 time=0.978 ms
64 bytes from 10.10.8.2: icmp_seq=502 ttl=63 time=0.669 ms
64 bytes from 10.10.8.2: icmp_seq=503 ttl=63 time=0.826 ms
64 bytes from 172.16.100.10: icmp_seq=504 ttl=63 time=0.399 ms
64 bytes from 172.16.100.10: icmp_seq=505 ttl=63 time=0.580 ms
64 bytes from 172.16.100.10: icmp_seq=506 ttl=63 time=0.653 ms
64 bytes from 172.16.100.10: icmp_seq=507 ttl=63 time=0.648 ms
64 bytes from 172.16.100.10: icmp_seq=508 ttl=63 time=0.668 ms
64 bytes from 172.16.100.10: icmp_seq=509 ttl=63 time=0.678 ms
64 bytes from 172.16.100.10: icmp_seq=510 ttl=63 time=0.646 ms
64 bytes from 172.16.100.10: icmp_seq=511 ttl=63 time=0.707 ms
64 bytes from 172.16.100.10: icmp_seq=512 ttl=63 time=0.687 ms
64 bytes from 172.16.100.10: icmp_seq=513 ttl=63 time=0.686 ms
The env:
Eucalyptus 1.6.2 CentOS 5.5 x86_64
2 CCs on same isolated network as CLC
each CC has 2 NCs on isolated netowrok
The CLC is the gateway for the CCs and the CCs are the gateways for the NCs
Configs:
CLC:
EUCALYPTUS="/opt/eucalyptus"
EUCA_USER="eucalyptus"
DISABLE_DNS="Y"
ENABLE_WS_SECURITY="Y"
LOGLEVEL="DEBUG"
VNET_PUBINTERFACE="bond0"
VNET_PRIVINTERFACE="bond1"
VNET_DHCPDAEMON="/usr/sbin/dhcpd"
VNET_DHCPUSER="root"
CC_PORT="8774"
SCHEDPOLICY="ROUNDROBIN"
POWER_IDLETHRESH="300"
POWER_WAKETHRESH="300"
VNET_MODE="MANAGED"
CC1
EUCALYPTUS="/opt/eucalyptus"
EUCA_USER="eucalyptus"
DISABLE_EBS="Y"
DISABLE_DNS="Y"
DISABLE_TUNNELING="N"
ENABLE_WS_SECURITY="Y"
LOGLEVEL="DEBUG"
VNET_PUBINTERFACE="br0"
VNET_PRIVINTERFACE="br1"
VNET_DHCPDAEMON="/usr/sbin/dhcpd"
VNET_DHCPUSER="root"
CC_PORT="8774"
SCHEDPOLICY="ROUNDROBIN"
POWER_IDLETHRESH="300"
POWER_WAKETHRESH="300"
NODES=" 10.1.2.101 10.1.2.100"
NC_SERVICE="axis2/services/EucalyptusNC"
NC_PORT="8775"
VNET_MODE="MANAGED"
VNET_SUBNET="10.10.0.0"
VNET_NETMASK="255.255.0.0"
VNET_DNS="172.18.10.135 172.18.10.136"
VNET_ADDRSPERNET="256"
VNET_PUBLICIPS="172.16.100.2-172.16.100.254"
VNET_LOCALIP="172.16.3.1"
VNET_CLOUDIP="172.16.1.3"
CC2:
EUCALYPTUS="/opt/eucalyptus"
EUCA_USER="eucalyptus"
DISABLE_EBS="Y"
DISABLE_DNS="Y"
DISABLE_TUNNELING="N"
ENABLE_WS_SECURITY="Y"
LOGLEVEL="DEBUG"
VNET_PUBINTERFACE="br0"
VNET_PRIVINTERFACE="br1"
VNET_DHCPDAEMON="/usr/sbin/dhcpd"
VNET_DHCPUSER="root"
CC_PORT="8774"
SCHEDPOLICY="ROUNDROBIN"
POWER_IDLETHRESH="300"
POWER_WAKETHRESH="300"
NODES=" 10.1.2.101 10.1.2.100"
NC_SERVICE="axis2/services/EucalyptusNC"
NC_PORT="8775"
VNET_MODE="MANAGED"
VNET_SUBNET="10.10.0.0"
VNET_NETMASK="255.255.0.0"
VNET_DNS="172.18.10.135 172.18.10.136"
VNET_ADDRSPERNET="256"
VNET_PUBLICIPS="172.16.101.2-172.16.101.254"
VNET_LOCALIP="172.16.2.1"
VNET_CLOUDIP="172.16.1.3"
This issue as I mentioned only shows up when tunneling is enabled. It happens in both MANAGED and MANAGED-NOVLAN. I have confirmed that the vtun tunnels are up and working:
CC1:
2010-08-22T00:25:14.612479-07:00 mcloud000-rz1-cc1 vtund[20857]: VTUN server ver 3.X 02/10/2008 (stand)
2010-08-22T00:25:14.614271-07:00 mcloud000-rz1-cc1 vtund[20859]: Session tun-0-1[172.16.3.1] opened
2010-08-22T00:25:14.615548-07:00 mcloud000-rz1-cc1 vtund[20859]: Blowfish-128-ECB encryption initialized
2010-08-22T00:25:19.593543-07:00 mcloud000-rz1-cc1 vtund[20940]: Session tun-1-0[172.16.3.10:59636] opened
2010-08-22T00:25:19.594650-07:00 mcloud000-rz1-cc1 vtund[20940]: Blowfish-128-ECB encryption initialized
CC2:
2010-08-22T00:25:14.607230-07:00 mcloud000-rz2-cc1 vtund[25067]: Session tun-0-1[172.16.2.10:57850] opened
2010-08-22T00:25:14.609105-07:00 mcloud000-rz2-cc1 vtund[25067]: Blowfish-128-ECB encryption initialized
2010-08-22T00:25:19.584822-07:00 mcloud000-rz2-cc1 vtund[18707]: Connecting to 172.16.2.1
2010-08-22T00:25:19.586348-07:00 mcloud000-rz2-cc1 vtund[18707]: Session tun-1-0[172.16.2.1] opened
2010-08-22T00:25:19.587412-07:00 mcloud000-rz2-cc1 vtund[18707]: Blowfish-128-ECB encryption initialized
Any help you can give on figuring this out would be greatly appreciated.
Aimon
I believe I have solved this issue by changing the internal network of the VMs to 192.168.0.0/16. This appears to have given it no way to improperly traverse around NAT.. or whatever it was doing.
I will update with any other info I find on this issue.
Aimon
Still happens. This is nasty. Any help would be appreciated.
64 bytes from 192.168.38.10: icmp_seq=186 ttl=63 time=0.799 ms
64 bytes from 192.168.38.10: icmp_seq=187 ttl=63 time=0.785 ms
64 bytes from 192.168.38.10: icmp_seq=188 ttl=63 time=0.804 ms
64 bytes from 192.168.38.10: icmp_seq=189 ttl=63 time=0.780 ms
64 bytes from 192.168.38.10: icmp_seq=190 ttl=63 time=0.801 ms
64 bytes from 172.16.100.106: icmp_seq=191 ttl=63 time=0.558 ms
64 bytes from 172.16.100.106: icmp_seq=192 ttl=63 time=0.631 ms
64 bytes from 172.16.100.106: icmp_seq=193 ttl=63 time=0.639 ms
Within a few minutes... But your 2.0 eucalyptus.conf pointed out what I believe to be the cause. Both CCs were on same broadcast network. I remember there being a post about this a while back with diagrams and everything, but when u migrated you lost the images. The install and config docs say nothing of this... If it is the cause.. it would have been nice to have it properly documented in the Netowrk Configuration section of the docs. Could this be causing the issues I see?
This has been part of the 1.6 instructions for a while,
"BOTH modes: do not run two CCs in the same broadcast domain with tunneling enabled, this will potentially lead to a broadcast storm as tunnels start forwarding packets in a loop on your local network."
http://open.eucalyptus.com/wiki/EucalyptusNetworking_v1.6
Sorry if that was not prominent enough.
Neil, Wow.. how did I miss that?! Did you guys add it later then the original 1.6 release? I had read that doc more then a few times when 1.6 came out. Anyways, yes it needs to be a bit more prominent :) . Given the issues it causes and the difficulty in finding the cause.. I would put it in large bold somewhere inside the multi-cluster part of the doc.. atm its above that section. Adding it to the eicalyptus.conf as you did in 2.0 was a great idea! Also you are in dire need of a real network diagram including switching and example network segmentation. As I mentioned that one post was great.. but that was lost.
Aimon
Hi, this shows up even after separating the subnets. CC1 is on 172.16.2.0/24 brd 172.16.2.255, CC2 on 172.16.3.0/24 brd 172.16.3.255. They do share the same switches however. Can this be cause? Any thing else I should look at?
Thanks,
Aimon
This is the post I was talking about: http://open.eucalyptus.com/forum/network-problems-are-occurring-multi-cl...
"Greetings,
Thank you for your detailed report. The multi-cluster tunneling implementation currently requires that each cluster (CC+NCs) be connected to its own physical switch (ethernet broadcast domain). From your diagram, I believe that both of your CCs are connected to the same switch, which will lead to this problem. We're evaluating a solution that will not require that each cluster be on a separate switch, but in the 1.6 series, this is still a requirement. "
So does this mean they not only need to be on their own broadcast domain but also need to be on their own private switches? Doers this mean each Zone should be completely physically separated from eachother? Can they share the same gateway? Or do I have to keep them on completely different physical networks? I understand the the CC private network needs to be physically separate. And they are in our environment. Does this same switch issue refer to the CC public network also?
Aimon
a note on reproducing.. when I ping from CLC i get bad pings every 10 minutes or so and only for a short while:
64 bytes from 172.16.100.100: icmp_seq=5363 ttl=63 time=0.462 ms
64 bytes from 172.16.100.100: icmp_seq=5364 ttl=63 time=0.495 ms
64 bytes from 172.16.100.100: icmp_seq=5365 ttl=63 time=0.481 ms
64 bytes from 10.10.8.4: icmp_seq=5366 ttl=63 time=0.699 ms
64 bytes from 10.10.8.4: icmp_seq=5367 ttl=63 time=0.734 ms
64 bytes from 10.10.8.4: icmp_seq=5368 ttl=63 time=0.594 ms
64 bytes from 10.10.8.4: icmp_seq=5369 ttl=63 time=0.603 ms
64 bytes from 10.10.8.4: icmp_seq=5370 ttl=63 time=0.655 ms
64 bytes from 10.10.8.4: icmp_seq=5371 ttl=63 time=0.523 ms
64 bytes from 10.10.8.4: icmp_seq=5372 ttl=63 time=0.631 ms
64 bytes from 10.10.8.4: icmp_seq=5373 ttl=63 time=0.676 ms
64 bytes from 10.10.8.4: icmp_seq=5374 ttl=63 time=0.592 ms
64 bytes from 10.10.8.4: icmp_seq=5375 ttl=63 time=0.636 ms
64 bytes from 172.16.100.100: icmp_seq=5376 ttl=63 time=0.399 ms
64 bytes from 172.16.100.100: icmp_seq=5377 ttl=63 time=0.473 ms
64 bytes from 172.16.100.100: icmp_seq=5378 ttl=63 time=0.440 ms
64 bytes from 172.16.100.100: icmp_seq=5379 ttl=63 time=0.436 ms
64 bytes from 172.16.100.100: icmp_seq=5380 ttl=63 time=0.461 ms
64 bytes from 172.16.100.100: icmp_seq=5381 ttl=63 time=0.378 ms
64 bytes from 172.16.100.100: icmp_seq=5382 ttl=63 time=0.520 ms
64 bytes from 172.16.100.100: icmp_seq=5856 ttl=63 time=0.604 ms
64 bytes from 172.16.100.100: icmp_seq=5857 ttl=63 time=0.476 ms
64 bytes from 172.16.100.100: icmp_seq=5858 ttl=63 time=0.532 ms
64 bytes from 172.16.100.100: icmp_seq=5859 ttl=63 time=0.453 ms
64 bytes from 10.10.8.4: icmp_seq=5860 ttl=63 time=0.589 ms
64 bytes from 10.10.8.4: icmp_seq=5861 ttl=63 time=0.596 ms
64 bytes from 10.10.8.4: icmp_seq=5862 ttl=63 time=0.600 ms
64 bytes from 10.10.8.4: icmp_seq=5863 ttl=63 time=0.667 ms
64 bytes from 10.10.8.4: icmp_seq=5864 ttl=63 time=0.618 ms
64 bytes from 10.10.8.4: icmp_seq=5865 ttl=63 time=0.567 ms
64 bytes from 10.10.8.4: icmp_seq=5866 ttl=63 time=0.663 ms
64 bytes from 10.10.8.4: icmp_seq=5867 ttl=63 time=0.642 ms
64 bytes from 10.10.8.4: icmp_seq=5868 ttl=63 time=0.748 ms
64 bytes from 10.10.8.4: icmp_seq=5869 ttl=63 time=0.685 ms
64 bytes from 10.10.8.4: icmp_seq=5870 ttl=63 time=0.683 ms
64 bytes from 10.10.8.4: icmp_seq=5871 ttl=63 time=0.601 ms
64 bytes from 10.10.8.4: icmp_seq=5872 ttl=63 time=0.697 ms
64 bytes from 10.10.8.4: icmp_seq=5873 ttl=63 time=0.633 ms
64 bytes from 10.10.8.4: icmp_seq=5874 ttl=63 time=0.713 ms
64 bytes from 172.16.100.100: icmp_seq=5875 ttl=63 time=0.450 ms
64 bytes from 172.16.100.100: icmp_seq=5876 ttl=63 time=0.534 ms
64 bytes from 172.16.100.100: icmp_seq=5877 ttl=63 time=0.443 ms
64 bytes from 172.16.100.100: icmp_seq=5878 ttl=63 time=0.437 ms
64 bytes from 172.16.100.100: icmp_seq=5879 ttl=63 time=0.441 ms
64 bytes from 172.16.100.100: icmp_seq=5880 ttl=63 time=0.378 ms
Hello aimon,
if the CC have 2 NICs, and the private NICs are on *completely separated* subnet facings the NCs, and you specify properly in eucalyptus.conf which one is private and which one is the public facing NIC, then you should be able to have the 2 CCs sharing the same network toward the gateway and the external world.
Are the pings between instances on different clusters?
cheers
graziano
Hi Graziano. The NCs networks are physically completely separate. The CC is gateway for its NCs. The CLC is gateway for all CCs. The ping above is from the CLC to the VMs external IP. Did you get the diagram I sent? It is 100% accurate. below the CLC the network is isolated. traffic goes: world -> eth0 NAT on CLC -> eth1 CLC -> eth0 CC Euca NAT -> eth1 CC -> eth0 NC/VM. The CCs eth0 can see all other CCs. NCs from different zones cannot see each other directly, must go through CC.
This all make sense? Any ideas on what is causing this? Obviously it has to do with tunneling.. do the tunnels get rebuilt at some interval? What if the tunnel breaks? Does it cause this kind of behavior?
Aimon
Hello,
I didn't have received your diagram yet: we have been busy with releasing 2.0 :) I'll go and bug dmitrii tomorrow to get it.
The tunnel gets rebuild or reconnected periodically to be sure that it's all up and good, but it's not tore down, so it should stay up if is already up. You should see the commands on the cc.log when this happens: can you correlate the bad ping times with some cc activity?
cheers
graziano
Hi I found the REAL error behind this all. This actually can happen to EIP and non EIP IPs.
There are two scenarios:
1. VM has its external IP on local CC from its own zone.
-- in this the VM should NEVER traverse the tunnel. Yet the traffic crosses the tunnel when it should'nt and causes what you see in code block below.
2. VM has it's external IP on other zones CC.
-- In this case the VM should ALWAYS traverse tunnel.. but sometimes does not and causes leak.
64 bytes from 172.16.101.103: icmp_seq=283 ttl=63 time=0.592 ms (same route)
64 bytes from 172.16.101.103: icmp_seq=284 ttl=63 time=0.569 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=285 ttl=63 time=0.694 ms
RR: 172.16.3.8
10.10.23.1
10.10.23.8
10.10.23.8
172.16.2.10
172.16.3.8
64 bytes from 10.10.23.8: icmp_seq=286 ttl=63 time=0.701 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=287 ttl=63 time=0.684 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=288 ttl=63 time=0.749 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=289 ttl=63 time=0.726 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=290 ttl=63 time=0.767 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=291 ttl=63 time=0.739 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=292 ttl=63 time=0.654 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=293 ttl=63 time=0.743 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=294 ttl=63 time=0.782 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=295 ttl=63 time=0.894 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=296 ttl=63 time=0.668 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=297 ttl=63 time=0.614 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=298 ttl=63 time=0.728 ms (same route)
64 bytes from 10.10.23.8: icmp_seq=299 ttl=63 time=0.642 ms (same route)
64 bytes from 172.16.101.103: icmp_seq=300 ttl=63 time=0.393 ms
RR: 172.16.3.8
10.10.23.1
10.10.23.8
10.10.23.8
172.16.3.11
172.16.3.8
64 bytes from 172.16.101.103: icmp_seq=301 ttl=63 time=0.509 ms (same route)
Th reason this is possible is htat the same VM gateway exiss on both CCs:
11: eucabr25: mtu 1500 qdisc noqueue
link/ether 00:26:5a:85:2e:50 brd ff:ff:ff:ff:ff:ff
inet 10.10.23.1/24 brd 10.10.23.255 scope global eucabr25
. but the NAT rules only exist on the VMs local CC.
Your attention to this would be greatly appreciated.
Aimon
Correct. 1.6.2 on CentOS 5.5 x86_64 (running latest kernel and packages via yum). Thank you for checking into this. It is a nasty little one to wort out.. we have some solutions...(maybe not ideal) involves adding routes dynamically for vms who's EIP is in another zone. We have tested a number of solutions (one of which we have that seems stable).. would be glad to help your team in any way we can.
Best regards,
Aimon
Hi, so the question is in.. What happens every 100 seconds? I did a long ping and saw a pattern. Almost exactly every 100 seconds the the leak occurs for about 20 seconds. Does the interval 100 seconds means anything in relation to your code?
Aimon