1、AWS VPC深入探討,主要議題,VPC設(shè)計原理, ,L2 L3,VPC新/高級特性, ,VPC S3 Endpoint VPC Flow Logs VPC Peering和其他連接方式,VPC實踐問題, ,Enhanced Networking Linux System Tuning,VPC設(shè)計原理,AWS云服務(wù),EBS,RDS,ElastiCache,Amazon Redshift,EC2,Elastic Load Balancing,客戶自有數(shù)據(jù)中心,白板工程化,EBS,RDS,ElastiCache,Amazon Redshift,EC2,Elastic Load Balancing,

2、EC2 曾經(jīng)是這樣, 7, 7,,為什么不工作,/16,路由表, /16: /32: 7/32: /32:,本地 AWS AWS AWS,/16,, 7,7,,需求, ,客戶指定的IP地址（段） 外部連接的路由聚合 與現(xiàn)有網(wǎng)絡(luò)設(shè)計的一致性,虛擬私有云,/1

3、6,路由表 /16: /18:,本地 AWS,/24 /18,/24, ,,2 1,這就是virtual networking!, ,子網(wǎng)= VLAN VPC = VRF (虛擬路由轉(zhuǎn)發(fā)) 但是,擴(kuò)展的挑戰(zhàn),VLAN ID數(shù)量上的限制,12位 = 4096個VLANs,VRF支持上的限制,大型路由器 = 1K - 2K 個VRF表,VLAN:VRF間的固定比率,路由器和容量緯度,Big Rout

4、er Control Plane Data Plane,Big Router Control Plane Data Plane,一個例子, ,路由配置平均每行: 每個VPC的配置數(shù): 每個VPC的子網(wǎng)數(shù): 每個子網(wǎng)的配置數(shù): 總VPC數(shù): 配置大小:,50 個字符 10 行 4 個 5 行 2,000 3 MB,但是,無法擴(kuò)展, ,12 位 VLAN ID = 4096 VLANs (遠(yuǎn)遠(yuǎn)不夠) 大型路由器最多支持到4000 VRFs ($200k+), ,大量的VLAN導(dǎo)致網(wǎng)絡(luò)工程師崩潰 受到供應(yīng)商Bug修復(fù)速度約束(6個月+) 需要日用品型的, 可替換型的網(wǎng)絡(luò)設(shè)備, ,少數(shù)幾家公司生產(chǎn)大型

5、虛擬路由器 高級特性等賣點(diǎn)并不包含好的互操作性,容量庫,A C G,C E G,0,1324/4,0 1,32/4,C A G,A C G,A A G,10 9 0 3 7,B D F,B D F,D D F,D D F,D F F,18/40,15/40,40 0 2 9,F B B B B,F B B B B,F B B B B,B B B B B,B B B B B,實現(xiàn)需求, ,縮放至 百萬個 A規(guī)模的環(huán)境 一個region中任何位置的任何服務(wù)器 都能夠 創(chuàng)建位于任意VPC的任意子網(wǎng)的實例,概念,Server Server 1

6、 ,Server Server ,VPC: ： ID: ： 實例 映射 VPC,客戶創(chuàng)建的服務(wù). 馬遜EC2實例 位于式查找 亞 分布所創(chuàng)建符， 的標(biāo)識,類似于 VPC + 實Cloud 映射 Private 例IP,映射服務(wù) 物理服務(wù)器: VPC亞馬遜數(shù)據(jù)中心的物理主機(jī) Virtualvpc-1a2B3c4d 到物理服務(wù)器,L2 - Ethernet,,,?,Ethernet Switch,收

7、到ARP 響應(yīng) 交換 Src: MAC() 對所有端口廣,播ARP請 具體地址為 求 L2 Dst: ff:ff:ff:ff:ff:ff MAC(),的端口 L3 Src:,L3 Who hasis at,L2 機(jī)會 MAC() 并了解 MAC() MAC() . ARP Dst: MAC() ICMP/TCP/UDP/,L2 - VPC,,Server Server 192

8、.168.1.4 ,映射服務(wù),L2 Src: Mapping Service MAC(),L2 Dst: Mapping Service ff:ff:ff:ff:ff:ff,ARP Who has Reply:,Host: Blue ?,Src: MAC() Dst: MAC() Query: is at MAC() MAC: MAC(),Server

9、 Server ,Src: ,Dst: Mapping Service,,Server Server ,Server Server ,Src: Mapping Service,Dst: ,L3 Dst: Mapping valid:,映射服務(wù) Src: 192

10、.168.0.3 Dst: VPC: Blue L2 MAC() L2 MAC() L3 Src: Validate: Blue isat ICMP/TCP/UDP/,L2 - VPC,VPC 隔離,,Server Server ,映射服務(wù),Server Server ,Gr

11、ey,L2 Src: ,L2 Dst: Mapping Service,ARP,Src: MAC() Dst: ff:ff:ff:ff:ff:ff Query: Who has ? ,VPC 隔離,,Server Server ,映射服務(wù),Server Server ,ARP,L2 Src: Src: is no

12、t,Dst: Dst: any L2 Mapping instances Service,Mapping Blue Denied, MAC() hosting ff:ff:ff:ff:ff:ff in VPC Blue. Query: Who has ? Alarm Raised,the instance.,VPC 隔離,Server Server ,Server

13、Server ,L3 Src: Mapping invalid!,Src: Mapping does not,L2 Src:the packet to Dst: Mapping Service ,映射服務(wù) Src: Dst: VPC: Blue Service deliver MAC() L2 Dst: MAC() Validate: Alarm is at 192.168.0

14、.4 ICMP/TCP/UDP/,L3 Dst: Raised. Blue ,L3 IP 路由,,,?,Ethernet Switch,ff:ff:ff:ff:ff:ff MAC(),L3 hasis Who ,L2 Src: MAC() L2 Dst: MAC() L3 Src: ARP Dst: at MAC() ICMP/TCP/UDP/,Router,Ethernet,Switch,L2 Src: MAC(1

15、) L2 Dst: MAC() L3 Src: L3 Dst: ICMP/TCP/UDP/,L3 - VPC,,Server Server ,映射服務(wù),L2 Src: Mapping Service,L2 Dst: Mapping Service ff:ff:ff:ff:ff:ff ,Reply: ARP Who has,? Host: Gateway

16、Blue,Src: MAC() Dst: MAC() Query: is at MAC() MAC: MAC(),Server Server ,L3 - VPC,Server Server ,Server Server 10.0.0.

17、4 ,映射服務(wù) Src: Dst: VPC: Blue Src: MAC() Dst: MAC() L3 Src: Validate: Host: ICMP/TCP/UDP/, Blue is at, MAC: MAC(),Src: L2 L2 Src: Mapping Service Mapping Service,L2

18、 MAC() L2Dst: Mapping Service ,L3 Dst: valid: Query: L3 Dst: Mapping Reply:,緩存,Server Server ,,Server Server ,映射服務(wù),L2 Src: MAC() L2 Dst: MAC() L3

19、 Src: L3 Dst: ICMP/TCP/UDP/,/18, /24,2 1 /24,與AWS之外互聯(lián) /16 Src: Dst: ? VPC: Blue L3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Edge Edge ,邊界（Edges） Server

20、 Server ,L3 Src: ,L3 Dst: ,/16 Edge,映射服務(wù) Src: Dst: VPC: Blue Host Host 7 ICMP/TCP/UDP/ ,Edges (三種情況),Src: Dst: VPC: Blue L

21、3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Src: 45 Dst: 4 IPSEC Stuff L3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Edge VPN,Edges (三種情況),Src: Dst: VPC: Blue L3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Src:

22、45 Dst: 4 802.1Q VLAN Tag L3 Src: L3 Dst: 7 ICMP/TCP/UDP/,Edge 直連DX,Edges (三種情況),Edge ,Src: Dst: VPC: Blue,L3 Src: L3 Dst: 90 ICMP/TCP/UDP/,,Internet,L3 Src: L3 Dst: 90

23、 ICMP/TCP/UDP/,Edges (三種情況),VPN DX Internet,Edge Edge Edge ,Src: Dst: VPC: Blue L3 Src: L3 Dst: 7 ICMP/TCP/UDP/ Src: Dst: VPC: Blue L3 Src: L3 Dst: 7 ICMP/TCP/UDP/ Src: 192.168.

24、0.3 Dst: VPC: Blue L3 Src: L3 Dst: 90 ICMP/TCP/UDP/,Src: 45 Dst: 4 IPSEC Stuff L3 Src: L3 Dst: 7 ICMP/TCP/UDP/ Src: 45 Dst: 4 802.1QVLAN Tag L3 Src: L3 Dst: 7 ICMP/TCP/UDP/ L3 Src:

25、 6 L3 Dst: 90 ICMP/TCP/UDP/,Image credit: Wikipedia /wiki/1918_Eighth_Avenue,有趣的,VPC 價格 每個VPC的價格: 每個suBnet的價格: 之上實例間流量價格:,$0.00 $0.00 $0.00,VPC的新特性和高級特性,私有子網(wǎng)對S3的高效訪問:,VPC S3 Endpoint,/0,VPC suBnet,VPC suBnet,NAT,/0,基于安全的VPC設(shè)計 Amazon-provide

26、d NAT instance image: amzn-ami-vpc-nat,, /24 /18,2,1 /24,為什么需要VPC S3 Endpoint,需要EMR運(yùn)行在 私有子網(wǎng)嗎,S3的VPC終端節(jié)點(diǎn),/24 /18,/24,,2,S3的VPC終端節(jié)點(diǎn),/24 /18,/24,172.31.1

27、.7,2,Server ,Edge ,Edge , ,,Server ,Src: Dst: ,邊界（Edges）,映射服務(wù),L3 Src: ,L3 Dst: ,/16 ,VPC: Blue Host Host 9 TCP/HTTP/ Ed

28、ge S3.us-east-1 Edge ,第四種Edge,Src: Dst: VPC: Blue L3 Src: L3 Dst: 9 TCP/HTTP/,Src: 45 Dst: 9 VPC Endpoint 1a2B3c4d L3 Src: L3 Dst: 9 TCP/HTTP/,Edge S3 endpoint, 172.31.

29、2.12,Statement: ,安全策略,/18,/24,/24,Statement: Sid: Access-to-specific-VPC-only, Principal: *, Action: s3:*, Effect: Deny, Resource: arn:aws:s3:my_secure_bucket, arn:aws:s3:my_secure_bucket/*, Condition: StringNotEquals: aws:sourceVpc: vpc-111bbb22 Sid: Access-to-specific

30、-bucket-only, Principal: *, Action: s3:GetObject, s3:PutObject , Effect: Allow, Resource: arn:aws:s3:my_secure_bucket, arn:aws:s3:my_secure_bucket/*, ,VPC Flow Logs,VPC Flow Logs: 了解網(wǎng)絡(luò)通信狀況, ,安全組規(guī)則效果立顯 網(wǎng)絡(luò)連通性排錯 大數(shù)據(jù)分析網(wǎng)絡(luò)流量 助力安全審計,可以在ENI，Subnet或者VPC級別開啟,創(chuàng)建Flow Log,CloudWatch中查看Log,10分鐘間隔累計和更新到CloudWatch

31、Logs,字段含義,VPC與VPC之間的互聯(lián):,VPC Peering以及其他連接方式,VPC對等的使用場景 通用的/核心服務(wù), ,認(rèn)證/AD 監(jiān)控 日志 遠(yuǎn)程管理 掃描,VPC peering,VPC Peering,/16,/16,步驟一：發(fā)起請求,/16,/16,步驟1 發(fā)起對等請求,步驟二：接受請求,/16,/16,步驟一 發(fā)起對等請求,步驟二 接受對等請求,步驟三：創(chuàng)建路由,/16,/16,步驟一,步驟三 創(chuàng)建路由,目標(biāo)地址為 發(fā)起對

32、等請求 到我的碗里來 步驟二 接受對等請求,VPC的,連接您的公司網(wǎng)絡(luò)和AWS,VPN,DX,CGW,VGW,HA IPSec隧道,VPN：基礎(chǔ)知識 /16,/16,192.168/16,您的網(wǎng)絡(luò)設(shè)備,VPC實踐問題,Enhanced Networking, HVM, SR-IO-V 提高性能 縮短延遲 降低抖動, 機(jī)型, ,C3 C4 D2 I2 M4 R3,第三方iperf測試（不代表AWS,僅供參考）,http:/B,Linux system Tuning,去掉nf_conntrack*相關(guān)的限制對小包性能提升顯著：,之前： PacketSize

33、,THROUGHPUT,Packet(W)/s,MEAN_LATENCY(ms),P99_LATENCY(ms),MAX_LATENCY(ms),Retrans/s,R eransRate,78B, 61.715,9.891, 1.28251, 2.888, 648.408, 272.35, .2753%,之后： PacketSize,THROUGHPUT,Packet(W)/s,MEAN_LATENCY(ms),P99_LATENCY(ms),MAX_LATENCY(ms),Retrans/s,R eransRate,78B, 155.957,24.995, 0.512579, 0.76,

34、612.417, 223.10, .0892%,Linux system Tuning,1. 接收方的滑動窗口,Linux system Tuning,2. 發(fā)送方的擁塞窗口和慢啟動 $ ip route list default via dev eth0 /24 dev eth0 proto kernel scope link 54 dev eth0 scope link,1448,1448,1448,= 4344 Bytes,Linux system Tuning # ip route change /

35、24 dev eth0 proto kernel scope link initcwnd 16 $ ip route list default via dev eth0 /24 dev eth0 proto kernel scope link initcwnd 16 54 dev eth0 scope link,1448,1448,1448,1448, + 12 ,= 23168 Bytes,Linux system Tuning,3. 調(diào)整擁塞算法 $ sysctl net.ipv4.tcp_available_conges

36、tion_control net.ipv4.tcp_available_congestion_control = cubic reno $ find /lib/modules -name tcp_* # modprobe tcp_illinois $ sysctl net.ipv4.tcp_available_congestion_control net.ipv4.tcp_available_congestion_control = cubic reno illinois # sysctl net.ipv4.tcp_congestion_control=illinois net.ipv4.tc

37、p_congestion_control = illinois,Socket 層分析, ,$ ss -ite State,Recv-Q Send-Q Local Address:Port Peer, ,Address:Port ESTAB 0 3829960 8:https 5:52008 timer:(on,012ms,0) uid:498 ino:7116021 sk:0001c286 ts sack cubic wscale:7,7 rto:204 rtt:1.423/0.14 ato:40 mss:1448 cwnd:138 ssthresh:8

38、0 send 1123.4Mbps unacked:138 retrans:0/11737 rcv_space:26847 TCP State,Socket 層分析, ,$ ss -ite State,Recv-Q Send-Q Local Address:Port Peer, ,Address:Port ESTAB 0 3829960 8:https 5:52008 timer:(on,012ms,0) uid:498 ino:7116021 sk:0001c286 ts sack cubic wscale:7,7 rto:204 rtt:1.423/

39、0.14 ato:40 mss:1448 cwnd:138 ssthresh:80 send 1123.4Mbps unacked:138 retrans:0/11737 rcv_space:26847 Bytes queued for transmission,Socket 層分析, ,$ ss -ite State,Recv-Q Send-Q Local Address:Port Peer, ,Address:Port ESTAB 0 3829960 8:https 5:52008 timer:(on,012ms,0) uid:498 ino:7116021 sk:0001c286 ts sack cubic