1、內(nèi)存熱點(diǎn)問題及案例分享,張永軍 企業(yè)客戶技術(shù)支持中心 支持服務(wù)事業(yè)部 中國惠普有限公司 2004.2,檢查內(nèi)存,1. dmesg: Memory Information: physical page size = 4096 bytes, logical page size = 4096 bytes Physical: 1048576 Kbytes, lockable: 790684 Kbytes, available: 918096 Kbytes 2.sam: samPerformance MonitorsSystem PropertiesMemory 3.syslog.log 4.glanc

2、e,Memory的角色,三個部分: 1. Physical memory 2. Available Memory The amount of main memory not reserved for the kernel is termed available memory. Available memory is used by the system for executing user processes. 3. Lockable Memory Pages kept in memory for the lifetime of a process by means of a system c

3、all (such as mlock, plock, or shmctl) are termed locked memory. Locked memory cannot be paged and processes with locked memory cannot be deactivated. Kernel parameter: unlockable_mem. Total available system memory - unlockable_mem = lockable memory,檢查空閑內(nèi)存,1. Top Memory: 126108K (79812K) real, 419716

4、K (324852K) virtual, 98184K free 2. Glance 3. Vmstat #vmstat 1 1 procs memory page faults cpu r b w avm free re at pi po fr de sr in sy cs us sy id 1 0 0 93405 5989 5 1 0 0 0 0 0 448 737 133 1 1 99,與內(nèi)存相關(guān)的內(nèi)核參數(shù),Buffer cache related: nbuf and bufpages =0 表示使用動態(tài)內(nèi)存 dbc_max_pct50 dbc_min_pct5 Buffer cache

5、 used by File system. 建議調(diào)低dbc_max_pct(尤其大量使用raw device)，否則占用太多內(nèi)存。,與內(nèi)存相關(guān)的內(nèi)核參數(shù)(續(xù)),vx_ninode=0 表示使用動態(tài)內(nèi)存,可能占用很大內(nèi)存, 造成系統(tǒng)內(nèi)存資源耗盡，還可能造成superpagepool corruption Or fragmentation。 unlockable_mem 使用缺省值(=0),內(nèi)存的分配,內(nèi)存分配的三種方式，需要相同數(shù)量的swap. plain memory as allocated with malloc(3C) system call. shared memory as allo

6、cated with shmget(2) system call. memory mapped files as allocated withmmap(2) system call.,內(nèi)存的分配（續(xù)）,一般內(nèi)存的分配由malloc(3C) 系統(tǒng)調(diào)用完成a) To report process memory usage, by largest first, run: # ps efl | sort rnk 10 | more And look at 10th column (SZ) to see the amount of memory used by this process for data

7、/text and stack (as this value is in pages, you can multiply by 4096 to determine the size in bytes.) Anytime you see that the size (SZ) is a four-digit number, watch it over time and see if it continues to grow. Alternative ps command - Alternatively, you can use the UNIX95 options to look at both

8、Virtual Size as well as the actual Size. Run: # UNIX95=1 ps -efo vsz,sz,pid,args |grep -v grep | sort -rnk 1 | more,內(nèi)存的分配（續(xù)）,b) And another way is to sort glances process list (or application list) by RSS (or Res Mem Virt Mem). For example: PROCESS LIST Users= 5 User CPU Util Cum Disk ThdProcess Nam

9、e PID PPID Pri Name ( 100% max) CPU IO Rate RSS Cnt- pax 13819 13818 148 root 2.7/ 5.8 273.3 9.4/32.8 284kb 1glance 14464 1822 158 root 2.1/ 3.1 3.0 0.0/ 2.1 4.3mb 1scopeux 1715 1 127 root 1.7/ 0.2 518.4 1.5/ 0.0 4.1mb 1swapper 0 0 127 root 1.5/ 0.8 2213.0 0.3/ 0.0 16kb 1java 10095 1 168 root 1.0/ 2

10、.7 348.7 0.0/ 4.2 42.0mb 28vxfsd 35 0 138 root 0.2/ 0.1 289.4 1.9/ 1.3 352kb 16 c) Unsupported tools: procsize & kmeminfo,內(nèi)存的分配（續(xù)）,2) 共享內(nèi)存的分配: shmget(2) system call. a) To see shared memory usage, ipcs can be used. For example, run: # ipcs mb| more To totalthe shared memory usage, run: # ipcs -mb |

11、sed -n /m/p | awk total+=$NFENDprintf(“%dn”, total) b) unsupported tool called shminfo,內(nèi)存的分配（續(xù)）,3) 內(nèi)存映像文件:mmap(2) system call. No tools available but unsupported shminfo.,用戶進(jìn)程內(nèi)存限制,maxdsiz,maxssiz,maxtsiz Shared memory limit: shmmax swap,32bit app,解釋,對通常的程序來說： 第一象限用來存放app text 第二象限用來存放data & stack 第三

12、四象限用來存放共享的數(shù)據(jù) The default usage for the quadrants is: Quadrant 1 Process text Quadrant 2 Process data Quadrant 3 Shared libraries Shared memory Memory mapped files Quadrant 4 Shared libraries Shared memory Memory mapped files The last part of quadrant 4 is reserved for system I/O.,32bit app Private 部

13、分,Maxdsiz & maxssiz 所決定的數(shù)據(jù)段和堆棧段位于同一個象限中，總和不能大于1GB。 Maxdsiz & maxssiz應(yīng)該合理配置，否則互相影響 Maxssiz一般設(shè)置成64MB已足夠 正常32bit應(yīng)用，數(shù)據(jù)段最大為940MB 安裝補(bǔ)丁，最大可以1.9GB。 編譯選項 ld -NOR“chatr +q3p enable executable_name” to enable third quadrant private,32bit app 共享部分,第3、4象限用來做32bit app的共享尋址 最大1.75GB SHMEM_MAGIC 將第二象限用于共享數(shù)據(jù)(這時最大為2.

14、75GB) To get SHMEM_MAGIC, the executable needs to have been linked (ld) with EXEC_MAGIC (the -N option) and chatred with SHMEM_MAGIC (the -M option).,64bit app,For 64 bit processes the quadrant sizes are 4tb, and therefore the maximum sizes for text, data, shared memory, etc, are quite large. For 64

15、 bit 11.00/11i executables there is currently no need to have different memory maps available as the standard one allows up to 4TB for the program text, another 4TB for its private data and a total of 8TB for shared areas.,考慮一下,16GB物理內(nèi)存 將多個數(shù)據(jù)庫置于同一臺機(jī)器 對32位數(shù)據(jù)庫，只有1.75GB內(nèi)存可用 怎么辦？,Memory Windows!,What do

16、es the Memory Windows feature do?,Enabling memory windows does change the kernels default allocation policy. With only the global memory window configured, the kernel allocates shared addresses from the 4th quadrant first and then the 3rd quadrant. When max_mem_window is set to a non-zero value (ena

17、bling memory windows) , the kernel changes the default allocation policy such that the 3rd quadrant is tried first followed by the 4th.,內(nèi)核設(shè)置,To enable the use of memory windows the kernel tunable max_mem_window must be set to the desired amount of memory windows. The disabled value is 0. A good defa

18、ult is 256. Note: This tunable is not dynamic.,內(nèi)核設(shè)置,To enable 256 memory windows in the kernel add the following line to the kernel configuration file (/stand/system): max_mem_window 256 Build a new kernel and reboot. Setting max_mem_window to 256 results in creating 256 memory windows, plus the glo

19、bal memory window. Setting max_mem_window to 1 would create 1 memory window, plus the global window, for a total of 2.,Services.window,Modify /etc/services.window . . database1 20 database2 30 database3 40,相關(guān)命令,getmemwindow is the command used to extract window ids of user processes from the /etc/se

20、rvices.window file. setmemwindow is the command that changes the window id of a running process or starts a specified program in a particular memory window.,限制,An application cannot dynamically switch between one window and another. Applications are only allowed to change their window at exec time.

21、Applications are not required to change their source code, but instead use the memory window command setmemwindow to start an application in a specified memory window.,例子,For example: setmemwindow -i 10 myprog arg1 arg2 arg3 This would start the executable myprog with 3 arguments in the memory windo

22、w corresponding to user key 10. There are other options to setmemwindow to control how memory windows are created.,例子（續(xù)）,getmemwindow is used to extract the user ids/keys from the /etc/services.window file given a particular string. This avoids having to embed the keys in the scripts themselves. syn

23、tax: getmemwindow ,例子（續(xù)）,Putting it all together in a real example: # cat startDB1.sh WinId=$(getmemwindow database1) setmemwindow -i $WinId /home/dave/memwinbb/startDB1 DB1 says hello!,例子（續(xù)）,# ./startDB1.sh writing to segment: DB1 says hello! Key is 1377312562,例子（續(xù)）,# ipcs -mob IPC status from /dev

24、/kmem as of Thu Jul 10 08:52:34 2003 T ID KEY MODE OWNER GROUP NATTCH SEGSZ Shared Memory: m 0 0 x4124288b -rw-rw-rw- root root 0 348 m 1 0 x4e000002 -rw-rw-rw- root root 1 61760 m 2 0 x41280050 -rw-rw-rw- root root 1 8192 m 5215 0 x52181f32 -rw-r-r- root sys 0 1024 Our key of 1377312562 converted t

25、o hex is 52181f32,例子（續(xù)）,# ./startDB2.sh writing to segment: DB2 says Bon Jour! Key is 1377312563 # ./startDB3.sh writing to segment: DB3 says Ola! Key is 1377042734,例子（續(xù)）,# more stopDB1.sh WinId=$(getmemwindow database1) setmemwindow -i $WinId /home/dave/memwinbb/stopDB1 # ./stopDB1.sh # ./stopDB2.s

26、h # ./stopDB3.sh,工具介紹,ps top glance,ps,# ps efl | sort rnk 10 | more And look at 10th column (SZ) to see the amount of memory used by this process for data/text and stack (as this value is in pages, you can multiply by 4096 to determine the size in bytes.) Anytime you see that the size (SZ) is a fou

27、r-digit number, watch it over time and see if it continues to grow. Alternative ps command - Alternatively, you can use the UNIX95 options to look at both Virtual Size as well as the actual Size. For example:# UNIX95=1 ps -efo vsz,sz,pid,args |grep -v grep | sort -rnk 1 | moreVSZ SZ PID COMMAND12252

28、 627 2745 /opt/OV/bin/ovdbrun -c /var/opt/OV/share/databases/analysis/9060 1214 2362 /opt/omni/lbin/rds -d8808 1892 2677 /opt/hpwebjet-5.5/hpwebjetd,top,Load averages: 0.64, 0.57, 0.56 195 processes: 194 sleeping, 1 running . . Memory: 1444296K (1238320K) real, 1967080K (1468152K) virtual, 84908K fr

29、ee | | | 1 23 45 Memory is not all of physcial, memory, it is: 分配給所有進(jìn)程的物理內(nèi)存DEDICATED to text, data or stack segments 所有runnable processes的物理內(nèi)存，不包括sleeping processes的。 分配給所有進(jìn)程的虛擬內(nèi)存DEDICATED to text, data or stack segments。不是所有都在物理內(nèi)存，有部分在swap區(qū)中。 分配給所有進(jìn)程的虛擬內(nèi)存，不包括sleeping or stopped processes 新進(jìn)程的可用物理內(nèi)存

30、，當(dāng)比較低時，swapping.,Top（續(xù)）,Size is virtual memory size RES means the part in RAM What if too many CPUs? top h What if I want to get all process info? top n500 f top.out,glance,Total VM : 121.1mb Sys Mem : 13.8mb User Mem: 91.3mb Phys Mem: 144.0mb Active VM: 73.7mb Buf Cache: 26.4mb Free Mem: 12.6mb Total VM: The total private virtual memory (in KBs unless otherwise specif