1、Monitoring IO PerformanceUsing iostat and pt-diskstatsWritten by Ben MildrenPresented by Andrew Moore & David BusbyApril 2014Your SpeakersAndrew MooreUK based rDBA for Perconamysqlboy6+ years Production DBA experienceeroomydnaPrevious roles with Nokia and PythianBristol MySQL MeetupInterests in foot

2、ball, comedy, ethics & philosophy, technologyYour SpeakersDavid BusbyUK based rDBA EMEA Lead / Security Leadicleus14+ years Production sysadminascrethyJu-Jitsu Ni-Dan instructor for Not for profit.Volunteer instructor teaching computing to children.Your SpeakersPercona Remote DBARemote DBAs, 24/7 co

3、verageUnique customer portal providing support metrics Key Contributer to mydumper projectMaintainers of Percona Monitoring PluginsBest in business backup serviceTiered support modelsMonitoring IO on LinuxThere are a number of tools available to monitor IO on Linux. This presentation looks at two to

4、ols, iostat and pt-diskstats. These tools look to provide an overview of block devices iops, throughput and latency.You can dive deeper:Tools such as iotop and atop can be used to expose process level IO performance by gathering data from /proc/process/io.blktrace can be used with blkparse, either i

5、ndependently or by using btrace. Output can also be analysed using seekwatcher.Tools such as fio, bonnie/bonnie+, iometer, iozone, and ORION canbe used to benchmark a block device.iostatiostat is part of the sysstat utilities which is maintained by Sebastien Godard.sysstat is open source software wr

6、itten in C, and is available under the GNU General Public License, version 2.Other utilities in the sysstat package include mpstat, pidstat, sar, nfsiostat, and cifsiostat.sysstat is likely available from your favourite repo, but the latest version can be found here:http:/sebastien.godard.pagesperso

7、-orange.fr/download.htmlGood news! Sysstat is now on GitHub, the public repository is available here: /sysstat/sysstat for all willing to contribute!If you have an old (or very old) version installed, it is recommended youinstall the latest stable version.”iostatAt the time of writi

8、ng the current stable version is 10.2.1.Note version 10 removes support for kernels older than 2.6.The version you have installed can be found with -V option.userlab $ iostat -Vsysstat version 10.0.3(C) Sebastien Godard (sysstat orange.fr)iostatInclusion of the CPU and Device Utilization reports can

9、 be controlled with the -c and -d options.The Network Filesystem report (-n) was deprecated in version 10 and replaced with the nfsiostat and cifsiostat utilities.benlab $ iostat -cLinux 3.8.4-102.fc17.x86_64 (lab.mysqlhome)04/10/2013_x86_64_(2 CPU)avg-cpu:%user%nice %system %iowait%steal%idle 17.01

10、2.663.692.350.0074.29benlab $ iostat -dLinux 3.8.4-102.fc17.x86_64 (lab.mysqlhome)04/10/2013_x86_64_(2 CPU)Device:tpskB_read/skB_wrtn/skB_readkB_wrtn sda8.2894.6045.60926915446773dm-04.6575.006.3473486162104dm-17.2819.0038.75186129379644dm-117015024iostatThe default Device Utilization re

11、port can be replaced with extended statistics using the -xoption.userlab $ iostat -cx|iostat -xLinux 3.8.4-102.fc17.x86_64 (lab.mysqlhome)04/10/2013_x86_64_ (2 CPU)avg-cpu:%user%nice %system %iowait%steal%idle 16.232.273.672.200.0075.63Device:rrqm/swrqm/sr/sw/srkB/swkB/s avgrq- sz avgqu-szawait r_aw

12、ait w_awaitsvctm%utilsda1.342.673.334.3981.1545.4432.810.5166.4129.5994.307.776.00dm-00.000.003.240.8264.105.7234.380.3791.2935.84309.174.942.01.iostatThe default Device Utilization report can be replaced with extended statistics using the -x option.benlab $ iostat -dxLinux 3.8.4-102.fc17.x86_64 (la

13、b.mysqlhome)04/10/2013_x86_64_(2 CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/s avgrq-sz avgqu-sz await r_await w_awaitsvctm%utilsda1.342.673.324.3981.0945.4232.800.5166.4029.5994.277.776.00dm-00.000.003.240.8264.055.7134.380.3791.2935.84309.174.942.00dm-10.000.001.345.7716.5339.1215.660.3143.2032.9545.586

14、.424.56dm-20.000.000.089.560.0168.7526.9689.0016.210.25iostatWhilst reviewing the stats since since system start up can be useful, more often you will want to review current activity.Current activity can be displayed by specifying an interval measured in seconds. Output will continue unt

15、il interrupted or a specified count has been reached.The above command will display extended device statistics since system start up in the first report, and the deltas for the last 3 seconds in subsequent reports until interrupted.The above command will display extended device statistics since syst

16、em start up in the first report, and the deltas for the last 3 seconds in two further reports.userlab $ iostat -dx 3 3userlab $ iostat -dx interval countuserlab $ iostat -dx 3iostatUSERlab $ iostat -dx 3 3Linux 3.8.4-102.fc17.x86_64 (lab.mysqlhome) 04/10/2013_x86_64_(2 CPU)Device:rrqm/swrqm/sr/sw/sr

17、kB/swkB/s avgrq-sz avgqu-szawait r_await w_awaitsvctm%util sda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69dm-00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80dm-10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37dm-20.000.000.0010.680.0163.3026.9677.3415.4

18、70.24Device:rrqm/swrqm/sr/sw/srkB/swkB/s avgrq-sz avgqu-szawait r_await w_awaitsvctm%util sda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33dm-00.000.000.000.000.000.000.000.000.000.000.000.000.00dm-10.000.000.005.330.0021.338.000.035.250.005.254.382.33dm-20.000.000.000.000.000.000.000.000

19、.000.000.000.000.00Device:rrqm/swrqm/sr/sw/srkB/swkB/s avgrq-sz avgqu-szawait r_await w_awaitsvctm%util sda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87dm-00.000.000.000.000.000.000.000.000.000.000.000.000.00dm-10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10dm-20.000.000.002.000.0

20、08.008.000.0945.500.0045.5018.833.77iostatSince version 7.1.3 the report can be made more readable by including the registered device mapper names using the -N option.Using the -N can skew the columns on each line, so it can be useful to also specify the -h option to keep the report easily readable.

21、Adding -k or -m will specify kB / mB per second respectively.(If the POSIXLY_CORRECT environment variable is NULL the data read / written is displayed in kB by default)Adding the -t option will include a timestamp with each report.Adding the -z option will exclude any inactive devices from the indiv

22、idual reports. From version 10.1.3, adding the -y option suppresses the first report showing statistics since system start up.iostat-t (timestamp reported with sample)-z (report only includes active devices for this sample)-N (registered device mapper name)-h (columns stay aligned even with long dev

23、ice names)userlab $ iostat -dxNhtz 3 2Linux 3.8.4-102.fc17.x86_64 (lab.mysqlhome) 04/10/2013_x86_64_(2 CPU)04/10/2013 11:37:08 AMDevice:rrqm/swrqm/sr/sw/srkB/swkB/s avgrq-sz avgqu-szawait r_await w_await svctm %util sda1.072.632.674.1765.0443.6031.760.4363.1729.5484.728.065.52vg_proddba-lv_root0.000

24、.002.590.7451.364.9433.800.3089.5135.81 276.285.091.70vg_proddba-lv_home0.000.001.085.5813.2737.9315.360.2841.3232.8242.976.394.26vg_proddba-lv_tmp0.000.000.0311.040.0160.8426.9672.70 15.280.2304/10/2013 11:37:11 AMDevice:rrqm/swrqm/sr/sw/srkB/swkB/s avgrq-sz avgqu-szawait r_await w_awai

25、t svctm %util sda0.003.330.003.330.0034.6720.800.0928.100.0028.10 16.705.57vg_proddba-lv_home0.000.000.006.000.0034.6711.560.1016.000.0016.009.285.57iostatIn version 10.0.5, the devices in the Device Utilization can be grouped using the -g option.It is possible to specify multiple groups by supplyin

26、g the -goption multiple times.Its probably easier not to use the -N option, as the group devices named would have to reference the registered device mapper names.Using the -T option will display the group totals only.iostatbenlab $ iostat -Vsysstat version 10.0.5(C) Sebastien Godard (sysstat orange.

27、fr)userlab $ iostat -dx -g MyLVM dm-0 dm-1 dm-2 -g Other sdaLinux 3.8.4-102.fc17.x86_64 (lab.mysqlhome) 04/10/2013_x86_64_(2 CPU)Device:rrqm/swrqm/sr/sw/s rkB/swkB/s avgrq-sz avgqu-sz await r_await w_awaitsvctm%util dm-00.000.002.280.7645.534.7333.150.2480.1232.12 224.365.081.54dm-10.000.000.865.401

28、0.5737.4515.360.2539.9632.6041.136.404.00dm-20.000.000.030.160.091.0211.630.0149.6426.9654.00 14.570.28MyLVM0.000.003.166.3156.1943.2020.970.5053.0132.2063.446.151.94sda0.832.592.364.0756.4243.2030.960.3757.7326.6775.738.145.24Other0.832.592.364.0756.4243.2030.960.3757.7326.6775.738.145.24userlab $

29、iostat -dxT -g MyLVM dm-0 dm-1 dm-2 -g Other sdaLinux 3.8.4-102.fc17.x86_64 (lab.mysqlhome) 04/10/2013_x86_64_(2 CPU)Device:rrqm/swrqm/sr/sw/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_awaitsvctm%util MyLVM0.000.003.166.3156.1843.2020.970.5053.0132.2063.446.141.94Other0.832.592.364.0856.4143.203

30、0.960.3757.7326.6775.728.145.24iostatThe Device Utilization report can be produced on a partition level using the -p option.Historically partition statistics were restricted when moving from the2.4 kernel to 2.6 kernel, however the enhanced statistics were made available again from the 2.6.25 kernel

31、.The -p option was mutually exclusive to the -x option up unto version 8.1.8The partition focused report, can be limited to specific devices but not specific partitions.Currently the group options (-g and -T) dont work well with the partition focused report.pt-diskstatspt-diskstats is part of the pe

32、rcona toolkit.pt-diskstats is open source software written in Perl, and is available under the GNU General Public License, version 2.Other utilities in the percona toolkit include pt-stalk, pt-table-checksum, pt- table-sync, pt-query-digest, and pt-summary.Percona toolkit can be downloaded from http

33、://software/percona-toolkit/ or more simply from the command line:wget /get/percona-toolkit.tar.gz Tools can also be downloaded individually: wget /get/TOOLe.g. wget /get/pt-diskstatspt-diskstatsAt the time of writing the current stable version of perc

34、ona toolkit is version 2.2.7Full documentation of pt-diskstats can be found here: /doc/percona-toolkit/2.2/pt-diskstats.htmlThe version of pt-diskstats you have installed can be found with -versionoption.userlab $ pt-diskstats version Pt-diskstats 2.2.1pt-diskstatsBy default pt-

35、diskstats produces useful stats reporting current device activity. Inactive devices are hidden, but subsequently added if they become active.The default invocation shows the statistics for the last interval (which by default is 1 second), and will continue until interrupted.Similar to: iostat -dxy 1

36、pt-diskstatsuserlab $ ./pt-diskstats#ts devicerd_s rd_avkb rd_mb_s rd_mrg rd_cncrd_rtwr_s wr_avkb wr_mb_s wr_mrg wr_cncwr_rt busy in_prg io_s qtime stime1.0 sda0.00.00.00%0.00.01.08.00.050%0.00.04%11.0-5.8 17.51.0 sda30.00.00.00%0.00.01.08.00.050%0.00.00%01.00.00.01.0 dm-20.00.00.00%0.00.03.04.00.00

37、%0.00.04%13.0-2.9 11.71.0 sda1.04.00.00%0.172.013.04.00.132%0.211.1 25%014.00.7 12.31.0 sda31.04.00.00%0.172.07.07.40.146%0.01.49%08.00.06.41.0 dm-11.04.00.00%0.172.016.03.00.00%0.18.8 21%017.00.4 12.11.0 dm-20.00.00.00%0.00.00.00.00.00%0.10.04%00.00.00.01.0 sda0.00.00.00%0.00.01.028.00.086%0.00.04%

38、11.0-0.65.01.0 sda30.00.00.00%0.00.01.028.00.086%0.00.00%01.00.00.01.0 dm-10.00.00.00%0.00.00.00.00.00%0.00.00%00.00.00.01.0 dm-20.00.00.00%0.00.00.00.00.00%0.00.00%00.00.00.0pt-diskstatsThe interval can be adjusted using the -interval optionThe number of samples can be limited using the -iterations

39、 option. Similar to: iostat -dxy 3 3userlab $ ./pt-diskstats -interval 3 -iterations 3#ts devicerd_s rd_avkb rd_mb_s rd_mrg rd_cncrd_rtwr_s wr_avkb wr_mb_s wr_mrg wr_cncwr_rt busy in_prgio_s qtime stime3.0 sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.4 28.23.0 sda30.00.00.00%0.00.01.04.00.00%0.022.

40、72%01.00.0 22.73.0 dm-10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.03.0 dm-20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.0 30.03.0 sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.7 14.63.0 sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.0- dm-00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.0 18.03.0 d

41、m-10.00.00.00%0.00.01.34.00.00%0.00.01%11.3-1.89.03.0 dm-20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.0 27.0pt-diskstatsDisplayed columns can be restricted using perl regex with the -columns- regex option.Displayed devices can be restricted using perl regex with the -devices- regex option.Adding the

42、-show-timestamps option will include a timestamp with each report.The output can be grouped by sample or by disk using the -group-byoption.Samples (of /proc/diskstats) can be saved for later analysis using the-save-samples option.A Primer on Linux IOA Primer on Linux IOApplications in the user space

43、 make requests to the VFS.The request is passed to the block layer if the page is not in the page cache or the request is made using direct IO.Requests in the block layer can be split if the request is across multiple devices, or remapped from a device partition to the underlying block device. Reque

44、sts are handled by the IO scheduler on a per block device basis.Dependent on the scheduler, the request could be merged to the front orback of existing requests (all schedulers), or sorted in the request queue (cfq & deadline). The anticipatory scheduler was removed from the kernel in version 2.6.33

45、.Statistics are calculated at the block layer.A Primer on Linux IOTo understand what is acceptable performance theres no substitute to understanding the block device hardware.A block device is an abstraction, potentially it could be a 10 disk array with a hardware RAID controller, it could be a LUN

46、exposed from a SAN, even if its a single disk, the expected disk performance could vary greatly dependent on its specification.Expected IOPs, latency and throughput can be gathered via benchmarks or estimated using calculations:/hardware/iops-calculator-and-raid-calculators-esti

47、mators/ /strcalc/As a rough estimate you can expect: 75-100 iops from a 7200 rpm disk 125-150 iops from 10k rpm disk 175-200 iops from 15k rpm disk1000s iops from SSD (+)Block Layer Disk StatisticsFrom the 2.6 kernel, statistics are held for all block devices and partitions in /p

48、roc/diskstats./proc/diskstats lists the block devices major number, minor number, and name as well as a statistic set of 11 counters.Prior to 2.6.25 the statistic set of partitions was only made up of 4 counters and the counters werent consistent with the underlying block device statistics.Statistic

49、s are also held for individual devices and partitions in sysfs./sys/block/dev/stat holds the statistic set for the device./sys/block/dev/partition/stat holds the statistic set for the device partition.Block Layer Disk StatisticsFrom the 2.6 kernel, statistics are held for all block devices and parti

50、tions in /proc/diskstats./proc/diskstats lists the block devices major number, minor number, and name as well as a statistic set of 11 counters.Prior to 2.6.25 the statistic set of partitions was only made up of 4 counters and the counters werent consistent with the underlying block device statistic

51、s.Statistics are also held for individual devices and partitions in sysfs./sys/block/dev/stat holds the statistic set for the device./sys/block/dev/partition/stat holds the statistic set for the device partition./proc/diskstats (2.6.25+)userlab $ cat /proc/diskstats70 loop0 0 0 0 0 0 0 0 0 0 0 071 l

52、oop1 0 0 0 0 0 0 0 0 0 0 072 loop2 0 0 0 0 0 0 0 0 0 0 073 loop3 0 0 0 0 0 0 0 0 0 0 074 loop4 0 0 0 0 0 0 0 0 0 0 075 loop5 0 0 0 0 0 0 0 0 0 0 076 loop6 0 0 0 0 0 0 0 0 0 0 077 loop7 0 0 0 0 0 0 0 0 0 0 080 sda 44783 15470 2257302 1210711 85999 54224 1808924 6087675 0 1087763729834981 sda1 463 163 4176 6464 2 0 4 1 0 6215 646582 sda2 267 31 2136 4146 0 0 0 0 0 4053 414683 sda3 43885 15276 2249646 1197369 73520 54224 1808920 5575620 0 6545526772954110 sr0 0 0 0 0 0 0 0 0 0 0 02530 dm-0 42736 0 1796226 1391325 15414 0 187656 3199366 0 304001 45906972531 d