Linux Health-Check Tool, Monitor Processes In Linux

Health-check Overview

The health-check tool monitors processes such as CPU usage, System call activity, Network connections and more to help you identify areas where it is consuming too many resources. It offers you the option to trace one or more processes and save the stats into a JSON formatted file for later analysis. At the moment I am writing this article only x86 (32 and 64 bit) and ARM architectures are supported.

Health-check can monitor:

– CPU usage
– Kernel wake-up events
– File I/O activity (open,read,write,close)
– System call activity
– Excessive polling of timeout wait blocked system calls (such as poll, select, etc)
– Memory usage (such as heap and stack growth)
– Network connections (to spot rogue internet activity)

Install Health-check On Ubuntu

sudo add-apt-repository ppa:colin-king/white
sudo apt-get update
sudo apt-get install health-check

Usage:

health-check [options] [command [options]]

Options:

-b          brief (terse) output
-c          find all child processes on start-up (only useful with -p option)
-d          specify the analysis duration in seconds (default is 60 seconds)
-f          follow fork/vfork/clone system calls
-h          show this help
-p pid[,pid]    specify process id(s) or process name(s) to be traced
-m max     specify maximum number of system calls to trace (default is 1000000)
-o file       output results to a json data file
-r          resolve IP addresses
-u user     run command as a specified user
-v         verbose output
-w         monitor wakelock count
-W         monitor wakelock usage (has overhead)

Test Health Check Using Application Name

Here i want to check the firefox health status. Enter the following command in Terminal. The firefox will be automatically open. After a minutes close firefox. The health check result will be shown on Terminal.

$ sudo health-check -b firefox

Sample output:

CPU usage:
 User:  13.91%, System:   8.29%, Total:  22.20% (medium load)

Wakeups:
 No wakeups detected.

Context Switches:
 7116.76 context switches/sec (high)

File I/O operations:
  I/O Operations per second: 188.26 open, 188.26 close, 163.40 read, 0.59 write

Polling system call analysis:
 firefox (10098), poll:
         30 immediate timed out calls with zero timeout (non-blocking peeks)
         10 repeated immediate timed out polled calls with zero timeouts (heavy polling peeks)

Filesystem Syncs:
 None.

Change in memory (K/second):
  PID  Process              Type        Size       RSS       PSS
 10098 firefox              Stack    9703.33     52.10     52.10 (growing very fast)
 10098 firefox              Heap    14836.21   3888.44   3888.44 (growing very fast)
 10098 firefox              Mapped 165303.52  11130.12   4908.20 (growing very fast)

Open Network Connections:
 None.

Analysis ran for just 3.3782 seconds, so rate calculations may be misleading

Test Health Check using PID

Enter the following command along with PID. You can find the PID using “top” command.

$ sudo health-check -p 9382

Sample output:

 CPU usage:
  PID  Process                USR%   SYS% TOTAL%   Duration
  9382 firefox               21.92   3.38  25.30      60.00  (medium load)

Wakeups:
  PID  Process               Wake/Sec Kernel Functions
  9382 firefox                   0.97 (hrtimer_start_range_ns, ehci_hrtimer_func) (low)
  9382 firefox                   0.08 (hrtimer_start_range_ns, hrtimer_wakeup) (very low)
 Total                           1.05

Context Switches:
  PID  Process                Voluntary   Involuntary     Total
                             Ctxt Sw/Sec  Ctxt Sw/Sec  Ctxt Sw/Sec
  9382 firefox                   3929.55        41.40      3970.95 (high)

File I/O operations:
  PID  Process               Count  Op  Filename
  9382 firefox                 184    R /home/sk/.mozilla/firefox/543j5fpy.default/content-prefs.sqlite
  9382 firefox                 119    W /home/sk/.mozilla/firefox/543j5fpy.default/cookies.sqlite-wal
  9382 firefox                  10    C /usr/share/fonts/truetype/msttcorefonts/Arial_Bold.ttf
  9382 firefox                  10    O /home/sk/.mozilla/firefox/543j5fpy.default/sessionstore.js.tmp
  9382 firefox                   2   CR /etc/hosts
  9382 firefox                   1  OCR /etc/hosts
  9382 firefox                   1   RW /home/sk/.mozilla/firefox/543j5fpy.default/cookies.sqlite-wal
  9382 firefox                   1   CW /home/sk/.mozilla/firefox/543j5fpy.default/sessionstore.js.tmp
 Total                         328
 Op: O=Open, R=Read, W=Write, C=Close

File I/O Operations per second:
  PID  Process                 Open   Close    Read   Write
  9382 firefox                 0.18    0.23    3.13    2.02

System calls traced:
  PID  Process              Syscall               Count    Rate/Sec
  9382 firefox              recvfrom              49658     827.6329
  9382 firefox              poll                  37694     628.2330
  9382 firefox              writev                11145     185.7499
  9382 firefox              futex                  5235      87.2500
  9382 firefox              read                   3873      64.5500
  9382 firefox              madvise                2944      49.0666
  9382 firefox              write                   210       3.5000
  9382 firefox              munmap                  108       1.8000
  9382 firefox              stat                     86       1.4333
  9382 firefox              mmap                     83       1.3833
  9382 firefox              getrusage                80       1.3333
  9382 firefox              getsockname              21       0.3500
  9382 firefox              fcntl                     5       0.0833
  9382 firefox              close                     5       0.0833
  9382 firefox              open                      5       0.0833
  9382 firefox              fstat                     5       0.0833
  9382 firefox              restart_syscall           1       0.0167
  9382 firefox              lseek                     1       0.0167
 Total                                           111159    1852.6491

Top polling system calls:
  PID  Process              Syscall             Rate/Sec   Infinite   Zero     Minimum    Maximum    Average
                                                           Timeouts Timeouts   Timeout    Timeout    Timeout
  9382 firefox              poll                  628.2330    23395    12627   0.0 sec  500.0 msec  18.5 msec

Distribution of poll timeout times:
                                                            10.0  100.0    1.0   10.0  100.0    1.0   10.0  100.0
                                                    up to    to     to     to     to     to     to     to  or more
                                              Zero    9.9   99.9  999.9    9.9   99.9  999.9    9.9   99.9        Infinite
  PID  Process              Syscall            sec   usec   usec   usec   msec   msec   msec    sec    sec    sec   Wait
  9382 firefox              poll             12627     -      -      -       6     54   1612     -      -      -   23395

Polling system call analysis:
 firefox (9382), poll:
      11793 immediate timed out calls with zero timeout (non-blocking peeks)
       7660 repeated immediate timed out polled calls with zero timeouts (heavy polling peeks)

Filesystem Syncs:
 None.

Per Process Memory (K):
  PID  Process              Type        Size       RSS       PSS
  9382 firefox              Stack     410408     79896     79896
  9382 firefox              Heap      631808    316412    316412
  9382 firefox              Mapped    732344     58324     43355

Change in memory (K/second):
  PID  Process              Type        Size       RSS       PSS
  9382 firefox              Stack       0.00    -40.73    -40.73 (shrinking moderately fast)
  9382 firefox              Heap     -313.60  -1428.60  -1428.60 (shrinking very fast)
  9382 firefox              Mapped    -16.60     -2.53     -1.33 (shrinking moderately fast)

Open Network Connections:
  PID  Process             Proto  Address:Port
  9382 firefox              UDP   127.0.1.1:53
  9382 firefox              TCP   37.59.47.177:443
  9382 firefox              TCP   198.50.164.162:80
  9382 firefox              TCP   184.173.90.195:80

For more details about health-check, refer the man pages.

$ man health-check