I was looking for a way to store (and graph) data for all of our physical servers. \u00a0Now sure, we could install SNMP on every machine and just use MRTG, but as a lot of them are leased by clients, I wanted something out of band which will ‘just work’. \u00a0We have a very nice IMPI based system which we use for provisioning every server on the network. \u00a0That seemed like a good way to go! \u00a0Pulling all the fun info via IPMI is really easy:<\/p>\n
$ ipmitool -I lan -H 10.x.x.x -U ADMINUSER -P PASSWORD sensor\r\nCPU1 Temp | 29.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 79.000 | 82.000 | 84.000 \r\nCPU2 Temp | 34.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 79.000 | 82.000 | 84.000 \r\nSystem Temp | 27.000 | degrees C | ok | -9.000 | -7.000 | -5.000 | 80.000 | 85.000 | 90.000 \r\nPeripheral Temp | 33.000 | degrees C | ok | -9.000 | -7.000 | -5.000 | 80.000 | 85.000 | 90.000 \r\nPCH Temp | 43.000 | degrees C | ok | -11.000 | -8.000 | -5.000 | 90.000 | 95.000 | 100.000 \r\nP1-DIMMA1 TEMP | 30.000 | degrees C | ok | 1.000 | 2.000 | 4.000 | 80.000 | 85.000 | 90.000 \r\nP1-DIMMB1 TEMP | 29.000 | degrees C | ok | 1.000 | 2.000 | 4.000 | 80.000 | 85.000 | 90.000 \r\nP1-DIMMC1 TEMP | na | | na | na | na | na | na | na | na \r\nP1-DIMMD1 TEMP | na | | na | na | na | na | na | na | na \r\nP2-DIMME1 TEMP | 31.000 | degrees C | ok | 1.000 | 2.000 | 4.000 | 80.000 | 85.000 | 90.000 \r\nP2-DIMMF1 TEMP | 31.000 | degrees C | ok | 1.000 | 2.000 | 4.000 | 80.000 | 85.000 | 90.000 \r\nP2-DIMMG1 TEMP | na | | na | na | na | na | na | na | na \r\nP2-DIMMH1 TEMP | na | | na | na | na | na | na | na | na \r\nFAN1 | na | | na | na | na | na | na | na | na \r\nFAN2 | 6375.000 | RPM | ok | 300.000 | 450.000 | 600.000 | 18975.000 | 19050.000 | 19125.000 \r\nFAN3 | na | | na | na | na | na | na | na | na \r\nFAN4 | 6450.000 | RPM | ok | 300.000 | 450.000 | 600.000 | 18975.000 | 19050.000 | 19125.000 \r\nFAN5 | 4575.000 | RPM | ok | 300.000 | 450.000 | 600.000 | 18975.000 | 19050.000 | 19125.000<\/pre>\nThat gives us quite a useful amount of info! \u00a0Though for our purposes, we’ll ignore everything except the first three columns.<\/p>\n
Handling that in PHP, we do something like the following:<\/p>\n
<?PHP\r\n$output = shell_exec('ipmitool -H ' . $server->internalIp . ' -U ' . $user . ' -P ' . $pw . ' sensor 2>&1');\r\n\/\/ Split into lines\r\n$lines = explode(\"\\n\", $output);\r\nforeach ($lines as $line) {\r\n \/\/ Remove whitespace\r\n $line = implode('', explode(' ', $line));\r\n \/\/ Explode by |\r\n $data = explode('|', $line);\r\n var_dump($data);\r\n \/\/ data is:\r\n \/\/ $data[0] = Name\r\n \/\/ $data[1] = Value\r\n \/\/ $data[2] = Units\r\n}<\/pre>\nThen we need to think about the RRD’s. \u00a0We need to check if we HAVE one, if so we put the data in. \u00a0Or we generate a new RRD and put the data in.<\/p>\n
if (!file_exists($rrdfile)) {\r\n \/\/ Create the RRD\r\n $options = array(\r\n \"--step\", \"300\", \/\/ Use a step-size of 5 minutes\r\n \"DS:\" . $data[0] . \":GAUGE:600:U:U\",\r\n \"RRA:AVERAGE:0.5:1:288\",\r\n \"RRA:MIN:0.5:1:288\",\r\n \"RRA:MAX:0.5:1:288\",\r\n \"RRA:AVERAGE:0.5:12:168\",\r\n \"RRA:MIN:0.5:12:168\",\r\n \"RRA:MAX:0.5:12:168\",\r\n \"RRA:AVERAGE:0.5:228:365\",\r\n \"RRA:MIN:0.5:228:365\",\r\n \"RRA:MAX:0.5:228:365\",\r\n );\r\n\r\n $ret = rrd_create($rrdfile, $options);\r\n if (!$ret) {\r\n echo \"<b>Creation error: <\/b>\" . rrd_error() . \"\\n\";\r\n }\r\n}<\/pre>\nWhat we’re doing above is creating the datastore, and creating Round Robin Archives for Min, Max, and Average. Storing a sample every 5 minutes for 24 hours (288 samples), a sample every hour for 7 days, and a sample every day for a year.<\/p>\n
Now we need to store our data. \u00a0That’s the easy part!<\/p>\n
\/\/ rewrite 'na' to 'U' for undefined\r\nif ($data[1]=='na') {\r\n $data[1] = 'U';\r\n}\r\n\/\/ insert the data\r\n$ret = shell_exec(\"\/usr\/bin\/rrdtool update \".$rrdfile.\" N:\".$data[1])<\/pre>\nAnd we’re now storing data. \u00a0Note the use of the shell_exec. \u00a0At least in PHP5.4 on Ubuntu, rrd_update and rrd_graph do not work.<\/p>\n
But that’s not much fun if we can’t display the graphs when we need them!<\/p>\n
I could write them out to file and then include them from static HTML etc. \u00a0But I’d rather have it all dynamically generated. \u00a0I have a php file which generates the HTML table referencing the graphs:<\/p>\n
echo '<table>';\r\n $rrds = ServerRRDS::model()->findAllByAttributes(array('serverId'=>$id));\r\n $i = 0;\r\n foreach ($rrds as $rrd) {\r\n\r\n \/\/ Table handling\r\n if ($i%2==0)\r\n echo '<tr>';\r\n\r\n echo '<td>';\r\n echo '<img src=\"\/index.php?r=graph\/drawGraph&orderId='.urlencode($rrd->orderId).'&dataName='.urlencode($rrd->dataName).'\">';\r\n echo '<\/td>';\r\n\r\n if ($i++%2==1)\r\n echo '<\/tr>';\r\n }\r\n echo '<\/table>';<\/pre>\nAnd then the actual graphing script:<\/p>\n
\/\/ Load the Server as well\r\n $server = Servers::model()->findByPk($rrd->orderId);\r\n \/\/ Clean up the data name to get what we use in the rrd file\r\n $dataname = str_replace(array('+','-','.'), array('PLUS', 'NEG', 'DOT'), $dataName);\r\n\r\n $options = array(\r\n \"--slope-mode\",\r\n \"--start \". $start,\r\n \"--title=\\\"\".$server->orderId.' - '.$rrd->dataName.\"\\\"\",\r\n \"--vertical-label=\".$rrd->dataUnit,\r\n \"--lower=0\",\r\n \"DEF:vname=\/path\/to\/rrds\/\".$rrd->fileName.\":\".$dataname.\":AVERAGE\",\r\n \"VDEF:vnamemin=vname,MINIMUM\",\r\n \"VDEF:vnamemax=vname,MAXIMUM\",\r\n \"VDEF:vnameavg=vname,AVERAGE\",\r\n \"COMMENT:\\\" \\\"\",\r\n \"COMMENT:\\\"Maximum \\\"\",\r\n \"COMMENT:\\\"Average \\\"\",\r\n \"COMMENT:\\\"Minimum \\l\\\"\",\r\n \"AREA:vname#0000BB:\".$rrd->dataName,\r\n \"GPRINT:vnamemin:\\\"%6.2lf \\\"\",\r\n \"GPRINT:vnamemax:\\\"%6.2lf \\\"\",\r\n \"GPRINT:vnameavg:\\\"%6.2lf \\l\\\"\",\r\n );\r\n\r\n $tmpfile = \"\/path\/to\/temp\/moo\".rand(11111111, 9999999).\".png\";\r\n\r\n $ret = shell_exec('\/usr\/bin\/rrdtool graph '.$tmpfile.' '.implode(\" \", $options).\" 2>&1\");\r\n\r\n header('Content-Type: image\/png');\r\n header('Content-Length: '. filesize($tmpfile));\r\n passthru($tmpfile);\r\n $fp = fopen($tmpfile,'rb');\r\n fpassthru($fp);\r\n fclose($fp);\r\n unlink($tmpfile);\r\n\r\n }<\/pre>\nAnd now it’s working very nicely indeed!<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
I was looking for a way to store (and graph) data for all of our physical servers. \u00a0Now sure, we could install SNMP on every machine and just use MRTG, but as a lot of them are leased by clients, I wanted something out of band which will ‘just work’. \u00a0We have a very nice […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[13,7,12,1],"tags":[],"class_list":["post-27","post","type-post","status-publish","format-standard","hentry","category-ipmi","category-linux","category-php","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=\/wp\/v2\/posts\/27"}],"collection":[{"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=27"}],"version-history":[{"count":2,"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=\/wp\/v2\/posts\/27\/revisions"}],"predecessor-version":[{"id":29,"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=\/wp\/v2\/posts\/27\/revisions\/29"}],"wp:attachment":[{"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=27"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=27"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/thewanderingsysadmin.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=27"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}