I was looking for a way to store (and graph) data for all of our physical servers. Now sure, we could install SNMP on every machine and just use MRTG, but as a lot of them are leased by clients, I wanted something out of band which will ‘just work’. We have a very nice IMPI based system which we use for provisioning every server on the network. That seemed like a good way to go! Pulling all the fun info via IPMI is really easy:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
$ ipmitool -I lan -H 10.x.x.x -U ADMINUSER -P PASSWORD sensor CPU1 Temp | 29.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 79.000 | 82.000 | 84.000 CPU2 Temp | 34.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 79.000 | 82.000 | 84.000 System Temp | 27.000 | degrees C | ok | -9.000 | -7.000 | -5.000 | 80.000 | 85.000 | 90.000 Peripheral Temp | 33.000 | degrees C | ok | -9.000 | -7.000 | -5.000 | 80.000 | 85.000 | 90.000 PCH Temp | 43.000 | degrees C | ok | -11.000 | -8.000 | -5.000 | 90.000 | 95.000 | 100.000 P1-DIMMA1 TEMP | 30.000 | degrees C | ok | 1.000 | 2.000 | 4.000 | 80.000 | 85.000 | 90.000 P1-DIMMB1 TEMP | 29.000 | degrees C | ok | 1.000 | 2.000 | 4.000 | 80.000 | 85.000 | 90.000 P1-DIMMC1 TEMP | na | | na | na | na | na | na | na | na P1-DIMMD1 TEMP | na | | na | na | na | na | na | na | na P2-DIMME1 TEMP | 31.000 | degrees C | ok | 1.000 | 2.000 | 4.000 | 80.000 | 85.000 | 90.000 P2-DIMMF1 TEMP | 31.000 | degrees C | ok | 1.000 | 2.000 | 4.000 | 80.000 | 85.000 | 90.000 P2-DIMMG1 TEMP | na | | na | na | na | na | na | na | na P2-DIMMH1 TEMP | na | | na | na | na | na | na | na | na FAN1 | na | | na | na | na | na | na | na | na FAN2 | 6375.000 | RPM | ok | 300.000 | 450.000 | 600.000 | 18975.000 | 19050.000 | 19125.000 FAN3 | na | | na | na | na | na | na | na | na FAN4 | 6450.000 | RPM | ok | 300.000 | 450.000 | 600.000 | 18975.000 | 19050.000 | 19125.000 FAN5 | 4575.000 | RPM | ok | 300.000 | 450.000 | 600.000 | 18975.000 | 19050.000 | 19125.000 |
That gives us quite a useful amount of info! Though for our purposes, we’ll ignore everything except the first three columns.
Handling that in PHP, we do something like the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
<?PHP $output = shell_exec('ipmitool -H ' . $server->internalIp . ' -U ' . $user . ' -P ' . $pw . ' sensor 2>&1'); // Split into lines $lines = explode("\n", $output); foreach ($lines as $line) { // Remove whitespace $line = implode('', explode(' ', $line)); // Explode by | $data = explode('|', $line); var_dump($data); // data is: // $data[0] = Name // $data[1] = Value // $data[2] = Units } |
Then we need to think about the RRD’s. We need to check if we HAVE one, if so we put the data in. Or we generate a new RRD and put the data in.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
if (!file_exists($rrdfile)) { // Create the RRD $options = array( "--step", "300", // Use a step-size of 5 minutes "DS:" . $data[0] . ":GAUGE:600:U:U", "RRA:AVERAGE:0.5:1:288", "RRA:MIN:0.5:1:288", "RRA:MAX:0.5:1:288", "RRA:AVERAGE:0.5:12:168", "RRA:MIN:0.5:12:168", "RRA:MAX:0.5:12:168", "RRA:AVERAGE:0.5:228:365", "RRA:MIN:0.5:228:365", "RRA:MAX:0.5:228:365", ); $ret = rrd_create($rrdfile, $options); if (!$ret) { echo "<b>Creation error: </b>" . rrd_error() . "\n"; } } |
What we’re doing above is creating the datastore, and creating Round Robin Archives for Min, Max, and Average. Storing a sample every 5 minutes for 24 hours (288 samples), a sample every hour for 7 days, and a sample every day for a year.
Now we need to store our data. That’s the easy part!
|
1 2 3 4 5 6 |
// rewrite 'na' to 'U' for undefined if ($data[1]=='na') { $data[1] = 'U'; } // insert the data $ret = shell_exec("/usr/bin/rrdtool update ".$rrdfile." N:".$data[1]) |
And we’re now storing data. Note the use of the shell_exec. At least in PHP5.4 on Ubuntu, rrd_update and rrd_graph do not work.
But that’s not much fun if we can’t display the graphs when we need them!
I could write them out to file and then include them from static HTML etc. But I’d rather have it all dynamically generated. I have a php file which generates the HTML table referencing the graphs:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
echo '<table>'; $rrds = ServerRRDS::model()->findAllByAttributes(array('serverId'=>$id)); $i = 0; foreach ($rrds as $rrd) { // Table handling if ($i%2==0) echo '<tr>'; echo '<td>'; echo '<img src="/index.php?r=graph/drawGraph&orderId='.urlencode($rrd->orderId).'&dataName='.urlencode($rrd->dataName).'">'; echo '</td>'; if ($i++%2==1) echo '</tr>'; } echo '</table>'; |
And then the actual graphing script:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
// Load the Server as well $server = Servers::model()->findByPk($rrd->orderId); // Clean up the data name to get what we use in the rrd file $dataname = str_replace(array('+','-','.'), array('PLUS', 'NEG', 'DOT'), $dataName); $options = array( "--slope-mode", "--start ". $start, "--title=\"".$server->orderId.' - '.$rrd->dataName."\"", "--vertical-label=".$rrd->dataUnit, "--lower=0", "DEF:vname=/path/to/rrds/".$rrd->fileName.":".$dataname.":AVERAGE", "VDEF:vnamemin=vname,MINIMUM", "VDEF:vnamemax=vname,MAXIMUM", "VDEF:vnameavg=vname,AVERAGE", "COMMENT:\" \"", "COMMENT:\"Maximum \"", "COMMENT:\"Average \"", "COMMENT:\"Minimum \l\"", "AREA:vname#0000BB:".$rrd->dataName, "GPRINT:vnamemin:\"%6.2lf \"", "GPRINT:vnamemax:\"%6.2lf \"", "GPRINT:vnameavg:\"%6.2lf \l\"", ); $tmpfile = "/path/to/temp/moo".rand(11111111, 9999999).".png"; $ret = shell_exec('/usr/bin/rrdtool graph '.$tmpfile.' '.implode(" ", $options)." 2>&1"); header('Content-Type: image/png'); header('Content-Length: '. filesize($tmpfile)); passthru($tmpfile); $fp = fopen($tmpfile,'rb'); fpassthru($fp); fclose($fp); unlink($tmpfile); } |
And now it’s working very nicely indeed!