Since upgrading to Proxmox 5, I’ve had very little visibility into my server’s health. HPE hasn’t released packages for debian stretch (proxmox 5) and really for detailed information, those packages are my only choice. I only care about the RAID array however, and after poking around I found that the jessie package of hpacucli installs and works just fine. I used to just pop into the hpe web interface to check on health but that interface isn’t fully functional now. It’s a good time to automate this and do daily checks.
hpacucli can perform a quick query and give me everything I need. So I’m going to perform the query and write it to a file, then set up a cron job to perform that same query daily and just run a diff true/false on it. If true, I’m going to shoot myself an email with the output of the query.
hpacucli
at the time of writing this, here’s my uname -a output
Linux capehenlopen 4.10.15-1-pve #1 SMP PVE 4.10.15-15 (Fri, 23 Jun 2017 08:57:55 +0200) x86_64 GNU/Linux
I added the following hpe source
deb http://downloads.linux.hpe.com/SDR/repo/mcp jessie/current non-free
and following hpe’s recommendation, that line is in
/etc/apt/sources.list.d/mcp.list
Once the sources is added, the only thing needed here is the hpacucli package, so run an
apt-get update
apt-get install hpacucli
If you want to add the pub keys for hpe, follow their instructions or just run the following, then install hpacucli again without warnings.
curl http://downloads.linux.hpe.com/SDR/hpPublicKey1024.pub | apt-key add - curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | apt-key add - curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add - curl http://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -
However with proxmox, you may need to install curl first
apt-get install curl
mandatory word of warning, be very careful running curl against urls without first verifying the code there. Curl offers convenience at a huge cost if you’re lazy. Here we’re piping it to apt-key, not huge deal.
Once that’s all set, we should be able to run hpacucli and get our output. Here’s what my two servers provide.
root@capehenlopen:/raidlogs# hpacucli ctrl all show config Smart Array P410 in Slot 2 (sn: PACCRID12050DTV) array A (SATA, Unused Space: 0 MB) logicaldrive 1 (2.7 TB, RAID 5, OK) physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA, 1 TB, OK) physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA, 1 TB, OK) physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA, 1 TB, OK) physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA, 1 TB, OK) array B (SATA, Unused Space: 0 MB) logicaldrive 2 (465.7 GB, RAID 1, OK) physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 500 GB, OK) physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 500 GB, OK) SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 250 (WWID: 50014380188F7EBF)
and
root@gordonspond:~# hpacucli ctrl all show config Smart Array P410 in Slot 3 (sn: PACCRCN81ZZ1W0X) array A (SATA, Unused Space: 0 MB) logicaldrive 1 (2.7 TB, RAID 5, OK) physicaldrive 1I:0:1 (port 1I:box 0:bay 1, SATA, 1 TB, OK) physicaldrive 1I:0:2 (port 1I:box 0:bay 2, SATA, 1 TB, OK) physicaldrive 1I:0:3 (port 1I:box 0:bay 3, SATA, 1 TB, OK) physicaldrive 1I:0:4 (port 1I:box 0:bay 4, SATA, 1 TB, OK) SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 250 (WWID: 50014380118CDA7F)
So all the information here is perfect. It’s static unless physical changes or errors occur. I can create the baseline file and never have to update it unless I add drives or move things around. Next step is to set up SMTP.
POSTFIX
Proxmox comes with postfix pre installed, so we just need to edit it’s config. I use Sendgrid for SMTP so I’ll just paste the needed blob to make that work. That being said, I had authentication errors initially. Sendgrid’s documentation however pointed out a needed SASL authentication library for authenticating with them
So if using sendgrid, be sure to
apt-get install libsasl2-modules
and you should be good to go. Here’s a working config
# Sendgrid Settings smtp_sasl_auth_enable = yes smtp_sasl_password_maps = static:username:password smtp_sasl_security_options = noanonymous smtp_tls_security_level = encrypt header_size_limit = 4096000 relayhost = [smtp.sendgrid.net]:587
Which can just be appended to /etc/postfix/main.cf
In case you’ve never played with postfix, look to the following for logs if you have issues.
/var/log/mail.err /var/log/mail.log
mail.err likely won’t be much use unless you use a sasl password db and have permission errors, for sendgrid we just throw our credentials right in the config.
All that’s left is a test. Restart postfix and paste the following.
systemctl restart postfix echo "test body" | mail -s "test subject" -a "From: test@mzimmerer.com" your_email@provider.com
Putting it together
For this, I created a directory in my root directory, not important where this goes so long as it’s not on the RAID array we’re querying . We need to get our baseline file. I just did a
test=$(hpacucli ctrl all show config) echo "$test" > baseline.txt
(we could have just done hpacucli ctrl all show config > baseline.txt but within the script we compare the files using echo, so a “>” character show ups and flags the files as different. This solves that problem. We use echo in the script so we can run the hpacucli command just once and create a variable, rather than running it to check the diff, then running it again just to email it.)
and now we can run our diff against /raidlogs/baseline.txt
Below is the script
root@capehenlopen:/raidlogs# more chkraid.sh
#!/bin/bash
#run hp raid check and compare to baseline file.
#if diff, email contents of output
raidhlth=$(hpacucli ctrl all show config)
diff -q /raidlogs/baseline.txt <(echo "$raidhlth") > /dev/null
if [[ $? == 0 ]]
then
echo "$raidhlth" | mail -s "raid check OK" -a "From: gordonspond@mzimmerer.com" youremail@gmail.com
else
echo "$raidhlth" | mail -s "RAID array ALERT!" -a "From: gordonspond@mzimmerer.com" youremail@gmail.com
fi
I will change this up a bit. Once I see it working consistently I will get rid of the OK action. A duplicate script is on the other server as well, with a different From: address so I can identify the two. Really that’s it though, a very simple short script takes care of this for us. We just now need to add a cronjob to run this every day. Copy the script into /etc/cron.daily/ and ensure the execute bit is set and file extension is removed, we’re good to go.
cp /raidlogs/chkraid.sh /etc/cron.daily/chkraid chmod 755 /etc/cron.daily/chkraid
Enjoy!