Application Server: Health System: Creating A Custom HealthCheck And Action

Health Checks

Resin provides a convinient framework for monitoring your application. Along with the many built-in checks that allow monitoring JVM and core Resin metrics the framework provides a way to build custom health checks and actions.

Checks are invoked repeatedly at a set interval and can raise status alerts that help take actions to mediate any arising problems. Such custom checks can cause the framework to notify an administrator of an abnormal condition e.g. running out of disk space. There isn't a built-in check for disk space so in this tutorial we will build one.

We will start with implementing com.caucho.health.check.AbstractHealthCheck abstract class. Method public HealthCheckResult checkHealth(). An implemented method should return an instance of a HealthCheckResult. Depending on the result of its check the method may return a HealthCheckResult that indicates a normal condition or a problem of various degree, from warning to fatal.

For this tutorial we will implement a disk space check and configure it to send an email when disk space reaches a predefined low mark. This can be useful in real systems and is portable across platforms.

Custom Check: File System Check

The code for the FSCheck will look like following.

package com.acme.health;

import java.io.File;

import javax.ejb.Startup;
import javax.inject.Named;
import javax.inject.Singleton;

import com.caucho.config.Configurable;
import com.caucho.config.types.Bytes;

import com.caucho.env.health.HealthCheckResult;
import com.caucho.env.health.HealthStatus;

import com.caucho.health.check.AbstractHealthCheck;
/**
 * Tracks free space on a volume.
 * 
 * Generates a WARNING if free space drops below a value set in warning-level
 * 
 * Generates a CRITICAL if free space drops below a value set in critical-level
 *
 */
@Startup
@Singleton
@Configurable
@Named
public class FSCheck extends AbstractHealthCheck
{
  private File _volume;
  private long _warningLevel;
  private long _criticalLevel;

  @Configurable
  public void setVolume(File volume) {
    _volume = volume;
  }

  @Configurable
  public void setWarningLevel(Bytes bytes) {
    _warningLevel = bytes.getBytes();
  }

  @Configurable
  public void setCriticalLevel(Bytes bytes) {
    _criticalLevel = bytes.getBytes();
  }
 
  public HealthCheckResult checkHealth() {
    long size = _volume.getFreeSpace();
 
    HealthCheckResult result;
    if (size <= _criticalLevel) {
      String message = "volume " + _volume + " is below critical-level at " + size;
      result = new HealthCheckResult(HealthStatus.CRITICAL, message);
    }
    else if (size <= _warningLevel) {
      String message = "volume " + _volume + " is below warning-level at " + size;
      result = new HealthCheckResult(HealthStatus.WARNING, message);
    }
    else {
      result = new HealthCheckResult(HealthStatus.OK);
    }

    return result;
  }
}

---

Custom Action: Issue email for failed check.

Next, let's create a custom action that will send the messages. We will call the class EmailAction. We will make the EmailAction generic enough to be reused with other checks.

package com.acme.health;

import com.caucho.health.action.*;

import com.caucho.config.Configurable;
import com.caucho.env.health.HealthActionResult;
import com.caucho.env.health.HealthCheckResult;
import com.caucho.env.health.HealthService;
import com.caucho.env.health.HealthStatus;
import com.caucho.health.check.HealthCheck;
import com.caucho.health.event.HealthEvent;
import com.caucho.hemp.services.MailService;

import javax.ejb.Startup;
import javax.annotation.PostConstruct;
import javax.mail.Address;
import javax.mail.internet.AddressException;
import javax.mail.internet.InternetAddress;

@Startup
@Configurable
public class EmailAction extends AbstractHealthAction
{
  private MailService _mailService = new MailService();
  private String _subject;
  private HealthStatus _lastStatus;

  private HealthCheck _healthCheck;
 
  public EmailAction()
  {
  }

  @Configurable
  public void setHealthCheck(HealthCheck healthCheck)
  {
    _healthCheck = healthCheck;
  }

  @Configurable
  public void setEmailTo(String to) throws AddressException
  {
    Address[] addresses = InternetAddress.parse(to);
    for (Address address : addresses) {
      _mailService.addTo(address);
    }
  }

  @Configurable
  public void setEmailFrom(String from) throws AddressException
  {
    Address[] addresses = InternetAddress.parse(from);
    for (Address address : addresses) {
      _mailService.addFrom(address);
    }
  }

  @PostConstruct
  void actionInit() {
  	_mailService.init();
  }

  @Configurable
  public void setSubject(String subject)
  {
    _subject = subject;
  }

  public HealthActionResult doActionImpl(HealthEvent healthEvent)
  {
    HealthService healthService = healthEvent.getHealthSystem();
    HealthCheckResult result = healthService.getLastResult(_healthCheck);
      
    HealthStatus status = result.getStatus();

    if (!status.equals(_lastStatus)) {
      _mailService.send(_subject, result.getMessage());

      _lastStatus = status;
   }
}

Register Health Check and Action in health.xml

We are almost done. Now, we just need to add this configuration to health.xml

First, lets add a namespace mapping that will allow Resin to configure classes from com.acme.health package.

<resin xmlns="http://caucho.com/ns/resin"
      xmlns:resin="urn:java:com.caucho.resin"
      xmlns:health="urn:java:com.caucho.health"
      xmlns:ee="urn:java:ee"
      xmlns:myhealth="urn:java:com.acme.health">

Next, we need to create the FSCheck, EmailAction and bind them together.

 <myhealth:FSCheck ee:Named="fscheck">
   <volume>/tmp</volume>
   <warning-level>50M</warning-level>
   <critical-level>10M</critical-level>
 </myhealth:FSCheck>

 <myhealth:EmailAction healthCheck="${fscheck}">
   <health:IfHealthCritical/>
   <email-to>admin@localhost</email-to>
   <email-from>admin@localhost</email-from>
   <subject>Critical-Warning: Volume is low on space</subject>
 </myhealth:EmailAction>

 <myhealth:EmailAction healthCheck="${fscheck}">
   <health:IfHealthWarning/>
   <email-to>admin@localhost</email-to>
   <email-from>admin@localhost</email-from>
   <subject>Warning: Volume is low on space</subject>
 </myhealth:EmailAction>

Note, that we've created two EmailActions and mapped them to same HealthCheck. WIth using health:IfHealthWarning and health:ifHealthCritical predicates we will make sure that one action is only executed for critical errors, while the other - for warning.

Last, let's change the default ActionSequence that handles persistent CRITICAL health condition. We don't want resin to restart if the system runs out of /tmp space.

We will add a health:Not for our check.

 <health:ActionSequence>
   <health:Not>
     <health:IfHealthCritical healthCheck="${fscheck}"/>
   </health:Not>
   <health:IfHealthCritical time="30s"/>

   <health:FailSafeRestart timeout="10m"/>
   <health:DumpJmx/>
   <health:DumpThreads/>
   <health:DumpHeap/>
   <health:DumpHeap hprof="true" hprof-path="${resin.logDirectory}/heap.hprof"/>
   <health:StartProfiler active-time="2m" wait="true"/>
   <health:Restart/>
 </health:ActionSequence>