Cloud: Cluster-Wide Configuration Using HTTP

Latest revision as of 00:00, 4 September 2012

Configuration Sharing: Cluster-Wide Configuration Using HTTP

In most cases, Resin can handle HTTP(S) URLs interchangeably with file paths. This provides the handy ability to pull configuration files from a central configuration repository over HTTP. Although it adds a possible single point of failure (the configuration server), for large clustered deployments the convenience of not having to synchronize configuration files to each machine can not be understated.

Note: Resin can also use cloud repository based configuration, which is stored in the Triad and shared between cluster members using Resin's internal .git implementation. While cloud configuration is slightly harder to work with than HTTP, it has the added advantage of having no SPOF, and requiring no additional HTTP server. An administrator can make the choice which works best for their architecture, or both can be used at the same time if desired.

Setup an HTTP Server

A single HTTP server will serve as your configuration repository for the entire cluster, or even multiple clusters/tiers if desired. You can use Resin for this also, but this server instance should have it's own local configuration and be entirely separate from other clusters.

For my example I've setup Apache on port 80 at "http://molson".

Setup Directory Structures

This can be as simple or complex as you desire. For this example, I've created a "resin" directory in my webserver's document root. Under that there is a directory for each environment: "dev", "qa", and "prod". Under each of these, there is a directory for each tier in the env: "web" and "app". (You may have more tiers also, such as "cache"). A tier is usually equivalent to a Resin cluster, and will used the same name.

/Library/WebServer/Documents
  |-resin
  |---dev
  |-----app
  |-----web
  |---prod
  |-----app
  |-----web
  |---qa
  |-----app
  |-----web

This makes the URL to the app cluster in my dev environment "http://molson/resin/dev/app".

Move Configuration Files

Move the following files from the Resin conf directory to the appropriate directory in your webserver document root, in my case "resin/dev/app"

• admin-users.xml
• cluster-default.xml
• health.xml
• resin.properties
• resin.xml

So now I can directly reference resin.xml at "http://molson/resin/dev/app/resin.xml"

You should notice that Resin XML configuration often references "${__DIR__}" for imports. This EL variable always refers to the current directory where the XML resides, and conveniently this works fine with HTTP paths. Thus with resin.xml loaded from http://molson/resin/dev/app/resin.xml, ${__DIR__} will refer to "http://molson/resin/dev/app/", making loading the other files automatic.

Some of these files, or parts of them, the triad server addresses in particular, should be shared between clusters in the same environment. I'll discuss that in more detail below.

Local resin.xml

Now we add a basic local conf/resin.xml that simply imports resin.xml from the HTTP configuration repository:

<resin xmlns="http://caucho.com/ns/resin"
      xmlns:resin="urn:java:com.caucho.resin">
  
 <resin:import path="http://molson/resin/dev/app/resin.xml"/>
 
</resin>

Use Environment Variables

The local resin.xml above has the server url hardcoded. It would be more convenient to define environment variables on each machine, or on the startup command line, so that our resin.xml is not environment specific.

Any environment variables can be referenced directly in resin.xml with no special setup.

For my example I'll export 3 environment variables that defined which environment this machine is a member of. This would most likely go in your ~/.bash_profile or /etc/profile file.

export RESIN_CONFIG_URL=http://molson/resin
export RESIN_ENV=dev
export RESIN_CLUSTER=app

Now we can modify the local resin.xml to use the machine's environment variables, making the local resin.xml quite generic:

<resin xmlns="http://caucho.com/ns/resin"
      xmlns:resin="urn:java:com.caucho.resin">
 
 <resin:import path="${RESIN_CONFIG_URL}/${RESIN_ENV}/${RESIN_CLUSTER}/resin.xml"/>
 
</resin>

Note: this will force all servers on the machine into the same environment and cluster. If you want to run multiple servers on the same machine in more than one cluster, you also have the option of setting variables on the command-line during startup, as show below.

$> RESIN_CLUSTER=app resinctl start

Environment Configuration Sharing

As I mentioned above, we can further eliminate copies of configuration files by sharing appropriate configuration between all clusters in the same environment.

These files make sense to share:

• admin-users.xml
• cluster-default.xml

So I'll move these up 1 directory in my webserver document root to "resin/dev".

And then modify the 2 import lines in resin/dev/app/resin.xml as such:

<resin:import path="${__DIR__}/../cluster-default.xml"/>

<resin:import path="${__DIR__}/../admin-users.xml" optional="true"/>

Finally, the web-tier must have a list of the app-tier servers for load balancing. Therefore it makes sense to create a separate file devoted to the server list alone. Comment out the servers from resin.properties, and create a file called servers.properties in the webserver root at resin/dev/servers.properties

web_servers      : 192.168.1.11:6810
app_servers      : 192.168.1.2:6800 192.168.1.12:6800 192.168.1.7:6800

Add add a reference to it in your repository resin.xml:

<resin:properties path="${__DIR__}/../servers.properties" optional="false"/>

Conclusion

To review, we did the following:

1. Setup a isolated HTTP server to act as our central configuration repository
2. Created separate subdirectories in the webserver document root to organize configuration files for each environment and cluster/tier
3. Moved configuration from the local resin conf directory to the appropriate directory on the conf server
4. Added environment variables to the local machine to configure the environment and cluster
5. Replaced the local resin.xml with a basic version that only contains a <resin:import> of the HTTP URL.
6. Create a shared servers.properties file that lists the addresses of all servers in the environment
7. Modified the remote resin.xml to import the shared configuration files

This architecture centralizes Resin configuration across all environments and tiers, while keeping things logically separate. For a large architecture with multiple environments and tiers, it should be a huge help avoiding the issues caused by copying around configuration files.

Sample Configuration

• The attached resin.tar.gz contains the entire webserver root directory I used in the examples.
• The attached resin.xml is the local resin.xml I used in the examples.