FusionReactor has many great features one of the most overlooked and possibly one of the most powerful is EnGuard Crash Protection (CP). This feature is able to alert you as well as prevent your servers from falling over by limiting the load based on a variety of criteria.
In this post I will walk you through how to setup the various settings for CP and the benefits of using these features.
Scenario - Application Server is running low on memory
Let us imagine the situation where, for whatever reason, the application server that you are running on is consistently running out of memory. By using FusionReactor's Crash Protection features we can be alerted to the situation and take steps to keep the system / application operational. Of course you will be investigating the cause of the memory issue, but in the mean time FusionReactor can buy you the time to ensure that service levels are maintained and you have minimized impact to your customers/business. By using the FusionReactor's "WebRequest Memory Protection" feature, you can limit the traffic on your server by either queuing or rejecting requests based on the current JVM memory usage. FusionReactor can also invoke system garbage collections if we exceed the threshold values, to help reduce the over all memory usage of the application.
Goal
In this situation the goal is to reduce the concurrent requests running on the application when the memory usage exceeds a specific value. What we want to do is make FusionReactor queue requests when the memory usage exceeds 80% for more that 15 seconds.
Steps
Our first action here is to enable the Crash Protection features. This does not require a restart and will take effect immediately.
- Go to Protection -> Settings
- Click on "WebRequest Memory Protection"
- Change the "WebRequest Memory Protection Threshold" to 80%
- Change the "Minimum condition Duration" to 15
- Change the "Memory Protection Strategy" to "Queue and Email Notification" or "Queue"
- Enable "Automatic Garbage Collection Trigger"
- Click "Save FusionReactor Settings"
[message type="warning"]If you are using the "Queue and Email Notifications" you will need to enable email notifications under Protection -> Settings -> Email Settings, you will also need valid Email settings in the FusionReactor -> Settings -> Email Settings page.[/message]
Results
When the system is setup correctly you will begin to see FusionReactor start queuing requests as they come in. These requests will be queued for 60 seconds, (configurable in the protection settings), before they timeout. This request queue is nicely displayed in the Request -> Activity screen, clearly showing the time in the queue and the time until the request is rejected. The request will be served as normal if the memory usage falls below 80% before the request times out.
The requests that are rejected based on the queue timeout, are tracked into a separate screen, Requests -> Rejections, and a summary is available on the WebMetrics page. Additionally the garbage collection metrics are tracked on the Resources -> Garbage Collection page (from FusionReactor 5 and later)
So hopefully at this stage you will have saved your system from collapsing from an out of memory error, kept the server running and bought yourself more time to diagnose the initial cause of the memory issues on the system. FusionReactor saves the day (AGAIN !).
Advanced Option
In addition to the setting described above the following options are available to improve the effectiveness of FusionReactor Crash Protection.
Garbage Collection
When using the memory protection settings feature, you might have noticed the "Automatic Garbage Collection Trigger". This feature is a very useful tool if you are experiencing memory issues on your application server. By using this setting you can configure FusionReactor to perform a garbage collection (GC) when the request starts. As you may be aware performing a garbage collection can be an expensive operation, as such, FusionReactor comes with a rate limiting option. The "Minimum Garbage Collection Interval" setting allows you to set the minimum time that must elapse before FusionReactor will invoke another garbage collection.
