This tutorial is prepared for Advanced WebLogic Administration and Automation. Setting up internal JMX MBean objects are the "best and ultimate practice" for applications' monitoring infrastructure.

Outline:

  1. How to develop and register JMX MBean to Oracle WebLogic Server?
  2. How to check SERVER disk space by using JMX MBean on WebLogic?
  3. How to define WLSDM User Defined Metric Actions?
  4. How to delete or move Application logs automatically when given threshold exceeds for disc usage on SERVER?
Problem: The application which is deployed to Oracle WebLogic Server 12c (12.1.3) is increasing server disk space continuously and in peak times it makes the server disk usage 100%. Meanwhile, the service goes down and causes the service down-time.

Solution:

  1. Develop JMX MBeans which check server disk usage (JMX StandardMBean)
  2. Create alarm and notify when the disk usage is over 95% (JMX MBean Monitoring - UsedDiskSpace)
  3. Create "User Defined Metric Action" on WLSDM which is targeted to "delete or move application logs"
  4. Couple "FreeDiskSpace or UsedDiskSpace WebLogic JMX MBean" and "ClearLogs Metric Action"
  5. When FreeDiskSpace percentage lower than given threshold, "ClearLogs" metric action would be triggered by WLSDM and all application logs would be deleted automatically. That means "NO Down-time" with advanced WebLogic monitoring and automation.
Video Tutorial & Custom JMX Source Code (Java):

An 8 Minutes tutorial is prepared for this problem and for any other similar problems.

The tutorial is short and targeted to overcome SERVER disk problem.

  1. Watch Tutorial:
  2. Download Source Code (Sample Project and WAR): ZIP file contains "WAR File" and "JAVA Eclipse Project"

    Download: WLSDMMonitoringSamples.zip
How to develop your own JMX MBean and JMX Metric Action? (SUMMARY)

For advanced WebLogic Administrators, DevOps or Developers above video tutorial and sample project quite enough to setup an advanced WebLogic monitoring and automation infrastructure for his/her organization or mission critical WebLogic domains that they manage.

  1. Login to WebLogic Console
  2. Go to WLSDM Console Metric Browser Page

    [caption id="attachment_929" align="alignnone" width="255"]WebLogic Monitoring Dashboard: WLSDM Console WebLogic Monitoring Dashboard: WLSDM Console[/caption]


  3. Play with WLSDM Metric Browser. It displays all WebLogic JMX MBeans dynamically.
    "WLSDM Console > Configuration > Metric Browser"

    [caption id="attachment_931" align="alignnone" width="288"]WLSDM Metric Browsers: JMX MBean Metric Browser WLSDM Metric Browser: JMX MBean Metric Browser[/caption]


  4. a) Develop JMX StandartMBean
    b) Register to WebLogic MBean Server

    We have prepared a sample JMX Bean which check "Free Disk Space" and "Used Disk Space" on server.
    Check Sample Java Source and develop your own JMX MBeans then deploy to your application or WebLogic domain.

    WLSDMMonitoringSamples.zip:

    Contains Eclipse Project and Deploy-able WAR File (Supports WebLogic 11g and 12c)

    Download: WLSDMMonitoringSamples.zip

    MBean Interface: DiskSpaceWatcherMBean.java

    package com.wlsdm.samples;
    

    public interface DiskSpaceWatcherMBean {

    public double getFreeDiskSpace();
    public void setFreeDiskSpace(double fds);
    
    public double getUsedDiskSpace();
    public void setUsedDiskSpace(double uds);
    

    }

    MBean Class: DiskSpaceWatcher.java

    package com.wlsdm.samples;
    

    import java.io.File;

    import javax.management.StandardMBean;
    import javax.swing.filechooser.FileSystemView;

    public class DiskSpaceWatcher extends StandardMBean implements DiskSpaceWatcherMBean {

    public double FreeDiskSpace = new Double(100.00);
    public double UsedDiskSpace = new Double(0.00);
    
    
    protected DiskSpaceWatcher() {
        super(DiskSpaceWatcherMBean.class, false);
        
        Thread t = new Thread(new CollectData());
        t.start();      
    }
    
    @Override
    public double getFreeDiskSpace() {
        return FreeDiskSpace;
    }
    
    @Override
    public void setFreeDiskSpace(double fds) {
        FreeDiskSpace = fds;        
    }
    
    @Override
    public double getUsedDiskSpace() {
        return UsedDiskSpace;
    }
    
    @Override
    public void setUsedDiskSpace(double uds) {
        UsedDiskSpace = uds;        
    }
    
    private class CollectData implements Runnable {
    
        @Override
        public void run() {
    
            while (true) {
                
                File f = FileSystemView.getFileSystemView().getHomeDirectory();
                
                double free = ((double)f.getUsableSpace() / f.getTotalSpace()) * 100.00d;
                
                setFreeDiskSpace(free);
                setUsedDiskSpace(100.00 - free);
                
                try {
                    Thread.sleep(60 * 1000);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }
    }
    

    }

    Listener Class: DiskSpaceWatcherListener.java

    package com.wlsdm.samples;
    

    import java.io.IOException;
    import java.util.Hashtable;

    import javax.management.MBeanServer;
    import javax.management.ObjectName;
    import javax.naming.Context;
    import javax.naming.InitialContext;
    import javax.servlet.GenericServlet;
    import javax.servlet.ServletException;
    import javax.servlet.ServletRequest;
    import javax.servlet.ServletResponse;

    public class DiskSpaceWatcherListener extends GenericServlet {

    private static final long serialVersionUID = 1L;
    
    public void destroy() {
        try {
    
            Hashtable<String, String> env = new Hashtable<String, String>();
            env.put(Context.SECURITY_PRINCIPAL, "weblogic");
            env.put(Context.SECURITY_CREDENTIALS, "welcome1");
            InitialContext ctx = new InitialContext(env);
    
            MBeanServer mbs = (MBeanServer) ctx.lookup("java:comp/env/jmx/runtime");
    
            ObjectName oname = new ObjectName("com.wlsdm.samples:type=DiskSpaceWatcherMBean,name=DiskSpaceWatcher");
    
            if (mbs.isRegistered(oname)) {
                mbs.unregisterMBean(oname);
            }
    
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
    
    public void init() {
        try {
    
            Hashtable<String, String> env = new Hashtable<String, String>();
            env.put(Context.SECURITY_PRINCIPAL, "weblogic");
            env.put(Context.SECURITY_CREDENTIALS, "welcome1");
            InitialContext ctx = new InitialContext(env);
            MBeanServer mbs = (MBeanServer) ctx.lookup("java:comp/env/jmx/runtime");
            
            DiskSpaceWatcherMBean mbean = new DiskSpaceWatcher();
            ObjectName oname = new ObjectName("com.wlsdm.samples:type=DiskSpaceWatcherMBean,name=DiskSpaceWatcher");
            mbs.registerMBean(mbean, oname);
    
        } catch (Exception e) {
            e.printStackTrace();
        } 
    }
    
    @Override
    public void service(ServletRequest req, ServletResponse res) throws ServletException, IOException {
    
    }
    

    }

  5. Deploy WebApplication to WebLogic Note: WLSDMMonitoringSamples.war can be used for completing this tutorial.

  6. Check new deployed JMX Custom MBeans on WLSDM Metric Browser page for DiskSpaceWatcher MBeans. Anymore FreeDiskSpace and UsedDiskSpace JMX MBeans are visible on WLSDM Metric Browser (Without WebLogic Server restart)

    [caption id="attachment_945" align="alignnone" width="452"]Custom WebLogic JMX MBeans Custom WebLogic JMX MBeans[/caption]


  7. Create new Dashboard and name it DiskSpaceWatch.

    [caption id="attachment_933" align="alignnone" width="302"]WLSDM Dashboard: Create New WLSDM Dashboard: Create New[/caption]


  8. Add FreeDiskSpace and UsedDiskSpace JMX Metrics to DiskSpaceWatch Dashboard and save all.

    [caption id="attachment_946" align="alignnone" width="679"]Adding Custom JMX Metrics to WLSDM Dashboard Adding Custom JMX Metrics to WLSDM Dashboard[/caption]


  9. Anymore Custom JMX MBeans that we developed and its values can be visualize on WLSDM Console Smart Dashboards. You would see two charts on DiskSpaceWatch dashboard as below.

    [caption id="attachment_953" align="alignnone" width="409"]WLSDM JMX Metric Dashboard: Grid View WLSDM JMX Metric Dashboard: Grid View[/caption]

    [caption id="attachment_947" align="alignnone" width="513"]Dynamic JMX MBean Charts-1: Free Disk Space Dynamic JMX MBean Charts-1: Free Disk Space[/caption]

    [caption id="attachment_948" align="alignnone" width="508"]Dynamic JMX MBean Charts-2: Used Disk Space Dynamic JMX MBean Charts-2: Used Disk Space[/caption]


  10. Set threshold values for FreeDiskSpace and UsedDiskSpace MBeans and see what will happen? Test it by setting low thresholds. For instance, set threshold value for UsedDiskSpace to "60" and get below notifications.

    JMX MBean monitoring alarms would be generated by WLSDM as page and email Notification.

    [caption id="attachment_975" align="alignnone" width="394"]WLSDM: JMX Metric Page Notification WLSDM: JMX Metric Page Notification[/caption]

    [caption id="attachment_949" align="alignnone" width="398"]WLSDM: JMX Metric Email Alarm WLSDM: JMX Metric Email Alarm[/caption]


  11. Now Define Custom Metric Actions for FreeDiskSpace:
    Custom metric actions can be defined on WLSDM. Now going to create a custom action for FreeDiskSpace metric.


  12. Go to "Configuration > User Defined Metric Actions" Page.

    [caption id="attachment_934" align="alignnone" width="286"]WLSDM Actions: User Defined Metric Actions WLSDM Actions: User Defined Metric Actions[/caption]


  13. Create new "User Defined Metric Action" as below.

    [caption id="attachment_935" align="alignnone" width="464"]WLSDM: User Defined JMX Metric Actions WLSDM: User Defined JMX Metric Actions[/caption]

    Note: "Executable Target Files" can be .SH or .BAT file. System Paremeters such as $METRIC_NAME, $METRIC_VALUE can be passed to "Executable Target File" as argument by WLSDM

    Content of "clearlogs.bat" file:

    cd C:\wlsdm\logs
    del /F /Q *.*
  14. Anymore "ClearLogs" Cusom Action can be listed on Actions Window. Select "ClearLogs".

    [caption id="attachment_951" align="alignnone" width="570"]WLSDM Metric Actions: Select ClearLogs WLSDM Metric Actions: Select ClearLogs[/caption]


  15. Reset UsedDiskSpace or FreeDiskSpace Metric threshold value and get METRIC CLEAR alarm from WLSDM. (For testing ClearLogs action)


  16. Reassign threshold value for FreeDiskSpace metric and get METRIC WARNING alarm from WLSDM. (ie. set lower than 30)


  17. Oops! FreeDiskSpace alarm generated! And ClearLogs action has been executed automatically. All logs under C:\wlsdm\logs has been deleted.

    [caption id="attachment_955" align="alignnone" width="756"]WLSDM JMX HTML Email Alert: User Defined Metric Action WLSDM JMX HTML Email Alert: User Defined Metric Action[/caption]

  18. End of tutorial. Monitoring is meaningful and powerful with automation.

Advanced WebLogic Monitoring: Best & Ultimate Practice


Think about your chronic runtime problems and develop your own JMX MBeans then overcome all manual processes. This method is the best and ultimate monitoring infrastructure for the mission critical applications. High level DevOps engineers, developers or architects should deliver their WebLogic IT projects with monitoring infrastructure. Should think about potential LIVE/PRODUCTION issues and should implement before going live. It is only possible with WebLogic and Metric Actions.