On the Back End Reporting, by default EJB and JDBC events are captured and alerted. This is useful. But there are options like File Read, File Write and Web Service calls on the Back End Reports. We enabled these options for the Managed Server but none of these events are captured or displayed in those reports. But while restarting the Managed Server, if we include this JVM option,
Then I am able see all those File Read and File Write events are displayed and alerted. After removing the JFR default recording, then no events are captured. Is there any option or config we need? The downside of keeping the defaultrecording ON is that the Managed Server generates the JFR recording and the disk space grows exponentially which we don’t want.
We analyzed your WebLogic monitoring request again and here our solution offer.
Please remove “-XX:FlightRecorderOptions=defaultrecording=true”, you do not need this option while using WLSDM.
If you still want to see File/Socket events you should increase your WLDF Diagnostic volume on WebLogic.
Go to “Servers > $ManageServerName > General (tab)” page and set “Diagnostic Volume” high. Then you are able to see file read/write events.
Note: This option means additional overhead on your managedservers; just activate only one managedserver which file read/write monitoring is important.
Log Monitoring cycle default value is 120 seconds. You can configure it from “Configuration > WLSDM System Settings” page’s “system.monitoring” tab, property as below. It effects back-end, loginspector and response times frequency. No problem until 60 seconds but we suggest 120 seconds for high transaction traffic.
We analyzed the problem with futher investigation and logs; that realized it is because of getting below error on WebLogic:
Request-URI Too Long
The requested URL's length exceeds the capacity limit for this server.
Accesing to WebLogic WLSDM console through a webserver (OHS, Apache… etc.) can cause above error while at the last step on WLSDM wizard screen when clicking on save button.
Accessing WebLogic WLSDM wizard console directly (bypassing webserver redirection) will solve the problem. We recommend not to use Webserver proxy while setting up WLSDM wizard at first. After the wizard completion you can use webserver proxy freely.
Webservers have limitations about posting data bacause of security issues. The limit can be increased on the configuration files.
We are supporting upgrade from 2.5.2 to 3.1.1.
Consider upgrade steps then your collected data will be kept and you won’t lost any data. But please take a back-up of WLSDM runtime folder which is located under $DOMAIN_HOME/WLSDM while your admin server is shut down.
We have Thread Dump action. Just activate ThreadDump action for Hoggers or Stuck metrics and then analyze thread dumps by using “Thread Dump Analyser”. If your Administrator good enough you can write your own scripts for detecting deadlocks on JVM threads.
Normally, WLSDM stores 24 hours data storage for MBeans and backend usage by default. But, all chart values can be stored on WLSDM database by turning it on. You can turn on “Store Data” for every data while setting up WLSDM wizard. After the installation, if you have not turned on “store date”, you need to open them by one by. Good news, you can sort them all on "
• > JMX MBean Metric Browser & Settings” page by sorting “Store Data” column. Just follow notifications on relevant notifications page. They’re stored for three months. Remember we have PURGE and ARCHIVE modules. You do not need to operation your monitoring console. It does everything for you.
Thank you for your interest. We trust our product very much and we thought every detail for WLSDM.
Here our comments;
First of all, your domain is really big and massive. But, we it is for sure we can handle it.
1 - For my experience what make the difference is collecting data… In order to monitoring historic information anda data (this help me in troubleshooting)
Yes, especially this size of domain is very hard to monitor. If you have any unused server, datasource, JMS definion please delete all, after that fresh resinstall WLSDM and do not track unused apps, servers, deployments… etc.
Historical data would be very useful for your problem troubleshooting. Especially, check notification pages.
2 - About deployments table your dashboard use to list applciation status in evey servers of my cluster and it is a little bit redundat
We have asynchronous backend health feature on health dashboard. Could you please enable it and let us know the result? (Go to Health Dashboard page, > Page Operations Menu> Switch the below button to ON)
By the way, we have JMX/t3 timeout on system.WLSDM tab at System Settings page. If your domain is not responsive has its own problem, we are cutting the request because of keeping WebLogic domain healthy. It’s default value is about 5sec. That’s why you can see the health status conflicts.
We support shell and batch (.sh and .bat/.cmd) OS script execution for custom metric actions. You can trigger any python/jython/wlst script by using WLSDM WebLogic console. We do not support script scheduling because of DevOps support. Scripts triggering is fully parametrical and can be assign to any specific MBean threshold.
WLSDM console runs under WebLogic /console context path. And we’re using default WebLogic security realm. The configuration and important pages on WLSDM console is only accessible for Administrators global role. So, a user which has Monitor global role, only able to see only monitoring pages on WLSDM console. In this case, What kind of requirement do you have for the custom access for accessing WLSDM console.
When we did the webex session we had seen that you are able to login WLSDM console. This situation is something about Webserver. We need to sort out this plain-text problem. Which web server are you using for the redirection for the WebLogic /console redirection? Your web server serving WLSDM console somehow as plain-text instead html. It could be something about mime-type. Could you please check this out and let me see the result? http://stackoverflow.com/questions/9142117/pages-are-displaying-plain-text-instead-of-html
WebLogic does not support these file for WLDF querying. Because, there are redirection and broadcast features on WebLogic.
For the .out file Go to “Servers > $SERVER_NAME > Logging (Tab) > General (Subtab)” then enable “Redirect stdout logging enabled” and “Redirect stderr logging enabled”. After the restart you are able to monitor these file by using WLSDM. Write your own WLDF query. If you’re having to define WLDF query please share your log record with us. Then we’ll support you for the relevant WLDF query.
For diagnostic log files WebLogic has broadcast feature for the all log file to broadcast logs to the DOMAIN log file. Go to “Servers > $SERVER_NAME > Logging (Tab) > General (Subtab)” and check “Domain log broadcaster” section and configure your broadcast settings. Decrease buffer size for getting quick results. After that by monitoring Domain.Log you’re able to monitor all kind of files by adding Domain.Log monitoring on WLSDM LogInspector. For the new definition; go to “WLSDM Configuration > Monitoring & Diagnostics > Log Inspector” page and add new from Page Operations menu on WLSDM console.
Hope, this will help you satisfy your requirement.
Could you please, check this solution offer and let us know the result.
The error occurs when serverRuntime is requested …
At javax.management.remote.rmi.RMIConnector $ RemoteMBeanServerConnection.invoke (RMIConnector.java:1020)
At weblogic.management.jmx.MBeanServerInvocationHandler.doInvoke (MBeanServerInvocationHandler.java:562)
At weblogic.management.jmx.MBeanServerInvocationHandler.invoke (MBeanServerInvocationHandler.java:391)
At com.sun.proxy. $ Proxy273.lookupServerRuntime (Unknown Source)
At com.wlsdm.L.a.getServerRuntimeMBean (MBSConnection.java:475)
At com.wlsdm.frontend.provider.GA.getServerHealthState (PerformanceHealthDatas.java:123)
At com.wlsdm.J.O. £ (Data.java:6914)
The WLSDM serverRuntime object is requested via the weblogic console API, and the admin console is also calling managedServers for RMI.
→ Is there any problem on the WebLogic admin console > Servers page? (Just want to diagnose problem)
→ Is there any problem on the WebLogic console in general? Sometimes, network issues can affect WebLogic console and WLSDM console… Please check ConnectionDelayTime on Datasources dashbaord and let us know any sudden increment for this MBean…
@sravanreddyg It’s standard feature of WLSDM. You should configure these settings while completing wizard. But it’s not problem. You still can update on Back-end tab on configuration pages. Just click on edit button for the JDBC entry; enable threshold and “store data”. That’s all. You can follow metric threshold documentation as a reference. Here is the doc URL: http://www.wlsdm.com/docs/help/#3.CompletingWLSDMWizard
Sometimes we get an error like following in the wlsdm logs. What could be the reason?
####<Dec 30, 2016 8:24:57 AM EET><java.lang.IllegalArgumentException: com.sun.org.apache.bcel.internal.generic.ClassGenException: Illegal opcode detected. java.lang.RuntimeException: java.lang.IllegalArgumentException: com.sun.org.apache.bcel.internal.generic.ClassGenException: Illegal opcode detected. at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237) at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223) at com.wlsdm.B.D.F_1036_WLStub.getFlightRecording(Unknown Source) at com.wlsdm.K.b.executeRemote(BackendDump.java:1424) at com.wlsdm.K.b.startBatchCollect(BackendDump.java:175) at com.wlsdm.K.c$_A.run(BackendDataCollector.java:43) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: com.sun.org.apache.bcel.internal.generic.ClassGenException: Illegal opcode detected. at com.jrockit.mc.flightrecorder.FlightRecordingLoader.loadFile(FlightRecordingLoader.java:194) at com.wlsdm.B.D.F.getFlightRecording(OperationImpl.java:453) at com.wlsdm.B.D.F_WLSkel.invoke(Unknown Source) at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667) at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146) at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518) at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221) Caused by: java.lang.IllegalArgumentException: com.sun.org.apache.bcel.internal.generic.ClassGenException: Illegal opcode detected. at com.jrockit.mc.flightrecorder.provider.bcel.BCELUtilities.createField(BCELUtilities.java:57) at com.jrockit.mc.flightrecorder.provider.DefaultFactories$BCELFieldFactory.createValueField(DefaultFactories.java:71) at com.jrockit.mc.flightrecorder.provider.DefaultFactories$BCELFieldFactory.createValueField(DefaultFactories.java:1) at com.jrockit.mc.flightrecorder.internal.parser.binary.EventParserBuilder.initEventType(EventParserBuilder.java:64) at com.jrockit.mc.flightrecorder.internal.parser.binary.EventParserBuilder.createEventParser(EventParserBuilder.java:49) at com.jrockit.mc.flightrecorder.internal.parser.binary.EventParserManager.(EventParserManager.java:59) at com.jrockit.mc.flightrecorder.internal.parser.binary.ChunkLoader.run(ChunkLoader.java:22) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: com.sun.org.apache.bcel.internal.generic.ClassGenException: Illegal opcode detected. at com.sun.org.apache.bcel.internal.generic.Instruction.readInstruction(Instruction.java:197) at com.sun.org.apache.bcel.internal.generic.InstructionList.(InstructionList.java:193) at com.sun.org.apache.bcel.internal.generic.MethodGen.(MethodGen.java:179) at com.jrockit.mc.flightrecorder.provider.bcel.EventClassLoader.createGetValueMethod(EventClassLoader.java:118) at com.jrockit.mc.flightrecorder.provider.bcel.EventClassLoader.createFieldJavaClass(EventClassLoader.java:110) at com.jrockit.mc.flightrecorder.provider.bcel.EventClassLoader.createBCELClass(EventClassLoader.java:101) at com.jrockit.mc.flightrecorder.provider.bcel.EventClassLoader.loadClass(EventClassLoader.java:67) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at com.jrockit.mc.flightrecorder.provider.bcel.BCELUtilities.createField(BCELUtilities.java:50) … 8 more >
This exception throws from JFR and Apache BCEL. It can be WebLogic/JRockit/Java defect while using Apache BCEL. By the way, do you have any additional agent on this managed server except wlsdm_agent?
We will reproduce this case and get back to you as soon as possible.
How can I remove shuttdown servers in Smart Dashboard. Would you help me pls. Thanks.
On WLSDM Health dashboard go to Servers grid. There are notification icon(bell) on every row. Click on it and select disable all. All the shutdown servers and its sub resources will be bypassed without monitoring.
I have successfully configured WLSDM into my domain, but due to some reason my system IP got changed, I have updated my new IP in all my weblogic domain files and is working as expected, but when I am launching WLSDM from console its redirecting me to my domain login screen everytime. I tried undeploying and deploying WLSDM as well but no luck, is their any file where I need to update my IP for WLSDM or were I am missing anything please help.
WLSDM works native just as WebLogic and there’s nothing to configure about IP addresses of WebLogic unless you did not setup boot.properties. We’ve tested your case and everything worked fine as expected.
If you’re still facing the same problem, could you please upload wlsdm.log which is located under “$WEBLOGIC_DOMAIN_HOME\WLSDM\logs” path.
Waiting your update about last status.
Thanks in advance and keep follow us 🙂