Hi Jeba,

First of all your WLSDM version is not the latest; please upgrade to v3.4.1 (it will took about 5 min). Please do a fresh installation. This domain was open for all the metrics about “STORE DATA” feature. That’s why you faced this issue.

Let us analyze more. Is this thread is in ManagedServer or AdminServer? We need below information to help you.

Could please at least take 3 thread dump for AdminServer and ManagedServer then send us to us? (thread dump: kill -3, or from WLSDM console) Could you please zip all WLSDM log files and send to us? (Available at WLSDM\logs folder) Could you please runt this command and share the output with us? cd $DOMAIN_HOME{WLSDM_RUNTIME_FOLDER}
du -sh * Could you please share the WLSDM version?

We are suspecting about data archive. As a workaround please DISABLE all your (make OFF) data archive settings. Go to “System Settings > system.archive (tab)” page and OFF all archive settings if they are currently ON. Then restart only AdminServer then observe the new states about high CPU consumption.

Kind Regards…

@whching said:

Hi @wlsdmsupport

I am just start looking at WLSDM, what is the difference of having one agent per cluster against one agent per managed server?

BTW, where I could find the resource overhead when WLSDM is used on managed servers? say, how much more heap memory it will use and how about any extra I/O it would introduced?
Thanks

Regards,
Rock

Hi,

If you have two member cluster and these members are on different machine then install two agent. If you have 4(four) member cluster and these members on two machine (two managed server per machine). Then install two agent (same with previous). Much agent, much traffic. Same behavior, same machine… Then go for one agent.

More information is available on README: http://www.wlsdm.com/docs/README#IV.Installation

WLSDM runs on AdminServer and you should not install any application on WebLogic admin console. So, it’s run-time console, should not affect your domain. WLSDM’s AdminServer overhead is negligible; one of the least overhead in the market; because we are not injecting any code to run-time classes. Maximum overhead for the managed server (considering backend monitoring is open) is about 3%-4%.

You had better, test on your current domain; because your domain is extra ordinary and you will see that WLSDM would be one of the best solution for this domain and its infrastructure.

Regards…

@jeba

Hi Jeba,

Also, check “store data” for back-end monitorings. It may generate too large data sets. especially if the “WebLogic Diagnostic Volume” set high for the related managed server.

Capture4.PNG

I enabled the store option for all of our datasources. So, in total there were around 75. Would this be an issue?

75 data sources cost you avg ~30 Mb per day (it depends on your metric instance count). If you have enough disk space it is not an issue.

How can I control the retention period for these metrics collections?

You can change the archive date range from the “Configuration> WLSDM System Settings” page. Archive module purges your data beyond the given date range. But do not reduce the date range instantly. You have too much data on your database and it may cause too much memory consumption. Prefer gradual reduction (like 3 or 5 days).

“system.archive.metric.data.range”

Capture2.PNG

Regards.

@jeba

Hi Jeba,

WLSDM does not have the feature for your requirement which is giving a threshold for the specific IP/hostname.

But it could be a nice feature, we will evaluate this feature for the next releases.

Regards.

Hi Tom,

You can filter back-end dashboard threshold value in “Configuration > WLSDM System Settings > system.monitoring > system.monitoring.diagnostic.backend.event.filter.threshold” property. Click edit button and give threshold value as 1000 (ms).

Regards.

1.png

@ozhana

Hi,

Are you sure all access and diagnostic logs are backuped?

Which WLSDM version are you using?

Kindly, please send reply at [email protected] email address.

Regards.

Hi Vivek;
Actually 11g and 12c offers same health dashboard. Something is wrong about your 12c WebLogic environment.
Could you please zip $domain_home/WLSDM/logs folder and attach screen capture to support (at) wlsdm.com email.
Regards…

Hi,

Today we have worked on SOA-11g environment. JVM Resources Page and GarbageCollection Page problem has been resolved.

Please do the following when you face such a problem in another domain.

Kind Regards…

Backup startWeblogic.sh file of the domain

Add the “-Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder” parameter to JAVA_OPTIONS in the file

JXM.Values.JVM.Resources.Dashborad.jpg

@asutosh said:

i have installed the WLSDM and set the java option as well. WLSDM console and agent loaded successfully.

But i am getting error when saving all monitoring setting

####<13-Mar-2018 13:50:16 GMT><Unparseable date: “-- 23:59:59”

java.text.ParseException: Unparseable date: “-- 23:59:59”
at java.text.DateFormat.parse(DateFormat.java:366)
at com.wlsdm.J.F.H(Unknown Source)
at com.wlsdm.J.D.hasLicense(Unknown Source)
at com.wlsdm.H.a$_A.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)

Please suggest how can we fix this

Hi,
wlsdm.log is available in $DOMAIN_HOME/WLSDM/logs folder. Just zip the whole folder and upload it. We will help you by investigating the real problem.
Regards…

Hi,

“Thread Dump Analyzer” requires to full thread dump for analyzing.

Normally WLSDM gets a full thread dump. But if there is an exception, like in your case (connection timeout); by default, WLSDM tries to get it using WebLogic API. And sometimes WebLogic does not provide a full thread dump.

That’s why you didn’t get a full thread dump even your managed WebLogic server has a WLSDM agent.

Thanks.

@hmurali said:

@wlsdmsupport Thanks for the update.

Do we also get alerts when the server health goes to warning/overloaded state and Server state goes to ADMIN/UNKNOWN status.

Kindly advise.

Thanks
Hari

Hi Hari,
Additionally please check this tutorial to find out and learn more: https://www.youtube.com/watch?v=PeKJusLuarI
Regards…

Hi @jorfergo,

The current WLSDM release doesn’t support this feature.

Please add your exact requirement on this topic: http://community.wlsdm.com/topic/18/wlsdm-new-features-requests

We will implement in the next release if our analyze team approve.

Regards.

Hi,

We are not supporting multiple log files (types) for a query/monitoring.

So currently you should repeat the monitoring queries.

But, you can assign the query to multiple servers for the “ServerLog”, “stdOut”, “stdErr” and “Diagnostic Log” types.

Kind Regards.

@catgovind said:

Thank you so much. It is working perfectly.

We will add additional NOTE to WLSDM readme in the next release. Thank you for reporting it.

Regards…

Hi,

WLSDM doesn’t support the multiple log files for log monitoring.

The current algorithm based on, “one log file, one search string” rule.

So, currently, there is no easy way to do this.

But we are going to report your request for the next WLSDM version.

Hi,

ProcessCPULoad metric values are sets between (zero) 0 and (one) 1 as “java.lang.Double” by the JVM.

So “1” means 100%.

To get an alert (for testing purpose), you should set the threshold value to less than “1”. eg. “0.01” (1%)
3.png

Hi,

We are not supporting multiple log files (types) for a query/monitoring.

So currently you should repeat the monitoring queries.

But, you can assign the query to multiple servers for the “ServerLog”, “stdOut”, “stdErr” and “Diagnostic Log” types.

Kind Regards…

Hi,

“[email protected]” is default sender (from) address for WLSDM and your mail client may not accept it.

To change sender address:

• Go to “Configuration > WLSDM System Settings” page
• Select the “mail.account” tab
• Edit the “mail.account.smtp.username” property (e.g. [email protected])

settings.jpg

Test changes :
On “Configuration > WLSDM System Settings” page; open “Page Operations” menu and click “Test Email Settings” button

poperations.jpg

testmail.jpg

Hi,

The right side of equal char which is attributeValue must be only numeric values.Probably your script contains string and you need to trim string chars.

Also the first “/” should be replaced, too. Here is sample blog for your reference. One of our community user; used below script for doing it.

http://mathiassalgado.blogspot.com/2017/06/add-metric-options-to-devops-mbean-with-wlsdm-weblogic-console.html

script→ df -H | grep -vE ‘^Filesystem|tmpfs|media’ | awk ‘{ print $6"="$5}’ | sed ‘s/%//’ | sed ‘2s///ROOT/’ | sed ‘s///_/g’

I recommend filter the result set; it does not make sense to use every result for “df –h” . i.e. you /etc, /dev, proc, system… Only consider the mount points which relevant with WebLogic JVM processes.

Regards…

Hi,

WLDF Query Language is not supporting in diagnostic log monitoring because diagnostic log monitoring is a special solution.

Firts of all, is strings connecting with “OR” or “AND” ?

If you are using“OR”, you must define two different search for two different defination.

If you are using “AND”, you can replace search string with exact defination.

You can search string as below.

Diagnostic log row: [ACTIVE].ExecuteThread: ‘2’ for queue: ‘weblogic.kernel.Default (self-tuning)’] [userId: ][APP: Service Bus Kernel] [partition-name: DOMAIN] [tenant-name: GLOBAL] [oracle.soa.tracking.InstanceId: 14216584] [oracle.soa.tracking.SCAEntityId: 10537] [oracle.soa.tracking.FaultId: 2252239] [FlowId: 0000Lu^vEciEoI04jz_Aio1PjgB80001jW] Outbound.B2BI/pipeline/vendor**/Unhandled error caught by system-level error handler: com.bea.wli.sb.pipeline.PipelineException: Service Unavailable**

Search string: “/Unhandled error caught by system-level error handler: com.bea.wli.sb.pipeline.PipelineException: Service Unavailable”

Hi,

Yo need to edit script according to your environment.

· http://community.wlsdm.com/topic/38/monitor-webservice-response-times-with-curl-alldetails-on-weblogic

Add soap-request.xml to server and replace à -d@/Users/admineer/u01/scripts.imac/curl.ws/soap-request.xml

Replace /Users/admineer/u01/scripts.imac/curl.ws/response.xml

Replace (Also add to server) “@/Users/admineer/u01/scripts.imac/curl.ws/curl-format.txt”

For testing below script the server must access the http://www.webservicex.net/globalweather.asmx URL. I am highly recommending test with an “inhouse” business Webservice. You need to reconfigure this according to your service.

Regards…

@Ramanpreet_Singh

Yes, everthing was just fine. If you want to remove buffer on WebLogic decrease the buffer size to “0” on “WebLogic console > $Server_Name > Logging > HTTP Logs > Advanced > Buffer Size” section.

Regards…

@Ramanpreet_Singh

Hi Ramanpreet,
We are happy to solve your issues. Thank you for the affirmative report.
Always use $IP:$PORT/console/WLSDM direct URL for accessing WebLogic WLSDM console.

@Ramanpreet_Singh Hi,

Export/Import feature is used for WLSDM settings. Nothing related about WebLogic configuration.

i.e. If you configured SMTP settings on WLSDM; these values will come from export file or DevOps Mbeans and any other WLSDM configuration you have done.

Kind Regards…

Hi @Ramanpreet_Singh,

Currently, you cannot. But we will add this feature to next release.

@jeba said:

Hi,

On the Back End Reporting, by default EJB and JDBC events are captured and alerted. This is useful. But there are options like File Read, File Write and Web Service calls on the Back End Reports. We enabled these options for the Managed Server but none of these events are captured or displayed in those reports. But while restarting the Managed Server, if we include this JVM option,
“-XX:FlightRecorderOptions=defaultrecording=true”
Then I am able see all those File Read and File Write events are displayed and alerted. After removing the JFR default recording, then no events are captured. Is there any option or config we need? The downside of keeping the defaultrecording ON is that the Managed Server generates the JFR recording and the disk space grows exponentially which we don’t want.

Thanks,
Jeba.

Hi Jeba,

We analyzed your WebLogic monitoring request again and here our solution offer.

Please remove “-XX:FlightRecorderOptions=defaultrecording=true”, you do not need this option while using WLSDM. If you still want to see File/Socket events you should increase your WLDF Diagnostic volume on WebLogic.
Go to “Servers > $ManageServerName > General (tab)” page and set “Diagnostic Volume” high. Then you are able to see file read/write events.

Note: This option means additional overhead on your managedservers; just activate only one managedserver which file read/write monitoring is important.

image.png

Keep your file and socket read/write definitions on WLSDM.

Restart your managed server (it is needed after JVM parameter removing).

This is the best setup for monitoring your file/socket events. By this configuration you will not face any disk problem. Because; WLSDM will take care of JFR files.

Regards…

Thank you so much. I will give it a try and let you know whether this works in our scenario or not. Thanks.

@Ramanpreet_Singh yes you can customize the subject.

Problem is resolved by adding username and removing password. Then the WLSDM email SMTP configurations has been setup successfully for WebLogic.

Log Monitoring cycle default value is 120 seconds. You can configure it from “Configuration > WLSDM System Settings” page’s “system.monitoring” tab, property as below. It effects back-end, loginspector and response times frequency. No problem until 60 seconds but we suggest 120 seconds for high transaction traffic.

LOG.png