OEM 13.4 - OMS and Agent Directory Structure - Important Diagnostic Logs and Trace Files Location on OEM 13c Release 4 - Part 4
Hello All,
In this video, demonstrated and discussed all about Oracle Enterprise Manager Cloud Control 13c Release 4 OMS and AGENT directory structure and diagnostic log and trace files location explored. Common OEM 13c agent issues and fixes with Log directory structure: Oracle Enterprise Manager Cloud Control 13c Release 4 - Directory Structure and Locations of Important Trace and Log Files:
Issues: You may face following issues in OEM 13c like : a) Agent is unable to communicate with the OMS. (REASON = Agent is Unreachable (REASON : Agent to OMS Communication is broken) b) Huge volume Alerts So, In this case, first try to connect the OMS like below: Please try to chekc through ssh or telnet for the connectivity [oracle@srlabgg1 ~]$ ssh srlabgg1.localdomain 3872 (or) telnet 192.168.58.201 3872 >> for checking host connectivity with agent port Web console: https://srlabgg1:9090/ or https://192.168.1.11:9090/ Last login: Sat Jan 23 03:05:35 2021 from 192.168.58.201 If there is no connectivity issue, then you can follow below steps and monitor the agent : Step 1: Increase the agent memory 1. Stop the agent. [oracle@srlabgg1 bin]$ pwd /u04/app/oracle/agent/agent134/agent_inst/bin [oracle@srlabgg1 bin]$ emctl stop agent 2. Take a backup of emd.properties and edit it [oracle@srlabgg1 config]$ pwd /u04/app/oracle/agent/agent134/agent_inst/sysman/config [oracle@srlabgg1 config]$ ls -lrt emd.properties -rw-r-----. 1 oracle oinstall 9452 Jan 23 02:56 emd.properties [oracle@srlabgg1 config]$ cp emd.properties emd.properties_bak [oracle@srlabgg1 config]$ ls -lrt emd* -rw-r-----. 1 oracle oinstall 9452 Jan 23 02:56 emd.properties -rw-r-----. 1 oracle oinstall 9452 Jan 23 03:14 emd.properties_bak From: agentJavaDefines=-Xmx270M -XX:MaxPermSize=96M To: agentJavaDefines=-Xmx512M -XX:MaxPermSize=96M In my case it is, [oracle@srlabgg1 config]$ cat emd.properties | grep agentJavaDefines agentJavaDefines=-Xmx211M -XX:MaxMetaspaceSize=224M 3.Backup the file s_jvm_options.opt and add following parameter at EOF [oracle@srlabgg1 config]$ pwd /u04/app/oracle/agent/agent134/agent_inst/sysman/config [oracle@srlabgg1 config]$ cp s_jvm_options.opt s_jvm_options.opt_bak [oracle@srlabgg1 config]$ ls -lrt s_jvm_options.opt* -rw-------. 1 oracle oinstall 607 Jan 14 10:30 s_jvm_options.opt -rw-------. 1 oracle oinstall 607 Jan 23 03:18 s_jvm_options.opt_bak -XX:+UseMembar 4.Start the agent [oracle@srlabgg1 bin]$ emctl start agent Then verify the log "emagent.nohup" file for rest of the agents and follow the above steps to avoid the same issue. [oracle@srlabgg1 log]$ ls -lrt *nohup* -rw-r-----. 1 oracle oinstall 4096 Jan 23 02:55 emagent.nohup.lr -rw-r-----. 1 oracle oinstall 266514 Jan 23 03:02 emagent.nohup [oracle@srlabgg1 log]$ tail -f emagent.nohup 5. Verify the status of the agent [oracle@srlabgg1 bin]$ emctl status agent [oracle@srlabgg1 bin]$ emctl pingOMS Oracle Enterprise Manager Cloud Control 13c Release 4 Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD pingOMS completed successfully Note: To get a report on this issue, you can connect to the OMS and run the below query to find out the history: Connected to the Repository database as the user SYSMAN, execute the following query: SQL> SELECT b.target_name agent , c.target_name buddy FROM em_agent_buddy_map a, em_targets b, em_targets c WHERE a.agent_target_guid = b.target_guid AND a.buddy_target_guid = c.target_guid; [oracle@srlabgg1 ~]$ sqlplus sysman@oem134plug SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jan 23 03:24:56 2021 Version 19.8.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Enter password: Last Successful login time: Sat Jan 23 2021 03:24:19 -06:00 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.8.0.0.0 SQL> show user USER is "SYSMAN" SQL> SQL> SQL> SELECT b.target_name agent , c.target_name buddy FROM em_agent_buddy_map a, em_targets b, em_targets c WHERE a.agent_target_guid = b.target_guid AND a.buddy_target_guid = c.target_guid; 2 3 4 no rows selected The Trace and Log Files of OMS: The Oracle Management Service has two applications: OMS Console (emgc) and OMS Platform (empbs). The trace and log files of these applications are located in "sysman/log" directory in the EM Instance Home (for example: /u04/app/oracle/oem13cr4/gc_inst/em/EMGC_OMS1/sysman/log ). These logs are very useful to diagnose OMS related problems. [oracle@srlabgg1 log]$ pwd /u04/app/oracle/oem13cr4/gc_inst/em/EMGC_OMS1/sysman/log [oracle@srlabgg1 log]$ ls -lrt total 19512 -rw-r--r--. 1 oracle oinstall 59179 Jan 14 02:47 secure.log -rw-r--r--. 1 oracle oinstall 55118 Jan 14 02:54 syncopss.log -rw-r-----. 1 oracle oinstall 0 Jan 14 07:14 emoms_pbs_startup.log drwxr-x---. 2 oracle oinstall 4096 Jan 14 07:25 pafLogs -rw-r-----. 1 oracle oinstall 1355 Jan 14 10:22 wls.log -rw-r-----. 1 oracle oinstall 5002834 Jan 15 05:33 emoms_startup.trc.1 drwxr-x---. 2 oracle oinstall 4096 Jan 23 01:52 jvmdlogs -rw-r-----. 1 oracle oinstall 5007279 Jan 23 01:53 emoms_pbs_startup.trc.1 -rw-r-----. 1 oracle oinstall 2198152 Jan 23 01:53 emoms_pbs_startup.trc -rw-r-----. 1 oracle oinstall 590 Jan 23 01:57 emoms_startup.log -rw-r-----. 1 oracle oinstall 3847548 Jan 23 01:58 emoms_startup.trc -rw-r-----. 1 oracle oinstall 116139 Jan 23 02:03 emoms_pbs.trc -rw-r-----. 1 oracle oinstall 116139 Jan 23 02:03 emoms_pbs.log -rw-r-----. 1 oracle oinstall 835114 Jan 23 07:00 emoms.trc -rw-r-----. 1 oracle oinstall 835114 Jan 23 07:00 emoms.log -rw-r-----. 1 oracle oinstall 1839951 Jan 23 07:07 emctl.log emoms.log : The main log file for the OMS Console (emgc) application. emoms.trc : The main trace file for the OMS Console (emgc) application. emoms_pbs.log : The main log file for the OMS pbs (empbs) application. emoms_pbs.trc : The main trace file for the OMS pbs (empbs) application. secure.log ; This log provides additional logs for “emctl secure oms” command. emctl.log : This log contains OMS startup/shutdown details. emctl.msg : This log is created by the OMS HealthMonitor thread, when it re-starts the OMS due to a critical error. The Trace and Log Files of EM Managed Server: EM Managed Server is the weblogic server hosting OMS applications. The logs are located in “logs” directory in EM Manager Server Home folder (for example: /u04/app/oracle/oem13cr4/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs): [oracle@srlabgg1 logs]$ pwd /u04/app/oracle/oem13cr4/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs [oracle@srlabgg1 logs]$ ls -lrt total 39816 drwxr-x---. 2 oracle oinstall 4096 Jan 14 02:07 diagnostic_images -rw-r-----. 1 oracle oinstall 0 Jan 14 02:11 EMGC_OMS1-clickhistory.log -rw-r-----. 1 oracle oinstall 0 Jan 14 02:11 DMSEventTraceLoggerDestination-event.log -rw-r-----. 1 oracle oinstall 0 Jan 14 02:11 oracle.dms.strace-event.log drwxr-x---. 2 oracle oinstall 4096 Jan 14 02:13 metrics drwxr-x---. 3 oracle oinstall 4096 Jan 14 02:13 iau drwxr-x---. 3 oracle oinstall 4096 Jan 14 02:16 owsm drwxr-x---. 2 oracle oinstall 4096 Jan 14 08:31 bipublisher -rw-r-----. 1 oracle oinstall 10485426 Jan 14 08:41 EMGC_OMS1-diagnostic-1.log -rw-r-----. 1 oracle oinstall 5274174 Jan 14 08:52 EMGC_OMS1.out00001 -rw-r-----. 1 oracle oinstall 5181597 Jan 15 00:55 EMGC_OMS1.out00002 -rw-r-----. 1 oracle oinstall 5120773 Jan 15 02:20 EMGC_OMS1.log00001 -rw-r-----. 1 oracle oinstall 292107 Jan 23 02:01 emgc-Versions.csv -rw-r-----. 1 oracle oinstall 2632180 Jan 23 07:00 EMGC_OMS1.log -rw-r-----. 1 oracle oinstall 1661786 Jan 23 07:05 access.log -rw-r-----. 1 oracle oinstall 3406198 Jan 23 07:09 EMGC_OMS1.out -rw-r-----. 1 oracle oinstall 6644478 Jan 23 07:09 EMGC_OMS1-diagnostic.log EMGC_OMS1.log : The EMGC_OMSn instance writes all messages from its subsystems and applications to this logfile. EMGC_OMS1.out : The messages written to sysout and syserr are saved in this file by the nodemanager. This log file is rotated by the nodemanager only at the time of Managed server start, and older files need to be manually purged. access.log : This log file contains details of HTTP requests processed by EM Managed Server. EMGC_OMS1-diagnostic.log: This log file contains application-related security errors. The Trace and Log Files of Oracle HTTP Server (OHS): Oracle HTTP Server (OHS) is the web server component of EM13c. It works as a reverse proxy for Oracle WebLogic Server and the framework for hosting static pages. It’s located in “logs” folder in the “OHS Home” (/u04/app/oracle/oem13cr4/gc_inst/user_projects/domains/GCDomain/servers/ohs1/logs): [oracle@srlabgg1 logs]$ pwd /u04/app/oracle/oem13cr4/gc_inst/user_projects/domains/GCDomain/servers/ohs1/logs [oracle@srlabgg1 logs]$ ls -lrt total 3164 -rw-r-----. 1 oracle oinstall 205502 Jan 15 05:59 admin_log.1610712000 -rw-r-----. 1 oracle oinstall 2574 Jan 23 02:06 lastinvocation.log -rw-r-----. 1 oracle oinstall 6 Jan 23 02:07 httpd.pid -rw-r-----. 1 oracle oinstall 260006 Jan 23 05:59 admin_log.1611403200 -rw-r-----. 1 oracle oinstall 806280 Jan 23 07:05 ohs1.log -rw-r-----. 1 oracle oinstall 1306088 Jan 23 07:23 em_upload_https_access_log -rw-r-----. 1 oracle oinstall 97897 Jan 23 07:24 em_upload_http_access_log -rw-r-----. 1 oracle oinstall 459453 Jan 23 07:24 access_log -rw-r-----. 1 oracle oinstall 57024 Jan 23 07:24 admin_log access_log : Access logs stores all the client requests. The HTTP Response code is useful in identifying whether the client request was successful or not. ohs1.log : This is the error log for the OHS. admin_log : This log file keeps Oracle Dynamic Monitoring Service (DMS) requests. em_upload_http_access_log : The log file contains entries for Agent uploads in http mode. em_upload_https_access_log : The log file contains entries for Agent uploads in https mode. The Trace and Log Files of EM Agents: [oracle@srlabgg1 log]$ pwd /u04/app/oracle/agent/agent134/agent_inst/sysman/log [oracle@srlabgg1 agent_inst]$ cd sysman/log/ [oracle@srlabgg1 log]$ ls -lrt total 36872 -rw-r-----. 1 oracle oinstall 0 Jan 14 10:30 gcagent_targetinstaller.log -rw-r-----. 1 oracle oinstall 5243525 Jan 14 10:36 emdctlj.log.6 -rw-r-----. 1 oracle oinstall 5242956 Jan 14 10:40 emdctlj.log.5 -rw-r-----. 1 oracle oinstall 5243101 Jan 14 10:43 emdctlj.log.4 -rw-r-----. 1 oracle oinstall 7182 Jan 14 10:48 secure.log -rw-r-----. 1 oracle oinstall 0 Jan 14 10:48 patchAgtStPlugin.log -rw-r-----. 1 oracle oinstall 5246353 Jan 14 10:49 emdctlj.log.3 -rw-r-----. 1 oracle oinstall 0 Jan 14 10:49 gcagent_ajts.log -rw-r-----. 1 oracle oinstall 0 Jan 14 10:49 gcagent_faup.log -rw-r-----. 1 oracle oinstall 0 Jan 14 10:49 gcagent_wsm.log.0.lck -rw-r-----. 1 oracle oinstall 0 Jan 14 10:49 gcagent_wsm.log.0 -rw-r-----. 1 oracle oinstall 0 Jan 14 10:49 gcagent_opss.log.0.lck -rw-r-----. 1 oracle oinstall 0 Jan 14 10:49 gcagent_opss.log.0 -rw-r-----. 1 oracle oinstall 0 Jan 14 10:49 gcagent_opss_cfg.log.0.lck -rw-r-----. 1 oracle oinstall 0 Jan 14 10:49 gcagent_opss_cfg.log.0 drwxr-----. 2 oracle oinstall 4096 Jan 14 10:50 remote_update -rw-r-----. 1 oracle oinstall 5242881 Jan 15 05:56 emdctlj.log.2 drwxr-----. 26 oracle oinstall 4096 Jan 15 06:51 targets -rw-r-----. 1 oracle oinstall 0 Jan 23 02:55 emctlLockFile.lck -rw-r-----. 1 oracle oinstall 164 Jan 23 02:55 agabend.log -rw-r-----. 1 oracle oinstall 4096 Jan 23 02:55 emagent.nohup.lr -rw-r-----. 1 oracle oinstall 5246704 Jan 23 02:56 emdctlj.log.1 -rw-r-----. 1 oracle oinstall 12339 Jan 23 02:56 startup.info -rw-r-----. 1 oracle oinstall 244 Jan 23 02:56 gcagent_pfu.log -rw-r-----. 1 oracle oinstall 49731 Jan 23 03:01 emctl.log -rw-r-----. 1 oracle oinstall 2084230 Jan 23 03:03 emdctlj.log -rw-r-----. 1 oracle oinstall 98695 Jan 23 03:05 gcagent_errors.log -rw-r-----. 1 oracle oinstall 125740 Jan 23 03:05 gcagent_mdu.log -rw-r-----. 1 oracle oinstall 176 Jan 23 03:08 patchList.info -rw-r-----. 1 oracle oinstall 266867 Jan 23 07:00 emagent.nohup -rw-r-----. 1 oracle oinstall 489537 Jan 23 07:39 gcagent_sdk.trc -rw-r-----. 1 oracle oinstall 135359 Jan 23 07:53 emagent_perl.trc -rw-r-----. 1 oracle oinstall 2908628 Jan 23 07:53 gcagent.log The agent log files are located in the "sysman/log" folder in the "Agent Instance Home" (for example:/u04/app/oracle/agent/agent134/agent_inst/sysman/log). gcagent.log : This log file contains trace, debug, information, error, or warning messages from the Agent. It can be used for debugging Agent framework issues. gcagent_errors.log : It’s similar to the gcagent.log but it contains only the log messages of ERROR and FATAL levels. The log has no size limit. emagent.nohup : Nohup logs of the agent. This log contains startup and shutdown messages including JVM flags. It can be used to determine when the agent is down, and the errors generated on startup of shutdown can be examined with this log. gcagent_mdu.log : This log tracks the metadata updates (such as save or remove targets, blackouts, deployments of metric extensions) to the Agent. gcagent_sdk.trc : The gcagent_sdk.trc file contains log messages from each of the plugins and can be used for debugging plugin related issues. emctl.log : emctl.log provides logging for emctl commands. emdctlj.log : emdctlj.log contains logging for the entry point and the exit codes of the emctl commands. secure.log : This log provides additional logs for “emctl secure agent” command, and can be useful to diagnose failures during securing of the Agent. emagent_perl.trc : Trace file for the PERL scripts. EM uses perl scripts to gather some metrics and target discovery. The log level can be changed by modifying EMAGENT_PERL_TRACE_LEVEL variable in “sysman/config/emd.properties” file.
Post a Comment: