kglic0

MMON Slave Process Reports ORA-7445 [kglic0] Error, Plus Database Hangs [ID 1487720.1]

Applies to:

Oracle Server - Enterprise Edition - Version 11.2.0.1 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.

Symptoms

When the MMON_SLAVE is attempting to perform an "Auto-Flush Slave Action" ,
the following error occurs.

ORA-07445: exception encountered: core dump [kglic0()+1075] [SIGSEGV] [ADDR:0x18] [PC:0x90F1CFF] [Address not mapped to object] []

Subsequently, the following message can be reported in the Alert log:

PMON failed to acquire latch...

The call stack as shown in the trace file at the time of the error includes:

kglic0 kksIterCursorStat kewrrtsq_rank_topsql kewrbtsq_build_topsql kewrftsq_flush_topsql kewrft_flush_table kewrftec_flush_table_ehdlcx kewrfat_flush_all_tables kewrfos_flush_onesnap kewrfsc_flush_snapshot_c kewrafs_auto_flush_slave kebm_slave_main ksvrdp

During this time, a database hang may be experienced 

Cause

If the Shared Pool has multiple subpools, and literal replacement is being used, then the ORA-7445[kglic0] error can occur.

This has been identified as:
BUG 13550185 - QUERIES ON V$SQLSTATS MAY CAUSE CRASH OR HANG
where the situation should match on the following criteria;

1. Using Oracle Database Versions >= 11.2.0.2 but BELOW 12.1, plus the fix for Bug 13550185 should not already be applied,
2. Reporting the ORA-7445[kglic0] error,
3. Experiencing a database hang,
4. The trace file for the ORA-7445[kglic0] error shows that Oracle function: kksIterCursorStat is on the Call Stack (typically, just prior to Oracle function: kglic0, that is reporting the error),
5. There is >1 Subpool (i.e. _kghdsidx_count=2 or more),
6. Literal replacements are being used (i.e. CURSOR_SHARING != EXACT or a select count(*) from v$sql_bind_capture where name like ':SYS%'; returns a count >0 ).

This is as per the Bug Note:
Note 13550185.8 Bug 13550185 - Hang / SGA memory corruption / ORA-7445 [kglic0] with CURSOR_SHARING using multiple shared pool subpools

ADDITIONAL
Please note that there are also the following similar bugs;
Note 12340939.8 Bug 12340939 - ORA-7445 [kglic0] can occur capturing cursor stats for V$SQLSTATS
Note 12409916.8 Bug 12409916 - ORA-7445 [kglic0] accessing V$SQL or V$SQLSTATS
where the symptoms are;

• Process May Dump (ORA-7445) / Abend / Abort
• Dump in or under kglic0
• Stack is likely to include kksIterCursorStat

These bugs differ from Bug 13550185 in that;

• No database hang is experienced,
• Literal replacement is not involved,
• The work-around is to disable the code path that leads to the crash by setting the hidden parameter "_cursor_stats_enabled"=FALSE.

Solution

• Download and apply interim Patch 13550185, if available for your platform and RDBMS release.If Patch 13550185 is not available for your version & platform, log a new Service Request and request a One-Off Back-port.

OR

• Apply PSU 11.2.0.3.4, where the fix for Bug 13550185 has already been included (PSU Patch 14275605)

OR

• Upgrade to 11.2.0.4 (when available) where the fix for Bug 13550185 has already been included.

OR

• WORKAROUND: Use only 1 Subpool (alter system set "_kghdsidx_count"=1 scope=spfile; then restart the instance)

References

NOTE:13550185.8 - Bug 13550185 - Hang / SGA memory corruption / ORA-7445 [kglic0] when using multiple shared pool subpools
BUG:13550185 - QUERIES ON V$SQLSTATS MAY CAUSE CRASH OR HANG
NOTE:12340939.8 - Bug 12340939 - ORA-7445 [kglic0] can occur capturing cursor stats for V$SQLSTATS
NOTE:12409916.8 - Bug 12409916 - ORA-7445 [kglic0] accessing V$SQL or V$SQLSTATS