2012-07-01

oracle rac 29 june 2012 - problems with 1 second


 https://bugzilla.redhat.com/show_bug.cgi?id=479765

                The kernel attempts to print a message when a leap second is inserted or removed. 
This can cause kernel versions prior to 2.6.29 to hang, due to a deadlock on xtime_lock.



See also the metalink note below .



NTP leap second event causing Oracle Clusterware node reboot [ID 759143.1]
Modified 27-JUN-2012     Type PROBLEM     Status PUBLISHED

In this Document
Symptoms
Changes
Cause
Solution
References

Applies to:

Oracle Server - Enterprise Edition - Version 10.1.0.2 to 11.1.0.7 [Release 10.1 to 11.1]
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)
Sun Solaris SPARC (64-bit)Sun Solaris x86-64 (64-bit)
Oracle Clusterware and patchsets 10.2.0.1 - 11.1.0.7
Sun Solaris 5.8 - 5.10 adjusting time through NTP daemon (xntpd)

Symptoms

UTC time (Coordinated Universal Time) is regularly adjusted by introducing a leap second based on the
accumulated difference between the atomic clock time (TAI) and UT1,
the time reflecting the Earth's rotational speed.

The last adjustment happened on Dec 31, 2008 at 23:59:59 UTC time by adding one leap second,
i.e. UTC -TAI was -1 meaning the UTC time was adjusted backwards by one second.

The decision to introduce a leap second in UTC is the responsibility of the
Earth Orientation Center of the International Earth Rotation and reference System Service (IERS).
IERS announce every six months if a new leap second will occur on the next date of Jun 30 or Dec 31.

Due to the backwards time setting NTP daemons had to adjust time in accordance to the leap second
requirements, affecting Oracle Clusterware, which resulted in node reboots.
The announcement of leap second events can be viewed at:

Changes

UTC time adjustment by leap second worldwide on upstream NTP servers.
In configurations using third party cluster solutions, e.g. Sun Cluster or Veritas SFRAC, the
oproc daemon does not get started. Systems using third party cluster configurations should therefore not be
affected. The third party cluster solution will have to accommodate to the leap second change, if needed.

Cause

A node reboot can occur in the event of a leap second due to both conditions occurring simultaneously:
  • C1. xntpd daemon does not have slewing enabled (default) or does not have PLL mode disabled (default)
     
  • C2. the Oracle Clusterware version does not have a fix for bug 5015469 or bug 6022204
    or the Oracle Clusterware version does have a fix for at least one these defects but due
    to Solaris CR#6595936 the alarm signal arrival has been delayed exceeding the oprocd
    0.5 sec default margin
Condition C1 can be viewed as a generic NTP configuration issue, i.e. other platforms can
be affected not only Solaris.
Typically slewing is enabled on all platforms by starting the NTP
daemon with -x option.

In Solaris 10 the -x option still exists but the "slewalways yes"
option in /etc/inet/ntp.conf should be used.
Note, both "-x" and "slewalways yes" are non-default.

In case condition C1 is not fullfilled the NTP daemon will decide
to step the time offset,i.e. when slewing is not enabled any time
adjustment over 0.5 sec (512 millisec) will
result in stepping the system time. And when PLL mode is enabled
time correction by more than 128 millisec will be stepped.
Therefore the need for both "slewalways yes" and "disable pll".
Condition C2 is partially generic as well, i.e. both bugs bugs 5015469 and 6022204 are generic.
The only strong Solaris specific issue in this constellation is Sun CR#6595936 which affects
oprocd daemon when a negative NTP time adjustement has occurred.

Regarding CR#6595936: this would affect only NTP configurations as per C1 which would
result in stepping the time offset thus being detected by oprocd as a time drift comparing
gettimeofday() values.
However the alarm signal for setitimer() needed by oprocd to wake up will be delayed up to
5 sec and will result in a fatal condition for oprocd initiating a fast reboot.
A fix for Solaris CR#6595936 will be included in Solaris 10 U7, which as of January 2009,
is not yet available.

Solution

To avoid node reboot due to a leap second event either of the two following solutions can be used:
  • S1. Configure xntpd Solaris daemon running on the Oracle Clusterware cluster node to
    disable PLL mode and enable slewing:

    I.e. adding to the /etc/inet/ntp.conf file the two lines:

                       slewalways yes
                       disable pll

    To restart xntpd the commands are:

    Solaris 10:      svcadm restart ntp
    Solaris 8 and 9: /etc/init.d/xntpd stop ; /etc/init.d/xntpd start
As a result a message is written to /var/adm/messages file like:

Jan 2 18:20:34 psyche xntpd[8724]: [ID 998766 daemon.warning] phase-lock loop disabled: will *not* use clock drift file.
Please note that restarting xntpd on a server having the system 
clock out of sync with the NTP server time might result in oprocd
causing a node reboot.
This is because the scripts used to restart xntpd
(/etc/init.d/xntpd in Solaris 8 and 9 and /lib/svc/method/xntp in Solaris 10)
execute the ntpdate command which steps a time difference.

Therefore after changing ntp.conf xntpd should be restarted only before
Oracle Clusterware restart or on next system reboot if it cannot be ensured
that NTP client and server times are in sync. 
OR 
  • S2. Apply recent Oracle Clusterware patch bundles or a recent MLR (i.e. MLR # 9 or higher)
    in order to resolve bugs 5015469 and 6022204.
    and
    increase oprocd daemon timeout margin to ignore alarm signal drifts due to Sun CR#6595936.

    Following patches are available:

    Bug 5015469: fix included in Oracle Clusterware 10.2.0.3 and higher, one off fixes exist for 10.1.0.3,
                         10.1.0.4, 10.1.0.5, 10.2.0.1, 10.2.0.2
                         This fix mainly rearms the oprocd timer after a negative time drift.

    Bug 6022204: fix included in Oracle Clusterware 10.2.0.4 and higher (including 11g),
                         included in Oracle Clusterware 10.2.0.3 MLR 9 and higher
                         and Oracle Clusterware 10.2.0.3 bundle patches 2 and 3
                         (patch # 6756433 and 7117233).
                         This fix doesn't exist for Oracle Clusterware 10.2.0.2 or 10.2.0.1.
                         This fix supersedes 5015469 and is needed in conjunction
                         with increasing the oprocd margin.

    For solution S2 it is needed to increase the oprocd margin to at least 5 sec to avoid the
    alarm signal delays imposed by CR#6595936.
    Please refer to Note 559365.1 how to increase oprocd margin.
    Please note that increasing the oprocd margin in this solution is needed only in order to
    avoid the OS alarm signal drift.
Veritas/Oracle/Sun (VOS) team recommends to use solution S1 and not modify the
oprocd margin while applying recent Oracle Clusterware patches would still be recommended.

References

BUG:5015469 - OPROCD REBOOTS NODE WHEN TIME IS SET BACK BY XNTPD
@ BUG:6022204 - OPROCD ENHANCED ERROR REPORTING MECHANISMS TO BE ADDED
NOTE:559365.1 - Using Diagwait as a diagnostic to get more information for diagnosing Oracle Clusterware Node evictions

Show Related Information Related
Products
  • Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition
Keywords
CLOCK DRIFT; CLUSTERWARE; SOLARIS; TIME DRIFT; TIME OUT

Publicat de la

Niciun comentariu:

Trimiteți un comentariu

Abonați-vă la: Postare comentarii (Atom)