As you probably know, on Tuesday June 30th a leap second will be added and the day will last 86,401 seconds. Considering the Leap Second Bug which happened in 2012, this event could be quite impacting for the internet and any computer systems. What does it mean and what should you know to be better prepared for this event?
In the end, there’s only one solution: Hitting the Earth with asteroids.
UTC: What is it again?
UTC stand for Coordinated Universal Time and is one of the successors of the Greenwich Mean Time (GMT). It was formalized in 1960 and is the reference time scale derived from the Temps Atomique International (TAI) as defined by the International Telecommunication Union TF.460 Standard.
Did you know…
The official abbreviation for Coordinated Universal Time is UTC. This abbreviation arose from a desire by the International Telecommunication Union and the International Astronomical Union to use the same abbreviation in all languages. English speakers originally proposed CUT (for "coordinated universal time"), while French speakers proposed TUC (for "temps universel coordonné"). The compromise that emerged was UTC, which conforms to the pattern for the abbreviations of the variants of Universal Time (UT0, UT1, UT2, UT1R, etc.).
The International Atomic Time (TAI) is the basis for UTC which is used for civil timekeeping. So, why the leap second?
UTC differs from TAI by an integer number of seconds; it is the basis of all activities in the world. UT1 is the time scale based on the observation of the Earth's rotation. It is now derived from Very Long Baseline Interferometry (VLBI). The various irregular fluctuations progressively detected in the rotation rate of the Earth lead in 1972 to the replacement of UT1 as the reference time scale . However, it was desired by the scientific community to maintain the difference UT1-UTC smaller than 0.9 second to ensure agreement between the physical and astronomical time scales.
Since the adoption of this system in 1972, firstly due to the initial choice of the value of the second (1/86400 mean solar day of the year 1820) and secondly to the general slowing down of the Earth's rotation, it has been necessary to add 25 s to UTC. The last additional second has been introduced on 1 July 2012, at 0hUTC.
This event is bringing so much attention that on 30 June 2015, the U.S. stock exchanges including the New York Stock Exchange’s NYSE Arca and Nasdaq, which trade in an after-hours session from 4 p.m. to 8 p.m. Eastern time, will close Tuesday session early at 23:30:00 UTC (ie 7:30 p.m ET) due to the leap second.
How to prevent 2012 to happen again?
The Leap Second Bug from 2012 is clearly understood and has been fixed in recent linux kernels. Basically, the kernel failed to call the
clock_was_set() syscall after the leap second insertion which led the high-resolution timer (hrtimer) to operate one second in the future. Timers started to expire one second sooner than they should have. Many application, including Java and MySQL use recurring timers that are set for less than a second. This resulted loop has been the source of the load spikes observed in 2012.
This Tuesday, when the leap second is being added you will see something simillar in your syslogs.
Jun 30 19:59:59 app06 kernel: [14049550.644667] Clock: inserting leap second 23:59:60 UTC
So, if you don’t want to feel the pain this year make sure to have the following:
- tzdata version >= 2015a
- Linux >= 3.2.29-1 or Linux 2.6 >= 2.6.32-46
Preventing Issues and Workarounds
If you run some old SUSE Linux Enterprise, make sure to review Novell Notes which contain workarounds.
If you are running RedHat Enterprise 4, 5, 6, or 7, make sure to review how to Resolve Leap Second Issues in RHEL.
Preventing the issue by disabling ntpd during the leap second
By disabling ntpd before the leap second and restarting it after the event, timers will be properly synchronized and no incident should occur.
- stop ntpd
- wait for the leap second
- start ntpd (the clock will be stepped)
Preventing the issue by running ntpd in slew mode on vulnerable systems
If your system is vulnerable to the Leap Second Bug, you can pro-actively change your NTPd configuration to run it in slew mode. In this case, the leap second event will be ignored by the Kernel and NTPd will slowly adjust the time instead.
Warning: some ntpd version have a bug that will still be inserting the leap second while in slew mode, see ntp Bug 2745.
To run ntpd in slew mode instead of step mode:
- stop ntpd
- run ntptime -s 0 -f 0 to reset the kernel status and frequency
- configure ntp with the -x option (ntp slew mode)
- start ntpd
Adressing a server impacted by the Leap Second Bug
If you get impacted by the Leap Second Bug, forcing the system to call the
clock_was_set() syscall will solve your problems. Restarting services won’t be enough. To fix the problem, just run the following on the system impacted:
date -s "$(LC_ALL=C date)"
- LWN.net - Leaping seconds and looping servers
- How Google does it?
- How to clear the Leap Second Insertion flag after it has been received?