Keeping Time with Amazon Time Sync Service(aws.amazon.com)
aws.amazon.com
Keeping Time with Amazon Time Sync Service
https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/
65 comments
What is leap second smearing? Spreading it out over some other period, vs. a sudden jump of one second?
Would be interesting to use smearing for daylights saving time too. Make the hour twice as long when changing to winter and only take half the time when changing to summer.
In some scenarios handling DST might not be necessary then!
It wouldn't work for a tv schedule though, since what is broadcasted still uses actual time. ;)
In some scenarios handling DST might not be necessary then!
It wouldn't work for a tv schedule though, since what is broadcasted still uses actual time. ;)
> Spreading it out over some other period
Yes.
They make a second last slightly long for some time period.
Yes.
They make a second last slightly long for some time period.
[deleted]
[deleted]
I setup a local chronyd server on AWS for long time, it's good that now I can use AWS local instead.
Argh! I wish the NTP servers were usable from outside of AWS.
Outside of North America and Europe, there aren't as many public NTP servers as there should be. For example, according to http://www.pool.ntp.org/zone/south-america, South America has less than 50 NTP servers.
It would be awesome if the large, widely-distributed providers (not just AWS with CloudFront, but also Akami, Cloudflare, etc.) provided public, Stratum-1 NTP servers at their POPs.
Outside of North America and Europe, there aren't as many public NTP servers as there should be. For example, according to http://www.pool.ntp.org/zone/south-america, South America has less than 50 NTP servers.
It would be awesome if the large, widely-distributed providers (not just AWS with CloudFront, but also Akami, Cloudflare, etc.) provided public, Stratum-1 NTP servers at their POPs.
Google hosts some: https://developers.google.com/time/
Note that they smear time, though (WRT leap seconds), as do Amazon.
You generally do not want to mix "smearing" and "non-smearing" time servers.
You generally do not want to mix "smearing" and "non-smearing" time servers.
You probably don't want to mix time servers that use a different algorithm for smearing, either.
Interesting point, we're looking at implementing this new ntp server. Wanted to know a bit more about the risks of mixing leap second algos. Is there any more info / links you can give on the subject?
time.windows.com, time.apple.com, and time1.google.com are all publicly accessible.
Apple's one is also available at time.asia.apple.com and time.euro.apple.com
Apple's one is also available at time.asia.apple.com and time.euro.apple.com
[deleted]
windows and apple tend to keep terribly inconsistent time, though, and windows time goes nutso at leap smears.
Do you have a citation for this? I've been monitoring NTP servers for years and Apple's are among the closest to my GPS reference.
Hmm...our data center (bare iron, not cloud) has always provided NTP servers. I assumed this was standard.
Anyone can provide an NTP server, but that's much different from having an atomic clock powered NTP server.
The servers your datacenter provide are probably clients of other NTP servers, so the quality and availability of NTP servers near your datacenter is still really important.
The servers your datacenter provide are probably clients of other NTP servers, so the quality and availability of NTP servers near your datacenter is still really important.
Having an atomic clock behind your NTP server doesn't really gain you anything unless GPS is lost for an extended period of time. As long as a timeserver has a GPS lock, it's going to be disciplining the atomic clock the same way any other clock is disciplined, so you don't really gain anything. Even a cheapie timekeeping GPS chip can get down into the < 500ns accuracy range (and the Thunderbolt on my desk will get down to < 50ns).
(You can gain a little accuracy if your timeserver itself is running its main clock (the one that drives the CPU and all the busses) off of a precision frequency reference, be it GPSDO or atomic clock, but that's a separate discussion)
Oh, and if you're using NTP and not directly connected to a precision timesource (not over the network), you may as well give up on that level of precision anyhow. For that you need PTP.
(You can gain a little accuracy if your timeserver itself is running its main clock (the one that drives the CPU and all the busses) off of a precision frequency reference, be it GPSDO or atomic clock, but that's a separate discussion)
Oh, and if you're using NTP and not directly connected to a precision timesource (not over the network), you may as well give up on that level of precision anyhow. For that you need PTP.
> As long as a timeserver has a GPS lock, it's going to be disciplining the atomic clock the same way any other clock is disciplined
Not true, you need a GPS lock as well as PPS.
Are you sure about 50ns?
Not true, you need a GPS lock as well as PPS.
Are you sure about 50ns?
The PPS is just something you get when the GPS chipset has a lock, though (and sometimes when it doesn't, depending on how you have it configured). Most non-timekeeping GPS chipsets give you a PPS output for free, and 100% of timekeeping GPS chipsets give you one.
Pretty sure about 50ns. Datasheet for the Trimble Thunderbolt E: http://trl.trimble.com/docushare/dsweb/Get/Document-383329/0... ...second page, "PPS accuracy: 15ns (one sigma)" (I picked a larger number than the spec sheet simply so that hopefully nobody would go "oh, that's the ONE SIGMA value, lets argue".)
My Thunderbolt, in a completely temperature-uncontrolled environment with ~20 degree daily swings, generally has a PPS accuracy of ±45ns, in what is pretty crap conditions (first rule of precision timekeeping is "temperature stability")
Pretty sure about 50ns. Datasheet for the Trimble Thunderbolt E: http://trl.trimble.com/docushare/dsweb/Get/Document-383329/0... ...second page, "PPS accuracy: 15ns (one sigma)" (I picked a larger number than the spec sheet simply so that hopefully nobody would go "oh, that's the ONE SIGMA value, lets argue".)
My Thunderbolt, in a completely temperature-uncontrolled environment with ~20 degree daily swings, generally has a PPS accuracy of ±45ns, in what is pretty crap conditions (first rule of precision timekeeping is "temperature stability")
I should have been more specific. You need GPS, PPS and you need to correctly configure them. Its a bad idea if people think you can just plug a USB into a GPS and be good to go.
I would assume a datacentre would put a GPS somewhere to provide time to an NTP server. It's quite cheap, easy to do, and a useful value-add for their customers; and it's something that can only can be done by someone with physical (and likely roof) access.
And I would add for the crowd that one can easily check what sort of NTP server is on offer. NTP tracks the "stratum" of a server, which is how far the server is from an authoritative time source:
https://en.wikipedia.org/wiki/Network_Time_Protocol#Clock_st...
If you're using ntpd, you can check it like this:
https://en.wikipedia.org/wiki/Network_Time_Protocol#Clock_st...
If you're using ntpd, you can check it like this:
$ ntpq
ntpq> peer
remote refid st t when poll reach delay offset jitter
==============================================================================
*y.ns.gin.ntt.ne 249.224.99.213 2 u 815 1024 377 1.230 -4.148 1.825
+hadb2.smatwebde 209.51.161.238 2 u 809 1024 377 49.672 -2.073 3.869
+six0.ntp3.mattn 128.227.205.3 2 u 409 1024 377 63.721 -4.328 2.669
-hadb1.smatwebde 45.79.187.10 3 u 808 1024 377 49.767 -6.600 2.927
As one example, Rackspace doesn’t (they upstream from pool) for all their data centers.
Source: recently asked them this.
Source: recently asked them this.
The datacenter I use does: http://www.he.net/adm/ntp.html
I've always assumed the DC's NTP servers were GPS fed, but I admit I have never asked. I don't need atomic-clock time accuracy for my work at present.
I’m amidst digging into getting some gps clock hardware to make sure an open source db we're developing at work can work in some interesting situations. (Hit me up if you wanna join the team ;) )
High accuracy time is a fascinating domain. It’s this weird mix of general relativity, hardware, and anthropology. Cesium atoms and satellites vs calendars vs astronomy: who wins? :)
Point being: more organizations making high accuracy time available to more folks is a good thing!
High accuracy time is a fascinating domain. It’s this weird mix of general relativity, hardware, and anthropology. Cesium atoms and satellites vs calendars vs astronomy: who wins? :)
Point being: more organizations making high accuracy time available to more folks is a good thing!
What trouble are you having with the GPSs? Getting them hooked up to ntpd is fairly easy after you get used to some of the common hiccups.
As far as who wins for a long time now it has been astronomers. The fact that Google and Amazon are doing leap smearing is a nice sign that the tide may be turning. It is very rare that you see organizations supporting leap seconds during big standards discussions that are not astronomers.
As far as who wins for a long time now it has been astronomers. The fact that Google and Amazon are doing leap smearing is a nice sign that the tide may be turning. It is very rare that you see organizations supporting leap seconds during big standards discussions that are not astronomers.
Getting a rig for some servers is easy. Figuring out a range of hardware to test and support for a range of different types of possible users with different budgets, from some tiny hobby user at home to the entire global economy... have very different needs and hardware budgets. :)
The entire global economy? Maybe you should narrow your market focus a little. NTPD and or GPSD support anything with NMEA and PPS. You should be able to satisfy a large range of users with any ublox 6, 7 or 8T.
Yep, you can buy a Neo-6M with breakout board and antenna for $5 and hook it up to a Raspberry Pi without much trouble.
Software can be fiddly but I found this guide to be helpful: http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html
Hell, you can hook it up to a NodeMCU as well: https://blog.squix.org/2016/05/esp8266-peripherals-ublox-gps...
Bam, $10 wifi enabled GPS.
Software can be fiddly but I found this guide to be helpful: http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html
Hell, you can hook it up to a NodeMCU as well: https://blog.squix.org/2016/05/esp8266-peripherals-ublox-gps...
Bam, $10 wifi enabled GPS.
For near term dev the 8T kit is probably what’s gonna happen. Gpsd is definitely a nice piece of tools
Anything that has PPS and is compatible with GPSD will work..
Cool, something like what happens in Google's datacentres to provide Spanner?
Yes! The system we're working on is one that benefits from the exact same sort of idea : having time info of better accuracy should allow better performance of the same system. (Logical time works too, but will often require much more coordination to resolve ordering or conflicts )
[deleted]
> uses a fleet of redundant satellite-connected and atomic clocks in each region
I am really curious to know what atomic frequency standards they are using. There is something a little suspicious about the wording. I have a feeling they mean "we use GPSDOs" and there are no atomic frequency standards physically located in each regional datacenter.
I'm also a little surprised to see that the remote clock is stratum 4. It would be interesting to see how they are distributing the time between their clocks.
If someone from amazon reads this it would be neat if you posted some more details about your setup.
I am really curious to know what atomic frequency standards they are using. There is something a little suspicious about the wording. I have a feeling they mean "we use GPSDOs" and there are no atomic frequency standards physically located in each regional datacenter.
I'm also a little surprised to see that the remote clock is stratum 4. It would be interesting to see how they are distributing the time between their clocks.
If someone from amazon reads this it would be neat if you posted some more details about your setup.
I have been hoping for this for a long time, with Google’s TrueTime system being so unique. Quite a lot more work would have to go into guaranteeing the results all the way into your application, but this is still a great start.
When is Google announcing TrueTime as a service on GCP? I see them doing this really soon since aws has it now
TrueTime is a lot more than just the NTP service Amazon just announced.
Can you expand on what you meant by that?
If you take it at face value, the TrueTime API is implementable on top of any clock synchronization protocol. It's just coincidentally true that if you're close to a stratum 0-equivalent clock, your skew will be low. In terms of implementation, TrueTime daemons essentially also use Marzullo's algorithm to estimate timestamps, just like NTP.
What am I missing?
If you take it at face value, the TrueTime API is implementable on top of any clock synchronization protocol. It's just coincidentally true that if you're close to a stratum 0-equivalent clock, your skew will be low. In terms of implementation, TrueTime daemons essentially also use Marzullo's algorithm to estimate timestamps, just like NTP.
What am I missing?
Amazon TimeSync is probably a lot more than NTP as well. It must be part of XEN/hw as it is not yet available on the Nitro based instances. That it provide's an NTP endpoint is just the top of the cake...
Interesting to know if they have an SLA for this and how it's formulated.
Time is important and hard. I doubt that customers who really need precise time would walk away from their ghetto solutions with no SLA because when something like this craps out, it's almost always a circus.
But hey, when time precision and accuracy is a bonus not a requirement for your AWS instance then 'free local NTP server!'
Time is important and hard. I doubt that customers who really need precise time would walk away from their ghetto solutions with no SLA because when something like this craps out, it's almost always a circus.
But hey, when time precision and accuracy is a bonus not a requirement for your AWS instance then 'free local NTP server!'
I'd be curious to know how this compares to Google's (internal ) TrueTime service[1], and if it allows for development of globally consistent but highly available storage systems like Spanner.
[1] https://research.google.com/pubs/pub45855.html
[1] https://research.google.com/pubs/pub45855.html
Considering this coincided with announcements of multi-region multi-master DBs (Aurora, DynamoDB) it seems likely.
Does this actually work? We had horrible issues with time drifting between services in AWS at scale in distributed systems.
Something like eg Google Spanner relies on atomic clocks - it's a combination of hardware and software that enable the use of wallclock time.
Something like eg Google Spanner relies on atomic clocks - it's a combination of hardware and software that enable the use of wallclock time.
System clock synchronization is a really hard problem but especially important for things like live video stream encoding where the frame timestamps need to correspond to the real wall-clock (within milliseconds) for things like sub-second play-by-play synchronization.
How accurate does that need to be...?
100ms? 10ms? 1ms? 100us?
100ms? 10ms? 1ms? 100us?
Finance centric:
Finra (Financial Industry Regulatory Authority) requires <50ms
MiFID II (EU centric) HFT requirement can be as low as 100 microseconds accuracy, or 0.1ms.
Finra (Financial Industry Regulatory Authority) requires <50ms
MiFID II (EU centric) HFT requirement can be as low as 100 microseconds accuracy, or 0.1ms.
It varies by sport and how accurate you need your timers to be. NBA needs a timer within "seconds" -- Formula 1 and NASCAR need timers within milliseconds.
The amount of dog fooding that happens at AWS is impressive - looks like they needed and built this for DynamoDB global tables (assumes clock sync across all regions) and made it available to all instances.
Curious if I’ll be able to replace the NTP servers inside our PCI-compliant environment with this.
Are there SLAs for this out? Can I get TrueTime-style assumptions about my clock skew?
Is it going to be monotonically increasing?
I see no reason that it would ever go backward, so yes. Especially with leap second smearing, meaning it's not only monotonically increasing, but also never stops (never a slope of zero).
Somewhat meta, but this is a bit ironic of an offering considering the time on their display boards at re:Invent are noticeably out of sync.
It's kinda neat they're using a link local address.