Debugging is hard, debugging security protocols doubly so. And there are not many tools and how-to’s availabe for NTS yet. So, here’s a (short) list of NTS problems I have seen and some tricks for debugging them.
(This is part 4 of the NTS series. Have you already read part 1?)
The network, your enemy
NTP packet loss
Already for plain NTP, it is known that sometimes, especially across transatlantic links, NTP packets sometimes magically disappear. Some people suggest that some ISPs are rate-limiting NTP packets to protect from DDoS. So, some of the problems you might see with NTS are older than (and independent of) NTS.
Medium-sized NTP packet loss
NTS packets are longer than plain NTP packets. NTP packets that have been used in amplification DDoS attacks several years ago are also longer than plain NTP packets. So some ISPs are aggressively filtering NTP packets longer than usual. Thus, depending on the ISPs your packets travel through, NTS packets might be very unreliable or even impossible to transmit. If it is your own ISP doing the filtering, you can try and talk to them, to remove these filter entirely or at least for your servers. Most servers should no longer be configured such that they can partake in this amplification attack and there are better ways than just blindly filtering port 123 traffic. Also, Chrony does not support that part of the NTP protocol that could be used for amplification.
Source ports
Probably also in a (misguided) attempt at preventing DDoS attacks, some ISPs block packets from privileged ports (<1024) to the NTP port (123). Maybe some also only block port-123-to-port-123 traffic.
This problem might manifest itself that Chrony clients can talk to your server without problems, but packets from NTPd and NTPsec clients are all dropped.
In Switzerland, Sunrise is currently known to block all incoming packets from privileged ports to port 123. This is the reason time.signorini.ch is listed as “Chrony-only” in the public server list.
Certificate issues
Public online tests for HTTPS and several other protocols with TLS are well-known and easy to come by. However, if you think that NTS-KE could be the culprit, there seem to be no online tests available.
However, Miroslav Lichvar provided a helpful one-liner to mimic an NTS-KE client:
printf '\x80\x1\x0\x2\x0\x0\x80\x4\x0\x2\x0\xf\x80\x0\x0\x0' | \
gnutls-cli -p 4460 --alpn=ntske/1 ntp.example.ch \
--logfile=/dev/stderr | hexdump -CFor all NTS-KE implementations, you should see gnutls-cli listing the certificate chain. This should include the entire TLS chain; i.e., if using Let’s Encrypt, it should include the following three certificates:
Processed 128 CA certificate(s).
Resolving 'ntp.trifence.ch:4460'...
Connecting to '109.202.196.249:4460'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
- subject `CN=ntp.trifence.ch', issuer `CN=R3,O=Let's Encrypt,C=US', serial 0x0366cc5bebce8b22d46ed99773b5cb293e21, RSA key 2048 bits, signed using RSA-SHA256, activated `2021-12-19 15:03:57 UTC', expires `2022-03-19 15:03:56 UTC', pin-sha256="YZ3NzoJSQofn3qJGVGxwr+zqK3TwKUkGKECRMUXDqKQ="
Public Key ID:
sha1:d440356a08558e52c97f3bc9a5c459b3ddd437c3
sha256:619dcdce82524287e7dea246546c70afecea2b74f02949062840913145c3a8a4
Public Key PIN:
pin-sha256:YZ3NzoJSQofn3qJGVGxwr+zqK3TwKUkGKECRMUXDqKQ=
- Certificate[1] info:
- subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15 16:00:00 UTC', pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
- Certificate[2] info:
- subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US', issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30 18:14:03 UTC', pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
- Status: The certificate is trusted.
- Description: (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
- Session ID: 7B:C9:66:88:E8:5B:90:0D:52:6C:0C:55:2D:A2:7E:C9:85:0A:21:AB:85:B5:B4:B9:71:52:A3:FE:71:A9:26:CD
- Options:
- Application protocol: ntske/1
- Handshake was completedYou should especially see the highlighted line above, “The certificate is trusted.”
When testing against a Chrony server, you will also get some 50 lines of hex dump, detailing the NTS-KE response. NTPsec servers do not seem to like the hand-crafted NTS-KE request message (the printf command above) and gnutls-cli will just report:
*** Fatal error: Error in the pull function.
*** Server has terminated the connection abnormally.NTPsec as a client will also provide extensive logging with the NTSc tag, which can also be helpful while diagnosing problems. On my system, these end up in /var/log/syslog. Your mileage may vary, though.
