problem with DNS resolution in CLI
old MacBook screen is dying, so switched to a new Ubuntu 22 laptop
immediately DNS problems, hitherto not experienced, ensued
I’m working with a staging box (AWS), xsubdomain.xdomain.com. Web browsers happily resolve the url, proving that I’ve set up AWS security groups correctly, and that DNS record is properly updated (web DNS lookup tools agree). But all the CLI tools, which work on the mac box, fail on Linux (curl and ssh is enough of a case). The symptom was revealed when I tried to ping the box – I’ve noticed a wrong IP address is showing.
nslookup/dig agreed with ping, resolving the url to the same wrong IP.
browsers (Chrome and Brave) are using a different DNS data than the commands in the terminal, that much is certain. The current hypothesis is that, somehow, my DNS cache got sour. I’ve tried (naively) to edit /etc/systemd/resolved.conf (DNS=184.108.40.206) and restarting resolved, but to no avail.
what else have I tried:
resolvectl flush-caches (reportedly what I need to clean the cache)
sudo systemctl restart systemd-resolved
resolvectl statistics (confirmed cache is clean)
but nslookup/dig still shows the weird IP…
What Am I Doing Wrong?
Thank you for your time
Modern web browsers may use DNS-over-SSL or DNS-over-HTTPS services and bypass the regular DNS services configured in the operating system, as those are claimed to be more secure than regular DNS. In your case, this might be producing the expected, correct URL resolution.
If you have the
unscd package installed in Ubuntu, it is a Name Server Cache Daemon – in that case, you might want to run
sudo nscd -i hosts to flush its cache.
DNS diagnostics tools like
bind usually bypass the
nscd cache and talk directly to the nameservers configured in
/etc/resolv.conf. However, if your system is using
systemd-resolved (as indicated by having
nameserver 127.0.0.53 in
resolvectl status displaying the actual DNS settings instead of an error message) they might still be affected by the local cache. But
resolvectl flush-caches should have already dealt with that cache.
As mentioned in the comments by QuartzCristal,
dig @220.127.116.11 xsubdomain.xdomain.com tells the
dig command to directly contact Google’s 18.104.22.168 public DNS service, skipping any DNS services configured to the operating system. If that results in the correct IP address being displayed, then the problem is either within your local system or in the nameservers configured for it; if that also results in the wrong IP address, then something (possibly some malware) might be manipulating any unencrypted DNS traffic between you and the internet.
nameserver lines in your
/etc/resolv.conf and/or the DNS server settings in
resolvectl status output point to your router, it might be a good idea to reset your router: turn it off for 30 seconds or so, then turn it back on. You might want to see if there are security updates available for your router: there are some pieces of malware in the wild that attack routers with known vulnerabilities, and some versions of them might try redirecting the users’ traffic, possibly for snooping or for injecting extra advertising. Your router might have been vulnerable to one of those.
As far as I know, most such malware infections in routers can be cleared by resetting the router, but if the vulnerable firmware version is still present in the router, it might get re-infected pretty quickly.
resolvectl status or
/etc/resolv.conf file points to your Internet Service Provider’s DNS servers, or if they point to your router and resetting the router and getting its firmware up to date did not help, then it’s possible your ISP’s DNS servers have a problem of some sort. In that case, you probably should contact your ISP and report the problem. While waiting for your ISP to fix things at their end, you might want to configure your Ubuntu and/or your router to use Google’s 22.214.171.124 or other public DNS service instead of your ISP’s nameservers, as a workaround.