Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not resolving using search domain #8

Closed
NoumanSaleem opened this issue Feb 20, 2015 · 95 comments
Closed

Not resolving using search domain #8

NoumanSaleem opened this issue Feb 20, 2015 · 95 comments

Comments

@NoumanSaleem
Copy link

Hostname is not properly resolving within the container when setting --dns-search on Docker.

Docker settings:

DOCKER_OPTS="--bip=172.17.42.1/16 --dns=172.17.42.1 --dns=10.0.2.15 --dns-search=service.consul"

gliderlabs/alpine

vagrant@local:~$ docker run --rm -it gliderlabs/alpine sh
/ # ping consul
ping: bad address 'consul'
/ # ping consul.service.consul
PING consul.service.consul (10.211.55.22): 56 data bytes
64 bytes from 10.211.55.22: seq=0 ttl=64 time=0.045 ms

progrium/busybox

vagrant@local:~$ docker run --rm -it progrium/busybox
/ # ping consul
PING consul (10.211.55.22): 56 data bytes

gliderlabs/registrator#111

@andyshinn
Copy link
Contributor

Looks like musl doesn't support domain and search in resolv.conf: http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc#Name_Resolver_.2F_DNS. We're researching workarounds to this issue. Thanks for reporting!

@brycekahle
Copy link

@andyshinn the musl behavior of querying all nameservers in /etc/resolv.conf in parallel and using the the first response is unexpected. Perhaps that could be added to the documentation?

@andyshinn
Copy link
Contributor

Yep, agree it should be noted somewhere. I'll add some more information to the docs about the caveats soon.

@pikeas
Copy link

pikeas commented Mar 22, 2015

@andyshinn The docs are a good start, but they raise further questions. It would be great if the section on dnsmasq (http://gliderlabs.viewdocs.io/docker-alpine/caveats) could be fleshed out to provide a full solution.

@andyshinn
Copy link
Contributor

I tried to provide a general example of the use case where you want to send internal requests to a Consul server and the rest out to upstream name servers. What specifically do you think the example is missing?

@pikeas
Copy link

pikeas commented Mar 24, 2015

A brief description of dnsmasq and how to use it - on the host, as an additional container, etc. In other words, specific instructions on use, rather than "maybe try this workaround".

@progrium
Copy link
Contributor

I think it could use a full dnsmasq config for the custom+upstream DNS
scenario so people can figure out how to run it on the host, then a pointer
to a good generic dnsmasq Docker container that would be easy to apply that
configuration for people that want to run it in a container, and lastly,
once available, a pointer to our magic resolve container.

On Tue, Mar 24, 2015 at 3:36 PM, Aris Pikeas notifications@github.com
wrote:

A brief description of dnsmasq and how to use it - on the host, as an
additional container, etc. In other words, specific instructions on use,
rather than "maybe try this workaround".


Reply to this email directly or view it on GitHub
#8 (comment)
.

Jeff Lindsay
http://progrium.com

@thockin
Copy link

thockin commented May 7, 2015

I was so hopeful for docker-alpine - I need a mini package for DNS utils. Alas, DNS in alpine seems to be totally busted. A shame.

@tifayuki
Copy link

I set up a dnsmasq container. In resolv.dnsmasq.conf, I added

nameserver 8.8.8.8
search example.com

In alpine container I set the dns pointing to that dnsmasq container.

The problem is when I do ping base in alpine container, the log of dnsmasq shows that only base is sent to 8.8.8.8, but base.example.com is never queried.

@andyshinn
Copy link
Contributor

The 3 dnsmasq options that might append a domain to unqualified names is --expand-hosts, --domain, and --auth-zone. Can you try setting these to see if it changes the behavior? Unfortunately, the only other way to do this is with some host file manipulation on the dnsmasq container.

@neilellis
Copy link

This cannot be solved using dnsmasq, dnsmasq assumes glibc has expanded the hostname from the search entry in /etc/resolv.conf.

If you are in an environment (like Tutum) where links don't work because of this:

https://gist.github.com/neilellis/8983d6977f45f126df28

Is my workaround in conjunction with running dnsmasq:

dnsmasq --resolv-file=/etc/dnsmasq-resolv.conf --addn-hosts=/etc/hosts.links --no-daemon

The full image is at https://github.com/vizzbuzz/base-alpine

@progrium
Copy link
Contributor

Alternatively we have a container called Resolvable that helps:
https://github.com/gliderlabs/resolvable

@hypergig
Copy link

Just because no one has mentioned it yet.. DNS search/domain resolution is really important in a kubernetes cluster. Kubernetes services are used for just about every integration point and they all resolve through DNS. Kubernetes injects the right information into /etc/resolv.conf to do this.

The fact that I can't connect to myservice.mynamespace from within an alpine container prevents me from doing anything useful. For example, I have a service called etcd.system that resolves to etcd host for http puts and gets. A very common thing is to write to etcd via curl for application configuration, etc. Alpine is the smallest base image that easily lets you download and use curl, but because my kubernetes service, etcd.system, isn't resolvable I am dead in the water.

I now have to use a 100 meg image to curl, when the 6 meg one would have done fine.

Just food for thought.

@progrium
Copy link
Contributor

Totally agree. Hopefully this puts pressure on musl developers to address this problem. If anybody knows the best way to do this, please share.

In the meantime, we're probably adding search domain support to Resolvable.

@JeanMertz
Copy link

I guess this is still dead in the water. Ran into the exact same use-case as @hypergig, using Kubernetes, hitting DNS problems, finding out Alpine doesn't support DNS search.

The people at Kubernetes (specifically @thockin) seem to be aware of it at kubernetes/kubernetes#10163, but no dice so far.

@macropin
Copy link

macropin commented Sep 3, 2015

I'd like to see this resolved too. We're using skydns for service discovery.

Has anyone raised this upstream with the musl devs?

@thockin
Copy link

thockin commented Sep 3, 2015

I seem to recall not finding a way to file a bug with them.
On Sep 2, 2015 6:11 PM, "Andrew Cutler" notifications@github.com wrote:

I'd like to see this resolved too. We're using skydns for service
discovery.

Has anyone raised this upstream with the musl devs?


Reply to this email directly or view it on GitHub
#8 (comment)
.

@progrium
Copy link
Contributor

progrium commented Sep 3, 2015

I've not found a way yet but if anybody finds a way or files an issue be sure to share the link.

@andyshinn
Copy link
Contributor

I've sent an email to the musl mailing list. It can be viewed at http://www.openwall.com/lists/musl/2015/09/04/3. I'm still gathering some information to answer all of Rich's questions (based on DNS use cases in Consul, Kubernetes and SkyDNS). If you have any specific use cases, please let me know or weigh in directly on the mailing list thread.

@thockin
Copy link

thockin commented Sep 4, 2015

Awesome. Let me know if there is anything I can do to help.

On Fri, Sep 4, 2015 at 12:22 PM, Andy Shinn notifications@github.com
wrote:

I've sent an email to the musl mailing list. It can be viewed at
http://www.openwall.com/lists/musl/2015/09/04/3. I'm still gathering some
information to answer all of Rich's questions (based on DNS use cases in
Consul, Kubernetes and SkyDNS). If you have any specific use cases, please
let me know or weigh in directly on the mailing list thread.


Reply to this email directly or view it on GitHub
#8 (comment)
.

@davidkbainbridge
Copy link

Any update on this? Just ran into it this morning.

@janeczku
Copy link

Sure, there is Alpine-kubernetes: A base image that comes with a special sauce for giving SEARCH domain powers to Docker Alpine. See here: https://github.com/janeczku/docker-alpine-kubernetes.

@mhart
Copy link

mhart commented Oct 22, 2015

It looks like Rich's questions here are still unanswered: http://www.openwall.com/lists/musl/2015/09/04/5

Can anyone familiar with what Kubernetes et al need in this regard respond there? Or should we get Rich involved here? (I can't find him on GitHub though, so that might be out of the question)

@andyshinn
Copy link
Contributor

I think @thockin just re-ignited the conversation. Sorry I've been so lax on the subject. I want to be respectful of everyone's goals and didn't really know how to reasonably answer the questions yet based on the communities goals.

@dchambers
Copy link

@danielo515 / @tuannvm: this is fixed from an Alpine perspective, but it's not as tolerant of non-conforming DNS servers, as was the case with Rancher's DNS until they recently fixed it for example.

@danielo515
Copy link

@dchambers do you know the minimum rancher version that works? I want to check mine. In wich Alpine version is this considered fixed ?

@dchambers
Copy link

I'm not 100% sure because I didn't perform the Rancher upgrade myself, but I know our cluster is definitely now working. More information in this issue.

@dchambers
Copy link

In wich Alpine version is this considered fixed ?

@danielo515, just saw the other part of your question and the answer is Alpine 3.4.

@tuannvm
Copy link

tuannvm commented May 9, 2017

Still have an issue with alpine 3.4 / kube-dns:1.9
Hope that kube-dns:2.x.x would fix this.

rk295 added a commit to rk295/docker-git2consul that referenced this issue May 19, 2017
Mostly to avoid
[this](gliderlabs/docker-alpine#8) issue.
@chenww
Copy link

chenww commented May 26, 2017

alpine does not support multiple nameservers as well. Only one

@maxbbender
Copy link

Any updates on this? Using Jenkins based alpine images with dynamic slaves through kubernetes 1.8.3. Some alpine images can resolve external addresses and others can't. Both have the same /etc/resolv.conf files.

@Jean85
Copy link

Jean85 commented Feb 6, 2018

I've stumbled onto this same issue, and I've solved it removing the a search directive, which was triggering a CloudFlare DNS (buggy) resolution.

I've found out about the solution with this blog post: https://blog.maio.me/alpine-kubernetes-dns/

@richfelker
Copy link

Cloudflare has EWONTFIX'd this issue according to:

kubernetes/dns#119 (comment)

@Jean85
Copy link

Jean85 commented Feb 7, 2018

Exactly, that's why the only solution was removing the triggering domain from the search directive on my end.

@inodb
Copy link

inodb commented May 22, 2018

This is closed, but I'm still having problems in the latest release running on kubernetes (kops on AWS):

kubectl run -i --tty alpine --image=alpine --restart=Never -- nslookup kubernetes.default
nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes.default
Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local

in busybox nslookup works fine:

kubectl run -i --tty busybox --image=busybox --restart=Never -- nslookup kubernetes.default
Server:    100.64.0.10
Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local

@Cloven
Copy link

Cloven commented May 22, 2018 via email

@inodb
Copy link

inodb commented May 22, 2018

@Cloven nice, thanks for clarifying! Seems like it could be nice to have a p big warning sign for using alpine images on k8s somewhere

@danlange
Copy link

I was experiencing a similar issue, and it was caused by a subtle configuration difference between alpine and glibc based linuxes.
If the "ndots" flag in /etc/resolv.conf is set to zero, then the search path will be disabled.
See: https://wiki.musl-libc.org/functional-differences-from-glibc.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests