Hi, I am facing an issue when running metaflow_ray with Jobset This is the error:

1 FailedCreatePodSandBox: Failed to create pod sandbox: failed to construct FQDN from pod hostname and cluster domain, FQDN js-5b374c0-control-0-0.js-5b374c0.xxxxxxxxxxxxxxx.svc.cluster.local is too long (64 characters is the max, 74 characters requested)

1. is there any way to fix it without changing our namespace? 2. If I set this

set_hostname_as_fqdn

to False, is there any impact?

@hallowed-glass-14538 thoughts?

What version of jobset / Metaflow r u running ?

@hallowed-glass-14538 Jobset : 0.9.1 Metaflow: 3.18.9

Unfortunately the host name setting is important. Let me verify if there is another way. I think there should be some way we can fix that

Hi @hallowed-glass-14538 I have an idea of changing the subdomain, is it a safe fix?

network:
                  description: Network defines the networking options for the jobset.
                  properties:
                    enableDNSHostnames:
                      description: >-
                        EnableDNSHostnames allows pods to be reached via their
                        hostnames.

                        Pods will be reachable using the fully qualified pod
                        hostname:

                        <jobSet.name>-<spec.replicatedJob.name>-<job-index>-<pod-index>.<subdomain>
                      type: boolean
                    publishNotReadyAddresses:
                      description: >-
                        Indicates if DNS records of pods should be published
                        before the pods are ready.

                        Defaults to True.
                      type: boolean
                    subdomain:
                      description: >-
                        Subdomain is an explicit choice for a network subdomain
                        name

                        When set, any replicated job in the set is added to this
                        network.

                        Defaults to <jobSet.name> if not set.
                      type: string
                  type: object

something like

network:
   subdomain: "j"
   enableDNSHostnames: true

the short explaination is that the jobset fqdn needs to be deterministic so that we can infer the address of the jobset beforehand and pass it as environment variables to the metaflow tasks so that they can communicate with each other. We set subdomain as jobset name because we can deterministically derive it and it has enough entropy that no two independent jobsets will have a shared network. The logic that derives the fqdn is here. One idea can be that we hash the jobsetname and then we use it. This is easy to implement for jobsets execution after the

run

command. Just need to add some hashing function here . Its slightly complicated for argo deployed jobsets. How are you deploying those jobsets ? via the run command or via argo workflows

Yes via argo workflows

The name of the jobsets in general is about 8/9 characters.

if i shorten the subdomain to like 3 character, does it sound good?

meaning ?

js-5b374c0-control-0-0.5b3.xxxxxxxxxxxxxxx.svc.cluster.local

Its a little complicated on the argo side because during dag compilation time, the names are actually present as template string. cant you reduce namespace length ?

Changing the namespace length is a bit difficult for us

If you really want to change the subdomain then you will probably have to chage a few things on the metaflow side. 1. Like here you will have to set subdomain to

"{{= sprig.substr(sprig.sha256sum(inputs.parameters.jobset-name), 0, 3) }}"

2. modify this function to take the subdomain value which you pass down over here 3. modify this function to reflect the change in the function signature ( you can keep passing names here) Once you have you can change the value of ``3` to what ever you like.

Sure. Thanks a lot.