Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set. Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?

Name:      "node2normalpod", Namespace: "ns", }, <ins> DeleteOptions: &metav1.DeleteOptions{GracePeriodSeconds: &zero}, </ins> }) } } createNode2MirrorPodEviction := func(client clientset.Interface) func() error { return func() error { <ins> zero := int64(0) </ins> return client.Policy().Evictions("ns").Evict(&policy.Eviction{ TypeMeta: metav1.TypeMeta{ APIVersion: "policy/v1beta1",


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set. Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?

Name:      "node2mirrorpod", Namespace: "ns", }, <ins> DeleteOptions: &metav1.DeleteOptions{GracePeriodSeconds: &zero}, </ins> }) } }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I'm running a K8s 1.13 cluster and a partial netsplit happened due to a defective linecard dropping only TCP packets above a certain sizes between two masters, but the third could still communicate with both (so the fault made the network non-transitive). etcd was run on the physical network, but Kubernetes traffic runs on a Wireguard overlay for security and routing is provided by OSPF (so internal communication was not affected since it was tunelled over UDP). Etcd experienced some temporary issues due to the connections dropping only certain packets (above ~1000 bytes), but quickly dropped one host out of the quorum, at which point it was degraded but all the remaining hosts could communicate freely. Each master node has an etcd instance and an apiserver. All apiservers are anycasted inside the overlay to balance the load and provide redundancy and the announcement is coupled to the health check, so in case any apiserver fails no traffic will be routed to it. The problem was that the apiserver did not fail its healthcheck even though all etcd write requests timed out. The non-working apiserver was still accessible and got routed its share of the anycast traffic since that was routed over the UDP overlay. Thus ~1/3 of all workloads and nodes were stuck on a non-working apiserver until I nuked the non-working apiserver. TLDR: One etcd server started timing out all writes due to a non-transitive netsplit and the Kubernetes apiserver healthcheck did not fail, causing 1/3 of K8s clients to not have a working control plane until manual intervention. What you expected to happen: The health check should've failed (in my opinion, I'm happy to discuss if you think otherwise). How to reproduce it (as minimally and precisely as possible): Set up 3 etcds with corresponding apiservers, introduce a split between two (shouldn't have to be a packet size limit, a full disconnect should reproduce cleanly), observe that all three report healthy even though at least one of them no longer works. Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.0 Cloud provider or hardware configuration: Physical x86_64 servers OS (e.g. from /etc/os-release): Custom Kernel (e.g. ): Custom, based on 4.19. No patches affect this functionality. Install tools: Custom Others:
/sig api-machinery (maybe cluster-ops?)
cc
Looks like the etcdctl endpoint health command does a random read after fetching the list of members - - we should probably do the same?
Looks sane and would have prevented my issue. Should I PR this?
sure. thanks
/assign

import ( "context" "fmt" <ins> "path" </ins> "sync" "sync/atomic" "time"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I'm running a K8s 1.13 cluster and a partial netsplit happened due to a defective linecard dropping only TCP packets above a certain sizes between two masters, but the third could still communicate with both (so the fault made the network non-transitive). etcd was run on the physical network, but Kubernetes traffic runs on a Wireguard overlay for security and routing is provided by OSPF (so internal communication was not affected since it was tunelled over UDP). Etcd experienced some temporary issues due to the connections dropping only certain packets (above ~1000 bytes), but quickly dropped one host out of the quorum, at which point it was degraded but all the remaining hosts could communicate freely. Each master node has an etcd instance and an apiserver. All apiservers are anycasted inside the overlay to balance the load and provide redundancy and the announcement is coupled to the health check, so in case any apiserver fails no traffic will be routed to it. The problem was that the apiserver did not fail its healthcheck even though all etcd write requests timed out. The non-working apiserver was still accessible and got routed its share of the anycast traffic since that was routed over the UDP overlay. Thus ~1/3 of all workloads and nodes were stuck on a non-working apiserver until I nuked the non-working apiserver. TLDR: One etcd server started timing out all writes due to a non-transitive netsplit and the Kubernetes apiserver healthcheck did not fail, causing 1/3 of K8s clients to not have a working control plane until manual intervention. What you expected to happen: The health check should've failed (in my opinion, I'm happy to discuss if you think otherwise). How to reproduce it (as minimally and precisely as possible): Set up 3 etcds with corresponding apiservers, introduce a split between two (shouldn't have to be a packet size limit, a full disconnect should reproduce cleanly), observe that all three report healthy even though at least one of them no longer works. Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.0 Cloud provider or hardware configuration: Physical x86_64 servers OS (e.g. from /etc/os-release): Custom Kernel (e.g. ): Custom, based on 4.19. No patches affect this functionality. Install tools: Custom Others:
/sig api-machinery (maybe cluster-ops?)
cc
Looks like the etcdctl endpoint health command does a random read after fetching the list of members - - we should probably do the same?
Looks sane and would have prevented my issue. Should I PR this?
sure. thanks
/assign

client := clientValue.Load().(*clientv3.Client) ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) defer cancel() <del> if _, err := client.Cluster.MemberList(ctx); err != nil { return fmt.Errorf("error listing etcd members: %v", err) </del> <ins> // See https://github.com/etcd-io/etcd/blob/master/etcdctl/ctlv3/command/ep_command.go#L118 _, err := client.Get(ctx, path.Join(c.Prefix, "health")) if err == nil { return nil </ins> } <del> return nil </del> <ins> return fmt.Errorf("error getting data from etcd: %v", err) </ins> }, nil }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.

} } <del> // Mount : mounts source to target as NTFS with given options. </del> <ins> // Mount : mounts source to target with given options. // currently only supports cifs(smb), bind mount(for disk) </ins> func (mounter *Mounter) Mount(source string, target string, fstype string, options []string) error { target = normalizeWindowsPath(target) if source == "tmpfs" { <del> klog.V(3).Infof("azureMount: mounting source (%q), target (%q), with options (%q)", source, target, options) </del> <ins> klog.V(3).Infof("mounting source (%q), target (%q), with options (%q)", source, target, options) </ins> return os.MkdirAll(target, 0755) }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.

return err } <del> klog.V(4).Infof("azureMount: mount options(%q) source:%q, target:%q, fstype:%q, begin to mount", </del> <ins> klog.V(4).Infof("mount options(%q) source:%q, target:%q, fstype:%q, begin to mount", </ins> options, source, target, fstype) <del> bindSource := "" </del> <ins> bindSource := source </ins> // tell it's going to mount azure disk or azure file according to options if bind, _, _ := isBind(options); bind {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.

bindSource = normalizeWindowsPath(source) } else { if len(options) < 2 { <del> klog.Warningf("azureMount: mount options(%q) command number(%d) less than 2, source:%q, target:%q, skip mounting", </del> <ins> klog.Warningf("mount options(%q) command number(%d) less than 2, source:%q, target:%q, skip mounting", </ins> options, len(options), source, target) return nil } // currently only cifs mount is supported if strings.ToLower(fstype) != "cifs" { <del> return fmt.Errorf("azureMount: only cifs mount is supported now, fstype: %q, mounting source (%q), target (%q), with options (%q)", fstype, source, target, options) </del> <ins> return fmt.Errorf("only cifs mount is supported now, fstype: %q, mounting source (%q), target (%q), with options (%q)", fstype, source, target, options) </ins> } <del> bindSource = source // use PowerShell Environment Variables to store user input string to prevent command line injection // https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables?view=powershell-5.1 cmdLine := fmt.Sprintf(`$PWord = ConvertTo-SecureString -String $Env:smbpassword -AsPlainText -Force` + `;$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $Env:smbuser, $PWord` + `;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Credential $Credential`) cmd := exec.Command("powershell", "/c", cmdLine) cmd.Env = append(os.Environ(), fmt.Sprintf("smbuser=%s", options[0]), fmt.Sprintf("smbpassword=%s", options[1]), fmt.Sprintf("smbremotepath=%s", source)) if output, err := cmd.CombinedOutput(); err != nil { return fmt.Errorf("azureMount: SmbGlobalMapping failed: %v, only SMB mount is supported now, output: %q", err, string(output)) </del> <ins> if output, err := newSMBMapping(options[0], options[1], source); err != nil { if isSMBMappingExist(source) { klog.V(2).Infof("SMB Mapping(%s) already exists, now begin to remove and remount", source) if output, err := removeSMBMapping(source); err != nil { return fmt.Errorf("Remove-SmbGlobalMapping failed: %v, output: %q", err, output) } if output, err := newSMBMapping(options[0], options[1], source); err != nil { return fmt.Errorf("New-SmbGlobalMapping remount failed: %v, output: %q", err, output) } } else { return fmt.Errorf("New-SmbGlobalMapping failed: %v, output: %q", err, output) } </ins> } }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.

return nil } <ins> // do the SMB mount with username, password, remotepath // return (output, error) func newSMBMapping(username, password, remotepath string) (string, error) { if username == "" || password == "" || remotepath == "" { return "", fmt.Errorf("invalid parameter(username: %s, password: %s, remoteapth: %s)", username, password, remotepath) } // use PowerShell Environment Variables to store user input string to prevent command line injection // https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables?view=powershell-5.1 cmdLine := `$PWord = ConvertTo-SecureString -String $Env:smbpassword -AsPlainText -Force` + `;$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $Env:smbuser, $PWord` + `;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Credential $Credential` cmd := exec.Command("powershell", "/c", cmdLine) cmd.Env = append(os.Environ(), fmt.Sprintf("smbuser=%s", username), fmt.Sprintf("smbpassword=%s", password), fmt.Sprintf("smbremotepath=%s", remotepath)) output, err := cmd.CombinedOutput() return string(output), err } // check whether remotepath is already mounted func isSMBMappingExist(remotepath string) bool { cmd := exec.Command("powershell", "/c", `Get-SmbGlobalMapping -RemotePath $Env:smbremotepath`) cmd.Env = append(os.Environ(), fmt.Sprintf("smbremotepath=%s", remotepath)) _, err := cmd.CombinedOutput() return err == nil } // remove SMB mapping func removeSMBMapping(remotepath string) (string, error) { cmd := exec.Command("powershell", "/c", `Remove-SmbGlobalMapping -RemotePath $Env:smbremotepath -Force`) cmd.Env = append(os.Environ(), fmt.Sprintf("smbremotepath=%s", remotepath)) output, err := cmd.CombinedOutput() return string(output), err } </ins> // Unmount unmounts the target. func (mounter *Mounter) Unmount(target string) error { klog.V(4).Infof("azureMount: Unmount target (%q)", target)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.

} } } <ins> func TestNewSMBMapping(t *testing.T) { tests := []struct { username    string password    string remotepath  string expectError bool }{ { "", "password", `remotepath`, true, }, { "username", "", `remotepath`, true, }, { "username", "password", "", true, }, } for _, test := range tests { _, err := newSMBMapping(test.username, test.password, test.remotepath) if test.expectError { assert.NotNil(t, err, "Expect error during newSMBMapping(%s, %s, %s, %v)", test.username, test.password, test.remotepath) } else { assert.Nil(t, err, "Expect error is nil during newSMBMapping(%s, %s, %s, %v)", test.username, test.password, test.remotepath) } } } </ins>


Question:
We need to add Windows-specific DNS tests to cover the following fields: Test cases needed for : Default, ClusterFirst, None Test cases needed for Test cases needed for Test cases needed for These need to run on Windows containers because the domain resolution policy is different, so different results need to be checked. Windows assumes that any name with a in it is a FQDN, not a PQDN. There is an to cover simple PQDN resolution, but they don't cover the Pod DNS configurations above. /sig windows /milestone v1.14
FYI I'm working on this. I plan to start out by porting over to Windows.
I just moved it into the in progress column. THanks!
What test cases are needed for , , and ? My thoughts were that covers and . I'm adding test cases for Default and ClusterFirst. Does cover the test cases needed for and or do I need more? Or should I just be porting over the entire file?
For the first one - yes. The main differences will be checking the output of instead of , and maybe using instead of but the flow is the same Checking second one now
Agree on second one. I think that will cover it. It may make sense to simplify it a bit because the common methods used there are all based on running bash + dig. If it works with busybox, that may be feasible. The description at matches those test cases
If at all possible, could we use native Windows cmdlets such as instead of relying on Linux cmdlets that have been ported to Windows like ? Am not 100% confident if they are going to behave consistently.
yeah, I think what David's suggesting makes sense if the test cases are forked for Windows.
Opened PR for enabling custom DNS on Azure CNI:
/milestone clear This will need to merge after 1.14 as we're waiting on CNI updates for these tests to pass. I this status note to the Windows kep
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten

name = "go_default_library", srcs = [ "density.go", <ins> "dns.go", </ins> "framework.go", "gmsa.go", "hybrid_network.go",


Question:
We need to add Windows-specific DNS tests to cover the following fields: Test cases needed for : Default, ClusterFirst, None Test cases needed for Test cases needed for Test cases needed for These need to run on Windows containers because the domain resolution policy is different, so different results need to be checked. Windows assumes that any name with a in it is a FQDN, not a PQDN. There is an to cover simple PQDN resolution, but they don't cover the Pod DNS configurations above. /sig windows /milestone v1.14
FYI I'm working on this. I plan to start out by porting over to Windows.
I just moved it into the in progress column. THanks!
What test cases are needed for , , and ? My thoughts were that covers and . I'm adding test cases for Default and ClusterFirst. Does cover the test cases needed for and or do I need more? Or should I just be porting over the entire file?
For the first one - yes. The main differences will be checking the output of instead of , and maybe using instead of but the flow is the same Checking second one now
Agree on second one. I think that will cover it. It may make sense to simplify it a bit because the common methods used there are all based on running bash + dig. If it works with busybox, that may be feasible. The description at matches those test cases
If at all possible, could we use native Windows cmdlets such as instead of relying on Linux cmdlets that have been ported to Windows like ? Am not 100% confident if they are going to behave consistently.
yeah, I think what David's suggesting makes sense if the test cases are forked for Windows.
Opened PR for enabling custom DNS on Azure CNI:
/milestone clear This will need to merge after 1.14 as we're waiting on CNI updates for these tests to pass. I this status note to the Windows kep
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package windows import ( "regexp" "strings" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/test/e2e/framework" e2elog "k8s.io/kubernetes/test/e2e/framework/log" imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" ) const dnsTestPodHostName = "dns-querier-1" const dnsTestServiceName = "dns-test-service" var _ = SIGDescribe("DNS", func() { ginkgo.BeforeEach(func() { framework.SkipUnlessNodeOSDistroIs("windows") }) f := framework.NewDefaultFramework("dns") ginkgo.It("should support configurable pod DNS servers", func() { ginkgo.By("Preparing a test DNS service with injected DNS names...") testInjectedIP := "1.1.1.1" testSearchPath := "resolv.conf.local" ginkgo.By("Creating a pod with dnsPolicy=None and customized dnsConfig...") testUtilsPod := generateDNSUtilsPod() testUtilsPod.Spec.DNSPolicy = v1.DNSNone testUtilsPod.Spec.DNSConfig = &v1.PodDNSConfig{ Nameservers: []string{testInjectedIP}, Searches:    []string{testSearchPath}, } testUtilsPod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(testUtilsPod) framework.ExpectNoError(err) e2elog.Logf("Created pod %v", testUtilsPod) defer func() { e2elog.Logf("Deleting pod %s...", testUtilsPod.Name) if err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(testUtilsPod.Name, metav1.NewDeleteOptions(0)); err != nil { e2elog.Failf("Failed to delete pod %s: %v", testUtilsPod.Name, err) } }() framework.ExpectNoError(f.WaitForPodRunning(testUtilsPod.Name), "failed to wait for pod %s to be running", testUtilsPod.Name) ginkgo.By("Verifying customized DNS option is configured on pod...") cmd := []string{"ipconfig", "/all"} stdout, _, err := f.ExecWithOptions(framework.ExecOptions{ Command:       cmd, Namespace:     f.Namespace.Name, PodName:       testUtilsPod.Name, ContainerName: "util", CaptureStdout: true, CaptureStderr: true, }) framework.ExpectNoError(err) e2elog.Logf("ipconfig /all:n%s", stdout) dnsRegex, err := regexp.Compile(`DNS Servers[s*.]*:(s*[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})+`) if dnsRegex.MatchString(stdout) { match := dnsRegex.FindString(stdout) if !strings.Contains(match, testInjectedIP) { e2elog.Failf("customized DNS options not found in ipconfig /all, got: %s", match) } } else { e2elog.Failf("cannot find DNS server info in ipconfig /all output: n%s", stdout) } // TODO: Add more test cases for other DNSPolicies. }) }) func generateDNSUtilsPod() *v1.Pod { return &v1.Pod{ TypeMeta: metav1.TypeMeta{ Kind: "Pod", }, ObjectMeta: metav1.ObjectMeta{ GenerateName: "e2e-dns-utils-", }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:    "util", Image:   imageutils.GetE2EImage(imageutils.Dnsutils), Command: []string{"sleep", "10000"}, }, }, }, } } </ins>


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In an otherwise-stable bare-metal setup, with only LocalVolume and NFS storage, and without any volume mount activity, I get 80,000+ of these warning messages per day: What you expected to happen: No warnings; and if there is a reason to generate these warnings, more info about what operation is being attempted should be to the warnings message. How to reproduce it (as minimally and precisely as possible): I don't know. This issue emerged after I about 40 services and 300 volumes on a 3-worker cluster. Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.0Cloud provider or hardware configuration: Bare-metal OS (e.g. from /etc/os-release): Ubuntu 18.04.1 bionic Kernel (e.g. ): 4.15.0-42-generic Install tools: kubeadm Others:
/sig storage

if spec.IsKubeletExpandable() { // for kubelet expandable volumes, return a noop plugin that // returns success for expand on the controller <del> klog.Warningf("FindExpandablePluginBySpec(%s) -> returning noopExpandableVolumePluginInstance", spec.Name()) </del> <ins> klog.V(4).Infof("FindExpandablePluginBySpec(%s) -> returning noopExpandableVolumePluginInstance", spec.Name()) </ins> return &noopExpandableVolumePluginInstance{spec}, nil } <del> klog.Warningf("FindExpandablePluginBySpec(%s) -> err:%v", spec.Name(), err) </del> <ins> klog.V(4).Infof("FindExpandablePluginBySpec(%s) -> err:%v", spec.Name(), err) </ins> return nil, err }


Question: Which jobs are failing: Which test(s) are failing: Job Reason for failure --- --- Since when has it been failing: 2019-02-11 Anything else we need to know: started failing with commit started failing with commit cc /kind failing-test /priority important-soon /sig api-machinery

optional APIServiceStatus status = 3; } <ins> // APIServiceCondition describes the state of an APIService at a particular point </ins> message APIServiceCondition { // Type is the type of the condition. optional string type = 1;


Question:
What happened: The preparation step of the e2e tests were failed sometimes on my local environment like What you expected to happen: These tests can be passed without any errors. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): 16.04.4 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-138-generic Install tools: kubeadm Others:
/sig storage
After discussing on the PR I have known I need to enable StorageObjectInUseProtection plugin of admission controller which is disabled by default. The corresponding e2e tests require the plugin is enabled, so I think it is better to enable the plugin by default for most users.
Investigation of StorageObjectInUseProtection plugin of admission controller under kubernetes/kubernetes: cluster/centos/config- Enabled cluster/gce/config- Enabled cluster/gce/config- Enabled cluster/kubemark/gce/config- Enabled hack/local-up- Enabled cluster/juju/layers/kubernetes- Disabled This is the reason why we don't face this issue on the upstream CI system, and I it on my local environment which is deployed with kubeadm.

# Admission Controllers to invoke prior to persisting objects in cluster ENABLE_ADMISSION_PLUGINS="LimitRanger,ResourceQuota" <del> DISABLE_ADMISSION_PLUGINS="ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook" </del> <ins> DISABLE_ADMISSION_PLUGINS="ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,StorageObjectInUseProtection" </ins> # Include RBAC (to exercise bootstrapping), and AlwaysAllow to allow all actions AUTHORIZATION_MODE="RBAC,AlwaysAllow"


Question:
What happened: The preparation step of the e2e tests were failed sometimes on my local environment like What you expected to happen: These tests can be passed without any errors. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): 16.04.4 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-138-generic Install tools: kubeadm Others:
/sig storage
After discussing on the PR I have known I need to enable StorageObjectInUseProtection plugin of admission controller which is disabled by default. The corresponding e2e tests require the plugin is enabled, so I think it is better to enable the plugin by default for most users.
Investigation of StorageObjectInUseProtection plugin of admission controller under kubernetes/kubernetes: cluster/centos/config- Enabled cluster/gce/config- Enabled cluster/gce/config- Enabled cluster/kubemark/gce/config- Enabled hack/local-up- Enabled cluster/juju/layers/kubernetes- Disabled This is the reason why we don't face this issue on the upstream CI system, and I it on my local environment which is deployed with kubeadm.

ETCDCTL=$(which etcdctl) KUBECTL="${KUBE_OUTPUT_HOSTBIN}/kubectl" UPDATE_ETCD_OBJECTS_SCRIPT="${KUBE_ROOT}/cluster/update-storage-objects.sh" <del> DISABLE_ADMISSION_PLUGINS="ServiceAccount,NamespaceLifecycle,LimitRanger,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PersistentVolumeLabel,DefaultStorageClass" </del> <ins> DISABLE_ADMISSION_PLUGINS="ServiceAccount,NamespaceLifecycle,LimitRanger,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PersistentVolumeLabel,DefaultStorageClass,StorageObjectInUseProtection" </ins> function startApiServer() { local storage_versions=${1:-""}


Question:
What happened: The preparation step of the e2e tests were failed sometimes on my local environment like What you expected to happen: These tests can be passed without any errors. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): 16.04.4 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-138-generic Install tools: kubeadm Others:
/sig storage
After discussing on the PR I have known I need to enable StorageObjectInUseProtection plugin of admission controller which is disabled by default. The corresponding e2e tests require the plugin is enabled, so I think it is better to enable the plugin by default for most users.
Investigation of StorageObjectInUseProtection plugin of admission controller under kubernetes/kubernetes: cluster/centos/config- Enabled cluster/gce/config- Enabled cluster/gce/config- Enabled cluster/kubemark/gce/config- Enabled hack/local-up- Enabled cluster/juju/layers/kubernetes- Disabled This is the reason why we don't face this issue on the upstream CI system, and I it on my local environment which is deployed with kubeadm.

// DefaultOffAdmissionPlugins get admission plugins off by default for kube-apiserver. func DefaultOffAdmissionPlugins() sets.String { defaultOnPlugins := sets.NewString( <del> lifecycle.PluginName,                //NamespaceLifecycle limitranger.PluginName,              //LimitRanger serviceaccount.PluginName,           //ServiceAccount setdefault.PluginName,               //DefaultStorageClass resize.PluginName,                   //PersistentVolumeClaimResize defaulttolerationseconds.PluginName, //DefaultTolerationSeconds mutatingwebhook.PluginName,          //MutatingAdmissionWebhook validatingwebhook.PluginName,        //ValidatingAdmissionWebhook resourcequota.PluginName,            //ResourceQuota </del> <ins> lifecycle.PluginName,                    //NamespaceLifecycle limitranger.PluginName,                  //LimitRanger serviceaccount.PluginName,               //ServiceAccount setdefault.PluginName,                   //DefaultStorageClass resize.PluginName,                       //PersistentVolumeClaimResize defaulttolerationseconds.PluginName,     //DefaultTolerationSeconds mutatingwebhook.PluginName,              //MutatingAdmissionWebhook validatingwebhook.PluginName,            //ValidatingAdmissionWebhook resourcequota.PluginName,                //ResourceQuota storageobjectinuseprotection.PluginName, //StorageObjectInUseProtection </ins> ) if utilfeature.DefaultFeatureGate.Enabled(features.PodPriority) {


Question:
I have been deploying master-branch kubernetes on Windows using flannel/host-gateway networking, with one master, one linux worker, and three windows worker. Recently, kube-proxy stopped running correctly. It exits with an error about not finding the HNS Network. I have confirmed that the HNS network exists on the host with the correct name (cbr0), and have identified the last commit where kube-proxy works. What happened: Starting with the commit merging , kube-proxy outputs the following error message: This is confusing because does show : (nat network removed from above output) What you expected to happen: kube-proxy to work in host-gateway mode. How to reproduce it (as minimally and precisely as possible): Build kubernetes at or after . Start flanneld, kubelet, and kube-proxy on a windows 2019 vm. See kube-proxy fail despite flanneld having created the network. Anything else we need to know?: Environment: Kubernetes version (use ): kube-proxy breaks on commit and later; works on Cloud provider or hardware configuration: vsphere OS (e.g: ): windows server 2019, with up-to-date patches Kernel (e.g. ): windows nodes are , linux master/worker are Install tools: deployed using /sig windows cc
I'm aware of this issue. The fix will be out within an hour.

var hns HostNetworkService hns = hnsV1{} supportedFeatures := hcn.GetSupportedFeatures() <del> if supportedFeatures.Api.V2 { </del> <ins> if supportedFeatures.RemoteSubnet { </ins> hns = hnsV2{} }


Question:
Which jobs are failing: All new tests introduced for api change Which test(s) are failing: New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Existing tests are working fine and the whole suite works correctly in the gce suite https://k8s- Since when has it been failing: 21st February 2019 (since merge of PR) Testgrid link: https://k8s- Reason for failure: Investigation shows that node e2e tests use a different framework pod creation pattern Doing some comparisons of tests that work and those that don't reveals differences in the standard client set as and the existing ones which do work go through the intermediate which uses The version has an additional function specifically for node tests
/sig storage
/sig node /sig test
/assign
/cc
/sig test

} By("creating the pod with failed condition") <del> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod) Expect(err).ToNot(HaveOccurred(), "while creating pod") err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) Expect(err).To(HaveOccurred(), "while waiting for pod to be running") </del> var podClient *framework.PodClient podClient = f.PodClient() <ins> pod = podClient.Create(pod) err := framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) Expect(err).To(HaveOccurred(), "while waiting for pod to be running") </ins> By("updating the pod") podClient.Update(podName, func(pod *v1.Pod) {


Question:
Which jobs are failing: All new tests introduced for api change Which test(s) are failing: New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Existing tests are working fine and the whole suite works correctly in the gce suite https://k8s- Since when has it been failing: 21st February 2019 (since merge of PR) Testgrid link: https://k8s- Reason for failure: Investigation shows that node e2e tests use a different framework pod creation pattern Doing some comparisons of tests that work and those that don't reveals differences in the standard client set as and the existing ones which do work go through the intermediate which uses The version has an additional function specifically for node tests
/sig storage
/sig node /sig test
/assign
/cc
/sig test

} By("creating the pod") <del> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod) </del> <ins> var podClient *framework.PodClient podClient = f.PodClient() pod = podClient.Create(pod) </ins> By("waiting for pod running") <del> err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) </del> <ins> err := framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) </ins> Expect(err).NotTo(HaveOccurred(), "while waiting for pod to be running") By("creating a file in subpath")


Question:
Which jobs are failing: All new tests introduced for api change Which test(s) are failing: New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Existing tests are working fine and the whole suite works correctly in the gce suite https://k8s- Since when has it been failing: 21st February 2019 (since merge of PR) Testgrid link: https://k8s- Reason for failure: Investigation shows that node e2e tests use a different framework pod creation pattern Doing some comparisons of tests that work and those that don't reveals differences in the standard client set as and the existing ones which do work go through the intermediate which uses The version has an additional function specifically for node tests
/sig storage
/sig node /sig test
/assign
/cc
/sig test

} By("updating the annotation value") <del> var podClient *framework.PodClient podClient = f.PodClient() </del> podClient.Update(podName, func(pod *v1.Pod) { pod.ObjectMeta.Annotations["mysubpath"] = "mynewpath" })


Question:
Which jobs are failing: All new tests introduced for api change Which test(s) are failing: New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Existing tests are working fine and the whole suite works correctly in the gce suite https://k8s- Since when has it been failing: 21st February 2019 (since merge of PR) Testgrid link: https://k8s- Reason for failure: Investigation shows that node e2e tests use a different framework pod creation pattern Doing some comparisons of tests that work and those that don't reveals differences in the standard client set as and the existing ones which do work go through the intermediate which uses The version has an additional function specifically for node tests
/sig storage
/sig node /sig test
/assign
/cc
/sig test

// Start pod By(fmt.Sprintf("Creating pod %s", pod.Name)) <del> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod) Expect(err).ToNot(HaveOccurred(), "while creating pod") </del> <ins> var podClient *framework.PodClient podClient = f.PodClient() pod = podClient.Create(pod) </ins> defer func() { framework.DeletePodWithWait(f, f.ClientSet, pod) }() <del> err = framework.WaitForPodRunningInNamespace(f.ClientSet, pod) </del> <ins> err := framework.WaitForPodRunningInNamespace(f.ClientSet, pod) </ins> Expect(err).ToNot(HaveOccurred(), "while waiting for pod to be running") <del> var podClient *framework.PodClient podClient = f.PodClient() </del> By("updating the pod") podClient.Update(podName, func(pod *v1.Pod) { pod.ObjectMeta.Annotations = map[string]string{"mysubpath": "newsubpath"}


Question:
Which jobs are failing: All new tests introduced for api change Which test(s) are failing: New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Existing tests are working fine and the whole suite works correctly in the gce suite https://k8s- Since when has it been failing: 21st February 2019 (since merge of PR) Testgrid link: https://k8s- Reason for failure: Investigation shows that node e2e tests use a different framework pod creation pattern Doing some comparisons of tests that work and those that don't reveals differences in the standard client set as and the existing ones which do work go through the intermediate which uses The version has an additional function specifically for node tests
/sig storage
/sig node /sig test
/assign
/cc
/sig test

func testPodFailSubpath(f *framework.Framework, pod *v1.Pod) { <del> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod) Expect(err).ToNot(HaveOccurred(), "while creating pod") </del> <ins> var podClient *framework.PodClient podClient = f.PodClient() pod = podClient.Create(pod) </ins> defer func() { framework.DeletePodWithWait(f, f.ClientSet, pod) }() <del> err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) </del> <ins> err := framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) </ins> Expect(err).To(HaveOccurred(), "while waiting for pod to be running") }


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: in Which test(s) are failing: Since when has it been failing: 2019-02-27 () -- since it was Testgrid link: Reason for failure: Timeout: Anything else we need to know: /kind failing-test /sig storage /priority important-soon /milestone v1.14 cc
cc Could you help with this please?
/assign
Neither of the failing tests mentioned in this issue are present in testgrid anymore. is all clear

var _ = utils.SIGDescribe("CSI mock volume", func() { type testParameters struct { <del> attachable      bool </del> <ins> disableAttach   bool </ins> attachLimit     int registerDriver  bool podInfoVersion  *string


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: in Which test(s) are failing: Since when has it been failing: 2019-02-27 () -- since it was Testgrid link: Reason for failure: Timeout: Anything else we need to know: /kind failing-test /sig storage /priority important-soon /milestone v1.14 cc
cc Could you help with this please?
/assign
Neither of the failing tests mentioned in this issue are present in testgrid anymore. is all clear

csics := f.CSIClientSet var err error <del> m.driver = drivers.InitMockCSIDriver(tp.registerDriver, tp.attachable, tp.podInfoVersion, tp.attachLimit) </del> <ins> m.driver = drivers.InitMockCSIDriver(tp.registerDriver, !tp.disableAttach, tp.podInfoVersion, tp.attachLimit) </ins> config, testCleanup := m.driver.PrepareTest(f) m.testCleanups = append(m.testCleanups, testCleanup) m.config = config


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: in Which test(s) are failing: Since when has it been failing: 2019-02-27 () -- since it was Testgrid link: Reason for failure: Timeout: Anything else we need to know: /kind failing-test /sig storage /priority important-soon /milestone v1.14 cc
cc Could you help with this please?
/assign
Neither of the failing tests mentioned in this issue are present in testgrid anymore. is all clear

// The CSIDriverRegistry feature gate is needed for this test in Kubernetes 1.12. Context("CSI attach test using mock driver [Feature:CSIDriverRegistry]", func() { tests := []struct { <del> name             string driverAttachable bool deployDriverCRD  bool </del> <ins> name            string disableAttach   bool deployDriverCRD bool </ins> }{ { <del> name:             "should not require VolumeAttach for drivers without attachment", driverAttachable: false, deployDriverCRD:  true, </del> <ins> name:            "should not require VolumeAttach for drivers without attachment", disableAttach:   true, deployDriverCRD: true, </ins> }, { <del> name:             "should require VolumeAttach for drivers with attachment", driverAttachable: true, deployDriverCRD:  true, </del> <ins> name:            "should require VolumeAttach for drivers with attachment", deployDriverCRD: true, </ins> }, { <del> name:             "should preserve attachment policy when no CSIDriver present", driverAttachable: true, deployDriverCRD:  false, </del> <ins> name:            "should preserve attachment policy when no CSIDriver present", deployDriverCRD: false, </ins> }, } for _, t := range tests { test := t It(t.name, func() { var err error <del> init(testParameters{registerDriver: test.deployDriverCRD, attachable: test.driverAttachable}) </del> <ins> init(testParameters{registerDriver: test.deployDriverCRD, disableAttach: test.disableAttach}) </ins> defer cleanup() _, claim, pod := createPod()


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: in Which test(s) are failing: Since when has it been failing: 2019-02-27 () -- since it was Testgrid link: Reason for failure: Timeout: Anything else we need to know: /kind failing-test /sig storage /priority important-soon /milestone v1.14 cc
cc Could you help with this please?
/assign
Neither of the failing tests mentioned in this issue are present in testgrid anymore. is all clear

_, err = m.cs.StorageV1beta1().VolumeAttachments().Get(attachmentName, metav1.GetOptions{}) if err != nil { if errors.IsNotFound(err) { <del> if test.driverAttachable { </del> <ins> if !test.disableAttach { </ins> framework.ExpectNoError(err, "Expected VolumeAttachment but none was found") } } else { framework.ExpectNoError(err, "Failed to find VolumeAttachment") } } <del> if !test.driverAttachable { </del> <ins> if test.disableAttach { </ins> Expect(err).To(HaveOccurred(), "Unexpected VolumeAttachment found") } })


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: in Which test(s) are failing: Since when has it been failing: 2019-02-27 () -- since it was Testgrid link: Reason for failure: Timeout: Anything else we need to know: /kind failing-test /sig storage /priority important-soon /milestone v1.14 cc
cc Could you help with this please?
/assign
Neither of the failing tests mentioned in this issue are present in testgrid anymore. is all clear

// define volume limit to be 2 for this test var err error <del> init(testParameters{attachable: true, nodeSelectorKey: "node-attach-limit-csi", attachLimit: 2}) </del> <ins> init(testParameters{nodeSelectorKey: "node-attach-limit-csi", attachLimit: 2}) </ins> defer cleanup() nodeName := m.config.ClientNodeName attachKey := v1.ResourceName(volumeutil.GetCSIAttachLimitKey(m.provisioner))


Question:
What happened: Currently starting Kubectl failing on Azure Stack. What you expected to happen: Kubectl should start on Azure Stack environment. How to reproduce it (as minimally and precisely as possible): Try to deploy Kubernets on Azure Stack environment.. Anything else we need to know?: Environment: Kubernetes version (use kubectl version): master Cloud provider or hardware configuration: OS (e.g: cat /etc/os-release): Kernel (e.g. uname -a): Install tools: Others: /sig azure /assign
is it ? is a tool ...
my bad its kubelet. I update the title.
then could you provide the kubelet log? I am not sure kubelet on Azure stack is running as a container or process, you could get the logs by: And how do you deploy k8s on azure stack? cc

func TestGetZone(t *testing.T) { cloud := &Cloud{ Config: Config{ <del> Location: "eastus", </del> <ins> Location:            "eastus", UseInstanceMetadata: true, </ins> }, } testcases := []struct {


Question:
What happened: Currently starting Kubectl failing on Azure Stack. What you expected to happen: Kubectl should start on Azure Stack environment. How to reproduce it (as minimally and precisely as possible): Try to deploy Kubernets on Azure Stack environment.. Anything else we need to know?: Environment: Kubernetes version (use kubectl version): master Cloud provider or hardware configuration: OS (e.g: cat /etc/os-release): Kernel (e.g. uname -a): Install tools: Others: /sig azure /assign
is it ? is a tool ...
my bad its kubelet. I update the title.
then could you provide the kubelet log? I am not sure kubelet on Azure stack is running as a container or process, you could get the logs by: And how do you deploy k8s on azure stack? cc

import ( "context" "fmt" <ins> "os" </ins> "strconv" "strings"


Question:
What happened: Currently starting Kubectl failing on Azure Stack. What you expected to happen: Kubectl should start on Azure Stack environment. How to reproduce it (as minimally and precisely as possible): Try to deploy Kubernets on Azure Stack environment.. Anything else we need to know?: Environment: Kubernetes version (use kubectl version): master Cloud provider or hardware configuration: OS (e.g: cat /etc/os-release): Kernel (e.g. uname -a): Install tools: Others: /sig azure /assign
is it ? is a tool ...
my bad its kubelet. I update the title.
then could you provide the kubelet log? I am not sure kubelet on Azure stack is running as a container or process, you could get the logs by: And how do you deploy k8s on azure stack? cc

// GetZone returns the Zone containing the current availability zone and locality region that the program is running in. // If the node is not running with availability zones, then it will fall back to fault domain. func (az *Cloud) GetZone(ctx context.Context) (cloudprovider.Zone, error) { <del> metadata, err := az.metadata.GetMetadata() if err != nil { return cloudprovider.Zone{}, err } </del> <ins> if az.UseInstanceMetadata { metadata, err := az.metadata.GetMetadata() if err != nil { return cloudprovider.Zone{}, err } </ins> <del> if metadata.Compute == nil { return cloudprovider.Zone{}, fmt.Errorf("failure of getting compute information from instance metadata") } </del> <ins> if metadata.Compute == nil { return cloudprovider.Zone{}, fmt.Errorf("failure of getting compute information from instance metadata") } </ins> <del> zone := "" if metadata.Compute.Zone != "" { zoneID, err := strconv.Atoi(metadata.Compute.Zone) if err != nil { return cloudprovider.Zone{}, fmt.Errorf("failed to parse zone ID %q: %v", metadata.Compute.Zone, err) </del> <ins> zone := "" if metadata.Compute.Zone != "" { zoneID, err := strconv.Atoi(metadata.Compute.Zone) if err != nil { return cloudprovider.Zone{}, fmt.Errorf("failed to parse zone ID %q: %v", metadata.Compute.Zone, err) } zone = az.makeZone(zoneID) } else { klog.V(3).Infof("Availability zone is not enabled for the node, falling back to fault domain") zone = metadata.Compute.FaultDomain </ins> } <del> zone = az.makeZone(zoneID) } else { klog.V(3).Infof("Availability zone is not enabled for the node, falling back to fault domain") zone = metadata.Compute.FaultDomain } return cloudprovider.Zone{ FailureDomain: zone, Region:        az.Location, }, nil </del> <ins> return cloudprovider.Zone{ FailureDomain: zone, Region:        az.Location, }, nil } // if UseInstanceMetadata is false, get Zone name by calling ARM hostname, err := os.Hostname() if err != nil { return cloudprovider.Zone{}, fmt.Errorf("failure getting hostname from kernel") } return az.vmSet.GetZoneByNodeName(strings.ToLower(hostname)) </ins> } // GetZoneByProviderID implements Zones.GetZoneByProviderID


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node

"//pkg/kubelet/apis/resourcemetrics/v1alpha1:go_default_library", "//pkg/kubelet/container:go_default_library", "//pkg/kubelet/prober:go_default_library", <ins> "//pkg/kubelet/server/metrics:go_default_library", </ins> "//pkg/kubelet/server/portforward:go_default_library", "//pkg/kubelet/server/remotecommand:go_default_library", "//pkg/kubelet/server/stats:go_default_library",


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node

name = "all-srcs", srcs = [ ":package-srcs", <ins> "//pkg/kubelet/server/metrics:all-srcs", </ins> "//pkg/kubelet/server/portforward:all-srcs", "//pkg/kubelet/server/remotecommand:all-srcs", "//pkg/kubelet/server/stats:all-srcs",


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node

<ins> package(default_visibility = ["//visibility:public"]) load( "@io_bazel_rules_go//go:def.bzl", "go_library", ) go_library( name = "go_default_library", srcs = ["metrics.go"], importpath = "k8s.io/kubernetes/pkg/kubelet/server/metrics", deps = [ "//vendor/github.com/prometheus/client_golang/prometheus:go_default_library", ], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], ) </ins>


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics import ( "sync" "time" "github.com/prometheus/client_golang/prometheus" ) const ( kubeletSubsystem = "kubelet" ) var ( // HTTPRequests tracks the number of the http requests received since the server started. HTTPRequests = prometheus.NewCounterVec( prometheus.CounterOpts{ Subsystem: kubeletSubsystem, Name:      "http_requests_total", Help:      "Number of the http requests received since the server started", }, // server_type aims to differentiate the readonly server and the readwrite server. // long_running marks whether the request is long-running or not. // Currently, long-running requests include exec/attach/portforward/debug. []string{"method", "path", "host", "server_type", "long_running"}, ) // HTTPRequestsDuration tracks the duration in seconds to serve http requests. HTTPRequestsDuration = prometheus.NewHistogramVec( prometheus.HistogramOpts{ Subsystem: kubeletSubsystem, Name:      "http_requests_duration_seconds", Help:      "Duration in seconds to serve http requests", // Use DefBuckets for now, will customize the buckets if necessary. Buckets: prometheus.DefBuckets, }, []string{"method", "path", "host", "server_type", "long_running"}, ) // HTTPInflightRequests tracks the number of the inflight http requests. HTTPInflightRequests = prometheus.NewGaugeVec( prometheus.GaugeOpts{ Subsystem: kubeletSubsystem, Name:      "http_inflight_requests", Help:      "Number of the inflight http requests", }, []string{"method", "path", "host", "server_type", "long_running"}, ) ) var registerMetrics sync.Once // Register all metrics. func Register() { registerMetrics.Do(func() { prometheus.MustRegister(HTTPRequests) prometheus.MustRegister(HTTPRequestsDuration) prometheus.MustRegister(HTTPInflightRequests) }) } // SinceInSeconds gets the time since the specified start in seconds. func SinceInSeconds(start time.Time) float64 { return time.Since(start).Seconds() } </ins>


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node

"k8s.io/kubernetes/pkg/kubelet/apis/resourcemetrics/v1alpha1" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/prober" <ins> servermetrics "k8s.io/kubernetes/pkg/kubelet/server/metrics" </ins> "k8s.io/kubernetes/pkg/kubelet/server/portforward" remotecommandserver "k8s.io/kubernetes/pkg/kubelet/server/remotecommand" "k8s.io/kubernetes/pkg/kubelet/server/stats"


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node

proxyStream(response.ResponseWriter, request.Request, url) } <ins> // trimURLPath trims a URL path. // For paths in the format of "/metrics/xxx", "metrics/xxx" is returned; // For all other paths, the first part of the path is returned. func trimURLPath(path string) string { parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3) if len(parts) == 0 { return path } if parts[0] == "metrics" && len(parts) > 1 { return fmt.Sprintf("%s/%s", parts[0], parts[1]) } return parts[0] } // isLongRunningRequest determines whether the request is long-running or not. func isLongRunningRequest(path string) bool { longRunningRequestPaths := []string{"exec", "attach", "portforward", "debug"} for _, p := range longRunningRequestPaths { if p == path { return true } } return false } </ins> // ServeHTTP responds to HTTP requests on the Kubelet. func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) { defer httplog.NewLogged(req, &w).StacktraceWhen(


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node

http.StatusSwitchingProtocols, ), ).Log() <ins> // monitor http requests var serverType string if s.auth == nil { serverType = "readonly" } else { serverType = "readwrite" } method, path, host := req.Method, trimURLPath(req.URL.Path), req.URL.Host longRunning := strconv.FormatBool(isLongRunningRequest(path)) servermetrics.HTTPRequests.WithLabelValues(method, path, host, serverType, longRunning).Inc() servermetrics.HTTPInflightRequests.WithLabelValues(method, path, host, serverType, longRunning).Inc() defer servermetrics.HTTPInflightRequests.WithLabelValues(method, path, host, serverType, longRunning).Dec() startTime := time.Now() defer servermetrics.HTTPRequestsDuration.WithLabelValues(method, path, host, serverType, longRunning).Observe(servermetrics.SinceInSeconds(startTime)) </ins> s.restfulCont.ServeHTTP(w, req) }


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node

assert.Equal(t, http.StatusOK, resp.StatusCode) } <ins> func TestTrimURLPath(t *testing.T) { tests := []struct { path, expected string }{ {"", ""}, {"//", ""}, {"/pods", "pods"}, {"pods", "pods"}, {"pods/", "pods"}, {"good/", "good"}, {"pods/probes", "pods"}, {"metrics", "metrics"}, {"metrics/resource", "metrics/resource"}, {"metrics/hello", "metrics/hello"}, } for _, test := range tests { assert.Equal(t, test.expected, trimURLPath(test.path), fmt.Sprintf("path is: %s", test.path)) } } </ins>


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When referencing init container's env value from its fields, kubelet complains What you expected to happen: Init container should be supported to get env values from its own fields, as to How to reproduce it (as minimally and precisely as possible): a pod with init container refering its fields value as env value kubectl create -f test-pod-1.yaml cannot be created pod events Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The err happens at this line , it seems that we should search for init containers as well

return &container, nil } } <ins> for _, container := range pod.Spec.InitContainers { if container.Name == containerName { return &container, nil } } </ins> return nil, fmt.Errorf("container %s not found", containerName) }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When referencing init container's env value from its fields, kubelet complains What you expected to happen: Init container should be supported to get env values from its own fields, as to How to reproduce it (as minimally and precisely as possible): a pod with init container refering its fields value as env value kubectl create -f test-pod-1.yaml cannot be created pod events Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The err happens at this line , it seems that we should search for init containers as well

pod:           getPod("foo", "", "", "10Mi", "100Mi"), expectedValue: "104857600", }, <ins> { fs: &v1.ResourceFieldSelector{ Resource: "limits.cpu", }, cName:         "init-foo", pod:           getPod("foo", "", "9", "", ""), expectedValue: "9", }, { fs: &v1.ResourceFieldSelector{ Resource: "requests.cpu", }, cName:         "init-foo", pod:           getPod("foo", "", "", "", ""), expectedValue: "0", }, { fs: &v1.ResourceFieldSelector{ Resource: "requests.cpu", }, cName:         "init-foo", pod:           getPod("foo", "8", "", "", ""), expectedValue: "8", }, { fs: &v1.ResourceFieldSelector{ Resource: "requests.cpu", }, cName:         "init-foo", pod:           getPod("foo", "100m", "", "", ""), expectedValue: "1", }, { fs: &v1.ResourceFieldSelector{ Resource: "requests.cpu", Divisor:  resource.MustParse("100m"), }, cName:         "init-foo", pod:           getPod("foo", "1200m", "", "", ""), expectedValue: "12", }, { fs: &v1.ResourceFieldSelector{ Resource: "requests.memory", }, cName:         "init-foo", pod:           getPod("foo", "", "", "100Mi", ""), expectedValue: "104857600", }, { fs: &v1.ResourceFieldSelector{ Resource: "requests.memory", Divisor:  resource.MustParse("1Mi"), }, cName:         "init-foo", pod:           getPod("foo", "", "", "100Mi", "1Gi"), expectedValue: "100", }, { fs: &v1.ResourceFieldSelector{ Resource: "limits.memory", }, cName:         "init-foo", pod:           getPod("foo", "", "", "10Mi", "100Mi"), expectedValue: "104857600", }, </ins> } as := assert.New(t) for idx, tc := range cases {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When referencing init container's env value from its fields, kubelet complains What you expected to happen: Init container should be supported to get env values from its own fields, as to How to reproduce it (as minimally and precisely as possible): a pod with init container refering its fields value as env value kubectl create -f test-pod-1.yaml cannot be created pod events Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The err happens at this line , it seems that we should search for init containers as well

Resources: resources, }, }, <ins> InitContainers: []v1.Container{ { Name:      "init-" + cname, Resources: resources, }, }, </ins> }, } }


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!

// BusyBoxImage is the image URI of BusyBox. BusyBoxImage = imageutils.GetE2EImage(imageutils.BusyBox) <ins> // AgnHostImage is the image URI of AgnHost AgnHostImage = imageutils.GetE2EImage(imageutils.Agnhost) </ins> // For parsing Kubectl version for version-skewed testing. gitVersionRegexp = regexp.MustCompile("GitVersion:"(v.+?)"")


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!

Containers: []v1.Container{ { Name:    "detector", <del> Image:   imageutils.GetE2EImage(imageutils.Agnhost), Command: []string{"/bin/sleep", "3600"}, </del> <ins> Image:   AgnHostImage, Command: []string{"pause"}, </ins> }, }, },


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!

} } <del> // PingCommand is the type to hold ping command. type PingCommand string const ( // IPv4PingCommand is a ping command for IPv4. IPv4PingCommand PingCommand = "ping" // IPv6PingCommand is a ping command for IPv6. IPv6PingCommand PingCommand = "ping6" ) </del> // CheckConnectivityToHost launches a pod to test connectivity to the specified // host. An error will be returned if the host is not reachable from the pod. // // An empty nodeName will use the schedule to choose where the pod is executed. <del> func CheckConnectivityToHost(f *Framework, nodeName, podName, host string, pingCmd PingCommand, timeout int) error { </del> <ins> func CheckConnectivityToHost(f *Framework, nodeName, podName, host string, port, timeout int) error { </ins> contName := fmt.Sprintf("%s-container", podName) command := []string{ <del> string(pingCmd), "-c", "3", // send 3 pings "-W", "2", // wait at most 2 seconds for a reply </del> <ins> "nc", "-vz", </ins> "-w", strconv.Itoa(timeout), host, <ins> strconv.Itoa(port), </ins> } pod := &v1.Pod{


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!

Containers: []v1.Container{ { Name:    contName, <del> Image:   BusyBoxImage, </del> <ins> Image:   AgnHostImage, </ins> Command: command, }, },


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!

Containers: []v1.Container{ { Name:  "agnhost", <del> Image: imageutils.GetE2EImage(imageutils.Agnhost), </del> <ins> Image: AgnHostImage, </ins> Args:  args, }, },


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!

}) ginkgo.It("should provide Internet connection for containers [Feature:Networking-IPv4]", func() { <del> ginkgo.By("Running container which tries to ping 8.8.8.8") </del> <ins> ginkgo.By("Running container which tries to connect to 8.8.8.8") </ins> framework.ExpectNoError( <del> framework.CheckConnectivityToHost(f, "", "ping-test", "8.8.8.8", framework.IPv4PingCommand, 30)) </del> <ins> framework.CheckConnectivityToHost(f, "", "connectivity-test", "8.8.8.8", 53, 30)) </ins> }) ginkgo.It("should provide Internet connection for containers [Feature:Networking-IPv6][Experimental]", func() { <del> ginkgo.By("Running container which tries to ping 2001:4860:4860::8888") </del> <ins> ginkgo.By("Running container which tries to connect to 2001:4860:4860::8888") </ins> framework.ExpectNoError( <del> framework.CheckConnectivityToHost(f, "", "ping-test", "2001:4860:4860::8888", framework.IPv6PingCommand, 30)) </del> <ins> framework.CheckConnectivityToHost(f, "", "connectivity-test", "2001:4860:4860::8888", 53, 30)) </ins> }) // First test because it has no dependencies on variables created later on.


Question:
What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

fs.BoolVar(&o.CleanupAndExit, "cleanup-iptables", o.CleanupAndExit, "If true cleanup iptables and ipvs rules and exit.") fs.MarkDeprecated("cleanup-iptables", "This flag is replaced by --cleanup.") fs.BoolVar(&o.CleanupAndExit, "cleanup", o.CleanupAndExit, "If true cleanup iptables and ipvs rules and exit.") <del> fs.BoolVar(&o.CleanupIPVS, "cleanup-ipvs", o.CleanupIPVS, "If true make kube-proxy cleanup ipvs rules before running.  Default is true") </del> <ins> fs.BoolVar(&o.CleanupIPVS, "cleanup-ipvs", o.CleanupIPVS, "If true and --cleanup is specified, kube-proxy will also flush IPVS rules, in addition to normal cleanup.") </ins> // All flags below here are deprecated and will eventually be removed.


Question:
What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

// NewProxyServer returns a new ProxyServer. func NewProxyServer(o *Options) (*ProxyServer, error) { <del> return newProxyServer(o.config, o.CleanupAndExit, o.CleanupIPVS, o.scheme, o.master) </del> <ins> return newProxyServer(o.config, o.CleanupAndExit, o.scheme, o.master) </ins> } func newProxyServer( config *proxyconfigapi.KubeProxyConfiguration, cleanupAndExit bool, <del> cleanupIPVS bool, </del> scheme *runtime.Scheme, master string) (*ProxyServer, error) {


Question:
What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

proxier = proxierIPTables serviceEventHandler = proxierIPTables endpointsEventHandler = proxierIPTables <del> // No turning back. Remove artifacts that might still exist from the userspace Proxier. klog.V(0).Info("Tearing down inactive rules.") // TODO this has side effects that should only happen when Run() is invoked. userspace.CleanupLeftovers(iptInterface) // IPVS Proxier will generate some iptables rules, need to clean them before switching to other proxy mode. // Besides, ipvs proxier will create some ipvs rules as well.  Because there is no way to tell if a given // ipvs rule is created by IPVS proxier or not.  Users should explicitly specify `--clean-ipvs=true` to flush // all ipvs rules when kube-proxy start up.  Users do this operation should be with caution. if canUseIPVS { ipvs.CleanupLeftovers(ipvsInterface, iptInterface, ipsetInterface, cleanupIPVS) } </del> } else if proxyMode == proxyModeIPVS { klog.V(0).Info("Using ipvs Proxier.") proxierIPVS, err := ipvs.NewProxier(


Question:
What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

proxier = proxierIPVS serviceEventHandler = proxierIPVS endpointsEventHandler = proxierIPVS <del> klog.V(0).Info("Tearing down inactive rules.") // TODO this has side effects that should only happen when Run() is invoked. userspace.CleanupLeftovers(iptInterface) iptables.CleanupLeftovers(iptInterface) </del> } else { klog.V(0).Info("Using userspace Proxier.") // This is a proxy.LoadBalancer which NewProxier needs but has methods we don't need for


Question:
What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

} serviceEventHandler = proxierUserspace proxier = proxierUserspace <del> // Remove artifacts from the iptables and ipvs Proxier, if not on Windows. klog.V(0).Info("Tearing down inactive rules.") // TODO this has side effects that should only happen when Run() is invoked. iptables.CleanupLeftovers(iptInterface) // IPVS Proxier will generate some iptables rules, need to clean them before switching to other proxy mode. // Besides, ipvs proxier will create some ipvs rules as well.  Because there is no way to tell if a given // ipvs rule is created by IPVS proxier or not.  Users should explicitly specify `--clean-ipvs=true` to flush // all ipvs rules when kube-proxy start up.  Users do this operation should be with caution. if canUseIPVS { ipvs.CleanupLeftovers(ipvsInterface, iptInterface, ipsetInterface, cleanupIPVS) } </del> } iptInterface.AddReloadFunc(proxier.Sync)


Question:
What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

} proxier = proxierUserspace serviceEventHandler = proxierUserspace <del> klog.V(0).Info("Tearing down pure-winkernel proxy rules.") winkernel.CleanupLeftovers() </del> } return &ProxyServer{


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: LoopbackHostPort code is calling the which depends on the machine on which this test is running. Especially in this particular testase it looks for the ipv6 ip address which does not exist because the machine is not configured for ipv6. Because of the LoopbackHostPort function bahaves based on the interfaces configured, test will fail if machine is not really configured with ipv6. Need to rewrite the testcase to handle such situation. What you expected to happen: Testcase should pass
/sig network
/cc
/triage unresolved I am a bot run by
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved

"testing" ) <del> func TestLoopbackHostPort(t *testing.T) { </del> <ins> func TestLoopbackHostPortIPv4(t *testing.T) { _, ipv6only, err := isIPv6LoopbackSupported() if err != nil { t.Fatalf("fail to enumerate network interface, %s", err) } if ipv6only { t.Fatalf("no ipv4 loopback interface") } </ins> host, port, err := LoopbackHostPort("1.2.3.4:443") if err != nil { t.Fatalf("unexpected error: %v", err)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: LoopbackHostPort code is calling the which depends on the machine on which this test is running. Especially in this particular testase it looks for the ipv6 ip address which does not exist because the machine is not configured for ipv6. Because of the LoopbackHostPort function bahaves based on the interfaces configured, test will fail if machine is not really configured with ipv6. Need to rewrite the testcase to handle such situation. What you expected to happen: Testcase should pass
/sig network
/cc
/triage unresolved I am a bot run by
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved

if port != "443" { t.Fatalf("expected 443 as port, got %q", port) } <ins> } func TestLoopbackHostPortIPv6(t *testing.T) { ipv6, _, err := isIPv6LoopbackSupported() if err != nil { t.Fatalf("fail to enumerate network interface, %s", err) } if !ipv6 { t.Fatalf("no ipv6 loopback interface") } </ins> <del> host, port, err = LoopbackHostPort("[ff06:0:0:0:0:0:0:c3]:443") </del> <ins> host, port, err := LoopbackHostPort("[ff06:0:0:0:0:0:0:c3]:443") </ins> if err != nil { t.Fatalf("unexpected error: %v", err) }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: LoopbackHostPort code is calling the which depends on the machine on which this test is running. Especially in this particular testase it looks for the ipv6 ip address which does not exist because the machine is not configured for ipv6. Because of the LoopbackHostPort function bahaves based on the interfaces configured, test will fail if machine is not really configured with ipv6. Need to rewrite the testcase to handle such situation. What you expected to happen: Testcase should pass
/sig network
/cc
/triage unresolved I am a bot run by
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved

if ip := net.ParseIP(host); ip == nil || !ip.IsLoopback() || ip.To4() != nil { t.Fatalf("expected IPv6 host to be loopback, got %q", host) } if port != "443" { t.Fatalf("expected 443 as port, got %q", port) } } <ins> func isIPv6LoopbackSupported() (ipv6 bool, ipv6only bool, err error) { addrs, err := net.InterfaceAddrs() if err != nil { return false, false, err } ipv4 := false for _, address := range addrs { ipnet, ok := address.(*net.IPNet) if !ok || !ipnet.IP.IsLoopback() { continue } if ipnet.IP.To4() == nil { ipv6 = true continue } ipv4 = true } ipv6only = ipv6 && !ipv4 return ipv6, ipv6only, nil } </ins>


Question:
What would you like to be : Change excludeCIDRs from []string to []*IPNet in Proxier and perform the parsing at construction of Proxier. Why is this needed: This was proposed by over comments in Pr This improves the performance of isIPInExcludeCIDRs since the cost of parsing becomes one time.
/sig network /kind cleanup
/remove-kind feature
/triage unresolved I am a bot run by
ping

syncPeriod    time.Duration minSyncPeriod time.Duration // Values are CIDR's to exclude when cleaning up IPVS rules. <del> excludeCIDRs []string </del> <ins> excludeCIDRs []*net.IPNet </ins> // Set to true to set sysctls arp_ignore and arp_announce strictARP      bool iptables       utiliptables.Interface