Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PR() fixed pull image error from a cross-subscription azure container registry by service principal, while for MSI, this issue is not fixed. Need to find a way how to fix this issue for MSI, related code is here: Here the difficulty is could not list all ACR since it's based on one subscriptionID: So if we could not get in a different subscription, how could we set the . What you expected to happen: How to reproduce it (as minimally and precisely as possible): error msg would be like following when pull a repository from an ACR in another subscription: Anything else we need to know?: Environment: Kubernetes version (use ): v.9 - v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
By further investigation, I think we should implement LazyProvider for azure acrProvider: Related code:
Update: It's hard to fix this since in the beginning we cannot list ACR in another subscription, will always list acr under current subscription, and without target loginServer address of ACR, we cannot get : And in func, we cannot get target repo, it's not passed on
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- is this still an issue? If not, let's it out.
it's still an issue, and currently it should be low priority, let's keep it
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} func (a *acrProvider) Provide(image string) credentialprovider.DockerConfig { <ins> klog.V(4).Infof("try to provide secret for image %s", image) </ins> cfg := credentialprovider.DockerConfig{} <del> ctx, cancel := getContextWithCancel() defer cancel() </del> if a.config.UseManagedIdentityExtension { <del> klog.V(4).Infof("listing registries") result, err := a.registryClient.List(ctx) if err != nil { klog.Errorf("Failed to list registries: %v", err) return cfg } for ix := range result { loginServer := getLoginServer(result[ix]) klog.V(2).Infof("loginServer: %s", loginServer) cred, err := getACRDockerEntryFromARMToken(a, loginServer) if err != nil { continue </del> <ins> if loginServer := parseACRLoginServerFromImage(image); loginServer == "" { klog.V(4).Infof("image(%s) is not from ACR, skip MSI authentication", image) } else { if cred, err := getACRDockerEntryFromARMToken(a, loginServer); err == nil { cfg[loginServer] = *cred </ins> } <del> cfg[loginServer] = *cred </del> } } else { // Add our entry for each of the supported container registry URLs


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PR() fixed pull image error from a cross-subscription azure container registry by service principal, while for MSI, this issue is not fixed. Need to find a way how to fix this issue for MSI, related code is here: Here the difficulty is could not list all ACR since it's based on one subscriptionID: So if we could not get in a different subscription, how could we set the . What you expected to happen: How to reproduce it (as minimally and precisely as possible): error msg would be like following when pull a repository from an ACR in another subscription: Anything else we need to know?: Environment: Kubernetes version (use ): v.9 - v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
By further investigation, I think we should implement LazyProvider for azure acrProvider: Related code:
Update: It's hard to fix this since in the beginning we cannot list ACR in another subscription, will always list acr under current subscription, and without target loginServer address of ACR, we cannot get : And in func, we cannot get target repo, it's not passed on
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- is this still an issue? If not, let's it out.
it's still an issue, and currently it should be low priority, let's keep it
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} func getACRDockerEntryFromARMToken(a *acrProvider, loginServer string) (*credentialprovider.DockerConfigEntry, error) { <ins> // Run EnsureFresh to make sure the token is valid and does not expire if err := a.servicePrincipalToken.EnsureFresh(); err != nil { klog.Errorf("Failed to ensure fresh service principal token: %v", err) return nil, err } </ins> armAccessToken := a.servicePrincipalToken.OAuthToken() klog.V(4).Infof("discovering auth redirects for: %s", loginServer)


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PR() fixed pull image error from a cross-subscription azure container registry by service principal, while for MSI, this issue is not fixed. Need to find a way how to fix this issue for MSI, related code is here: Here the difficulty is could not list all ACR since it's based on one subscriptionID: So if we could not get in a different subscription, how could we set the . What you expected to happen: How to reproduce it (as minimally and precisely as possible): error msg would be like following when pull a repository from an ACR in another subscription: Anything else we need to know?: Environment: Kubernetes version (use ): v.9 - v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
By further investigation, I think we should implement LazyProvider for azure acrProvider: Related code:
Update: It's hard to fix this since in the beginning we cannot list ACR in another subscription, will always list acr under current subscription, and without target loginServer address of ACR, we cannot get : And in func, we cannot get target repo, it's not passed on
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- is this still an issue? If not, let's it out.
it's still an issue, and currently it should be low priority, let's keep it
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

}, nil } <ins> // parseACRLoginServerFromImage takes image as parameter and returns login server of it. // Parameter `image` is expected in following format: foo.azurecr.io/bar/imageName:version // If the provided image is not an acr image, this function will return an empty string. func parseACRLoginServerFromImage(image string) string { match := acrRE.FindAllString(image, -1) if len(match) == 1 { return match[0] } return "" } </ins> func (a *acrProvider) LazyProvide(image string) *credentialprovider.DockerConfigEntry { return nil }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PR() fixed pull image error from a cross-subscription azure container registry by service principal, while for MSI, this issue is not fixed. Need to find a way how to fix this issue for MSI, related code is here: Here the difficulty is could not list all ACR since it's based on one subscriptionID: So if we could not get in a different subscription, how could we set the . What you expected to happen: How to reproduce it (as minimally and precisely as possible): error msg would be like following when pull a repository from an ACR in another subscription: Anything else we need to know?: Environment: Kubernetes version (use ): v.9 - v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
By further investigation, I think we should implement LazyProvider for azure acrProvider: Related code:
Update: It's hard to fix this since in the beginning we cannot list ACR in another subscription, will always list acr under current subscription, and without target loginServer address of ACR, we cannot get : And in func, we cannot get target repo, it's not passed on
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- is this still an issue? If not, let's it out.
it's still an issue, and currently it should be low priority, let's keep it
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} } } <ins> func TestParseACRLoginServerFromImage(t *testing.T) { tests := []struct { image    string expected string }{ { image:    "invalidImage", expected: "", }, { image:    "docker.io/library/busybox:latest", expected: "", }, { image:    "foo.azurecr.io/bar/image:version", expected: "foo.azurecr.io", }, { image:    "foo.azurecr.cn/bar/image:version", expected: "foo.azurecr.cn", }, { image:    "foo.azurecr.de/bar/image:version", expected: "foo.azurecr.de", }, { image:    "foo.azurecr.us/bar/image:version", expected: "foo.azurecr.us", }, } for _, test := range tests { if loginServer := parseACRLoginServerFromImage(test.image); loginServer != test.expected { t.Errorf("function parseACRLoginServerFromImage returns "%s" for image %s, expected "%s"", loginServer, test.image, test.expected) } } } </ins>


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

"github.com/golang/glog" "github.com/xanzy/go-cloudstack/cloudstack" "k8s.io/api/core/v1" <ins> cloudprovider "k8s.io/cloud-provider" </ins> ) type loadBalancer struct {


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

// GetLoadBalancerName retrieves the name of the LoadBalancer. func (cs *CSCloud) GetLoadBalancerName(ctx context.Context, clusterName string, service *v1.Service) string { <del> return cs.GetLoadBalancerName(ctx, clusterName, service) </del> <ins> return cloudprovider.DefaultLoadBalancerName(service) </ins> } // getLoadBalancer retrieves the IP address and ID and all the existing rules it can find.


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

"//pkg/api/legacyscheme:go_default_library", "//pkg/apis/core:go_default_library", "//pkg/capabilities:go_default_library", <del> "//pkg/client/chaosclient:go_default_library", </del> "//pkg/cloudprovider:go_default_library", "//pkg/cloudprovider/providers:go_default_library", "//pkg/credentialprovider:go_default_library",


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

"k8s.io/kubernetes/pkg/api/legacyscheme" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/capabilities" <del> "k8s.io/kubernetes/pkg/client/chaosclient" </del> "k8s.io/kubernetes/pkg/cloudprovider" "k8s.io/kubernetes/pkg/credentialprovider" "k8s.io/kubernetes/pkg/features"


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

clientConfig.QPS = float32(s.KubeAPIQPS) clientConfig.Burst = int(s.KubeAPIBurst) <del> addChaosToClientConfig(s, clientConfig) </del> return clientConfig, nil } <del> // addChaosToClientConfig injects random errors into client connections if configured. func addChaosToClientConfig(s *options.KubeletServer, config *restclient.Config) { if s.ChaosChance != 0.0 { config.WrapTransport = func(rt http.RoundTripper) http.RoundTripper { seed := chaosclient.NewSeed(1) // TODO: introduce a standard chaos package with more tunables - this is just a proof of concept // TODO: introduce random latency and stalls return chaosclient.NewChaosRoundTripper(rt, chaosclient.LogChaos, seed.P(s.ChaosChance, chaosclient.ErrSimulatedConnectionResetByPeer)) } } } </del> // RunKubelet is responsible for setting up and running a kubelet.  It is used in three different applications: //   1 Integration tests //   2 Kubelet binary


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

"//pkg/auth/authorizer/abac:all-srcs", "//pkg/auth/nodeidentifier:all-srcs", "//pkg/capabilities:all-srcs", <del> "//pkg/client/chaosclient:all-srcs", </del> "//pkg/client/clientset_generated/internalclientset:all-srcs", "//pkg/client/conditions:all-srcs", "//pkg/client/informers/informers_generated/internalversion:all-srcs",


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

<del> package(default_visibility = ["//visibility:public"]) load( "@io_bazel_rules_go//go:def.bzl", "go_library", "go_test", ) go_library( name = "go_default_library", srcs = ["chaosclient.go"], importpath = "k8s.io/kubernetes/pkg/client/chaosclient", deps = ["//staging/src/k8s.io/apimachinery/pkg/util/net:go_default_library"], ) go_test( name = "go_default_test", srcs = ["chaosclient_test.go"], embed = [":go_default_library"], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], ) </del>


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

<del> reviewers: - smarterclayton - liggitt - davidopp - eparis - resouer </del>


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

<del> /* Copyright 2015 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // Package chaosclient makes it easy to simulate network latency, misbehaving // servers, and random errors from servers. It is intended to stress test components // under failure conditions and expose weaknesses in the error handling logic // of the codebase. package chaosclient import ( "errors" "fmt" "log" "math/rand" "net/http" "reflect" "runtime" "k8s.io/apimachinery/pkg/util/net" ) // chaosrt provides the ability to perform simulations of HTTP client failures // under the Golang http.Transport interface. type chaosrt struct { rt     http.RoundTripper notify ChaosNotifier c      []Chaos } // Chaos intercepts requests to a remote HTTP endpoint and can inject arbitrary // failures. type Chaos interface { // Intercept should return true if the normal flow should be skipped, and the // return response and error used instead. Modifications to the request will // be ignored, but may be used to make decisions about types of failures. Intercept(req *http.Request) (bool, *http.Response, error) } // ChaosNotifier notifies another component that the ChaosRoundTripper has simulated // a failure. type ChaosNotifier interface { // OnChaos is invoked when a chaotic outcome was triggered. fn is the // source of Chaos and req was the outgoing request OnChaos(req *http.Request, c Chaos) } // ChaosFunc takes an http.Request and decides whether to alter the response. It // returns true if it wishes to mutate the response, with a http.Response or // error. type ChaosFunc func(req *http.Request) (bool, *http.Response, error) // Intercept calls the nested method `Intercept` func (fn ChaosFunc) Intercept(req *http.Request) (bool, *http.Response, error) { return fn.Intercept(req) } func (fn ChaosFunc) String() string { return runtime.FuncForPC(reflect.ValueOf(fn).Pointer()).Name() } // NewChaosRoundTripper creates an http.RoundTripper that will intercept requests // based on the provided Chaos functions. The notifier is invoked when a Chaos // Intercept is fired. func NewChaosRoundTripper(rt http.RoundTripper, notify ChaosNotifier, c ...Chaos) http.RoundTripper { return &chaosrt{rt, notify, c} } // RoundTrip gives each ChaosFunc an opportunity to intercept the request. The first // interceptor wins. func (rt *chaosrt) RoundTrip(req *http.Request) (*http.Response, error) { for _, c := range rt.c { if intercept, resp, err := c.Intercept(req); intercept { rt.notify.OnChaos(req, c) return resp, err } } return rt.rt.RoundTrip(req) } var _ = net.RoundTripperWrapper(&chaosrt{}) func (rt *chaosrt) WrappedRoundTripper() http.RoundTripper { return rt.rt } // Seed represents a consistent stream of chaos. type Seed struct { *rand.Rand } // NewSeed creates an object that assists in generating random chaotic events // based on a deterministic seed. func NewSeed(seed int64) Seed { return Seed{rand.New(rand.NewSource(seed))} } type pIntercept struct { Chaos s Seed p float64 } // P returns a ChaosFunc that fires with a probability of p (p between 0.0 // and 1.0 with 0.0 meaning never and 1.0 meaning always). func (s Seed) P(p float64, c Chaos) Chaos { return pIntercept{c, s, p} } // Intercept intercepts requests with the provided probability p. func (c pIntercept) Intercept(req *http.Request) (bool, *http.Response, error) { if c.s.Float64() < c.p { return c.Chaos.Intercept(req) } return false, nil, nil } func (c pIntercept) String() string { return fmt.Sprintf("P{%f %s}", c.p, c.Chaos) } // ErrSimulatedConnectionResetByPeer emulates the golang net error when a connection // is reset by a peer. // TODO: make this more accurate // TODO: add other error types // TODO: add a helper for returning multiple errors randomly. var ErrSimulatedConnectionResetByPeer = Error{errors.New("connection reset by peer")} // Error returns the nested error when C() is invoked. type Error struct { error } // Intercept returns the nested error func (e Error) Intercept(_ *http.Request) (bool, *http.Response, error) { return true, nil, e.error } // LogChaos is the default ChaosNotifier and writes a message to the Golang log. var LogChaos = ChaosNotifier(logChaos{}) type logChaos struct{} func (logChaos) OnChaos(req *http.Request, c Chaos) { log.Printf("Triggered chaotic behavior for %s %s: %v", req.Method, req.URL.String(), c) } </del>


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

<del> /* Copyright 2015 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package chaosclient import ( "net/http" "net/http/httptest" "net/url" "testing" ) type TestLogChaos struct { *testing.T } func (t TestLogChaos) OnChaos(req *http.Request, c Chaos) { t.Logf("CHAOS: chaotic behavior for %s %s: %v", req.Method, req.URL.String(), c) } func unwrapURLError(err error) error { if urlErr, ok := err.(*url.Error); ok && urlErr != nil { return urlErr.Err } return err } func TestChaos(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })) defer server.Close() client := http.Client{ Transport: NewChaosRoundTripper(http.DefaultTransport, TestLogChaos{t}, ErrSimulatedConnectionResetByPeer), } resp, err := client.Get(server.URL) if unwrapURLError(err) != ErrSimulatedConnectionResetByPeer.error { t.Fatalf("expected reset by peer: %v", err) } if resp != nil { t.Fatalf("expected no response object: %#v", resp) } } func TestPartialChaos(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })) defer server.Close() seed := NewSeed(1) client := http.Client{ Transport: NewChaosRoundTripper( http.DefaultTransport, TestLogChaos{t}, seed.P(0.5, ErrSimulatedConnectionResetByPeer), ), } success, fail := 0, 0 for { _, err := client.Get(server.URL) if err != nil { fail++ } else { success++ } if success > 1 && fail > 1 { break } } } </del>


Question:
/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?

k8s.io/kubernetes/pkg/apis/rbac/validation,erictune,0, k8s.io/kubernetes/pkg/apis/storage/validation,caesarxuchao,1, k8s.io/kubernetes/pkg/auth/authorizer/abac,liggitt,0, <del> k8s.io/kubernetes/pkg/client/chaosclient,deads2k,1, </del> k8s.io/kubernetes/pkg/client/legacylisters,jsafrane,1, k8s.io/kubernetes/pkg/client/listers/batch/internalversion,mqliang,0, k8s.io/kubernetes/pkg/client/listers/extensions/internalversion,eparis,1,


Question:
We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by

// must unmounted prior to detach. // If a volume with the name volumeName does not exist in the list of // attached volumes, an error is returned. <del> SetVolumeGloballyMounted(volumeName v1.UniqueVolumeName, globallyMounted bool) error </del> <ins> SetVolumeGloballyMounted(volumeName v1.UniqueVolumeName, globallyMounted bool, devicePath string) error </ins> // DeletePodFromVolume removes the given pod from the given volume in the // cache indicating the volume has been successfully unmounted from the pod.


Question:
We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by

} func (asw *actualStateOfWorld) MarkDeviceAsMounted( <del> volumeName v1.UniqueVolumeName) error { return asw.SetVolumeGloballyMounted(volumeName, true /* globallyMounted */) </del> <ins> volumeName v1.UniqueVolumeName, devicePath string) error { return asw.SetVolumeGloballyMounted(volumeName, true /* globallyMounted */, devicePath) </ins> } func (asw *actualStateOfWorld) MarkDeviceAsUnmounted( volumeName v1.UniqueVolumeName) error { <del> return asw.SetVolumeGloballyMounted(volumeName, false /* globallyMounted */) </del> <ins> return asw.SetVolumeGloballyMounted(volumeName, false /* globallyMounted */, "" /*devicePath*/) </ins> } // addVolume adds the given volume to the cache indicating the specified


Question:
We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by

} func (asw *actualStateOfWorld) SetVolumeGloballyMounted( <del> volumeName v1.UniqueVolumeName, globallyMounted bool) error { </del> <ins> volumeName v1.UniqueVolumeName, globallyMounted bool, devicePath string) error { </ins> asw.Lock() defer asw.Unlock()


Question:
We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by

} volumeObj.globallyMounted = globallyMounted <ins> volumeObj.devicePath = devicePath </ins> asw.attachedVolumes[volumeName] = volumeObj return nil }


Question:
We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by

} // Act <del> err = asw.MarkDeviceAsMounted(generatedVolumeName) </del> <ins> err = asw.MarkDeviceAsMounted(generatedVolumeName, devicePath) </ins> // Assert if err != nil {


Question:
We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by

continue } if volume.pluginIsAttachable { <del> err = rc.actualStateOfWorld.MarkDeviceAsMounted(volume.volumeName) </del> <ins> err = rc.actualStateOfWorld.MarkDeviceAsMounted(volume.volumeName, volume.devicePath) </ins> if err != nil { glog.Errorf("Could not mark device is mounted to actual state of world: %v", err) continue


Question:
We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by

MarkVolumeAsUnmounted(podName volumetypes.UniquePodName, volumeName v1.UniqueVolumeName) error // Marks the specified volume as having been globally mounted. <del> MarkDeviceAsMounted(volumeName v1.UniqueVolumeName) error </del> <ins> MarkDeviceAsMounted(volumeName v1.UniqueVolumeName, devicePath string) error </ins> // Marks the specified volume as having its global mount unmounted. MarkDeviceAsUnmounted(volumeName v1.UniqueVolumeName) error


Question:
We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by

// Update actual state of world to reflect volume is globally mounted markDeviceMountedErr := actualStateOfWorld.MarkDeviceAsMounted( <del> volumeToMount.VolumeName) </del> <ins> volumeToMount.VolumeName, devicePath) </ins> if markDeviceMountedErr != nil { // On failure, return error. Caller will log and retry. return volumeToMount.GenerateErrorDetailed("MountVolume.MarkDeviceAsMounted failed", markDeviceMountedErr)


Question:
We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by

// Update actual state of world to reflect volume is globally mounted markDeviceMappedErr := actualStateOfWorld.MarkDeviceAsMounted( <del> volumeToMount.VolumeName) </del> <ins> volumeToMount.VolumeName, devicePath) </ins> if markDeviceMappedErr != nil { // On failure, return error. Caller will log and retry. return volumeToMount.GenerateErrorDetailed("MapVolume.MarkDeviceAsMounted failed", markDeviceMappedErr)


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: CSI Plugin calls to get node ID information (see ). If that call is not implemented by driver, registration with kubelet will fail. What you expected to happen: Per CSI Spec, the deprecated call should be supported until CSI 1.0 is released (see ). Therefore if a by Kubernetes fails (because it is not implemented by the driver), we should fall back and try call instead. How to reproduce it (as minimally and precisely as possible): Use any CSI driver that implements but not -- says OpenSDS driver is currently hitting this, and may be used for repro. Anything else we need to know?: When CSI moves to 1.0 we can drop this fall back logic. If a fix is available within , we can include it, but 1.12 release should NOT block on this. /sig storage Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
please verify this is the issue you hit. Specifically see if you can find in your kubelet logs.
CC who volunteered to take a look
I don't see "error during CSI NodeGetInfo" in my kubelet logs. I don't see glog.Infof(log("Register new plugin with name: %s at endpoint: %s", pluginName, endpoint)) so it seems that RegisterPlugin is not called. I do see a message: "Plugin Watcher Start at /var/lib/kubelet/plugins". I don't know why RegisterPlugin is not called. I'll continue to check.
Here's an update. I implemented NodeGetInfo. I followed documentation about the Kubelet Plugin Watcher: https://kubernetes- to get my plugin registered. After that I don’t see the “driver name csi-opensdsplugin not found in the list of registred CSI drivers” any more. I see that now RegisterPlugin gets called and I see the following message in I0914 14:44:50. ] Register new plugin with name: csi-opensdsplugin at endpoint: /var/lib/kubelet/plugins/csi- So this confirmed that missing NodeGetInfo is the problem. I still have problems but I think that’s a config issue on my side. I the following as cluster role rule for attacher and node: apiGroups: [""] resources: ["csidrivers"] verbs: ["get", "list", "watch", "update"] However I’m still getting the following errors in the logs: I0914 16:12:31. ] Listing and watching v1alpha1.CSIDriver from E0914 16:12:31. ] Failed to list v1alpha1.CSIDriver: is forbidden: User "system:serviceaccount:kube-system:attachdetach-controller" cannot list resource "csidrivers" in API group "" at the cluster scope kube-controller- I0914 16:11:51. ] Listing and watching v1alpha1.CSIDriver from E0914 16:11:51. ] Failed to list v1alpha1.CSIDriver: is forbidden: User "system:node:127.0.0.1" cannot list resource "csidrivers" in API group "" at the cluster scope
Did more investigation on this. Here's an update. I set the following feature gate to true on my local cluster: alpha: CSICRDAutoInstall CSISkipAttach CSIPodInfo I got some new errors in and kube-controller- I0916 07:22:06. ] Listing and watching v1alpha1.CSIDriver from E0916 07:22:06. ] Failed to list v1alpha1.CSIDriver: the server could not find the requested resource (get ) I0916 07:25:24. ] Listing and watching v1alpha1.CSIDriver from E0916 07:25:24. ] Failed to list v1alpha1.CSIDriver: the server could not find the requested resource (get ) In addition, I found the following in kube-controller- E0916 07:22:01. ] failed to create CSIDrivers CRD: &v1beta1.CustomResourceDefinition{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.CustomResourceDefinitionSpec{Group:"", Version:"", Names:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, Scope:"", Validation:(v1beta1.CustomResourceValidation)(nil), Subresources:(v1beta1.CustomResourceSubresources)(nil), Versions:[]v1beta1.CustomResourceDefinitionVersion(nil), AdditionalPrinterColumns:[]v1beta1.CustomResourceColumnDefinition(nil)}, Status:v1beta1.CustomResourceDefinitionStatus{Conditions:[]v1beta1.CustomResourceDefinitionCondition(nil), AcceptedNames:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, StoredVersions:[]string(nil)}}, err: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:" is forbidden: User "system:serviceaccount:kube-system:attachdetach-controller" cannot create resource "customresourcedefinitions" in API group "" at the cluster scope", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc4212af560), Code:403}} So I made this change to add RBAC rule for customresourcedefinitions: After that, my CSI plugin is functional. I can create/attach/mount volumes successfully now.
/cc
(As we discussed offline) the latest code in master no longer has CSICRDAutoInstall, CSISkipAttach, CSIPodInfo flags. New CSI features are now flag gated under CSINodeInfo and CSIDriverRegistry flags (both alpha and disabled by default). KubeletPluginsWatcher must be enabled as well, but that's in beta now so it's enabled by default. Since this PR merged: , CRDs must be installed manually, and you can find an example of it here: (note they have to be created with --validation=false kubectl flag for the time being). This PR has not been cherry-picked to 1.12.0-rc.0 yet.
Thanks Tested latest code. If NodeGetInfo is implemented, we just need to make necessary changes when deploying driver-registrar sidecar container so KubeletPluginsWatcher can work. NodeGetInfo is needed for kubelet to register CSI plugins. Also tested with and without "CSINodeInfo=true,CSIDriverRegistry=true" and manually installed CRDs using when they are enabled.
I have the same problem when deploying my CSI plugin based on CSI v0.2.0 on Kubernetes v.1.12.0. Daemonset pods of CSI plugin cannot launch successfully. Kubelet logs:
Considering Kubernetes v1.12 is already out, and we planned to deprecate the old call in CSI 1.0/Kubernetes 1.13 anyway, we'll pull of the bandaid and deprecate it now. If you are running Kubernetes 1.13, make sure your CSI driver implements the .
looks like you might be running in to a different issue: Kubernetes v1.13 also enables Kubelet device plugin registration by default. Before upgrading to v1.13, ensure the CSI sidecar container for your CSI driver is configured to handle plugin registration (set the parameter on to expose a new unix domain socket to handle Kubelet Plugin Registration). From
Hi Because goes Beta and is enabled by default in Kubernetes v1.12. After disabling this feater gate, I can run CSI plugin based on CSI spec v0.2.0 in Kubernetes v1.12.2. When Kubelet enable , Kubelet would call gRPC call to register CSI plugin. gRPC call was first proposed at CSI spec v0.3.0. Therefore, user should disable feature gate when running CSI plugin based on CSI spec v0.2.0 in Kubernetes v1.12.
I have implement the NodeGetInfo call. But it still has this issue. My kubernetes version is 1.14.3.

* kube-apiserver: the `Priority` admission plugin is now enabled by default when using `--enable-admission-plugins`. If using `--admission-control` to fully specify the set of admission plugins, the `Priority` admission plugin should be added if using the `PodPriority` feature, which is enabled by default in 1.11. ([#65739](https://github.com/kubernetes/kubernetes/pull/65739), [@liggitt](https://github.com/liggitt)) * The `system-node-critical` and `system-cluster-critical` priority classes are now limited to the `kube-system` namespace by the `PodPriority` admission plugin. ([#65593](https://github.com/kubernetes/kubernetes/pull/65593), [@bsalamat](https://github.com/bsalamat)) * kubernetes-worker juju charm: Added support for setting the --enable-ssl-chain-completion option on the ingress proxy.  "action required": if your installation relies on supplying incomplete certificate chains and using OCSP to fill them in, you must set "ingress-ssl-chain-completion" to "true" in your juju configuration. ([#63845](https://github.com/kubernetes/kubernetes/pull/63845), [@paulgear](https://github.com/paulgear)) <ins> * In anticipation of CSI 1.0 in the next release, Kubernetes 1.12 calls the CSI `NodeGetInfo` RPC instead of `NodeGetId` RPC. Ensure your CSI Driver implements `NodeGetInfo(...)` before upgrading to 1.12. [@saad-ali](https://github.com/kubernetes/kubernetes/issues/68688) * Kubernetes 1.12 also enables [Kubelet device plugin registration](https://github.com/kubernetes/features/issues/595) feature by default. Before upgrading to 1.12, ensure the `driver-registrar` CSI sidecar container for your CSI driver is configured to handle plugin registration (set the `--kubelet-registration-path` parameter on `driver-registrar` to expose a new unix domain socket to handle Kubelet Plugin Registration). </ins> ### Other notable changes


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: After running (on master branch), a new file named left as a surprise. and the file is not matched by git ignore so it's tracked by git which is really disturbing. I tried to find out where it's created but failed. Any one has a context w/ it? Environment: OS (e.g. from /etc/os-release): Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASEX8664 x86_64
/sig instrumentation
the only piece of code in local-up-cluster that touches the file is
can't reproduce this. Two questions: Can you see this everytime? What is the SHA of the commit on master when you tried this? Can check with that too.
finally i realize that the cause is command is built receiving different flags between Linux and OSX. on Linux, you'll find that for linux sed the is explained as: '--in-place[=SUFFIX]' This option specifies that files are to be edited in-place. GNU 'sed' does this by creating a temporary file and sending output to this file rather than to the standard output.(1). so the is optional. but for OSX, it is: Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved. It is not recommended to give a zero- length extension when in-place editing files, as you risk corruption or partial content in situations where disk space is exhausted, etc. the difference is that on OSX, flag is requiring a variable , so when we're using on OSX in the script: the which is supposed to be received as a flag will be a suffix for the backup file. then the magic happened.will raise a fix for this. /assign
linking the difference between and

set -e source "${KUBE_ROOT}/hack/lib/init.sh" <ins> kube::util::ensure-gnu-sed </ins> function usage { echo "This script starts a local kube cluster. "


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: After running (on master branch), a new file named left as a surprise. and the file is not matched by git ignore so it's tracked by git which is really disturbing. I tried to find out where it's created but failed. Any one has a context w/ it? Environment: OS (e.g. from /etc/os-release): Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASEX8664 x86_64
/sig instrumentation
the only piece of code in local-up-cluster that touches the file is
can't reproduce this. Two questions: Can you see this everytime? What is the SHA of the commit on master when you tried this? Can check with that too.
finally i realize that the cause is command is built receiving different flags between Linux and OSX. on Linux, you'll find that for linux sed the is explained as: '--in-place[=SUFFIX]' This option specifies that files are to be edited in-place. GNU 'sed' does this by creating a temporary file and sending output to this file rather than to the standard output.(1). so the is optional. but for OSX, it is: Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved. It is not recommended to give a zero- length extension when in-place editing files, as you risk corruption or partial content in situations where disk space is exhausted, etc. the difference is that on OSX, flag is requiring a variable , so when we're using on OSX in the script: the which is supposed to be received as a flag will be a suffix for the backup file. then the magic happened.will raise a fix for this. /assign
linking the difference between and

#   foo: true #   bar: false for gate in $(echo ${FEATURE_GATES} | tr ',' ' '); do <del> echo $gate | sed -e 's/(.*)=(.*)/  1: 2/' </del> <ins> echo $gate | ${SED} -e 's/(.*)=(.*)/  1: 2/' </ins> done fi >>/tmp/kube-proxy.yaml


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: After running (on master branch), a new file named left as a surprise. and the file is not matched by git ignore so it's tracked by git which is really disturbing. I tried to find out where it's created but failed. Any one has a context w/ it? Environment: OS (e.g. from /etc/os-release): Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASEX8664 x86_64
/sig instrumentation
the only piece of code in local-up-cluster that touches the file is
can't reproduce this. Two questions: Can you see this everytime? What is the SHA of the commit on master when you tried this? Can check with that too.
finally i realize that the cause is command is built receiving different flags between Linux and OSX. on Linux, you'll find that for linux sed the is explained as: '--in-place[=SUFFIX]' This option specifies that files are to be edited in-place. GNU 'sed' does this by creating a temporary file and sending output to this file rather than to the standard output.(1). so the is optional. but for OSX, it is: Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved. It is not recommended to give a zero- length extension when in-place editing files, as you risk corruption or partial content in situations where disk space is exhausted, etc. the difference is that on OSX, flag is requiring a variable , so when we're using on OSX in the script: the which is supposed to be received as a flag will be a suffix for the backup file. then the magic happened.will raise a fix for this. /assign
linking the difference between and

function start_kubedns { if [[ "${ENABLE_CLUSTER_DNS}" = true ]]; then cp "${KUBE_ROOT}/cluster/addons/dns/kube-dns/kube-dns.yaml.in" kube-dns.yaml <del> sed -i -e "s/{{ pillar['dns_domain'] }}/${DNS_DOMAIN}/g" kube-dns.yaml sed -i -e "s/{{ pillar['dns_server'] }}/${DNS_SERVER_IP}/g" kube-dns.yaml </del> <ins> ${SED} -i -e "s/{{ pillar['dns_domain'] }}/${DNS_DOMAIN}/g" kube-dns.yaml ${SED} -i -e "s/{{ pillar['dns_server'] }}/${DNS_SERVER_IP}/g" kube-dns.yaml </ins> # TODO update to dns role once we have one. # use kubectl to create kubedns addon ${KUBECTL} --kubeconfig="${CERT_DIR}/admin.kubeconfig" --namespace=kube-system create -f kube-dns.yaml


Question:
/kind bug What happened: After upgrading 1.11.1 -1.12.0 , when a node gets terminated in AWS, the controller-manager doesn't delete the node in the API. This has lead to several side effect for us : pods don't get GC Prometheus alerts firing because 100 of nodes are in the NotReady state for a long period What you expected to happen: kube-controller-manager deletes the node when it stops reporting, after checking with the cloud provider. How to reproduce it (as minimally and precisely as possible): Looking at the code, I believe this is what introduced the behaviour change : - The node gets tainted with because it's in a Terminated state. But it doesn't get removed because it still "exists" in the cloud provider. The behaviour is probably correct for a shutdown node, but in AWS a terminated node can't be restarted so there's no good reason to keep it there ? Environment: Kubernetes version : 1.2.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS (sorry for the all the sigs, not sure who it belongs to) /sig aws /sig cloud-provider /sig cluster-lifecycle
as I see it should delete node if instance is in terminated state have you tested that change in that row? I do not see what could be the issue there. Or could it be that the is not in place for all binaries in 1.12.0?
I might be wrong, but I think that part of the code never gets executed once the node is tainted :
good finding. So we should remove terminated state from this row. Then it does not add shutdown taint if node is terminated - instead it will delete node. also this should return false.. because if it returns true node is not deleted
I had the same issue. After I scale down my worker node instance number from 3 to 1. One node is stuck in status and never got deleted. However, one interesting thing is one of the other node is removed successfully. do you see similar behavior too?
- can we make sure this gets cherry picked for 1.12 releases ? I think we already missed the cut for 1.12.2

} if len(instances) == 0 { glog.Warningf("the instance %s does not exist anymore", providerID) <del> return true, nil </del> <ins> // returns false, because otherwise node is not deleted from cluster // false means that it will continue to check InstanceExistsByProviderID return false, nil </ins> } if len(instances) > 1 { return false, fmt.Errorf("multiple instances found for instance: %s", instanceID)


Question:
/kind bug What happened: After upgrading 1.11.1 -1.12.0 , when a node gets terminated in AWS, the controller-manager doesn't delete the node in the API. This has lead to several side effect for us : pods don't get GC Prometheus alerts firing because 100 of nodes are in the NotReady state for a long period What you expected to happen: kube-controller-manager deletes the node when it stops reporting, after checking with the cloud provider. How to reproduce it (as minimally and precisely as possible): Looking at the code, I believe this is what introduced the behaviour change : - The node gets tainted with because it's in a Terminated state. But it doesn't get removed because it still "exists" in the cloud provider. The behaviour is probably correct for a shutdown node, but in AWS a terminated node can't be restarted so there's no good reason to keep it there ? Environment: Kubernetes version : 1.2.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS (sorry for the all the sigs, not sure who it belongs to) /sig aws /sig cloud-provider /sig cluster-lifecycle
as I see it should delete node if instance is in terminated state have you tested that change in that row? I do not see what could be the issue there. Or could it be that the is not in place for all binaries in 1.12.0?
I might be wrong, but I think that part of the code never gets executed once the node is tainted :
good finding. So we should remove terminated state from this row. Then it does not add shutdown taint if node is terminated - instead it will delete node. also this should return false.. because if it returns true node is not deleted
I had the same issue. After I scale down my worker node instance number from 3 to 1. One node is stuck in status and never got deleted. However, one interesting thing is one of the other node is removed successfully. do you see similar behavior too?
- can we make sure this gets cherry picked for 1.12 releases ? I think we already missed the cut for 1.12.2

if instance.State != nil { state := aws.StringValue(instance.State.Name) // valid state for detaching volumes <del> if state == ec2.InstanceStateNameStopped || state == ec2.InstanceStateNameTerminated { </del> <ins> if state == ec2.InstanceStateNameStopped { </ins> return true, nil } }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: What happened: After upgrade of Kubernetes 1.11.3 to 1.12.1 via kubeadm in bare-metal setup I see these errors in kubelet logs: What you expected to happen: Since this is a bare-metal setup I expect no messages concerning Azure cloud How to reproduce it (as minimally and precisely as possible): Do an upgrade from 1.11.3 to 1.12.1 Anything else we need to know?: kubelet-1.12.1- was downloaded from Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Oracle Linux Server 7.4 Kernel (e.g. ): 62 Install tools: Others:
Lines from kubelet log which say that no cloud provider was specified (that is a bare-metal setup): ..
/sig azure
have the same issue related to the azure.. also a bare-metal cluster.. these messages are flooding my centralized logging elasticsearch (though they aren't really problematic more than being an annoyance). i also upgraded from 1.11.3 to 1.12.1 on a 3-master HA set up. i also noted (perhaps worthy of a separate issue) that the two of my three control-planes that are showing the azure error also previously had been having cert authority errors from the api-server container, when having used kubeadm upgrade.. (on the second two control plane systems, with the issue) (and on the first control without the issue) (and i needed to kubeadm reset and then kubeadm init all over again with 1.12.1 on the 2nd two control-plane nodes).. the first control-plane node that didn't need to do a reset/init didn't have the azure errors (nor the cert errors). so i kind of suspect it might not be an kubeadm upgrade issue but a general issue going from 1.11.3 to 1.12.1 ?
Same problems +1
Same problem on brand new bm 1.12.1 cluster built with kubeadm and latest 1alpha3 config based off of customized kubeadm config print-default example
/assign
Same problems +1
Oct 9 11:11:32 localhost kubelet: E1009 11:11:32. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:11:42 localhost kubelet: E1009 11:11:42. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:11:52 localhost kubelet: E1009 11:11:52. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:02 localhost kubelet: E1009 11:12:02. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:12 localhost kubelet: E1009 11:12:12. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:23 localhost kubelet: E1009 11:12:23. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:33 localhost kubelet: E1009 11:12:33. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:43 localhost kubelet: E1009 11:12:43. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:53 localhost kubelet: E1009 11:12:53. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:13:03 localhost kubelet: E1009 11:13:03. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1
how often did you hit this error log: I am planning to make this log as , is that ok for you? Usually V(2) is the default info(see ) log for k8s on azure I am not sure what the default info log by kubeadmin installation.
It is logged every 10 seconds. I disagree. At least previous incarnation of GetVolumeLimits in version 1.11.3 : would have logged an error "No cloudprovider present". Which I didn't see in kubelet log. In fact, I didn't see any message in kubelet logs with 'azure' substring at all. I think it means that AzureDataDisk plugin was not instantiated at all in 1.11.3. So the question is: why azuredd is instantiated now in 1.12.1 in bare-metal setup (that is without cloud provider)? Do you mind checking this idea? (Besides your method of lowering message priority won't work as I said this is a bare-metal setup where kubelet systemd unit file has no --v option at all )
That's because feature reaches beta in v1.12, in v1.11.3, even the code is there, it won't be invoked since feature is still alpha in v1.11
And in v1.12, aws, gce, and azure disk plugins are all loaded, take aws ebs for example, it will return error directly: And logging as V(4): I suppose V(4) logs won't display in logs. Let me make the same change as aws and gce.
also tries to address this error.
The same mistake, I also encountered NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 172.16.16.13 Ready <none5m27s v1.12.1 172.16.16.13 <noneCentOS Linux 7 (Core) 4.18.12- docker://18.6.0 failed to get azure cloud in GetVolumeLimits, 172.16.16.13 failed to get azure cloud in GetVolumeLimits, 172.16.16.13 failed to get azure cloud in GetVolumeLimits, 172.16.16.13
I have cherry picked the fix to v1.12
I understand that the feature Node-specific Volume Limits has now reached beta in 1.12, but for those of us running Kubernetes on-premises, this log error is pretty useless. In fact, I would even want to disable this plugin alltogether, since it will never be used on-prem. Is there a way to stop the plugin from loading, or in the very least make sure the log does not appear? Right now, it is polluting the system log file quite a lot.
I second that. That was the key message of my bug report. That is we must not even instantiate cloud-specific plugins if no cloud provider is configured.
I'm having exactly the same problem, doing an installation from scratch on DigitalOcean
+1
+1
Hi I have the same issue. It would be really nice to know how to disable this error message that floods the logs for no reason.
For those using logstash it was sufficient to just temporarily add this to your syslog filter: if [message] =~ "" { drop { } }
After a fresh installation of Raspbian and kubernetes, same issue is occurring in my 4 node cluster. Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/arm"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/arm"} flood of this: epsilon kubelet[2412]: E1024 12:25:58. 2412 ] failed to get azure cloud in GetVolumeLimits, epsilon
disable in controller plane should work: "--feature-gates=AttachVolumeLimit=false"
I have tried "--feature-gates=AttachVolumeLimit=false" and it's not working. I still see messages kubelet[]: E1024 22:51:02. ] failed to get azure cloud in GetVolumeLimits,
I have the same problem here.., ] failed to get azure cloud in GetVolumeLimits,
+1 same case here, also tried with --feature-gates=AttachVolumeLimit=false
Will this be disabled by default in the next release?
It's already fixed in v1.12.2 and v1.13.0. do you know how to disable in kubelet in v1.12? Would add in kubelet command arguments work?
I haven't updated to 1.12.2 yet (still on 1.12.1) but just to confirm, disabling that feature works. For those that are also running on CentOS and/or RHEL, just to the following: Edit file on your master server (control plane), and add the following: If you already have extra arguments in there, just add this one at the end of the line. Then restart kubelet on your master server:
Yes that will work.
--feature-gates=AttachVolumeLimit property doesn't exist in the documentation
Update tho 1.12.2 resolve for me Em qui, 1 de nov de 2018 9:22 PM, Abhilash Pallerlamudi < escreveu:
add follow line to "your kubelet.service" file. like this: [root ~]# systemctl daemon-reload [root ~]# sudo systemctl restart kubelet then it's ok
So to get rid of a bug we should disable an entire feature? Says it is supported by Azure-Disk. Why isn't node reporting the limit? It is getting instance metadata okay as all my nodes are tagged with SKU. Being that SKU determines attach limit as well, I do not understand why this cannot be made working? I am on 1.14 in Azure on Ubuntu.
how do you deploy k8s on Azure? To get volume limit, kubelet on agent node needs azure identity, e.g. service principal, usually it's under file
I deployed with kubespray. I have cloudconfig set as such: KUBELETCLOUDPROVIDER="--cloud-provider=azure --cloud-config=/etc/kubernetes/cloudconfig" Service principle is set in cloudconfig file.
I triple checked and location was misspelled on some of the nodes for the cloud provider. Thanks!
the get vm size list call depends on location, so now it works?
Yes it does. Thanks!

// hoping external CCM or admin can set it. Returning // default values from here will mean, no one can // override them. <del> glog.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) return volumeLimits, nil </del> <ins> return nil, fmt.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) </ins> } instances, ok := az.Instances()


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "[sig-network] Network [It] should set TCP CLOSEWAIT timeout" requires the test target OS supports /proc/net/nfconntrack, but Ubuntu 16.04 doesn't support it. Then the test fails today like the following: What you expected to happen: The test succeeded or was skipped. How to reproduce it (as minimally and precisely as possible): Just kick Anything else we need to know?: The existence of /proc/net/nfconntrack depends on the Linux kernel config CONFIGNFCONNTRACKPROCFS, but that is disabled on Ubuntu 16.04. In addition, the test is skipped if the node number is less than 2. So I guess that is one of reasons this issue was not present until today. Environment: Kubernetes version (use ): v1.11.3 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): 4.4.0-134-generic Install tools: kubeadm Others:
/sig network
Some kube-proxy functionalities depends on conntrack, is it possible to run a conformance cluster without having conntrack enable?

LogSSHResult(result) if result.Code != 0 || err != nil { <del> return nil, fmt.Errorf("failed running %q: %v (exit code %d)", cmd, err, result.Code) </del> <ins> return nil, fmt.Errorf("failed running %q: %v (exit code %d, stderr %v)", cmd, err, result.Code, result.Stderr) </ins> } return &result, nil


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "[sig-network] Network [It] should set TCP CLOSEWAIT timeout" requires the test target OS supports /proc/net/nfconntrack, but Ubuntu 16.04 doesn't support it. Then the test fails today like the following: What you expected to happen: The test succeeded or was skipped. How to reproduce it (as minimally and precisely as possible): Just kick Anything else we need to know?: The existence of /proc/net/nfconntrack depends on the Linux kernel config CONFIGNFCONNTRACKPROCFS, but that is disabled on Ubuntu 16.04. In addition, the test is skipped if the node number is less than 2. So I guess that is one of reasons this issue was not present until today. Environment: Kubernetes version (use ): v1.11.3 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): 4.4.0-134-generic Install tools: kubeadm Others:
/sig network
Some kube-proxy functionalities depends on conntrack, is it possible to run a conformance cluster without having conntrack enable?

zero := int64(0) <ins> // Some distributions (Ubuntu 16.04 etc.) don't support the proc file. _, err := framework.IssueSSHCommandWithResult( "ls /proc/net/nf_conntrack", framework.TestContext.Provider, clientNodeInfo.node) if err != nil && strings.Contains(err.Error(), "No such file or directory") { framework.Skipf("The node %s does not support /proc/net/nf_conntrack", clientNodeInfo.name) } framework.ExpectNoError(err) </ins> clientPodSpec := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "e2e-net-client",


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the for a CRDs Established condition is always null What you expected to happen: it should have a timestamp How to reproduce it (as minimally and precisely as possible): create a CRD and then examine its yaml/json. For example: Anything else we need to know?: Environment: community master
/assign
/area custom-resource
/sig api-machinery /area custom-resources
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
edit: actually, that issue is about kubectl display and clock skew. this issue is about the condition timestamp updating, which still requires correction
Closing this issue. <detailsIn response to : duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now), which means that this PR must merged by then. Do we still target this issue to be fixed for 1.16?
yes, will take a look at it tomorrow /assign

// SetCRDCondition sets the status condition. It either overwrites the existing one or creates a new one. func SetCRDCondition(crd *CustomResourceDefinition, newCondition CustomResourceDefinitionCondition) { <ins> newCondition.LastTransitionTime = metav1.NewTime(time.Now()) </ins> existingCondition := FindCRDCondition(crd, newCondition.Type) if existingCondition == nil { <del> newCondition.LastTransitionTime = metav1.NewTime(time.Now()) </del> crd.Status.Conditions = append(crd.Status.Conditions, newCondition) return } <del> if existingCondition.Status != newCondition.Status { existingCondition.Status = newCondition.Status </del> <ins> if existingCondition.Status != newCondition.Status || existingCondition.LastTransitionTime.IsZero() { </ins> existingCondition.LastTransitionTime = newCondition.LastTransitionTime } <ins> existingCondition.Status = newCondition.Status </ins> existingCondition.Reason = newCondition.Reason existingCondition.Message = newCondition.Message }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the for a CRDs Established condition is always null What you expected to happen: it should have a timestamp How to reproduce it (as minimally and precisely as possible): create a CRD and then examine its yaml/json. For example: Anything else we need to know?: Environment: community master
/assign
/area custom-resource
/sig api-machinery /area custom-resources
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
edit: actually, that issue is about kubectl display and clock skew. this issue is about the condition timestamp updating, which still requires correction
Closing this issue. <detailsIn response to : duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now), which means that this PR must merged by then. Do we still target this issue to be fixed for 1.16?
yes, will take a look at it tomorrow /assign

}, }, }, <ins> { name: "set new condition which doesn't have lastTransitionTime set", crdCondition: []CustomResourceDefinitionCondition{ { Type:               Established, Status:             ConditionTrue, Reason:             "Accepted", Message:            "the initial names have been accepted", LastTransitionTime: metav1.Date(2018, 1, 1, 0, 0, 0, 0, time.UTC), }, }, newCondition: CustomResourceDefinitionCondition{ Type:    Established, Status:  ConditionFalse, Reason:  "NotAccepted", Message: "Not accepted", }, expectedcrdCondition: []CustomResourceDefinitionCondition{ { Type:               Established, Status:             ConditionFalse, Reason:             "NotAccepted", Message:            "Not accepted", LastTransitionTime: metav1.Date(2018, 1, 2, 0, 0, 0, 0, time.UTC), }, }, }, { name: "append new condition which doesn't have lastTransitionTime set", crdCondition: []CustomResourceDefinitionCondition{ { Type:               Established, Status:             ConditionTrue, Reason:             "Accepted", Message:            "the initial names have been accepted", LastTransitionTime: metav1.Date(2018, 1, 1, 0, 0, 0, 0, time.UTC), }, }, newCondition: CustomResourceDefinitionCondition{ Type:    Terminating, Status:  ConditionFalse, Reason:  "NeverEstablished", Message: "resource was never established", }, expectedcrdCondition: []CustomResourceDefinitionCondition{ { Type:               Established, Status:             ConditionTrue, Reason:             "Accepted", Message:            "the initial names have been accepted", LastTransitionTime: metav1.Date(2018, 1, 1, 0, 0, 0, 0, time.UTC), }, { Type:               Terminating, Status:             ConditionFalse, Reason:             "NeverEstablished", Message:            "resource was never established", LastTransitionTime: metav1.Date(2018, 2, 1, 0, 0, 0, 0, time.UTC), }, }, }, </ins> } for _, tc := range tests { crd := generateCRDwithCondition(tc.crdCondition)


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the for a CRDs Established condition is always null What you expected to happen: it should have a timestamp How to reproduce it (as minimally and precisely as possible): create a CRD and then examine its yaml/json. For example: Anything else we need to know?: Environment: community master
/assign
/area custom-resource
/sig api-machinery /area custom-resources
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
edit: actually, that issue is about kubectl display and clock skew. this issue is about the condition timestamp updating, which still requires correction
Closing this issue. <detailsIn response to : duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now), which means that this PR must merged by then. Do we still target this issue to be fixed for 1.16?
yes, will take a look at it tomorrow /assign

if !IsCRDConditionEquivalent(&tc.expectedcrdCondition[i], &crd.Status.Conditions[i]) { t.Errorf("%v expected %v, got %v", tc.name, tc.expectedcrdCondition, crd.Status.Conditions) } <ins> if crd.Status.Conditions[i].LastTransitionTime.IsZero() { t.Errorf("%q lastTransitionTime should not be null: %v", tc.name, i, crd.Status.Conditions) } </ins> } } }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Here's an example from an e2e test: What you expected to happen: leader election success logs to be at a higher log level.
Leader election library is not a part of the scheduler. The scheduler is only a user of the library. It is located in client-go. It is currently . I am not sure if it is a good idea to reduce verbosity further. The log messages seem to be excessive in the scheduler logs when the scheduler is idle in our tests.
I think this is a common problem for all components, not just for scheduler.
In that case, we should consider reducing its verbosity to 5 or 6.
For some reason I thought that this was something new but I went back to 1.9 tests and checked the logs and I also see it printed every 2 seconds. So maybe this is fine for test environments.
This is certainly not new, but I am still in favor of lowering it verbosity unless there is a good reason for having it in tests by default.
I'm ok with changing the verbosity but the code now lives in the api-machinery land.
I'd like to pick this up. Could I do that?
The log level seems correct following these guidelines: I am not sold on the value of glog's infinite log levels feature, however I could be convinced that our logging guidelines need a TRACE level.
Thank you for your response. I'm on your side, How about ?

le.maybeReportTransition() desc := le.config.Lock.Describe() if err == nil { <del> glog.V(4).Infof("successfully renewed lease %v", desc) </del> <ins> glog.V(5).Infof("successfully renewed lease %v", desc) </ins> return } le.config.Lock.RecordEvent("stopped leading")


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: In PR we tightened the --provider parameter checking of the binary such that it only accepted known providers. This turned out to be problematic () because people had started to depend on the previous behavior, which was to accept anything and merely warn about it. For the sake of giving people time to adapt, PR reverted to the older behavior. What you expected to happen: In 1.14 we want to allow only known providers. While at it, the error handling for unknown providers should be enhanced (print proper error message instead of calling Failf, which panics). /assign /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
kubeadm was one of the tools which got broken by the tightening of the parameter checking. Does kubeadm still rely on the acceptance of unknown provider parameters or can the current warning be turned into an error now? /remove-lifecycle stale
yes the kubernetes-anywhere deployer is still bound to provider=kubernetes-anywhere (unknown) in test infra jobs. we went back and forth on this topic, but if you wish to replace this support for unknown deployers i can switch those jobs to . LMK.
given one release has passed with the deprecated message on the CLI, it's probably safe to remove this functionality now. :)
"Lubomir I. Ivanov" <: Yes, please do. Once that is complete, I'll turn the warning into an error.
done:

providers[name] = factory } <ins> // GetProviders returns the names of all currently registered providers. func GetProviders() []string { mutex.Lock() defer mutex.Unlock() var providerNames []string for name := range providers { providerNames = append(providerNames, name) } return providerNames } </ins> func init() { // "local" or "skeleton" can always be used. RegisterProvider("local", func() (ProviderInterface, error) {


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: In PR we tightened the --provider parameter checking of the binary such that it only accepted known providers. This turned out to be problematic () because people had started to depend on the previous behavior, which was to accept anything and merely warn about it. For the sake of giving people time to adapt, PR reverted to the older behavior. What you expected to happen: In 1.14 we want to allow only known providers. While at it, the error handling for unknown providers should be enhanced (print proper error message instead of calling Failf, which panics). /assign /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
kubeadm was one of the tools which got broken by the tightening of the parameter checking. Does kubeadm still rely on the acceptance of unknown provider parameters or can the current warning be turned into an error now? /remove-lifecycle stale
yes the kubernetes-anywhere deployer is still bound to provider=kubernetes-anywhere (unknown) in test infra jobs. we went back and forth on this topic, but if you wish to replace this support for unknown deployers i can switch those jobs to . LMK.
given one release has passed with the deprecated message on the CLI, it's probably safe to remove this functionality now. :)
"Lubomir I. Ivanov" <: Yes, please do. Once that is complete, I'll turn the warning into an error.
done:

RegisterProvider("skeleton", func() (ProviderInterface, error) { return NullProvider{}, nil }) <del> // The empty string also works, but triggers a warning. RegisterProvider("", func() (ProviderInterface, error) { Logf("The --provider flag is not set.  Treating as a conformance test.  Some tests may not be run.") return NullProvider{}, nil }) </del> <ins> // The empty string used to be accepted in the past, but is not // a valid value anymore. </ins> } // SetupProviderConfig validates the chosen provider and creates


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: In PR we tightened the --provider parameter checking of the binary such that it only accepted known providers. This turned out to be problematic () because people had started to depend on the previous behavior, which was to accept anything and merely warn about it. For the sake of giving people time to adapt, PR reverted to the older behavior. What you expected to happen: In 1.14 we want to allow only known providers. While at it, the error handling for unknown providers should be enhanced (print proper error message instead of calling Failf, which panics). /assign /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
kubeadm was one of the tools which got broken by the tightening of the parameter checking. Does kubeadm still rely on the acceptance of unknown provider parameters or can the current warning be turned into an error now? /remove-lifecycle stale
yes the kubernetes-anywhere deployer is still bound to provider=kubernetes-anywhere (unknown) in test infra jobs. we went back and forth on this topic, but if you wish to replace this support for unknown deployers i can switch those jobs to . LMK.
given one release has passed with the deprecated message on the CLI, it's probably safe to remove this functionality now. :)
"Lubomir I. Ivanov" <: Yes, please do. Once that is complete, I'll turn the warning into an error.
done:

"fmt" "io/ioutil" "os" <ins> "sort" "strings" </ins> "time" "github.com/onsi/ginkgo/config"