<!-- NEW RELEASE NOTES ENTRY --> <del> # v1.19.0 </del> <ins> # v1.20.0 </ins> [문서](https://docs.k8s.io) <del> ## v1.19.0 다운로드 </del> <ins> ## v1.20.0 다운로드 </ins> 파일명 | sha512 해시 -------- | ----------- <del> [kubernetes.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes.tar.gz) | `448b941e973a519a500eb24786f6deb7eebd0e1ecb034941e382790ff69dfc2838715a222cfc53bea7b75f2c6aedc7425eded4aad69bf88773393155c737f9c0` [kubernetes-src.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-src.tar.gz) | `47d253e6eb1f6da730f4f3885e205e6bfde88ffe66d92915465108c9eaf8e3c5d1ef515f8bf804a726db057433ecd25008ecdef624ee68ad9c103d1c7a615aad` </del> <ins> [kubernetes.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes.tar.gz) | `ebfe49552bbda02807034488967b3b62bf9e3e507d56245e298c4c19090387136572c1fca789e772a5e8a19535531d01dcedb61980e42ca7b0461d3864df2c14` [kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-src.tar.gz) | `bcbd67ed0bb77840828c08c6118ad0c9bf2bcda16763afaafd8731fd6ce735be654feef61e554bcc34c77c65b02a25dae565adc5e1dc49a2daaa0d115bf1efe6` </ins> ### 클라이언트 바이너리 파일명 | sha512 해시 -------- | ----------- <del> [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-client-darwin-amd64.tar.gz) | `7093a34298297e46bcd1ccb77a9c83ca93b8ccb63ce2099d3d8cd8911ccc384470ac202644843406f031c505a8960d247350a740d683d8910ca70a0b58791a1b` [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-client-linux-386.tar.gz) | `891569cff7906732a42b20b86d1bf20a9fe873f87b106e717a5c0f80728b5823c2a00c7ccea7ec368382509f095735089ddd582190bc51dcbbcef6b8ebdbd5cc` [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-client-linux-amd64.tar.gz) | `1590d4357136a71a70172e32820c4a68430d1b94cf0ac941ea17695fbe0c5440d13e26e24a2e9ebdd360c231d4cd16ffffbbe5b577c898c78f7ebdc1d8d00fa3` [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-client-linux-arm.tar.gz) | `bc0fb19fb6af47f591adc64b5a36d3dffcadc35fdfd77a4a222e037dbd2ee53fafb84f13c4e307910cfa36b3a46704063b42a14ceaad902755ec14c492ccd51d` [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-client-linux-arm64.tar.gz) | `6ff47f4fdfb3b5f2bfe18fd792fe9bfc747f06bf52de062ee803cda87ac4a98868d8e1211742e32dd443a4bdb770018bbdde704dae6abfc6d80c02bdfb4e0311` [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-client-linux-ppc64le.tar.gz) | `d8816518adc3a7fc00f996f23ff84e6782a3ebbba7ef37ba44def47b0e6506fefeeaf37d0e197cecf0deb5bd1a8f9dd1ba82af6c29a6b9d21b8e62af965b6b81` [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-client-linux-s390x.tar.gz) | `662fd4618f2b747d2b0951454b9148399f6cd25d3ca7c40457b6e02cb20df979138cad8cccd18fc8b265d9426c90828d3f0b2a6b40d9cd1a1bdc17219e35ed33` [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-client-windows-386.tar.gz) | `d90cb92eb33ecbfab7a0e3a2da60ab10fc59132e4bc9abe0a1461a13222b5016704a7cfe0bf9bcf5d4ec55f505ffbbf53162dfe570e8f210e3f68b0d3a6bf7e3` [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-client-windows-amd64.tar.gz) | `6ec32a8a62b69363a524c4f8db765ff4bd16ea7e5b0eb04aa5a667f8653eda18c357a97513d9e12f0ba1612516acb150deffb6e3608633c62b97a15b6efa7cc0` </del> <ins> [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-darwin-amd64.tar.gz) | `3609f6483f4244676162232b3294d7a2dc40ae5bdd86a842a05aa768f5223b8f50e1d6420fd8afb2d0ce19de06e1d38e5e5b10154ba0cb71a74233e6dc94d5a0` [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-386.tar.gz) | `e06c08016a08137d39804383fdc33a40bb2567aa77d88a5c3fd5b9d93f5b581c635b2c4faaa718ed3bb2d120cb14fe91649ed4469ba72c3a3dda1e343db545ed` [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-amd64.tar.gz) | `081472833601aa4fa78e79239f67833aa4efcb4efe714426cd01d4ddf6f36fbf304ef7e1f5373bff0fdff44a845f7560165c093c108bd359b5ab4189f36b1f2f` [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-arm.tar.gz) | `037f84a2f29fe62d266cab38ac5600d058cce12cbc4851bcf062fafba796c1fbe23a0c2939cd15784854ca7cd92383e5b96a11474fc71fb614b47dbf98a477d9` [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-arm64.tar.gz) | `275727e1796791ca3cbe52aaa713a2660404eab6209466fdc1cfa8559c9b361fe55c64c6bcecbdeba536b6d56213ddf726e58adc60f959b6f77e4017834c5622` [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-ppc64le.tar.gz) | `7a9965293029e9fcdb2b7387467f022d2026953b8461e6c84182abf35c28b7822d2389a6d8e4d8e532d2ea5d5d67c6fee5fb6c351363cb44c599dc8800649b04` [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-linux-s390x.tar.gz) | `85fc449ce1980f5f030cc32e8c8e2198c1cc91a448e04b15d27debc3ca56aa85d283f44b4f4e5fed26ac96904cc12808fa3e9af3d8bf823fc928befb9950d6f5` [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-windows-386.tar.gz) | `4c0a27dba1077aaee943e0eb7a787239dd697e1d968e78d1933c1e60b02d5d233d58541d5beec59807a4ffe3351d5152359e11da120bf64cacb3ee29fbc242e6` [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-client-windows-amd64.tar.gz) | `29336faf7c596539b8329afbbdceeddc843162501de4afee44a40616278fa1f284d8fc48c241fc7d52c65dab70f76280cc33cec419c8c5dbc2625d9175534af8` </ins> ### 서버 바이너리 파일명 | sha512 해시 -------- | ----------- <del> [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-server-linux-amd64.tar.gz) | `7c268bd58e67d3c5016f3fcc9f4b6d2da7558af5a2c708ff3baf767b39e847e3d35d4fd2fa0f640bedbfb09a445036cafbe2f04357a88dada405cfc2ded76972` [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-server-linux-arm.tar.gz) | `fcbf8d9004f1cd244a82b685abaf81f9638c3cc1373d78e705050042cfa6a004f8eed92f4721539dcd169c55b662d10416af19cff7537a8dfef802dc41b4088b` [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-server-linux-arm64.tar.gz) | `e21f54a35ff29e919e98fe81758f654ea735983d5a9d08dab9484598b116843830a86ceb5cf0a23d27b7f9aba77e5f0aa107c171a0837ba781d508ebbea76f55` [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-server-linux-ppc64le.tar.gz) | `c7014c782683f8f612c7805654b632aab4c5dce895ee8f9ef24360616e24240ce59ddf3cf27c3170df5450d8fe14fbca3fb7cddfc9b74ae37943081f0fa4b6b3` [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-server-linux-s390x.tar.gz) | `3ac2d6b273e5b650f63260aae164fc6781ad5760f63cca911f5db9652c4bf32e7e7b25728987befc6dfda89c5c56969681b75f12b17141527d4e1d12f3d41f3c` </del> <ins> [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-amd64.tar.gz) | `fb56486a55dbf7dbacb53b1aaa690bae18d33d244c72a1e2dc95fb0fcce45108c44ba79f8fa04f12383801c46813dc33d2d0eb2203035cdce1078871595e446e` [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-arm.tar.gz) | `735ed9993071fe35b292bf06930ee3c0f889e3c7edb983195b1c8e4d7113047c12c0f8281fe71879fc2fcd871e1ee587f03b695a03c8512c873abad444997a19` [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-arm64.tar.gz) | `ffab155531d5a9b82487ee1abf4f6ef49626ea58b2de340656a762e46cf3e0f470bdbe7821210901fe1114224957c44c1d9cc1e32efb5ee24e51fe63990785b2` [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-ppc64le.tar.gz) | `9d5730d35c4ddfb4c5483173629fe55df35d1e535d96f02459468220ac2c97dc01b995f577432a6e4d1548b6edbfdc90828dc9c1f7cf7464481af6ae10aaf118` [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-server-linux-s390x.tar.gz) | `6e4c165306940e8b99dd6e590f8542e31aed23d2c7a6808af0357fa425cec1a57016dd66169cf2a95f8eb8ef70e1f29e2d500533aae889e2e3d9290d04ab8721` </ins> ### 노드 바이너리 파일명 | sha512 해시 -------- | ----------- <del> [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-node-linux-amd64.tar.gz) | `d5e21432a4ab019f00cd1a52bbbdb00feb3db2ce96b41a58b1ee27d8847c485f5d0efe13036fd1155469d6d15f5873a5a892ecc0198f1bae1bf5b586a0129e75` [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-node-linux-arm.tar.gz) | `bd57adf060813b06be2b33439d6f60d13630c0251ef96ba473274073200ea118f5622ec31ed714cc57bd9da410655e958a7700a5742ae7e4b6406ab12fbf21f3` [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-node-linux-arm64.tar.gz) | `3ee70abc0a5cbf1ef5dde0d27055f4d17084585c36a2cf41e3fd925d206df0b583f50dc1c118472f198788b65b2c447aa40ad41646b88791659d2dfb69b3890b` [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-node-linux-ppc64le.tar.gz) | `0f4368f229c082b2a75e7089a259e487d60b20bc8edf650dd7ca0fe23c51632397c2ef24c9c6cef078c95fce70d9229a5b4ff682c34f65a44bc4be3329c8ccde` [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-node-linux-s390x.tar.gz) | `8f0b6839fc0ad51300221fa7f32134f8c687073715cc0839f7aacb21a075c66dab113369707d03e9e0e53be62ca2e1bdf04d4b26cff805ae9c7a5a4b864e3eae` [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.19.0/kubernetes-node-windows-amd64.tar.gz) | `587651158c9999e64e06186ef2e65fe14d46ffdae28c5d8ee6261193bfe4967717f997ebe13857fa1893bbf492e1cc1f816bce86a94c6df9b7a0264848391397` </del> <ins> [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-amd64.tar.gz) | `3e6c90561dd1c27fa1dff6953c503251c36001f7e0f8eff3ec918c74ae2d9aa25917d8ac87d5b4224b8229f620b1830442e6dce3b2a497043f8497eee3705696` [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-arm.tar.gz) | `26db385d9ae9a97a1051a638e7e3de22c4bbff389d5a419fe40d5893f9e4fa85c8b60a2bd1d370fd381b60c3ca33c5d72d4767c90898caa9dbd4df6bd116a247` [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-arm64.tar.gz) | `5b8b63f617e248432b7eb913285a8ef8ba028255216332c05db949666c3f9e9cb9f4c393bbd68d00369bda77abf9bfa2da254a5c9fe0d79ffdad855a77a9d8ed` [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-ppc64le.tar.gz) | `60da7715996b4865e390640525d6e98593ba3cd45c6caeea763aa5355a7f989926da54f58cc5f657f614c8134f97cd3894b899f8b467d100dca48bc22dd4ff63` [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-linux-s390x.tar.gz) | `9407dc55412bd04633f84fcefe3a1074f3eaa772a7cb9302242b8768d6189b75d37677a959f91130e8ad9dc590f9ba8408ba6700a0ceff6827315226dd5ee1e6` [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0/kubernetes-node-windows-amd64.tar.gz) | `9d4261af343cc330e6359582f80dbd6efb57d41f882747a94bbf47b4f93292d43dd19a86214d4944d268941622dfbc96847585e6fec15fddc4dbd93d17015fa8` </ins> <del> ## v1.18.0 이후 체인지로그 </del> <ins> ## v1.19.0 이후 변경로그(Changelog) </ins> <del> ## 새로운 소식 (주요 테마) </del> <ins> ## 새로운 소식(주요 테마) </ins> <del> ### 사용 중단 경고 </del> <ins> ### Dockershim 사용 중단(deprecation) </ins> <del> SIG API Machinery는 `kubectl` 사용자 및 API 사용자에게 표시되는 [사용 중단된 API 사용 시 경고](https://kubernetes.io/docs/reference/using-api/deprecation-policy/#rest-resources-aka-api-objects)와 클러스터 관리자에게 표시되는 메트릭을 구현하였다. 사용 중단된 API에 대한 요청은 대상 제거 릴리스 및 대체 API를 포함하는 경고와 함께 반환된다. 경고는 [어드미션 웹훅](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admissionreview-response-warning)에서도 반환될 수 있으며, [사용자 정의 리소스의 사용 중단된 버전](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#version-deprecation)에 대해 지정된다. </del> <ins> Docker as an underlying runtime is being deprecated. Docker-produced images will continue to work in your cluster with all runtimes, as they always have. The Kubernetes community [has written a blog post about this in detail](https://blog.k8s.io/2020/12/02/dont-panic-kubernetes-and-docker/) with [a dedicated FAQ page for it](https://blog.k8s.io/2020/12/02/dockershim-faq/). </ins> <del> ### 영구적인 베타(beta) 회피하기 </del> <ins> ### client-go를 위한 외부 자격증명(credential) 제공자 </ins> <del> 쿠버네티스 1.20에서부터, SIG Architecture는 9개월 이내에 모든 REST API들을 베타에서 안정 버전으로 전환하는 새로운 정책을 구현한다. 새로운 정책의 이면에 있는 아이디어는 기능이 오랫동안 베타 상태로 유지되는 것을 피하기 위함이다. 일단 새로운 API가 베타 단계에 진입하면, 9개월 동안 다음 중 하나를 수행하게 된다. </del> <ins> The client-go credential plugins can now be passed in the current cluster information via the `KUBERNETES_EXEC_INFO` environment variable. Learn more about this on [client-go credential plugins documentation](https://docs.k8s.io/reference/access-authn-authz/authentication/#client-go-credential-plugins/). </ins> <del> - GA에 도달하여, 베타 지원 중단 혹은 - 새로운 베타 버전을 보유 _(및 이전 베타 지원 중단)_. </del> <ins> ### 기능 게이트(feature gate)를 통해 크론잡(CronJob) 컨트롤러 v2 활성화 가능 </ins> <del> REST API가 9개월의 카운트 다운이 끝나면, 다음 쿠버네티스 릴리스에서 해당 API 버전이 사용 중단된다. 더 자세한 내용은 [쿠버네티스 블로그](https://kubernetes.io/blog/2020/08/21/moving-forward-from-beta/)에서 확인할 수 있다. </del> <ins> An alternative implementation of `CronJob` controller is now available as an alpha feature in this release, which has experimental performance improvement by using informers instead of polling. While this will be the default behavior in the future, you can [try them in this release through a feature gate](https://docs.k8s.io/concepts/workloads/controllers/cron-jobs/). </ins> <del> ### 워크로드 및 노드 디버깅을 위한 확장된 CLI 지원 </del> <ins> ### PID 제한(PID Limits)이 안정 기능(General Availability)으로 전환 </ins> <del> SIG CLI는 `kubectl` 로 디버깅을 확장하여, 다음의 두 가지 새로운 디버깅 워크플로(workflows)를 지원한다. 복사본을 생성하여 워크로드를 디버깅하고, 호스트 네임스페이스에 컨테이너를 생성하여 노드를 디버깅한다. 이는 아래의 작업을 할 때 유용하다. - 임시(Ephemeral) 컨테이너가 활성화되지 않은 클러스터에서 디버그 컨테이너 삽입 - busybox와 같은 이미지나 `sleep 1d` 와 같은 명령어를 변경하여 쉽게 디버깅할 수 있도록, 손상된 컨테이너를 수정하여 `kubectl exec` 를 사용할 수 있는 시간 확보 - 노드의 호스트 파일시스템에서 구성 파일 검사 </del> <ins> PID Limits features are now generally available on both `SupportNodePidsLimit` (node-to-pod PID isolation) and `SupportPodPidsLimit` (ability to limit PIDs per pod), after being enabled-by-default in beta stage for a year. </ins> <del> 이러한 새로운 워크플로에는 신규 클러스터 기능이 필요하지 않으므로, `kubectl alpha debug` 를 통해 기존 클러스터로 테스트할 수 있다. `kubectl` 을 사용한 디버깅에 대해 사용자의 의견을 기대한다. 이슈를 열거나, [#sig-cli](https://kubernetes.slack.com/messages/sig-cli)를 방문하거나 기능 개선에 대한 이슈 [#1441](https://features.k8s.io/1441)에 코멘트를 남겨서 SIG CLI와 연락할 수 있다. </del> <ins> ### API 우선순위 및 공정성(API Priority and Fairness)이 베타 단계로 전환 </ins> <del> ### 구조화된 로깅 </del> <ins> Initially introduced in 1.18, Kubernetes 1.20 now enables API Priority and Fairness (APF) by default. This allows `kube-apiserver` to [categorize incoming requests by priority levels](https://docs.k8s.io/concepts/cluster-administration/flow-control/). </ins> <del> SIG Instrumentation은 로그 메시지의 구조와 쿠버네티스 오브젝트에 대한 참조를 표준화했다. 구조화된 로깅을 사용하면 로그를 더 쉽게 구문 분석, 처리, 저장, 질의 및 분석할 수 있다. klog 라이브러리의 새 메소드는 로그 메시지 구조를 강제 적용한다. </del> <ins> ### IPv4/IPv6이 작동 </ins> <del> ### 엔드포인트슬라이스(EndpointSlices)가 기본적으로 활성화 </del> <ins> IPv4/IPv6 dual-stack has been reimplemented for 1.20 to support dual-stack Services, based on user and community feedback. If your cluster has dual-stack enabled, you can create Services which can use IPv4, IPv6, or both, and you can change this setting for existing Services. Details are available in updated [IPv4/IPv6 dual-stack docs](https://docs.k8s.io/concepts/services-networking/dual-stack/), which cover the nuanced array of options. </ins> <del> 엔드포인트슬라이스는 엔드포인트(EndPoint) API에 대한 스케일 및 확장 가능한 대안을 제공하는 훌륭한 신규 API이다. 엔드포인트슬라이스는 서비스를 지원하는 파드에 대한 IP 주소, 포트, 준비성 게이트(readiness gate) 상태 및 토폴로지 정보를 추적한다. </del> <ins> We expect this implementation to progress from alpha to beta and GA in coming releases, so we’re eager to have you comment about your dual-stack experiences in [#k8s-dual-stack](https://kubernetes.slack.com/messages/k8s-dual-stack) or in [enhancements #563](https://features.k8s.io/563). </ins> <del> 쿠버네티스 1.19에서 이 기능은, 엔드포인트 대신 엔드포인트슬라이스에서 읽는 kube-proxy와 함께 기본적으로 활성화된다. 이는 대부분 눈에 띄지 않는 변경이지만, 대규모 클러스터에서는 눈에 띄게 확장성이 향상된다. 또한 토폴로지 인식 라우팅과 같은 향후 쿠버네티스 릴리스에서 중요한 신규 기능을 사용할 수 있다. </del> <ins> ### go1.15.5 </ins> <del> ### 인그레스(ingress)를 안정 기능(General Availability)으로 전환 </del> <ins> go1.15.5 has been integrated to Kubernets project as of this release, [including other infrastructure related updates on this effort](https://github.com/kubernetes/kubernetes/pull/95776). </ins> <del> SIG Network는 널리 사용되고 있는 [인그레스 API](https://kubernetes.io/ko/docs/concepts/services-networking/ingress/)를 쿠버네티스 1.19의 안정 기능으로 전환했다. 이 변경은 쿠버네티스 기여자들의 수 년간의 노력을 인정하고, 쿠버네티스의 향후 네트워킹 API에 대한 추가 작업을 위한 기반을 마련한다. </del> <ins> ### CSI 볼륨 스냅샷(CSI Volume Snapshot)이 안정 기능으로 전환 </ins> <del> ### seccomp를 안정 기능으로 전환 </del> <ins> CSI Volume Snapshot moves to GA in the 1.20 release. This feature provides a standard way to trigger volume snapshot operations in Kubernetes and allows Kubernetes users to incorporate snapshot operations in a portable manner on any Kubernetes environment regardless of supporting underlying storage providers. Additionally, these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced, enterprise grade, storage administration features for Kubernetes: including application or cluster level backup solutions. Note that snapshot support will require Kubernetes distributors to bundle the Snapshot controller, Snapshot CRDs, and validation webhook. In addition, a CSI driver supporting the snapshot functionality must also be deployed on the cluster. </ins> <del> 쿠버네티스에 대한 seccomp(보안 컴퓨팅 모드) 지원이 안정 기능(GA)로 전환되었다. 이 기능은 파드(모든 컨테이너에 적용) 또는 단일 컨테이너에 대한 시스템 호출을 제한하여 워크로드 보안을 강화하는 데 사용될 수 있다. </del> <ins> ### 비재귀적 볼륨 소유(Non-recursive Volume Ownership (FSGroup))가 베타 단계로 전환 </ins> <del> 기술적으로 이것은 일급 객체(first class)인 `seccompProfile` 필드가 파드 및 컨테이너 `securityContext` 오브젝트에 추가되었음을 의미한다. </del> <ins> By default, the `fsgroup` setting, if specified, recursively updates permissions for every file in a volume on every mount. This can make mount, and pod startup, very slow if the volume has many files. This setting enables a pod to specify a `PodFSGroupChangePolicy` that indicates that volume ownership and permissions will be changed only when permission and ownership of the root directory does not match with expected permissions on the volume. </ins> <del> ```yaml securityContext: seccompProfile: type: RuntimeDefault|Localhost|Unconfined # 셋 중 하나를 선택 localhostProfile: my-profiles/profile-allow.json # type == Localhost 일때만 필요 ``` </del> <ins> ### FSGroup를 위한 CSIDriver 정책이 베타 단계로 전환 </ins> <del> `seccomp.security.alpha.kubernetes.io/pod` 및 `container.seccomp.security.alpha.kubernetes.io/...` 어노테이션에 대한 지원은 이제 사용 중단되었으며, 쿠버네티스 v1.22.0에서 제거된다. 현재 자동 버전 차이(skew)에 대한 처리는 새로운 필드를 어노테이션으로 변환하며 그 반대의 경우도 동일하게 동작한다. 즉, 클러스터의 기존 워크로드를 변환하는데 별도의 조치가 필요하지 않다. </del> <ins> The FSGroup's CSIDriver Policy is now beta in 1.20. This allows CSIDrivers to explicitly indicate if they want Kubernetes to manage permissions and ownership for their volumes via `fsgroup`. </ins> <del> 새로운 [Kubernetes.io의 문서 페이지][seccomp-docs]에서 seccomp로 컨테이너 시스템 호출을 제한하는 방법에 대한 자세한 정보를 찾을 수 있다. </del> <ins> ### CSI 드라이버의 보안성 향상(알파) </ins> <del> [seccomp-docs]: https://kubernetes.io/docs/tutorials/clusters/seccomp/ </del> <ins> In 1.20, we introduce a new alpha feature `CSIServiceAccountToken`. This feature allows CSI drivers to impersonate the pods that they mount the volumes for. This improves the security posture in the mounting process where the volumes are ACL’ed on the pods’ service account without handing out unnecessary permissions to the CSI drivers’ service account. This feature is especially important for secret-handling CSI drivers, such as the secrets-store-csi-driver. Since these tokens can be rotated and short-lived, this feature also provides a knob for CSI drivers to receive `NodePublishVolume` RPC calls periodically with the new token. This knob is also useful when volumes are short-lived, e.g. certificates. ### 그레이스풀 노드 종료(Graceful Node Shutdown) 기능 소개(알파) </ins> <del> ### 운영용 이미지가 커뮤니티 제어로 이동 </del> <ins> The `GracefulNodeShutdown` feature is now in Alpha. This allows kubelet to be aware of node system shutdowns, enabling graceful termination of pods during a system shutdown. This feature can be [enabled through feature gate](https://docs.k8s.io/concepts/architecture/nodes/#graceful-node-shutdown). </ins> <del> 쿠버네티스 v1.19부터 쿠버네티스 컨테이너 이미지는 `{asia,eu,us}.gcr.io/k8s-artifacts-prod` 에 있는 커뮤니티 제어 기반의 스토리지 버킷에 저장된다. `k8s.gcr.io` 가상 도메인(vanity domain)이 새 버킷으로 업데이트되었다. 이는 커뮤니티 제어 하에 운영 아티팩트(production artifacts)를 가져온다. </del> <ins> ### 런타임 로그 관리(sanitation) </ins> <del> ### KubeSchedulerConfiguration를 베타로 전환 </del> <ins> Logs can now be configured to use runtime protection from leaking sensitive data. [Details for this experimental feature is available in documentation](https://docs.k8s.io/concepts/cluster-administration/system-logs/#log-sanitization). </ins> <del> SIG Scheduling은 `KubeSchedulerConfiguration` 를 베타로 전환했다. [KubeSchedulerConfiguration](https://kubernetes.io/docs/reference/scheduling/config) 기능을 사용하면 kube-scheduler의 알고리즘 및 기타 설정을 조정할 수 있다. 남은 구성을 다시 작성하지 않고도, 선택한 일정 단계에서 특정 기능(플러그인에 포함된)을 쉽게 활성화하거나 비활성화할 수 있다. 또한 단일 kube-scheduler 인스턴스는 프로파일이라고 불리는 다양한 구성을 제공한다. 파드는 `.spec.schedulerName` 필드를 통해 스케줄링하려는 프로파일을 선택할 수 있다. </del> <ins> ### 파드 리소스 메트릭 </ins> <del> ### CSI 마이그레이션 - AzureDisk 및 vSphere (베타) 인-트리(In-tree) 볼륨 플러그인 및 모든 클라우드 공급자의 의존성이 쿠버네티스 코어 밖으로 이동된다. CSI 마이그레이션 기능을 사용하면 모든 볼륨 작업을 해당 CSI 드라이버로 라우팅하여 코드가 제거된 경우에도 레거시 API를 사용하는 기존 볼륨이 계속 동작할 수 있다. 이 기능의 AzureDisk 및 vSphere 구현이 베타로 승격되었다. </del> <ins> On-demand metrics calculation is now available through `/metrics/resources`. [When enabled]( https://docs.k8s.io/concepts/cluster-administration/system-metrics#kube-scheduler-metrics), the endpoint will report the requested resources and the desired limits of all running pods. </ins> <del> ### 스토리지 용량 추적 </del> <ins> ### `RootCAConfigMap` 소개 </ins> <del> 전통적으로 쿠버네티스 스케줄러는 추가 퍼시스턴트 스토리지가 클러스터의 모든 곳에서 사용할 수 있고 그 용량이 무한하다고 가정했다. 토폴로지 제약 사항은 첫 번째 요점을 다루었지만, 남은 스토리지 용량이 새 파드를 시작하기에 충분하지 않을 수 있다는 점을 고려하지 않은 채로 파드 스케줄링이 수행되었다. 새로운 알파(alpha) 기능인 [스토리지 용량 추적](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1472-storage-capacity-tracking)은 CSI 드라이버에 API를 추가하여, 스토리지 용량을 전달하고 파드의 노드를 선택할 때 쿠버네티스 스케줄러에서 해당 정보를 사용한다. 이 기능은 용량이 더 제한된 로컬 볼륨 및 기타 볼륨 유형에 대한 동적 프로비저닝을 지원하기 위한 디딤돌 역할이다. </del> <ins> `RootCAConfigMap` graduates to Beta, seperating from `BoundServiceAccountTokenVolume`. The `kube-root-ca.crt` ConfigMap is now available to every namespace, by default. It contains the Certificate Authority bundle for verify kube-apiserver connections. </ins> <del> ### CSI 볼륨 상태 모니터링 </del> <ins> ### `kubectl debug` 이 베타 단계로 전환 </ins> <del> CSI 상태 모니터링의 알파 버전은 쿠버네티스 1.19와 함께 릴리스된다. 이 기능을 사용하면 CSI 드라이버가 기본 스토리지 시스템의 비정상적인 볼륨 조건을 쿠버네티스와 공유하여 PVC 또는 파드의 이벤트를 전달할 수 있다. 이 기능은 쿠버네티스 개별 볼륨 상태 문제를 프로그래밍 방식으로 감지하고 해결하기 위한 디딤돌 역할이다. </del> <ins> `kubectl alpha debug` graduates from alpha to beta in 1.20, becoming `kubectl debug`. `kubectl debug` provides support for common debugging workflows directly from kubectl. Troubleshooting scenarios supported in this release of `kubectl` include: Troubleshoot workloads that crash on startup by creating a copy of the pod that uses a different container image or command. Troubleshoot distroless containers by adding a new container with debugging tools, either in a new copy of the pod or using an ephemeral container. (Ephemeral containers are an alpha feature that are not enabled by default.) Troubleshoot on a node by creating a container running in the host namespaces and with access to the host’s filesystem. Note that as a new builtin command, `kubectl debug` takes priority over any `kubectl` plugin named “debug”. You will need to rename the affected plugin. Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug` instead of `kubectl alpha debug`! For more information about kubectl debug, see Debugging Running Pods on the Kubernetes website, kubectl help debug, or reach out to SIG CLI by visiting #sig-cli or commenting on [enhancement #1441](https://features.k8s.io/1441). </ins> <del> ### 일반적인 임시 볼륨 </del> <ins> ### kubeadm에서 사용 중단된 플래그 삭제 </ins> <del> 쿠버네티스는 라이프사이클이 파드에 연결되어 있고 스크래치 공간(예: 기본 제공되는 "empty dir" 볼륨 타입)으로 사용되거나 일부 데이터를 파드에 로드(예: 기본 제공되는 컨피그맵(ConfigMap) 및 시크릿(Secret) 볼륨 유형 또는 "CSI 인라인 볼륨")할 수 있는 볼륨 플러그인을 제공한다. 새로운 [일반적인 임시 볼륨](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1698-generic-ephemeral-volumes) 알파 기능은 동적 프로비저닝을 지원하는 기존 스토리지 드라이버를 볼륨의 라이프사이클이 포함된 파드에 바인딩된 임시 볼륨으로 사용한다. - 예를 들어 퍼시스턴트 메모리나 해당 노드의 별도 로컬 디스크와 같은, 루트 디스크와는 다른 스크래치 스토리지를 제공하는데 사용 - 볼륨 프로비저닝을 위한 모든 스토리지클래스(StorageClass) 파라미터 지원 - 스토리지 용량 추적, 스냅샷 및 복원, 볼륨 크기 조정과 같이 퍼시스턴트볼륨클레임(PersistentVolumeClaim)에서 지원하는 모든 기능을 지원 </del> <ins> `kubeadm` applies a number of deprecations and removals of deprecated features in this release. More details are available in the Urgent Upgrade Notes and Kind / Deprecation sections. </ins> <del> ### 변경할 수 없는(immutable) 시크릿 및 컨피그맵 (베타) </del> <ins> ### 파드의 호스트네임을 FQDN으로 사용하는 것이 베타 단계로 전환 </ins> <del> 시크릿 및 컨피그맵 볼륨은 immutable로 표시될 수 있으므로, 클러스터에 시크릿 및 컨피그맵 볼륨이 많은 경우 API 서버의 로드가 크게 줄어든다. 자세한 내용은 [컨피그맵](https://kubernetes.io/ko/docs/concepts/configuration/configmap/) 및 [시크릿](https://kubernetes.io/ko/docs/concepts/configuration/secret/)을 참고한다. </del> <ins> Previously introduced in 1.19 behind a feature gate, `SetHostnameAsFQDN` is now enabled by default. More details on this behavior is available in [documentation for DNS for Services and Pods](https://docs.k8s.io/concepts/services-networking/dns-pod-service/#pod-sethostnameasfqdn-field) </ins> <del> ### 윈도우에 대한 CSI 프록시 </del> <ins> ### `TokenRequest` / `TokenRequestProjection` 이 안정 기능으로 전환 </ins> <del> 윈도우용 CSI 프록시는 1.19 릴리스와 함께 베타로 승격되었다. 이 CSI 프록시를 사용하면 윈도우의 컨테이너가 권한 있는 스토리지 작업을 수행할 수 있도록 CSI 드라이버를 윈도우에서 실행할 수 있다. 베타에서 윈도우용 CSI 프록시는 직접 연결된 디스크 및 SMB를 사용하는 스토리지 드라이버를 지원한다. </del> <ins> Service account tokens bound to pod is now a stable feature. The feature gates will be removed in 1.21 release. For more information, refer to notes below on the changelogs. </ins> <del> ### 대시보드 v2 </del> <ins> ### 런타임클래스(RuntimeClass)가 안정 기능으로 전환 </ins> <del> SIG UI는 쿠버네티스 대시보드 애드온 v2를 릴리스했다. [쿠버네티스/대시보드](https://github.com/kubernetes/dashboard/releases) 리포지터리에서 최신 릴리스를 찾을 수 있다. 쿠버네티스 대시보드에는 이제 CRD 지원, 새로운 번역 및 AngularJS의 업데이트된 버전이 포함된다. </del> <ins> The `node.k8s.io` API groups are promoted from `v1beta1` to `v1`. `v1beta1` is now deprecated and will be removed in a future release, please start using `v1`. ([#95718](https://github.com/kubernetes/kubernetes/pull/95718), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Apps, Auth, Node, Scheduling and Testing] </ins> <del> ### 윈도우 컨테이너 지원을 베타로 전환 </del> <ins> ### 클라우드 컨트롤러 관리자(Cloud Controller Manager)가 이제 각 클라우드 공급자를 통해서만 제공 </ins> <del> 쿠버네티스 1.18에서 처음 도입된 윈도우 컨테이너 지원이 이번 릴리스에서 베타로 전환된다. 여기에는 윈도우 서버 버전 2004에 대한 추가 지원이 포함된다(전체 버전 호환성은 [윈도우용 문서](https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#cri-containerd)에서 찾을 수 있다). SIG Windows는 이번 릴리스에 대한 몇 가지 추가 사항도 포함한다. - DSR(Direct Server Return) 모드 지원으로 많은 서비스를 효율적으로 확장 가능 - 윈도우 컨테이너가 CPU 한계를 준수 - 메트릭 및 요약 모음에 대한 성능 개선 ### 쿠버네티스 지원 기간을 1년으로 늘림 쿠버네티스 1.19부터, 쿠버네티스 마이너(minor) 릴리스에 대한 패치 릴리스를 통한 버그 수정 지원이 9개월에서 1년으로 늘었다. 2019년 초에 워킹그룹(WG) 장기 지원(LTS)에서 실시한 설문 조사에 따르면 쿠버네티스 최종 사용자의 상당 부분이 이전 9개월 지원 기간 내에 업그레이드에 실패한 것으로 보인다. 연간 지원 기간은 최종 사용자가 원하는 대로 '추가적인 기간(쿠션)'을 제공하며, 이러한 방식을 통해 연간 계획 주기와도 더 적절하게 어울린다. </del> <ins> Kubernetes will no longer ship an instance of the Cloud Controller Manager binary. Each Cloud Provider is expected to ship their own instance of this binary. Details for a Cloud Provider to create an instance of such a binary can be found under [here](https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/cloud-provider/sample). Anyone with questions on building a Cloud Controller Manager should reach out to SIG Cloud Provider. Questions about the Cloud Controller Manager on a Managed Kubernetes solution should go to the relevant Cloud Provider. Questions about the Cloud Controller Manager on a non managed solution can be brought up with SIG Cloud Provider. </ins> ## 알려진 이슈 <del> 새로운 스토리지 용량 추적 알파 기능은 WaitForFirstConsumer 볼륨 바인딩 모드 제한의 영향을 받는 것으로 알려져 있다. [#94217](https://github.com/kubernetes/kubernetes/issues/94217) </del> <ins> ### kubelet의 요약(Summary) API는 가속기(accelerator) 메트릭을 가지고 있지 않음 Currently, cadvisor_stats_provider provides AcceleratorStats but cri_stats_provider does not. As a result, when using cri_stats_provider, kubelet's Summary API does not have accelerator metrics. [There is an open work in progress to fix this](https://github.com/kubernetes/kubernetes/pull/96873). </ins> ## 긴급 업그레이드 노트 <del> ### (업그레이드 전에 반드시 읽어야 함) - 조치 필요: 코어 마스터 기본 이미지(kube-controller-manager)를 debian에서 distroless로 전환한다. 스크립트를 사용하여 Flex 볼륨 지원이 필요한 경우 필요한 패키지(bash와 같은)로 자체 이미지를 빌드하라. ([#91329](https://github.com/kubernetes/kubernetes/pull/91329), [@dims](https://github.com/dims)) [SIG Cloud Provider, Release, Storage 및 Testing] - 조치 필요: --basic-auth-file 플래그를 통한 기본 인증 지원이 제거되었다. 사용자는 유사한 기능을 위해 --token-auth-file 플래그로 마이그레이션해야 한다. ([#89069](https://github.com/kubernetes/kubernetes/pull/89069), [@enj](https://github.com/enj)) [SIG API Machinery] - Azure blob 디스크 기능(`kind`: `Shared`, `Dedicated`)이 더 이상 사용되지 않는다. `kubernetes.io/azure-disk` 스토리지 클래스에 있는 `kind`: `Managed` 를 사용해야 한다. ([#92905](https://github.com/kubernetes/kubernetes/pull/92905), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Storage] - CVE-2020-8559(Medium): 손상된 노드에서 클러스터로 권한 상승. 자세한 내용은 https://github.com/kubernetes/kubernetes/issues/92914 를 참고한다. API 서버는 더이상 업그레이드 요청에 대해 101이 아닌 응답을 프록시하지 않는다. 이로 인해 101이 아닌 응답 코드로 업그레이드 요청에 응답하는 프록시된 백엔드(확장 API 서버와 같은)가 손상될 수 있다. ([#92941](https://github.com/kubernetes/kubernetes/pull/92941), [@tallclair](https://github.com/tallclair)) [SIG API Machinery] - Kubeadm은 /var/lib/kubelet/kubeadm-flags.env에서 사용 중단된 '--cgroup-driver' 플래그를 설정하지 않으며, kubelet의 config.yaml 파일에 설정한다. /var/lib/kubelet/kubeadm-flags.env 파일이나 /etc/default/kubelet 파일에 (RPM의 경우 /etc/sysconfig/kubelet) 이 플래그가 있는 경우 제거 후에 KubeletConfiguration을 사용하여 값을 지정해야 한다. ([#90513](https://github.com/kubernetes/kubernetes/pull/90513), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] - Kubeadm은 이제 ClusterConfiguration에서 사용자가 지정한 etcd 버전을 준수하고 올바르게 사용한다. 사용자가 ClusterConfiguration에 지정된 버전을 유지하지 않으려면, kubeadm-config 컨피그맵을 편집하고 삭제해야 한다. ([#89588](https://github.com/kubernetes/kubernetes/pull/89588), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] - Kubeadm은 systemd-resolved 서비스가 활성화된 경우에도 사용자가 설정한 resolvConfg 값을 따른다. kubeadm은 더이상 /var/lib/kubelet/kubeadm-flags.env 파일에 있는 '--resolv-conf' 플래그를 설정하지 않는다. /var/lib/kubelet/kubeadm-flags.env 파일이나 /etc/default/kubelet 파일에(RPM의 경우 /etc/sysconfig/kubelet) 이 플래그가 있는 경우 제거 후에 KubeletConfiguration을 사용하여 값을 지정해야 한다. ([#90394](https://github.com/kubernetes/kubernetes/pull/90394), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] - Kubeadm: "kubeadm init 단계: kubelet-start"가 "kubeconfig" 단계 이후인, 초기화 워크 플로의 후반부로 이동한다. 이렇게 하면 kubeadm이 KubeletConfiguration 구성 설정 파일이 생성된 후에만 kubelet을 시작하고, OpenRC와 같은 초기화 시스템이 kubelet 서비스를 크래시루프(crashloop)할 수 없는 문제가 해결된다. - 'kubeadm config upload' 명령어는 전체 GA 사용 중단 주기 후에 최종적으로 제거되었다. 만약 그래도 사용해야 할 경우, 대안으로 'kubeadm init phase upload-config'를 사용한다. ([#92610](https://github.com/kubernetes/kubernetes/pull/92610), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] - kubescheduler.config.k8s.io/v1alpha2를 kubescheduler.config.k8s.io/v1beta1로 업그레이드 - `.bindTimeoutSeconds` 가 `VolumeBinding` 에 대한 플러그인 인수의 일부로 이동되었으며, [프로파일](#profiles)별로 별도 구성할 수 있다. - `.extenders` 가 API 표준을 충족하도록 업데이트되었다. 다음을 참고한다. - `.extenders` 디코딩은 대소문자를 구분한다. 모든 필드가 영향을 받는다. - `.extenders[*].httpTimeout` 은 `metav1.Duration` 유형이다. - `.extenders[*].enableHttps` 가 `.extenders[*].enableHTTPS` 로 이름이 변경되었다. - `RequestedToCapacityRatio` 인수 디코딩은 대소문자를 구분한다. 모든 필드가 영향을 받는다. - `DefaultPodTopologySpread` [플러그인](#scheduling-plugins)이 `SelectorSpread` 로 이름이 변경되었다. - `Unreserve` 익스텐션 포인트가 프로파일 정의에서 제거되었다. 모든 `Reserve` 플러그인은 `Unreserve` 호출을 구현한다. - `.disablePreemption` 이 제거되었다. 사용자는 "DefaultPreemption" PostFilter 플러그인을 비활성화하여 선점에 대해 비활성화할 수 있다. ([#91420](https://github.com/kubernetes/kubernetes/pull/91420), [@pancernik](https://github.com/pancernik)) [SIG Scheduling] ## 종류(Kind)별 변경 </del> <ins> ### (주의. 업그레이드 전에 반드시 읽어야 함) - A bug was fixed in kubelet where exec probe timeouts were not respected. This may result in unexpected behavior since the default timeout (if not specified) is `1s` which may be too small for some exec probes. Ensure that pods relying on this behavior are updated to correctly handle probe timeouts. See [configure probe](https://docs.k8s.io/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes) section of the documentation for more details. - This change in behavior may be unexpected for some clusters and can be disabled by turning off the `ExecProbeTimeout` feature gate. This gate will be locked and removed in future releases so that exec probe timeouts are always respected. ([#94115](https://github.com/kubernetes/kubernetes/pull/94115), [@andrewsykim](https://github.com/andrewsykim)) [SIG Node and Testing] - RuntimeClass feature graduates to General Availability. Promote `node.k8s.io` API groups from `v1beta1` to `v1`. `v1beta1` is now deprecated and will be removed in a future release, please start using `v1`. ([#95718](https://github.com/kubernetes/kubernetes/pull/95718), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Apps, Auth, Node, Scheduling and Testing] - API priority and fairness graduated to beta. 1.19 servers with APF turned on should not be run in a multi-server cluster with 1.20+ servers. ([#96527](https://github.com/kubernetes/kubernetes/pull/96527), [@adtac](https://github.com/adtac)) [SIG API Machinery and Testing] - For CSI drivers, kubelet no longer creates the target_path for NodePublishVolume in accordance with the CSI spec. Kubelet also no longer checks if staging and target paths are mounts or corrupted. CSI drivers need to be idempotent and do any necessary mount verification. ([#88759](https://github.com/kubernetes/kubernetes/pull/88759), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage] - Kubeadm: http://git.k8s.io/enhancements/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint/README.md ([#95382](https://github.com/kubernetes/kubernetes/pull/95382), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - The label applied to control-plane nodes "node-role.kubernetes.io/master" is now deprecated and will be removed in a future release after a GA deprecation period. - Introduce a new label "node-role.kubernetes.io/control-plane" that will be applied in parallel to "node-role.kubernetes.io/master" until the removal of the "node-role.kubernetes.io/master" label. - Make "kubeadm upgrade apply" add the "node-role.kubernetes.io/control-plane" label on existing nodes that only have the "node-role.kubernetes.io/master" label during upgrade. - Please adapt your tooling built on top of kubeadm to use the "node-role.kubernetes.io/control-plane" label. - The taint applied to control-plane nodes "node-role.kubernetes.io/master:NoSchedule" is now deprecated and will be removed in a future release after a GA deprecation period. - Apply toleration for a new, future taint "node-role.kubernetes.io/control-plane:NoSchedule" to the kubeadm CoreDNS / kube-dns managed manifests. Note that this taint is not yet applied to kubeadm control-plane nodes. - Please adapt your workloads to tolerate the same future taint preemptively. - Kubeadm: improve the validation of serviceSubnet and podSubnet. ServiceSubnet has to be limited in size, due to implementation details, and the mask can not allocate more than 20 bits. PodSubnet validates against the corresponding cluster "--node-cidr-mask-size" of the kube-controller-manager, it fail if the values are not compatible. kubeadm no longer sets the node-mask automatically on IPv6 deployments, you must check that your IPv6 service subnet mask is compatible with the default node mask /64 or set it accordenly. Previously, for IPv6, if the podSubnet had a mask lower than /112, kubeadm calculated a node-mask to be multiple of eight and splitting the available bits to maximise the number used for nodes. ([#95723](https://github.com/kubernetes/kubernetes/pull/95723), [@aojea](https://github.com/aojea)) [SIG Cluster Lifecycle] - The deprecated flag --experimental-kustomize is now removed from kubeadm commands. Use --experimental-patches instead, which was introduced in 1.19. Migration infromation available in --help description for --exprimental-patches. ([#94871](https://github.com/kubernetes/kubernetes/pull/94871), [@neolit123](https://github.com/neolit123)) - Windows hyper-v container featuregate is deprecated in 1.20 and will be removed in 1.21 ([#95505](https://github.com/kubernetes/kubernetes/pull/95505), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] - The kube-apiserver ability to serve on an insecure port, deprecated since v1.10, has been removed. The insecure address flags `--address` and `--insecure-bind-address` have no effect in kube-apiserver and will be removed in v1.24. The insecure port flags `--port` and `--insecure-port` may only be set to 0 and will be removed in v1.24. ([#95856](https://github.com/kubernetes/kubernetes/pull/95856), [@knight42](https://github.com/knight42), [SIG API Machinery, Node, Testing]) - Add dual-stack Services (alpha).  This is a BREAKING CHANGE to an alpha API. It changes the dual-stack API wrt Service from a single ipFamily field to 3 fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), ipFamilies (a list of families assigned), and clusterIPs (inclusive of clusterIP).  Most users do not need to set anything at all, defaulting will handle it for them.  Services are single-stack unless the user asks for dual-stack.  This is all gated by the "IPv6DualStack" feature gate. ([#91824](https://github.com/kubernetes/kubernetes/pull/91824), [@khenidak](https://github.com/khenidak)) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing] - `TokenRequest` and `TokenRequestProjection` are now GA features. The following flags are required by the API server: - `--service-account-issuer`, should be set to a URL identifying the API server that will be stable over the cluster lifetime. - `--service-account-key-file`, set to one or more files containing one or more public keys used to verify tokens. - `--service-account-signing-key-file`, set to a file containing a private key to use to sign service account tokens. Can be the same file given to `kube-controller-manager` with `--service-account-private-key-file`. ([#95896](https://github.com/kubernetes/kubernetes/pull/95896), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth, Cluster Lifecycle] - kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags: - apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration. - cluster-name: use clusterName from ClusterConfiguration - cert-dir: use certificatesDir from ClusterConfiguration ([#94879](https://github.com/kubernetes/kubernetes/pull/94879), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] - Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of `OwnerRefInvalidNamespace` are recorded when namespace mismatches between child and owner objects are detected. The [kubectl-check-ownerreferences](https://github.com/kubernetes-sigs/kubectl-check-ownerreferences) tool can be run prior to upgrading to locate existing objects with invalid ownerReferences. - A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted. - A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. ([#92743](https://github.com/kubernetes/kubernetes/pull/92743), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Apps and Testing] ## 종류(Kind)별 변경 사항 </ins> ### 사용 중단 <del> - vSphere CSI 드라이버로의 vSphere 인-트리 볼륨 마이그레이션에 대한 지원이 추가되었다. 인-트리 vSphere 볼륨 플러그인은 더이상 사용되지 않으며, 향후 릴리스에서 제거된다. vSphere에 쿠버네티스를 자체 배포하는 사용자는 CSIMigration + CSIMigrationSphere 기능을 활성화하고 vSphere CSI 드라이버(https://github.com/kubernetes-sigs/vsphere-csi-driver)를 설치하여 기존 파드 및 PVC 오브젝트에 대한 중단을 방지해야 한다. 사용자는 새 볼륨에 대해 vSphere CSI 드라이버를 직접 사용하기 시작해야 한다. vSphere 볼륨용 CSI 마이그레이션 기능을 사용하려면 최소 vSphere vCenter/ESXi 버전이 7.0u1이고 최소 HW 버전이 VM 버전 15여야 한다. vSAN 원시 정책 파라미터는 인-트리 vSphere 볼륨 플러그인에서 더이상 사용되지 않으며 향후 릴리스에서 제거될 예정이다. ([#90911](https://github.com/kubernetes/kubernetes/pull/90911), [@divyenpatel](https://github.com/divyenpatel)) [SIG API Machinery, Node 및 Storage] - Apiextensions.k8s.io/v1beta1은 사용 중단되었으며, apiextensions.k8s.io/v1을 사용한다. ([#90673](https://github.com/kubernetes/kubernetes/pull/90673), [@deads2k](https://github.com/deads2k)) [SIG API Machinery] - Apiregistration.k8s.io/v1beta1은 사용 중단되었으며, apiregistration.k8s.io/v1을 사용한다. ([#90672](https://github.com/kubernetes/kubernetes/pull/90672), [@deads2k](https://github.com/deads2k)) [SIG API Machinery] - Authentication.k8s.io/v1beta1 및 authorization.k8s.io/v1beta1은 1.19에서 사용 중단되었으며, v1 level을 사용하고, 1.22에서 제거될 예정이다. ([#90458](https://github.com/kubernetes/kubernetes/pull/90458), [@deads2k](https://github.com/deads2k)) [SIG API Machinery 및 Auth] - Autoscaling/v2beta1은 사용 중단되었으며, autoscaling/v2beta2를 사용한다. ([#90463](https://github.com/kubernetes/kubernetes/pull/90463), [@deads2k](https://github.com/deads2k)) [SIG Autoscaling] - Coordination.k8s.io/v1beta1은 1.19에서 사용 중단되어, 1.22에서 제거될 예정이며, 대신 v1을 사용한다. ([#90559](https://github.com/kubernetes/kubernetes/pull/90559), [@deads2k](https://github.com/deads2k)) [SIG Scalability] - nodeExpansion CSI 호출에 볼륨 기능 및 스테이징 대상 필드가 있는지 확인해야 한다. NodeStage 및 NodePublish 간에 호출되는 NodeExpandVolume의 동작은 CSI 볼륨에 대해 사용 중단된다. CSI 드라이버는 노드 EXPAND_VOLUME 기능이 있는 경우 NodePublish 후 NodeExpandVolume 호출을 지원해야 한다. ([#86968](https://github.com/kubernetes/kubernetes/pull/86968), [@gnufied](https://github.com/gnufied)) [SIG Storage] - 기능: azure 디스크 마이그레이션이 1.19에서 베타 버전으로 변경된다. 기능 게이트 CSIMigration이 베타(기본적으로 설정되는)로 전환되며, CSIMigrationAzureDisk가 베타(AzureDisk CSI 드라이버가 설치되야 하므로 기본적으로 설정되어 있지 않은)로 전환된다. 인-트리 AzureDisk 플러그인 "kubernetes.io/azure-disk"는 이제 사용 중단되며, 1.23에서 제거된다. 사용자는 CSIMigration + CSIMigrationAzureDisk 기능을 사용하도록 설정하고 AzureDisk CSI 드라이버(https://github.com/kubernetes-sigs/azuredisk-csi-driver)를 설치하여 해당 시점에 기존 파드 및 PVC 오브젝트가 중단되지 않도록 해야 한다. 사용자는 모든 새 볼륨에 대해 AzureDisk CSI 드라이버를 직접 사용해야 한다. ([#90896](https://github.com/kubernetes/kubernetes/pull/90896), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Storage] - Kube-apiserver: componentstatus API가 사용 중단된다. 이 API는 etcd, kube-scheduler 및 kube-controller-manager 컴포넌트의 상태를 제공했지만, 해당 컴포넌트가 API 서버에 대해 로컬이며, kube-scheduler 및 kube-controller-manager가 보안되지 않은 상태의 엔드포인트를 노출한 경우에만 동작했다. 이 API 대신 etcd의 상태 확인은 kube-apiserver의 상태 확인에 포함되고, kube-scheduler/kube-controller-manager 상태 확인은 해당 컴포넌트의 상태 엔드포인트에 대해 직접 수행될 수 있다. ([#93570](https://github.com/kubernetes/kubernetes/pull/93570), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Apps 및 Cluster Lifecycle] - Kubeadm: `kubeadm config view` 명령은 사용 중단되며, 기능 릴리스에서 제거될 예정이다. `kubectl get cm -o yaml -n kube-system kubeadm-config` 를 사용하여 kubeadm 설정을 직접 가져와야 한다. ([#92740](https://github.com/kubernetes/kubernetes/pull/92740), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] - Kubeadm: "kubeadm alpha kubelet config enable-dynamic" 명령이 사용 중단된다. 이 기능을 계속 사용하려면 k8s.io에서 "동적 Kubelet 구성"을 참조한다. ([#92881](https://github.com/kubernetes/kubernetes/pull/92881), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: `--experimental-kustomize` 기능이 사용 중단되며, 대신 `--experimental-patches` 기능을 사용한다. 지원되는 패치 형식은 "kubectl patch"와 동일하다. 디렉터리에서 파일로 읽히며, 초기화/조인/업그레이드 도중에 kubeadm 컴포넌트에 적용될 수 있다. 당분간은 정적 파드의 패치만 지원된다. ([#92017](https://github.com/kubernetes/kubernetes/pull/92017), [@neolit123](https://github.com/neolit123)) - Kubeadm: "kubeadm alpha certs renew" 명령어에 대해 사용 중단된 "--use-api"  플래그가 제거되었다. ([#90143](https://github.com/kubernetes/kubernetes/pull/90143), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - 쿠버네티스는 더이상 hyperkube 이미지 빌드를 지원하지 않는다. ([#88676](https://github.com/kubernetes/kubernetes/pull/88676), [@dims](https://github.com/dims)) [SIG Cluster Lifecycle 및 Release] - kubectl get 명령어에서 --export 플래그가 제거되었다. ([#88649](https://github.com/kubernetes/kubernetes/pull/88649), [@oke-py](https://github.com/oke-py)) [SIG CLI 및 Testing] - 스케줄러의 알파 기능인 'ResourceLimitsPriorityFunction'은 사용을 많이 하지 않아 완전히 제거되었다. ([#91883](https://github.com/kubernetes/kubernetes/pull/91883), [@SataQiu](https://github.com/SataQiu)) [SIG Scheduling 및 Testing] - Storage.k8s.io/v1beta1은 사용 중단되었으며, 대신 storage.k8s.io/v1이 사용된다. ([#90671](https://github.com/kubernetes/kubernetes/pull/90671), [@deads2k](https://github.com/deads2k)) [SIG Storage] </del> <ins> - Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. ([#94624](https://github.com/kubernetes/kubernetes/pull/94624), [@dims](https://github.com/dims)) [SIG Node] - Kubeadm: deprecate self-hosting support. The experimental command "kubeadm alpha self-hosting" is now deprecated and will be removed in a future release. ([#95125](https://github.com/kubernetes/kubernetes/pull/95125), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: graduate the "kubeadm alpha certs" command to a parent command "kubeadm certs". The command "kubeadm alpha certs" is deprecated and will be removed in a future release. Please migrate. ([#94938](https://github.com/kubernetes/kubernetes/pull/94938), [@yagonobre](https://github.com/yagonobre)) [SIG Cluster Lifecycle] - Kubeadm: remove the deprecated "kubeadm alpha kubelet config enable-dynamic" command. To continue using the feature please defer to the guide for "Dynamic Kubelet Configuration" at k8s.io. This change also removes the parent command "kubeadm alpha kubelet" as there are no more sub-commands under it for the time being. ([#94668](https://github.com/kubernetes/kubernetes/pull/94668), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: remove the deprecated --kubelet-config flag for the command "kubeadm upgrade node" ([#94869](https://github.com/kubernetes/kubernetes/pull/94869), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubectl: deprecate --delete-local-data ([#95076](https://github.com/kubernetes/kubernetes/pull/95076), [@dougsland](https://github.com/dougsland)) [SIG CLI, Cloud Provider and Scalability] - Kubelet's deprecated endpoint `metrics/resource/v1alpha1` has been removed, please adopt `metrics/resource`. ([#94272](https://github.com/kubernetes/kubernetes/pull/94272), [@RainbowMango](https://github.com/RainbowMango)) [SIG Instrumentation and Node] - Removes deprecated scheduler metrics DeprecatedSchedulingDuration, DeprecatedSchedulingAlgorithmPredicateEvaluationSecondsDuration, DeprecatedSchedulingAlgorithmPriorityEvaluationSecondsDuration ([#94884](https://github.com/kubernetes/kubernetes/pull/94884), [@arghya88](https://github.com/arghya88)) [SIG Instrumentation and Scheduling] - Scheduler alpha metrics binding_duration_seconds and scheduling_algorithm_preemption_evaluation_seconds are deprecated, Both of those metrics are now covered as part of framework_extension_point_duration_seconds, the former as a PostFilter the latter and a Bind plugin. The plan is to remove both in 1.21 ([#95001](https://github.com/kubernetes/kubernetes/pull/95001), [@arghya88](https://github.com/arghya88)) [SIG Instrumentation and Scheduling] - Support 'controlplane' as a valid EgressSelection type in the EgressSelectorConfiguration API. 'Master' is deprecated and will be removed in v1.22. ([#95235](https://github.com/kubernetes/kubernetes/pull/95235), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery] - The v1alpha1 PodPreset API and admission plugin has been removed with no built-in replacement. Admission webhooks can be used to modify pods on creation. ([#94090](https://github.com/kubernetes/kubernetes/pull/94090), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Apps, CLI, Cloud Provider, Scalability and Testing] </ins> ### API 변경 <del> - CSI드라이버(CSIDriver)가 볼륨 소유권 및 권한 수정을 지원하는지 여부를 지정할 수 있도록, 새로운 알파 수준의 필드인, `SupportsFsGroup` 가 도입되었다. 이 필드를 사용하려면 `CSIVolumeSupportFSGroup` 기능 게이트를 활성화해야 한다. ([#92001](https://github.com/kubernetes/kubernetes/pull/92001), [@huffmanca](https://github.com/huffmanca)) [SIG API Machinery, CLI 및 Storage] - 사용 중단된 어노테이션을 신규 API 서버의 필드와 동기화하기 위해 seccomp 프로파일에 대한 파드 버전 비대칭 전략이 추가되었다. 자세한 설명은 [KEP의](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/20190717-seccomp-ga.md#version-skew-strategy) 해당 섹션을 참고한다. ([#91408](https://github.com/kubernetes/kubernetes/pull/91408), [@saschagrunert](https://github.com/saschagrunert)) [SIG Apps, Auth, CLI 및 Node] - Kubelet에서 수집한 가속기/GPU 메트릭을 비활성화하는 기능이 추가되었다. ([#91930](https://github.com/kubernetes/kubernetes/pull/91930), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) [SIG Node] - 이제 어드미션 웹훅은 어드미션 리뷰 응답의 `.response.warnings` 필드를 사용하여 API 클라이언트에 표시되는 경고 메시지를 반환할 수 있다. ([#92667](https://github.com/kubernetes/kubernetes/pull/92667), [@liggitt](https://github.com/liggitt)) [SIG API Machinery 및 Testing] - CertificateSigningRequest API의 조건이 업데이트되었다. - `status` 필드가 추가되었다. 이 필드의 기본값은 `True` 이며 `Approved`, `Denied` 및 `Failed` 조건에 대해서만 `True` 로 설정될 수 있다. - `lastTransitionTime` 필드가 추가되었다. - 서명자가 영구적인 실패를 표시할 수 있도록 `Failed` 조건 유형이 추가되었다. 이 조건은 `certificatesigningrequests/status` 하위 리소스를 통해 추가될 수 있다. - `Approved` 및 `Denied` 조건은 상호 배타적이다. - `Approved`, `Denied` , `Failed` 조건은 더이상 CSR에서 제거할 수 없다. ([#90191](https://github.com/kubernetes/kubernetes/pull/90191), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Apps, Auth, CLI 및 Node] - 클러스터 관리자는 이제 kubelet 구성 파일에서 enableSystemLogHandler를 false로 설정하여 kubelet에서 /logs 엔드포인트를 끌 수 있다. enableSystemLogHandler는 오직 enableDebuggingHandlers도 true로 설정된 경우에만 true로 설정할 수 있다. ([#87273](https://github.com/kubernetes/kubernetes/pull/87273), [@SaranBalaji90](https://github.com/SaranBalaji90)) [SIG Node] - 사용자 정의 엔드포인트는 이제 새로운 EndpointSliceMirroring 컨트롤러에 의해 엔드포인트슬라이스에 미러링된다. ([#91637](https://github.com/kubernetes/kubernetes/pull/91637), [@robscott](https://github.com/robscott)) [SIG API Machinery, Apps, Auth, Cloud Provider, Instrumentation, Network 및 Testing] - CustomResourceDefinitions은 `spec.versions[*].deprecated` 를 `true` 로 설정하여 버전을 사용 중단됨으로 표시하고, 선택적으로 `spec.versions[*].deprecationWarning` 필드로 기본 지원 중단 경고를 재정의할 수 있도록 지원한다.  ([#92329](https://github.com/kubernetes/kubernetes/pull/92329), [@liggitt](https://github.com/liggitt)) [SIG API Machinery] - EnvVarSource api 문서의 버그가 수정되었다. ([#91194](https://github.com/kubernetes/kubernetes/pull/91194), [@wawa0210](https://github.com/wawa0210)) [SIG Apps] - "Too large resource version" 오류에서 복구할 수 없는 리플렉터(reflector)의 버그를 수정하였다. ([#92537](https://github.com/kubernetes/kubernetes/pull/92537), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery] - 고침: 이제 로그 타임스탬프에 고정 너비를 유지하기 위해 후행에 0이 포함된다. ([#91207](https://github.com/kubernetes/kubernetes/pull/91207), [@iamchuckss](https://github.com/iamchuckss)) [SIG Apps 및 Node] - `GenericEphemeralVolume` 기능 게이트의 새로운 알파 기능인 일반 임시 볼륨은 `EmptyDir` 볼륨에 대한 보다 유연한 대안을 제공한다. `EmptyDir` 과 마찬가지로 쿠버네티스에 의해 각 파드에 대한 볼륨이 자동으로 생성되고 삭제된다. 그러나 일반적인 프로비저닝 프로세스(`PersistentVolumeClaim`)를 사용하기 때문에, 스토리지는 타사 스토리지 공급 업체에서 제공할 수 있으며, 일반적인 모든 볼륨 기능이 작동한다. 볼륨은 비워둘 필요가 없다. 예를 들어, 스냅샷으로부터의 복원이 지원된다. ([#92784](https://github.com/kubernetes/kubernetes/pull/92784), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Instrumentation, Node, Scheduling, Storage 및 Testing] - 이제 쿠버네티스 빌드를 위하여 최소 Go1.14.4 이상의 버전이 필요하다. ([#92438](https://github.com/kubernetes/kubernetes/pull/92438), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage 및 Testing] - kubectl edit 명령어에서 managedFields가 숨겨졌다. ([#91946](https://github.com/kubernetes/kubernetes/pull/91946), [@soltysh](https://github.com/soltysh)) [SIG CLI] - K8s.io/apimachinery - scheme.Convert()는 이제 명시적으로 등록된 변환만을 사용한다. 기본 리플렉션 기반 변환은 더이상 사용할 수 없다. `+k8s:conversion-gen` 태그는 `k8s.io/code-generator` 컴포넌트와 함께 사용하여 변환을 생성할 수 있다. ([#90018](https://github.com/kubernetes/kubernetes/pull/90018), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery, Apps 및 Testing] - Kube-proxy: 포트 바인딩 실패를 치명적인 오류로 처리하기 위해 `--bind-address-hard-fail` 플래그가 추가되었다. ([#89350](https://github.com/kubernetes/kubernetes/pull/89350), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle 및 Network] - Kubebuilder 유효성 검사 태그는 CRD 생성을 위해 metav1.Condition에 설정된다. ([#92660](https://github.com/kubernetes/kubernetes/pull/92660), [@damemi](https://github.com/damemi)) [SIG API Machinery] - Kubelet의 --runonce 옵션은 이제 Kubelet의 설정 파일에서 `runOnce` 로도 사용할 수 있다. ([#89128](https://github.com/kubernetes/kubernetes/pull/89128), [@vincent178](https://github.com/vincent178)) [SIG Node] - Kubelet: 구조화된 로깅을 지원하기 위해 '--logging-format' 플래그가 추가되었다. ([#91532](https://github.com/kubernetes/kubernetes/pull/91532), [@afrouzMashaykhi](https://github.com/afrouzMashaykhi)) [SIG API Machinery, Cluster Lifecycle, Instrumentation 및 Node] - 쿠버네티스는 이제 golang 1.15.0-rc.1로 빌드된다. - 주체 대체 이름(Subject Alternative Names)이 없을 때, 인증서를 제공하는 X.509의 CommonName 필드를 호스트 이름으로 처리하는, 더이상 사용되지 않는 레거시 동작은 이제 기본적으로 비활성화된다. GODEBUG 환경 변수에 x509ignoreCN=0 값을 추가하여 일시적으로 다시 활성화할 수도 있다. ([#93264](https://github.com/kubernetes/kubernetes/pull/93264), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Storage 및 Testing] - 변경할 수 없는 시크릿/컨피그맵 기능을 베타로 승격하고 기본적으로 기능이 활성화된다. 이를 통해 시크릿 또는 컨피그맵 오브젝트의 `Immutable` 필드를 설정하여 내용을 immutable로 표시할 수 있다. ([#89594](https://github.com/kubernetes/kubernetes/pull/89594), [@wojtek-t](https://github.com/wojtek-t)) [SIG Apps 및 Testing] - 스케줄 구성 `KubeSchedulerConfiguration` 에서 `BindTimeoutSeconds` 가 제거되었다. ([#91580](https://github.com/kubernetes/kubernetes/pull/91580), [@cofyc](https://github.com/cofyc)) [SIG Scheduling 및 Testing] - kubescheduler.config.k8s.io/v1alpha1이 제거되었다. ([#89298](https://github.com/kubernetes/kubernetes/pull/89298), [@gavinfish](https://github.com/gavinfish)) [SIG Scheduling] - 예약에 실패한 예약 플러그인은 예약 해제 익스텐션 포인트를 트리거하게 된다. ([#92391](https://github.com/kubernetes/kubernetes/pull/92391), [@adtac](https://github.com/adtac)) [SIG Scheduling 및 Testing] - 예전 API 클라이언트가 제출한 업데이트/패치 요청에서 처리하는 `metadata.managedFields` 의 회귀 문제가 해결되었다. ([#91748](https://github.com/kubernetes/kubernetes/pull/91748), [@apelisse](https://github.com/apelisse)) - 스케줄러: 바인딩되지 않은 볼륨이 있는 파드를 예약하기 전에 선택적으로 사용 가능한 스토리지 용량을 확인한다. (신규 기능 게이트인 `CSIStorageCapacity` 가 있는 알파 기능, CSI 드라이버에서만 작동하며, CSI 드라이버 배포의 기능 지원에 따라 상이하다.) ([#92387](https://github.com/kubernetes/kubernetes/pull/92387), [@pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Scheduling, Storage 및 Testing] - Seccomp 지원이 GA로 전환되었다. 새로운 `seccompProfile` 필드가 파드 및 컨테이너의 securityContext 오브젝트에 추가된다. `seccomp.security.alpha.kubernetes.io/pod` 및 `container.seccomp.security.alpha.kubernetes.io/...` 어노테이션에 대한 지원은 사용 중단되며, v1.22에서 제거된다. ([#91381](https://github.com/kubernetes/kubernetes/pull/91381), [@pjbgf](https://github.com/pjbgf)) [SIG Apps, Auth, Node, Release, Scheduling 및 Testing] - ServiceAppProtocol 기능 게이트는 이제 베타 버전이며 기본적으로 활성화되어 서비스 및 엔드포인트에 새 AppProtocol 필드를 추가한다. ([#90023](https://github.com/kubernetes/kubernetes/pull/90023), [@robscott](https://github.com/robscott)) [SIG Apps 및 Network] - SetHostnameAsFQDN은 PodSpec의 새 필드이다. true로 설정하면, 파드의 정규화된 도메인 이름(FQDN)이 해당 컨테이너의 호스트 이름으로 설정된다. 리눅스 컨테이너에서 이는 커널의 hostname 필드(utsname 구조체의 nodename 필드)에 FQDN을 설정하는 것을 의미한다. 윈도우 컨테이너에서 이는 레지스트리 키 HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters에 대한 호스트 이름의 레지스트리 값을 FQDN으로 설정하는 것을 의미한다. 파드에 FQDN이 없는 경우에는 아무런 효과가 없다. ([#91699](https://github.com/kubernetes/kubernetes/pull/91699), [@javidiaz](https://github.com/javidiaz)) [SIG Apps, Network, Node 및 Testing] - The CertificateSigningRequest API는 다음의 변경 사항과 함께 certificates.k8s.io/v1로 승격되었다. - 이제 `spec.signerName` 이 필요하며, `certificates.k8s.io/v1` API를 통해 `kubernetes.io/legacy-unknown` 요청을 생성하는것이 허용되지 않는다. - `spec.usages` 가 필요하며, 중복값을 포함할 수 없으며, 알려진 사용법만 포함해야 한다. - `status.conditions` 는 중복 유형을 포함할 수 없다. - `status.conditions[*].status` 가 필요하다. - `status.certificate` 는 PEM으로 반드시 인코딩되어야 하며, CERTIFICATE 블록만을 포함해야 한다. ([#91685](https://github.com/kubernetes/kubernetes/pull/91685), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Architecture, Auth, CLI 및 Testing] - 이제 HugePageStorageMediumSize 기능 게이트는 기본적으로 켜져 있으며, 컨테이너 수준에서 여러 크기의 huge page 리소스를 사용할 수 있다. ([#90592](https://github.com/kubernetes/kubernetes/pull/90592), [@bart0sh](https://github.com/bart0sh)) [SIG Node] - Kubelet의 --node-status-max-images 옵션은 이제 Kubelet 구성 파일의 nodeStatusMaxImage 필드를 통해 사용할 수 있다. ([#91275](https://github.com/kubernetes/kubernetes/pull/91275), [@knabben](https://github.com/knabben)) [SIG Node] - Kubelet의 --seccomp-profile-root 옵션은 사용 중단된 것으로 표시된다. ([#91182](https://github.com/kubernetes/kubernetes/pull/91182), [@knabben](https://github.com/knabben)) [SIG Node] - Kubelet의 `--bootstrap-checkpoint-path` 옵션이 제거되었다. ([#91577](https://github.com/kubernetes/kubernetes/pull/91577), [@knabben](https://github.com/knabben)) [SIG Apps 및 Node] - Kubelet의 `--cloud-provider` 및 `--cloud-config` 옵션은 사용 중단된 것으로 표시된다. ([#90408](https://github.com/kubernetes/kubernetes/pull/90408), [@knabben](https://github.com/knabben)) [SIG Cloud Provider 및 Node] - Kubelet의 `--enable-server` 및 `--provider-id` 옵션은 이제 각각 Kubelet 구성 파일의 `enableServer` 및 `providerID` 필드를 통해 사용할 수 있다. ([#90494](https://github.com/kubernetes/kubernetes/pull/90494), [@knabben](https://github.com/knabben)) [SIG Node] - Kubelet의 `--kernel-memcg-notification` 옵션은 이제 Kubelet 구성 파일의 kernelMemcgNotification 필드를 통해 사용할 수 있다. ([#91863](https://github.com/kubernetes/kubernetes/pull/91863), [@knabben](https://github.com/knabben)) [SIG Cloud Provider, Node 및 Testing] - Kubelet의 `--really-crash-for-testing` 및 `--chaos-chance` 옵션은 사용 중단된 것으로 표시된다. ([#90499](https://github.com/kubernetes/kubernetes/pull/90499), [@knabben](https://github.com/knabben)) [SIG Node] - Kubelet의 `--volume-plugin-dir` 옵션은 이제 Kubelet 구성 파일의 `VolumePluginDir` 필드를 통해 사용할 수 있다. ([#88480](https://github.com/kubernetes/kubernetes/pull/88480), [@savitharaghunathan](https://github.com/savitharaghunathan)) [SIG Node] - `DefaultIngressClass` 기능은 이제 GA로 전환되었다. `--feature-gate` 파라미터는 1.20에서 제거된다. ([#91957](https://github.com/kubernetes/kubernetes/pull/91957), [@cmluciano](https://github.com/cmluciano)) [SIG API Machinery, Apps, Network 및 Testing] - 알파 `DynamicAuditing` 기능 게이트 및 `auditregistration.k8s.io/v1alpha1` API가 제거되었으며 더이상 지원되지 않는다. ([#91502](https://github.com/kubernetes/kubernetes/pull/91502), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Auth 및 Testing] - kube-controller-manager 관리 서명자는 이제 고유한 서명 인증서와 키를 소유할 수 있다. `--cluster-signing-[signer-name]-{cert,key}-file` 에 대한 도움말을 참고한다. `--cluster-signing-{cert,key}-file` 는 여전히 기본값이다. ([#90822](https://github.com/kubernetes/kubernetes/pull/90822), [@deads2k](https://github.com/deads2k)) [SIG API Machinery, Apps 및 Auth] - v1.14부터 사용 중단된, 사용하지 않는 `series.state` 필드는 `events.k8s.io/v1beta1` 및 `v1` 이벤트 유형에서 삭제되었다. ([#90449](https://github.com/kubernetes/kubernetes/pull/90449), [@wojtek-t](https://github.com/wojtek-t)) [SIG Apps] - 스케줄러 플러그인에 대한 예약 해제 익스텐션 포인트가 예약 익스텐션 포인트로 병합되었다. ([#92200](https://github.com/kubernetes/kubernetes/pull/92200), [@adtac](https://github.com/adtac)) [SIG Scheduling 및 Testing] - Golang 버전이 v1.14.4로 변경되었다. ([#88638](https://github.com/kubernetes/kubernetes/pull/88638), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Cloud Provider, Release 및 Testing] - Service.Spec.IPFamily에 대한 API 문서를 변경하여, 이중 스택 기능이 GA 되기 전에 정확한 의미가 변경될 수 있으며, 사용자는 기존 서비스가 IPv4, IPv6 혹은 이중 스택인지 확인하기 위해 IPFamily가 아닌 ClusterIP 또는 엔드포인트를 살펴봐야 한다. ([#91527](https://github.com/kubernetes/kubernetes/pull/91527), [@danwinship](https://github.com/danwinship)) [SIG Apps 및 Network] - 사용자는 리소스 그룹을 무시하도록 리소스 접두사를 구성할 수 있다. ([#88842](https://github.com/kubernetes/kubernetes/pull/88842), [@angao](https://github.com/angao)) [SIG Node 및 Scheduling] - `Ingress` 및 `IngressClass` 리소스가 `networking.k8s.io/v1` 로 변경되었다. `extensions/v1beta1` 및 `networking.k8s.io/v1beta1` API 버전의 인그레스 및 인그레스클래스 유형은 사용 중단되었으며, 더이상 1.22 이상에서 제공되지 않는다. 지속형 오브젝트는 `networking.k8s.io/v1` API를 통해 접근할 수 있다. v1 인그레스 오브젝트의 주목할만한 변경 사항은 다음과 같다. (v1beta1 필드 이름은 변경되지 않았다.) - `spec.backend` -> `spec.defaultBackend` - `serviceName` -> `service.name` - `servicePort` -> `service.port.name` (string 값에 대하여) - `servicePort` -> `service.port.number` (numeric 값에 대하여) - `pathType` 은 더이상 v1에서 기본값을 갖지 않는다. "Exact", "Prefix" 또는 "ImplementationSpecific" 중에서 지정되어야 한다. 기타 인그레스 API 변경 사항: - 백엔드는 이제 리소스 또는 서비스 백엔드가 될 수 있다. - `path` 는 더이상 유효한 정규 표현식이 아니어도 된다. ([#89778](https://github.com/kubernetes/kubernetes/pull/89778), [@cmluciano](https://github.com/cmluciano)) [SIG API Machinery, Apps, CLI, Network 및 Testing] - `NodeResourcesLeastAllocated` 및 `NodeResourcesMostAllocated` 플러그인은 이제 CPU 및 메모리에 대한 사용자 정의 가중치를 지원한다. [#90544](https://github.com/kubernetes/kubernetes/pull/90544), [@chendave](https://github.com/chendave)) [SIG Scheduling] - v1beta1 버전의 스케줄러 구성 설정 API에 `PostFilter` 유형이 추가되었다. ([#91547](https://github.com/kubernetes/kubernetes/pull/91547), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] - `RequestedToCapacityRatioArgs` 인코딩은 이제 엄격하게 작동한다. ([#91603](https://github.com/kubernetes/kubernetes/pull/91603), [@pancernik](https://github.com/pancernik)) [SIG Scheduling] - `v1beta1` 스케줄러의 `Extender` 인코딩은 대소문자를 구분하며(`v1alpha1`/`v1alpha2` 는 대소문자를 구분하지 않았다.), `httpTimeout` 필드는 기간 인코딩을 사용하며(예를 들어, 1초는 `"1s"` 로 지정된다.), `v1alpha1`/`v1alpha2` 의 `enableHttps` 필드 이름이 `enableHTTPS` 로 변경되었다. ([#91625](https://github.com/kubernetes/kubernetes/pull/91625), [@pancernik](https://github.com/pancernik)) [SIG Scheduling] ### 기능 - defaultpreemption 플러그인은 기존 하드코딩된 파드 선점 로직을 대체하는 스케줄러에 등록되고 활성화된다. ([#92049](https://github.com/kubernetes/kubernetes/pull/92049), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling 및 Testing] - 스케줄링 필터 실패를 해결하기 위해, 필터 단계 후에 실행되는 스케줄러 프레임워크에 새로운 익스텐션 포인트인 `PostFilter` 가 도입되었다. 전형적인 구현은 선점 로직을 실행하는 것이다. ([#91314](https://github.com/kubernetes/kubernetes/pull/91314), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling 및 Testing] - 조치 필요: CoreDNS v1.7.0에서, [메트릭의 이름이 변경되어](https://github.com/coredns/coredns/blob/master/notes/coredns-1.7.0.md#metric-changes) 이전 메트릭의 이름을 사용하는 기존 리포팅 공식과 역호환(backward incompatible)되지 않는다. 업그레이드하기 전에 공식을 새 이름으로 조정해야 한다. Kubeadm은 이제 CoreDNS 버전 v1.7.0이 포함된다. 주요 변경 사항은 다음과 같다. -  CoreDNS가 서비스 기록 업데이트를 중단할 수 있는 버그가 수정되었다. -  어떤 정책이 설정되어 있어도 항상 첫 번째 업스트림 서버만 선택되는 포워드 플러그인의 버그가 수정되었다. -  쿠버네티스 플러그인에서 이미 사용 중단된 `resyncperiod` 및 `upstream` 이 제거되었다. -  프로메테우스(Prometheus) 메트릭 이름 변경을 포함한다. (표준 프로메테우스 메트릭 명명 규칙에 맞게 변경). 이전 메트릭의 이름을 사용하는 기존 리포팅 공식과 역호환된다. -  페더레이션 플러그인(v1 쿠버네티스 페더레이션을 허용)이 제거되었다. 자세한 내용은 https://coredns.io/2020/06/15/coredns-1.7.0-release/ 에서 확인할 수 있다. ([#92651](https://github.com/kubernetes/kubernetes/pull/92651), [@rajansandeep](https://github.com/rajansandeep)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle 및 Instrumentation] - 사용 중단된 버전에 대한 API 요청은 이제 API 응답에서 경고 헤더를 수신하고, 사용 중단된 API의 사용을 나타내는 메트릭이 표시되도록 한다. - `kubectl` 은 표준 에러 출력으로 경고를 출력하며, `--warnings-as-errors` 옵션을 허용하여 경고를 치명적인 오류로 처리한다. - `k8s.io/client-go` 은 기본적으로 표준 에러 출력에 경고를 출력한다. `config.WarningHandler` 를 설정하여 클라이언트 별로 재정의하거나, `rest.SetDefaultWarningHandler()` 로 프로세스별로 재정의한다. - `kube-apiserver` 는 요청된, 사용 중단된 API에 대해 `group`, `version`, `resource`, `subresource` 및 `removed_release` 레이블을 표시하고 `apiserver_requested_deprecated_apis` 게이지 메트릭을 `1` 로 설정한다. ([#73032](https://github.com/kubernetes/kubernetes/pull/73032), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, CLI, Instrumentation 및 Testing] - component-base에 --logging-format 플래그가 추가되었다. 기본값은 변경되지 않은 klog를 사용하는 "text"이다. ([#89683](https://github.com/kubernetes/kubernetes/pull/89683), [@yuzhiquan](https://github.com/yuzhiquan)) [SIG Instrumentation] - kubectl create deployment 명령어에 --port 플래그가 추가되었다. ([#91113](https://github.com/kubernetes/kubernetes/pull/91113), [@soltysh](https://github.com/soltysh)) [SIG CLI 및 Testing] - cmd/cloud-controller-manager에 .import-restrictions 파일이 추가되었다. ([#90630](https://github.com/kubernetes/kubernetes/pull/90630), [@nilo19](https://github.com/nilo19)) [SIG API Machinery 및 Cloud Provider] - CRI-API ImageSpec 오브젝트에 어노테이션이 추가되었다. ([#90061](https://github.com/kubernetes/kubernetes/pull/90061), [@marosset](https://github.com/marosset)) [SIG Node 및 Windows] - 스케줄러의 PodSchedulingDuration 메트릭에 시도(attempts)에 대한 레이블이 추가되었다. ([#92650](https://github.com/kubernetes/kubernetes/pull/92650), [@ahg-g](https://github.com/ahg-g)) [SIG Instrumentation 및 Scheduling] - kubectl scale 명령어에 클라이언트측과 서버측에 대한 드라이런(dry-run) 지원이 추가되었다. ([#89666](https://github.com/kubernetes/kubernetes/pull/89666), [@julianvmodesto](https://github.com/julianvmodesto)) [SIG CLI 및 Testing] - kubectl diff에 셀렉터가 추가되었다. ([#90857](https://github.com/kubernetes/kubernetes/pull/90857), [@sethpollack](https://github.com/sethpollack)) [SIG CLI] - cgroups v2 노드의 유효성 검사에 대한 지원이 추가되었다. ([#89901](https://github.com/kubernetes/kubernetes/pull/89901), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle  및 Node] - 노드 수준에서, 다른 크기로 사전 할당된 huge page에 대한 지원이 추가되었다. ([#89252](https://github.com/kubernetes/kubernetes/pull/89252), [@odinuge](https://github.com/odinuge)) [SIG Apps 및 Node] - Azure 파일 드라이버에 대한 태그 지원이 추가되었다. ([#92825](https://github.com/kubernetes/kubernetes/pull/92825), [@ZeroMagic](https://github.com/ZeroMagic)) [SIG Cloud Provider 및 Storage] - azure 디스크 드라이버에 대한 태그 지원이 추가되었다. ([#92356](https://github.com/kubernetes/kubernetes/pull/92356), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Storage] - kubectl run 명령어에 --privileged 플래그가 추가되었다. ([#90569](https://github.com/kubernetes/kubernetes/pull/90569), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] - `v1beta1` 장비 플러그인 API에 새로운 `GetPreferredAllocation()` 호출이 추가되었다. ([#92665](https://github.com/kubernetes/kubernetes/pull/92665), [@klueska](https://github.com/klueska)) [SIG Node 및 Testing] - 쿠버네티스 서비스의 세션 어피니티를 설정하기 위해 윈도우에 대한 기능 지원이 추가되었다. 필요: [윈도우 서버 vNext Insider Preview Build 19551](https://blogs.windows.com/windowsexperience/2020/01/28/announcing-windows-server-vnext-insider-preview-build-19551/)(또는 그 이상) ([#91701](https://github.com/kubernetes/kubernetes/pull/91701), [@elweb9858](https://github.com/elweb9858)) [SIG Network 및 Windows] - kube-apiserver 메트릭 추가: apiserver_current_inflight_request_measures. (API 우선순위(Priority) 및 공정성(Fairness)이 활성화된 경우, windowed_request_stats). ([#91177](https://github.com/kubernetes/kubernetes/pull/91177), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery, Instrumentation 및 Testing] - AWS 로드밸런서 서비스의 대상 노드에 service.beta.kubernetes.io/aws-load-balancer-target-node-labels 어노테이션이 추가되었다. ([#90943](https://github.com/kubernetes/kubernetes/pull/90943), [@foobarfran](https://github.com/foobarfran)) [SIG Cloud Provider] - 서로 다른 세분성을 갖는 흐름-제어 시스템의 내부 상태를 덤프하기 위해 접두사 "/debug/flowcontrol/*" 아래에 디버깅 엔드포인트 집합이 추가되었다. ([#90967](https://github.com/kubernetes/kubernetes/pull/90967), [@yue9944882](https://github.com/yue9944882)) [SIG API Machinery] - kube-scheduler의 메트릭인 framework_extension_point_duration_seconds에 프로파일 레이블이 추가되었다.([#92268](https://github.com/kubernetes/kubernetes/pull/92268), [@alculquicondor](https://github.com/alculquicondor)) [SIG Instrumentation 및 Scheduling] - kube-scheduler의 메트릭인 schedule_attempts_total에 프로파일 레이블이 추가되었다. - e2e_scheduling_duration_seconds에 결과 및 프로파일 레이블이 추가되었다. 이제 스케줄링 불가능 및 오류 시도 횟수가 기록된다. ([#92202](https://github.com/kubernetes/kubernetes/pull/92202), [@alculquicondor](https://github.com/alculquicondor)) [SIG Instrumentation 및 Scheduling] - 사용 중단된 API 버전에 대한 API 요청의 감사(audit) 이벤트는 이제 `"k8s.io/deprecated": "true"` 감사 어노테이션을 포함한다. 대상에 대한 제거 릴리스(removal release)가 식별되면, 해당 감사 이벤트는 `"k8s.io/removal-release": "<majorVersion>.<minorVersion>"` 감사 어노테이션도 포함한다. ([#92842](https://github.com/kubernetes/kubernetes/pull/92842), [@liggitt](https://github.com/liggitt)) [SIG API Machinery 및 Instrumentation] - 대시보드가 v2.0.1로 격상되었다. ([#91526](https://github.com/kubernetes/kubernetes/pull/91526), [@maciaszczykm](https://github.com/maciaszczykm)) [SIG Cloud Provider] - 클라우드 노드-컨트롤러는 InstancesV2를 사용한다. ([#91319](https://github.com/kubernetes/kubernetes/pull/91319), [@gongguan](https://github.com/gongguan)) [SIG Apps, Cloud Provider, Scalability 및 Storage] - 의존성: Golang 버전이 1.13.9로 변경되었다. - 빌드: kube-cross 이미지 빌드가 제거되었다. ([#89275](https://github.com/kubernetes/kubernetes/pull/89275), [@justaugustus](https://github.com/justaugustus)) [SIG Release 및 Testing] - 세부 스케줄러 점수 결과는 레벨(verbose) 10에서 표시된다. ([#89384](https://github.com/kubernetes/kubernetes/pull/89384), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] - E2e.test는 클러스터가 적합하다고 평가되기 위해, 통과해야 하는 적합성 테스트 목록을 표시할 수 있다. ([#88924](https://github.com/kubernetes/kubernetes/pull/88924), [@dims](https://github.com/dims)) [SIG Architecture 및 Testing] - PodTopologySpread 플러그인을 사용하여 defaultspreading을 수행하려면 DefaultPodTopologySpread 기능 게이트를 활성화해야 한다. 이렇게 하면, 레거시 DefaultPodTopologySpread 플러그인이 비활성화된다. ([#91793](https://github.com/kubernetes/kubernetes/pull/91793), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - EndpointSlice 컨트롤러가 실패한 동기화를 재시도하기 위해 더 오래 대기하게 된다. ([#89438](https://github.com/kubernetes/kubernetes/pull/89438), [@robscott](https://github.com/robscott)) [SIG Apps 및 Network] - 로컬 영역(AWS local zone)을 지원하기 위해, AWS azToRegion 메서드가 확장된다. ([#90874](https://github.com/kubernetes/kubernetes/pull/90874), [@Jeffwan](https://github.com/Jeffwan)) [SIG Cloud Provider] - 기능: azure 공유 디스크 지원이 추가되었다. ([#89511](https://github.com/kubernetes/kubernetes/pull/89511), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Storage] - 기능: azure 디스크 api-version이 변경되었다. ([#89250](https://github.com/kubernetes/kubernetes/pull/89250), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Storage] - 기능: [Azure 공유 디스크](https://docs.microsoft.com/ko-kr/azure/virtual-machines/disks-shared-enable?tabs=azure-cli)를 지원하며, azure 디스크 스토리지 클래스에 새로운 필드(`maxShares`)가 추가되었다. kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: shared-disk provisioner: kubernetes.io/azure-disk parameters: skuname: Premium_LRS  # 현재는 프리미엄 SSD만 사용 가능하다. cachingMode: None  # 읽기전용 호스트 캐싱은 maxShares>1 인 프리미엄 SSD에 사용할 수 없다. maxShares: 2 ([#89328](https://github.com/kubernetes/kubernetes/pull/89328), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Storage] - 가상 서버 주소가 이미 바인딩되지 않은 경우, `EnsureDummyInterface` 만 실행하여 IPVS 프록시 성능을 향상시킬 수 있다. ([#92609](https://github.com/kubernetes/kubernetes/pull/92609), [@andrewsykim](https://github.com/andrewsykim)) [SIG Network] - 이제 Kube-Proxy는 EndpointSliceProxying 기능 게이트를 통해 윈도우에서 엔드포인트슬라이스를 지원한다. ([#90909](https://github.com/kubernetes/kubernetes/pull/90909), [@kumarvin123](https://github.com/kumarvin123)) [SIG Network 및 Windows] - 이제 Kube-Proxy는 IPv6DualStack 기능 게이트를 사용하여 윈도우에서 IPv6DualStack 기능을 지원한다. ([#90853](https://github.com/kubernetes/kubernetes/pull/90853), [@kumarvin123](https://github.com/kumarvin123)) [SIG Network, Node 및 Windows] - Kube-addon-manager가 v9.1.1로 변경되어, 허용된 리소스의 기본 목록을 재정의할 수 있다. (https://github.com/kubernetes/kubernetes/pull/91018) ([#91240](https://github.com/kubernetes/kubernetes/pull/91240), [@tosi3k](https://github.com/tosi3k)) [SIG Cloud Provider, Scalability 및 Testing] - etcd3에서 지원하는 Kube-apiserver는 데이터베이스 파일 크기를 표시하는 메트릭을 추출할 수 있다. ([#89151](https://github.com/kubernetes/kubernetes/pull/89151), [@jingyih](https://github.com/jingyih)) [SIG API Machinery] - Kube-apiserver, kube-scheduler 및 kube-controller manager는 이제 유닉스 시스템에서 실행되는 경우 --bind-address 및 --secure-port 플래그로 정의된 주소에서 수신 대기할 때, SO_REUSEPORT 소켓 옵션을 사용한다(윈도우는 지원되지 않는다). 이를 통해 동일한 구성으로 단일 호스트에서 이러한 프로세스의 여러 인스턴스를 실행할 수 있으므로, 다운타임 없이 정상적으로 변경/재시작이 가능하다. ([#88893](https://github.com/kubernetes/kubernetes/pull/88893), [@invidian](https://github.com/invidian)) [SIG API Machinery, Scheduling 및 Testing] - Kube-apiserver: NodeRestriction 어드미션 플러그인은 이제 새 노드를 만들 때, kubelet이 설정할 수 있는 노드 레이블을 1.16 이상에서 kubelet이 허용하는, `--node-labels` 파라미터로 제한한다. ([#90307](https://github.com/kubernetes/kubernetes/pull/90307), [@liggitt](https://github.com/liggitt)) [SIG Auth 및 Node] - Kube-controller-manager: 구조화된 로깅을 지원하기 위해 '--logging-format' 플래그가 추가되었다. ([#91521](https://github.com/kubernetes/kubernetes/pull/91521), [@SataQiu](https://github.com/SataQiu)) [SIG API Machinery 및 Instrumentation] - Kube-controller-manager: `--experimental-cluster-signing-duration` 플래그는 v1.22에서 사용 중단된 것으로 표시되며, `--cluster-signing-duration` 으로 대체된다. ([#91154](https://github.com/kubernetes/kubernetes/pull/91154), [@liggitt](https://github.com/liggitt)) [SIG Auth 및 Cloud Provider] - 이제 Kube-proxy는 리눅스에서 기본적으로 엔드포인트 대신 엔드포인트슬라이스를 사용한다. 새로운 알파 `WindowsEndpointSliceProxying` 기능 게이트를 통해 윈도우에서 기능을 활성화할 수 있다. ([#92736](https://github.com/kubernetes/kubernetes/pull/92736), [@robscott](https://github.com/robscott)) [SIG Network] - Kube-scheduler: 구조화된 로깅을 지원하기 위해 '--logging-format' 플래그가 추가되었다. ([#91522](https://github.com/kubernetes/kubernetes/pull/91522), [@SataQiu](https://github.com/SataQiu)) [SIG API Machinery, Cluster Lifecycle, Instrumentation 및 Scheduling] - Kubeadm은 이제 생성된 컴포넌트와 사용자 제공 컴포넌트의 설정을 구분하여, 구성 업그레이드가 필요한 경우 이전 컴포넌트를 다시 생성한다. ([#86070](https://github.com/kubernetes/kubernetes/pull/86070), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] - Kubeadm: 수동으로 업그레이드된 컴포넌트의 설정이, 업그레이드 계획 및 적용 중에 --config 옵션을 통해 YAML 파일에서 적용될 수 있다.  클러스터에 저장된 모든 것을 덮어쓰는 kubeadm 구성 및 컴포넌트의 설정이 보존되는 기존의 --config 동작도 유지된다. --config와 함께 사용되는 동작은 이제 kubeadm 설정 API 오브젝트(API group "kubeadm.kubernetes.io")가 파일에 제공되었는지의 여부에 따라 결정된다. ([#91980](https://github.com/kubernetes/kubernetes/pull/91980), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] - Kubeadm: 구동이 느린 컨테이너를 보호하기 위해 정적 파드에 대한 시작 프로브가 추가되었다. ([#91179](https://github.com/kubernetes/kubernetes/pull/91179), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] - Kubeadm: "kubeadm init phase certs" 하위 명령어의 "--csr-only" 및 "--csr-dir" 플래그는 사용 중단되었다. 대신 "kubeadm alpha certs generate-csr" 를 사용해야 한다. 이 신규 명령어를 사용하면 모든 컨트롤 플레인 컴포넌트에 대한 새로운 개인키 및 인증서 서명 요청을 생성할 수 있으므로, 외부 CA로 인증서에 서명할 수 있다. ([#92183](https://github.com/kubernetes/kubernetes/pull/92183), [@wallrj](https://github.com/wallrj)) [SIG Cluster Lifecycle] - Kubeadm: 'upgrade apply' 도중에, kube-proxy 컨피그맵이 누락된 경우, kube-proxy가 업그레이드되지 않아야 한다고 가정한다. DNS 서버 애드온에서 kube-dns/coredns 컨피그맵이 누락된 경우에도 동일하게 적용된다. 이는 'upgrade apply' 지원 단계까지의 일시적인 해결 방법이다. 해당 단계가 지원되면, kube-proxy/dns 수동으로 넘겨져야 한다. ([#89593](https://github.com/kubernetes/kubernetes/pull/89593), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: 컨트롤-플레인 정적 파드는 "시스템-노드-크리티컬" 우선순위 클래스로 전환되었다. ([#90063](https://github.com/kubernetes/kubernetes/pull/90063), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: upgrade plan은 이제 업그레이드 전에, 알려진 컴포넌트의 설정 상태를 나타내는 표를 출력한다. ([#88124](https://github.com/kubernetes/kubernetes/pull/88124), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] - Kubectl 명령어는 지정하지 않고(전체 리소스 이름을 입력할 필요 없음) 바로 taint no와 같이 사용할 수 있다. ([#88723](https://github.com/kubernetes/kubernetes/pull/88723), [@wawa0210](https://github.com/wawa0210)) [SIG CLI] - Kubelet: 다음의 메트릭 이름이 변경되었다. kubelet_running_container_count --> kubelet_running_containers kubelet_running_pod_count --> kubelet_running_pods ([#92407](https://github.com/kubernetes/kubernetes/pull/92407), [@RainbowMango](https://github.com/RainbowMango)) [SIG API Machinery, Cluster Lifecycle, Instrumentation 및 Node] - 클라이언트 인증서를 교체하도록 설정된 kubelet은 이제 인증서 만료까지 남은 시간(초)을 나타내는 `certificate_manager_server_ttl_seconds` 게이지 메트릭을 표시한다. ([#91148](https://github.com/kubernetes/kubernetes/pull/91148), [@liggitt](https://github.com/liggitt)) [SIG Auth 및 Node] - PodTopologySpreading이 더 잘 확산될 수 있도록 새로운 점수 계산 함수가 도입되었다. ([#90475](https://github.com/kubernetes/kubernetes/pull/90475), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - 유틸리티 라이브러리가 소폭 향상되었으며, 별도의 조치가 필요하진 않다. ([#92440](https://github.com/kubernetes/kubernetes/pull/92440), [@luigibk](https://github.com/luigibk)) [SIG Network] - PodTolerationRestriction: 에러에서 네임스페이스 허용 목록이 표시된다. ([#87582](https://github.com/kubernetes/kubernetes/pull/87582), [@mrueg](https://github.com/mrueg)) [SIG Scheduling] - 공급자별 참고 사항: vsphere: vsphere.conf - 성능 문제를 위해 자격증명 시크릿 관리를 비활성화하는 새로운 옵션 ([#90836](https://github.com/kubernetes/kubernetes/pull/90836), [@Danil-Grigorev](https://github.com/Danil-Grigorev)) [SIG Cloud Provider] - pod_preemption_metrics의 이름이 preemption_metrics로 변경되었다. ([#93256](https://github.com/kubernetes/kubernetes/pull/93256), [@ahg-g](https://github.com/ahg-g)) [SIG Instrumentation 및 Scheduling] - Rest.Config는 이제, 이전에는 환경 변수를 통해서만 구성할 수 있었던 프록시 구성을 재정의하는 플래그를 지원한다. ([#81443](https://github.com/kubernetes/kubernetes/pull/81443), [@mikedanese](https://github.com/mikedanese)) [SIG API Machinery 및 Node] - PodTopologySpreading의 점수는 maxSkew가 증가함에 따라 차이가 감소되었다. ([#90820](https://github.com/kubernetes/kubernetes/pull/90820), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - 적용된 구성으로부터 필드가 제거된 경우 서버측 적용 동작이 정규화되었다. 다른 소유자가 없는 제거된 필드는 생성된 오브젝트에서 제거되거나 기본값이 있는 경우, 기본값으로 재설정된다. 기본값으로 재설정하지 않고 HPA로 `replicas` 필드를 이전하는 것과 같은 안전한 소유권 이전은 [소유권 이전](https://kubernetes.io/docs/reference/using-api/api-concepts/#transferring-ownership)에 설명되어 있다. ([#92661](https://github.com/kubernetes/kubernetes/pull/92661), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation 및 Testing] - 서비스 컨트롤러: 노드의 관련 필드가 변경될 때에만 LB 노드 풀이 동기화된다. ([#90769](https://github.com/kubernetes/kubernetes/pull/90769), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps 및 Network] - CSIMigrationvSphere 기능 게이트가 베타로 전환되었다. 사용자는 CSIMigration + CSIMigrationvSphere 기능을 활성화하고 vSphere CSI 드라이버(https://github.com/kubernetes-sigs/vsphere-csi-driver)를 설치하여 인-트리 vSphere 플러그인 "kubernetes.io/vsphere-volume" 에서 vSphere CSI 드라이버로 워크로드를 이동해야 한다. 요구사항: vSphere vCenter/ESXi 버전: 7.0u1, HW 버전: VM 버전 15 ([#92816](https://github.com/kubernetes/kubernetes/pull/92816), [@divyenpatel](https://github.com/divyenpatel)) [SIG Cloud Provider 및 Storage] - `kubectl create deployment` 명령에 대하여 --replicas 플래그가 지원된다. ([#91562](https://github.com/kubernetes/kubernetes/pull/91562), [@zhouya0](https://github.com/zhouya0)) - 클라이언트측 적용에서 서버측 적용으로의 원활한 업그레이드를 지원하며, 해당하는 다운그레이드 또한 지원한다. ([#90187](https://github.com/kubernetes/kubernetes/pull/90187), [@julianvmodesto](https://github.com/julianvmodesto)) [SIG API Machinery 및 Testing] - VMSS의 비동기식 생성/변경을 지원한다. ([#89248](https://github.com/kubernetes/kubernetes/pull/89248), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - cgroups v2 통합 모드를 사용하는 호스트에서의 실행을 지원한다. ([#85218](https://github.com/kubernetes/kubernetes/pull/85218), [@giuseppe](https://github.com/giuseppe)) [SIG Node] - 코어 마스터 베이스 이미지(kube-apiserver, kube-scheduler)를 debian에서 distroless로 전환하였다. ([#90674](https://github.com/kubernetes/kubernetes/pull/90674), [@dims](https://github.com/dims)) [SIG Cloud Provider, Release 및 Scalability] - etcd 이미지(마이그레이션 스크립트 포함)를 debian에서 distroless로 전환하였다. ([#91171](https://github.com/kubernetes/kubernetes/pull/91171), [@dims](https://github.com/dims)) [SIG API Machinery 및 Cloud Provider] - RotateKubeletClientCertificate 기능 게이트가 GA로 승격되었으며, kubelet --feature-gate RotateKubeletClientCertificate 파라미터는 1.20에서 제거된다. ([#91780](https://github.com/kubernetes/kubernetes/pull/91780), [@liggitt](https://github.com/liggitt)) [SIG Auth 및 Node] - SCTPSupport 기능이 기본적으로 활성화된다. ([#88932](https://github.com/kubernetes/kubernetes/pull/88932), [@janosi](https://github.com/janosi)) [SIG Network] - `certificatesigningrequests/approval` 하위 리소스는 이제 패치 API 요청을 지원한다. ([#91558](https://github.com/kubernetes/kubernetes/pull/91558), [@liggitt](https://github.com/liggitt)) [SIG Auth 및 Testing] - `kubernetes_build_info` 의 메트릭 레이블 이름이 `camel case` 에서 `snake case` 로 변경되었다. - gitVersion --> git_version - gitCommit --> git_commit - gitTreeState --> git_tree_state - buildDate --> build_date - goVersion --> go_version 이 변경 사항은 `kube-apiserver`、`kube-scheduler`、`kube-proxy` 및 `kube-controller-manager` 에서 발생한다. ([#91805](https://github.com/kubernetes/kubernetes/pull/91805), [@RainbowMango](https://github.com/RainbowMango)) [SIG API Machinery, Cluster Lifecycle 및 Instrumentation] - apiserver 로그의 추적 출력이 더 체계적이고 포괄적으로 변경되었다. 추적은 중첩되며, 오래 실행되지 않은 모든 요청 엔드포인트에 대해 전체 필터 체인이 계측된다(예: 인증 확인이 포함된다.). ([#88936](https://github.com/kubernetes/kubernetes/pull/88936), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation 및 Scheduling] - 시간 초과 직전에 감시 북마크를 보내는 것 외에도, 주기적으로(요청된 경우) 보내도록 변경되었다. ([#90560](https://github.com/kubernetes/kubernetes/pull/90560), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery] - cri-tools이 v1.18.0으로 변경되었다. ([#89720](https://github.com/kubernetes/kubernetes/pull/89720), [@saschagrunert](https://github.com/saschagrunert)) [SIG Cloud Provider, Cluster Lifecycle, Release 및 Scalability] - etcd 클라이언트측이 v3.4.4로 변경되었다. ([#89169](https://github.com/kubernetes/kubernetes/pull/89169), [@jingyih](https://github.com/jingyih)) [SIG API Machinery 및 Cloud Provider] - etcd 클라이언트측이 v3.4.7로 변경되었다. ([#89822](https://github.com/kubernetes/kubernetes/pull/89822), [@jingyih](https://github.com/jingyih)) [SIG API Machinery 및 Cloud Provider] - etcd 클라이언트측이 v3.4.9로 변경되었다. ([#92075](https://github.com/kubernetes/kubernetes/pull/92075), [@jingyih](https://github.com/jingyih)) [SIG API Machinery, Cloud Provider 및 Instrumentation] - azure-sdk가 v40.2.0으로 변경되었다. ([#89105](https://github.com/kubernetes/kubernetes/pull/89105), [@andyzhangx](https://github.com/andyzhangx)) [SIG CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Storage 및 Testing] - 이제 `kubectl port-forward` 가 UDP를 지원하지 않음을 사용자에게 경고하게 된다. ([#91616](https://github.com/kubernetes/kubernetes/pull/91616), [@knight42](https://github.com/knight42)) [SIG CLI] - PodTopologySpread의 가중치 확산 스케줄링 점수가 2배로 늘어났다. ([#91258](https://github.com/kubernetes/kubernetes/pull/91258), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - `EventRecorder()` 는 `FrameworkHandle` 인터페이스에 노출되어 스케줄러 플러그인 개발자가 클러스터 수준 이벤트를 기록하도록 선택할 수 있다. ([#92010](https://github.com/kubernetes/kubernetes/pull/92010), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] - `kubectl alpha debug` 명령어는 이제 원본을 복사하여 파드 디버깅을 지원한다. ([#90094](https://github.com/kubernetes/kubernetes/pull/90094), [@aylei](https://github.com/aylei)) [SIG CLI] - `kubectl alpha debug` 명령어는 이제 노드의 호스트 네임스페이스에서 실행되는 디버깅 컨테이너를 생성하여 노드를 디버깅하는 방식을 지원한다. ([#92310](https://github.com/kubernetes/kubernetes/pull/92310), [@verb](https://github.com/verb)) [SIG CLI] - `local-up-cluster.sh` 는 이제 기본적으로 CSI snapshotter를 설치하며, `ENABLE_CSI_SNAPSHOTTER=false` 로 비활성화할 수 있다. ([#91504](https://github.com/kubernetes/kubernetes/pull/91504), [@pohly](https://github.com/pohly)) - `ImageLocality` 플러그인의 `maxThreshold` 는 이제 파드의 이미지 수에 따라 크기가 조정되어, 여러 이미지가 있는 파드의 노드 우선순위를 구분하는데 도움을 줄 수 있다. ([#91138](https://github.com/kubernetes/kubernetes/pull/91138), [@chendave](https://github.com/chendave)) [SIG Scheduling] </del> <ins> - `TokenRequest` and `TokenRequestProjection` features have been promoted to GA. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection for details on configuring and using this feature. The `TokenRequest` and `TokenRequestProjection` feature gates will be removed in v1.21. - kubeadm's kube-apiserver Pod manifest now includes the following flags by default "--service-account-key-file", "--service-account-signing-key-file", "--service-account-issuer". ([#93258](https://github.com/kubernetes/kubernetes/pull/93258), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Auth, Cluster Lifecycle, Storage and Testing] - A new `nofuzz` go build tag now disables gofuzz support. Release binaries enable this. ([#92491](https://github.com/kubernetes/kubernetes/pull/92491), [@BenTheElder](https://github.com/BenTheElder)) [SIG API Machinery] - Add WindowsContainerResources and Annotations to CRI-API UpdateContainerResourcesRequest ([#95741](https://github.com/kubernetes/kubernetes/pull/95741), [@katiewasnothere](https://github.com/katiewasnothere)) [SIG Node] - Add a `serving` and `terminating` condition to the EndpointSlice API. `serving` tracks the readiness of endpoints regardless of their terminating state. This is distinct from `ready` since `ready` is only true when pods are not terminating. `terminating` is true when an endpoint is terminating. For pods this is any endpoint with a deletion timestamp. ([#92968](https://github.com/kubernetes/kubernetes/pull/92968), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps and Network] - Add dual-stack Services (alpha).  This is a BREAKING CHANGE to an alpha API. It changes the dual-stack API wrt Service from a single ipFamily field to 3 fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), ipFamilies (a list of families assigned), and clusterIPs (inclusive of clusterIP).  Most users do not need to set anything at all, defaulting will handle it for them.  Services are single-stack unless the user asks for dual-stack.  This is all gated by the "IPv6DualStack" feature gate. ([#91824](https://github.com/kubernetes/kubernetes/pull/91824), [@khenidak](https://github.com/khenidak)) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing] - Add support for hugepages to downward API ([#86102](https://github.com/kubernetes/kubernetes/pull/86102), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing] - Adds kubelet alpha feature, `GracefulNodeShutdown` which makes kubelet aware of node system shutdowns and result in graceful termination of pods during a system shutdown. ([#96129](https://github.com/kubernetes/kubernetes/pull/96129), [@bobbypage](https://github.com/bobbypage)) [SIG Node] - AppProtocol is now GA for Endpoints and Services. The ServiceAppProtocol feature gate will be deprecated in 1.21. ([#96327](https://github.com/kubernetes/kubernetes/pull/96327), [@robscott](https://github.com/robscott)) [SIG Apps and Network] - Automatic allocation of NodePorts for services with type LoadBalancer can now be disabled by setting the (new) parameter Service.spec.allocateLoadBalancerNodePorts=false. The default is to allocate NodePorts for services with type LoadBalancer which is the existing behavior. ([#92744](https://github.com/kubernetes/kubernetes/pull/92744), [@uablrek](https://github.com/uablrek)) [SIG Apps and Network] - Certain fields on  Service objects will be automatically cleared when changing the service's `type` to a mode that does not need those fields.  For example, changing from type=LoadBalancer to type=ClusterIP will clear the NodePort assignments, rather than forcing the user to clear them. ([#95196](https://github.com/kubernetes/kubernetes/pull/95196), [@thockin](https://github.com/thockin)) [SIG API Machinery, Apps, Network and Testing] - Document that ServiceTopology feature is required to use `service.spec.topologyKeys`. ([#96528](https://github.com/kubernetes/kubernetes/pull/96528), [@andrewsykim](https://github.com/andrewsykim)) [SIG Apps] - EndpointSlice has a new NodeName field guarded by the EndpointSliceNodeName feature gate. - EndpointSlice topology field will be deprecated in an upcoming release. - EndpointSlice "IP" address type is formally removed after being deprecated in Kubernetes 1.17. - The discovery.k8s.io/v1alpha1 API is deprecated and will be removed in Kubernetes 1.21. ([#96440](https://github.com/kubernetes/kubernetes/pull/96440), [@robscott](https://github.com/robscott)) [SIG API Machinery, Apps and Network] - External facing API podresources is now available under k8s.io/kubelet/pkg/apis/ ([#92632](https://github.com/kubernetes/kubernetes/pull/92632), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) [SIG Node and Testing] - Fewer candidates are enumerated for preemption to improve performance in large clusters. ([#94814](https://github.com/kubernetes/kubernetes/pull/94814), [@adtac](https://github.com/adtac)) - Fix conversions for custom metrics. ([#94481](https://github.com/kubernetes/kubernetes/pull/94481), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Instrumentation] - GPU metrics provided by kubelet are now disabled by default. ([#95184](https://github.com/kubernetes/kubernetes/pull/95184), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) - If BoundServiceAccountTokenVolume is enabled, cluster admins can use metric `serviceaccount_stale_tokens_total` to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting `kube-apiserver` with flag `--service-account-extend-token-expiration=false` ([#96273](https://github.com/kubernetes/kubernetes/pull/96273), [@zshihang](https://github.com/zshihang)) [SIG API Machinery and Auth] - Introduce alpha support for exec-based container registry credential provider plugins in the kubelet. ([#94196](https://github.com/kubernetes/kubernetes/pull/94196), [@andrewsykim](https://github.com/andrewsykim)) [SIG Node and Release] - Introduces a metric source for HPAs which allows scaling based on container resource usage. ([#90691](https://github.com/kubernetes/kubernetes/pull/90691), [@arjunrn](https://github.com/arjunrn)) [SIG API Machinery, Apps, Autoscaling and CLI] - Kube-apiserver now deletes expired kube-apiserver Lease objects: - The feature is under feature gate `APIServerIdentity`. - A flag is added to kube-apiserver: `identity-lease-garbage-collection-check-period-seconds` ([#95895](https://github.com/kubernetes/kubernetes/pull/95895), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery, Apps, Auth and Testing] - Kube-controller-manager: volume plugins can be restricted from contacting local and loopback addresses by setting `--volume-host-allow-local-loopback=false`, or from contacting specific CIDR ranges by setting `--volume-host-cidr-denylist` (for example, `--volume-host-cidr-denylist=127.0.0.1/28,feed::/16`) ([#91785](https://github.com/kubernetes/kubernetes/pull/91785), [@mattcary](https://github.com/mattcary)) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing] - Migrate scheduler, controller-manager and cloud-controller-manager to use LeaseLock ([#94603](https://github.com/kubernetes/kubernetes/pull/94603), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery, Apps, Cloud Provider and Scheduling] - Modify DNS-1123 error messages to indicate that RFC 1123 is not followed exactly ([#94182](https://github.com/kubernetes/kubernetes/pull/94182), [@mattfenwick](https://github.com/mattfenwick)) [SIG API Machinery, Apps, Auth, Network and Node] - Move configurable fsgroup change policy for pods to beta ([#96376](https://github.com/kubernetes/kubernetes/pull/96376), [@gnufied](https://github.com/gnufied)) [SIG Apps and Storage] - New flag is introduced, i.e. --topology-manager-scope=container|pod. The default value is the "container" scope. ([#92967](https://github.com/kubernetes/kubernetes/pull/92967), [@cezaryzukowski](https://github.com/cezaryzukowski)) [SIG Instrumentation, Node and Testing] - New parameter `defaultingType` for `PodTopologySpread` plugin allows to use k8s defined or user provided default constraints ([#95048](https://github.com/kubernetes/kubernetes/pull/95048), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - NodeAffinity plugin can be configured with AddedAffinity. ([#96202](https://github.com/kubernetes/kubernetes/pull/96202), [@alculquicondor](https://github.com/alculquicondor)) [SIG Node, Scheduling and Testing] - Promote RuntimeClass feature to GA. Promote node.k8s.io API groups from v1beta1 to v1. ([#95718](https://github.com/kubernetes/kubernetes/pull/95718), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG Apps, Auth, Node, Scheduling and Testing] - Reminder: The labels "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" are deprecated in favor of "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" respectively.  All users of the "failure-domain.beta..." labels should switch to the "topology..." equivalents. ([#96033](https://github.com/kubernetes/kubernetes/pull/96033), [@thockin](https://github.com/thockin)) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Node, Scheduling, Storage and Testing] - Server Side Apply now treats LabelSelector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. ([#93901](https://github.com/kubernetes/kubernetes/pull/93901), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing] - Services will now have a `clusterIPs` field to go with `clusterIP`.  `clusterIPs[0]` is a synonym for `clusterIP` and will be syncronized on create and update operations. ([#95894](https://github.com/kubernetes/kubernetes/pull/95894), [@thockin](https://github.com/thockin)) [SIG Network] - The ServiceAccountIssuerDiscovery feature gate is now Beta and enabled by default. ([#91921](https://github.com/kubernetes/kubernetes/pull/91921), [@mtaufen](https://github.com/mtaufen)) [SIG Auth] - The status of v1beta1 CRDs without "preserveUnknownFields:false" now shows a violation, "spec.preserveUnknownFields: Invalid value: true: must be false". ([#93078](https://github.com/kubernetes/kubernetes/pull/93078), [@vareti](https://github.com/vareti)) - The usage of mixed protocol values in the same LoadBalancer Service is possible if the new feature gate MixedProtocolLBService is enabled. The feature gate is disabled by default. The user has to enable it for the API Server. ([#94028](https://github.com/kubernetes/kubernetes/pull/94028), [@janosi](https://github.com/janosi)) [SIG API Machinery and Apps] - This PR will introduce a feature gate CSIServiceAccountToken with two additional fields in `CSIDriverSpec`. ([#93130](https://github.com/kubernetes/kubernetes/pull/93130), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing] - Users can try the cronjob controller v2 using the feature gate. This will be the default controller in future releases. ([#93370](https://github.com/kubernetes/kubernetes/pull/93370), [@alaypatel07](https://github.com/alaypatel07)) [SIG API Machinery, Apps, Auth and Testing] - VolumeSnapshotDataSource moves to GA in 1.20 release ([#95282](https://github.com/kubernetes/kubernetes/pull/95282), [@xing-yang](https://github.com/xing-yang)) [SIG Apps] - WinOverlay feature graduated to beta ([#94807](https://github.com/kubernetes/kubernetes/pull/94807), [@ksubrmnn](https://github.com/ksubrmnn)) [SIG Windows] ### 기능(feature) - **Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.**: - A new metric `apiserver_request_filter_duration_seconds` has been introduced that measures request filter latency in seconds. ([#95207](https://github.com/kubernetes/kubernetes/pull/95207), [@tkashem](https://github.com/tkashem)) [SIG API Machinery and Instrumentation] - A new set of alpha metrics are reported by the Kubernetes scheduler under the `/metrics/resources` endpoint that allow administrators to easily see the resource consumption (requests and limits for all resources on the pods) and compare it to actual pod usage or node capacity. ([#94866](https://github.com/kubernetes/kubernetes/pull/94866), [@smarterclayton](https://github.com/smarterclayton)) [SIG API Machinery, Instrumentation, Node and Scheduling] - Add --experimental-logging-sanitization flag enabling runtime protection from leaking sensitive data in logs ([#96370](https://github.com/kubernetes/kubernetes/pull/96370), [@serathius](https://github.com/serathius)) [SIG API Machinery, Cluster Lifecycle and Instrumentation] - Add a StorageVersionAPI feature gate that makes API server update storageversions before serving certain write requests. This feature allows the storage migrator to manage storage migration for built-in resources. Enabling internal.apiserver.k8s.io/v1alpha1 API and APIServerIdentity feature gate are required to use this feature. ([#93873](https://github.com/kubernetes/kubernetes/pull/93873), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery, Auth and Testing] - Add a metric for time taken to perform recursive permission change ([#95866](https://github.com/kubernetes/kubernetes/pull/95866), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation and Storage] - Add a new `vSphere` metric: `cloudprovider_vsphere_vcenter_versions`. It's content show `vCenter` hostnames with the associated server version. ([#94526](https://github.com/kubernetes/kubernetes/pull/94526), [@Danil-Grigorev](https://github.com/Danil-Grigorev)) [SIG Cloud Provider and Instrumentation] - Add a new flag to set priority for the kubelet on Windows nodes so that workloads cannot overwhelm the node there by disrupting kubelet process. ([#96051](https://github.com/kubernetes/kubernetes/pull/96051), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) [SIG Node and Windows] - Add feature to size memory backed volumes ([#94444](https://github.com/kubernetes/kubernetes/pull/94444), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Storage and Testing] - Add foreground cascading deletion to kubectl with the new `kubectl delete foreground|background|orphan` option. ([#93384](https://github.com/kubernetes/kubernetes/pull/93384), [@zhouya0](https://github.com/zhouya0)) - Add metrics for azure service operations (route and loadbalancer). ([#94124](https://github.com/kubernetes/kubernetes/pull/94124), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider and Instrumentation] - Add network rule support in Azure account creation. ([#94239](https://github.com/kubernetes/kubernetes/pull/94239), [@andyzhangx](https://github.com/andyzhangx)) - Add node_authorizer_actions_duration_seconds metric that can be used to estimate load to node authorizer. ([#92466](https://github.com/kubernetes/kubernetes/pull/92466), [@mborsz](https://github.com/mborsz)) [SIG API Machinery, Auth and Instrumentation] - Add pod_ based CPU and memory metrics to Kubelet's  /metrics/resource endpoint ([#95839](https://github.com/kubernetes/kubernetes/pull/95839), [@egernst](https://github.com/egernst)) [SIG Instrumentation, Node and Testing] - Added `get-users` and `delete-user` to the `kubectl config` subcommand ([#89840](https://github.com/kubernetes/kubernetes/pull/89840), [@eddiezane](https://github.com/eddiezane)) [SIG CLI] - Added counter metric "apiserver_request_self" to count API server self-requests with labels for verb, resource, and subresource. ([#94288](https://github.com/kubernetes/kubernetes/pull/94288), [@LogicalShark](https://github.com/LogicalShark)) [SIG API Machinery, Auth, Instrumentation and Scheduling] - Added new k8s.io/component-helpers repository providing shared helper code for (core) components. ([#92507](https://github.com/kubernetes/kubernetes/pull/92507), [@ingvagabund](https://github.com/ingvagabund)) [SIG Apps, Node, Release and Scheduling] - Adds `create ingress` command to `kubectl` ([#78153](https://github.com/kubernetes/kubernetes/pull/78153), [@amimof](https://github.com/amimof)) [SIG CLI and Network] - Adds a headless service on node-local-cache addon. ([#88412](https://github.com/kubernetes/kubernetes/pull/88412), [@stafot](https://github.com/stafot)) [SIG Cloud Provider and Network] - Allow cross compilation of kubernetes on different platforms. ([#94403](https://github.com/kubernetes/kubernetes/pull/94403), [@bnrjee](https://github.com/bnrjee)) [SIG Release] - Azure: Support multiple services sharing one IP address ([#94991](https://github.com/kubernetes/kubernetes/pull/94991), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - CRDs: For structural schemas, non-nullable null map fields will now be dropped and defaulted if a default is available. null items in list will continue being preserved, and fail validation if not nullable. ([#95423](https://github.com/kubernetes/kubernetes/pull/95423), [@apelisse](https://github.com/apelisse)) [SIG API Machinery] - Changed: default "Accept: */*" header added to HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (https://github.com/kubernetes/website/pull/24756) ([#95641](https://github.com/kubernetes/kubernetes/pull/95641), [@fonsecas72](https://github.com/fonsecas72)) [SIG Network and Node] - Client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. ([#95489](https://github.com/kubernetes/kubernetes/pull/95489), [@ankeesler](https://github.com/ankeesler)) [SIG API Machinery and Auth] - Command to start network proxy changes from 'KUBE_ENABLE_EGRESS_VIA_KONNECTIVITY_SERVICE ./cluster/kube-up.sh' to 'KUBE_ENABLE_KONNECTIVITY_SERVICE=true ./hack/kube-up.sh' ([#92669](https://github.com/kubernetes/kubernetes/pull/92669), [@Jefftree](https://github.com/Jefftree)) [SIG Cloud Provider] - Configure AWS LoadBalancer health check protocol via service annotations. ([#94546](https://github.com/kubernetes/kubernetes/pull/94546), [@kishorj](https://github.com/kishorj)) - DefaultPodTopologySpread graduated to Beta. The feature gate is enabled by default. ([#95631](https://github.com/kubernetes/kubernetes/pull/95631), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling and Testing] - E2e test for PodFsGroupChangePolicy ([#96247](https://github.com/kubernetes/kubernetes/pull/96247), [@saikat-royc](https://github.com/saikat-royc)) [SIG Storage and Testing] - Ephemeral containers now apply the same API defaults as initContainers and containers ([#94896](https://github.com/kubernetes/kubernetes/pull/94896), [@wawa0210](https://github.com/wawa0210)) [SIG Apps and CLI] - Gradudate the Pod Resources API to G.A Introduces the pod_resources_endpoint_requests_total metric which tracks the total number of requests to the pod resources API ([#92165](https://github.com/kubernetes/kubernetes/pull/92165), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) [SIG Instrumentation, Node and Testing] - In dual-stack bare-metal clusters, you can now pass dual-stack IPs to `kubelet --node-ip`. eg: `kubelet --node-ip 10.1.0.5,fd01::0005`. This is not yet supported for non-bare-metal clusters. In dual-stack clusters where nodes have dual-stack addresses, hostNetwork pods will now get dual-stack PodIPs. ([#95239](https://github.com/kubernetes/kubernetes/pull/95239), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] - Introduce api-extensions category which will return: mutating admission configs, validating admission configs, CRDs and APIServices when used in kubectl get, for example. ([#95603](https://github.com/kubernetes/kubernetes/pull/95603), [@soltysh](https://github.com/soltysh)) [SIG API Machinery] - Introduces a new GCE specific cluster creation variable KUBE_PROXY_DISABLE. When set to true, this will skip over the creation of kube-proxy (whether the daemonset or static pod). This can be used to control the lifecycle of kube-proxy separately from the lifecycle of the nodes. ([#91977](https://github.com/kubernetes/kubernetes/pull/91977), [@varunmar](https://github.com/varunmar)) [SIG Cloud Provider] - Kube-apiserver now maintains a Lease object to identify itself: - The feature is under feature gate `APIServerIdentity`. - Two flags are added to kube-apiserver: `identity-lease-duration-seconds`, `identity-lease-renew-interval-seconds` ([#95533](https://github.com/kubernetes/kubernetes/pull/95533), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] - Kube-apiserver: The timeout used when making health check calls to etcd can now be configured with `--etcd-healthcheck-timeout`. The default timeout is 2 seconds, matching the previous behavior. ([#93244](https://github.com/kubernetes/kubernetes/pull/93244), [@Sh4d1](https://github.com/Sh4d1)) [SIG API Machinery] - Kube-apiserver: added support for compressing rotated audit log files with `--audit-log-compress` ([#94066](https://github.com/kubernetes/kubernetes/pull/94066), [@lojies](https://github.com/lojies)) [SIG API Machinery and Auth] - Kubeadm now prints warnings instead of throwing errors if the current system time is outside of the NotBefore and NotAfter bounds of a loaded certificate.  ([#94504](https://github.com/kubernetes/kubernetes/pull/94504), [@neolit123](https://github.com/neolit123)) - Kubeadm: Add a preflight check that the control-plane node has at least 1700MB of RAM ([#93275](https://github.com/kubernetes/kubernetes/pull/93275), [@xlgao-zju](https://github.com/xlgao-zju)) [SIG Cluster Lifecycle] - Kubeadm: add the "--cluster-name" flag to the "kubeadm alpha kubeconfig user" to allow configuring the cluster name in the generated kubeconfig file ([#93992](https://github.com/kubernetes/kubernetes/pull/93992), [@prabhu43](https://github.com/prabhu43)) [SIG Cluster Lifecycle] - Kubeadm: add the "--kubeconfig" flag to the "kubeadm init phase upload-certs" command to allow users to pass a custom location for a kubeconfig file. ([#94765](https://github.com/kubernetes/kubernetes/pull/94765), [@zhanw15](https://github.com/zhanw15)) [SIG Cluster Lifecycle] - Kubeadm: make etcd pod request 100m CPU, 100Mi memory and 100Mi ephemeral_storage by default ([#94479](https://github.com/kubernetes/kubernetes/pull/94479), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] - Kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags: - apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration. - cluster-name: use clusterName from ClusterConfiguration - cert-dir: use certificatesDir from ClusterConfiguration ([#94879](https://github.com/kubernetes/kubernetes/pull/94879), [@knight42](https://github.com/knight42)) [SIG Cluster Lifecycle] - Kubectl create now supports creating ingress objects. ([#94327](https://github.com/kubernetes/kubernetes/pull/94327), [@rikatz](https://github.com/rikatz)) [SIG CLI and Network] - Kubectl rollout history sts/sts-name --revision=some-revision will start showing the detailed view of  the sts on that specified revision ([#86506](https://github.com/kubernetes/kubernetes/pull/86506), [@dineshba](https://github.com/dineshba)) [SIG CLI] - Kubectl: Previously users cannot provide arguments to a external diff tool via KUBECTL_EXTERNAL_DIFF env. This release now allow users to specify args to KUBECTL_EXTERNAL_DIFF env. ([#95292](https://github.com/kubernetes/kubernetes/pull/95292), [@dougsland](https://github.com/dougsland)) [SIG CLI] - Kubemark now supports both real and hollow nodes in a single cluster. ([#93201](https://github.com/kubernetes/kubernetes/pull/93201), [@ellistarn](https://github.com/ellistarn)) [SIG Scalability] - Kubernetes E2E test image manifest lists now contain Windows images. ([#77398](https://github.com/kubernetes/kubernetes/pull/77398), [@claudiubelu](https://github.com/claudiubelu)) [SIG Testing and Windows] - Kubernetes is now built using go1.15.2 - build: Update to k/repo-infra@v0.1.1 (supports go1.15.2) - build: Use go-runner:buster-v2.0.1 (built using go1.15.1) - bazel: Replace --features with Starlark build settings flag - hack/lib/util.sh: some bash cleanups - switched one spot to use kube::logging - make kube::util::find-binary return an error when it doesn't find anything so that hack scripts fail fast instead of with '' binary not found errors. - this required deleting some genfeddoc stuff. the binary no longer exists in k/k repo since we removed federation/, and I don't see it in https://github.com/kubernetes-sigs/kubefed/ either. I'm assuming that it's gone for good now. - bazel: output go_binary rule directly from go_binary_conditional_pure From: [@mikedanese](https://github.com/mikedanese): Instead of aliasing. Aliases are annoying in a number of ways. This is specifically bugging me now because they make the action graph harder to analyze programmatically. By using aliases here, we would need to handle potentially aliased go_binary targets and dereference to the effective target. The comment references an issue with `pure = select(...)` which appears to be resolved considering this now builds. - make kube::util::find-binary not dependent on bazel-out/ structure Implement an aspect that outputs go_build_mode metadata for go binaries, and use that during binary selection. ([#94449](https://github.com/kubernetes/kubernetes/pull/94449), [@justaugustus](https://github.com/justaugustus)) [SIG Architecture, CLI, Cluster Lifecycle, Node, Release and Testing] - Kubernetes is now built using go1.15.5 - build: Update to k/repo-infra@v0.1.2 (supports go1.15.5) ([#95776](https://github.com/kubernetes/kubernetes/pull/95776), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] - New default scheduling plugins order reduces scheduling and preemption latency when taints and node affinity are used ([#95539](https://github.com/kubernetes/kubernetes/pull/95539), [@soulxu](https://github.com/soulxu)) [SIG Scheduling] - Only update Azure data disks when attach/detach ([#94265](https://github.com/kubernetes/kubernetes/pull/94265), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - Promote SupportNodePidsLimit to GA to provide node-to-pod PID isolation. Promote SupportPodPidsLimit to GA to provide ability to limit PIDs per pod. ([#94140](https://github.com/kubernetes/kubernetes/pull/94140), [@derekwaynecarr](https://github.com/derekwaynecarr)) - SCTP support in API objects (Pod, Service, NetworkPolicy) is now GA. Note that this has no effect on whether SCTP is enabled on nodes at the kernel level, and note that some cloud platforms and network plugins do not support SCTP traffic. ([#95566](https://github.com/kubernetes/kubernetes/pull/95566), [@danwinship](https://github.com/danwinship)) [SIG Apps and Network] - Scheduler now ignores Pod update events if the resourceVersion of old and new Pods are identical. ([#96071](https://github.com/kubernetes/kubernetes/pull/96071), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] - Scheduling Framework: expose Run[Pre]ScorePlugins functions to PreemptionHandle which can be used in PostFilter extention point. ([#93534](https://github.com/kubernetes/kubernetes/pull/93534), [@everpeace](https://github.com/everpeace)) [SIG Scheduling and Testing] - SelectorSpreadPriority maps to PodTopologySpread plugin when DefaultPodTopologySpread feature is enabled ([#95448](https://github.com/kubernetes/kubernetes/pull/95448), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - Send GCE node startup scripts logs to console and journal. ([#95311](https://github.com/kubernetes/kubernetes/pull/95311), [@karan](https://github.com/karan)) - SetHostnameAsFQDN has been graduated to Beta and therefore it is enabled by default. ([#95267](https://github.com/kubernetes/kubernetes/pull/95267), [@javidiaz](https://github.com/javidiaz)) [SIG Node] - Support [service.beta.kubernetes.io/azure-pip-ip-tags] annotations to allow customers to specify ip-tags to influence public-ip creation in Azure [Tag1=Value1, Tag2=Value2, etc.] ([#94114](https://github.com/kubernetes/kubernetes/pull/94114), [@MarcPow](https://github.com/MarcPow)) [SIG Cloud Provider] - Support custom tags for cloud provider managed resources ([#96450](https://github.com/kubernetes/kubernetes/pull/96450), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - Support customize load balancer health probe protocol and request path ([#96338](https://github.com/kubernetes/kubernetes/pull/96338), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - Support for Windows container images (OS Versions: 1809, 1903, 1909, 2004) was added the pause:3.4 image. ([#91452](https://github.com/kubernetes/kubernetes/pull/91452), [@claudiubelu](https://github.com/claudiubelu)) [SIG Node, Release and Windows] - Support multiple standard load balancers in one cluster ([#96111](https://github.com/kubernetes/kubernetes/pull/96111), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - The beta `RootCAConfigMap` feature gate is enabled by default and causes kube-controller-manager to publish a "kube-root-ca.crt" ConfigMap to every namespace. This ConfigMap contains a CA bundle used for verifying connections to the kube-apiserver. ([#96197](https://github.com/kubernetes/kubernetes/pull/96197), [@zshihang](https://github.com/zshihang)) [SIG API Machinery, Apps, Auth and Testing] - The kubelet_runtime_operations_duration_seconds metric buckets were set to 0.005 0.0125 0.03125 0.078125 0.1953125 0.48828125 1.220703125 3.0517578125 7.62939453125 19.073486328125 47.6837158203125 119.20928955078125 298.0232238769531 and 745.0580596923828 seconds ([#96054](https://github.com/kubernetes/kubernetes/pull/96054), [@alvaroaleman](https://github.com/alvaroaleman)) [SIG Instrumentation and Node] - There is a new pv_collector_total_pv_count metric that counts persistent volumes by the volume plugin name and volume mode. ([#95719](https://github.com/kubernetes/kubernetes/pull/95719), [@tsmetana](https://github.com/tsmetana)) [SIG Apps, Instrumentation, Storage and Testing] - Volume snapshot e2e test to validate PVC and VolumeSnapshotContent finalizer ([#95863](https://github.com/kubernetes/kubernetes/pull/95863), [@RaunakShah](https://github.com/RaunakShah)) [SIG Cloud Provider, Storage and Testing] - Warns user when executing kubectl apply/diff to resource currently being deleted. ([#95544](https://github.com/kubernetes/kubernetes/pull/95544), [@SaiHarshaK](https://github.com/SaiHarshaK)) [SIG CLI] - `kubectl alpha debug` has graduated to beta and is now `kubectl debug`. ([#96138](https://github.com/kubernetes/kubernetes/pull/96138), [@verb](https://github.com/verb)) [SIG CLI and Testing] - `kubectl debug` gains support for changing container images when copying a pod for debugging, similar to how `kubectl set image` works. See `kubectl help debug` for more information. ([#96058](https://github.com/kubernetes/kubernetes/pull/96058), [@verb](https://github.com/verb)) [SIG CLI] </ins> ### 문서 <del> - 샘플 앱 배포 지침이 변경되었다. ([#82785](https://github.com/kubernetes/kubernetes/pull/82785), [@ashish-billore](https://github.com/ashish-billore)) [SIG API Machinery] </del> <ins> - Fake dynamic client: document that List does not preserve TypeMeta in UnstructuredList ([#95117](https://github.com/kubernetes/kubernetes/pull/95117), [@andrewsykim](https://github.com/andrewsykim)) [SIG API Machinery] - Kubelet: remove alpha warnings for CNI flags. ([#94508](https://github.com/kubernetes/kubernetes/pull/94508), [@andrewsykim](https://github.com/andrewsykim)) [SIG Network and Node] - Updates docs and guidance on cloud provider InstancesV2 and Zones interface for external cloud providers: - removes experimental warning for InstancesV2 - document that implementation of InstancesV2 will disable calls to Zones - deprecate Zones in favor of InstancesV2 ([#96397](https://github.com/kubernetes/kubernetes/pull/96397), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider] </ins> ### 실패 테스트 <del> - Kube-proxy iptables min-sync-period 의 기본값은 1초로 변경되었다. (이전 값은 0 이다.) ([#92836](https://github.com/kubernetes/kubernetes/pull/92836), [@aojea](https://github.com/aojea)) [SIG Network] ### 버그 또는 회귀(Regression) - 인-트리 소스의 PV 집합은 CSIPersistentVolumeSource로 변환될 때 노드 어피니티에서 주문된 요구사항 값을 갖는다. ([#88987](https://github.com/kubernetes/kubernetes/pull/88987), [@jiahuif](https://github.com/jiahuif)) [SIG Storage] - `informer-sync` 헬스 체커로 인한 apiserver의 패닉이 수정되었다. ([#93600](https://github.com/kubernetes/kubernetes/pull/93600), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG API Machinery] - 이제 GCP cloud-controller-manager 가 클러스터 외부에서 실행되어 새 노드를 초기화하지 못하는 문제가 수정되었다. ([#90057](https://github.com/kubernetes/kubernetes/pull/90057), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Apps 및 Cloud Provider] - Kubelets 용 GCE CloudProvider를 초기화 할 때 GCE API 호출을 하지 않는다. ([#90218](https://github.com/kubernetes/kubernetes/pull/90218), [@wojtek-t](https://github.com/wojtek-t)) [SIG Cloud Provider 및 Scalability] - IP 별칭을 추가하거나 GCE 클라우드 제공 업체의 노드 객체에 반영할 때, 불필요한 GCE API 호출을 하지 않는다. ([#90242](https://github.com/kubernetes/kubernetes/pull/90242), [@wojtek-t](https://github.com/wojtek-t)) [SIG Apps, Cloud Provider 및 Network] - 파드가 스케줄 되는 동안, 어노테이션이 변경될 때 불필요한 스케줄링 변동을 방지한다. ([#90373](https://github.com/kubernetes/kubernetes/pull/90373), [@fabiokung](https://github.com/fabiokung)) [SIG Scheduling] - kubectl 용 Azure 인증 모듈은 이제 갱신 토큰이 만료된 후에 로그인을 요청한다. ([#86481](https://github.com/kubernetes/kubernetes/pull/86481), [@tdihp](https://github.com/tdihp)) [SIG API Machinery 및 Auth] - Azure: lb 생성시 발생하는 동시성 해결 ([#89604](https://github.com/kubernetes/kubernetes/pull/89604), [@aramase](https://github.com/aramase)) [SIG Cloud Provider] - Azure: 연결된 VMSS 가 많은 클러스터에서 스로틀링(throttling)을 방지하기 위해 전역 VMSS VMs 캐시를 VMSS 별 VM 캐시로 전환하였다. ([#93107](https://github.com/kubernetes/kubernetes/pull/93107), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider] - Azure: IPv6 인바운드 보안 규칙에 대한 대상 접두사 및 포트를 설정하였다. ([#91831](https://github.com/kubernetes/kubernetes/pull/91831), [@aramase](https://github.com/aramase)) [SIG Cloud Provider] - 베이스-이미지: kube-cross:v1.13.9-5 으로 업데이트 되었다. ([#90963](https://github.com/kubernetes/kubernetes/pull/90963), [@justaugustus](https://github.com/justaugustus)) [SIG Release 및 Testing] - 기존 servicePort의 nodePort가 수동으로 변경된 경우 AWS NLB 서비스에 대한 버그가 수정되었다. ([#89562](https://github.com/kubernetes/kubernetes/pull/89562), [@M00nF1sh](https://github.com/M00nF1sh)) [SIG Cloud Provider] - APIServer에 연결할 수 없거나 kubelet에 아직 올바른 자격증명이 없는 경우, CSINode 초기화는 시작할때 kubelet을 중단하지 않는다. ([#89589](https://github.com/kubernetes/kubernetes/pull/89589), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] - CVE-2020-8557 (Medium): 컨테이너의 /etc/hosts 파일을 통한 노드-로컬 간 서비스 거부. 자세한 내용은 https://github.com/kubernetes/kubernetes/issues/93032 을 참조한다. ([#92916](https://github.com/kubernetes/kubernetes/pull/92916), [@joelsmith](https://github.com/joelsmith)) [SIG Node] - Client-go: watch를 재구성하는 대신 타임아웃이 발생하면, 정보 제공자가 전체 목록 요청으로 돌아가는 문제가 해결되었다. ([#89652](https://github.com/kubernetes/kubernetes/pull/89652), [@liggitt](https://github.com/liggitt)) [SIG API Machinery 및 Testing] - CloudNodeLifecycleController는 노드 모니터링을 할 때, 셧다운 상태 전에 노드의 존재 상태를 확인하게 된다. ([#90737](https://github.com/kubernetes/kubernetes/pull/90737), [@jiahuif](https://github.com/jiahuif)) [SIG Apps 및 Cloud Provider] - `startupProbe` 는 지정하지만 `readinessProbe` 는 지정하지 않은 컨테이너는 이전에는 `startupProbe` 가 완료되기 전에 "준비(ready)"된 것으로 간주되었지만 이제는 "준비되지 않은(not-ready)" 것으로 간주된다. ([#92196](https://github.com/kubernetes/kubernetes/pull/92196), [@thockin](https://github.com/thockin)) [SIG Node] - 이제 Cordon 된 노드가 AWS 대상 그룹에서 등록 취소된다. ([#85920](https://github.com/kubernetes/kubernetes/pull/85920), [@hoelzro](https://github.com/hoelzro)) [SIG Cloud Provider] - kubernetes.azure.com/managed=false 레이블이 지정된 노드를 로드 밸런서의 백엔드 풀에 추가하지 않는다. ([#93034](https://github.com/kubernetes/kubernetes/pull/93034), [@matthias50](https://github.com/matthias50)) [SIG Cloud Provider] - CSI 드라이버가 FailedPrecondition 오류를 반환하는 경우 볼륨 확장을 재시도 하지 않는다. ([#92986](https://github.com/kubernetes/kubernetes/pull/92986), [@gnufied](https://github.com/gnufied)) [SIG Node 및 Storage] - Dockershim 보안: 이제 파드 샌드박스가 항상 `no-new-privileges` 및 `runtime/default` seccomp 프로필로 실행된다. dockershim seccomp: 이제 파드 수준에서 설정할 때 사용자 지정 프로필이 더 작은 seccomp 프로필을 가질 수 있다. ([#90948](https://github.com/kubernetes/kubernetes/pull/90948), [@pjbgf](https://github.com/pjbgf)) [SIG Node] - 이중-스택: 서비스 clusterIP가 지정된 ipFamily를 따르지 않는 버그가 수정되었다. ([#89612](https://github.com/kubernetes/kubernetes/pull/89612), [@SataQiu](https://github.com/SataQiu)) [SIG Network] - EndpointSliceMirroring 컨트롤러는 이제 엔드포인트에서 엔드포인트슬라이스로 레이블을 복사한다. ([#93442](https://github.com/kubernetes/kubernetes/pull/93442), [@robscott](https://github.com/robscott)) [SIG Apps 및 Network] - Azure 가용성 영역(availability zone)이 항상 소문자인지 확인한다. ([#89722](https://github.com/kubernetes/kubernetes/pull/89722), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] - DeletionTimestamp가 0이 아닌 파드에 대한 축출 요청이 항상 성공하게 된다. ([#91342](https://github.com/kubernetes/kubernetes/pull/91342), [@michaelgugino](https://github.com/michaelgugino)) [SIG Apps] - 리소스 이름이 내장 오브젝트와 동일한 CRD를 설명한다. ([#89505](https://github.com/kubernetes/kubernetes/pull/89505), [@knight42](https://github.com/knight42)) [SIG API Machinery, CLI 및 Testing] - 내부 정보가 동기화 되었는지를 확인하는 새로운 "informer-sync" 검사로 kube-apiserver의 /readyz 를 확장한다. ([#92644](https://github.com/kubernetes/kubernetes/pull/92644), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery 및 Testing] - winkernel kube-proxy 에 있는 DSR 로드밸런서 버전이 HNS 버전 9.3-9.max, 10.2+ 로 확장되었다. ([#93080](https://github.com/kubernetes/kubernetes/pull/93080), [@elweb9858](https://github.com/elweb9858)) [SIG Network] - 필수 어피니티 용어가 있는 첫번째 파드는 토폴로지 키가 일치하는 노드에서만 예약할 수 있게 된다. ([#91168](https://github.com/kubernetes/kubernetes/pull/91168), [@ahg-g](https://github.com/ahg-g)) [SIG Scheduling] - 연결 해제된 상태의 CIDR이 존재하는 경우에 대한 AWS 로드밸런서 VPC CIDR 계산이 수정되었다. ([#92227](https://github.com/kubernetes/kubernetes/pull/92227), [@M00nF1sh](https://github.com/M00nF1sh)) [SIG Cloud Provider] - 관리되지 않는 노드에 대한 InstanceMetadataByProviderID 가 수정되었다. ([#92572](https://github.com/kubernetes/kubernetes/pull/92572), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] - 고객이 VMSS orchestrationMode를 설정한 경우 `VirtualMachineScaleSets.virtualMachines.GET` 이 허용되지 않았던 문제가 수정되었다. ([#91097](https://github.com/kubernetes/kubernetes/pull/91097), [@feiskyer](https://github.com/feiskyer)) - 앞에 0이 있는 IPv6 주소 사용을 허용하지 않던 버그가 수정되었다. ([#89341](https://github.com/kubernetes/kubernetes/pull/89341), [@aojea](https://github.com/aojea)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle 및 Instrumentation] - ExternalTrafficPolicy 가 ExternalIPs 서비스에 적용되지 않던 버그를 수정하였다. ([#90537](https://github.com/kubernetes/kubernetes/pull/90537), [@freehan](https://github.com/freehan)) [SIG Network] - VMSS 캐시에서 nil VM 항목이 만료될 때의 조건이 수정되었다. ([#92681](https://github.com/kubernetes/kubernetes/pull/92681), [@ArchangelSDY](https://github.com/ArchangelSDY)) [SIG Cloud Provider] - 스케줄러가 불필요한 스케줄 시도를 수행할 수 있는 레이싱(racing) 이슈가 수정되었다. ([#90660](https://github.com/kubernetes/kubernetes/pull/90660), [@Huang-Wei](https://github.com/Huang-Wei)) - 수정된 컨피그맵 또는 시크릿 하위 경로(subpath)의 볼륨 마운트를 사용하여 컨테이너가 다시 시작되는 문제가 수정되었다. ([#89629](https://github.com/kubernetes/kubernetes/pull/89629), [@fatedier](https://github.com/fatedier)) [SIG Architecture, Storage 및 Testing] - 다중 마스터 HA 클러스터에서 정적으로 할당 된 portNumber가 있는 NodePort 생성이 충돌하는 포트 할당 로직 버그가 수정되었다. ([#89937](https://github.com/kubernetes/kubernetes/pull/89937), [@aojea](https://github.com/aojea)) [SIG Network 및 Testing] - xfs_repair가 xfs 마운트를 중지하는 버그가 수정되었다. ([#89444](https://github.com/kubernetes/kubernetes/pull/89444), [@gnufied](https://github.com/gnufied)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation 및 Storage] - 클러스터 덤프 정보에 대한 네임스페이스 플래그가 작동하지 않는 문제가 수정되었다. ([#91890](https://github.com/kubernetes/kubernetes/pull/91890), [@zhouya0](https://github.com/zhouya0)) [SIG CLI] - OOM 으로 컨테이너가 손상된 경우, SystemOOM 감지에 대한 버그가 수정되었다. ([#88871](https://github.com/kubernetes/kubernetes/pull/88871), [@dashpole](https://github.com/dashpole)) [SIG Node] - 이미지 파일 시스템 감지, devicemapper에 대한 디스크 메트릭, 5.0+ 리눅스 커널에서 OOM Kill 감지에 대한 버그가 수정되었다. ([#92919](https://github.com/kubernetes/kubernetes/pull/92919), [@dashpole](https://github.com/dashpole)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation 및 Node] - etcd 이미지에서 etcd 버전 마이그레이션 스크립트가 수정되었다. ([#91925](https://github.com/kubernetes/kubernetes/pull/91925), [@wenjiaswe](https://github.com/wenjiaswe)) [SIG API Machinery] - Azure 파일 CSI 번역의 결함이 수정되었다. ([#90162](https://github.com/kubernetes/kubernetes/pull/90162), [@rfranzke](https://github.com/rfranzke)) [SIG Release 및 Storage] - 짧은 시간에 Azure 노드를 다시 만들 경우, 인스턴스를 찾을 수 없는 문제가 수정되었다. ([#93316](https://github.com/kubernetes/kubernetes/pull/93316), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] - 지원되는 거대 페이지의 크기를 변경할 시 발생하는 문제가 수정되었다. ([#80831](https://github.com/kubernetes/kubernetes/pull/80831), [@odinuge](https://github.com/odinuge)) [SIG Node 및 Testing] - 준비 상태(readiness)를 보고하기 전까지 APIServices가 HTTP 핸들러에 설치될 때까지 기다리도록 kube-apiserver 시작 작업이 수정되었다. ([#89147](https://github.com/kubernetes/kubernetes/pull/89147), [@sttts](https://github.com/sttts)) [SIG API Machinery] - kubectl create --dryrun 클라이언트가 네임스페이스를 무시하는 문제가 수정되었다. ([#90502](https://github.com/kubernetes/kubernetes/pull/90502), [@zhouya0](https://github.com/zhouya0)) - kubectl create secret docker-registry 명령어의 --from-file 플래그를 사용할 수 없는 문제가 수정되었다. ([#90960](https://github.com/kubernetes/kubernetes/pull/90960), [@zhouya0](https://github.com/zhouya0)) [SIG CLI 및 Testing] - kubectl describe CSINode 명령어의 nil pointer 에러가 수정되었다. ([#89646](https://github.com/kubernetes/kubernetes/pull/89646), [@zhouya0](https://github.com/zhouya0)) [SIG CLI] - 리스(Lease) 정보에 접근할 수 없는 사용자를 위해 kubectl describe node 명령어가 수정되었다. ([#90469](https://github.com/kubernetes/kubernetes/pull/90469), [@uthark](https://github.com/uthark)) [SIG CLI] - 빈 어노테이션에 대한 kubectl describe 명령어의 출력 형식이 수정되었다. ([#91405](https://github.com/kubernetes/kubernetes/pull/91405), [@iyashu](https://github.com/iyashu)) [SIG CLI] - 실제로 패치를 지속하지 않도록 kubectl diff 명령어가 수정되었다. ([#89795](https://github.com/kubernetes/kubernetes/pull/89795), [@julianvmodesto](https://github.com/julianvmodesto)) [SIG CLI 및 Testing] - kubectl run 명령어의 --dry-run=client 플래그가 네임스페이스를 무시하는 문제가 수정되었다. ([#90785](https://github.com/kubernetes/kubernetes/pull/90785), [@zhouya0](https://github.com/zhouya0)) [SIG CLI] - kubectl version 명령이 설정 파일 없이 버전 정보를 표시하도록 변경되었다. ([#89913](https://github.com/kubernetes/kubernetes/pull/89913), [@zhouya0](https://github.com/zhouya0)) [SIG API Machinery 및 CLI] - `kubectl alpha debug` 명령어의 `--container` 플래그에 대하여 `-c` 축약표현이 누락된 문제가 수정되었다. ([#89674](https://github.com/kubernetes/kubernetes/pull/89674), [@superbrothers](https://github.com/superbrothers)) [SIG CLI] - 오브젝트의 평균 값이 무시되어 출력되는 문제가 수정되었다. ([#89142](https://github.com/kubernetes/kubernetes/pull/89142), [@zhouya0](https://github.com/zhouya0)) [SIG API Machinery] - Azure VMs에 공용 IP를 할당 한 후 공용 IP가 표시되지 않는 문제가 수정되었다. ([#90886](https://github.com/kubernetes/kubernetes/pull/90886), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] - 파드보다 먼저 노드를 제거할 때 스케줄러의 충돌이 일어나는 문제를 수정하였다. ([#89908](https://github.com/kubernetes/kubernetes/pull/89908), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - 로드밸런서 backendPools 에 대해 Azure VMSS를 업데이트 할 때 VMSS 이름 및 리소스 그룹의 이름 수정에 대한 버그가 수정되었다. ([#89337](https://github.com/kubernetes/kubernetes/pull/89337), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] - Azure VM 컴퓨터 이름의 접두사가 VMSS 이름과 다른 경우의 스로틀링 문제가 해결되었다. ([#92793](https://github.com/kubernetes/kubernetes/pull/92793), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] - 수정: Azure 할당 해제 노드가 종료된 것으로 간주 ([#92257](https://github.com/kubernetes/kubernetes/pull/92257), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - 수정: azure 디스크 PV에 대한 GetLabelsForVolume 패닉 문제 ([#92166](https://github.com/kubernetes/kubernetes/pull/92166), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - 수정: 어노테이션 지원에 Azure 파일 마이그레이션 지원 추가 ([#91093](https://github.com/kubernetes/kubernetes/pull/91093), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Node] - 수정: API 스로틀링(throttling)을 유발할 수 있는 Azure 디스크 댕글링(dangling) 연결 문제 ([#90749](https://github.com/kubernetes/kubernetes/pull/90749), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - 수정: ip 제품군을 기반으로 올바른 ip 결정 ([#93043](https://github.com/kubernetes/kubernetes/pull/93043), [@aramase](https://github.com/aramase)) [SIG Cloud Provider] - 수정: 비어있는 경우 도커 구성 캐시를 사용하지 않음 ([#92330](https://github.com/kubernetes/kubernetes/pull/92330), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - 수정: Azure 디스크 스토리지 클래스 마이그레이션에서 토폴로지 문제 수정 ([#91196](https://github.com/kubernetes/kubernetes/pull/91196), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - 수정: 최대 개수 테이블에 항목이 누락되어 디스크 연결 오류 발생 ([#89768](https://github.com/kubernetes/kubernetes/pull/89768), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Storage] - 수정: 잘못된 최대 Azure 디스크 최대 개수 ([#92331](https://github.com/kubernetes/kubernetes/pull/92331), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Storage] - 수정: Azure 디스크 및 파일 마운트 시 초기 지연 ([#93052](https://github.com/kubernetes/kubernetes/pull/93052), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider 및 Storage] - 수정: Azure에서 삭제된 비 VMSS 인스턴스에 의해 지원되는 노드 제거 지원 ([#91184](https://github.com/kubernetes/kubernetes/pull/91184), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider] - 수정: Azure 디스크에 강제 분리 사용 ([#91948](https://github.com/kubernetes/kubernetes/pull/91948), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - wait.Forever에서 첫 번째 반복에서 백오프 기간을 건너 뛰는 1.18 회귀 문제가 수정되었다. ([#90476](https://github.com/kubernetes/kubernetes/pull/90476), [@zhan849](https://github.com/zhan849)) [SIG API Machinery] - 엔드포인트 슬라이스 업데이트에서 newObj 를 oldObj 로 잘못 사용하는 버그가 수정되었다. ([#92339](https://github.com/kubernetes/kubernetes/pull/92339), [@fatkun](https://github.com/fatkun)) [SIG Apps 및 Network] - 중첩 범위가 있는 jsonpath 출력 표현식으로 kubectl 명령을 실행하면, 중첩 범위 다음의 표현식이 무시되는 버그가 수정되었다. ([#88464](https://github.com/kubernetes/kubernetes/pull/88464), [@brianpursley](https://github.com/brianpursley)) [SIG API Machinery] - TopologyManager가 활성화 되었을 때, 재사용 가능한 CPU 및 장치 할당이 적용되지 않는 버그가 수정되었다. ([#93189](https://github.com/kubernetes/kubernetes/pull/93189), [@klueska](https://github.com/klueska)) [SIG Node] - 깊이 중첩된 오브젝트에 json 패치를 적용하는 성능 문제가 수정되었다. ([#92069](https://github.com/kubernetes/kubernetes/pull/92069), [@tapih](https://github.com/tapih)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle 및 Instrumentation] - RBAC 역할 및 바인딩 오브젝트의 가비지 수집을 방지하는 회귀 문제가 수정되었다. ([#90534](https://github.com/kubernetes/kubernetes/pull/90534), [@apelisse](https://github.com/apelisse)) [SIG Auth] - kubeconfig 파일이 없을 때 --local 또는 --dry-run 플래그를 사용하여 kubectl 명령을 실행하는 회귀 문제가 수정되었다. ([#90243](https://github.com/kubernetes/kubernetes/pull/90243), [@soltysh](https://github.com/soltysh)) [SIG API Machinery, CLI 및 Testing] - 베어러 토큰(kubectl --token=..)과 exec 자격증명 플러그인이 동일한 컨텍스트로 구성되었을 때 모호하게 동작하는 문제가 수정되었다. 이제 전달자 토큰이 우선하게 된다. ([#91745](https://github.com/kubernetes/kubernetes/pull/91745), [@anderseknert](https://github.com/anderseknert)) [SIG API Machinery, Auth 및 Testing] - 이름에 `.` 문자가 포함된 서비스 어카운트의 사용자 인증 정보 마운트 문제가 수정되었다. ([#89696](https://github.com/kubernetes/kubernetes/pull/89696), [@nabokihms](https://github.com/nabokihms)) [SIG Auth] - 노드 삭제 시 파드의 nominatedNodeName을 지울 수 없는 문제가 수정되었다. ([#91750](https://github.com/kubernetes/kubernetes/pull/91750), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling 및 Testing] - zsh 완료가 성공적으로 초기화되었지만 zsh 완료를 초기화 할때 0이 아닌 종료 코드가 반환되는 버그가 수정되었다. ([#88165](https://github.com/kubernetes/kubernetes/pull/88165), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] - endpointSliceTracker의 메모리 누수 문제가 수정되었다. ([#92838](https://github.com/kubernetes/kubernetes/pull/92838), [@tnqn](https://github.com/tnqn)) [SIG Apps 및 Network] - iSCSI 및 FibreChannel 볼륨 플러그인의 고정 마운트 옵션이 수정되었다. ([#89172](https://github.com/kubernetes/kubernetes/pull/89172), [@jsafrane](https://github.com/jsafrane)) [SIG Storage] - 영역 전체의 노드 수에 불균형이 있는 클러스터의 kube-scheduler 에서 손실된 노드 데이터에 대한 문제가 수정되었다. ([#93355](https://github.com/kubernetes/kubernetes/pull/93355), [@maelk](https://github.com/maelk)) - IPv6DualStack 기능 게이트가 활성화된 클러스터에서 서비스를 생성하거나 변경할 때 IPFamily 필드와 관련된 몇 가지 버그가 수정되었다. IPFamily 필드의 동작이 이상하고 일관성이 없으며, 이중-스택 기능이 GA가 되기 전에 변경될 가능성이 있다. 사용자는 현재 이 필드를 "쓰기 전용" 으로 취급해야 하며 현재 IPFamily 값을 기반으로 서비스에 대한 어떤 가정도 하지 않아야 한다. ([#91400](https://github.com/kubernetes/kubernetes/pull/91400), [@danwinship](https://github.com/danwinship)) [SIG Apps 및 Network] - OwnerReferencesPermissionEnforcement 유효성 검사 승인 플러그인이 활성화된 클러스터에서 엔드포인트슬라이스 컨트롤러가 오류없이 실행되도록 수정되었다. ([#89741](https://github.com/kubernetes/kubernetes/pull/89741), [@marun](https://github.com/marun)) [SIG Auth 및 Networking] - IPv6-전용 파드에 대한 엔드포인트를 올바르게 생성하도록 엔드포인트슬라이스컨트롤러가 수정되었다. IPv6DualStack 기능 게이트가 활성화된 경우 서비스에서 `ipFamily: IPv6` 을 지정하여 IPv6 헤드리스 서비스를 허용하도록 엔드포인트컨트롤러가 수정되었다. (이것은 이미 엔드포인트슬라이스컨트롤러와 함께 작동되었다.) ([#91399](https://github.com/kubernetes/kubernetes/pull/91399), [@danwinship](https://github.com/danwinship)) [SIG Apps 및 Network] - 여러 파드에서 읽기 전용 iSCSI 볼륨을 사용할 때 발생하는 문제가 수정되었다. ([#91738](https://github.com/kubernetes/kubernetes/pull/91738), [@jsafrane](https://github.com/jsafrane)) [SIG Storage 및 Testing] - 정보제공자(informer)를 사용하는 CSI 볼륨 첨부 스케일링의 문제가 수정되었다. ([#91307](https://github.com/kubernetes/kubernetes/pull/91307), [@yuga711](https://github.com/yuga711)) [SIG API Machinery, Apps, Node, Storage 및 Testing] - 스케일 하위리소스가 활성화된 사용자 지정 리소스 정의의 복제본 필드에 대한 기본값을 수정하는 버그가 수정되었다. ([#89833](https://github.com/kubernetes/kubernetes/pull/89833), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle 및 Instrumentation] - 디렉토리가 아닌 hostpath 타입이 HostPathFile로 인식될 수 있는 버그가 수정되었으며, HostPathType에 대한 e2e 테스트가 추가되었다. ([#64829](https://github.com/kubernetes/kubernetes/pull/64829), [@dixudx](https://github.com/dixudx)) [SIG Apps, Storage 및 Testing] - 1.16 에서 처음 널리 알려졌던, 일부 VXLAN-기반 네트워크 플러그인에서 63-초 또는 1-초 연결 지연 문제가 수정되었다. (일부 사용자는 특정 네트워크 플러그인에서 이 문제를 더 일찍 확인했다.) 이전에 ethtool 을 사용하여 기본 네트워크 인터페이스에서 체크섬 오프로드를 비활성화한 경우 이제 그 작업을 중지할 수 있다. ([#92035](https://github.com/kubernetes/kubernetes/pull/92035), [@danwinship](https://github.com/danwinship)) [SIG Network 및 Node] - API 요청에서 캐시-제어 헤더가 삭제된 1.17 의 회귀문제가 수정되었다. ([#90468](https://github.com/kubernetes/kubernetes/pull/90468), [@liggitt](https://github.com/liggitt)) [SIG API Machinery 및 Testing] - 잘못된 주석이 있는 HorizontalPodAutoscaler 오브젝트의 변환 오류가 수정되었다. ([#89963](https://github.com/kubernetes/kubernetes/pull/89963), [@liggitt](https://github.com/liggitt)) [SIG Autoscaling] - 오류가 발생했을 경우, 중지하는 대신 올바르게 빌드된 모든 오브젝트에 적용하도록 kubectl 이 수정되었다. ([#89848](https://github.com/kubernetes/kubernetes/pull/89848), [@seans3](https://github.com/seans3)) [SIG CLI 및 Testing] - 부정확한 시간에 독점 CPU를 해제하는 CPUManager 의 회귀 문제가 수정되었다. ([#90377](https://github.com/kubernetes/kubernetes/pull/90377), [@cbf123](https://github.com/cbf123)) [SIG Cloud Provider 및 Node] - 초기화(init) 컨테이너로부터 상속된 앱 컨테이너에서 독점 CPU를 해제할 가능성이 거의 없는 CPUManager 의 회귀 문제가 수정되었다. ([#90419](https://github.com/kubernetes/kubernetes/pull/90419), [@klueska](https://github.com/klueska)) [SIG Node] - 로컬 및 원격 포트를 지정할 때 `kubectl port-forward` 에서 v1.18.0-rc.1 로의 회귀문제가 수정되었다. ([#89401](https://github.com/kubernetes/kubernetes/pull/89401), [@liggitt](https://github.com/liggitt)) - 엔드포인트슬라이스 컨트롤러 가비지 수집에 대한 경합(race) 조건이 수정되었다. ([#91311](https://github.com/kubernetes/kubernetes/pull/91311), [@robscott](https://github.com/robscott)) [SIG Apps, Network 및 Testing] - GCE 클러스터 공급자의 경우, 단일 영역에서 노드가 1000 개 이상인 클러스터에 대해 내부 유형의 로드 밸런서를 만들수 없는 버그가 수정되었다. ([#89902](https://github.com/kubernetes/kubernetes/pull/89902), [@wojtek-t](https://github.com/wojtek-t)) [SIG Cloud Provider, Network 및 Scalability] - 외부 스토리지 e2e 테스트 스위트(suite) 의 경우, VolumeSnapshotClass가 입력으로 명시적으로 제공되는 경우, VolumeSnapshotClass에서 스냅샷 프로비저닝 도구를 선택하도록 외부 드라이버가 변경되었다. ([#90878](https://github.com/kubernetes/kubernetes/pull/90878), [@saikat-royc](https://github.com/saikat-royc)) [SIG Storage 및 Testing] - Get-kube.sh: 올바른 버킷에서 바이너리를 가져오도록 순서가 수정되었다. ([#91635](https://github.com/kubernetes/kubernetes/pull/91635), [@cpanato](https://github.com/cpanato)) [SIG Release] - firstTimestamp가 설정되지 않은 경우 이벤트를 출력할 때, eventTime 을 사용하도록 수정되었다. ([#89999](https://github.com/kubernetes/kubernetes/pull/89999), [@soltysh](https://github.com/soltysh)) [SIG CLI] - 파라미터 cgroupPerQos=false 및 cgroupRoot=/docker 를 설정하면 이 함수는 nodeAllocatableRoot=/docker/kubepods 를 반환하는데 이는 옳지 않다. 올바른 반환은 /docker.cm.NodeAllocatableRoot(s.CgroupRoot, s.CgroupDriver) 이어야 한다. kubeDeps.CAdvisorInterface, err = cadvisor.New(imageFsInfoProvider, s.RootDirectory, cgroupRoots, cadvisor.UsingLegacyCadvisorStats(s.ContainerRuntime, s.RemoteRuntimeEndpoint)) 위 함수는 cgroupRoots를 사용하여 cadvisor 인터페이스를 만들 때, 잘못된 파라미터 cgroupRoots 로 인해 제거 관리자가 /docker 에서 메트릭을 수집하지 않도록 유도하고 kubelet은 이러한 오류를 자주 표시하게된다. E0303 17:25:03.436781 63839 summary_sys_containers.go:47] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03.436809 63839 helpers.go:680] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics ([#88970](https://github.com/kubernetes/kubernetes/pull/88970), [@mysunshine92](https://github.com/mysunshine92)) [SIG Node] - HA 환경에서, 대기 스케줄러가 API 서버와의 연결이 끊어진 기간 동안 파드를 삭제하고 다시 생성한 후 대기 스케줄러가 마스터가 되면 스케줄러 캐시가 손상될 수 있다. 이 PR은 이 문제를 해결한다. ([#91126](https://github.com/kubernetes/kubernetes/pull/91126), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] - /metrics/resource 의 kubelet 리소스 메트릭 엔드포인트에서 다음 메트릭의 이름이 변경되었다. - node_cpu_usage_seconds --> node_cpu_usage_seconds_total - container_cpu_usage_seconds --> container_cpu_usage_seconds_total 이것은 1.18.0에서 추가된 #86282의 부분적인 되돌리기이며, 처음에는 _total 접미사가 제거되었다. ([#89540](https://github.com/kubernetes/kubernetes/pull/89540), [@dashpole](https://github.com/dashpole)) [SIG Instrumentation 및 Node] - Ipvs: 지원되는 커널에서만 sysctlconnreuse 설정을 시도한다. ([#88541](https://github.com/kubernetes/kubernetes/pull/88541), [@cmluciano](https://github.com/cmluciano)) [SIG Network] - kubectl/client-go의 Jsonpath 지원은 복잡한 유형(맵/슬라이스/구조체)을 Go-구문 대신 json으로 직렬화 한다. ([#89660](https://github.com/kubernetes/kubernetes/pull/89660), [@pjferrell](https://github.com/pjferrell)) [SIG API Machinery, CLI 및 Cluster Lifecycle] - Kube-aggregator 인증서는 디스크에서 변경시 동적으로 로드된다. ([#92791](https://github.com/kubernetes/kubernetes/pull/92791), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery] - Kube-apiserver: 불필요한 409 충돌 오류를 클라이언트에 반환하지 않도록 스케일 하위 리소스 패치 처리가 수정되었다. ([#90342](https://github.com/kubernetes/kubernetes/pull/90342), [@liggitt](https://github.com/liggitt)) [SIG Apps, Autoscaling 및 Testing] - Kube-apiserver: 연속적인 재귀 하강 연산자가 있는 jsonpath 표현식은 더 이상 사용자 정의 리소스 표시 열에 대해 해석되지 않는다. ([#93408](https://github.com/kubernetes/kubernetes/pull/93408), [@joelsmith](https://github.com/joelsmith)) [SIG API Machinery] - Kube-apiserver: 단일 X-Stream-Protocol-Version 헤더의 여러 쉼표로 구분된 프로토콜이 이제 RFC2616을 준수하는 여러 헤더와 함께 인식된다. ([#89857](https://github.com/kubernetes/kubernetes/pull/89857), [@tedyu](https://github.com/tedyu)) [SIG API Machinery] - Kube-proxy IP 군은 프록시가 사용하는 nodeIP에 의해 결정된다. 우선순위는: 1. 바인드 주소가 0.0.0.0 또는 ::가 아닌 경우 구성된 --bind-address 의 값 2. 설정된 경우, 노드 개체의 기본 IP 3. IP를 찾을 수 없는 경우, NodeIP 의 기본값은 127.0.0.1 이며 IP 군은 IPv4 이다. ([#91725](https://github.com/kubernetes/kubernetes/pull/91725), [@aojea](https://github.com/aojea)) [SIG Network] - 이중-스택 모드에서 Kube-proxy는 `Service.Spec.IPFamily` 필드를 사용하는 대신 ClusterIP에서 서비스 IP 계열을 유추한다. ([#91357](https://github.com/kubernetes/kubernetes/pull/91357), [@aojea](https://github.com/aojea)) - 이제 Kube-up에 CoreDNS 버전 v1.7.0이 포함된다. 주요 변경 사항은 다음과 같다. -  CoreDNS가 서비스 기록 업데이트를 중단할 수 있는 버그가 수정되었다. -  어떤 정책이 설정되어도 항상 첫 번째 업스트림 서버만 선택되는 포워드 플러그인의 버그가 수정되었다. -  쿠버네티스 플러그인에서 이미 사용 중단된 옵션 `resyncperiod` 및 `upstream` 가 제거되었다. -  Prometheus 메트릭 이름 변경을 포함한다(표준 Prometheus 메트릭 명명 규칙에 맞게 변경). 이전 메트릭의 이름을 사용하는 기존 보고 공식과 역호환 된다. -  페더레이션 플러그인 (v1 쿠버네티스 페더레이션 허용) 이 제거되었다. 자세한 내용은 https://coredns.io/2020/06/15/coredns-1.7.0-release/ 에서 확인할 수 있다. ([#92718](https://github.com/kubernetes/kubernetes/pull/92718), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cloud Provider] - Kube-up: 승인 웹훅 자격증명 구성의 확인 설정이 수정되었다. ([#91995](https://github.com/kubernetes/kubernetes/pull/91995), [@liggitt](https://github.com/liggitt)) [SIG Cloud Provider 및 Cluster Lifecycle] - Kubeadm의 TLS 부트스트랩 프로세스가 조인 시 완료되는 시간 제한이 5 분으로 증가하였다. ([#89735](https://github.com/kubernetes/kubernetes/pull/89735), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] - Kubeadm: 이 작업을 반복 재시도 함으로써, 업데이트 상태를 보다 탄력적으로 만들기 위하여, join / UpdateStatus에 대한 재시도를 추가하였다. ([#91952](https://github.com/kubernetes/kubernetes/pull/91952), [@xlgao-zju](https://github.com/xlgao-zju)) [SIG Cluster Lifecycle] - Kubeadm: 안전하지 않은 서비스를 비활성화하기 위하여 사용 중단된 플래그인 --port=0 을 kube-controller-manager 및 kube-scheduler 매니페스트에 추가하였다. 이 플래그가 없으면 기본적으로 구성 요소(예: /metrics)가 (--adress에 의해 제어되는)기본노드 인터페이스에서 안전하지 않게 제공된다. 이 동작을 재정의하고 안전하지 않은 서비스를 활성화하려는 사용자의 경우, 이러한 구성 요소에 대한 kubeadm의 "extraArgs" 메커니즘을 통해 사용자 정의 포트로 --port=X 를 전달할 수 있다. ([#92720](https://github.com/kubernetes/kubernetes/pull/92720), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: "조인" 과정에서, 이미 클러스터에 존재하는 경우 etcd 구성원을 다시 추가하지 않는다. ([#92118](https://github.com/kubernetes/kubernetes/pull/92118), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: "리셋" 과정에서, 클러스터에 남아서 쌓여있는 etcd 멤버만 제거하지 않고, 로컬 etcd 스토리지의 정리를 계속 진행한다. ([#91145](https://github.com/kubernetes/kubernetes/pull/91145), [@tnqn](https://github.com/tnqn)) [SIG Cluster Lifecycle] - Kubeadm: 조인 중에 동일한 이름의 노드가 클러스터에 이미 존재하는지 확인하는 경우 NodeReady 조건이 올바르게 검증되었는지를 확인한다. ([#89602](https://github.com/kubernetes/kubernetes/pull/89602), [@kvaps](https://github.com/kvaps)) [SIG Cluster Lifecycle] - Kubeadm: 업그레이드 단계에서 `image-pull-timeout` 플래그가 준수되었는지 확인한다. ([#90328](https://github.com/kubernetes/kubernetes/pull/90328), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] - Kubeadm: 1.18.x로 업그레이드 후 RBAC 누락으로 인해 노드가 클러스터에 조인할 수 없는 버그가 수정되었다. ([#89537](https://github.com/kubernetes/kubernetes/pull/89537), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: 'kubeadm join'에서 컨트롤 플레인과 관련된 플래그 전달에 대한 잘못된 경고가 수정되었다. ([#89596](https://github.com/kubernetes/kubernetes/pull/89596), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: 느린 설정에서 etcd 멤버를 추가할 때 "kubeadm join" 에 대한 견고성이 향상되었다. ([#90645](https://github.com/kubernetes/kubernetes/pull/90645), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: 생성된 인증서에서 중복된 DNS 이름 및 IP 주소가 제거된다. ([#92753](https://github.com/kubernetes/kubernetes/pull/92753), [@QianChenglong](https://github.com/QianChenglong)) [SIG Cluster Lifecycle] - Kubectl azure 인증: 1.18.0에서 "spn:" 접두사가 예기치 않게 kubeconfig 파일의 `apiserver-id` 구성에 추가되는 회귀현상을 수정하였다. ([#89706](https://github.com/kubernetes/kubernetes/pull/89706), [@weinong](https://github.com/weinong)) [SIG API Machinery 및 Auth] - Kubectl: 오토스케일이 '--name' 플래그를 따르지 않던 버그가 수정되었다. ([#91855](https://github.com/kubernetes/kubernetes/pull/91855), [@SataQiu](https://github.com/SataQiu)) [SIG CLI] - Kubectl: kubectl scale이 '--timeout' 플래그를 따르지 않던 버그가 수정되었다. ([#91858](https://github.com/kubernetes/kubernetes/pull/91858), [@SataQiu](https://github.com/SataQiu)) [SIG CLI] - Kubelet: kubelet 도움말 정보가 올바른 유형의 플래그를 표시할 수 없던 버그가 수정되었다. ([#88515](https://github.com/kubernetes/kubernetes/pull/88515), [@SataQiu](https://github.com/SataQiu)) [SIG Docs 및 Node] - Kuberuntime 보안: 파드 샌드박스는 이제 항상 `runtime/default` seccomp 프로필로 실행된다. kuberuntime seccomp: 사용자 정의 프로필은 이제 파드 수준에서 설정할 때 더 작은 seccomp 프로필을 가질 수 있다. ([#90949](https://github.com/kubernetes/kubernetes/pull/90949), [@pjbgf](https://github.com/pjbgf)) [SIG Node] - Kubelet 부트 스트랩 인증서 신호를 인식하게 되었다. ([#92786](https://github.com/kubernetes/kubernetes/pull/92786), [@answer1991](https://github.com/answer1991)) [SIG API Machinery, Auth 및 Node] - 노드 ([#89677](https://github.com/kubernetes/kubernetes/pull/89677), [@zhouya0](https://github.com/zhouya0)) [SIG CLI] - 여러 네트워크 인터페이스가 있는 AWS 노드에서, kubelet은 이제 보조 인터페이스의 주소를 보다 안정적으로 보고해야 한다. ([#91889](https://github.com/kubernetes/kubernetes/pull/91889), [@anguslees](https://github.com/anguslees)) [SIG Cloud Provider] - 리스케줄링 시도를 위한 파드의 조건 업데이트는 생략된다. ([#91252](https://github.com/kubernetes/kubernetes/pull/91252), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - 이제 이전에 지정된 노드가 예약 및 확인할 수 없게 된 후 파드를 선점 대상으로 고려할 수 있게 된다. ([#92604](https://github.com/kubernetes/kubernetes/pull/92604), [@soulxu](https://github.com/soulxu)) - 볼륨을 확장하거나 생성할 때 PVC 요청크기에 대한 오버플로를 방지한다. ([#90907](https://github.com/kubernetes/kubernetes/pull/90907), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider 및 Storage] - 프라이빗 Azure 클라우드의 클러스터가 동일한 클라우드의 ACR에 인증할 수 있도록 수정을 허용한다. ([#90425](https://github.com/kubernetes/kubernetes/pull/90425), [@DavidParks8](https://github.com/DavidParks8)) [SIG Cloud Provider] - AWS 로드밸런서 워커 노드의 SG 규칙 논리를 결정적으로 변경하였다. ([#92224](https://github.com/kubernetes/kubernetes/pull/92224), [@M00nF1sh](https://github.com/M00nF1sh)) [SIG Cloud Provider] - 서버측 적용을 사용하지 않는 생성/업데이트/패치 요청에서 처리되는 metadata.managedFields의 회귀 문제가 해결되었다. ([#91690](https://github.com/kubernetes/kubernetes/pull/91690), [@apelisse](https://github.com/apelisse)) [SIG API Machinery 및 Testing] - 윈도우 볼륨을 마운트하는 v1.18.0-rc.1의 회귀 문제가 해결되었다. ([#89319](https://github.com/kubernetes/kubernetes/pull/89319), [@mboersma](https://github.com/mboersma)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation 및 Storage] - v1 CSR API를 제공하는 서버에 대해 `kubectl certificate approve/deny` 를 사용하는 문제가 해결되었다. ([#91691](https://github.com/kubernetes/kubernetes/pull/91691), [@liggitt](https://github.com/liggitt)) [SIG Auth 및 CLI] - --namespace 플래그 없이 `kubectl apply --prune` 기능이 복원되었다. 1.17부터 `kubectl apply --prune` 은 기본 네임스페이스(또는 kubeconfig에서) 또는 명령줄 플래그에 명시적으로 지정된 리소스만 제거하였다. 그러나 이것은 구성 파일의 모든 네임스페이스에서 리소스를 제거할 수 있는 kubectl 1.16의 변경사항을 호환되지 않는다. 이 패치는 kubectl 1.16 의 동작을 복원한다. ([#89551](https://github.com/kubernetes/kubernetes/pull/89551), [@tatsuhiro-t](https://github.com/tatsuhiro-t)) [SIG CLI 및 Testing] - 클러스터/gce/매니페스트의 컨트롤 플레인 매니페스트에서 정적 컨트롤 플레인 파드의 우선순위를 복원한다. ([#89970](https://github.com/kubernetes/kubernetes/pull/89970), [@liggitt](https://github.com/liggitt)) [SIG Cluster Lifecycle 및 Node] - 1.19-rc1에 추가된, 윈도우 노드를 위한 devicemanager 가 복귀되었다. ([#93263](https://github.com/kubernetes/kubernetes/pull/93263), [@liggitt](https://github.com/liggitt)) Node 및 Windows] - 스케줄러 v1 정책 구성 또는 알고리즘 제공 업체 설정을 v1beta1 컴포넌트컨피그(ComponentConfig)와 함께 전달하여 정책에서 CC로의 전환을 지원한다. ([#92531](https://github.com/kubernetes/kubernetes/pull/92531), [@damemi](https://github.com/damemi)) [SIG Scheduling] - 사용 가능한 노드가 없어서 발생하는 스케줄링 실패의 경우 이제 ```schedule_attempts_total``` 메트릭 하위에 스케줄링 불가능으로 보고된다. ([#90989](https://github.com/kubernetes/kubernetes/pull/90989), [@ahg-g](https://github.com/ahg-g)) [SIG Scheduling] - 이제 파드에 바인딩 된 서비스 어카운트 토큰을 파드 삭제 유예 기간 동안 사용할 수 있다. ([#89583](https://github.com/kubernetes/kubernetes/pull/89583), [@liggitt](https://github.com/liggitt)) [SIG Auth] - 서비스 로드 밸런서는 더 이상 스케줄링 불가능으로 표시된 노드를 후보 노드에서 제외하지 않는다. 대신 서비스 로드 밸런서 제외 레이블을 사용해야 한다. 코돈(cordon) 노드를 보유하고 있는 1.18 클러스터에서 업그레이드 하려는 사용자는, 해당 노드가 서비스 로드 밸런서에서 제외시키려 할 경우, 업그레이드 하기 전에 영향을 받는 노드에 `node.kubernetes.io/exclude-from-external-load-balancers` 레이블을 설정해야 한다. ([#90823](https://github.com/kubernetes/kubernetes/pull/90823), [@smarterclayton](https://github.com/smarterclayton)) [SIG Apps, Cloud Provider 및 Network] - kubectl annotate 명령어의 --list 옵션이 지원된다. ([#92576](https://github.com/kubernetes/kubernetes/pull/92576), [@zhouya0](https://github.com/zhouya0)) [SIG CLI] - 노드 추가/삭제 이벤트에서 서비스 `Type=LoadBalancer`에 대한 LB 백엔드 노드를 동기화한다. ([#81185](https://github.com/kubernetes/kubernetes/pull/81185), [@andrewsykim](https://github.com/andrewsykim)) - 비어있지 않고, 인수가 필요하지 않는 다음의 구성요소는 이제 인수가 지정된 경우 오류 메세지를 출력하고 종료된다. cloud-controller-manager, kube-apiserver, kube-controller-manager, kube-proxy, kubeadm {alpha|config|token|version}, kubemark. 플래그는 짦은 형식의 경우 단일 대시 "-" (0x45), 긴 형식의 경우 이중 대시 "--"로 시작해야 한다. 이 변경 이전에는 잘못된 플래그 (예를 들어, 0x8211: "–" 와 같은 아스키가 아닌 대시문자로 시작)가 위치 인수로 처리되고 무시되었다. ([#91349](https://github.com/kubernetes/kubernetes/pull/91349), [@neolit123](https://github.com/neolit123)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Network 및 Scheduling] - 파드 사양의 terminationGracePeriodSeconds가 미러 파드에 적용된다. ([#92442](https://github.com/kubernetes/kubernetes/pull/92442), [@tedyu](https://github.com/tedyu)) [SIG Node 및 Testing] - 이전 커널과의 IPVS 호환성 문제를 해결하기 위해 github.com/moby/ipvs를 v1.0.1로 변경하였다. ([#90555](https://github.com/kubernetes/kubernetes/pull/90555), [@andrewsykim](https://github.com/andrewsykim)) [SIG Network] - status 하위 리소스를 경유하는 파드의 상태 업데이트는 이제 `status.podIP` 및 `status.podIPs` 필드의 형식이 올바른지 확인한다. ([#90628](https://github.com/kubernetes/kubernetes/pull/90628), [@liggitt](https://github.com/liggitt)) [SIG 앱 및 Node] - 준비상태(readiness)를 보고하기 전에 모든 CRD는 디스커버리 엔드포인트가 확인될 때까지 기다린다. ([#89145](https://github.com/kubernetes/kubernetes/pull/89145), [@sttts](https://github.com/sttts)) [SIG API Machinery] - 파드를 제거할 때, PDB를 확인하지 않고 Pending 상태의 파드가 제거된다. ([#83906](https://github.com/kubernetes/kubernetes/pull/83906), [@michaelgugino](https://github.com/michaelgugino)) [SIG API Machinery, Apps, Node 및 Scheduling] - [보안] golang.org/x/text/encoding/unicode 의 취약성 ([#92219](https://github.com/kubernetes/kubernetes/pull/92219), [@voor](https://github.com/voor)) Cloud Provider, Cluster Lifecycle, Instrumentation 및 Node ### 기타 (정리(Cleanup) or Flake(플레이크)) - --cache-dir은 http 및 디스커버리 모두에 대해 캐시 디렉토리를 설정하며 기본값은 $HOME/.kube/cache이다. ([#92910](https://github.com/kubernetes/kubernetes/pull/92910), [@soltysh](https://github.com/soltysh)) [SIG API Machinery 및 CLI] - 이미지 로그에 `pod.Namespace` 가 추가되었다. ([#91945](https://github.com/kubernetes/kubernetes/pull/91945), [@zhipengzuo](https://github.com/zhipengzuo)) - DISABLE_KUBECONFIG_LOCK 환경 변수를 통해 kubeconfig 파일 잠금을 비활성화 하는 기능이 추가되었다. ([#92513](https://github.com/kubernetes/kubernetes/pull/92513), [@soltysh](https://github.com/soltysh)) [SIG API Machinery 및 CLI] - udp 파드의 conntrack 모듈이 정리되었는지 확인하기 위한 별도의 테스트가 추가되었다. ([#90180](https://github.com/kubernetes/kubernetes/pull/90180), [@JacobTanenbaum](https://github.com/JacobTanenbaum)) [SIG Architecture, Network 및 Testing] - fsType이 지정되지 않은 경우, cinder 값의 fsType 을 `ext4` 로 조정한다. ([#90608](https://github.com/kubernetes/kubernetes/pull/90608), [@huffmanca](https://github.com/huffmanca)) [SIG Storage] - 베이스-이미지(base-image): debian-base:v2.1.0를 사용한다. ([#90697](https://github.com/kubernetes/kubernetes/pull/90697), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery 및 Release] - 베이스-이미지: debian-iptables:v12.1.0를 사용한다. ([#90782](https://github.com/kubernetes/kubernetes/pull/90782), [@justaugustus](https://github.com/justaugustus)) [SIG Release] - Beta.kubernetes.io/arch는 v1.14 이후 이미 사용 중단되었으며, v1.18은 제거 대상이다. ([#89462](https://github.com/kubernetes/kubernetes/pull/89462), [@wawa0210](https://github.com/wawa0210)) [SIG Testing] - 빌드: debian-base@v2.1.2 및 debian-iptables@v12.1.1로 변경되었다. ([#93667](https://github.com/kubernetes/kubernetes/pull/93667), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Release 및 Testing] - beta.kubernetes.io/os를 kubernetes.io/os로 변경하였다. ([#89460](https://github.com/kubernetes/kubernetes/pull/89460), [@wawa0210](https://github.com/wawa0210)) [SIG Testing 및 Windows] - beta.kubernetes.io/os를 kubernetes.io/os로 변경하였다. ([#89461](https://github.com/kubernetes/kubernetes/pull/89461), [@wawa0210](https://github.com/wawa0210)) [SIG Cloud Provider 및 Cluster Lifecycle] - `kubectl get` 을 사용하여 네임스페이스가 지정되지 않는 리소스를 검색할 때, 리소스가 없을 때의 출력 메세지가 변경되었다. ([#89861](https://github.com/kubernetes/kubernetes/pull/89861), [@rccrdpccl](https://github.com/rccrdpccl)) [SIG CLI] - CoreDNS는 더 이상 kube-dns 컨피그맵에 대한 페더레이션 데이터 변환을 지원하지 않는다. ([#92716](https://github.com/kubernetes/kubernetes/pull/92716), [@rajansandeep](https://github.com/rajansandeep)) [SIG Cluster Lifecycle] - 힙스터와 관련된 kubectl top 명령어의 플래그가 사용 중단 되었다. kubectl top 에서 힙스터 지원이 중단 되었다. ([#87498](https://github.com/kubernetes/kubernetes/pull/87498), [@serathius](https://github.com/serathius)) [SIG CLI] - 더 이상 사용되지 않는 `--target-ram-md` 플래그가 사용 중단되었다. ([#91818](https://github.com/kubernetes/kubernetes/pull/91818), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery] - Kubelet API에 직접 의존하는 일부 적합성 테스트가 삭제되었다. ([#90615](https://github.com/kubernetes/kubernetes/pull/90615), [@dims](https://github.com/dims)) [SIG Architecture, Network, Release 및 Testing] - 바인딩 되지 않은 PVC가 지연 바인딩 모드에 있지만 파드에서 사용하는 경우, `WaitingForPodScheduled` 이벤트를 보내게 된다. ([#91455](https://github.com/kubernetes/kubernetes/pull/91455), [@cofyc](https://github.com/cofyc)) [SIG Storage] - 수정: blob 디스크 기능의 라이선스 문제 ([#92824](https://github.com/kubernetes/kubernetes/pull/92824), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - 전용 kubectl 하위명령어의 필드 관리자를 설정하여 서버측 적용의 충돌 오류를 개선하였다. ([#88885](https://github.com/kubernetes/kubernetes/pull/88885), [@julianvmodesto](https://github.com/julianvmodesto)) [SIG CLI 및 Testing] - IsFullyQualifiedDomainName()은 IsDNS1123Label을 기반으로 각 레이블의 유효성을 검사한다. ([#90172](https://github.com/kubernetes/kubernetes/pull/90172), [@nak3](https://github.com/nak3)) [SIG API Machinery 및 Network] - 이제 서비스 어노테이션 `cloud.google.com/network-tier: Standard` 을 사용하여 GCE 로드밸런서의 네트워크 등급을 구성할 수 있다. ([#88532](https://github.com/kubernetes/kubernetes/pull/88532), [@zioproto](https://github.com/zioproto)) [SIG Cloud Provider, Network 및 Testing] - Kube-aggregator: aggregator_unavailable_apiservice_count 메트릭의 이름을 aggregator_unavailable_apiservice_total로 변경하였다. ([#88156](https://github.com/kubernetes/kubernetes/pull/88156), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery] - Kube-apiserver: 사용자 정의 리소스 용으로 게시된 openapi 스키마는 이제 표준 ListMeta 스키마 정의를 참조한다. ([#92546](https://github.com/kubernetes/kubernetes/pull/92546), [@liggitt](https://github.com/liggitt)) [SIG API Machinery] - Kube-proxy는 kube-proxy에 대한 변경 사항이 적용되기 위해 대기열에 마지막으로 대기한 시간을 나타내는 새로운 메트릭인, `kubeproxy_sync_proxy_rules_last_queued_timestamp_seconds` 를 노출해야 한다. ([#90175](https://github.com/kubernetes/kubernetes/pull/90175), [@squeed](https://github.com/squeed)) [SIG Instrumentation 및 Network] - Kube-scheduler: 메트릭 항목 `scheduler_total_preemption_attempts` 의 이름이 `scheduler_preemption_attempts_total` 로 변경되었다. ([#91448](https://github.com/kubernetes/kubernetes/pull/91448), [@RainbowMango](https://github.com/RainbowMango)) [SIG API Machinery, Cluster Lifecycle, Instrumentation 및 Scheduling] - Kube-up: 1.17 이전의 동작과 일치하도록 중요 파드를 kube-system 네임스페이스로 기본적으로 제한한다. ([#93121](https://github.com/kubernetes/kubernetes/pull/93121), [@liggitt](https://github.com/liggitt)) [SIG Cloud Provider 및 Scheduling] - Kubeadm은 이제 kubelet 명령줄 대신 kubelet 구성 요소 설정을 사용하여 IPv6DualStack 기능 게이트를 전달한다. ([#90840](https://github.com/kubernetes/kubernetes/pull/90840), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] - Kubeadm: "kubeadm upgrade apply" 중에 컨트롤 플레인 이미지를 미리 가져오기 위하여 데몬셋(DaemonSet)을 사용하지 않는다. 개별 노드의 업그레이드는 이제 사전 구동 검사를 하여 필요한 이미지를 가져온다. "kubeadm upgrade apply" 에 대한 "--image-pull-timeout" 플래그는 이제 사용 중단되었으며, GA 지원 중단 정책에 따라 향후 릴리스에서 제거된다. ([#90788](https://github.com/kubernetes/kubernetes/pull/90788), [@xlgao-zju](https://github.com/xlgao-zju)) [SIG Cluster Lifecycle] - Kubeadm: /healthz 를 사용하는 대신 kube-apiserver의 스태틱(static) 파드에 대해 /livez 및 /readyz 에 대해 두 개의 개별 검사를 사용한다. ([#90970](https://github.com/kubernetes/kubernetes/pull/90970), [@johscheuer](https://github.com/johscheuer)) [SIG Cluster Lifecycle] - 없음 ([#91597](https://github.com/kubernetes/kubernetes/pull/91597), [@elmiko](https://github.com/elmiko)) [SIG Autoscaling 및 Testing] - Openapi-controller: 등급 제한 메트릭 `APIServiceOpenAPIAggregationControllerQueue1` 에서 후행의 `1` 문자리터럴을 제거하고, 프로메테우스의 모범 사례를 준수하도록 이름을 `open_api_aggregation_controller` 로 변경하였다. ([#77979](https://github.com/kubernetes/kubernetes/pull/77979), [@s-urbaniak](https://github.com/s-urbaniak)) [SIG API Machinery] - 볼륨 작동 오류시 이벤트 스팸이 감소하였다. ([#89794](https://github.com/kubernetes/kubernetes/pull/89794), [@msau42](https://github.com/msau42)) [SIG Storage] - 로컬 nodeipam 범위 할당자를 리팩토링하고 다음의 메트릭을 사용하여, 할당된 CIDR을 저장하는데 사용된 cidr집합을 계측한다. "cidrset_cidrs_allocations_total", "cidrset_cidrs_releases_total", "cidrset_usage_cidrs", "cidrset_allocation_tries_per_request", ([#90288](https://github.com/kubernetes/kubernetes/pull/90288), [@aojea](https://github.com/aojea)) [SIG 앱, Instrumentation, Network 및 확장성] - kubectl apply 명령어에서 사용 중단된 --server-dry-run 플래그가 제거되었다. ([#91308](https://github.com/kubernetes/kubernetes/pull/91308), [@julianvmodesto](https://github.com/julianvmodesto)) [SIG CLI 및 Testing] - 기능 게이트 DefaultPodTopologySpread와의 이름 충돌을 피하기 위해 DefaultPodTopologySpread 플러그인의 이름을 SelectorSpread 플러그인으로 변경하였다. ([#92501](https://github.com/kubernetes/kubernetes/pull/92501), [@rakeshreddybandi](https://github.com/rakeshreddybandi)) [SIG Release, Scheduling 및 Testing] - framework.Failf가 ExpectNoError로 변경되었다. ([#91811](https://github.com/kubernetes/kubernetes/pull/91811), [@lixiaobing1](https://github.com/lixiaobing1)) [SIG Instrumentation, Storage 및 Testing] - 필터링 된 노드가 하나 이하인 경우, 스케줄러 PreScore 플러그인이 실행되지 않는다. ([#89370](https://github.com/kubernetes/kubernetes/pull/89370), [@ahg-g](https://github.com/ahg-g)) [SIG Scheduling] - "HostPath는 볼륨에 올바른 모드를 제공해야 한다."는 이제 적합성 테스트에서 제외된다. ([#90861](https://github.com/kubernetes/kubernetes/pull/90861), [@dims](https://github.com/dims)) [SIG Architecture 및 Testing] - Kubelet의 `--experimental-allocatable-ignore-eviction` 옵션은 이제 사용 중단된 것으로 표시된다. ([#91578](https://github.com/kubernetes/kubernetes/pull/91578), [@knabben](https://github.com/knabben)) [SIG Node] - Kubelet의 `--experimental-mounter-path` 및 `--experimental-check-node-capabilities-before-mount` 옵션은 이제 사용 중단된 것으로 표시된다. ([#91373](https://github.com/kubernetes/kubernetes/pull/91373), [@knabben](https://github.com/knabben)) - PR은 PV 또는 PVC 처리에서 특정 실패가 발생할 때 이벤트를 생성하는 기능을 추가한다. 이 이벤트는 사용자가 사용 실패 이유를 파악하여 필요한 복구 조치를 취할 수 있도록 도와준다. ([#89845](https://github.com/kubernetes/kubernetes/pull/89845), [@yuga711](https://github.com/yuga711)) [SIG Apps] - PodShareProcessNamespace 기능 게이트가 제거되고, PodShareProcessNamespace 는 항상 활성화된다. ([#90099](https://github.com/kubernetes/kubernetes/pull/90099), [@tanjunchen](https://github.com/tanjunchen)) [SIG Node] - kube-apiserver의 `--kubelet-https` 플래그가 사용 중단 된다. kubelet에 대한 kube-apiserver의 연결은 이제 항상 `https` 를 사용한다. (kubelet은 v1.0 이전부터 apiserver가 통신하는 엔드포인트를 제공하기 위해 언제나 `https` 를 사용하였다.) ([#91630](https://github.com/kubernetes/kubernetes/pull/91630), [@liggitt](https://github.com/liggitt)) [SIG API Machinery 및 Node] - CNI가 v0.8.6로 변경되었다. ([#91370](https://github.com/kubernetes/kubernetes/pull/91370), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Network, Release 및 Testing] - Golang이 v1.14.5로 변경되었다. - Update repo-infra to 0.0.7 (go1.14.5 및 go1.13.13을 지원하기 위해) - 포함: - bazelbuild/bazel-toolchains@3.3.2 - bazelbuild/rules_go@v0.22.7 ([#93088](https://github.com/kubernetes/kubernetes/pull/93088), [@justaugustus](https://github.com/justaugustus)) [SIG Release 및 Testing] - Golang이 v1.14.6로 변경되었다. - repo-infra가 0.0.8로 변경되었다. (go1.14.6 및 go1.13.14을 지원하기 위해) - 포함: - bazelbuild/bazel-toolchains@3.4.0 - bazelbuild/rules_go@v0.22.8 ([#93198](https://github.com/kubernetes/kubernetes/pull/93198), [@justaugustus](https://github.com/justaugustus)) [SIG Release 및 Testing] - corefile-migration 라이브러리가 1.0.8로 변경되었다. ([#91856](https://github.com/kubernetes/kubernetes/pull/91856), [@wawa0210](https://github.com/wawa0210)) [SIG Node] - 기본 etcd 서버 버전이 3.4.4로 변경되었다. ([#89214](https://github.com/kubernetes/kubernetes/pull/89214), [@jingyih](https://github.com/jingyih)) [SIG API Machinery, Cluster Lifecycle 및 Testing] - 기본 etcd 서버 버전이 3.4.7로 변경되었다. ([#89895](https://github.com/kubernetes/kubernetes/pull/89895), [@jingyih](https://github.com/jingyih)) [SIG API Machinery, Cluster Lifecycle 및 Testing] - 기본 etcd 서버 버전이 3.4.9로 변경되었다. ([#92349](https://github.com/kubernetes/kubernetes/pull/92349), [@jingyih](https://github.com/jingyih)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle 및 Testing] - go.etcd.io/bbolt가 v1.3.5로 변경되었다. ([#92350](https://github.com/kubernetes/kubernetes/pull/92350), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery 및 Cloud Provider] - opencontainers/runtime-spec 의존성이 v1.0.2로 변경되었다. ([#89644](https://github.com/kubernetes/kubernetes/pull/89644), [@saschagrunert](https://github.com/saschagrunert)) [SIG Node] - `beta.kubernetes.io/os` 및 `beta.kubernetes.io/arch` 노드 레이블은 사용 중단되었다. `kubernetes.io/os` 및 `kubernetes.io/arch` 를 사용하도록 노드 셀렉터를 변경해야한다. ([#91046](https://github.com/kubernetes/kubernetes/pull/91046), [@wawa0210](https://github.com/wawa0210)) [SIG Apps 및 Node] - `kubectl config view` 는 이제 클라이언트 인증서와 유사하게 기본적으로 베어러 토큰을 수정한다. `--raw` 플래그는 여전히 전체 컨텐츠를 출력하는데 사용될 수 있다. ([#88985](https://github.com/kubernetes/kubernetes/pull/88985), [@puerco](https://github.com/puerco)) </del> <ins> - Resolves an issue running Ingress conformance tests on clusters which use finalizers on Ingress objects to manage releasing load balancer resources ([#96742](https://github.com/kubernetes/kubernetes/pull/96742), [@spencerhance](https://github.com/spencerhance)) [SIG Network and Testing] - The Conformance test "validates that there is no conflict between pods with same hostPort but different hostIP and protocol" now validates the connectivity to each hostPort, in addition to the functionality. ([#96627](https://github.com/kubernetes/kubernetes/pull/96627), [@aojea](https://github.com/aojea)) [SIG Scheduling and Testing] ### 버그 또는 회귀(regression) - Add kubectl wait  --ignore-not-found flag ([#90969](https://github.com/kubernetes/kubernetes/pull/90969), [@zhouya0](https://github.com/zhouya0)) [SIG CLI] - Added support to kube-proxy for externalTrafficPolicy=Local setting via Direct Server Return (DSR) load balancers on Windows. ([#93166](https://github.com/kubernetes/kubernetes/pull/93166), [@elweb9858](https://github.com/elweb9858)) [SIG Network] - Alter wording to describe pods using a pvc ([#95635](https://github.com/kubernetes/kubernetes/pull/95635), [@RaunakShah](https://github.com/RaunakShah)) [SIG CLI] - An issues preventing volume expand controller to annotate the PVC with `volume.kubernetes.io/storage-resizer` when the PVC StorageClass is already updated to the out-of-tree provisioner is now fixed. ([#94489](https://github.com/kubernetes/kubernetes/pull/94489), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG API Machinery, Apps and Storage] - Azure ARM client: don't segfault on empty response and http error ([#94078](https://github.com/kubernetes/kubernetes/pull/94078), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider] - Azure armclient backoff step defaults to 1 (no retry). ([#94180](https://github.com/kubernetes/kubernetes/pull/94180), [@feiskyer](https://github.com/feiskyer)) - Azure: fix a bug that kube-controller-manager would panic if wrong Azure VMSS name is configured ([#94306](https://github.com/kubernetes/kubernetes/pull/94306), [@knight42](https://github.com/knight42)) [SIG Cloud Provider] - Both apiserver_request_duration_seconds metrics and RequestReceivedTimestamp fields of an audit event now take into account the time a request spends in the apiserver request filters. ([#94903](https://github.com/kubernetes/kubernetes/pull/94903), [@tkashem](https://github.com/tkashem)) - Build/lib/release: Explicitly use '--platform' in building server images When we switched to go-runner for building the apiserver, controller-manager, and scheduler server components, we no longer reference the individual architectures in the image names, specifically in the 'FROM' directive of the server image Dockerfiles. As a result, server images for non-amd64 images copy in the go-runner amd64 binary instead of the go-runner that matches that architecture. This commit explicitly sets the '--platform=linux/${arch}' to ensure we're pulling the correct go-runner arch from the manifest list. Before: `FROM ${base_image}` After: `FROM --platform=linux/${arch} ${base_image}` ([#94552](https://github.com/kubernetes/kubernetes/pull/94552), [@justaugustus](https://github.com/justaugustus)) [SIG Release] - Bump node-problem-detector version to v0.8.5 to fix OOM detection in with Linux kernels 5.1+ ([#96716](https://github.com/kubernetes/kubernetes/pull/96716), [@tosi3k](https://github.com/tosi3k)) [SIG Cloud Provider, Scalability and Testing] - CSIDriver object can be deployed during volume attachment. ([#93710](https://github.com/kubernetes/kubernetes/pull/93710), [@Jiawei0227](https://github.com/Jiawei0227)) [SIG Apps, Node, Storage and Testing] - Ceph RBD volume expansion now works even when ceph.conf was not provided. ([#92027](https://github.com/kubernetes/kubernetes/pull/92027), [@juliantaylor](https://github.com/juliantaylor)) - Change plugin name in fsgroupapplymetrics of csi and flexvolume to distinguish different driver ([#95892](https://github.com/kubernetes/kubernetes/pull/95892), [@JornShen](https://github.com/JornShen)) [SIG Instrumentation, Storage and Testing] - Change the calculation of pod UIDs so that static pods get a unique value - will cause all containers to be killed and recreated after in-place upgrade. ([#87461](https://github.com/kubernetes/kubernetes/pull/87461), [@bboreham](https://github.com/bboreham)) [SIG Node] - Change the mount way from systemd to normal mount except ceph and glusterfs intree-volume. ([#94916](https://github.com/kubernetes/kubernetes/pull/94916), [@smileusd](https://github.com/smileusd)) [SIG Apps, Cloud Provider, Network, Node, Storage and Testing] - Changes to timeout parameter handling in 1.20.0-beta.2 have been reverted to avoid breaking backwards compatibility with existing clients. ([#96727](https://github.com/kubernetes/kubernetes/pull/96727), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing] - Clear UDP conntrack entry on endpoint changes when using nodeport ([#71573](https://github.com/kubernetes/kubernetes/pull/71573), [@JacobTanenbaum](https://github.com/JacobTanenbaum)) [SIG Network] - Cloud node controller: handle empty providerID from getProviderID ([#95342](https://github.com/kubernetes/kubernetes/pull/95342), [@nicolehanjing](https://github.com/nicolehanjing)) [SIG Cloud Provider] - Disable watchcache for events ([#96052](https://github.com/kubernetes/kubernetes/pull/96052), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery] - Disabled `LocalStorageCapacityIsolation` feature gate is honored during scheduling. ([#96092](https://github.com/kubernetes/kubernetes/pull/96092), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling] - Do not fail sorting empty elements. ([#94666](https://github.com/kubernetes/kubernetes/pull/94666), [@soltysh](https://github.com/soltysh)) [SIG CLI] - Dual-stack: make nodeipam compatible with existing single-stack clusters when dual-stack feature gate become enabled by default ([#90439](https://github.com/kubernetes/kubernetes/pull/90439), [@SataQiu](https://github.com/SataQiu)) [SIG API Machinery] - Duplicate owner reference entries in create/update/patch requests now get deduplicated by the API server. The client sending the request now receives a warning header in the API response. Clients should stop sending requests with duplicate owner references. The API server may reject such requests as early as 1.24. ([#96185](https://github.com/kubernetes/kubernetes/pull/96185), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery and Testing] - Endpoint slice controller now mirrors parent's service label to its corresponding endpoint slices. ([#94443](https://github.com/kubernetes/kubernetes/pull/94443), [@aojea](https://github.com/aojea)) - Ensure getPrimaryInterfaceID not panic when network interfaces for Azure VMSS are null ([#94355](https://github.com/kubernetes/kubernetes/pull/94355), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] - Exposes and sets a default timeout for the SubjectAccessReview client for DelegatingAuthorizationOptions ([#95725](https://github.com/kubernetes/kubernetes/pull/95725), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery and Cloud Provider] - Exposes and sets a default timeout for the TokenReview client for DelegatingAuthenticationOptions ([#96217](https://github.com/kubernetes/kubernetes/pull/96217), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery and Cloud Provider] - Fix CVE-2020-8555 for Quobyte client connections. ([#95206](https://github.com/kubernetes/kubernetes/pull/95206), [@misterikkit](https://github.com/misterikkit)) [SIG Storage] - Fix IP fragmentation of UDP and TCP packets not supported issues on LoadBalancer rules ([#96464](https://github.com/kubernetes/kubernetes/pull/96464), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - Fix a bug that DefaultPreemption plugin is disabled when using (legacy) scheduler policy. ([#96439](https://github.com/kubernetes/kubernetes/pull/96439), [@Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling and Testing] - Fix a bug where loadbalancer deletion gets stuck because of missing resource group. ([#93962](https://github.com/kubernetes/kubernetes/pull/93962), [@phiphi282](https://github.com/phiphi282)) - Fix a concurrent map writes error in kubelet ([#93773](https://github.com/kubernetes/kubernetes/pull/93773), [@knight42](https://github.com/knight42)) [SIG Node] - Fix a panic in `kubectl debug` when a pod has multiple init or ephemeral containers. ([#94580](https://github.com/kubernetes/kubernetes/pull/94580), [@kiyoshim55](https://github.com/kiyoshim55)) - Fix a regression where kubeadm bails out with a fatal error when an optional version command line argument is supplied to the "kubeadm upgrade plan" command ([#94421](https://github.com/kubernetes/kubernetes/pull/94421), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] - Fix azure disk attach failure for disk size bigger than 4TB ([#95463](https://github.com/kubernetes/kubernetes/pull/95463), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - Fix azure disk data loss issue on Windows when unmount disk ([#95456](https://github.com/kubernetes/kubernetes/pull/95456), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] - Fix azure file migration panic ([#94853](https://github.com/kubernetes/kubernetes/pull/94853), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - Fix bug in JSON path parser where an error occurs when a range is empty ([#95933](https://github.com/kubernetes/kubernetes/pull/95933), [@brianpursley](https://github.com/brianpursley)) [SIG API Machinery] - Fix client-go prometheus metrics to correctly present the API path accessed in some environments. ([#74363](https://github.com/kubernetes/kubernetes/pull/74363), [@aanm](https://github.com/aanm)) [SIG API Machinery] - Fix detach azure disk issue when vm not exist ([#95177](https://github.com/kubernetes/kubernetes/pull/95177), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - Fix etcd_object_counts metric reported by kube-apiserver ([#94773](https://github.com/kubernetes/kubernetes/pull/94773), [@tkashem](https://github.com/tkashem)) [SIG API Machinery] - Fix incorrectly reported verbs for kube-apiserver metrics for CRD objects ([#93523](https://github.com/kubernetes/kubernetes/pull/93523), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Instrumentation] - Fix k8s.io/apimachinery/pkg/api/meta.SetStatusCondition to update ObservedGeneration ([#95961](https://github.com/kubernetes/kubernetes/pull/95961), [@KnicKnic](https://github.com/KnicKnic)) [SIG API Machinery] - Fix kubectl SchemaError on CRDs with schema using x-kubernetes-preserve-unknown-fields on array types. ([#94888](https://github.com/kubernetes/kubernetes/pull/94888), [@sttts](https://github.com/sttts)) [SIG API Machinery] - Fix memory leak in kube-apiserver when underlying time goes forth and back. ([#96266](https://github.com/kubernetes/kubernetes/pull/96266), [@chenyw1990](https://github.com/chenyw1990)) [SIG API Machinery] - Fix missing csi annotations on node during parallel csinode update. ([#94389](https://github.com/kubernetes/kubernetes/pull/94389), [@pacoxu](https://github.com/pacoxu)) [SIG Storage] - Fix network_programming_latency metric reporting for Endpoints/EndpointSlice deletions, where we don't have correct timestamp ([#95363](https://github.com/kubernetes/kubernetes/pull/95363), [@wojtek-t](https://github.com/wojtek-t)) [SIG Network and Scalability] - Fix paging issues when Azure API returns empty values with non-empty nextLink ([#96211](https://github.com/kubernetes/kubernetes/pull/96211), [@feiskyer](https://github.com/feiskyer)) [SIG Cloud Provider] - Fix pull image error from multiple ACRs using azure managed identity ([#96355](https://github.com/kubernetes/kubernetes/pull/96355), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - Fix race condition on timeCache locks. ([#94751](https://github.com/kubernetes/kubernetes/pull/94751), [@auxten](https://github.com/auxten)) - Fix regression on `kubectl portforward` when TCP and UCP services were configured on the same port. ([#94728](https://github.com/kubernetes/kubernetes/pull/94728), [@amorenoz](https://github.com/amorenoz)) - Fix scheduler cache snapshot when a Node is deleted before its Pods ([#95130](https://github.com/kubernetes/kubernetes/pull/95130), [@alculquicondor](https://github.com/alculquicondor)) [SIG Scheduling] - Fix the `cloudprovider_azure_api_request_duration_seconds` metric buckets to correctly capture the latency metrics. Previously, the majority of the calls would fall in the "+Inf" bucket. ([#94873](https://github.com/kubernetes/kubernetes/pull/94873), [@marwanad](https://github.com/marwanad)) [SIG Cloud Provider and Instrumentation] - Fix vSphere volumes that could be erroneously attached to wrong node ([#96224](https://github.com/kubernetes/kubernetes/pull/96224), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] - Fix verb & scope reporting for kube-apiserver metrics (LIST reported instead of GET) ([#95562](https://github.com/kubernetes/kubernetes/pull/95562), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Testing] - Fix vsphere detach failure for static PVs ([#95447](https://github.com/kubernetes/kubernetes/pull/95447), [@gnufied](https://github.com/gnufied)) [SIG Cloud Provider and Storage] - Fix: azure disk resize error if source does not exist ([#93011](https://github.com/kubernetes/kubernetes/pull/93011), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - Fix: detach azure disk broken on Azure Stack ([#94885](https://github.com/kubernetes/kubernetes/pull/94885), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - Fix: resize Azure disk issue when it's in attached state ([#96705](https://github.com/kubernetes/kubernetes/pull/96705), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - Fix: smb valid path error ([#95583](https://github.com/kubernetes/kubernetes/pull/95583), [@andyzhangx](https://github.com/andyzhangx)) [SIG Storage] - Fix: use sensitiveOptions on Windows mount ([#94126](https://github.com/kubernetes/kubernetes/pull/94126), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] - Fixed a bug causing incorrect formatting of `kubectl describe ingress`. ([#94985](https://github.com/kubernetes/kubernetes/pull/94985), [@howardjohn](https://github.com/howardjohn)) [SIG CLI and Network] - Fixed a bug in client-go where new clients with customized `Dial`, `Proxy`, `GetCert` config may get stale HTTP transports. ([#95427](https://github.com/kubernetes/kubernetes/pull/95427), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery] - Fixed a bug that prevents kubectl to validate CRDs with schema using x-kubernetes-preserve-unknown-fields on object fields. ([#96369](https://github.com/kubernetes/kubernetes/pull/96369), [@gautierdelorme](https://github.com/gautierdelorme)) [SIG API Machinery and Testing] - Fixed a bug that prevents the use of ephemeral containers in the presence of a validating admission webhook. ([#94685](https://github.com/kubernetes/kubernetes/pull/94685), [@verb](https://github.com/verb)) [SIG Node and Testing] - Fixed a bug where aggregator_unavailable_apiservice metrics were reported for deleted apiservices. ([#96421](https://github.com/kubernetes/kubernetes/pull/96421), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG API Machinery and Instrumentation] - Fixed a bug where improper storage and comparison of endpoints led to excessive API traffic from the endpoints controller ([#94112](https://github.com/kubernetes/kubernetes/pull/94112), [@damemi](https://github.com/damemi)) [SIG Apps, Network and Testing] - Fixed a regression which prevented pods with `docker/default` seccomp annotations from being created in 1.19 if a PodSecurityPolicy was in place which did not allow `runtime/default` seccomp profiles. ([#95985](https://github.com/kubernetes/kubernetes/pull/95985), [@saschagrunert](https://github.com/saschagrunert)) [SIG Auth] - Fixed bug in reflector that couldn't recover from "Too large resource version" errors with API servers 1.17.0-1.18.5 ([#94316](https://github.com/kubernetes/kubernetes/pull/94316), [@janeczku](https://github.com/janeczku)) [SIG API Machinery] - Fixed bug where kubectl top pod output is not sorted when --sort-by and --containers flags are used together ([#93692](https://github.com/kubernetes/kubernetes/pull/93692), [@brianpursley](https://github.com/brianpursley)) [SIG CLI] - Fixed kubelet creating extra sandbox for pods with RestartPolicyOnFailure after all containers succeeded ([#92614](https://github.com/kubernetes/kubernetes/pull/92614), [@tnqn](https://github.com/tnqn)) [SIG Node and Testing] - Fixes an issue proxying to ipv6 pods without specifying a port ([#94834](https://github.com/kubernetes/kubernetes/pull/94834), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Network] - Fixes code generation for non-namespaced create subresources fake client test. ([#96586](https://github.com/kubernetes/kubernetes/pull/96586), [@Doude](https://github.com/Doude)) [SIG API Machinery] - Fixes high CPU usage in kubectl drain ([#95260](https://github.com/kubernetes/kubernetes/pull/95260), [@amandahla](https://github.com/amandahla)) [SIG CLI] - For vSphere Cloud Provider, If VM of worker node is deleted, the node will also be deleted by node controller ([#92608](https://github.com/kubernetes/kubernetes/pull/92608), [@lubronzhan](https://github.com/lubronzhan)) [SIG Cloud Provider] - Gracefully delete nodes when their parent scale set went missing ([#95289](https://github.com/kubernetes/kubernetes/pull/95289), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider] - HTTP/2 connection health check is enabled by default in all Kubernetes clients. The feature should work out-of-the-box. If needed, users can tune the feature via the HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS environment variables. The feature is disabled if HTTP2_READ_IDLE_TIMEOUT_SECONDS is set to 0. ([#95981](https://github.com/kubernetes/kubernetes/pull/95981), [@caesarxuchao](https://github.com/caesarxuchao)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node] - If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400. - If the user specifies a timeout in the request URL that exceeds the maximum request deadline allowed by the apiserver, the request will be aborted with an HTTP 400. ([#96061](https://github.com/kubernetes/kubernetes/pull/96061), [@tkashem](https://github.com/tkashem)) [SIG API Machinery, Network and Testing] - If we set SelectPolicy MinPolicySelect on scaleUp behavior or scaleDown behavior,Horizontal Pod Autoscaler doesn`t automatically scale the number of pods correctly ([#95647](https://github.com/kubernetes/kubernetes/pull/95647), [@JoshuaAndrew](https://github.com/JoshuaAndrew)) [SIG Apps and Autoscaling] - Ignore apparmor for non-linux operating systems ([#93220](https://github.com/kubernetes/kubernetes/pull/93220), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] - Ignore root user check when windows pod starts ([#92355](https://github.com/kubernetes/kubernetes/pull/92355), [@wawa0210](https://github.com/wawa0210)) [SIG Node and Windows] - Improve error messages related to nodePort endpoint changes conntrack entries cleanup. ([#96251](https://github.com/kubernetes/kubernetes/pull/96251), [@ravens](https://github.com/ravens)) [SIG Network] - In dual-stack clusters, kubelet will now set up both IPv4 and IPv6 iptables rules, which may fix some problems, eg with HostPorts. ([#94474](https://github.com/kubernetes/kubernetes/pull/94474), [@danwinship](https://github.com/danwinship)) [SIG Network and Node] - Increase maximum IOPS of AWS EBS io1 volume to current maximum (64,000). ([#90014](https://github.com/kubernetes/kubernetes/pull/90014), [@jacobmarble](https://github.com/jacobmarble)) - Ipvs: ensure selected scheduler kernel modules are loaded ([#93040](https://github.com/kubernetes/kubernetes/pull/93040), [@cmluciano](https://github.com/cmluciano)) [SIG Network] - K8s.io/apimachinery: runtime.DefaultUnstructuredConverter.FromUnstructured now handles converting integer fields to typed float values ([#93250](https://github.com/kubernetes/kubernetes/pull/93250), [@liggitt](https://github.com/liggitt)) [SIG API Machinery] - Kube-proxy now trims extra spaces found in loadBalancerSourceRanges to match Service validation. ([#94107](https://github.com/kubernetes/kubernetes/pull/94107), [@robscott](https://github.com/robscott)) [SIG Network] - Kubeadm ensures "kubeadm reset" does not unmount the root "/var/lib/kubelet" directory if it is mounted by the user. ([#93702](https://github.com/kubernetes/kubernetes/pull/93702), [@thtanaka](https://github.com/thtanaka)) - Kubeadm now makes sure the etcd manifest is regenerated upon upgrade even when no etcd version change takes place ([#94395](https://github.com/kubernetes/kubernetes/pull/94395), [@rosti](https://github.com/rosti)) [SIG Cluster Lifecycle] - Kubeadm now warns (instead of error out) on missing "ca.key" files for root CA, front-proxy CA and etcd CA, during "kubeadm join --control-plane" if the user has provided all certificates, keys and kubeconfig files which require signing with the given CA keys. ([#94988](https://github.com/kubernetes/kubernetes/pull/94988), [@neolit123](https://github.com/neolit123)) - Kubeadm: add missing "--experimental-patches" flag to "kubeadm init phase control-plane" ([#95786](https://github.com/kubernetes/kubernetes/pull/95786), [@Sh4d1](https://github.com/Sh4d1)) [SIG Cluster Lifecycle] - Kubeadm: avoid a panic when determining if the running version of CoreDNS is supported during upgrades ([#94299](https://github.com/kubernetes/kubernetes/pull/94299), [@zouyee](https://github.com/zouyee)) [SIG Cluster Lifecycle] - Kubeadm: ensure the etcd data directory is created with 0700 permissions during control-plane init and join ([#94102](https://github.com/kubernetes/kubernetes/pull/94102), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: fix coredns migration should be triggered when there are newdefault configs during kubeadm upgrade ([#96907](https://github.com/kubernetes/kubernetes/pull/96907), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] - Kubeadm: fix the bug that kubeadm tries to call 'docker info' even if the CRI socket was for another CR ([#94555](https://github.com/kubernetes/kubernetes/pull/94555), [@SataQiu](https://github.com/SataQiu)) [SIG Cluster Lifecycle] - Kubeadm: for Docker as the container runtime, make the "kubeadm reset" command stop containers before removing them ([#94586](https://github.com/kubernetes/kubernetes/pull/94586), [@BedivereZero](https://github.com/BedivereZero)) [SIG Cluster Lifecycle] - Kubeadm: make the kubeconfig files for the kube-controller-manager and kube-scheduler use the LocalAPIEndpoint instead of the ControlPlaneEndpoint. This makes kubeadm clusters more reseliant to version skew problems during immutable upgrades: https://kubernetes.io/docs/setup/release/version-skew-policy/#kube-controller-manager-kube-scheduler-and-cloud-controller-manager ([#94398](https://github.com/kubernetes/kubernetes/pull/94398), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: relax the validation of kubeconfig server URLs. Allow the user to define custom kubeconfig server URLs without erroring out during validation of existing kubeconfig files (e.g. when using external CA mode). ([#94816](https://github.com/kubernetes/kubernetes/pull/94816), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubectl: print error if users place flags before plugin name ([#92343](https://github.com/kubernetes/kubernetes/pull/92343), [@knight42](https://github.com/knight42)) [SIG CLI] - Kubelet: assume that swap is disabled when `/proc/swaps` does not exist ([#93931](https://github.com/kubernetes/kubernetes/pull/93931), [@SataQiu](https://github.com/SataQiu)) [SIG Node] - New Azure instance types do now have correct max data disk count information. ([#94340](https://github.com/kubernetes/kubernetes/pull/94340), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG Cloud Provider and Storage] - Port mapping now allows the same `containerPort` of different containers to different `hostPort` without naming the mapping explicitly. ([#94494](https://github.com/kubernetes/kubernetes/pull/94494), [@SergeyKanzhelev](https://github.com/SergeyKanzhelev)) - Print go stack traces at -v=4 and not -v=2 ([#94663](https://github.com/kubernetes/kubernetes/pull/94663), [@soltysh](https://github.com/soltysh)) [SIG CLI] - Recreate EndpointSlices on rapid Service creation. ([#94730](https://github.com/kubernetes/kubernetes/pull/94730), [@robscott](https://github.com/robscott)) - Reduce volume name length for vsphere volumes ([#96533](https://github.com/kubernetes/kubernetes/pull/96533), [@gnufied](https://github.com/gnufied)) [SIG Storage] - Remove ready file and its directory (which is created during volume SetUp) during emptyDir volume TearDown. ([#95770](https://github.com/kubernetes/kubernetes/pull/95770), [@jingxu97](https://github.com/jingxu97)) [SIG Storage] - Reorganized iptables rules to fix a performance issue ([#95252](https://github.com/kubernetes/kubernetes/pull/95252), [@tssurya](https://github.com/tssurya)) [SIG Network] - Require feature flag CustomCPUCFSQuotaPeriod if setting a non-default cpuCFSQuotaPeriod in kubelet config. ([#94687](https://github.com/kubernetes/kubernetes/pull/94687), [@karan](https://github.com/karan)) [SIG Node] - Resolves a regression in 1.19+ with workloads targeting deprecated beta os/arch labels getting stuck in NodeAffinity status on node startup. ([#96810](https://github.com/kubernetes/kubernetes/pull/96810), [@liggitt](https://github.com/liggitt)) [SIG Node] - Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of `OwnerRefInvalidNamespace` are recorded when namespace mismatches between child and owner objects are detected. The [kubectl-check-ownerreferences](https://github.com/kubernetes-sigs/kubectl-check-ownerreferences) tool can be run prior to upgrading to locate existing objects with invalid ownerReferences. - A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted. - A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. ([#92743](https://github.com/kubernetes/kubernetes/pull/92743), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Apps and Testing] - Skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:162]: Driver azure-disk doesn't support snapshot type DynamicSnapshot -- skipping skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:185]: Driver azure-disk doesn't support ntfs -- skipping ([#96144](https://github.com/kubernetes/kubernetes/pull/96144), [@qinpingli](https://github.com/qinpingli)) [SIG Storage and Testing] - StatefulSet Controller now waits for PersistentVolumeClaim deletion before creating pods. ([#93457](https://github.com/kubernetes/kubernetes/pull/93457), [@ymmt2005](https://github.com/ymmt2005)) - StreamWatcher now calls HandleCrash at appropriate sequence. ([#93108](https://github.com/kubernetes/kubernetes/pull/93108), [@lixiaobing1](https://github.com/lixiaobing1)) - Support the node label `node.kubernetes.io/exclude-from-external-load-balancers` ([#95542](https://github.com/kubernetes/kubernetes/pull/95542), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - The AWS network load balancer attributes can now be specified during service creation ([#95247](https://github.com/kubernetes/kubernetes/pull/95247), [@kishorj](https://github.com/kishorj)) [SIG Cloud Provider] - The `/debug/api_priority_and_fairness/dump_requests` path at an apiserver will no longer return a phantom line for each exempt priority level. ([#93406](https://github.com/kubernetes/kubernetes/pull/93406), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery] - The kube-apiserver will no longer serve APIs that should have been deleted in GA non-alpha levels.  Alpha levels will continue to serve the removed APIs so that CI doesn't immediately break. ([#96525](https://github.com/kubernetes/kubernetes/pull/96525), [@deads2k](https://github.com/deads2k)) [SIG API Machinery] - The kubelet recognizes the --containerd-namespace flag to configure the namespace used by cadvisor. ([#87054](https://github.com/kubernetes/kubernetes/pull/87054), [@changyaowei](https://github.com/changyaowei)) [SIG Node] - Unhealthy pods covered by PDBs can be successfully evicted if enough healthy pods are available. ([#94381](https://github.com/kubernetes/kubernetes/pull/94381), [@michaelgugino](https://github.com/michaelgugino)) [SIG Apps] - Update Calico to v3.15.2 ([#94241](https://github.com/kubernetes/kubernetes/pull/94241), [@lmm](https://github.com/lmm)) [SIG Cloud Provider] - Update default etcd server version to 3.4.13 ([#94287](https://github.com/kubernetes/kubernetes/pull/94287), [@jingyih](https://github.com/jingyih)) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing] - Update max azure data disk count map ([#96308](https://github.com/kubernetes/kubernetes/pull/96308), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider and Storage] - Update the PIP when it is not in the Succeeded provisioning state during the LB update. ([#95748](https://github.com/kubernetes/kubernetes/pull/95748), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - Update the frontend IP config when the service's `pipName` annotation is changed ([#95813](https://github.com/kubernetes/kubernetes/pull/95813), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - Update the route table tag in the route reconcile loop ([#96545](https://github.com/kubernetes/kubernetes/pull/96545), [@nilo19](https://github.com/nilo19)) [SIG Cloud Provider] - Use NLB Subnet CIDRs instead of VPC CIDRs in Health Check SG Rules ([#93515](https://github.com/kubernetes/kubernetes/pull/93515), [@t0rr3sp3dr0](https://github.com/t0rr3sp3dr0)) [SIG Cloud Provider] - Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server  would mean deletion of all the resources from container runtime. ([#92817](https://github.com/kubernetes/kubernetes/pull/92817), [@kmala](https://github.com/kmala)) [SIG Node] - Very large patches may now be specified to `kubectl patch` with the `--patch-file` flag instead of including them directly on the command line. The `--patch` and `--patch-file` flags are mutually exclusive. ([#93548](https://github.com/kubernetes/kubernetes/pull/93548), [@smarterclayton](https://github.com/smarterclayton)) [SIG CLI] - Volume binding: report UnschedulableAndUnresolvable status instead of an error when bound PVs not found ([#95541](https://github.com/kubernetes/kubernetes/pull/95541), [@cofyc](https://github.com/cofyc)) [SIG Apps, Scheduling and Storage] - Warn instead of fail when creating Roles and ClusterRoles with custom verbs via kubectl ([#92492](https://github.com/kubernetes/kubernetes/pull/92492), [@eddiezane](https://github.com/eddiezane)) [SIG CLI] - When creating a PVC with the volume.beta.kubernetes.io/storage-provisioner annotation already set, the PV controller might have incorrectly deleted the newly provisioned PV instead of binding it to the PVC, depending on timing and system load. ([#95909](https://github.com/kubernetes/kubernetes/pull/95909), [@pohly](https://github.com/pohly)) [SIG Apps and Storage] - [kubectl] Fail when local source file doesn't exist ([#90333](https://github.com/kubernetes/kubernetes/pull/90333), [@bamarni](https://github.com/bamarni)) [SIG CLI] ### 기타 (정리 또는 플레이크(flake)) - **Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.**: <!-- This section can be blank if this pull request does not require a release note. When adding links which point to resources within git repositories, like KEPs or supporting documentation, please reference a specific commit and avoid linking directly to the master branch. This ensures that links reference a specific point in time, rather than a document that may change over time. See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files Please use the following format for linking documentation: - [KEP]: <link> - [Usage]: <link> - [Other doc]: <link> --> ([#96443](https://github.com/kubernetes/kubernetes/pull/96443), [@alaypatel07](https://github.com/alaypatel07)) [SIG Apps] - --redirect-container-streaming is no longer functional. The flag will be removed in v1.22 ([#95935](https://github.com/kubernetes/kubernetes/pull/95935), [@tallclair](https://github.com/tallclair)) [SIG Node] - A new metric `requestAbortsTotal` has been introduced that counts aborted requests for each `group`, `version`, `verb`, `resource`, `subresource` and `scope`. ([#95002](https://github.com/kubernetes/kubernetes/pull/95002), [@p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery, Cloud Provider, Instrumentation and Scheduling] - API priority and fairness metrics use snake_case in label names ([#96236](https://github.com/kubernetes/kubernetes/pull/96236), [@adtac](https://github.com/adtac)) [SIG API Machinery, Cluster Lifecycle, Instrumentation and Testing] - Add fine grained debugging to intra-pod conformance test to troubleshoot networking issues for potentially unhealthy nodes when running conformance or sonobuoy tests. ([#93837](https://github.com/kubernetes/kubernetes/pull/93837), [@jayunit100](https://github.com/jayunit100)) - Add the following metrics: - network_plugin_operations_total - network_plugin_operations_errors_total ([#93066](https://github.com/kubernetes/kubernetes/pull/93066), [@AnishShah](https://github.com/AnishShah)) - Adds a bootstrapping ClusterRole, ClusterRoleBinding and group for /metrics, /livez/*, /readyz/*, & /healthz/- endpoints. ([#93311](https://github.com/kubernetes/kubernetes/pull/93311), [@logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Auth, Cloud Provider and Instrumentation] - AdmissionReview objects sent for the creation of Namespace API objects now populate the `namespace` attribute consistently (previously the `namespace` attribute was empty for Namespace creation via POST requests, and populated for Namespace creation via server-side-apply PATCH requests) ([#95012](https://github.com/kubernetes/kubernetes/pull/95012), [@nodo](https://github.com/nodo)) [SIG API Machinery and Testing] - Applies translations on all command descriptions ([#95439](https://github.com/kubernetes/kubernetes/pull/95439), [@HerrNaN](https://github.com/HerrNaN)) [SIG CLI] - Base-images: Update to debian-iptables:buster-v1.3.0 - Uses iptables 1.8.5 - base-images: Update to debian-base:buster-v1.2.0 - cluster/images/etcd: Build etcd:3.4.13-1 image - Uses debian-base:buster-v1.2.0 ([#94733](https://github.com/kubernetes/kubernetes/pull/94733), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Release and Testing] - Changed: default "Accept-Encoding" header removed from HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes ([#96127](https://github.com/kubernetes/kubernetes/pull/96127), [@fonsecas72](https://github.com/fonsecas72)) [SIG Network and Node] - Client-go header logging (at verbosity levels >= 9) now masks `Authorization` header contents ([#95316](https://github.com/kubernetes/kubernetes/pull/95316), [@sfowl](https://github.com/sfowl)) [SIG API Machinery] - Decrease warning message frequency on setting volume ownership for configmap/secret. ([#92878](https://github.com/kubernetes/kubernetes/pull/92878), [@jvanz](https://github.com/jvanz)) - Enhance log information of verifyRunAsNonRoot, add pod, container information ([#94911](https://github.com/kubernetes/kubernetes/pull/94911), [@wawa0210](https://github.com/wawa0210)) [SIG Node] - Fix func name NewCreateCreateDeploymentOptions ([#91931](https://github.com/kubernetes/kubernetes/pull/91931), [@lixiaobing1](https://github.com/lixiaobing1)) [SIG CLI] - Fix kubelet to properly log when a container is started. Previously, kubelet may log that container is dead and was restarted when it was actually started for the first time. This behavior only happened on pods with initContainers and regular containers. ([#91469](https://github.com/kubernetes/kubernetes/pull/91469), [@rata](https://github.com/rata)) - Fixes the message about no auth for metrics in scheduler. ([#94035](https://github.com/kubernetes/kubernetes/pull/94035), [@zhouya0](https://github.com/zhouya0)) [SIG Scheduling] - Generators for services are removed from kubectl ([#95256](https://github.com/kubernetes/kubernetes/pull/95256), [@Git-Jiro](https://github.com/Git-Jiro)) [SIG CLI] - Introduce kubectl-convert plugin. ([#96190](https://github.com/kubernetes/kubernetes/pull/96190), [@soltysh](https://github.com/soltysh)) [SIG CLI and Testing] - Kube-scheduler now logs processed component config at startup ([#96426](https://github.com/kubernetes/kubernetes/pull/96426), [@damemi](https://github.com/damemi)) [SIG Scheduling] - Kubeadm: Separate argument key/value in log msg ([#94016](https://github.com/kubernetes/kubernetes/pull/94016), [@mrueg](https://github.com/mrueg)) [SIG Cluster Lifecycle] - Kubeadm: remove the CoreDNS check for known image digests when applying the addon ([#94506](https://github.com/kubernetes/kubernetes/pull/94506), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle] - Kubeadm: update the default pause image version to 1.4.0 on Windows. With this update the image supports Windows versions 1809 (2019LTS), 1903, 1909, 2004 ([#95419](https://github.com/kubernetes/kubernetes/pull/95419), [@jsturtevant](https://github.com/jsturtevant)) [SIG Cluster Lifecycle and Windows] - Kubectl: the `generator` flag of `kubectl autoscale` has been deprecated and has no effect, it will be removed in a feature release ([#92998](https://github.com/kubernetes/kubernetes/pull/92998), [@SataQiu](https://github.com/SataQiu)) [SIG CLI] - Lock ExternalPolicyForExternalIP to default, this feature gate will be removed in 1.22. ([#94581](https://github.com/kubernetes/kubernetes/pull/94581), [@knabben](https://github.com/knabben)) [SIG Network] - Mask ceph RBD adminSecrets in logs when logLevel >= 4. ([#95245](https://github.com/kubernetes/kubernetes/pull/95245), [@sfowl](https://github.com/sfowl)) - Remove offensive words from kubectl cluster-info command. ([#95202](https://github.com/kubernetes/kubernetes/pull/95202), [@rikatz](https://github.com/rikatz)) - Remove support for "ci/k8s-master" version label in kubeadm, use "ci/latest" instead. See [kubernetes/test-infra#18517](https://github.com/kubernetes/test-infra/pull/18517). ([#93626](https://github.com/kubernetes/kubernetes/pull/93626), [@vikkyomkar](https://github.com/vikkyomkar)) - Remove the dependency of csi-translation-lib module on apiserver/cloud-provider/controller-manager ([#95543](https://github.com/kubernetes/kubernetes/pull/95543), [@wawa0210](https://github.com/wawa0210)) [SIG Release] - Scheduler framework interface moved from pkg/scheduler/framework/v1alpha to pkg/scheduler/framework ([#95069](https://github.com/kubernetes/kubernetes/pull/95069), [@farah](https://github.com/farah)) [SIG Scheduling, Storage and Testing] - Service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset is removed.  All Standard load balancers will always enable tcp resets. ([#94297](https://github.com/kubernetes/kubernetes/pull/94297), [@MarcPow](https://github.com/MarcPow)) [SIG Cloud Provider] - Stop propagating SelfLink (deprecated in 1.16) in kube-apiserver ([#94397](https://github.com/kubernetes/kubernetes/pull/94397), [@wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Testing] - Strip unnecessary security contexts on Windows ([#93475](https://github.com/kubernetes/kubernetes/pull/93475), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) [SIG Node, Testing and Windows] - To ensure the code be strong,  add unit test for GetAddressAndDialer ([#93180](https://github.com/kubernetes/kubernetes/pull/93180), [@FreeZhang61](https://github.com/FreeZhang61)) [SIG Node] - UDP and SCTP protocols can left stale connections that need to be cleared to avoid services disruption, but they can cause problems that are hard to debug. Kubernetes components using a loglevel greater or equal than 4 will log the conntrack operations and its output, to show the entries that were deleted. ([#95694](https://github.com/kubernetes/kubernetes/pull/95694), [@aojea](https://github.com/aojea)) [SIG Network] - Update CNI plugins to v0.8.7 ([#94367](https://github.com/kubernetes/kubernetes/pull/94367), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Network, Node, Release and Testing] - Update cri-tools to [v1.19.0](https://github.com/kubernetes-sigs/cri-tools/releases/tag/v1.19.0) ([#94307](https://github.com/kubernetes/kubernetes/pull/94307), [@xmudrii](https://github.com/xmudrii)) [SIG Cloud Provider] - Update etcd client side to v3.4.13 ([#94259](https://github.com/kubernetes/kubernetes/pull/94259), [@jingyih](https://github.com/jingyih)) [SIG API Machinery and Cloud Provider] - Users will now be able to configure all supported values for AWS NLB health check interval and thresholds for new resources. ([#96312](https://github.com/kubernetes/kubernetes/pull/96312), [@kishorj](https://github.com/kishorj)) [SIG Cloud Provider] - V1helpers.MatchNodeSelectorTerms now accepts just a Node and a list of Terms ([#95871](https://github.com/kubernetes/kubernetes/pull/95871), [@damemi](https://github.com/damemi)) [SIG Apps, Scheduling and Storage] - Vsphere: improve logging message on node cache refresh event ([#95236](https://github.com/kubernetes/kubernetes/pull/95236), [@andrewsykim](https://github.com/andrewsykim)) [SIG Cloud Provider] - `MatchNodeSelectorTerms` function moved to `k8s.io/component-helpers` ([#95531](https://github.com/kubernetes/kubernetes/pull/95531), [@damemi](https://github.com/damemi)) [SIG Apps, Scheduling and Storage] - `kubectl api-resources` now prints the API version (as 'API group/version', same as output of `kubectl api-versions`). The column APIGROUP is now APIVERSION ([#95253](https://github.com/kubernetes/kubernetes/pull/95253), [@sallyom](https://github.com/sallyom)) [SIG CLI] - `kubectl get ingress` now prefers the `networking.k8s.io/v1` over `extensions/v1beta1` (deprecated since v1.14). To explicitly request the deprecated version, use `kubectl get ingress.v1beta1.extensions`. ([#94309](https://github.com/kubernetes/kubernetes/pull/94309), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and CLI] </ins> ## 의존성 ### 추가 <del> - cloud.google.com/go/bigquery: v1.0.1 - cloud.google.com/go/datastore: v1.0.0 - cloud.google.com/go/pubsub: v1.0.1 - cloud.google.com/go/storage: v1.0.0 - dmitri.shuralyov.com/gpu/mtl: 666a987 - github.com/cespare/xxhash/v2: [v2.1.1](https://github.com/cespare/xxhash/v2/tree/v2.1.1) - github.com/checkpoint-restore/go-criu/v4: [v4.0.2](https://github.com/checkpoint-restore/go-criu/v4/tree/v4.0.2) - github.com/chzyer/logex: [v1.1.10](https://github.com/chzyer/logex/tree/v1.1.10) - github.com/chzyer/readline: [2972be2](https://github.com/chzyer/readline/tree/2972be2) - github.com/chzyer/test: [a1ea475](https://github.com/chzyer/test/tree/a1ea475) - github.com/containerd/cgroups: [0dbf7f0](https://github.com/containerd/cgroups/tree/0dbf7f0) - github.com/containerd/continuity: [aaeac12](https://github.com/containerd/continuity/tree/aaeac12) - github.com/containerd/fifo: [a9fb20d](https://github.com/containerd/fifo/tree/a9fb20d) - github.com/containerd/go-runc: [5a6d9f3](https://github.com/containerd/go-runc/tree/5a6d9f3) - github.com/containerd/ttrpc: [v1.0.0](https://github.com/containerd/ttrpc/tree/v1.0.0) - github.com/coreos/bbolt: [v1.3.2](https://github.com/coreos/bbolt/tree/v1.3.2) - github.com/coreos/go-systemd/v22: [v22.1.0](https://github.com/coreos/go-systemd/v22/tree/v22.1.0) - github.com/cpuguy83/go-md2man/v2: [v2.0.0](https://github.com/cpuguy83/go-md2man/v2/tree/v2.0.0) - github.com/docopt/docopt-go: [ee0de3b](https://github.com/docopt/docopt-go/tree/ee0de3b) - github.com/go-gl/glfw/v3.3/glfw: [12ad95a](https://github.com/go-gl/glfw/v3.3/glfw/tree/12ad95a) - github.com/go-ini/ini: [v1.9.0](https://github.com/go-ini/ini/tree/v1.9.0) - github.com/godbus/dbus/v5: [v5.0.3](https://github.com/godbus/dbus/v5/tree/v5.0.3) - github.com/ianlancetaylor/demangle: [5e5cf60](https://github.com/ianlancetaylor/demangle/tree/5e5cf60) - github.com/ishidawataru/sctp: [7c296d4](https://github.com/ishidawataru/sctp/tree/7c296d4) - github.com/moby/ipvs: [v1.0.1](https://github.com/moby/ipvs/tree/v1.0.1) - github.com/moby/sys/mountinfo: [v0.1.3](https://github.com/moby/sys/mountinfo/tree/v0.1.3) - github.com/moby/term: [672ec06](https://github.com/moby/term/tree/672ec06) - github.com/russross/blackfriday/v2: [v2.0.1](https://github.com/russross/blackfriday/v2/tree/v2.0.1) - github.com/shurcooL/sanitized_anchor_name: [v1.0.0](https://github.com/shurcooL/sanitized_anchor_name/tree/v1.0.0) - github.com/ugorji/go: [v1.1.4](https://github.com/ugorji/go/tree/v1.1.4) - github.com/yuin/goldmark: [v1.1.27](https://github.com/yuin/goldmark/tree/v1.1.27) - google.golang.org/protobuf: v1.24.0 - gotest.tools/v3: v3.0.2 - k8s.io/klog/v2: v2.2.0 </del> <ins> - cloud.google.com/go/firestore: v1.1.0 - github.com/Azure/go-autorest: [v14.2.0+incompatible](https://github.com/Azure/go-autorest/tree/v14.2.0) - github.com/armon/go-metrics: [f0300d1](https://github.com/armon/go-metrics/tree/f0300d1) - github.com/armon/go-radix: [7fddfc3](https://github.com/armon/go-radix/tree/7fddfc3) - github.com/bketelsen/crypt: [5cbc8cc](https://github.com/bketelsen/crypt/tree/5cbc8cc) - github.com/form3tech-oss/jwt-go: [v3.2.2+incompatible](https://github.com/form3tech-oss/jwt-go/tree/v3.2.2) - github.com/fvbommel/sortorder: [v1.0.1](https://github.com/fvbommel/sortorder/tree/v1.0.1) - github.com/hashicorp/consul/api: [v1.1.0](https://github.com/hashicorp/consul/api/tree/v1.1.0) - github.com/hashicorp/consul/sdk: [v0.1.1](https://github.com/hashicorp/consul/sdk/tree/v0.1.1) - github.com/hashicorp/errwrap: [v1.0.0](https://github.com/hashicorp/errwrap/tree/v1.0.0) - github.com/hashicorp/go-cleanhttp: [v0.5.1](https://github.com/hashicorp/go-cleanhttp/tree/v0.5.1) - github.com/hashicorp/go-immutable-radix: [v1.0.0](https://github.com/hashicorp/go-immutable-radix/tree/v1.0.0) - github.com/hashicorp/go-msgpack: [v0.5.3](https://github.com/hashicorp/go-msgpack/tree/v0.5.3) - github.com/hashicorp/go-multierror: [v1.0.0](https://github.com/hashicorp/go-multierror/tree/v1.0.0) - github.com/hashicorp/go-rootcerts: [v1.0.0](https://github.com/hashicorp/go-rootcerts/tree/v1.0.0) - github.com/hashicorp/go-sockaddr: [v1.0.0](https://github.com/hashicorp/go-sockaddr/tree/v1.0.0) - github.com/hashicorp/go-uuid: [v1.0.1](https://github.com/hashicorp/go-uuid/tree/v1.0.1) - github.com/hashicorp/go.net: [v0.0.1](https://github.com/hashicorp/go.net/tree/v0.0.1) - github.com/hashicorp/logutils: [v1.0.0](https://github.com/hashicorp/logutils/tree/v1.0.0) - github.com/hashicorp/mdns: [v1.0.0](https://github.com/hashicorp/mdns/tree/v1.0.0) - github.com/hashicorp/memberlist: [v0.1.3](https://github.com/hashicorp/memberlist/tree/v0.1.3) - github.com/hashicorp/serf: [v0.8.2](https://github.com/hashicorp/serf/tree/v0.8.2) - github.com/jmespath/go-jmespath/internal/testify: [v1.5.1](https://github.com/jmespath/go-jmespath/internal/testify/tree/v1.5.1) - github.com/mitchellh/cli: [v1.0.0](https://github.com/mitchellh/cli/tree/v1.0.0) - github.com/mitchellh/go-testing-interface: [v1.0.0](https://github.com/mitchellh/go-testing-interface/tree/v1.0.0) - github.com/mitchellh/gox: [v0.4.0](https://github.com/mitchellh/gox/tree/v0.4.0) - github.com/mitchellh/iochan: [v1.0.0](https://github.com/mitchellh/iochan/tree/v1.0.0) - github.com/pascaldekloe/goe: [57f6aae](https://github.com/pascaldekloe/goe/tree/57f6aae) - github.com/posener/complete: [v1.1.1](https://github.com/posener/complete/tree/v1.1.1) - github.com/ryanuber/columnize: [9b3edd6](https://github.com/ryanuber/columnize/tree/9b3edd6) - github.com/sean-/seed: [e2103e2](https://github.com/sean-/seed/tree/e2103e2) - github.com/subosito/gotenv: [v1.2.0](https://github.com/subosito/gotenv/tree/v1.2.0) - github.com/willf/bitset: [d5bec33](https://github.com/willf/bitset/tree/d5bec33) - gopkg.in/ini.v1: v1.51.0 - gopkg.in/yaml.v3: 9f266ea - rsc.io/quote/v3: v3.1.0 - rsc.io/sampler: v1.3.0 </ins> ### 변경 <del> - cloud.google.com/go: v0.38.0 → v0.51.0 - github.com/Azure/azure-sdk-for-go: [v35.0.0+incompatible → v43.0.0+incompatible](https://github.com/Azure/azure-sdk-for-go/compare/v35.0.0...v43.0.0) - github.com/Azure/go-autorest/autorest/adal: [v0.5.0 → v0.8.2](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.5.0...v0.8.2) - github.com/Azure/go-autorest/autorest/date: [v0.1.0 → v0.2.0](https://github.com/Azure/go-autorest/autorest/date/compare/v0.1.0...v0.2.0) - github.com/Azure/go-autorest/autorest/mocks: [v0.2.0 → v0.3.0](https://github.com/Azure/go-autorest/autorest/mocks/compare/v0.2.0...v0.3.0) - github.com/Azure/go-autorest/autorest: [v0.9.0 → v0.9.6](https://github.com/Azure/go-autorest/autorest/compare/v0.9.0...v0.9.6) - github.com/GoogleCloudPlatform/k8s-cloud-provider: [27a4ced → 7901bc8](https://github.com/GoogleCloudPlatform/k8s-cloud-provider/compare/27a4ced...7901bc8) - github.com/Microsoft/go-winio: [v0.4.14 → fc70bd9](https://github.com/Microsoft/go-winio/compare/v0.4.14...fc70bd9) - github.com/Microsoft/hcsshim: [672e52e → 5eafd15](https://github.com/Microsoft/hcsshim/compare/672e52e...5eafd15) - github.com/alecthomas/template: [a0175ee → fb15b89](https://github.com/alecthomas/template/compare/a0175ee...fb15b89) - github.com/alecthomas/units: [2efee85 → c3de453](https://github.com/alecthomas/units/compare/2efee85...c3de453) - github.com/beorn7/perks: [v1.0.0 → v1.0.1](https://github.com/beorn7/perks/compare/v1.0.0...v1.0.1) - github.com/cilium/ebpf: [95b36a5 → 1c8d4c9](https://github.com/cilium/ebpf/compare/95b36a5...1c8d4c9) - github.com/containerd/console: [84eeaae → v1.0.0](https://github.com/containerd/console/compare/84eeaae...v1.0.0) - github.com/containerd/containerd: [v1.0.2 → v1.3.3](https://github.com/containerd/containerd/compare/v1.0.2...v1.3.3) - github.com/containerd/typeurl: [2a93cfd → v1.0.0](https://github.com/containerd/typeurl/compare/2a93cfd...v1.0.0) - github.com/containernetworking/cni: [v0.7.1 → v0.8.0](https://github.com/containernetworking/cni/compare/v0.7.1...v0.8.0) - github.com/coredns/corefile-migration: [v1.0.6 → v1.0.10](https://github.com/coredns/corefile-migration/compare/v1.0.6...v1.0.10) - github.com/coreos/pkg: [97fdf19 → 399ea9e](https://github.com/coreos/pkg/compare/97fdf19...399ea9e) - github.com/docker/docker: [be7ac8b → aa6a989](https://github.com/docker/docker/compare/be7ac8b...aa6a989) - github.com/docker/go-connections: [v0.3.0 → v0.4.0](https://github.com/docker/go-connections/compare/v0.3.0...v0.4.0) - github.com/evanphx/json-patch: [v4.2.0+incompatible → e83c0a1](https://github.com/evanphx/json-patch/compare/v4.2.0...e83c0a1) - github.com/fsnotify/fsnotify: [v1.4.7 → v1.4.9](https://github.com/fsnotify/fsnotify/compare/v1.4.7...v1.4.9) - github.com/go-kit/kit: [v0.8.0 → v0.9.0](https://github.com/go-kit/kit/compare/v0.8.0...v0.9.0) - github.com/go-logfmt/logfmt: [v0.3.0 → v0.4.0](https://github.com/go-logfmt/logfmt/compare/v0.3.0...v0.4.0) - github.com/go-logr/logr: [v0.1.0 → v0.2.0](https://github.com/go-logr/logr/compare/v0.1.0...v0.2.0) - github.com/golang/groupcache: [02826c3 → 215e871](https://github.com/golang/groupcache/compare/02826c3...215e871) - github.com/golang/protobuf: [v1.3.2 → v1.4.2](https://github.com/golang/protobuf/compare/v1.3.2...v1.4.2) - github.com/google/cadvisor: [v0.35.0 → v0.37.0](https://github.com/google/cadvisor/compare/v0.35.0...v0.37.0) - github.com/google/go-cmp: [v0.3.0 → v0.4.0](https://github.com/google/go-cmp/compare/v0.3.0...v0.4.0) - github.com/google/pprof: [3ea8567 → d4f498a](https://github.com/google/pprof/compare/3ea8567...d4f498a) - github.com/googleapis/gax-go/v2: [v2.0.4 → v2.0.5](https://github.com/googleapis/gax-go/v2/compare/v2.0.4...v2.0.5) - github.com/googleapis/gnostic: [v0.1.0 → v0.4.1](https://github.com/googleapis/gnostic/compare/v0.1.0...v0.4.1) - github.com/gorilla/mux: [v1.7.0 → v1.7.3](https://github.com/gorilla/mux/compare/v1.7.0...v1.7.3) - github.com/json-iterator/go: [v1.1.8 → v1.1.10](https://github.com/json-iterator/go/compare/v1.1.8...v1.1.10) - github.com/jstemmer/go-junit-report: [af01ea7 → v0.9.1](https://github.com/jstemmer/go-junit-report/compare/af01ea7...v0.9.1) - github.com/konsorten/go-windows-terminal-sequences: [v1.0.1 → v1.0.3](https://github.com/konsorten/go-windows-terminal-sequences/compare/v1.0.1...v1.0.3) - github.com/kr/pretty: [v0.1.0 → v0.2.0](https://github.com/kr/pretty/compare/v0.1.0...v0.2.0) - github.com/mattn/go-isatty: [v0.0.9 → v0.0.4](https://github.com/mattn/go-isatty/compare/v0.0.9...v0.0.4) - github.com/matttproud/golang_protobuf_extensions: [v1.0.1 → c182aff](https://github.com/matttproud/golang_protobuf_extensions/compare/v1.0.1...c182aff) - github.com/mistifyio/go-zfs: [v2.1.1+incompatible → f784269](https://github.com/mistifyio/go-zfs/compare/v2.1.1...f784269) - github.com/mrunalp/fileutils: [7d4729f → abd8a0e](https://github.com/mrunalp/fileutils/compare/7d4729f...abd8a0e) - github.com/opencontainers/runc: [v1.0.0-rc10 → 819fcc6](https://github.com/opencontainers/runc/compare/v1.0.0-rc10...819fcc6) - github.com/opencontainers/runtime-spec: [v1.0.0 → 237cc4f](https://github.com/opencontainers/runtime-spec/compare/v1.0.0...237cc4f) - github.com/opencontainers/selinux: [5215b18 → v1.5.2](https://github.com/opencontainers/selinux/compare/5215b18...v1.5.2) - github.com/pkg/errors: [v0.8.1 → v0.9.1](https://github.com/pkg/errors/compare/v0.8.1...v0.9.1) - github.com/prometheus/client_golang: [v1.0.0 → v1.7.1](https://github.com/prometheus/client_golang/compare/v1.0.0...v1.7.1) - github.com/prometheus/common: [v0.4.1 → v0.10.0](https://github.com/prometheus/common/compare/v0.4.1...v0.10.0) - github.com/prometheus/procfs: [v0.0.2 → v0.1.3](https://github.com/prometheus/procfs/compare/v0.0.2...v0.1.3) - github.com/rubiojr/go-vhd: [0bfd3b3 → 02e2102](https://github.com/rubiojr/go-vhd/compare/0bfd3b3...02e2102) - github.com/sirupsen/logrus: [v1.4.2 → v1.6.0](https://github.com/sirupsen/logrus/compare/v1.4.2...v1.6.0) - github.com/spf13/cobra: [v0.0.5 → v1.0.0](https://github.com/spf13/cobra/compare/v0.0.5...v1.0.0) - github.com/spf13/viper: [v1.3.2 → v1.4.0](https://github.com/spf13/viper/compare/v1.3.2...v1.4.0) - github.com/tmc/grpc-websocket-proxy: [89b8d40 → 0ad062e](https://github.com/tmc/grpc-websocket-proxy/compare/89b8d40...0ad062e) - github.com/urfave/cli: [v1.20.0 → v1.22.2](https://github.com/urfave/cli/compare/v1.20.0...v1.22.2) - github.com/vishvananda/netlink: [v1.0.0 → v1.1.0](https://github.com/vishvananda/netlink/compare/v1.0.0...v1.1.0) - github.com/vishvananda/netns: [be1fbed → 52d707b](https://github.com/vishvananda/netns/compare/be1fbed...52d707b) - go.etcd.io/bbolt: v1.3.3 → v1.3.5 - go.etcd.io/etcd: 3cf2f69 → 18dfb9c - go.opencensus.io: v0.21.0 → v0.22.2 - go.uber.org/atomic: v1.3.2 → v1.4.0 - golang.org/x/crypto: bac4c82 → 75b2880 - golang.org/x/exp: 4b39c73 → da58074 - golang.org/x/image: 0694c2d → cff245a - golang.org/x/lint: 959b441 → fdd1cda - golang.org/x/mobile: d3739f8 → d2bd2a2 - golang.org/x/mod: 4bf6d31 → v0.3.0 - golang.org/x/net: 13f9640 → ab34263 - golang.org/x/oauth2: 0f29369 → 858c2ad - golang.org/x/sys: fde4db3 → ed371f2 - golang.org/x/text: v0.3.2 → v0.3.3 - golang.org/x/time: 9d24e82 → 555d28b - golang.org/x/tools: 65e3620 → c1934b7 - golang.org/x/xerrors: a985d34 → 9bdfabe - google.golang.org/api: 5213b80 → v0.15.1 - google.golang.org/appengine: v1.5.0 → v1.6.5 - google.golang.org/genproto: 24fa4b2 → cb27e3a - google.golang.org/grpc: v1.26.0 → v1.27.0 - gopkg.in/check.v1: 788fd78 → 41f04d3 - honnef.co/go/tools: v0.0.1-2019.2.2 → v0.0.1-2019.2.3 - k8s.io/gengo: 36b2048 → 8167cfd - k8s.io/kube-openapi: bf4fb3b → 656914f - k8s.io/system-validators: v1.0.4 → v1.1.2 - k8s.io/utils: 0a110f9 → d5654de - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.7 → v0.0.9 - sigs.k8s.io/structured-merge-diff/v3: v3.0.0 → 43c19bb </del> <ins> - cloud.google.com/go/bigquery: v1.0.1 → v1.4.0 - cloud.google.com/go/datastore: v1.0.0 → v1.1.0 - cloud.google.com/go/pubsub: v1.0.1 → v1.2.0 - cloud.google.com/go/storage: v1.0.0 → v1.6.0 - cloud.google.com/go: v0.51.0 → v0.54.0 - github.com/Azure/go-autorest/autorest/adal: [v0.8.2 → v0.9.5](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.8.2...v0.9.5) - github.com/Azure/go-autorest/autorest/date: [v0.2.0 → v0.3.0](https://github.com/Azure/go-autorest/autorest/date/compare/v0.2.0...v0.3.0) - github.com/Azure/go-autorest/autorest/mocks: [v0.3.0 → v0.4.1](https://github.com/Azure/go-autorest/autorest/mocks/compare/v0.3.0...v0.4.1) - github.com/Azure/go-autorest/autorest: [v0.9.6 → v0.11.1](https://github.com/Azure/go-autorest/autorest/compare/v0.9.6...v0.11.1) - github.com/Azure/go-autorest/logger: [v0.1.0 → v0.2.0](https://github.com/Azure/go-autorest/logger/compare/v0.1.0...v0.2.0) - github.com/Azure/go-autorest/tracing: [v0.5.0 → v0.6.0](https://github.com/Azure/go-autorest/tracing/compare/v0.5.0...v0.6.0) - github.com/Microsoft/go-winio: [fc70bd9 → v0.4.15](https://github.com/Microsoft/go-winio/compare/fc70bd9...v0.4.15) - github.com/aws/aws-sdk-go: [v1.28.2 → v1.35.24](https://github.com/aws/aws-sdk-go/compare/v1.28.2...v1.35.24) - github.com/blang/semver: [v3.5.0+incompatible → v3.5.1+incompatible](https://github.com/blang/semver/compare/v3.5.0...v3.5.1) - github.com/checkpoint-restore/go-criu/v4: [v4.0.2 → v4.1.0](https://github.com/checkpoint-restore/go-criu/v4/compare/v4.0.2...v4.1.0) - github.com/containerd/containerd: [v1.3.3 → v1.4.1](https://github.com/containerd/containerd/compare/v1.3.3...v1.4.1) - github.com/containerd/ttrpc: [v1.0.0 → v1.0.2](https://github.com/containerd/ttrpc/compare/v1.0.0...v1.0.2) - github.com/containerd/typeurl: [v1.0.0 → v1.0.1](https://github.com/containerd/typeurl/compare/v1.0.0...v1.0.1) - github.com/coreos/etcd: [v3.3.10+incompatible → v3.3.13+incompatible](https://github.com/coreos/etcd/compare/v3.3.10...v3.3.13) - github.com/docker/docker: [aa6a989 → bd33bbf](https://github.com/docker/docker/compare/aa6a989...bd33bbf) - github.com/go-gl/glfw/v3.3/glfw: [12ad95a → 6f7a984](https://github.com/go-gl/glfw/v3.3/glfw/compare/12ad95a...6f7a984) - github.com/golang/groupcache: [215e871 → 8c9f03a](https://github.com/golang/groupcache/compare/215e871...8c9f03a) - github.com/golang/mock: [v1.3.1 → v1.4.1](https://github.com/golang/mock/compare/v1.3.1...v1.4.1) - github.com/golang/protobuf: [v1.4.2 → v1.4.3](https://github.com/golang/protobuf/compare/v1.4.2...v1.4.3) - github.com/google/cadvisor: [v0.37.0 → v0.38.5](https://github.com/google/cadvisor/compare/v0.37.0...v0.38.5) - github.com/google/go-cmp: [v0.4.0 → v0.5.2](https://github.com/google/go-cmp/compare/v0.4.0...v0.5.2) - github.com/google/pprof: [d4f498a → 1ebb73c](https://github.com/google/pprof/compare/d4f498a...1ebb73c) - github.com/google/uuid: [v1.1.1 → v1.1.2](https://github.com/google/uuid/compare/v1.1.1...v1.1.2) - github.com/gorilla/mux: [v1.7.3 → v1.8.0](https://github.com/gorilla/mux/compare/v1.7.3...v1.8.0) - github.com/gorilla/websocket: [v1.4.0 → v1.4.2](https://github.com/gorilla/websocket/compare/v1.4.0...v1.4.2) - github.com/jmespath/go-jmespath: [c2b33e8 → v0.4.0](https://github.com/jmespath/go-jmespath/compare/c2b33e8...v0.4.0) - github.com/karrick/godirwalk: [v1.7.5 → v1.16.1](https://github.com/karrick/godirwalk/compare/v1.7.5...v1.16.1) - github.com/opencontainers/go-digest: [v1.0.0-rc1 → v1.0.0](https://github.com/opencontainers/go-digest/compare/v1.0.0-rc1...v1.0.0) - github.com/opencontainers/runc: [819fcc6 → v1.0.0-rc92](https://github.com/opencontainers/runc/compare/819fcc6...v1.0.0-rc92) - github.com/opencontainers/runtime-spec: [237cc4f → 4d89ac9](https://github.com/opencontainers/runtime-spec/compare/237cc4f...4d89ac9) - github.com/opencontainers/selinux: [v1.5.2 → v1.6.0](https://github.com/opencontainers/selinux/compare/v1.5.2...v1.6.0) - github.com/prometheus/procfs: [v0.1.3 → v0.2.0](https://github.com/prometheus/procfs/compare/v0.1.3...v0.2.0) - github.com/quobyte/api: [v0.1.2 → v0.1.8](https://github.com/quobyte/api/compare/v0.1.2...v0.1.8) - github.com/spf13/cobra: [v1.0.0 → v1.1.1](https://github.com/spf13/cobra/compare/v1.0.0...v1.1.1) - github.com/spf13/viper: [v1.4.0 → v1.7.0](https://github.com/spf13/viper/compare/v1.4.0...v1.7.0) - github.com/storageos/go-api: [343b3ef → v2.2.0+incompatible](https://github.com/storageos/go-api/compare/343b3ef...v2.2.0) - github.com/stretchr/testify: [v1.4.0 → v1.6.1](https://github.com/stretchr/testify/compare/v1.4.0...v1.6.1) - github.com/vishvananda/netns: [52d707b → db3c7e5](https://github.com/vishvananda/netns/compare/52d707b...db3c7e5) - go.etcd.io/etcd: 17cef6e → dd1b699 - go.opencensus.io: v0.22.2 → v0.22.3 - golang.org/x/crypto: 75b2880 → 7f63de1 - golang.org/x/exp: da58074 → 6cc2880 - golang.org/x/lint: fdd1cda → 738671d - golang.org/x/net: ab34263 → 69a7880 - golang.org/x/oauth2: 858c2ad → bf48bf1 - golang.org/x/sys: ed371f2 → 5cba982 - golang.org/x/text: v0.3.3 → v0.3.4 - golang.org/x/time: 555d28b → 3af7569 - golang.org/x/xerrors: 9bdfabe → 5ec99f8 - google.golang.org/api: v0.15.1 → v0.20.0 - google.golang.org/genproto: cb27e3a → 8816d57 - google.golang.org/grpc: v1.27.0 → v1.27.1 - google.golang.org/protobuf: v1.24.0 → v1.25.0 - honnef.co/go/tools: v0.0.1-2019.2.3 → v0.0.1-2020.1.3 - k8s.io/gengo: 8167cfd → 83324d8 - k8s.io/klog/v2: v2.2.0 → v2.4.0 - k8s.io/kube-openapi: 6aeccd4 → d219536 - k8s.io/system-validators: v1.1.2 → v1.2.0 - k8s.io/utils: d5654de → 67b214c - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.9 → v0.0.14 - sigs.k8s.io/structured-merge-diff/v4: v4.0.1 → v4.0.2 </ins> ### 제거 <del> - github.com/OpenPeeDeeP/depguard: [v1.0.1](https://github.com/OpenPeeDeeP/depguard/tree/v1.0.1) - github.com/Rican7/retry: [v0.1.0](https://github.com/Rican7/retry/tree/v0.1.0) - github.com/StackExchange/wmi: [5d04971](https://github.com/StackExchange/wmi/tree/5d04971) - github.com/anmitsu/go-shlex: [648efa6](https://github.com/anmitsu/go-shlex/tree/648efa6) - github.com/bazelbuild/bazel-gazelle: [70208cb](https://github.com/bazelbuild/bazel-gazelle/tree/70208cb) - github.com/bazelbuild/buildtools: [69366ca](https://github.com/bazelbuild/buildtools/tree/69366ca) - github.com/bazelbuild/rules_go: [6dae44d](https://github.com/bazelbuild/rules_go/tree/6dae44d) - github.com/bradfitz/go-smtpd: [deb6d62](https://github.com/bradfitz/go-smtpd/tree/deb6d62) - github.com/cespare/prettybench: [03b8cfe](https://github.com/cespare/prettybench/tree/03b8cfe) - github.com/checkpoint-restore/go-criu: [17b0214](https://github.com/checkpoint-restore/go-criu/tree/17b0214) - github.com/client9/misspell: [v0.3.4](https://github.com/client9/misspell/tree/v0.3.4) - github.com/coreos/go-etcd: [v2.0.0+incompatible](https://github.com/coreos/go-etcd/tree/v2.0.0) - github.com/cpuguy83/go-md2man: [v1.0.10](https://github.com/cpuguy83/go-md2man/tree/v1.0.10) - github.com/docker/libnetwork: [c8a5fca](https://github.com/docker/libnetwork/tree/c8a5fca) - github.com/gliderlabs/ssh: [v0.1.1](https://github.com/gliderlabs/ssh/tree/v0.1.1) - github.com/go-critic/go-critic: [1df3008](https://github.com/go-critic/go-critic/tree/1df3008) - github.com/go-lintpack/lintpack: [v0.5.2](https://github.com/go-lintpack/lintpack/tree/v0.5.2) - github.com/go-ole/go-ole: [v1.2.1](https://github.com/go-ole/go-ole/tree/v1.2.1) - github.com/go-toolsmith/astcast: [v1.0.0](https://github.com/go-toolsmith/astcast/tree/v1.0.0) - github.com/go-toolsmith/astcopy: [v1.0.0](https://github.com/go-toolsmith/astcopy/tree/v1.0.0) - github.com/go-toolsmith/astequal: [v1.0.0](https://github.com/go-toolsmith/astequal/tree/v1.0.0) - github.com/go-toolsmith/astfmt: [v1.0.0](https://github.com/go-toolsmith/astfmt/tree/v1.0.0) - github.com/go-toolsmith/astinfo: [9809ff7](https://github.com/go-toolsmith/astinfo/tree/9809ff7) - github.com/go-toolsmith/astp: [v1.0.0](https://github.com/go-toolsmith/astp/tree/v1.0.0) - github.com/go-toolsmith/pkgload: [v1.0.0](https://github.com/go-toolsmith/pkgload/tree/v1.0.0) - github.com/go-toolsmith/strparse: [v1.0.0](https://github.com/go-toolsmith/strparse/tree/v1.0.0) - github.com/go-toolsmith/typep: [v1.0.0](https://github.com/go-toolsmith/typep/tree/v1.0.0) - github.com/gobwas/glob: [v0.2.3](https://github.com/gobwas/glob/tree/v0.2.3) - github.com/godbus/dbus: [2ff6f7f](https://github.com/godbus/dbus/tree/2ff6f7f) - github.com/golangci/check: [cfe4005](https://github.com/golangci/check/tree/cfe4005) - github.com/golangci/dupl: [3e9179a](https://github.com/golangci/dupl/tree/3e9179a) - github.com/golangci/errcheck: [ef45e06](https://github.com/golangci/errcheck/tree/ef45e06) - github.com/golangci/go-misc: [927a3d8](https://github.com/golangci/go-misc/tree/927a3d8) - github.com/golangci/go-tools: [e32c541](https://github.com/golangci/go-tools/tree/e32c541) - github.com/golangci/goconst: [041c5f2](https://github.com/golangci/goconst/tree/041c5f2) - github.com/golangci/gocyclo: [2becd97](https://github.com/golangci/gocyclo/tree/2becd97) - github.com/golangci/gofmt: [0b8337e](https://github.com/golangci/gofmt/tree/0b8337e) - github.com/golangci/golangci-lint: [v1.18.0](https://github.com/golangci/golangci-lint/tree/v1.18.0) - github.com/golangci/gosec: [66fb7fc](https://github.com/golangci/gosec/tree/66fb7fc) - github.com/golangci/ineffassign: [42439a7](https://github.com/golangci/ineffassign/tree/42439a7) - github.com/golangci/lint-1: [ee948d0](https://github.com/golangci/lint-1/tree/ee948d0) - github.com/golangci/maligned: [b1d8939](https://github.com/golangci/maligned/tree/b1d8939) - github.com/golangci/misspell: [950f5d1](https://github.com/golangci/misspell/tree/950f5d1) - github.com/golangci/prealloc: [215b22d](https://github.com/golangci/prealloc/tree/215b22d) - github.com/golangci/revgrep: [d9c87f5](https://github.com/golangci/revgrep/tree/d9c87f5) - github.com/golangci/unconvert: [28b1c44](https://github.com/golangci/unconvert/tree/28b1c44) - github.com/google/go-github: [v17.0.0+incompatible](https://github.com/google/go-github/tree/v17.0.0) - github.com/google/go-querystring: [v1.0.0](https://github.com/google/go-querystring/tree/v1.0.0) - github.com/gostaticanalysis/analysisutil: [v0.0.3](https://github.com/gostaticanalysis/analysisutil/tree/v0.0.3) - github.com/jellevandenhooff/dkim: [f50fe3d](https://github.com/jellevandenhooff/dkim/tree/f50fe3d) - github.com/klauspost/compress: [v1.4.1](https://github.com/klauspost/compress/tree/v1.4.1) - github.com/logrusorgru/aurora: [a7b3b31](https://github.com/logrusorgru/aurora/tree/a7b3b31) - github.com/mattn/go-shellwords: [v1.0.5](https://github.com/mattn/go-shellwords/tree/v1.0.5) - github.com/mattn/goveralls: [v0.0.2](https://github.com/mattn/goveralls/tree/v0.0.2) - github.com/mesos/mesos-go: [v0.0.9](https://github.com/mesos/mesos-go/tree/v0.0.9) - github.com/mitchellh/go-ps: [4fdf99a](https://github.com/mitchellh/go-ps/tree/4fdf99a) - github.com/mozilla/tls-observatory: [8791a20](https://github.com/mozilla/tls-observatory/tree/8791a20) - github.com/nbutton23/zxcvbn-go: [eafdab6](https://github.com/nbutton23/zxcvbn-go/tree/eafdab6) - github.com/pquerna/ffjson: [af8b230](https://github.com/pquerna/ffjson/tree/af8b230) - github.com/quasilyte/go-consistent: [c6f3937](https://github.com/quasilyte/go-consistent/tree/c6f3937) - github.com/ryanuber/go-glob: [256dc44](https://github.com/ryanuber/go-glob/tree/256dc44) - github.com/shirou/gopsutil: [c95755e](https://github.com/shirou/gopsutil/tree/c95755e) - github.com/shirou/w32: [bb4de01](https://github.com/shirou/w32/tree/bb4de01) - github.com/shurcooL/go-goon: [37c2f52](https://github.com/shurcooL/go-goon/tree/37c2f52) - github.com/shurcooL/go: [9e1955d](https://github.com/shurcooL/go/tree/9e1955d) - github.com/sourcegraph/go-diff: [v0.5.1](https://github.com/sourcegraph/go-diff/tree/v0.5.1) - github.com/tarm/serial: [98f6abe](https://github.com/tarm/serial/tree/98f6abe) - github.com/timakin/bodyclose: [87058b9](https://github.com/timakin/bodyclose/tree/87058b9) - github.com/ugorji/go/codec: [d75b2dc](https://github.com/ugorji/go/codec/tree/d75b2dc) - github.com/ultraware/funlen: [v0.0.2](https://github.com/ultraware/funlen/tree/v0.0.2) - github.com/valyala/bytebufferpool: [v1.0.0](https://github.com/valyala/bytebufferpool/tree/v1.0.0) - github.com/valyala/fasthttp: [v1.2.0](https://github.com/valyala/fasthttp/tree/v1.2.0) - github.com/valyala/quicktemplate: [v1.1.1](https://github.com/valyala/quicktemplate/tree/v1.1.1) - github.com/valyala/tcplisten: [ceec8f9](https://github.com/valyala/tcplisten/tree/ceec8f9) - go4.org: 417644f - golang.org/x/build: 2835ba2 - golang.org/x/perf: 6e6d33e - gopkg.in/airbrake/gobrake.v2: v2.0.9 - gopkg.in/gemnasium/logrus-airbrake-hook.v2: v2.1.2 - gotest.tools/gotestsum: v0.3.5 - grpc.go4.org: 11d0a25 - k8s.io/klog: v1.0.0 - k8s.io/repo-infra: v0.0.1-alpha.1 - mvdan.cc/interfacer: c200402 - mvdan.cc/lint: adc824a - mvdan.cc/unparam: fbb5962 - sourcegraph.com/sqs/pbtypes: d3ebe8f </del> <ins> - github.com/armon/consul-api: [eb2c6b5](https://github.com/armon/consul-api/tree/eb2c6b5) - github.com/go-ini/ini: [v1.9.0](https://github.com/go-ini/ini/tree/v1.9.0) - github.com/ugorji/go: [v1.1.4](https://github.com/ugorji/go/tree/v1.1.4) - github.com/xlab/handysort: [fb3537e](https://github.com/xlab/handysort/tree/fb3537e) - github.com/xordataexchange/crypt: [b2862e3](https://github.com/xordataexchange/crypt/tree/b2862e3) - vbom.ml/util: db5cfe1 </ins> ## 의존성 ### 추가 <del> - cloud.google.com/go/bigquery: v1.0.1 - cloud.google.com/go/datastore: v1.0.0 - cloud.google.com/go/pubsub: v1.0.1 - cloud.google.com/go/storage: v1.0.0 - dmitri.shuralyov.com/gpu/mtl: 666a987 - github.com/cespare/xxhash/v2: [v2.1.1](https://github.com/cespare/xxhash/v2/tree/v2.1.1) - github.com/checkpoint-restore/go-criu/v4: [v4.0.2](https://github.com/checkpoint-restore/go-criu/v4/tree/v4.0.2) - github.com/chzyer/logex: [v1.1.10](https://github.com/chzyer/logex/tree/v1.1.10) - github.com/chzyer/readline: [2972be2](https://github.com/chzyer/readline/tree/2972be2) - github.com/chzyer/test: [a1ea475](https://github.com/chzyer/test/tree/a1ea475) - github.com/containerd/cgroups: [0dbf7f0](https://github.com/containerd/cgroups/tree/0dbf7f0) - github.com/containerd/continuity: [aaeac12](https://github.com/containerd/continuity/tree/aaeac12) - github.com/containerd/fifo: [a9fb20d](https://github.com/containerd/fifo/tree/a9fb20d) - github.com/containerd/go-runc: [5a6d9f3](https://github.com/containerd/go-runc/tree/5a6d9f3) - github.com/containerd/ttrpc: [v1.0.0](https://github.com/containerd/ttrpc/tree/v1.0.0) - github.com/coreos/bbolt: [v1.3.2](https://github.com/coreos/bbolt/tree/v1.3.2) - github.com/coreos/go-systemd/v22: [v22.1.0](https://github.com/coreos/go-systemd/v22/tree/v22.1.0) - github.com/cpuguy83/go-md2man/v2: [v2.0.0](https://github.com/cpuguy83/go-md2man/v2/tree/v2.0.0) - github.com/docopt/docopt-go: [ee0de3b](https://github.com/docopt/docopt-go/tree/ee0de3b) - github.com/go-gl/glfw/v3.3/glfw: [12ad95a](https://github.com/go-gl/glfw/v3.3/glfw/tree/12ad95a) - github.com/go-ini/ini: [v1.9.0](https://github.com/go-ini/ini/tree/v1.9.0) - github.com/godbus/dbus/v5: [v5.0.3](https://github.com/godbus/dbus/v5/tree/v5.0.3) - github.com/ianlancetaylor/demangle: [5e5cf60](https://github.com/ianlancetaylor/demangle/tree/5e5cf60) - github.com/ishidawataru/sctp: [7c296d4](https://github.com/ishidawataru/sctp/tree/7c296d4) - github.com/moby/ipvs: [v1.0.1](https://github.com/moby/ipvs/tree/v1.0.1) - github.com/moby/sys/mountinfo: [v0.1.3](https://github.com/moby/sys/mountinfo/tree/v0.1.3) - github.com/moby/term: [672ec06](https://github.com/moby/term/tree/672ec06) - github.com/russross/blackfriday/v2: [v2.0.1](https://github.com/russross/blackfriday/v2/tree/v2.0.1) - github.com/shurcooL/sanitized_anchor_name: [v1.0.0](https://github.com/shurcooL/sanitized_anchor_name/tree/v1.0.0) - github.com/ugorji/go: [v1.1.4](https://github.com/ugorji/go/tree/v1.1.4) - github.com/yuin/goldmark: [v1.1.27](https://github.com/yuin/goldmark/tree/v1.1.27) - google.golang.org/protobuf: v1.24.0 - gotest.tools/v3: v3.0.2 - k8s.io/klog/v2: v2.2.0 - sigs.k8s.io/structured-merge-diff/v4: v4.0.1 </del> <ins> - cloud.google.com/go/firestore: v1.1.0 - github.com/Azure/go-autorest: [v14.2.0+incompatible](https://github.com/Azure/go-autorest/tree/v14.2.0) - github.com/armon/go-metrics: [f0300d1](https://github.com/armon/go-metrics/tree/f0300d1) - github.com/armon/go-radix: [7fddfc3](https://github.com/armon/go-radix/tree/7fddfc3) - github.com/bketelsen/crypt: [5cbc8cc](https://github.com/bketelsen/crypt/tree/5cbc8cc) - github.com/form3tech-oss/jwt-go: [v3.2.2+incompatible](https://github.com/form3tech-oss/jwt-go/tree/v3.2.2) - github.com/fvbommel/sortorder: [v1.0.1](https://github.com/fvbommel/sortorder/tree/v1.0.1) - github.com/hashicorp/consul/api: [v1.1.0](https://github.com/hashicorp/consul/api/tree/v1.1.0) - github.com/hashicorp/consul/sdk: [v0.1.1](https://github.com/hashicorp/consul/sdk/tree/v0.1.1) - github.com/hashicorp/errwrap: [v1.0.0](https://github.com/hashicorp/errwrap/tree/v1.0.0) - github.com/hashicorp/go-cleanhttp: [v0.5.1](https://github.com/hashicorp/go-cleanhttp/tree/v0.5.1) - github.com/hashicorp/go-immutable-radix: [v1.0.0](https://github.com/hashicorp/go-immutable-radix/tree/v1.0.0) - github.com/hashicorp/go-msgpack: [v0.5.3](https://github.com/hashicorp/go-msgpack/tree/v0.5.3) - github.com/hashicorp/go-multierror: [v1.0.0](https://github.com/hashicorp/go-multierror/tree/v1.0.0) - github.com/hashicorp/go-rootcerts: [v1.0.0](https://github.com/hashicorp/go-rootcerts/tree/v1.0.0) - github.com/hashicorp/go-sockaddr: [v1.0.0](https://github.com/hashicorp/go-sockaddr/tree/v1.0.0) - github.com/hashicorp/go-uuid: [v1.0.1](https://github.com/hashicorp/go-uuid/tree/v1.0.1) - github.com/hashicorp/go.net: [v0.0.1](https://github.com/hashicorp/go.net/tree/v0.0.1) - github.com/hashicorp/logutils: [v1.0.0](https://github.com/hashicorp/logutils/tree/v1.0.0) - github.com/hashicorp/mdns: [v1.0.0](https://github.com/hashicorp/mdns/tree/v1.0.0) - github.com/hashicorp/memberlist: [v0.1.3](https://github.com/hashicorp/memberlist/tree/v0.1.3) - github.com/hashicorp/serf: [v0.8.2](https://github.com/hashicorp/serf/tree/v0.8.2) - github.com/jmespath/go-jmespath/internal/testify: [v1.5.1](https://github.com/jmespath/go-jmespath/internal/testify/tree/v1.5.1) - github.com/mitchellh/cli: [v1.0.0](https://github.com/mitchellh/cli/tree/v1.0.0) - github.com/mitchellh/go-testing-interface: [v1.0.0](https://github.com/mitchellh/go-testing-interface/tree/v1.0.0) - github.com/mitchellh/gox: [v0.4.0](https://github.com/mitchellh/gox/tree/v0.4.0) - github.com/mitchellh/iochan: [v1.0.0](https://github.com/mitchellh/iochan/tree/v1.0.0) - github.com/pascaldekloe/goe: [57f6aae](https://github.com/pascaldekloe/goe/tree/57f6aae) - github.com/posener/complete: [v1.1.1](https://github.com/posener/complete/tree/v1.1.1) - github.com/ryanuber/columnize: [9b3edd6](https://github.com/ryanuber/columnize/tree/9b3edd6) - github.com/sean-/seed: [e2103e2](https://github.com/sean-/seed/tree/e2103e2) - github.com/subosito/gotenv: [v1.2.0](https://github.com/subosito/gotenv/tree/v1.2.0) - github.com/willf/bitset: [d5bec33](https://github.com/willf/bitset/tree/d5bec33) - gopkg.in/ini.v1: v1.51.0 - gopkg.in/yaml.v3: 9f266ea - rsc.io/quote/v3: v3.1.0 - rsc.io/sampler: v1.3.0 </ins> ### 변경 <del> - cloud.google.com/go: v0.38.0 → v0.51.0 - github.com/Azure/azure-sdk-for-go: [v35.0.0+incompatible → v43.0.0+incompatible](https://github.com/Azure/azure-sdk-for-go/compare/v35.0.0...v43.0.0) - github.com/Azure/go-autorest/autorest/adal: [v0.5.0 → v0.8.2](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.5.0...v0.8.2) - github.com/Azure/go-autorest/autorest/date: [v0.1.0 → v0.2.0](https://github.com/Azure/go-autorest/autorest/date/compare/v0.1.0...v0.2.0) - github.com/Azure/go-autorest/autorest/mocks: [v0.2.0 → v0.3.0](https://github.com/Azure/go-autorest/autorest/mocks/compare/v0.2.0...v0.3.0) - github.com/Azure/go-autorest/autorest: [v0.9.0 → v0.9.6](https://github.com/Azure/go-autorest/autorest/compare/v0.9.0...v0.9.6) - github.com/GoogleCloudPlatform/k8s-cloud-provider: [27a4ced → 7901bc8](https://github.com/GoogleCloudPlatform/k8s-cloud-provider/compare/27a4ced...7901bc8) - github.com/Microsoft/go-winio: [v0.4.14 → fc70bd9](https://github.com/Microsoft/go-winio/compare/v0.4.14...fc70bd9) - github.com/Microsoft/hcsshim: [672e52e → 5eafd15](https://github.com/Microsoft/hcsshim/compare/672e52e...5eafd15) - github.com/alecthomas/template: [a0175ee → fb15b89](https://github.com/alecthomas/template/compare/a0175ee...fb15b89) - github.com/alecthomas/units: [2efee85 → c3de453](https://github.com/alecthomas/units/compare/2efee85...c3de453) - github.com/beorn7/perks: [v1.0.0 → v1.0.1](https://github.com/beorn7/perks/compare/v1.0.0...v1.0.1) - github.com/cilium/ebpf: [95b36a5 → 1c8d4c9](https://github.com/cilium/ebpf/compare/95b36a5...1c8d4c9) - github.com/containerd/console: [84eeaae → v1.0.0](https://github.com/containerd/console/compare/84eeaae...v1.0.0) - github.com/containerd/containerd: [v1.0.2 → v1.3.3](https://github.com/containerd/containerd/compare/v1.0.2...v1.3.3) - github.com/containerd/typeurl: [2a93cfd → v1.0.0](https://github.com/containerd/typeurl/compare/2a93cfd...v1.0.0) - github.com/containernetworking/cni: [v0.7.1 → v0.8.0](https://github.com/containernetworking/cni/compare/v0.7.1...v0.8.0) - github.com/coredns/corefile-migration: [v1.0.6 → v1.0.10](https://github.com/coredns/corefile-migration/compare/v1.0.6...v1.0.10) - github.com/coreos/pkg: [97fdf19 → 399ea9e](https://github.com/coreos/pkg/compare/97fdf19...399ea9e) - github.com/docker/docker: [be7ac8b → aa6a989](https://github.com/docker/docker/compare/be7ac8b...aa6a989) - github.com/docker/go-connections: [v0.3.0 → v0.4.0](https://github.com/docker/go-connections/compare/v0.3.0...v0.4.0) - github.com/evanphx/json-patch: [v4.2.0+incompatible → v4.9.0+incompatible](https://github.com/evanphx/json-patch/compare/v4.2.0...v4.9.0) - github.com/fsnotify/fsnotify: [v1.4.7 → v1.4.9](https://github.com/fsnotify/fsnotify/compare/v1.4.7...v1.4.9) - github.com/go-kit/kit: [v0.8.0 → v0.9.0](https://github.com/go-kit/kit/compare/v0.8.0...v0.9.0) - github.com/go-logfmt/logfmt: [v0.3.0 → v0.4.0](https://github.com/go-logfmt/logfmt/compare/v0.3.0...v0.4.0) - github.com/go-logr/logr: [v0.1.0 → v0.2.0](https://github.com/go-logr/logr/compare/v0.1.0...v0.2.0) - github.com/golang/groupcache: [02826c3 → 215e871](https://github.com/golang/groupcache/compare/02826c3...215e871) - github.com/golang/protobuf: [v1.3.2 → v1.4.2](https://github.com/golang/protobuf/compare/v1.3.2...v1.4.2) - github.com/google/cadvisor: [v0.35.0 → v0.37.0](https://github.com/google/cadvisor/compare/v0.35.0...v0.37.0) - github.com/google/go-cmp: [v0.3.0 → v0.4.0](https://github.com/google/go-cmp/compare/v0.3.0...v0.4.0) - github.com/google/pprof: [3ea8567 → d4f498a](https://github.com/google/pprof/compare/3ea8567...d4f498a) - github.com/googleapis/gax-go/v2: [v2.0.4 → v2.0.5](https://github.com/googleapis/gax-go/v2/compare/v2.0.4...v2.0.5) - github.com/googleapis/gnostic: [v0.1.0 → v0.4.1](https://github.com/googleapis/gnostic/compare/v0.1.0...v0.4.1) - github.com/gorilla/mux: [v1.7.0 → v1.7.3](https://github.com/gorilla/mux/compare/v1.7.0...v1.7.3) - github.com/json-iterator/go: [v1.1.8 → v1.1.10](https://github.com/json-iterator/go/compare/v1.1.8...v1.1.10) - github.com/jstemmer/go-junit-report: [af01ea7 → v0.9.1](https://github.com/jstemmer/go-junit-report/compare/af01ea7...v0.9.1) - github.com/konsorten/go-windows-terminal-sequences: [v1.0.1 → v1.0.3](https://github.com/konsorten/go-windows-terminal-sequences/compare/v1.0.1...v1.0.3) - github.com/kr/pretty: [v0.1.0 → v0.2.0](https://github.com/kr/pretty/compare/v0.1.0...v0.2.0) - github.com/mattn/go-isatty: [v0.0.9 → v0.0.4](https://github.com/mattn/go-isatty/compare/v0.0.9...v0.0.4) - github.com/matttproud/golang_protobuf_extensions: [v1.0.1 → c182aff](https://github.com/matttproud/golang_protobuf_extensions/compare/v1.0.1...c182aff) - github.com/mistifyio/go-zfs: [v2.1.1+incompatible → f784269](https://github.com/mistifyio/go-zfs/compare/v2.1.1...f784269) - github.com/mrunalp/fileutils: [7d4729f → abd8a0e](https://github.com/mrunalp/fileutils/compare/7d4729f...abd8a0e) - github.com/opencontainers/runc: [v1.0.0-rc10 → 819fcc6](https://github.com/opencontainers/runc/compare/v1.0.0-rc10...819fcc6) - github.com/opencontainers/runtime-spec: [v1.0.0 → 237cc4f](https://github.com/opencontainers/runtime-spec/compare/v1.0.0...237cc4f) - github.com/opencontainers/selinux: [5215b18 → v1.5.2](https://github.com/opencontainers/selinux/compare/5215b18...v1.5.2) - github.com/pkg/errors: [v0.8.1 → v0.9.1](https://github.com/pkg/errors/compare/v0.8.1...v0.9.1) - github.com/prometheus/client_golang: [v1.0.0 → v1.7.1](https://github.com/prometheus/client_golang/compare/v1.0.0...v1.7.1) - github.com/prometheus/common: [v0.4.1 → v0.10.0](https://github.com/prometheus/common/compare/v0.4.1...v0.10.0) - github.com/prometheus/procfs: [v0.0.2 → v0.1.3](https://github.com/prometheus/procfs/compare/v0.0.2...v0.1.3) - github.com/rubiojr/go-vhd: [0bfd3b3 → 02e2102](https://github.com/rubiojr/go-vhd/compare/0bfd3b3...02e2102) - github.com/sirupsen/logrus: [v1.4.2 → v1.6.0](https://github.com/sirupsen/logrus/compare/v1.4.2...v1.6.0) - github.com/spf13/cobra: [v0.0.5 → v1.0.0](https://github.com/spf13/cobra/compare/v0.0.5...v1.0.0) - github.com/spf13/viper: [v1.3.2 → v1.4.0](https://github.com/spf13/viper/compare/v1.3.2...v1.4.0) - github.com/tmc/grpc-websocket-proxy: [89b8d40 → 0ad062e](https://github.com/tmc/grpc-websocket-proxy/compare/89b8d40...0ad062e) - github.com/urfave/cli: [v1.20.0 → v1.22.2](https://github.com/urfave/cli/compare/v1.20.0...v1.22.2) - github.com/vishvananda/netlink: [v1.0.0 → v1.1.0](https://github.com/vishvananda/netlink/compare/v1.0.0...v1.1.0) - github.com/vishvananda/netns: [be1fbed → 52d707b](https://github.com/vishvananda/netns/compare/be1fbed...52d707b) - go.etcd.io/bbolt: v1.3.3 → v1.3.5 - go.etcd.io/etcd: 3cf2f69 → 17cef6e - go.opencensus.io: v0.21.0 → v0.22.2 - go.uber.org/atomic: v1.3.2 → v1.4.0 - golang.org/x/crypto: bac4c82 → 75b2880 - golang.org/x/exp: 4b39c73 → da58074 - golang.org/x/image: 0694c2d → cff245a - golang.org/x/lint: 959b441 → fdd1cda - golang.org/x/mobile: d3739f8 → d2bd2a2 - golang.org/x/mod: 4bf6d31 → v0.3.0 - golang.org/x/net: 13f9640 → ab34263 - golang.org/x/oauth2: 0f29369 → 858c2ad - golang.org/x/sys: fde4db3 → ed371f2 - golang.org/x/text: v0.3.2 → v0.3.3 - golang.org/x/time: 9d24e82 → 555d28b - golang.org/x/tools: 65e3620 → c1934b7 - golang.org/x/xerrors: a985d34 → 9bdfabe - google.golang.org/api: 5213b80 → v0.15.1 - google.golang.org/appengine: v1.5.0 → v1.6.5 - google.golang.org/genproto: 24fa4b2 → cb27e3a - google.golang.org/grpc: v1.26.0 → v1.27.0 - gopkg.in/check.v1: 788fd78 → 41f04d3 - honnef.co/go/tools: v0.0.1-2019.2.2 → v0.0.1-2019.2.3 - k8s.io/gengo: 36b2048 → 8167cfd - k8s.io/kube-openapi: bf4fb3b → 6aeccd4 - k8s.io/system-validators: v1.0.4 → v1.1.2 - k8s.io/utils: a9aa75a → d5654de - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.7 → v0.0.9 </del> <ins> - cloud.google.com/go/bigquery: v1.0.1 → v1.4.0 - cloud.google.com/go/datastore: v1.0.0 → v1.1.0 - cloud.google.com/go/pubsub: v1.0.1 → v1.2.0 - cloud.google.com/go/storage: v1.0.0 → v1.6.0 - cloud.google.com/go: v0.51.0 → v0.54.0 - github.com/Azure/go-autorest/autorest/adal: [v0.8.2 → v0.9.5](https://github.com/Azure/go-autorest/autorest/adal/compare/v0.8.2...v0.9.5) - github.com/Azure/go-autorest/autorest/date: [v0.2.0 → v0.3.0](https://github.com/Azure/go-autorest/autorest/date/compare/v0.2.0...v0.3.0) - github.com/Azure/go-autorest/autorest/mocks: [v0.3.0 → v0.4.1](https://github.com/Azure/go-autorest/autorest/mocks/compare/v0.3.0...v0.4.1) - github.com/Azure/go-autorest/autorest: [v0.9.6 → v0.11.1](https://github.com/Azure/go-autorest/autorest/compare/v0.9.6...v0.11.1) - github.com/Azure/go-autorest/logger: [v0.1.0 → v0.2.0](https://github.com/Azure/go-autorest/logger/compare/v0.1.0...v0.2.0) - github.com/Azure/go-autorest/tracing: [v0.5.0 → v0.6.0](https://github.com/Azure/go-autorest/tracing/compare/v0.5.0...v0.6.0) - github.com/Microsoft/go-winio: [fc70bd9 → v0.4.15](https://github.com/Microsoft/go-winio/compare/fc70bd9...v0.4.15) - github.com/aws/aws-sdk-go: [v1.28.2 → v1.35.24](https://github.com/aws/aws-sdk-go/compare/v1.28.2...v1.35.24) - github.com/blang/semver: [v3.5.0+incompatible → v3.5.1+incompatible](https://github.com/blang/semver/compare/v3.5.0...v3.5.1) - github.com/checkpoint-restore/go-criu/v4: [v4.0.2 → v4.1.0](https://github.com/checkpoint-restore/go-criu/v4/compare/v4.0.2...v4.1.0) - github.com/containerd/containerd: [v1.3.3 → v1.4.1](https://github.com/containerd/containerd/compare/v1.3.3...v1.4.1) - github.com/containerd/ttrpc: [v1.0.0 → v1.0.2](https://github.com/containerd/ttrpc/compare/v1.0.0...v1.0.2) - github.com/containerd/typeurl: [v1.0.0 → v1.0.1](https://github.com/containerd/typeurl/compare/v1.0.0...v1.0.1) - github.com/coreos/etcd: [v3.3.10+incompatible → v3.3.13+incompatible](https://github.com/coreos/etcd/compare/v3.3.10...v3.3.13) - github.com/docker/docker: [aa6a989 → bd33bbf](https://github.com/docker/docker/compare/aa6a989...bd33bbf) - github.com/go-gl/glfw/v3.3/glfw: [12ad95a → 6f7a984](https://github.com/go-gl/glfw/v3.3/glfw/compare/12ad95a...6f7a984) - github.com/golang/groupcache: [215e871 → 8c9f03a](https://github.com/golang/groupcache/compare/215e871...8c9f03a) - github.com/golang/mock: [v1.3.1 → v1.4.1](https://github.com/golang/mock/compare/v1.3.1...v1.4.1) - github.com/golang/protobuf: [v1.4.2 → v1.4.3](https://github.com/golang/protobuf/compare/v1.4.2...v1.4.3) - github.com/google/cadvisor: [v0.37.0 → v0.38.5](https://github.com/google/cadvisor/compare/v0.37.0...v0.38.5) - github.com/google/go-cmp: [v0.4.0 → v0.5.2](https://github.com/google/go-cmp/compare/v0.4.0...v0.5.2) - github.com/google/pprof: [d4f498a → 1ebb73c](https://github.com/google/pprof/compare/d4f498a...1ebb73c) - github.com/google/uuid: [v1.1.1 → v1.1.2](https://github.com/google/uuid/compare/v1.1.1...v1.1.2) - github.com/gorilla/mux: [v1.7.3 → v1.8.0](https://github.com/gorilla/mux/compare/v1.7.3...v1.8.0) - github.com/gorilla/websocket: [v1.4.0 → v1.4.2](https://github.com/gorilla/websocket/compare/v1.4.0...v1.4.2) - github.com/jmespath/go-jmespath: [c2b33e8 → v0.4.0](https://github.com/jmespath/go-jmespath/compare/c2b33e8...v0.4.0) - github.com/karrick/godirwalk: [v1.7.5 → v1.16.1](https://github.com/karrick/godirwalk/compare/v1.7.5...v1.16.1) - github.com/opencontainers/go-digest: [v1.0.0-rc1 → v1.0.0](https://github.com/opencontainers/go-digest/compare/v1.0.0-rc1...v1.0.0) - github.com/opencontainers/runc: [819fcc6 → v1.0.0-rc92](https://github.com/opencontainers/runc/compare/819fcc6...v1.0.0-rc92) - github.com/opencontainers/runtime-spec: [237cc4f → 4d89ac9](https://github.com/opencontainers/runtime-spec/compare/237cc4f...4d89ac9) - github.com/opencontainers/selinux: [v1.5.2 → v1.6.0](https://github.com/opencontainers/selinux/compare/v1.5.2...v1.6.0) - github.com/prometheus/procfs: [v0.1.3 → v0.2.0](https://github.com/prometheus/procfs/compare/v0.1.3...v0.2.0) - github.com/quobyte/api: [v0.1.2 → v0.1.8](https://github.com/quobyte/api/compare/v0.1.2...v0.1.8) - github.com/spf13/cobra: [v1.0.0 → v1.1.1](https://github.com/spf13/cobra/compare/v1.0.0...v1.1.1) - github.com/spf13/viper: [v1.4.0 → v1.7.0](https://github.com/spf13/viper/compare/v1.4.0...v1.7.0) - github.com/storageos/go-api: [343b3ef → v2.2.0+incompatible](https://github.com/storageos/go-api/compare/343b3ef...v2.2.0) - github.com/stretchr/testify: [v1.4.0 → v1.6.1](https://github.com/stretchr/testify/compare/v1.4.0...v1.6.1) - github.com/vishvananda/netns: [52d707b → db3c7e5](https://github.com/vishvananda/netns/compare/52d707b...db3c7e5) - go.etcd.io/etcd: 17cef6e → dd1b699 - go.opencensus.io: v0.22.2 → v0.22.3 - golang.org/x/crypto: 75b2880 → 7f63de1 - golang.org/x/exp: da58074 → 6cc2880 - golang.org/x/lint: fdd1cda → 738671d - golang.org/x/net: ab34263 → 69a7880 - golang.org/x/oauth2: 858c2ad → bf48bf1 - golang.org/x/sys: ed371f2 → 5cba982 - golang.org/x/text: v0.3.3 → v0.3.4 - golang.org/x/time: 555d28b → 3af7569 - golang.org/x/xerrors: 9bdfabe → 5ec99f8 - google.golang.org/api: v0.15.1 → v0.20.0 - google.golang.org/genproto: cb27e3a → 8816d57 - google.golang.org/grpc: v1.27.0 → v1.27.1 - google.golang.org/protobuf: v1.24.0 → v1.25.0 - honnef.co/go/tools: v0.0.1-2019.2.3 → v0.0.1-2020.1.3 - k8s.io/gengo: 8167cfd → 83324d8 - k8s.io/klog/v2: v2.2.0 → v2.4.0 - k8s.io/kube-openapi: 6aeccd4 → d219536 - k8s.io/system-validators: v1.1.2 → v1.2.0 - k8s.io/utils: d5654de → 67b214c - sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.9 → v0.0.14 - sigs.k8s.io/structured-merge-diff/v4: v4.0.1 → v4.0.2 </ins> ### 제거 <del> - github.com/OpenPeeDeeP/depguard: [v1.0.1](https://github.com/OpenPeeDeeP/depguard/tree/v1.0.1) - github.com/Rican7/retry: [v0.1.0](https://github.com/Rican7/retry/tree/v0.1.0) - github.com/StackExchange/wmi: [5d04971](https://github.com/StackExchange/wmi/tree/5d04971) - github.com/anmitsu/go-shlex: [648efa6](https://github.com/anmitsu/go-shlex/tree/648efa6) - github.com/bazelbuild/bazel-gazelle: [70208cb](https://github.com/bazelbuild/bazel-gazelle/tree/70208cb) - github.com/bazelbuild/buildtools: [69366ca](https://github.com/bazelbuild/buildtools/tree/69366ca) - github.com/bazelbuild/rules_go: [6dae44d](https://github.com/bazelbuild/rules_go/tree/6dae44d) - github.com/bradfitz/go-smtpd: [deb6d62](https://github.com/bradfitz/go-smtpd/tree/deb6d62) - github.com/cespare/prettybench: [03b8cfe](https://github.com/cespare/prettybench/tree/03b8cfe) - github.com/checkpoint-restore/go-criu: [17b0214](https://github.com/checkpoint-restore/go-criu/tree/17b0214) - github.com/client9/misspell: [v0.3.4](https://github.com/client9/misspell/tree/v0.3.4) - github.com/coreos/go-etcd: [v2.0.0+incompatible](https://github.com/coreos/go-etcd/tree/v2.0.0) - github.com/cpuguy83/go-md2man: [v1.0.10](https://github.com/cpuguy83/go-md2man/tree/v1.0.10) - github.com/docker/libnetwork: [c8a5fca](https://github.com/docker/libnetwork/tree/c8a5fca) - github.com/gliderlabs/ssh: [v0.1.1](https://github.com/gliderlabs/ssh/tree/v0.1.1) - github.com/go-critic/go-critic: [1df3008](https://github.com/go-critic/go-critic/tree/1df3008) - github.com/go-lintpack/lintpack: [v0.5.2](https://github.com/go-lintpack/lintpack/tree/v0.5.2) - github.com/go-ole/go-ole: [v1.2.1](https://github.com/go-ole/go-ole/tree/v1.2.1) - github.com/go-toolsmith/astcast: [v1.0.0](https://github.com/go-toolsmith/astcast/tree/v1.0.0) - github.com/go-toolsmith/astcopy: [v1.0.0](https://github.com/go-toolsmith/astcopy/tree/v1.0.0) - github.com/go-toolsmith/astequal: [v1.0.0](https://github.com/go-toolsmith/astequal/tree/v1.0.0) - github.com/go-toolsmith/astfmt: [v1.0.0](https://github.com/go-toolsmith/astfmt/tree/v1.0.0) - github.com/go-toolsmith/astinfo: [9809ff7](https://github.com/go-toolsmith/astinfo/tree/9809ff7) - github.com/go-toolsmith/astp: [v1.0.0](https://github.com/go-toolsmith/astp/tree/v1.0.0) - github.com/go-toolsmith/pkgload: [v1.0.0](https://github.com/go-toolsmith/pkgload/tree/v1.0.0) - github.com/go-toolsmith/strparse: [v1.0.0](https://github.com/go-toolsmith/strparse/tree/v1.0.0) - github.com/go-toolsmith/typep: [v1.0.0](https://github.com/go-toolsmith/typep/tree/v1.0.0) - github.com/gobwas/glob: [v0.2.3](https://github.com/gobwas/glob/tree/v0.2.3) - github.com/godbus/dbus: [2ff6f7f](https://github.com/godbus/dbus/tree/2ff6f7f) - github.com/golangci/check: [cfe4005](https://github.com/golangci/check/tree/cfe4005) - github.com/golangci/dupl: [3e9179a](https://github.com/golangci/dupl/tree/3e9179a) - github.com/golangci/errcheck: [ef45e06](https://github.com/golangci/errcheck/tree/ef45e06) - github.com/golangci/go-misc: [927a3d8](https://github.com/golangci/go-misc/tree/927a3d8) - github.com/golangci/go-tools: [e32c541](https://github.com/golangci/go-tools/tree/e32c541) - github.com/golangci/goconst: [041c5f2](https://github.com/golangci/goconst/tree/041c5f2) - github.com/golangci/gocyclo: [2becd97](https://github.com/golangci/gocyclo/tree/2becd97) - github.com/golangci/gofmt: [0b8337e](https://github.com/golangci/gofmt/tree/0b8337e) - github.com/golangci/golangci-lint: [v1.18.0](https://github.com/golangci/golangci-lint/tree/v1.18.0) - github.com/golangci/gosec: [66fb7fc](https://github.com/golangci/gosec/tree/66fb7fc) - github.com/golangci/ineffassign: [42439a7](https://github.com/golangci/ineffassign/tree/42439a7) - github.com/golangci/lint-1: [ee948d0](https://github.com/golangci/lint-1/tree/ee948d0) - github.com/golangci/maligned: [b1d8939](https://github.com/golangci/maligned/tree/b1d8939) - github.com/golangci/misspell: [950f5d1](https://github.com/golangci/misspell/tree/950f5d1) - github.com/golangci/prealloc: [215b22d](https://github.com/golangci/prealloc/tree/215b22d) - github.com/golangci/revgrep: [d9c87f5](https://github.com/golangci/revgrep/tree/d9c87f5) - github.com/golangci/unconvert: [28b1c44](https://github.com/golangci/unconvert/tree/28b1c44) - github.com/google/go-github: [v17.0.0+incompatible](https://github.com/google/go-github/tree/v17.0.0) - github.com/google/go-querystring: [v1.0.0](https://github.com/google/go-querystring/tree/v1.0.0) - github.com/gostaticanalysis/analysisutil: [v0.0.3](https://github.com/gostaticanalysis/analysisutil/tree/v0.0.3) - github.com/jellevandenhooff/dkim: [f50fe3d](https://github.com/jellevandenhooff/dkim/tree/f50fe3d) - github.com/klauspost/compress: [v1.4.1](https://github.com/klauspost/compress/tree/v1.4.1) - github.com/logrusorgru/aurora: [a7b3b31](https://github.com/logrusorgru/aurora/tree/a7b3b31) - github.com/mattn/go-shellwords: [v1.0.5](https://github.com/mattn/go-shellwords/tree/v1.0.5) - github.com/mattn/goveralls: [v0.0.2](https://github.com/mattn/goveralls/tree/v0.0.2) - github.com/mesos/mesos-go: [v0.0.9](https://github.com/mesos/mesos-go/tree/v0.0.9) - github.com/mitchellh/go-ps: [4fdf99a](https://github.com/mitchellh/go-ps/tree/4fdf99a) - github.com/mozilla/tls-observatory: [8791a20](https://github.com/mozilla/tls-observatory/tree/8791a20) - github.com/nbutton23/zxcvbn-go: [eafdab6](https://github.com/nbutton23/zxcvbn-go/tree/eafdab6) - github.com/pquerna/ffjson: [af8b230](https://github.com/pquerna/ffjson/tree/af8b230) - github.com/quasilyte/go-consistent: [c6f3937](https://github.com/quasilyte/go-consistent/tree/c6f3937) - github.com/ryanuber/go-glob: [256dc44](https://github.com/ryanuber/go-glob/tree/256dc44) - github.com/shirou/gopsutil: [c95755e](https://github.com/shirou/gopsutil/tree/c95755e) - github.com/shirou/w32: [bb4de01](https://github.com/shirou/w32/tree/bb4de01) - github.com/shurcooL/go-goon: [37c2f52](https://github.com/shurcooL/go-goon/tree/37c2f52) - github.com/shurcooL/go: [9e1955d](https://github.com/shurcooL/go/tree/9e1955d) - github.com/sourcegraph/go-diff: [v0.5.1](https://github.com/sourcegraph/go-diff/tree/v0.5.1) - github.com/tarm/serial: [98f6abe](https://github.com/tarm/serial/tree/98f6abe) - github.com/timakin/bodyclose: [87058b9](https://github.com/timakin/bodyclose/tree/87058b9) - github.com/ugorji/go/codec: [d75b2dc](https://github.com/ugorji/go/codec/tree/d75b2dc) - github.com/ultraware/funlen: [v0.0.2](https://github.com/ultraware/funlen/tree/v0.0.2) - github.com/valyala/bytebufferpool: [v1.0.0](https://github.com/valyala/bytebufferpool/tree/v1.0.0) - github.com/valyala/fasthttp: [v1.2.0](https://github.com/valyala/fasthttp/tree/v1.2.0) - github.com/valyala/quicktemplate: [v1.1.1](https://github.com/valyala/quicktemplate/tree/v1.1.1) - github.com/valyala/tcplisten: [ceec8f9](https://github.com/valyala/tcplisten/tree/ceec8f9) - go4.org: 417644f - golang.org/x/build: 2835ba2 - golang.org/x/perf: 6e6d33e - gopkg.in/airbrake/gobrake.v2: v2.0.9 - gopkg.in/gemnasium/logrus-airbrake-hook.v2: v2.1.2 - gotest.tools/gotestsum: v0.3.5 - grpc.go4.org: 11d0a25 - k8s.io/klog: v1.0.0 - k8s.io/repo-infra: v0.0.1-alpha.1 - mvdan.cc/interfacer: c200402 - mvdan.cc/lint: adc824a - mvdan.cc/unparam: fbb5962 - sigs.k8s.io/structured-merge-diff/v3: v3.0.0 - sourcegraph.com/sqs/pbtypes: d3ebe8f </del> <ins> - github.com/armon/consul-api: [eb2c6b5](https://github.com/armon/consul-api/tree/eb2c6b5) - github.com/go-ini/ini: [v1.9.0](https://github.com/go-ini/ini/tree/v1.9.0) - github.com/ugorji/go: [v1.1.4](https://github.com/ugorji/go/tree/v1.1.4) - github.com/xlab/handysort: [fb3537e](https://github.com/xlab/handysort/tree/fb3537e) - github.com/xordataexchange/crypt: [b2862e3](https://github.com/xordataexchange/crypt/tree/b2862e3) - vbom.ml/util: db5cfe1 </ins> <del> # v1.19.0-rc.4 </del> <ins> # v1.20.0-rc.0 </ins> <del> ## Downloads for v1.19.0-rc.4 </del> <ins> ## Downloads for v1.20.0-rc.0 </ins> ### Source Code filename | sha512 hash -------- | ----------- <del> [kubernetes.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes.tar.gz) | 98bb6e2ac98a3176a9592982fec04b037d189de73cb7175d51596075bfd008c7ec0a2301b9511375626581f864ea74b5731e2ef2b4c70363f1860d11eb827de1 [kubernetes-src.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-src.tar.gz) | d4686f8d07fe6f324f46880a4dc5af9afa314a6b7dca82d0edb50290b769d25d18babcc5257a96a51a046052c7747e324b025a90a36ca5e62f67642bb03c44f6 </del> <ins> [kubernetes.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes.tar.gz) | acfee8658831f9503fccda0904798405434f17be7064a361a9f34c6ed04f1c0f685e79ca40cef5fcf34e3193bacbf467665e8dc277e0562ebdc929170034b5ae [kubernetes-src.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-src.tar.gz) | 9d962f8845e1fa221649cf0c0e178f0f03808486c49ea15ab5ec67861ec5aa948cf18bc0ee9b2067643c8332227973dd592e6a4457456a9d9d80e8ef28d5f7c3 </ins> ### Client binaries filename | sha512 hash -------- | ----------- <del> [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-client-darwin-amd64.tar.gz) | e9184ceb37491764c1ea2ef0b1eca55f27109bb973c7ff7c78e83c5945840baf28fdead21ef861b0c5cb81f4dc39d0af86ed7b17ed6f087f211084d0033dad11 [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-client-linux-386.tar.gz) | c9f1ec4e8d9c6245f955b2132c0fae6d851a6a49a5b7a2333c01ba9fafa3c4e8a07c6462e525179c25e308520502544ab4dc570e1b9d0090d58b6d18bcfcba47 [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-client-linux-amd64.tar.gz) | d23858b03c3554ad12517ce5f7855ceccaa9425c2d19fbc9cf902c0e796a8182f8b0e8eeeeefff0f46e960dfee96b2a2033a04a3194ac34dfd2a32003775d060 [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-client-linux-arm.tar.gz) | a745b3a06fe992713e4d7f921e2f36c5b39222d7b1a3e13299d15925743dd99965c2bdf05b4deda30a6f6232a40588e154fdd83f40d9d260d7ac8f70b18cad48 [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-client-linux-arm64.tar.gz) | 719b1f30e4bbb05d638ee78cf0145003a1e783bbd0c2f0952fcb30702dd27dfd44c3bc85baaf9a776e490ed53c638327ed1c0e5a882dc93c24d7cac20e4f1dd0 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-client-linux-ppc64le.tar.gz) | fba0794e9dc0f231da5a4e85e37c2d8260e5205574e0421f5122a7d60a05ca6546912519a28e8d6c241904617234e1b0b5c94f890853ad5ae4e329ef8085a092 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-client-linux-s390x.tar.gz) | edce96e37e37fd2b60e34fe56240461094e5784985790453becdfe09011305fcbd8a50ee5bf6d82a70d208d660796d0ddf160bed0292271b6617049db800962f [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-client-windows-386.tar.gz) | 06c849b35d886bebedfd8d906ac37ccda10e05b06542fefe6440268c5e937f235915e53daafe35076b68e0af0d4ddeab4240da55b09ee52fa26928945f1a2ecd [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-client-windows-amd64.tar.gz) | a13e6ec70f76d6056d5668b678ba6f223e35756cded6c84dfb58e28b3741fecfa7cb5e6ae2239392d770028d1b55ca8eb520c0b24e13fc3bd38720134b472d53 </del> <ins> [kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-darwin-amd64.tar.gz) | 062b57f1a450fe01d6184f104d81d376bdf5720010412821e315fd9b1b622a400ac91f996540daa66cee172006f3efade4eccc19265494f1a1d7cc9450f0b50a [kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-386.tar.gz) | 86e96d2c2046c5e62e02bef30a6643f25e01f1b3eba256cab7dd61252908540c26cb058490e9cecc5a9bad97d2b577f5968884e9f1a90237e302419f39e068bc [kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-amd64.tar.gz) | 619d3afb9ce902368390e71633396010e88e87c5fd848e3adc71571d1d4a25be002588415e5f83afee82460f8a7c9e0bd968335277cb8f8cb51e58d4bb43e64e [kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-arm.tar.gz) | 60965150a60ab3d05a248339786e0c7da4b89a04539c3719737b13d71302bac1dd9bcaa427d8a1f84a7b42d0c67801dce2de0005e9e47d21122868b32ac3d40f [kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-arm64.tar.gz) | 688e064f4ef6a17189dbb5af468c279b9de35e215c40500fb97b1d46692d222747023f9e07a7f7ba006400f9532a8912e69d7c5143f956b1dadca144c67ee711 [kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-ppc64le.tar.gz) | 47b8abc02b42b3b1de67da184921b5801d7e3cb09befac840c85913193fc5ac4e5e3ecfcb57da6b686ff21af9a3bd42ae6949d4744dbe6ad976794340e328b83 [kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-linux-s390x.tar.gz) | 971b41d3169f30e6c412e0254c180636abb7ccc8dcee6641b0e9877b69752fc61aa30b76c19c108969df654fe385da3cb3a44dd59d3c28dc45561392d7e08874 [kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-windows-386.tar.gz) | 2d34e8387e31531d9aca5655f2f0d18e75b01825dc1c39b7beb73a7b7b610e2ba429e5ca97d5c41a71b67e75e7096c86ab63fda9baab4c0878c1ccb3a1aefac8 [kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-client-windows-amd64.tar.gz) | f909640f4140693bb871936f10a40e79b43502105d0adb318b35bb7a64a770ad9d05a3a732368ccd3d15d496d75454789165bd1f5c2571da9a00569b3e6c007c </ins> ### Server binaries filename | sha512 hash -------- | ----------- <del> [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-server-linux-amd64.tar.gz) | ff7fbf211c29b94c19466337e6c142e733c8c0bac815a97906168e57d21ad1b2965e4b0033b525de8fed9a91ab943e3eb6d330f8095660e32be2791f8161a6a2 [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-server-linux-arm.tar.gz) | 218a35466ebcc0bc0e8eff9bbb2e58f0ac3bec6a75f45a7c1487aa4fc3e2bddb90b74e91a2b81bbbbb1eb1de2df310adab4c07c2a2c38a9973580b4f85734a1f [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-server-linux-arm64.tar.gz) | 8a81d727e63875d18336fda8bb6f570084553fc346b7e7df2fc3e1c04a8ef766f61d96d445537e4660ce2f46b170a04218a4d8a270b3ad373caa3f815c0fec93 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-server-linux-ppc64le.tar.gz) | 9b5afa44ac2e1232cd0c54b3602a2027bc8a08b30809b3ef973f75793b35a596491e6056d7995e493a1e4f48d83389240ac2e609b9f76d2715b8e115e6648716 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-server-linux-s390x.tar.gz) | f3034b2e88b5c1d362d84f78dfd1761d0fc21ada1cd6b1a6132a709c663a1206651df96c948534b3661f6b70b651e33021aced3a7574a0e5fc88ace73fff2a6f </del> <ins> [kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-amd64.tar.gz) | 0ea4458ae34108c633b4d48f1f128c6274dbc82b613492e78b3e0a2f656ac0df0bb9a75124e15d67c8e81850adcecf19f4ab0234c17247ee7ddf84f2df3e5eaa [kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-arm.tar.gz) | aef6a4d457faa29936603370f29a8523bb274211c3cb5101bd31aaf469c91ba6bd149ea99a4ccdd83352cf37e4d6508c5ee475ec10292bccd2f77ceea31e1c28 [kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-arm64.tar.gz) | 4829f473e9d60f9929ad17c70fdc2b6b6509ed75418be0b23a75b28580949736cb5b0bd6382070f93aa0a2a8863f0b1596daf965186ca749996c29d03ef7d8b8 [kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-ppc64le.tar.gz) | 9ab0790d382a3e28df1c013762c09da0085449cfd09d176d80be932806c24a715ea829be0075c3e221a2ad9cf06e726b4c39ab41987c1fb0fee2563e48206763 [kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-server-linux-s390x.tar.gz) | 98670b587e299856dd9821b7517a35f9a65835b915b153de08b66c54d82160438b66f774bf5306c07bc956d70ff709860bc23162225da5e89f995d3fdc1f0122 </ins> ### Node binaries filename | sha512 hash -------- | ----------- <del> [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-node-linux-amd64.tar.gz) | 2061a8f5bc2060b071564c92b693950eda7768a9ceb874982f0e91aa78284fb477becb55ecf099acae73c447271240caecefc19b3b29024e9b818e0639c2fc70 [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-node-linux-arm.tar.gz) | c06b817b191ff9a4b05bf70fc14edcf01d4ded204e489966b1761dd68d45d054028870301e45ebba648c0be097c7c42120867c8b28fdd625c8eb5a5bc3ace71d [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-node-linux-arm64.tar.gz) | 21efb3bf23628546de97210074f48e928fec211b81215eff8b10c3f5f7e79bb5911c1393a66a8363a0183fe299bf98b316c0c2d77a13c8c5b798255c056bd806 [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-node-linux-ppc64le.tar.gz) | ce31dd65b9cbfaabdc3c93e8afee0ea5606c64e6bf4452078bee73b1d328d23ebdbc871a00edd16fa4e676406a707cf9113fdaec76489681c379c35c3fd6aeb0 [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-node-linux-s390x.tar.gz) | 523a8e1d6e0eff70810e846c171b7f74a4aaecb25237addf541a9f8adb3797402b6e57abf9030f62d5bb333d5f5e8960199a44fe48696a4da98f7ed7d993cde1 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.19.0-rc.4/kubernetes-node-windows-amd64.tar.gz) | a7fbcd11ea8b6427e7846e39b2fdeae41d484320faa8f3e9b6a777d87ba62e7399ad0ec6a33d9a4675001898881e444f336eebd5c97b3903dee803834a646f3d </del> <ins> [kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-amd64.tar.gz) | 699e9c8d1837198312eade8eb6fec390f6a2fea9e08207d2f58e8bb6e3e799028aca69e4670aac0a4ba7cf0af683aee2c158bf78cc520c80edc876c8d94d521a [kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-arm.tar.gz) | f3b5eab0669490e3cd7e802693daf3555d08323dfff6e73a881fce00fed4690e8bdaf1610278d9de74036ca37631016075e5695a02158b7d3e7582b20ef7fa35 [kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-arm64.tar.gz) | e5012f77363561a609aaf791baaa17d09009819c4085a57132e5feb5366275a54640094e6ed1cba527f42b586c6d62999c2a5435edf5665ff0e114db4423c2ae [kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-ppc64le.tar.gz) | 2a6d6501620b1a9838dff05c66a40260cc22154a28027813346eb16e18c386bc3865298a46a0f08da71cd55149c5e7d07c4c4c431b4fd231486dd9d716548adb [kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-linux-s390x.tar.gz) | 5eca02777519e31428a1e5842fe540b813fb8c929c341bbc71dcfd60d98deb89060f8f37352e8977020e21e053379eead6478eb2d54ced66fb9d38d5f3142bf0 [kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.20.0-rc.0/kubernetes-node-windows-amd64.tar.gz) | 8ace02e7623dff894e863a2e0fa7dfb916368431d1723170713fe82e334c0ae0481b370855b71e2561de0fb64fed124281be604761ec08607230b66fb9ed1c03 </ins> <del> ## Changelog since v1.19.0-rc.3 </del> <ins> ## Changelog since v1.20.0-beta.2 </ins> ## Changes by Kind <del> ### Deprecation </del> <ins> ### Feature </ins> <del> - Kube-apiserver: the componentstatus API is deprecated. This API provided status of etcd, kube-scheduler, and kube-controller-manager components, but only worked when those components were local to the API server, and when kube-scheduler and kube-controller-manager exposed unsecured health endpoints. Instead of this API, etcd health is included in the kube-apiserver health check and kube-scheduler/kube-controller-manager health checks can be made directly against those components' health endpoints. ([#93570](https://github.com/kubernetes/kubernetes/pull/93570), [@liggitt](https://github.com/liggitt)) [SIG API Machinery, Apps and Cluster Lifecycle] </del> <ins> - Kubernetes is now built using go1.15.5 - build: Update to k/repo-infra@v0.1.2 (supports go1.15.5) ([#95776](https://github.com/kubernetes/kubernetes/pull/95776), [@justaugustus](https://github.com/justaugustus)) [SIG Cloud Provider, Instrumentation, Release and Testing] </ins> <del> ### Bug or Regression </del> <ins> ### Failing Test </ins> <del> - A panic in the apiserver caused by the `informer-sync` health checker is now fixed. ([#93600](https://github.com/kubernetes/kubernetes/pull/93600), [@ialidzhikov](https://github.com/ialidzhikov)) [SIG API Machinery] - EndpointSliceMirroring controller now copies labels from Endpoints to EndpointSlices. ([#93442](https://github.com/kubernetes/kubernetes/pull/93442), [@robscott](https://github.com/robscott)) [SIG Apps and Network] - Kube-apiserver: jsonpath expressions with consecutive recursive descent operators are no longer evaluated for custom resource printer columns ([#93408](https://github.com/kubernetes/kubernetes/pull/93408), [@joelsmith](https://github.com/joelsmith)) [SIG API Machinery] </del> <ins> - Resolves an issue running Ingress conformance tests on clusters which use finalizers on Ingress objects to manage releasing load balancer resources ([#96742](https://github.com/kubernetes/kubernetes/pull/96742), [@spencerhance](https://github.com/spencerhance)) [SIG Network and Testing] - The Conformance test "validates that there is no conflict between pods with same hostPort but different hostIP and protocol" now validates the connectivity to each hostPort, in addition to the functionality. ([#96627](https://github.com/kubernetes/kubernetes/pull/96627), [@aojea](https://github.com/aojea)) [SIG Scheduling and Testing] </ins> <del> ### Other (Cleanup or Flake) </del> <ins> ### Bug or Regression </ins> <del> - Build: Update to debian-base@v2.1.0 and debian-iptables@v12.1.1 ([#93667](https://github.com/kubernetes/kubernetes/pull/93667), [@justaugustus](https://github.com/justaugustus)) [SIG API Machinery, Release and Testing] </del> <ins> - Bump node-problem-detector version to v0.8.5 to fix OOM detection in with Linux kernels 5.1+ ([#96716](https://github.com/kubernetes/kubernetes/pull/96716), [@tosi3k](https://github.com/tosi3k)) [SIG Cloud Provider, Scalability and Testing] - Changes to timeout parameter handling in 1.20.0-beta.2 have been reverted to avoid breaking backwards compatibility with existing clients. ([#96727](https://github.com/kubernetes/kubernetes/pull/96727), [@liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing] - Duplicate owner reference entries in create/update/patch requests now get deduplicated by the API server. The client sending the request now receives a warning header in the API response. Clients should stop sending requests with duplicate owner references. The API server may reject such requests as early as 1.24. ([#96185](https://github.com/kubernetes/kubernetes/pull/96185), [@roycaihw](https://github.com/roycaihw)) [SIG API Machinery and Testing] - Fix: resize Azure disk issue when it's in attached state ([#96705](https://github.com/kubernetes/kubernetes/pull/96705), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider] - Fixed a bug where aggregator_unavailable_apiservice metrics were reported for deleted apiservices. ([#96421](https://github.com/kubernetes/kubernetes/pull/96421), [@dgrisonnet](https://github.com/dgrisonnet)) [SIG API Machinery and Instrumentation] - Fixes code generation for non-namespaced create subresources fake client test. ([#96586](https://github.com/kubernetes/kubernetes/pull/96586), [@Doude](https://github.com/Doude)) [SIG API Machinery] - HTTP/2 connection health check is enabled by default in all Kubernetes clients. The feature should work out-of-the-box. If needed, users can tune the feature via the HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS environment variables. The feature is disabled if HTTP2_READ_IDLE_TIMEOUT_SECONDS is set to 0. ([#95981](https://github.com/kubernetes/kubernetes/pull/95981), [@caesarxuchao](https://github.com/caesarxuchao)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node] - Kubeadm: fix coredns migration should be triggered when there are newdefault configs during kubeadm upgrade ([#96907](https://github.com/kubernetes/kubernetes/pull/96907), [@pacoxu](https://github.com/pacoxu)) [SIG Cluster Lifecycle] - Reduce volume name length for vsphere volumes ([#96533](https://github.com/kubernetes/kubernetes/pull/96533), [@gnufied](https://github.com/gnufied)) [SIG Storage] - Resolves a regression in 1.19+ with workloads targeting deprecated beta os/arch labels getting stuck in NodeAffinity status on node startup. ([#96810](https://github.com/kubernetes/kubernetes/pull/96810), [@liggitt](https://github.com/liggitt)) [SIG Node] </ins> ## Dependencies