Histogram	 Admission controller latency histogram in seconds, identified by name and broken out for each operation and API resource and type (validate or admit).
name

operation

rejected

type

   None

apiserver_admission_step_admission_duration_seconds	 STABLE	 Histogram	 Admission sub-step latency histogram in seconds, broken out for each operation and API resource and step type (validate or admit).
operation

rejected

type

   None

apiserver_admission_webhook_admission_duration_seconds	 STABLE	 Histogram	 Admission webhook latency histogram in seconds, identified by name and broken out for each operation and API resource and type (validate or admit).
name

operation

rejected

type

   None

apiserver_current_inflight_requests	 STABLE	 Gauge	 Maximal number of currently used inflight request limit of this apiserver per request kind in last second.
request_kind

   None

apiserver_longrunning_requests	 STABLE	 Gauge	 Gauge of all active long-running apiserver requests broken out by verb, group, version, resource, scope and component. Not all requests are tracked this way.
component

group

resource

scope

subresource

verb

version

   None

apiserver_request_duration_seconds	 STABLE	 Histogram	 Response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope and component.
component

dry_run

group

resource

scope

subresource

verb

version

   None

apiserver_request_total	 STABLE	 Counter	 Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, and HTTP response code.
code

component

dry_run

group

resource

scope

subresource

verb

version

   None

apiserver_requested_deprecated_apis	 STABLE	 Gauge	 Gauge of deprecated APIs that have been requested, broken out by API group, version, resource, subresource, and removed_release.
group

removed_release

resource

subresource

version

   None

apiserver_response_sizes	 STABLE	 Histogram	 Response size distribution in bytes for each group, version, verb, resource, subresource, scope and component.
component

group

resource

scope

subresource

verb

version

   None

apiserver_storage_objects	 STABLE	 Gauge	 Number of stored objects at the time of last check split by kind.
resource

cronjob_controller_job_creation_skew_duration_seconds	 STABLE	 Histogram	 Time between when a cronjob is scheduled to be run, and when the corresponding job is created

job_controller_job_pods_finished_total	 STABLE	 Counter	 The number of finished Pods that are fully tracked
completion_mode

result

job_controller_job_sync_duration_seconds	 STABLE	 Histogram	 The time it took to sync a job
action

completion_mode

result

job_controller_job_syncs_total	 STABLE	 Counter	 The number of job syncs
action

completion_mode

result

job_controller_jobs_finished_total	 STABLE	 Counter	 The number of finished jobs
completion_mode

reason

result

   None

node_collector_evictions_total	 STABLE	 Counter	 Number of Node evictions that happened since current instance of NodeController started.
zone

   None

scheduler_framework_extension_point_duration_seconds	 STABLE	 Histogram	 Latency for running all plugins of a specific extension point.
extension_point

profile

status

   None

scheduler_pending_pods	 STABLE	 Gauge	  Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulablePods that the scheduler attempted to schedule and failed; 'gated' is the number of unschedulable pods that the scheduler never attempted to schedule because they are gated.	   Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulablePods.
queue

   None

scheduler_pod_scheduling_attempts	 STABLE	 Histogram	 Number of attempts to successfully schedule a pod.
   None	 None

scheduler_pod_scheduling_duration_seconds	 STABLE	 Histogram	 E2e latency for a pod being scheduled which may include multiple scheduling attempts.
attempts

   None

scheduler_preemption_attempts_total	 STABLE	 Counter	 Total preemption attempts in the cluster till now
   None	 None

scheduler_preemption_victims	 STABLE	 Histogram	 Number of selected preemption victims
   None	 None

scheduler_queue_incoming_pods_total	 STABLE	 Counter	 Number of pods added to scheduling queues by event and queue type.
event

queue

   None

scheduler_schedule_attempts_total	 STABLE	 Counter	 Number of attempts to schedule pods, by the result. 'unschedulable' means a pod could not be scheduled, while 'error' means an internal scheduler problem.
profile

result

   None

scheduler_scheduling_attempt_duration_seconds	 STABLE	 Histogram	 Scheduling attempt latency in seconds (scheduling algorithm + binding)
profile

result

   None


Help	 Labels	 Const Labels	  Deprecated Version


Counter	 Counter of OpenAPI v2 spec regeneration count broken down by causing APIService name and reason.
apiservice

reason

   None

aggregator_openapi_v2_regeneration_duration	 ALPHA	 Gauge	 Gauge of OpenAPI v2 spec regeneration duration in seconds.
reason

   None

aggregator_unavailable_apiservice	 ALPHA	 Custom	 Gauge of APIServices which are marked as unavailable broken down by APIService name.
name

   None

aggregator_unavailable_apiservice_total	 ALPHA	 Counter	 Counter of APIServices which are marked as unavailable broken down by APIService name and reason.
name

reason

   None

apiextensions_openapi_v2_regeneration_count	 ALPHA	 Counter	 Counter of OpenAPI v2 spec regeneration count broken down by causing CRD name and reason.
crd

reason

   None

apiextensions_openapi_v3_regeneration_count	 ALPHA	 Counter	 Counter of OpenAPI v3 spec regeneration count broken down by group, version, causing CRD and reason.
crd

group

reason

version

   None

apiserver_admission_step_admission_duration_seconds_summary	 ALPHA	 Summary	 Admission sub-step latency summary in seconds, broken out for each operation and API resource and step type (validate or admit).
operation

rejected

type

   None

apiserver_admission_webhook_fail_open_count	 ALPHA	 Counter	 Admission webhook fail open count, identified by name and broken out for each admission type (validating or mutating).
name

type

   None

apiserver_admission_webhook_rejection_count	 ALPHA	 Counter	 Admission webhook rejection count, identified by name and broken out for each admission type (validating or admit) and operation. Additional labels specify an error type (calling_webhook_error or apiserver_internal_error if an error occurred; no_error otherwise) and optionally a non-zero rejection code if the webhook rejects the request with an HTTP status code (honored by the apiserver when the code is greater or equal to 400). Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
error_type

name

operation

rejection_code

type

   None

apiserver_admission_webhook_request_total	 ALPHA	 Counter	 Admission webhook request total, identified by name and broken out for each admission type (validating or mutating) and operation. Additional labels specify whether the request was rejected or not and an HTTP status code. Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
code

name

operation

rejected

type

   None

apiserver_audit_error_total	 ALPHA	 Counter	 Counter of audit events that failed to be audited properly. Plugin identifies the plugin affected by the error.
plugin

   None

apiserver_audit_event_total	 ALPHA	 Counter	 Counter of audit events generated and sent to the audit backend.
   None	 None

apiserver_audit_level_total	 ALPHA	 Counter	 Counter of policy levels for audit events (1 per request).
level

   None

apiserver_audit_requests_rejected_total	 ALPHA	 Counter	 Counter of apiserver requests rejected due to an error in audit logging backend.
   None	 None

apiserver_cache_list_fetched_objects_total	 ALPHA	 Counter	 Number of objects read from watch cache in the course of serving a LIST request
index

resource_prefix

   None

apiserver_cache_list_returned_objects_total	 ALPHA	 Counter	 Number of objects returned for a LIST request from watch cache
resource_prefix

   None

apiserver_cache_list_total	 ALPHA	 Counter	 Number of LIST requests served from watch cache
index

resource_prefix

   None

apiserver_cel_compilation_duration_seconds	 ALPHA	 Histogram
   None	 None

apiserver_cel_evaluation_duration_seconds	 ALPHA	 Histogram
   None	 None

apiserver_certificates_registry_csr_honored_duration_total	 ALPHA	 Counter	 Total number of issued CSRs with a requested duration that was honored, sliced by signer (only kubernetes.io signer names are specifically identified)
signerName

   None

apiserver_certificates_registry_csr_requested_duration_total	 ALPHA	 Counter	 Total number of issued CSRs with a requested duration, sliced by signer (only kubernetes.io signer names are specifically identified)
signerName

   None

apiserver_client_certificate_expiration_seconds	 ALPHA	 Histogram	 Distribution of the remaining lifetime on the certificate used to authenticate a request.
   None	 None

apiserver_crd_webhook_conversion_duration_seconds	 ALPHA	 Histogram	 CRD webhook conversion duration in seconds
crd_name

from_version

succeeded

to_version

   None

apiserver_current_inqueue_requests	 ALPHA	 Gauge	 Maximal number of queued requests in this apiserver per request kind in last second.
request_kind

   None

apiserver_delegated_authn_request_duration_seconds	 ALPHA	 Histogram	 Request latency in seconds. Broken down by status code.
code

   None

apiserver_delegated_authn_request_total	 ALPHA	 Counter	 Number of HTTP requests partitioned by status code.
code

   None

apiserver_delegated_authz_request_duration_seconds	 ALPHA	 Histogram	 Request latency in seconds. Broken down by status code.
code

   None

apiserver_delegated_authz_request_total	 ALPHA	 Counter	 Number of HTTP requests partitioned by status code.
code

   None

apiserver_egress_dialer_dial_duration_seconds	 ALPHA	 Histogram	 Dial latency histogram in seconds, labeled by the protocol (http-connect or grpc), transport (tcp or uds)
protocol

transport

   None

apiserver_egress_dialer_dial_failure_count	 ALPHA	 Counter	 Dial failure count, labeled by the protocol (http-connect or grpc), transport (tcp or uds), and stage (connect or proxy). The stage indicates at which stage the dial failed
protocol

stage

transport

apiserver_egress_dialer_dial_start_total	 ALPHA	 Counter	 Dial starts, labeled by the protocol (http-connect or grpc) and transport (tcp or uds).
protocol

transport

   None

apiserver_envelope_encryption_dek_cache_fill_percent	 ALPHA	 Gauge	 Percent of the cache slots currently occupied by cached DEKs.
   None	 None

apiserver_envelope_encryption_dek_cache_inter_arrival_time_seconds	 ALPHA	 Histogram	 Time (in seconds) of inter arrival of transformation requests.
transformation_type

   None

apiserver_flowcontrol_current_executing_requests	 ALPHA	 Gauge	 Number of requests in initial (for a WATCH) or any (for a non-WATCH) execution stage in the API Priority and Fairness subsystem
flow_schema

priority_level

   None

apiserver_flowcontrol_current_inqueue_requests	 ALPHA	 Gauge	 Number of requests currently pending in queues of the API Priority and Fairness subsystem
flow_schema

priority_level

apiserver_flowcontrol_current_limit_seats	 ALPHA	 Gauge	 current derived number of execution seats available to each priority level
priority_level

   None

apiserver_flowcontrol_current_r	 ALPHA	 Gauge	 R(time of last change)
priority_level

apiserver_flowcontrol_demand_seats	 ALPHA	 TimingRatioHistogram	 Observations, at the end of every nanosecond, of (the number of seats each priority level could use) / (nominal number of seats for that level)
priority_level

apiserver_flowcontrol_demand_seats_average	 ALPHA	 Gauge	 Time-weighted average, over last adjustment period, of demand_seats
priority_level

apiserver_flowcontrol_demand_seats_high_watermark	 ALPHA	 Gauge	 High watermark, over last adjustment period, of demand_seats
priority_level

apiserver_flowcontrol_demand_seats_smoothed	 ALPHA	 Gauge	 Smoothed seat demands
priority_level

apiserver_flowcontrol_demand_seats_stdev	 ALPHA	 Gauge	 Time-weighted standard deviation, over last adjustment period, of demand_seats
priority_level

   None

apiserver_flowcontrol_dispatch_r	 ALPHA	 Gauge	 R(time of last dispatch)
priority_level

   None

apiserver_flowcontrol_dispatched_requests_total	 ALPHA	 Counter	 Number of requests executed by API Priority and Fairness subsystem
flow_schema

priority_level

   None

apiserver_flowcontrol_epoch_advance_total	 ALPHA	 Counter	 Number of times the queueset's progress meter jumped backward
priority_level

success

   None

apiserver_flowcontrol_latest_s	 ALPHA	 Gauge	 S(most recently dispatched request)
priority_level

apiserver_flowcontrol_lower_limit_seats	 ALPHA	 Gauge	 Configured lower bound on number of execution seats available to each priority level
priority_level

   None

apiserver_flowcontrol_next_discounted_s_bounds	 ALPHA	 Gauge	 min and max, over queues, of S(oldest waiting request in queue) - estimated work in progress
bound

priority_level

   None

apiserver_flowcontrol_next_s_bounds	 ALPHA	 Gauge	 min and max, over queues, of S(oldest waiting request in queue)
bound

priority_level

apiserver_flowcontrol_nominal_limit_seats	 ALPHA	 Gauge	 Nominal number of execution seats configured for each priority level
priority_level

   None

apiserver_flowcontrol_priority_level_request_utilization	 ALPHA	 TimingRatioHistogram	 Observations, at the end of every nanosecond, of number of requests (as a fraction of the relevant limit) waiting or in any stage of execution (but only initial stage for WATCHes)
phase

priority_level

   None

apiserver_flowcontrol_priority_level_seat_utilization	 ALPHA	 TimingRatioHistogram	 Observations, at the end of every nanosecond, of utilization of seats for any stage of execution (but only initial stage for WATCHes)
priority_level

phase:executing

   map[phase:executing]

apiserver_flowcontrol_read_vs_write_current_requests	 ALPHA	 TimingRatioHistogram	 Observations, at the end of every nanosecond, of the number of requests (as a fraction of the relevant limit) waiting or in regular stage of execution
phase

request_kind

   None

apiserver_flowcontrol_rejected_requests_total	 ALPHA	 Counter	 Number of requests rejected by API Priority and Fairness subsystem
flow_schema

priority_level

reason

   None

apiserver_flowcontrol_request_concurrency_in_use	 ALPHA	 Gauge	 Concurrency (number of seats) occupied by the currently executing (initial stage for a WATCH, any stage otherwise) requests in the API Priority and Fairness subsystem
flow_schema

priority_level

   None

apiserver_flowcontrol_request_concurrency_limit	 ALPHA	 Gauge	 Shared concurrency limit in the API Priority and Fairness subsystem
priority_level

   None

apiserver_flowcontrol_request_dispatch_no_accommodation_total	 ALPHA	 Counter	 Number of times a dispatch attempt resulted in a non accommodation due to lack of available seats
flow_schema

priority_level

   None

apiserver_flowcontrol_request_execution_seconds	 ALPHA	 Histogram	 Duration of initial stage (for a WATCH) or any (for a non-WATCH) stage of request execution in the API Priority and Fairness subsystem
flow_schema

priority_level

type

   None

apiserver_flowcontrol_request_queue_length_after_enqueue	 ALPHA	 Histogram	 Length of queue in the API Priority and Fairness subsystem, as seen by each request after it is enqueued
flow_schema

priority_level

   None

apiserver_flowcontrol_request_wait_duration_seconds	 ALPHA	 Histogram	 Length of time a request spent waiting in its queue
execute

flow_schema

priority_level

apiserver_flowcontrol_seat_fair_frac	 ALPHA	 Gauge	 Fair fraction of server's concurrency to allocate to each priority level that can use it

apiserver_flowcontrol_target_seats	 ALPHA	 Gauge	 Seat allocation targets
priority_level

apiserver_flowcontrol_upper_limit_seats	 ALPHA	 Gauge	 Configured upper bound on number of execution seats available to each priority level
priority_level

   None

apiserver_flowcontrol_watch_count_samples	 ALPHA	 Histogram	 count of watchers for mutating requests in API Priority and Fairness
flow_schema

priority_level

   None

apiserver_flowcontrol_work_estimated_seats	 ALPHA	 Histogram	 Number of estimated seats (maximum of initial and final seats) associated with requests in API Priority and Fairness
flow_schema

priority_level

   None

apiserver_init_events_total	 ALPHA	 Counter	 Counter of init events processed in watch cache broken by resource type.
resource

   None

apiserver_kube_aggregator_x509_insecure_sha1_total	 ALPHA	 Counter	 Counts the number of requests to servers with insecure SHA1 signatures in their serving certificate OR the number of connection failures due to the insecure SHA1 signatures (either/or, based on the runtime environment)
   None	 None

apiserver_kube_aggregator_x509_missing_san_total	 ALPHA	 Counter	 Counts the number of requests to servers missing SAN extension in their serving certificate OR the number of connection failures due to the lack of x509 certificate SAN extension missing (either/or, based on the runtime environment)
   None	 None

apiserver_request_aborts_total	 ALPHA	 Counter	 Number of requests which apiserver aborted possibly due to a timeout, for each group, version, verb, resource, subresource and scope
group

resource

scope

subresource

verb

version

   None

apiserver_request_body_sizes	 ALPHA	 Histogram	 Apiserver request body sizes broken out by size.
resource

verb

   None

apiserver_request_filter_duration_seconds	 ALPHA	 Histogram	 Request filter latency distribution in seconds, for each filter type
filter

   None

apiserver_request_post_timeout_total	 ALPHA	 Counter	 Tracks the activity of the request handlers after the associated requests have been timed out by the apiserver
source

status

apiserver_request_sli_duration_seconds	 ALPHA	 Histogram	 Response latency distribution (not counting webhook duration) in seconds for each verb, group, version, resource, subresource, scope and component.
component

group

resource

scope

subresource

verb

version

   None

apiserver_request_slo_duration_seconds	 ALPHA	 Histogram	 Response latency distribution (not counting webhook duration) in seconds for each verb, group, version, resource, subresource, scope and component.
component

group

resource

scope

subresource

verb

version
	  	 1.27.0
   None

apiserver_request_terminations_total	 ALPHA	 Counter	 Number of requests which apiserver terminated in self-defense.
code

component

group

resource

scope

subresource

verb

version

   None

apiserver_request_timestamp_comparison_time	 ALPHA	 Histogram	 Time taken for comparison of old vs new objects in UPDATE or PATCH requests
code_path

   None

apiserver_selfrequest_total	 ALPHA	 Counter	 Counter of apiserver self-requests broken out for each verb, API resource and subresource.
resource

subresource

verb

   None

apiserver_storage_data_key_generation_duration_seconds	 ALPHA	 Histogram	 Latencies in seconds of data encryption key(DEK) generation operations.
   None	 None

apiserver_storage_data_key_generation_failures_total	 ALPHA	 Counter	 Total number of failed data encryption key(DEK) generation operations.
   None	 None

apiserver_storage_db_total_size_in_bytes	 ALPHA	 Gauge	 Total size of the storage database file physically allocated in bytes.
endpoint

   None

apiserver_storage_envelope_transformation_cache_misses_total	 ALPHA	 Counter	 Total number of cache misses while accessing key decryption key(KEK).
   None	 None

apiserver_storage_list_evaluated_objects_total	 ALPHA	 Counter	 Number of objects tested in the course of serving a LIST request from storage
resource

   None

apiserver_storage_list_fetched_objects_total	 ALPHA	 Counter	 Number of objects read from storage in the course of serving a LIST request
resource

   None

apiserver_storage_list_returned_objects_total	 ALPHA	 Counter	 Number of objects returned for a LIST request from storage
resource

   None

apiserver_storage_list_total	 ALPHA	 Counter	 Number of LIST requests served from storage
resource

   None

apiserver_storage_transformation_duration_seconds	 ALPHA	 Histogram	 Latencies in seconds of value transformation operations.
transformation_type

   None

apiserver_storage_transformation_operations_total	 ALPHA	 Counter	 Total number of transformations.
status

transformation_type

transformer_prefix

   None

apiserver_terminated_watchers_total	 ALPHA	 Counter	 Counter of watchers closed due to unresponsiveness broken by resource type.
resource

   None

apiserver_tls_handshake_errors_total	 ALPHA	 Counter	 Number of requests dropped with 'TLS handshake error from' error
   None	 None

apiserver_validating_admission_policy_check_duration_seconds	 ALPHA	 Histogram	  Validation admission latency for individual validation expressions in seconds, labeled by policy and further including binding, state and enforcement action taken.
enforcement_action

policy

policy_binding

state

   Validation admission latency for individual validation expressions in seconds, labeled by policy and param resource, further including binding, state and enforcement action taken.
enforcement_action

params

policy

policy_binding

state

validation_expression
	 None

apiserver_validating_admission_policy_check_total	 ALPHA	 Counter	  Validation admission policy check total, labeled by policy and further identified by binding, enforcement action taken, and state.
enforcement_action

policy

policy_binding

state

   Validation admission policy check total, labeled by policy and param resource, and further identified by binding, validation expression, enforcement action taken, and state.
enforcement_action

params

policy

policy_binding

state

validation_expression
	 None

apiserver_validating_admission_policy_definition_total	 ALPHA	 Counter	 Validation admission policy count total, labeled by state and enforcement action.
enforcement_action

state

   None

apiserver_watch_cache_events_dispatched_total	 ALPHA	 Counter	 Counter of events dispatched in watch cache broken by resource type.
resource

   None

apiserver_watch_cache_initializations_total	 ALPHA	 Counter	 Counter of watch cache initializations broken by resource type.
resource

   None

apiserver_watch_events_sizes	 ALPHA	 Histogram	 Watch event size distribution in bytes
group

kind

version

   None

apiserver_watch_events_total	 ALPHA	 Counter	 Number of events sent in watch clients
group

kind

version

   None

apiserver_webhooks_x509_insecure_sha1_total	 ALPHA	 Counter	 Counts the number of requests to servers with insecure SHA1 signatures in their serving certificate OR the number of connection failures due to the insecure SHA1 signatures (either/or, based on the runtime environment)
   None	 None

apiserver_webhooks_x509_missing_san_total	 ALPHA	 Counter	 Counts the number of requests to servers missing SAN extension in their serving certificate OR the number of connection failures due to the lack of x509 certificate SAN extension missing (either/or, based on the runtime environment)
   None	 None

attachdetach_controller_forced_detaches	 ALPHA	 Counter	 Number of times the A/D Controller performed a forced detach
   None	 None

attachdetach_controller_total_volumes	 ALPHA	 Custom	 Number of volumes in A/D Controller
plugin_name

state

   None

authenticated_user_requests	 ALPHA	 Counter	 Counter of authenticated requests broken out by username.
username

   None

authentication_attempts	 ALPHA	 Counter	 Counter of authenticated attempts.
result

   None

authentication_duration_seconds	 ALPHA	 Histogram	 Authentication duration in seconds broken out by result.
result

   None

authentication_token_cache_active_fetch_count	 ALPHA	 Gauge
status

   None

authentication_token_cache_fetch_total	 ALPHA	 Counter
status

   None

authentication_token_cache_request_duration_seconds	 ALPHA	 Histogram
status

   None

authentication_token_cache_request_total	 ALPHA	 Counter
status

   None

cloudprovider_aws_api_request_duration_seconds	 ALPHA	 Histogram	 Latency of AWS API calls
request

   None

cloudprovider_aws_api_request_errors	 ALPHA	 Counter	 AWS API errors
request

   None

cloudprovider_aws_api_throttled_requests_total	 ALPHA	 Counter	 AWS API throttled requests
operation_name

   None

cloudprovider_azure_api_request_duration_seconds	 ALPHA	 Histogram	 Latency of an Azure API call
request

resource_group

source

subscription_id

   None

cloudprovider_azure_api_request_errors	 ALPHA	 Counter	 Number of errors for an Azure API call
request

resource_group

source

subscription_id

   None

cloudprovider_azure_api_request_ratelimited_count	 ALPHA	 Counter	 Number of rate limited Azure API calls
request

resource_group

source

subscription_id

   None

cloudprovider_azure_api_request_throttled_count	 ALPHA	 Counter	 Number of throttled Azure API calls
request

resource_group

source

subscription_id

   None

cloudprovider_azure_op_duration_seconds	 ALPHA	 Histogram	 Latency of an Azure service operation
request

resource_group

source

subscription_id

   None

cloudprovider_azure_op_failure_count	 ALPHA	 Counter	 Number of failed Azure service operations
request

resource_group

source

subscription_id

   None

cloudprovider_gce_api_request_duration_seconds	 ALPHA	 Histogram	 Latency of a GCE API call
region

request

version

zone

   None

cloudprovider_gce_api_request_errors	 ALPHA	 Counter	 Number of errors for an API call
region

request

version

zone

   None

cloudprovider_vsphere_api_request_duration_seconds	 ALPHA	 Histogram	 Latency of vsphere api call
request

   None

cloudprovider_vsphere_api_request_errors	 ALPHA	 Counter	 vsphere Api errors
request

   None

cloudprovider_vsphere_operation_duration_seconds	 ALPHA	 Histogram	 Latency of vsphere operation call
operation

   None

cloudprovider_vsphere_operation_errors	 ALPHA	 Counter	 vsphere operation errors
operation

   None

cloudprovider_vsphere_vcenter_versions	 ALPHA	 Custom	 Versions for connected vSphere vCenters
hostname

version

build

   None

container_cpu_usage_seconds_total	 ALPHA	 Custom	 Cumulative cpu time consumed by the container in core-seconds
container

pod

namespace

   None

container_memory_working_set_bytes	 ALPHA	 Custom	 Current working set of the container in bytes
container

pod

namespace

   None

container_start_time_seconds	 ALPHA	 Custom	 Start time of the container since unix epoch in seconds
container

pod

namespace

   None

cronjob_controller_cronjob_job_creation_skew_duration_seconds	 ALPHA	 Histogram	 Time between when a cronjob is scheduled to be run, and when the corresponding job is created	 None	 None

csi_operations_seconds	 ALPHA	 Histogram	 Container Storage Interface operation duration with gRPC error code status total
driver_name

grpc_status_code

method_name

migrated

   None

endpoint_slice_controller_changes	 ALPHA	 Counter	 Number of EndpointSlice changes
operation

   None

endpoint_slice_controller_desired_endpoint_slices	 ALPHA	 Gauge	 Number of EndpointSlices that would exist with perfect endpoint allocation
   None	 None

endpoint_slice_controller_endpoints_added_per_sync	 ALPHA	 Histogram	 Number of endpoints added on each Service sync
   None	 None

endpoint_slice_controller_endpoints_desired	 ALPHA	 Gauge	 Number of endpoints desired
   None	 None

endpoint_slice_controller_endpoints_removed_per_sync	 ALPHA	 Histogram	 Number of endpoints removed on each Service sync
   None	 None

endpoint_slice_controller_endpointslices_changed_per_sync	 ALPHA	 Histogram	 Number of EndpointSlices changed on each Service sync
topology

   None

endpoint_slice_controller_num_endpoint_slices	 ALPHA	 Gauge	 Number of EndpointSlices
   None	 None

endpoint_slice_controller_syncs	 ALPHA	 Counter	 Number of EndpointSlice syncs
result

   None

endpoint_slice_mirroring_controller_addresses_skipped_per_sync	 ALPHA	 Histogram	 Number of addresses skipped on each Endpoints sync due to being invalid or exceeding MaxEndpointsPerSubset
   None	 None

endpoint_slice_mirroring_controller_changes	 ALPHA	 Counter	 Number of EndpointSlice changes
operation

   None

endpoint_slice_mirroring_controller_desired_endpoint_slices	 ALPHA	 Gauge	 Number of EndpointSlices that would exist with perfect endpoint allocation
   None	 None

endpoint_slice_mirroring_controller_endpoints_added_per_sync	 ALPHA	 Histogram	 Number of endpoints added on each Endpoints sync
   None	 None

endpoint_slice_mirroring_controller_endpoints_desired	 ALPHA	 Gauge	 Number of endpoints desired
   None	 None

endpoint_slice_mirroring_controller_endpoints_removed_per_sync	 ALPHA	 Histogram	 Number of endpoints removed on each Endpoints sync
   None	 None

endpoint_slice_mirroring_controller_endpoints_sync_duration	 ALPHA	 Histogram	 Duration of syncEndpoints() in seconds
   None	 None

endpoint_slice_mirroring_controller_endpoints_updated_per_sync	 ALPHA	 Histogram	 Number of endpoints updated on each Endpoints sync
   None	 None

endpoint_slice_mirroring_controller_num_endpoint_slices	 ALPHA	 Gauge	 Number of EndpointSlices
   None	 None

ephemeral_volume_controller_create_failures_total	 ALPHA	 Counter	 Number of PersistenVolumeClaims creation requests
   None	 None

ephemeral_volume_controller_create_total	 ALPHA	 Counter	 Number of PersistenVolumeClaims creation requests
   None	 None

etcd_bookmark_counts	 ALPHA	 Gauge	 Number of etcd bookmarks (progress notify events) split by kind.
resource

   None

etcd_lease_object_counts	 ALPHA	 Histogram	 Number of objects attached to a single etcd lease.
   None	 None

etcd_request_duration_seconds	 ALPHA	 Histogram	 Etcd request latency in seconds for each operation and object type.
operation

type

   None

etcd_version_info	 ALPHA	 Gauge	 Etcd server's binary version
binary_version

   None

field_validation_request_duration_seconds	 ALPHA	 Histogram	 Response latency distribution in seconds for each field validation value and whether field validation is enabled or not
enabled

field_validation

   None

garbagecollector_controller_resources_sync_error_total	 ALPHA	 Counter	 Number of garbage collector resources sync errors
   None	 None

get_token_count	 ALPHA	 Counter	 Counter of total Token() requests to the alternate token source
   None	 None

get_token_fail_count	 ALPHA	 Counter	 Counter of failed Token() requests to the alternate token source
   None	 None

job_controller_job_finished_total	 ALPHA	 Counter	 The number of finished job
completion_mode

reason

result
	 None

job_controller_job_pods_finished_total	 ALPHA	 Counter	 The number of finished Pods that are fully tracked
completion_mode

result
	 None

job_controller_job_sync_duration_seconds	 ALPHA	 Histogram	 The time it took to sync a job
action

completion_mode

result
	 None

job_controller_job_sync_total	 ALPHA	 Counter	 The number of job syncs
action

completion_mode

result
	 None

job_controller_pod_failures_handled_by_failure_policy_total	 ALPHA	 Counter	 `The number of failed Pods handled by failure policy with, 			respect to the failure policy action applied based on the matched, 			rule. Possible values of the action label correspond to the, 			possible values for the failure policy rule action, which are:, 			"FailJob", "Ignore" and "Count".`
action

   None

job_controller_terminated_pods_tracking_finalizer_total	 ALPHA	 Counter	 `The number of terminated pods (phase=Failed|Succeeded), that have the finalizer batch.kubernetes.io/job-tracking, The event label can be "add" or "delete".`
event

   None

kube_apiserver_clusterip_allocator_allocated_ips	 ALPHA	 Gauge	 Gauge measuring the number of allocated IPs for Services
cidr

   None

kube_apiserver_clusterip_allocator_allocation_errors_total	 ALPHA	 Counter	 Number of errors trying to allocate Cluster IPs
cidr

scope

   None

kube_apiserver_clusterip_allocator_allocation_total	 ALPHA	 Counter	 Number of Cluster IPs allocations
cidr

scope

   None

kube_apiserver_clusterip_allocator_available_ips	 ALPHA	 Gauge	 Gauge measuring the number of available IPs for Services
cidr

   None

kube_apiserver_pod_logs_pods_logs_backend_tls_failure_total	 ALPHA	 Counter	 Total number of requests for pods/logs that failed due to kubelet server TLS verification
   None	 None

kube_apiserver_pod_logs_pods_logs_insecure_backend_total	 ALPHA	 Counter	 Total number of requests for pods/logs sliced by usage type: enforce_tls, skip_tls_allowed, skip_tls_denied
usage

   None

kube_pod_resource_limit	 ALPHA	 Custom	 Resources limit for workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any.
namespace

pod

node

scheduler

priority

resource

unit

   None

kube_pod_resource_request	 ALPHA	 Custom	 Resources requested by workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any.
namespace

pod

node

scheduler

priority

resource

unit

   None

kubelet_certificate_manager_client_expiration_renew_errors	 ALPHA	 Counter	 Counter of certificate renewal errors.
   None	 None

kubelet_certificate_manager_client_ttl_seconds	 ALPHA	 Gauge	 Gauge of the TTL (time-to-live) of the Kubelet's client certificate. The value is in seconds until certificate expiry (negative if already expired). If client certificate is invalid or unused, the value will be +INF.
   None	 None

kubelet_certificate_manager_server_rotation_seconds	 ALPHA	 Histogram	 Histogram of the number of seconds the previous certificate lived before being rotated.
   None	 None

kubelet_certificate_manager_server_ttl_seconds	 ALPHA	 Gauge	 Gauge of the shortest TTL (time-to-live) of the Kubelet's serving certificate. The value is in seconds until certificate expiry (negative if already expired). If serving certificate is invalid or unused, the value will be +INF.
   None	 None

kubelet_cgroup_manager_duration_seconds	 ALPHA	 Histogram	 Duration in seconds for cgroup manager operations. Broken down by method.
operation_type

   None

kubelet_container_log_filesystem_used_bytes	 ALPHA	 Custom	 Bytes used by the container's logs on the filesystem.
uid

namespace

pod

container

   None

kubelet_containers_per_pod_count	 ALPHA	 Histogram	 The number of containers per pod.
   None	 None

kubelet_cpu_manager_pinning_errors_total	 ALPHA	 Counter	 The number of cpu core allocations which required pinning failed.
   None	 None

kubelet_cpu_manager_pinning_requests_total	 ALPHA	 Counter	 The number of cpu core allocations which required pinning.

kubelet_credential_provider_plugin_duration	 ALPHA	 Histogram	 Duration of execution in seconds for credential provider plugin
plugin_name

kubelet_credential_provider_plugin_errors	 ALPHA	 Counter	 Number of errors from credential provider plugin
plugin_name

   None	 None

kubelet_device_plugin_alloc_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to serve a device plugin Allocation request. Broken down by resource name.
resource_name

   None

kubelet_device_plugin_registration_total	 ALPHA	 Counter	 Cumulative number of device plugin registrations. Broken down by resource name.
resource_name

   None

kubelet_eviction_stats_age_seconds	 ALPHA	 Histogram	 Time between when stats are collected, and when pod is evicted based on those stats by eviction signal
eviction_signal

   None

kubelet_evictions	 ALPHA	 Counter	 Cumulative number of pod evictions by eviction signal
eviction_signal

   None

kubelet_graceful_shutdown_end_time_seconds	 ALPHA	 Gauge	 Last graceful shutdown start time since unix epoch in seconds
   None	 None

kubelet_graceful_shutdown_start_time_seconds	 ALPHA	 Gauge	 Last graceful shutdown start time since unix epoch in seconds
   None	 None

kubelet_http_inflight_requests	 ALPHA	 Gauge	 Number of the inflight http requests
long_running

method

path

server_type

   None

kubelet_http_requests_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to serve http requests
long_running

method

path

server_type

   None

kubelet_http_requests_total	 ALPHA	 Counter	 Number of the http requests received since the server started
long_running

method

path

server_type

   None

kubelet_kubelet_credential_provider_plugin_duration	 ALPHA	 Histogram	 Duration of execution in seconds for credential provider plugin
plugin_name
	 None

kubelet_kubelet_credential_provider_plugin_errors	 ALPHA	 Counter	 Number of errors from credential provider plugin
plugin_name
	 None

kubelet_lifecycle_handler_http_fallbacks_total	 ALPHA	 Counter	 The number of times lifecycle handlers successfully fell back to http from https.
   None	 None

kubelet_managed_ephemeral_containers	 ALPHA	 Gauge	 Current number of ephemeral containers in pods managed by this kubelet. Ephemeral containers will be ignored if disabled by the EphemeralContainers feature gate, and this number will be 0.
   None	 None

kubelet_node_name	 ALPHA	 Gauge	 The node's name. The count is always 1.
node

   None

kubelet_pleg_discard_events	 ALPHA	 Counter	 The number of discard events in PLEG.
   None	 None

kubelet_pleg_last_seen_seconds	 ALPHA	 Gauge	 Timestamp in seconds when PLEG was last seen active.
   None	 None

kubelet_pleg_relist_duration_seconds	 ALPHA	 Histogram	 Duration in seconds for relisting pods in PLEG.
   None	 None

kubelet_pleg_relist_interval_seconds	 ALPHA	 Histogram	 Interval in seconds between relisting in PLEG.
   None	 None

kubelet_pod_resources_endpoint_errors_get_allocatable	 ALPHA	 Counter	 Number of requests to the PodResource GetAllocatableResources endpoint which returned error. Broken down by server api version.
server_api_version

   None

kubelet_pod_resources_endpoint_errors_list	 ALPHA	 Counter	 Number of requests to the PodResource List endpoint which returned error. Broken down by server api version.
server_api_version

   None

kubelet_pod_resources_endpoint_requests_get_allocatable	 ALPHA	 Counter	 Number of requests to the PodResource GetAllocatableResources endpoint. Broken down by server api version.
server_api_version

   None

kubelet_pod_resources_endpoint_requests_list	 ALPHA	 Counter	 Number of requests to the PodResource List endpoint. Broken down by server api version.
server_api_version

   None

kubelet_pod_resources_endpoint_requests_total	 ALPHA	 Counter	 Cumulative number of requests to the PodResource endpoint. Broken down by server api version.
server_api_version

   None

kubelet_pod_start_duration_seconds	 ALPHA	 Histogram	 Duration in seconds from kubelet seeing a pod for the first time to the pod starting to run

kubelet_pod_start_sli_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to start a pod, excluding time to pull images and run init containers, measured from pod creation timestamp to when all its containers are reported as started and observed via watch
   None	 None

kubelet_pod_status_sync_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to sync a pod status update. Measures time from detection of a change to pod status until the API is successfully updated for that pod, even if multiple intevening changes to pod status occur.
   None	 None

kubelet_pod_worker_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to sync a single pod. Broken down by operation type: create, update, or sync
operation_type

   None

kubelet_pod_worker_start_duration_seconds	 ALPHA	 Histogram	 Duration in seconds from kubelet seeing a pod to starting a worker.
   None	 None

kubelet_preemptions	 ALPHA	 Counter	 Cumulative number of pod preemptions by preemption resource
preemption_signal

   None

kubelet_run_podsandbox_duration_seconds	 ALPHA	 Histogram	 Duration in seconds of the run_podsandbox operations. Broken down by RuntimeClass.Handler.
runtime_handler

   None

kubelet_run_podsandbox_errors_total	 ALPHA	 Counter	 Cumulative number of the run_podsandbox operation errors by RuntimeClass.Handler.
runtime_handler

   None

kubelet_running_containers	 ALPHA	 Gauge	 Number of containers currently running
container_state

   None

kubelet_running_pods	 ALPHA	 Gauge	 Number of pods that have a running pod sandbox
   None	 None

kubelet_runtime_operations_duration_seconds	 ALPHA	 Histogram	 Duration in seconds of runtime operations. Broken down by operation type.
operation_type

   None

kubelet_runtime_operations_errors_total	 ALPHA	 Counter	 Cumulative number of runtime operation errors by operation type.
operation_type

   None

kubelet_runtime_operations_total	 ALPHA	 Counter	 Cumulative number of runtime operations by operation type.
operation_type

   None

kubelet_server_expiration_renew_errors	 ALPHA	 Counter	 Counter of certificate renewal errors.
   None	 None

kubelet_started_containers_errors_total	 ALPHA	 Counter	 Cumulative number of errors when starting containers
code

container_type

   None

kubelet_started_containers_total	 ALPHA	 Counter	 Cumulative number of containers started
container_type

   None

kubelet_started_host_process_containers_errors_total	 ALPHA	 Counter	 Cumulative number of errors when starting hostprocess containers. This metric will only be collected on Windows and requires WindowsHostProcessContainers feature gate to be enabled.
code

container_type

   None

kubelet_started_host_process_containers_total	 ALPHA	 Counter	 Cumulative number of hostprocess containers started. This metric will only be collected on Windows and requires WindowsHostProcessContainers feature gate to be enabled.
container_type

   None

kubelet_started_pods_errors_total	 ALPHA	 Counter	 Cumulative number of errors when starting pods
   None	 None

kubelet_started_pods_total	 ALPHA	 Counter	 Cumulative number of pods started
   None	 None

kubelet_volume_metric_collection_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to calculate volume stats
metric_source

   None

kubelet_volume_stats_available_bytes	 ALPHA	 Custom	 Number of available bytes in the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_capacity_bytes	 ALPHA	 Custom	 Capacity in bytes of the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_health_status_abnormal	 ALPHA	 Custom	 Abnormal volume health status. The count is either 1 or 0. 1 indicates the volume is unhealthy, 0 indicates volume is healthy
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_inodes	 ALPHA	 Custom	 Maximum number of inodes in the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_inodes_free	 ALPHA	 Custom	 Number of free inodes in the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_inodes_used	 ALPHA	 Custom	 Number of used inodes in the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_used_bytes	 ALPHA	 Custom	 Number of used bytes in the volume
namespace

persistentvolumeclaim

   None

kubeproxy_network_programming_duration_seconds	 ALPHA	 Histogram	 In Cluster Network Programming Latency in seconds
   None	 None

kubeproxy_sync_proxy_rules_duration_seconds	 ALPHA	 Histogram	 SyncProxyRules latency in seconds
   None	 None

kubeproxy_sync_proxy_rules_endpoint_changes_pending	 ALPHA	 Gauge	 Pending proxy rules Endpoint changes
   None	 None

kubeproxy_sync_proxy_rules_endpoint_changes_total	 ALPHA	 Counter	 Cumulative proxy rules Endpoint changes

kubeproxy_sync_proxy_rules_iptables_partial_restore_failures_total	 ALPHA	 Counter	 Cumulative proxy iptables partial restore failures
   None	 None

kubeproxy_sync_proxy_rules_iptables_restore_failures_total	 ALPHA	 Counter	 Cumulative proxy iptables restore failures
   None	 None

kubeproxy_sync_proxy_rules_iptables_total	 ALPHA	 Gauge	 Number of proxy iptables rules programmed
table

   None

kubeproxy_sync_proxy_rules_last_queued_timestamp_seconds	 ALPHA	 Gauge	 The last time a sync of proxy rules was queued
   None	 None

kubeproxy_sync_proxy_rules_last_timestamp_seconds	 ALPHA	 Gauge	 The last time proxy rules were successfully synced
   None	 None

kubeproxy_sync_proxy_rules_no_local_endpoints_total	 ALPHA	 Gauge	 Number of services with a Local traffic policy and no endpoints
traffic_policy

   None

kubeproxy_sync_proxy_rules_service_changes_pending	 ALPHA	 Gauge	 Pending proxy rules Service changes
   None	 None

kubeproxy_sync_proxy_rules_service_changes_total	 ALPHA	 Counter	 Cumulative proxy rules Service changes
   None	 None

kubernetes_build_info	 ALPHA	 Gauge	 A metric with a constant '1' value labeled by major, minor, git version, git commit, git tree state, build date, Go version, and compiler from which Kubernetes was built, and platform on which it is running.
build_date

compiler

git_commit

git_tree_state

git_version

go_version

major

minor

platform

   None

kubernetes_feature_enabled	 ALPHA	 Gauge	 This metric records the data about the stage and enablement of a k8s feature.
name

stage

   None

kubernetes_healthcheck	 ALPHA	 Gauge	 This metric records the result of a single healthcheck.
name

type

   None

kubernetes_healthchecks_total	 ALPHA	 Counter	 This metric records the results of all healthcheck.
name

status

type

   None

leader_election_master_status	 ALPHA	 Gauge	 Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.
name

   None

node_authorizer_graph_actions_duration_seconds	 ALPHA	 Histogram	 Histogram of duration of graph actions in node authorizer.
operation

   None

node_collector_evictions_number	 ALPHA	 Counter	 Number of Node evictions that happened since current instance of NodeController started, This metric is replaced by node_collector_evictions_total.
zone
	  	 1.24.0
   None

node_collector_unhealthy_nodes_in_zone	 ALPHA	 Gauge	 Gauge measuring number of not Ready Nodes per zones.
zone

   None

node_collector_zone_health	 ALPHA	 Gauge	 Gauge measuring percentage of healthy nodes per zone.
zone

   None

node_collector_zone_size	 ALPHA	 Gauge	 Gauge measuring number of registered Nodes per zones.
zone

   None

node_cpu_usage_seconds_total	 ALPHA	 Custom	 Cumulative cpu time consumed by the node in core-seconds
   None	 None

node_ipam_controller_cidrset_allocation_tries_per_request	 ALPHA	 Histogram	 Number of endpoints added on each Service sync
clusterCIDR

   None

node_ipam_controller_cidrset_cidrs_allocations_total	 ALPHA	 Counter	 Counter measuring total number of CIDR allocations.
clusterCIDR

   None

node_ipam_controller_cidrset_cidrs_releases_total	 ALPHA	 Counter	 Counter measuring total number of CIDR releases.
clusterCIDR

   None

node_ipam_controller_cidrset_usage_cidrs	 ALPHA	 Gauge	 Gauge measuring percentage of allocated CIDRs.
clusterCIDR

   None

node_ipam_controller_multicidrset_allocation_tries_per_request	 ALPHA	 Histogram	 Histogram measuring CIDR allocation tries per request.
clusterCIDR

   None

node_ipam_controller_multicidrset_cidrs_allocations_total	 ALPHA	 Counter	 Counter measuring total number of CIDR allocations.
clusterCIDR

   None

node_ipam_controller_multicidrset_cidrs_releases_total	 ALPHA	 Counter	 Counter measuring total number of CIDR releases.
clusterCIDR

   None

node_ipam_controller_multicidrset_usage_cidrs	 ALPHA	 Gauge	 Gauge measuring percentage of allocated CIDRs.
clusterCIDR

   None

node_memory_working_set_bytes	 ALPHA	 Custom	 Current working set of the node in bytes
   None	 None

number_of_l4_ilbs	 ALPHA	 Gauge	 Number of L4 ILBs
feature

   None

plugin_manager_total_plugins	 ALPHA	 Custom	 Number of plugins in Plugin Manager
socket_path

state

   None

pod_cpu_usage_seconds_total	 ALPHA	 Custom	 Cumulative cpu time consumed by the pod in core-seconds
pod

namespace

pod_gc_collector_force_delete_pod_errors_total	 ALPHA	 Counter	 Number of errors encountered when forcefully deleting the pods since the Pod GC Controller started.

pod_gc_collector_force_delete_pods_total	 ALPHA	 Counter	 Number of pods that are being forcefully deleted since the Pod GC Controller started.
   None

pod_memory_working_set_bytes	 ALPHA	 Custom	 Current working set of the pod in bytes
pod

namespace

   None

pod_security_errors_total	 ALPHA	 Counter	 Number of errors preventing normal evaluation. Non-fatal errors may result in the latest restricted profile being used for evaluation.
fatal

request_operation

resource

subresource

   None

pod_security_evaluations_total	 ALPHA	 Counter	 Number of policy evaluations that occurred, not counting ignored or exempt requests.
decision

mode

policy_level

policy_version

request_operation

resource

subresource

   None

pod_security_exemptions_total	 ALPHA	 Counter	 Number of exempt requests, not counting ignored or out of scope requests.
request_operation

resource

subresource

   None

prober_probe_duration_seconds	 ALPHA	 Histogram	 Duration in seconds for a probe response.
container

namespace

pod

probe_type

   None

prober_probe_total	 ALPHA	 Counter	 Cumulative number of a liveness, readiness or startup probe for a container by result.
container

namespace

pod

pod_uid

probe_type

result

   None

pv_collector_bound_pv_count	 ALPHA	 Custom	 Gauge measuring number of persistent volume currently bound
storage_class

   None

pv_collector_bound_pvc_count	 ALPHA	 Custom	 Gauge measuring number of persistent volume claim currently bound
namespace

   None

pv_collector_total_pv_count	 ALPHA	 Custom	 Gauge measuring total number of persistent volumes
plugin_name

volume_mode

   None

pv_collector_unbound_pv_count	 ALPHA	 Custom	 Gauge measuring number of persistent volume currently unbound
storage_class

   None

pv_collector_unbound_pvc_count	 ALPHA	 Custom	 Gauge measuring number of persistent volume claim currently unbound
namespace

   None

replicaset_controller_sorting_deletion_age_ratio	 ALPHA	 Histogram	 The ratio of chosen deleted pod's ages to the current youngest pod's age (at the time). Should be <2.The intent of this metric is to measure the rough efficacy of the LogarithmicScaleDown feature gate's effect onthe sorting (and deletion) of pods when a replicaset scales down. This only considers Ready pods when calculating and reporting.
   None	 None

rest_client_exec_plugin_call_total	 ALPHA	 Counter	 Number of calls to an exec plugin, partitioned by the type of event encountered (no_error, plugin_execution_error, plugin_not_found_error, client_internal_error) and an optional exit code. The exit code will be set to 0 if and only if the plugin call was successful.
call_status

code

   None

rest_client_exec_plugin_certificate_rotation_age	 ALPHA	 Histogram	 Histogram of the number of seconds the last auth exec plugin client certificate lived before being rotated. If auth exec plugin client certificates are unused, histogram will contain no data.
   None	 None

rest_client_exec_plugin_ttl_seconds	 ALPHA	 Gauge	 Gauge of the shortest TTL (time-to-live) of the client certificate(s) managed by the auth exec plugin. The value is in seconds until certificate expiry (negative if already expired). If auth exec plugins are unused or manage no TLS certificates, the value will be +INF.
   None	 None

rest_client_rate_limiter_duration_seconds	 ALPHA	 Histogram	 Client side rate limiter latency in seconds. Broken down by verb, and host.
host

verb

   None

rest_client_request_duration_seconds	 ALPHA	 Histogram	 Request latency in seconds. Broken down by verb, and host.
host

verb

   None

rest_client_request_size_bytes	 ALPHA	 Histogram	 Request size in bytes. Broken down by verb and host.
host

verb

   None

rest_client_requests_total	 ALPHA	 Counter	 Number of HTTP requests, partitioned by status code, method, and host.
code

host

method

   None

rest_client_response_size_bytes	 ALPHA	 Histogram	 Response size in bytes. Broken down by verb and host.
host

verb

   None

retroactive_storageclass_errors_total	 ALPHA	 Counter	 Total number of failed retroactive StorageClass assignments to persistent volume claim
   None	 None

retroactive_storageclass_total	 ALPHA	 Counter	 Total number of retroactive StorageClass assignments to persistent volume claim
   None	 None

root_ca_cert_publisher_sync_duration_seconds	 ALPHA	 Histogram	 Number of namespace syncs happened in root ca cert publisher.
code

   None

root_ca_cert_publisher_sync_total	 ALPHA	 Counter	 Number of namespace syncs happened in root ca cert publisher.
code

   None

running_managed_controllers	 ALPHA	 Gauge	 Indicates where instances of a controller are currently running
manager

name

   None

scheduler_e2e_scheduling_duration_seconds	 ALPHA	 Histogram	 E2e scheduling latency in seconds (scheduling algorithm + binding). This metric is replaced by scheduling_attempt_duration_seconds.
profile

result
	  	 1.23.0
   None

scheduler_goroutines	 ALPHA	 Gauge	 Number of running goroutines split by the work they do such as binding.
operation

   None

scheduler_permit_wait_duration_seconds	 ALPHA	 Histogram	 Duration of waiting on permit.
result

   None

scheduler_plugin_execution_duration_seconds	 ALPHA	 Histogram	 Duration for running a plugin at a specific extension point.
extension_point

plugin

status

   None

scheduler_scheduler_cache_size	 ALPHA	 Gauge	 Number of nodes, pods, and assumed (bound) pods in the scheduler cache.
type

   None

scheduler_scheduler_goroutines	 ALPHA	 Gauge	 Number of running goroutines split by the work they do such as binding. This metric is replaced by the "goroutines" metric.
work
	  	 1.26.0
   None

scheduler_scheduling_algorithm_duration_seconds	 ALPHA	 Histogram	 Scheduling algorithm latency in seconds
   None	 None

scheduler_unschedulable_pods	 ALPHA	 Gauge	 The number of unschedulable pods broken down by plugin name. A pod will increment the gauge for all plugins that caused it to not schedule and so this metric have meaning only when broken down by plugin.
plugin

profile

   None

scheduler_volume_binder_cache_requests_total	 ALPHA	 Counter	 Total number for request volume binding cache
operation

   None

scheduler_volume_scheduling_stage_error_total	 ALPHA	 Counter	 Volume scheduling stage error count
operation

   None

scrape_error	 ALPHA	 Custom	 1 if there was an error while getting container metrics, 0 otherwise
   None	 None

service_controller_nodesync_latency_seconds	 ALPHA	 Histogram	 A metric measuring the latency for nodesync which updates loadbalancer hosts on cluster node updates.
   None	 None

service_controller_update_loadbalancer_host_latency_seconds	 ALPHA	 Histogram	 A metric measuring the latency for updating each load balancer hosts.
   None	 None

serviceaccount_legacy_tokens_total	 ALPHA	 Counter	 Cumulative legacy service account tokens used
   None	 None

serviceaccount_stale_tokens_total	 ALPHA	 Counter	 Cumulative stale projected service account tokens used
   None	 None

serviceaccount_valid_tokens_total	 ALPHA	 Counter	 Cumulative valid projected service account tokens used
   None	 None

storage_count_attachable_volumes_in_use	 ALPHA	 Custom	 Measure number of volumes in use
node

volume_plugin

   None

storage_operation_duration_seconds	 ALPHA	 Histogram	 Storage operation duration
migrated

operation_name

status

volume_plugin

   None

ttl_after_finished_controller_job_deletion_duration_seconds	 ALPHA	 Histogram	 The time it took to delete the job since it became eligible for deletion
   None	 None

volume_manager_selinux_container_errors_total	 ALPHA	 Gauge	 Number of errors when kubelet cannot compute SELinux context for a container. Kubelet can't start such a Pod then and it will retry, therefore value of this metric may not represent the actual nr. of containers.
   None	 None

volume_manager_selinux_container_warnings_total	 ALPHA	 Gauge	 Number of errors when kubelet cannot compute SELinux context for a container that are ignored. They will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
   None	 None

volume_manager_selinux_pod_context_mismatch_errors_total	 ALPHA	 Gauge	 Number of errors when a Pod defines different SELinux contexts for its containers that use the same volume. Kubelet can't start such a Pod then and it will retry, therefore value of this metric may not represent the actual nr. of Pods.
   None	 None

volume_manager_selinux_pod_context_mismatch_warnings_total	 ALPHA	 Gauge	 Number of errors when a Pod defines different SELinux contexts for its containers that use the same volume. They are not errors yet, but they will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
   None	 None

volume_manager_selinux_volume_context_mismatch_errors_total	 ALPHA	 Gauge	 Number of errors when a Pod uses a volume that is already mounted with a different SELinux context than the Pod needs. Kubelet can't start such a Pod then and it will retry, therefore value of this metric may not represent the actual nr. of Pods.
   None	 None

volume_manager_selinux_volume_context_mismatch_warnings_total	 ALPHA	 Gauge	 Number of errors when a Pod uses a volume that is already mounted with a different SELinux context than the Pod needs. They are not errors yet, but they will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
   None	 None

volume_manager_selinux_volumes_admitted_total	 ALPHA	 Gauge	 Number of volumes whose SELinux context was fine and will be mounted with mount -o context option.
   None	 None

volume_manager_total_volumes	 ALPHA	 Custom	 Number of volumes in Volume Manager
plugin_name

state

   None

volume_operation_total_errors	 ALPHA	 Counter	 Total volume operation errors
operation_name

plugin_name

   None

volume_operation_total_seconds	 ALPHA	 Histogram	 Storage operation end to end duration in seconds
operation_name

plugin_name

   None

watch_cache_capacity	 ALPHA	 Gauge	 Total capacity of watch cache broken by resource type.
resource

   None

watch_cache_capacity_decrease_total	 ALPHA	 Counter	 Total number of watch cache capacity decrease events broken by resource type.
resource

   None

watch_cache_capacity_increase_total	 ALPHA	 Counter	 Total number of watch cache capacity increase events broken by resource type.
resource

   None

workqueue_adds_total	 ALPHA	 Counter	 Total number of adds handled by workqueue
name

   None

workqueue_depth	 ALPHA	 Gauge	 Current depth of workqueue
name

   None

workqueue_longest_running_processor_seconds	 ALPHA	 Gauge	 How many seconds has the longest running processor for workqueue been running.
name

   None

workqueue_queue_duration_seconds	 ALPHA	 Histogram	 How long in seconds an item stays in workqueue before being requested.
name

   None

workqueue_retries_total	 ALPHA	 Counter	 Total number of retries handled by workqueue
name

   None

workqueue_unfinished_work_seconds	 ALPHA	 Gauge	 How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
name

   None

workqueue_work_duration_seconds	 ALPHA	 Histogram	 How long in seconds processing an item from workqueue takes.
name

   None


outline: none; padding: .5em 0 .5em 0; } <ins> /* CSS for 'figure' full-screen display */ /* Define styles for full-screen overlay */ .figure-fullscreen-overlay { position: fixed; inset: 0; z-index: 9999; background-color: rgba(255, 255, 255, 0.95); /* White background with some transparency */ display: flex; justify-content: center; align-items: center; padding: calc(5% + 20px); box-sizing: border-box; } /* CSS class to scale the image when zoomed */ .figure-zoomed { transform: scale(1.2); } /* Define styles for full-screen image */ .figure-fullscreen-img { max-width: 100%; max-height: 100%; object-fit: contain; /* Maintain aspect ratio and fit within the container */ } /* Define styles for close button */ .figure-close-button { position: absolute; top: 1%; right: 2%; cursor: pointer; font-size: calc(5vw + 10px); color: #333; } </ins>  No newline at end of file


This document outlines the various components you need to have for a complete and working Kubernetes cluster. <del> {{< figure src="/images/docs/components-of-kubernetes.svg" alt="Components of Kubernetes" caption="The components of a Kubernetes cluster" class="diagram-large" >}} </del> <ins> {{< figure src="/images/docs/components-of-kubernetes.svg" alt="Components of Kubernetes" caption="The components of a Kubernetes cluster" class="diagram-large" clicktozoom="true" >}} </ins> <!-- body --> ## Control Plane Components


<script src="https://cdnjs.cloudflare.com/ajax/libs/iframe-resizer/4.3.9/iframeResizer.min.js" integrity="sha384-hHTwgxzjpO1G1NI0wMHWQYUxnGtpWyDjVSZrFnDrlWa5OL+DFY57qnDWw/5WSJOl" crossorigin="anonymous"></script> {{- end -}} <ins> <!-- Enable zoom-on-click for figures that opt in to this --> {{- if .HasShortcode "figure" -}} <script defer src="{{ "js/zoom.js" | relURL }}"></script> {{- end -}} </ins> {{- if eq (lower .Params.cid) "community" -}} {{- if eq .Params.community_styles_migrated true -}} <link href="/css/community.css" rel="stylesheet"><!-- legacy styles -->


 {{ $src := (.Page.Resources.GetMatch (printf "**%s*" (.Get "src"))) }} {{ $clickToZoom := .Get "clicktozoom" }}
 {{- if .Get "link" -}}  {{- end }}   {{- if .Get "link" }}{{ end -}} {{- if or (or (.Get "title") (.Get "caption")) (.Get "attr") -}}
 {{ with (.Get "title") -}}
{{ . }}
 {{- end -}} {{- if or (.Get "caption") (.Get "attr") -}}
 {{- .Get "caption" | markdownify -}} {{- with .Get "attrlink" }}  {{- end -}} {{- .Get "attr" | markdownify -}} {{- if .Get "attrlink" }}{{ end }}
 {{- end }}
 {{- end }}
   No newline at end of file


 // The page and script is loaded successfully $(document).ready(function() { // Function to handle hover over
 elements function handleFigureHover() { // Only change cursor to zoom-in if figure has 'clickable-zoom' class if ($(this).hasClass('clickable-zoom') && !$(this).hasClass('figure-fullscreen-content')) { $(this).css('cursor', 'zoom-in'); } } // Attach hover event to
 elements with 'clickable-zoom' class $('figure.clickable-zoom').hover(handleFigureHover, function() { // Mouse out - revert cursor back to default $(this).css('cursor', 'default'); }); // Attach click event to
 elements with 'clickable-zoom' class $('figure.clickable-zoom').click(function() { var $figure = $(this); // Check if the figure has 'clickable-zoom' class if ($figure.hasClass('clickable-zoom')) { var $img = $figure.find('img'); // Get the  element within the clicked
 // Toggle 'figure-zoomed' class to scale the image $img.toggleClass('figure-zoomed'); // Create a full-screen overlay var $fullscreenOverlay = $('

'); // Clone the  element to display in full-screen var $fullscreenImg = $img.clone(); $fullscreenImg.addClass('figure-fullscreen-img'); // Append the full-screen image to the overlay $fullscreenOverlay.append($fullscreenImg); // Create a close button for the full-screen overlay var $closeButton = $('×'); $closeButton.click(function() { // Remove the full-screen overlay when close button is clicked $fullscreenOverlay.remove(); $('body').css('overflow', 'auto'); // Restore scrolling to the underlying page // Remove 'figure-zoomed' class to reset image scale $img.removeClass('figure-zoomed'); }); $fullscreenOverlay.append($closeButton); // Append the overlay to the body $('body').append($fullscreenOverlay); // Disable scrolling on the underlying page $('body').css('overflow', 'hidden'); // Close full-screen figure when clicking outside of it (on the overlay) $fullscreenOverlay.click(function(event) { if (event.target === this) { // Clicked on the overlay area (outside the full-screen image) $fullscreenOverlay.remove(); $('body').css('overflow', 'auto'); // Restore scrolling to the underlying page // Remove 'figure-zoomed' class to reset image scale $img.removeClass('figure-zoomed'); } }); } }); });


<!-- overview --> <del> Kubernetes runs your workload by placing containers into Pods to run on _Nodes_. </del> <ins> Kubernetes runs your {{< glossary_tooltip text="workload" term_id="workload" >}} by placing containers into Pods to run on _Nodes_. </ins> A node may be a virtual or physical machine, depending on the cluster. Each node is managed by the {{< glossary_tooltip text="control plane" term_id="control-plane" >}}


section [Graceful Node Shutdown](#graceful-node-shutdown) for more details. When a node is shutdown but not detected by kubelet's Node Shutdown Manager, the pods <del> that are part of a StatefulSet will be stuck in terminating status on </del> <ins> that are part of a {{< glossary_tooltip text="StatefulSet" term_id="statefulset" >}} will be stuck in terminating status on </ins> the shutdown node and cannot move to a new running node. This is because kubelet on the shutdown node is not available to delete the pods so the StatefulSet cannot create a new pod with the same name. If there are volumes used by the pods, the


To mitigate the above situation, a  user can manually add the taint `node.kubernetes.io/out-of-service` with either `NoExecute` or `NoSchedule` effect to a Node marking it out-of-service. If the `NodeOutOfServiceVolumeDetach`[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) <del> is enabled on `kube-controller-manager`, and a Node is marked out-of-service with this taint, the </del> <ins> is enabled on {{< glossary_tooltip text="kube-controller-manager" term_id="kube-controller-manager" >}}, and a Node is marked out-of-service with this taint, the </ins> pods on the node will be forcefully deleted if there are no matching tolerations on it and volume detach operations for the pods terminating on the node will happen immediately. This allows the Pods on the out-of-service node to recover quickly on a different node.


## {{% heading "whatsnext" %}} <del> * Learn about the [components](/docs/concepts/overview/components/#node-components) that make up a node. * Read the [API definition for Node](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#node-v1-core). * Read the [Node](https://git.k8s.io/design-proposals-archive/architecture/architecture.md#the-kubernetes-node) section of the architecture design document. * Read about [taints and tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/). </del> <ins> Learn more about the following: * [Components](/docs/concepts/overview/components/#node-components) that make up a node. * [API definition for Node](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#node-v1-core). * [Node](https://git.k8s.io/design-proposals-archive/architecture/architecture.md#the-kubernetes-node) section of the architecture design document. * [Taints and Tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/). * [Node Resource Managers](/docs/concepts/policy/node-resource-managers/). * [Resource Management for Windows nodes](/docs/concepts/configuration/windows-resource-management/). </ins>


- operation --- <del> एग्रीगेशन लेयर आपको अपने क्लस्टर में अतिरिक्त कुबेरनेट्स-शैली API स्थापित करने देता है।``` </del> <ins> एग्रीगेशन लेयर आपको अपने क्लस्टर में अतिरिक्त कुबेरनेट्स-शैली API स्थापित करने देता है। </ins> <!--more--> <del> जब आपने {{< glossary_tooltip text="कुबेरनेट्स API सर्वर" term_id="kube-apiserver" >}} को [अतिरिक्त API का समर्थन](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) करने के लिए कॉन्फ़िगर किया हो, आप कुबेरनेट्स एपीआई में URL पथ का "दावा" करने के लिए `APIService` ऑब्जेक्ट जोड़ सकते हैं। </del> <ins> जब आपने {{< glossary_tooltip text="कुबेरनेट्स API सर्वर" term_id="kube-apiserver" >}} को [अतिरिक्त API का समर्थन](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) करने के लिए कॉन्फ़िगर किया हो, आप कुबेरनेट्स API में URL पाथ का "दावा" करने के लिए `APIService` ऑब्जेक्ट जोड़ सकते हैं। </ins>


<ins> --- title: 动态资源分配 content_type: concept weight: 65 --- <!-- reviewers: - klueska - pohly title: Dynamic Resource Allocation content_type: concept weight: 65 --> <!-- overview --> {{< feature-state for_k8s_version="v1.26" state="alpha" >}} <!-- Dynamic resource allocation is a new API for requesting and sharing resources between pods and containers inside a pod. It is a generalization of the persistent volumes API for generic resources. Third-party resource drivers are responsible for tracking and allocating resources. Different kinds of resources support arbitrary parameters for defining requirements and initialization. --> 动态资源分配是一个用于在 Pod 之间和 Pod 内部容器之间请求和共享资源的新 API。 它是对为通用资源所提供的持久卷 API 的泛化。第三方资源驱动程序负责跟踪和分配资源。 不同类型的资源支持用任意参数进行定义和初始化。 ## {{% heading "prerequisites" %}} <!-- Kubernetes v{{< skew currentVersion >}} includes cluster-level API support for dynamic resource allocation, but it [needs to be enabled](#enabling-dynamic-resource-allocation) explicitly.  You also must install a resource driver for specific resources that are meant to be managed using this API.  If you are not running Kubernetes v{{< skew currentVersion>}}, check the documentation for that version of Kubernetes. --> Kubernetes v{{< skew currentVersion >}} 包含用于动态资源分配的集群级 API 支持， 但它需要被[显式启用](#enabling-dynamic-resource-allocation)。 你还必须为此 API 要管理的特定资源安装资源驱动程序。 如果你未运行 Kubernetes v{{< skew currentVersion>}}， 请查看对应版本的 Kubernetes 文档。 <!-- body --> ## API {#api} <!-- The new `resource.k8s.io/v1alpha1` {{< glossary_tooltip text="API group" term_id="api-group" >}} provides four new types: --> 新的 `resource.k8s.io/v1alpha1` {{< glossary_tooltip text="API 组" term_id="api-group" >}}提供四种新类型： <!-- ResourceClass : Defines which resource driver handles a certain kind of resource and provides common parameters for it. ResourceClasses are created by a cluster administrator when installing a resource driver. ResourceClaim : Defines a particular resource instances that is required by a workload. Created by a user (lifecycle managed manually, can be shared between different Pods) or for individual Pods by the control plane based on a ResourceClaimTemplate (automatic lifecycle, typically used by just one Pod). ResourceClaimTemplate : Defines the spec and some meta data for creating ResourceClaims. Created by a user when deploying a workload. PodScheduling : Used internally by the control plane and resource drivers to coordinate pod scheduling when ResourceClaims need to be allocated for a Pod. --> ResourceClass : 定义由哪个资源驱动程序处理某种资源，并为其提供通用参数。 集群管理员在安装资源驱动程序时创建 ResourceClass。 ResourceClaim : 定义工作负载所需的特定资源实例。 由用户创建（手动管理生命周期，可以在不同的 Pod 之间共享）， 或者由控制平面基于 ResourceClaimTemplate 为特定 Pod 创建 （自动管理生命周期，通常仅由一个 Pod 使用）。 ResourceClaimTemplate : 定义用于创建 ResourceClaim 的 spec 和一些元数据。 部署工作负载时由用户创建。 PodScheduling : 供控制平面和资源驱动程序内部使用， 在需要为 Pod 分配 ResourceClaim 时协调 Pod 调度。 <!-- Parameters for ResourceClass and ResourceClaim are stored in separate objects, typically using the type defined by a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CRD" >}} that was created when installing a resource driver. --> ResourceClass 和 ResourceClaim 的参数存储在单独的对象中， 通常使用安装资源驱动程序时创建的 {{< glossary_tooltip term_id="CustomResourceDefinition" text="CRD" >}} 所定义的类型。 <!-- The `core/v1` `PodSpec` defines ResourceClaims that are needed for a Pod in a new `resourceClaims` field. Entries in that list reference either a ResourceClaim or a ResourceClaimTemplate. When referencing a ResourceClaim, all Pods using this PodSpec (for example, inside a Deployment or StatefulSet) share the same ResourceClaim instance. When referencing a ResourceClaimTemplate, each Pod gets its own instance. --> `core/v1` 的 `PodSpec` 在新的 `resourceClaims` 字段中定义 Pod 所需的 ResourceClaim。 该列表中的条目引用 ResourceClaim 或 ResourceClaimTemplate。 当引用 ResourceClaim 时，使用此 PodSpec 的所有 Pod （例如 Deployment 或 StatefulSet 中的 Pod）共享相同的 ResourceClaim 实例。 引用 ResourceClaimTemplate 时，每个 Pod 都有自己的实例。 <!-- The `resources.claims` list for container resources defines whether a container gets access to these resource instances, which makes it possible to share resources between one or more containers. Here is an example for a fictional resource driver. Two ResourceClaim objects will get created for this Pod and each container gets access to one of them. --> 容器资源的 `resources.claims` 列表定义容器可以访问的资源实例， 从而可以实现在一个或多个容器之间共享资源。 下面是一个虚构的资源驱动程序的示例。 该示例将为此 Pod 创建两个 ResourceClaim 对象，每个容器都可以访问其中一个。 ```yaml apiVersion: resource.k8s.io/v1alpha1 kind: ResourceClass name: resource.example.com driverName: resource-driver.example.com --- apiVersion: cats.resource.example.com/v1 kind: ClaimParameters name: large-black-cat-claim-parameters spec: color: black size: large --- apiVersion: resource.k8s.io/v1alpha1 kind: ResourceClaimTemplate metadata: name: large-black-cat-claim-template spec: spec: resourceClassName: resource.example.com parametersRef: apiGroup: cats.resource.example.com kind: ClaimParameters name: large-black-cat-claim-parameters –-- apiVersion: v1 kind: Pod metadata: name: pod-with-cats spec: containers: - name: container0 image: ubuntu:20.04 command: ["sleep", "9999"] resources: claims: - name: cat-0 - name: container1 image: ubuntu:20.04 command: ["sleep", "9999"] resources: claims: - name: cat-1 resourceClaims: - name: cat-0 source: resourceClaimTemplateName: large-black-cat-claim-template - name: cat-1 source: resourceClaimTemplateName: large-black-cat-claim-template ``` <!-- ## Scheduling --> ## 调度  {#scheduling} <!-- In contrast to native resources (CPU, RAM) and extended resources (managed by a device plugin, advertised by kubelet), the scheduler has no knowledge of what dynamic resources are available in a cluster or how they could be split up to satisfy the requirements of a specific ResourceClaim. Resource drivers are responsible for that. They mark ResourceClaims as "allocated" once resources for it are reserved. This also then tells the scheduler where in the cluster a ResourceClaim is available. --> 与原生资源（CPU、RAM）和扩展资源（由设备插件管理，并由 kubelet 公布）不同， 调度器不知道集群中有哪些动态资源， 也不知道如何将它们拆分以满足特定 ResourceClaim 的要求。 资源驱动程序负责这些任务。 资源驱动程序在为 ResourceClaim 保留资源后将其标记为“已分配（Allocated）”。 然后告诉调度器集群中可用的 ResourceClaim 的位置。 <!-- ResourceClaims can get allocated as soon as they are created ("immediate allocation"), without considering which Pods will use them. The default is to delay allocation until a Pod gets scheduled which needs the ResourceClaim (i.e. "wait for first consumer"). --> ResourceClaim 可以在创建时就进行分配（“立即分配”），不用考虑哪些 Pod 将使用它。 默认情况下采用延迟分配，直到需要 ResourceClaim 的 Pod 被调度时 （即“等待第一个消费者”）再进行分配。 <!-- In that mode, the scheduler checks all ResourceClaims needed by a Pod and creates a PodScheduling object where it informs the resource drivers responsible for those ResourceClaims about nodes that the scheduler considers suitable for the Pod. The resource drivers respond by excluding nodes that don't have enough of the driver's resources left. Once the scheduler has that information, it selects one node and stores that choice in the PodScheduling object. The resource drivers then allocate their ResourceClaims so that the resources will be available on that node. Once that is complete, the Pod gets scheduled. --> 在这种模式下，调度器检查 Pod 所需的所有 ResourceClaim，并创建一个 PodScheduling 对象， 通知负责这些 ResourceClaim 的资源驱动程序，告知它们调度器认为适合该 Pod 的节点。 资源驱动程序通过排除没有足够剩余资源的节点来响应调度器。 一旦调度器有了这些信息，它就会选择一个节点，并将该选择存储在 PodScheduling 对象中。 然后，资源驱动程序为分配其 ResourceClaim，以便资源可用于该节点。 完成后，Pod 就会被调度。 <!-- As part of this process, ResourceClaims also get reserved for the Pod. Currently ResourceClaims can either be used exclusively by a single Pod or an unlimited number of Pods. --> 作为此过程的一部分，ResourceClaim 会为 Pod 保留。 目前，ResourceClaim 可以由单个 Pod 独占使用或不限数量的多个 Pod 使用。 <!-- One key feature is that Pods do not get scheduled to a node unless all of their resources are allocated and reserved. This avoids the scenario where a Pod gets scheduled onto one node and then cannot run there, which is bad because such a pending Pod also blocks all other resources like RAM or CPU that were set aside for it. --> 除非 Pod 的所有资源都已分配和保留，否则 Pod 不会被调度到节点，这是一个重要特性。 这避免了 Pod 被调度到一个节点但无法在那里运行的情况， 这种情况很糟糕，因为被挂起 Pod 也会阻塞为其保留的其他资源，如 RAM 或 CPU。 <!-- ## Limitations --> ## 限制 {#limitations} <!-- The scheduler plugin must be involved in scheduling Pods which use ResourceClaims. Bypassing the scheduler by setting the `nodeName` field leads to Pods that the kubelet refuses to start because the ResourceClaims are not reserved or not even allocated. It may be possible to [remove this limitation](https://github.com/kubernetes/kubernetes/issues/114005) in the future. --> 调度器插件必须参与调度那些使用 ResourceClaim 的 Pod。 通过设置 `nodeName` 字段绕过调度器会导致 kubelet 拒绝启动 Pod， 因为 ResourceClaim 没有被保留或甚至根本没有被分配。 未来可能[去除该限制](https://github.com/kubernetes/kubernetes/issues/114005)。 <!-- ## Enabling dynamic resource allocation --> ## 启用动态资源分配 {#enabling-dynamic-resource-allocation} <!-- Dynamic resource allocation is an *alpha feature* and only enabled when the `DynamicResourceAllocation` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) and the `resource.k8s.io/v1alpha1` {{< glossary_tooltip text="API group" term_id="api-group" >}} are enabled. For details on that, see the `--feature-gates` and `--runtime-config` [kube-apiserver parameters](/docs/reference/command-line-tools-reference/kube-apiserver/). kube-scheduler, kube-controller-manager and kubelet also need the feature gate. --> 动态资源分配是一个 **alpha 特性**，只有在启用 `DynamicResourceAllocation` [特性门控](/zh-cn/docs/reference/command-line-tools-reference/feature-gates/) 和 `resource.k8s.io/v1alpha1` {{< glossary_tooltip text="API 组" term_id="api-group" >}} 时才启用。 有关详细信息，参阅 `--feature-gates` 和 `--runtime-config` [kube-apiserver 参数](/zh-cn/docs/reference/command-line-tools-reference/kube-apiserver/)。 kube-scheduler、kube-controller-manager 和 kubelet 也需要设置该特性门控。 <!-- A quick check whether a Kubernetes cluster supports the feature is to list ResourceClass objects with: --> 快速检查 Kubernetes 集群是否支持该功能的方法是列出 ResourceClass 对象： ```shell kubectl get resourceclasses ``` <!-- If your cluster supports dynamic resource allocation, the response is either a list of ResourceClass objects or: --> 如果你的集群支持动态资源分配，则响应是 ResourceClass 对象列表或： ``` No resources found ``` <!-- If not supported, this error is printed instead: --> 如果不支持，则会输出如下错误： ``` error: the server doesn't have a resource type "resourceclasses" ``` <!-- The default configuration of kube-scheduler enables the "DynamicResources" plugin if and only if the feature gate is enabled. Custom configurations may have to be modified to include it. --> kube-scheduler 的默认配置仅在启用特性门控时才启用 "DynamicResources" 插件。 自定义配置可能需要被修改才能启用它。 <!-- In addition to enabling the feature in the cluster, a resource driver also has to be installed. Please refer to the driver's documentation for details. --> 除了在集群中启用该功能外，还必须安装资源驱动程序。 欲了解详细信息，请参阅驱动程序的文档。 ## {{% heading "whatsnext" %}} <!-- - For more information on the design, see the [Dynamic Resource Allocation KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3063-dynamic-resource-allocation/README.md). --> - 了解更多该设计的信息， 参阅[动态资源分配 KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3063-dynamic-resource-allocation/README.md)。 </ins>  No newline at end of file


<ins> --- title: ファイナライザー(Finalizers) content_type: concept weight: 80 --- <!-- overview --> {{<glossary_definition term_id="finalizer" length="long">}} ファイナライザーを利用すると、対象のリソースを削除する前に特定のクリーンアップを行うように{{<glossary_tooltip text="コントローラー" term_id="controller">}}に警告することで、{{<glossary_tooltip text="ガベージコレクション" term_id="garbage-collection">}}を管理することができます。 大抵の場合ファイナライザーは実行されるコードを指定することはありません。 その代わり、一般的にはアノテーションのように特定のリソースに関するキーのリストになります。 Kubernetesはいくつかのファイナライザーを自動的に追加しますが、自分で追加することもできます。 ## ファイナライザーはどのように動作するか マニフェストファイルを使ってリソースを作るとき、`metadata.finalizers`フィールドの中でファイナライザーを指定することができます。 リソースを削除しようとするとき、削除リクエストを扱うAPIサーバーは`finalizers`フィールドの値を確認し、以下のように扱います。 * 削除を開始した時間をオブジェクトの`metadata.deletionTimestamp`フィールドに設定します。 * `metadata.finalizers`フィールドが空になるまでオブジェクトが削除されるのを阻止します。 * ステータスコード`202`(HTTP "Accepted")を返します。 ファイナライザーを管理しているコントローラーは、オブジェクトの削除がリクエストされたことを示す`metadata.deletionTimestamp`がオブジェクトに設定されたことを検知します。 するとコントローラーはリソースに指定されたファイナライザーの要求を満たそうとします。 ファイナライザーの条件が満たされるたびに、そのコントローラーはリソースの`finalizers`フィールドの対象のキーを削除します。 `finalizers`フィールドが空になったとき、`deletionTimestamp`フィールドが設定されたオブジェクトは自動的に削除されます。管理外のリソース削除を防ぐためにファイナライザーを利用することもできます。 ファイナライザーの一般的な例は`kubernetes.io/pv-protection`で、これは `PersistentVolume`オブジェクトが誤って削除されるのを防ぐためのものです。 `PersistentVolume`オブジェクトをPodが利用中の場合、Kubernetesは`pv-protection`ファイナライザーを追加します。 `PersistentVolume`を削除しようとすると`Terminating`ステータスになりますが、ファイナライザーが存在しているためコントローラーはボリュームを削除することができません。 Podが`PersistentVolume`の利用を停止するとKubernetesは`pv-protection`ファイナライザーを削除し、コントローラーがボリュームを削除します。 ## オーナーリファレンス、ラベル、ファイナライザー {#owners-labels-finalizers} {{<glossary_tooltip text="ラベル" term_id="label">}}のように、 [オーナーリファレンス](/docs/concepts/overview/working-with-objects/owners-dependents/)はKubernetesのオブジェクト間の関係性を説明しますが、利用される目的が異なります。 {{<glossary_tooltip text="コントローラー" term_id="controller">}} がPodのようなオブジェクトを管理するとき、関連するオブジェクトのグループの変更を追跡するためにラベルを利用します。 例えば、{{<glossary_tooltip text="Job" term_id="job">}}がいくつかのPodを作成するとき、JobコントローラーはそれらのPodにラベルを付け、クラスター内の同じラベルを持つPodの変更を追跡します。 Jobコントローラーは、Podを作成したJobを指す*オーナーリファレンス*もそれらのPodに追加します。 Podが実行されているときにJobを削除すると、Kubernetesはオーナーリファレンス(ラベルではない)を使って、クリーンアップする必要のあるPodをクラスター内から探し出します。 また、Kubernetesは削除対象のリソースのオーナーリファレンスを認識して、ファイナライザーを処理します。 状況によっては、ファイナライザーが依存オブジェクトの削除をブロックしてしまい、対象のオーナーオブジェクトが完全に削除されず予想以上に長時間残ってしまうことがあります。 このような状況では、対象のオーナーと依存オブジェクトの、ファイナライザーとオーナーリファレンスを確認して問題を解決する必要があります。 {{<note>}} オブジェクトが削除中の状態で詰まってしまった場合、削除を続行するために手動でファイナライザーを削除することは避けてください。 通常、ファイナライザーは理由があってリソースに追加されているものであるため、強制的に削除してしまうとクラスターで何らかの問題を引き起こすことがあります。 そのファイナライザーの目的を理解しているかつ、別の方法で達成できる場合にのみ行うべきです(例えば、依存オブジェクトを手動で削除するなど)。 {{</note>}} ## {{% heading "whatsnext" %}} * Kubernetesブログの[ファイナライザーを利用した削除の制御](/blog/2021/05/14/using-finalizers-to-control-deletion/)をお読みください。 </ins>


<ins> --- title: ファイナライザー id: finalizer date: 2021-07-07 full_link: /ja/docs/concepts/overview/working-with-objects/finalizers/ short_description: > 削除対象としてマークされたオブジェクトを完全に削除する前に、特定の条件が満たされるまでKubernetesを待機させるための名前空間付きのキーです。 aka: tags: - fundamental - operation --- ファイナライザーは、削除対象としてマークされたリソースを完全に削除する前に、特定の条件が満たされるまでKubernetesを待機させるための名前空間付きのキーです。 ファイナライザーは、削除されたオブジェクトが所有していたリソースをクリーンアップするように{{<glossary_tooltip text="コントローラー" term_id="controller">}}に警告します。 <!--more--> Kubernetesにファイナライザーが指定されたオブジェクトを削除するように指示すると、Kubernetes APIはそのオブジェクトに`.metadata.deletionTimestamp`を追加し削除対象としてマークして、ステータスコード`202`(HTTP "Accepted")を返します。 コントロールプレーンやその他のコンポーネントがファイナライザーによって定義されたアクションを実行している間、対象のオブジェクトは終了中の状態のまま残っています。 それらのアクションが完了したら、そのコントローラーは関係しているファイナライザーを対象のオブジェクトから削除します。 `metadata.finalizers`フィールドが空になったら、Kubernetesは削除が完了したと判断しオブジェクトを削除します。 ファイナライザーはリソースの{{<glossary_tooltip text="ガベージコレクション" term_id="garbage-collection">}}を管理するために使うことができます。 例えば、コントローラーが対象のリソースを削除する前に関連するリソースやインフラをクリーンアップするためにファイナライザーを定義することができます。 </ins>


## Start a Kubelet process configured via the config file {{< note >}} <del> If you use kubeadm to initialize your cluster, use the kubelet-config while creating your cluster with `kubeadmin init`. </del> <ins> If you use kubeadm to initialize your cluster, use the kubelet-config while creating your cluster with `kubeadm init`. </ins> See [configuring kubelet using kubeadm](/docs/setup/production-environment/tools/kubeadm/kubelet-integration/) for details. {{< /note >}}


<ins> --- title: cgroup v2について content_type: concept weight: 50 --- <!-- overview --> Linuxでは、{{< glossary_tooltip text="コントロールグループ" term_id="cgroup" >}}がプロセスに割り当てられるリソースを制限しています。 コンテナ化されたワークロードの、CPU/メモリーの要求と制限を含む[Podとコンテナのリソース管理](/docs/concepts/configuration/manage-resources-containers/)を強制するために、 {{< glossary_tooltip text="kubelet" term_id="kubelet" >}}と基盤となるコンテナランタイムはcgroupをインターフェースとして接続する必要があります。 Linuxではcgroup v1とcgroup v2の2つのバージョンのcgroupがあります。 cgroup v2は新世代の`cgroup` APIです。 <!-- body --> ## cgroup v2とは何か？ {#cgroup-v2} {{< feature-state for_k8s_version="v1.25" state="stable" >}} cgroup v2はLinuxの`cgroup` APIの次のバージョンです。 cgroup v2はリソース管理機能を強化した統合制御システムを提供しています。 以下のように、cgroup v2はcgroup v1からいくつかの点を改善しています。 - 統合された単一階層設計のAPI - より安全なコンテナへのサブツリーの移譲 - [Pressure Stall Information](https://www.kernel.org/doc/html/latest/accounting/psi.html)などの新機能 - 強化されたリソース割り当て管理と複数リソース間の隔離 - 異なるタイプのメモリー割り当ての統一(ネットワークメモリー、カーネルメモリーなど) - ページキャッシュの書き戻しといった、非即時のリソース変更 Kubernetesのいくつかの機能では、強化されたリソース管理と隔離のためにcgroup v2のみを使用しています。 例えば、[MemoryQoS](/blog/2021/11/26/qos-memory-resources/)機能はメモリーQoSを改善し、cgroup v2の基本的な機能に依存しています。 ## cgroup v2を使う {#using-cgroupv2} cgroup v2を使うおすすめの方法は、デフォルトでcgroup v2が有効で使うことができるLinuxディストリビューションを使うことです。 あなたのディストリビューションがcgroup v2を使っているかどうかを確認するためには、[Linux Nodeのcgroupバージョンを特定する](#check-cgroup-version)を参照してください。 ### 必要要件 {#requirements} cgroup v2を使うには以下のような必要要件があります。 * OSディストリビューションでcgroup v2が有効であること * Linuxカーネルバージョンが5.8以上であること * コンテナランタイムがcgroup v2をサポートしていること。例えば、 * [containerd](https://containerd.io/) v1.4以降 * [cri-o](https://cri-o.io/) v1.20以降 * kubeletとコンテナランタイムが[systemd cgroupドライバー](/docs/setup/production-environment/container-runtimes#systemd-cgroup-driver)を使うように設定されていること ### Linuxディストリビューションのcgroup v2サポート cgroup v2を使っているLinuxディストリビューションの一覧は[cgroup v2ドキュメント](https://github.com/opencontainers/runc/blob/main/docs/cgroup-v2.md)をご覧ください。 <!-- 一覧は https://github.com/opencontainers/runc/blob/main/docs/cgroup-v2.md と同期してください --> * Container-Optimized OS (M97以降) * Ubuntu (21.10以降, 22.04以降推奨) * Debian GNU/Linux (Debian 11 bullseye以降) * Fedora (31以降) * Arch Linux (April 2021以降) * RHEL and RHEL-like distributions (9以降) あなたのディストリビューションがcgroup v2を使っているかどうかを確認するためには、あなたのディストリビューションのドキュメントを参照するか、[Linux Nodeのcgroupバージョンを特定する](#check-cgroup-version)の説明に従ってください。 カーネルのcmdlineの起動時引数を修正することで、手動であなたのLinuxディストリビューションのcgroup v2を有効にすることもできます。 あなたのディストリビューションがGRUBを使っている場合は、 `/etc/default/grub`の中の`GRUB_CMDLINE_LINUX`に`systemd.unified_cgroup_hierarchy=1`を追加し、`sudo update-grub`を実行してください。 ただし、おすすめの方法はデフォルトですでにcgroup v2が有効になっているディストリビューションを使うことです。 ### cgroup v2への移行 {#migrating-cgroupv2} cgroup v2に移行するには、[必要要件](#requirements)を満たすことを確認し、 cgroup v2がデフォルトで有効であるカーネルバージョンにアップグレードします。 kubeletはOSがcgroup v2で動作していることを自動的に検出し、それに応じて処理を行うため、追加設定は必要ありません。 ノード上やコンテナ内からユーザーが直接cgroupファイルシステムにアクセスしない限り、cgroup v2に切り替えたときのユーザー体験に目立った違いはないはずです。 cgroup v2はcgroup v1とは違うAPIを利用しているため、cgroupファイルシステムに直接アクセスしているアプリケーションはcgroup v2をサポートしている新しいバージョンに更新する必要があります。例えば、 * サードパーティーの監視またはセキュリティエージェントはcgroupファイルシステムに依存していることがあります。 エージェントをcgroup v2をサポートしているバージョンに更新してください。 * Podやコンテナを監視するために[cAdvisor](https://github.com/google/cadvisor)をスタンドアローンのDaemonSetとして起動している場合、v0.43.0以上に更新してください。 * JDKを利用している場合、[cgroup v2を完全にサポートしている](https://bugs.openjdk.org/browse/JDK-8230305)JDK 11.0.16以降、またはJDK15以降を利用することが望ましいです。 ## Linux Nodeのcgroupバージョンを特定する {#check-cgroup-version} cgroupバージョンは利用されているLinuxディストリビューションと、OSで設定されているデフォルトのcgroupバージョンに依存します。 あなたのディストリビューションがどちらのcgroupバージョンを利用しているのかを確認するには、`stat -fc %T /sys/fs/cgroup/`コマンドをノード上で実行してください。 ```shell stat -fc %T /sys/fs/cgroup/ ``` cgroup v2では、`cgroup2fs`と出力されます。 cgroup v1では、`tmpfs`と出力されます。 ## {{% heading "whatsnext" %}} - [cgroups](https://man7.org/linux/man-pages/man7/cgroups.7.html)についてもっと学習しましょう。 - [コンテナランタイム](/ja/docs/concepts/architecture/cri)についてもっと学習しましょう。 - [cgroupドライバー](/docs/setup/production-environment/container-runtimes#cgroup-drivers)についてもっと学習しましょう。 </ins>


Being vulnerable does not necessarily mean that your service will be exploited. Though your services are vulnerable in some ways unknown to you, offenders still need to identify these vulnerabilities and then exploit them. If offenders fail to exploit your service vulnerabilities, you win! In other words, having a vulnerability that can’t be exploited, represents a risk that can’t be realized. <del> {{< figure src="Example.png" alt="Image of an example of offender gaining foothold in a service" class="diagram-large" caption="Figure 1. An Offender gaining foothold in a vulnerable service" >}} </del> <ins> {{< figure src="security_behavior_figure_1.svg" alt="Image of an example of offender gaining foothold in a service" class="diagram-large" caption="Figure 1. An Offender gaining foothold in a vulnerable service" >}} </ins> The above diagram shows an example in which the offender does not yet have a foothold in the service; that is, it is assumed that your service does not run code controlled by the offender on day 1. In our example the service has vulnerabilities in the API exposed to clients. To gain an initial foothold the offender uses a malicious client to try and exploit one of the service API vulnerabilities. The malicious client sends an exploit that triggers some unplanned behavior of the service.


Kubernetes is often used to support workloads designed with microservice architecture. By design, microservices aim to follow the UNIX philosophy of "Do One Thing And Do It Well". Each microservice has a bounded context and a clear interface. In other words, you can expect the microservice clients to send relatively regular requests and the microservice to present a relatively regular behavior as a response to these requests. Consequently, a microservice architecture is an excellent candidate for security-behavior monitoring. <del> {{< figure src="Microservices.png" alt="Image showing why microservices are well suited for security-behavior monitoring" class="diagram-large" caption="Figure 2. Microservices are well suited for security-behavior monitoring" >}} </del> <ins> {{< figure src="security_behavior_figure_2.svg" alt="Image showing why microservices are well suited for security-behavior monitoring" class="diagram-large" caption="Figure 2. Microservices are well suited for security-behavior monitoring" >}} </ins> The diagram above clarifies how dividing a monolithic service to a set of microservices improves our ability to perform security-behavior monitoring and control. In a monolithic service approach, different client requests are intertwined, resulting in a diminished ability to identify irregular client behaviors. Without prior knowledge, an observer of the intertwined client requests will find it hard to distinguish between types of requests and their related characteristics. Further, internal client requests are not exposed to the observer. Lastly, the aggregated behavior of the monolithic service is a compound of the many different internal behaviors of its components, making it hard to identify irregular service behavior.