| nohup: ignoring input | |
| ============================================================ | |
| GPU监控脚本启动 | |
| ============================================================ | |
| [配置] 检查间隔: 10秒 | |
| [配置] 显存阈值: 1000MB | |
| [配置] GPU利用率阈值: 5% | |
| [配置] 服务端口: 8004 | |
| ============================================================ | |
| [2025-12-18 14:38:11] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率45%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率65%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率44%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:38:21] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率33%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率33%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率44%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:38:31] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率14%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率16%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率40%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:38:41] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率32%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率35%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率34%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:38:52] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率19%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率39%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率24%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:39:02] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率37%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率29%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率25%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:39:12] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率15%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率18%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率33%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:39:22] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率69%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率15%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率48%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:39:33] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率34%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率37%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率23%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:39:43] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率29%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率14%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率41%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:39:53] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率34%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率50%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率35%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:40:04] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率33%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率37%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率31%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:40:14] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率30%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率34%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率23%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:40:24] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率59%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率20%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率55%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:40:34] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率32%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率25%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率25%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:40:45] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率41%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率45%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率43%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:40:55] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率58%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率15%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率28%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:41:05] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率36%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率33%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率52%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:41:15] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率76%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率52%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率44%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:41:26] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率31%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率33%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率51%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:41:36] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率17%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率38%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率24%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:41:46] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率64%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率57%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率22%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:41:56] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率32%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率31%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率21%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:42:07] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率25%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率32%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率43%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:42:17] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率38%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率50%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率20%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:42:27] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率64%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率43%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率44%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:42:37] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率39%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率40%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率45%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:42:48] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率35%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率28%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率26%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:42:58] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率35%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率45%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率24%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:43:08] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率32%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率39%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率30%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:43:18] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率27%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率32%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率42%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:43:29] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率57%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率75%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率37%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:43:39] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率49%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率33%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率25%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:43:49] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率64%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率48%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率40%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:43:59] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率47%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率36%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率43%, GPU6: 40169MB/40960MB (98.1%), 利用率0%, GPU7: 40169MB/40960MB (98.1%), 利用率0% | |
| [2025-12-18 14:44:10] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率37%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率24%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率46%, GPU6: 40169MB/40960MB (98.1%), 利用率89%, GPU7: 40169MB/40960MB (98.1%), 利用率91% | |
| [2025-12-18 14:44:20] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率34%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率47%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率26%, GPU6: 40169MB/40960MB (98.1%), 利用率91%, GPU7: 40169MB/40960MB (98.1%), 利用率91% | |
| [2025-12-18 14:44:30] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率35%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率36%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率42%, GPU6: 40169MB/40960MB (98.1%), 利用率87%, GPU7: 40169MB/40960MB (98.1%), 利用率88% | |
| [2025-12-18 14:44:40] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率35%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率60%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率31%, GPU6: 40169MB/40960MB (98.1%), 利用率89%, GPU7: 40169MB/40960MB (98.1%), 利用率91% | |
| [2025-12-18 14:44:51] 无空闲GPU - GPU0: 25685MB/40960MB (62.7%), 利用率45%, GPU1: 35787MB/40960MB (87.4%), 利用率0%, GPU2: 31263MB/40960MB (76.3%), 利用率34%, GPU3: 36619MB/40960MB (89.4%), 利用率0%, GPU4: 37299MB/40960MB (91.1%), 利用率0%, GPU5: 39683MB/40960MB (96.9%), 利用率22%, GPU6: 40169MB/40960MB (98.1%), 利用率90%, GPU7: 40169MB/40960MB (98.1%), 利用率91% | |
| [INFO] 收到中断信号，正在清理... | |