!qsTqPdqMvjZiJfeuGc:matrix.org

tidb-chinese

79 Members
3 Servers

Load older messages

SenderMessageTime
29 Dec 2023
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang 应该正常启动的吧 07:51:48
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang [2023/12/29 11:09:57.423 +08:00] [INFO] [kvstore_compaction.go:56] ["finished scheduled compaction"] [compact-revision=47643] [took=52.193089ms] [2023/12/29 12:09:57.374 +08:00] [INFO] [index.go:189] ["compact tree index"] [revision=48794] [2023/12/29 12:09:57.387 +08:00] [INFO] [kvstore_compaction.go:56] ["finished scheduled compaction"] [compact-revision=48794] [took=12.68299ms] [2023/12/29 13:09:57.380 +08:00] [INFO] [index.go:189] ["compact tree index"] [revision=50054] [2023/12/29 13:09:57.393 +08:00] [INFO] [kvstore_compaction.go:56] ["finished scheduled compaction"] [compact-revision=50054] [took=13.179535ms] [2023/12/29 14:09:57.387 +08:00] [INFO] [index.go:189] ["compact tree index"] [revision=51314] [2023/12/29 14:09:57.403 +08:00] [INFO] [kvstore_compaction.go:56] ["finished scheduled compaction"] [compact-revision=51314] [took=14.965205ms] [2023/12/29 15:09:57.395 +08:00] [INFO] [index.go:189] ["compact tree index"] [revision=52574] [2023/12/29 15:09:57.407 +08:00] [INFO] [kvstore_compaction.go:56] ["finished scheduled compaction"] [compact-revision=52574] [took=12.086877ms] 07:52:53
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang [2023/12/29 15:48:02.888 +08:00] [ERROR] [tidb_requests.go:64] ["fail to send schema request"] [component=TiDB] [error=error.tidb.no_alive_tidb] [2023/12/29 15:49:02.888 +08:00] [WARN] [forwarder.go:106] ["Unable to resolve connection address since no alive TiDB instance"] [2023/12/29 15:49:02.888 +08:00] [ERROR] [tidb_requests.go:64] ["fail to send schema request"] [component=TiDB] [error=error.tidb.no_alive_tidb] [2023/12/29 15:50:02.888 +08:00] [WARN] [forwarder.go:106] ["Unable to resolve connection address since no alive TiDB instance"] [2023/12/29 15:50:02.888 +08:00] [ERROR] [tidb_requests.go:64] ["fail to send schema request"] [component=TiDB] [error=error.tidb.no_alive_tidb] [2023/12/29 15:51:02.888 +08:00] [WARN] [forwarder.go:106] ["Unable to resolve connection address since no alive TiDB instance"] [2023/12/29 15:51:02.888 +08:00] [ERROR] [tidb_requests.go:64] ["fail to send schema request"] [component=TiDB] [error=error.tidb.no_alive_tidb] 07:53:04
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang 使用tidb-operator安装起来的，这个问题比较容易出现 08:11:03
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang kubectl logs tidb-cluster-pd-1 -n tidb-cluster | grep "cluster id" [2023/12/29 10:10:01.368 +08:00] [INFO] [server.go:411] ["init cluster id"] [cluster-id=7317539057677356381] kubectl logs tidb-cluster-pd-0 -n tidb-cluster | grep "cluster id" [2023/12/29 10:10:01.627 +08:00] [INFO] [server.go:411] ["init cluster id"] [cluster-id=7317539057677356381] kubectl logs tidb-cluster-pd-2 -n tidb-cluster | grep "cluster id" [2023/12/29 10:10:01.371 +08:00] [INFO] [server.go:411] ["init cluster id"] [cluster-id=7317539057677356381] 08:12:23
@telegram_236842890:t2bot.io我能能现在是不是又正常了08:22:18
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang 不正常，恢复不了 08:23:15
@xuanwo:matrix.org@xuanwo:matrix.org left the room.08:24:47
@telegram_236842890:t2bot.io我能能PD正常的话可能是连接的问题吧，从日志看之前是成功连接过的，不然不会创建cluster id08:24:14
@telegram_236842890:t2bot.io我能能要不你从tikv那边试着ping pd，或者调用pd的API，比如 http://tidb-cluster-pd:2379/pd/api/v1/stores08:24:55
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang 容器里面http://tidb-cluster-pd:2379/pd/api/v1/stores 这个能调通 08:28:09
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang [root@tidb-cluster-tikv-0 /]# curl http://tidb-cluster-pd:2379/pd/api/v1/stores { "count": 3, "stores": [ { "store": { "id": 2001, "address": "tidb-cluster-tikv-0.tidb-cluster-tikv-peer.tidb-cluster.svc:20160", "version": "7.1.2", "peer_address": "tidb-cluster-tikv-0.tidb-cluster-tikv-peer.tidb-cluster.svc:20160", "status_address": "tidb-cluster-tikv-0.tidb-cluster-tikv-peer.tidb-cluster.svc:20180", "git_hash": "8632b3952d931e510d00953f89477ce095b3d902", "start_timestamp": 1703756506, "deploy_path": "/", "last_heartbeat": 1703812921932741743, "node_state": 1, "state_name": "Down" }, "status": { "capacity": "0B", "available": "0B", "used_size": "0B", "leader_count": 0, "leader_weight": 1, "leader_score": 0, "leader_size": 0, "region_count": 6, "region_weight": 1, "region_score": 0, "region_size": 0, "start_ts": "2023-12-28T17:41:46+08:00", "last_heartbeat_ts": "2023-12-29T09:22:01.932741743+08:00", "uptime": "15h40m15.932741743s" } }, { "store": { "id": 2004, "address": "tidb-cluster-tikv-2.tidb-cluster-tikv-peer.tidb-cluster.svc:20160", "version": "7.1.2", "peer_address": "tidb-cluster-tikv-2.tidb-cluster-tikv-peer.tidb-cluster.svc:20160", "status_address": "tidb-cluster-tikv-2.tidb-cluster-tikv-peer.tidb-cluster.svc:20180", "git_hash": "8632b3952d931e510d00953f89477ce095b3d902", "start_timestamp": 1703756505, "deploy_path": "/", "last_heartbeat": 1703812921542561872, "node_state": 1, "state_name": "Down" }, "status": { "capacity": "0B", "available": "0B", "used_size": "0B", "leader_count": 0, "leader_weight": 1, "leader_score": 0, "leader_size": 0, "region_count": 6, "region_weight": 1, "region_score": 0, "region_size": 0, "start_ts": "2023-12-28T17:41:45+08:00", "last_heartbeat_ts": "2023-12-29T09:22:01.542561872+08:00", "uptime": "15h40m16.542561872s" } }, { "store": { "id": 3001, "address": "tidb-cluster-tikv-1.tidb-cluster-tikv-peer.tidb-cluster.svc:20160", "version": "7.1.2", "peer_address": "tidb-cluster-tikv-1.tidb-cluster-tikv-peer.tidb-cluster.svc:20160", "status_address": "tidb-cluster-tikv-1.tidb-cluster-tikv-peer.tidb-cluster.svc:20180", "git_hash": "8632b3952d931e510d00953f89477ce095b3d902", "start_timestamp": 1703756506, "deploy_path": "/", "last_heartbeat": 1703812921870843688, "node_state": 1, "state_name": "Down" }, "status": { "capacity": "0B", "available": "0B", "used_size": "0B", "leader_count": 0, "leader_weight": 1, "leader_score": 0, "leader_size": 0, "region_count": 6, "region_weight": 1, "region_score": 0, "region_size": 0, "start_ts": "2023-12-28T17:41:46+08:00", "last_heartbeat_ts": "2023-12-29T09:22:01.870843688+08:00", "uptime": "15h40m15.870843688s" } } ] } 08:28:51
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang 亲人们，这问题太奇怪了 08:32:46
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang 有谁碰到过吗 08:34:14
@telegram_236842890:t2bot.io我能能嗯，看起来之前tikv连上过，然后从上午9点22分开始就都连不上了08:31:27
@telegram_236842890:t2bot.io我能能要不把tikv重启一遍试试08:31:40
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang 重启过了，没用！ 08:36:27
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang pd应该是好的，但是tikv就是连不上 08:40:08
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang [2023/12/29 16:38:29.653 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:31.654 +08:00] [INFO] [util.rs:567] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:31.654 +08:00] [WARN] [client.rs:166] ["validate PD endpoints failed"] [err="Other(\"[components/pd_client/src/util.rs:600]: PD cluster failed to respond\")"] [2023/12/29 16:38:31.955 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:33.956 +08:00] [INFO] [util.rs:567] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:34.257 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:36.258 +08:00] [INFO] [util.rs:567] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:36.559 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:38.560 +08:00] [INFO] [util.rs:567] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:38.861 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:40.862 +08:00] [INFO] [util.rs:567] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:41.163 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:43.164 +08:00] [INFO] [util.rs:567] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:43.465 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:45.466 +08:00] [INFO] [util.rs:567] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:45.767 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:47.768 +08:00] [INFO] [util.rs:567] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:48.069 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:50.070 +08:00] [INFO] [util.rs:567] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-cluster-pd:2379] [2023/12/29 16:38:50.371 +08:00] [INFO] [util.rs:605] ["connecting to PD endpoint"] [endpoints=http://tidb-cluster-pd:2379] 08:40:48
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang yanqing (PingCAP) 接下去怎么定位啊，我重置了集群，pd，tikv，tidb的个数都设置为1，tikv还是无法连接pd 08:55:34
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang [root@tidb-cluster-tikv-0 /]# curl http://tidb-cluster-pd:2379/pd/api/v1/stores "[PD:cluster:ErrNotBootstrapped]TiKV cluster not bootstrapped, please start TiKV first" 08:55:58
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang [2023/12/29 18:40:11.037 +08:00] [INFO] [config.rs:726] ["kernel parameters"] [value=0] [param=vm.swappiness] [thread_id=0x4] [2023/12/29 18:40:11.037 +08:00] [WARN] [server.rs:1619] ["check: kernel"] [err="kernel parameters net.core.somaxconn got 4096, expect 32768"] [thread_id=0x4] [2023/12/29 18:40:11.037 +08:00] [WARN] [server.rs:1619] ["check: kernel"] [err="kernel parameters net.ipv4.tcp_syncookies got 1, expect 0"] [thread_id=0x4] 10:46:50
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang 日志里面看到有这个告警，和这个有关系吗？ 10:47:06
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang 看了下，和这个也没有关系 11:27:33
@_slack_tidbcommunity_U0523P40G6M:matrix.orgsimonzhang yanqing (PingCAP) 有空了帮忙定位一下吧，这个问题很困扰我 11:27:55
@telegram_121134544:t2bot.io@telegram_121134544:t2bot.io left the room.14:17:52
@telegram_202571881:t2bot.io@telegram_202571881:t2bot.io left the room.14:29:06
9 Jan 2024
@_slack_tidbcommunity_UK0QZ8N4D:matrix.orgWink Yao - PingCAP changed their display name from Wink Yao(PingCAP) to Wink Yao - PingCAP.06:58:20
@_slack_tidbcommunity_UK0QZ8N4D:matrix.orgWink Yao - PingCAP changed their profile picture.06:58:21
@_slack_tidbcommunity_UK0QZ8N4D:matrix.orgWink Yao - PingCAP simonzhang 中文的话，请发到 asktug.com 吧，那边会有版主帮忙一起看看问题。 06:58:21

There are no newer messages yet.

Back to Room ListRoom Version: 6