Log Viewer

< Previous

September 2025

Next >

Timestamp Int Value Tag Log Level String Value Tag Status Actions
9/11/2025 10:11:06 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0033cb0 qid=0x0000 qp_destroy_elapsed=51131us res_destroy_elapsed=14us
9/11/2025 10:11:06 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00f3910, qp_create_elapsed=25115us, res_elapsed=39us
9/11/2025 10:11:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/11/2025 10:11:06 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:88:00.0
9/11/2025 10:11:06 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:88:00
9/11/2025 10:11:06 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:88:00.0
9/11/2025 10:11:06 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f3910 Final Negotiated Queue Depth=256 R/W Depth=16
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_4]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00e7500 qid=0x0001 ctrlr=0x7f52cc2e7320 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000e
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00e7500, qp_create_elapsed=27316us, res_elapsed=43us
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f3910 qid=0x0000 qp_destroy_elapsed=38217us res_destroy_elapsed=15us
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00e7500 Final Negotiated Queue Depth=256 R/W Depth=16
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_3]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d0035480 qid=0x0001 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d0035480, qp_create_elapsed=24021us, res_elapsed=36us
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0035480 Final Negotiated Queue Depth=256 R/W Depth=16
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00f71b0 qid=0x0000 ctrlr=0x7f52d4198010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000e
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000e
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f71b0, qp_create_elapsed=23999us, res_elapsed=12us
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f71b0 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 ctrlr=0x7f52d42efc40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00f9af0 qid=0x0000 ctrlr=0x7f52d42efc40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f9af0 qid=0x0000
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00f9af0 qid=0x0000 ctrlr=0x7f52d42efc40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:86:00.0
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:86:00
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:86:00.0
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d010db90 qid=0x0000 ctrlr=0x7f52e81f7010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0013
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0013
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d010db90, qp_create_elapsed=25378us, res_elapsed=15us
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme105n1 ENABLED, Status NVG_SUCCESS
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme105n1 for latency reports
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010db90 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme104n1 ENABLED, Status NVG_SUCCESS
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme104n1 for latency reports
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme103n1 ENABLED, Status NVG_SUCCESS
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme103n1 for latency reports
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00f2680 qid=0x0000 ctrlr=0x7f52cc2e7320 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000e
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000e
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00f2680, qp_create_elapsed=24229us, res_elapsed=14us
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme102n1 ENABLED, Status NVG_SUCCESS
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme102n1 for latency reports
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f2680 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 ctrlr=0x7f52c8206f60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0012
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 4
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme105n1 at 0000:07:00.0 [8CE38EE300FF11DF] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 with nsid=1
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f6780 qid=0x0000 ctrlr=0x7f52c8206f60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0012
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f6780 qid=0x0000
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f6780 qid=0x0000 ctrlr=0x7f52c8206f60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0012
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 ctrlr=0x7f52e819b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000d, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03, event: type=0002, info=0000, log page id=0004
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52e819b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0033cb0 qid=0x0000
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52e819b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme105n1 eui64:8CE38EE300FF11DF sn:2ET0A01D0AW4 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme105n1 bdf:0000:07:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 eui64:8CE38EE300FF11DF sn:2ET0A01D0AW4
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:85:00.0
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:85:00
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:85:00.0
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme104n1 at 0000:05:00.0 [8CE38EE202891501] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 with nsid=1
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000d, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01, event: type=0002, info=0000, log page id=0004
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d0035320 qid=0x0000 ctrlr=0x7f52c8205470 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme104n1 eui64:8CE38EE202891501 sn:70G0A00FTBX7 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme104n1 bdf:0000:05:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 eui64:8CE38EE202891501 sn:70G0A00FTBX7
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d0035320, qp_create_elapsed=23854us, res_elapsed=9us
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme103n1 at 0000:8b:00.0 [002538E9410026BC] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 with nsid=1
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0035320 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 ctrlr=0x7f52cc19d450 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0012, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09, event: type=0002, info=0000, log page id=0004
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme103n1 eui64:002538E9410026BC sn:S7WBNE0X900403 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme103n1 bdf:0000:8b:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 eui64:002538E9410026BC sn:S7WBNE0X900403
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52cc19d450 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f3910 qid=0x0000
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52cc19d450 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme102n1 at 0000:04:00.0 [0000000000000000] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 with nsid=1
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00, event: type=0002, info=0000, log page id=0004
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme102n1 eui64:0000000000000000 sn:S80DNG0Y200115 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00
9/11/2025 10:11:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme102n1 bdf:0000:04:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 eui64:0000000000000000 sn:S80DNG0Y200115
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:07:00.0 (HotInNvme105)
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:07:00.0 name=HotInNvme105 ctrlr=0x200139fa2200
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme105
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme105n1 sn:(2ET0A01D0AW4 ) mn:(KCM77680P5AxFTRI ) blocklen:512 blockcnt:15002931888 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:10 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:05:00.0 (HotInNvme104)
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:05:00.0 name=HotInNvme104 ctrlr=0x200131960400
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme104
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme104n1 sn:(70G0A00FTBX7 ) mn:(TCM615T4P5xnFTRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:7 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:13 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:8b:00.0 (HotInNvme103)
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme103n1 sn:(S7WBNE0X900403 ) mn:(AELN30T7P5xnFQRI ) blocklen:512 blockcnt:60001615872 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:8b:00.0 name=HotInNvme103 ctrlr=0x200136874500
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme103
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:07:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:8b:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:8b:00.0
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:8b:00.0] !!! Setting CC.EN = 1
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:07:00.0] !!! Setting CC.EN = 1
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:05:00.0] !!! Setting CC.EN = 1
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:05:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:8b:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:07:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:05:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:8b:00
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:8b:00.0
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:07:00.0
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:07:00
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:07:00.0
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:05:00
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:05:00.0
9/11/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:05:00.0
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:04:00.0 (HotInNvme102)
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:04:00.0 name=HotInNvme102 ctrlr=0x200134121340
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme102
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme102n1 sn:(S80DNG0Y200115 ) mn:(SAMSUNG MZ3MO15THCLC-00AW7 ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:6 (262144) optimal_io_boundary:0 max_segment_size:262144 max_num_segments:256
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:04:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:04:00.0
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:04:00.0] !!! Setting CC.EN = 1
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:04:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:04:00
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:04:00.0
9/11/2025 10:11:03 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 10:09:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 10:09:37 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 0
9/11/2025 10:09:37 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c6:00.0 with driver=nvme
9/11/2025 10:09:37 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:85:00.0 with driver=nvme
9/11/2025 10:09:37 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:84:00.0 with driver=nvme
9/11/2025 10:09:37 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0106220 qid=0x0001 qp_destroy_elapsed=48644us res_destroy_elapsed=110us
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010bca0 qid=0x0000 qp_destroy_elapsed=37684us res_destroy_elapsed=16us
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme99
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:84:00
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0000, state=6
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0000
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:84:00.0] Prepare to destruct SSD: 0000:84:00.0
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:84:00.0] already in failed state
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:84:00.0]: unregistering thread=app_thread
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:84:00.0
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:84:00.0)
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:84:00.0: destruct=1
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 23 times ***
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52d41b2010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f5c20, qp_create_elapsed=24796us, res_elapsed=11us
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 1 time ***
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 1 time ***
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme99n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme99n1 from latency reports
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f5c20 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme99n1
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 ctrlr=0x7f52c85f0e30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme93n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme93n1 from latency reports
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d0106220 qid=0x0001 ctrlr=0x7f52c85f0e30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0106220 qid=0x0001
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d0106220 qid=0x0001 ctrlr=0x7f52c85f0e30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d010bca0 qid=0x0000 ctrlr=0x7f52c85f0e30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010bca0 qid=0x0000
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d010bca0 qid=0x0000 ctrlr=0x7f52c85f0e30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme93n1
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme100n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme100n1 from latency reports
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme100n1
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 5 times ***
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 0
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c6:00.0 is missing
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:85:00.0 is missing
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:84:00.0 is missing
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme99n1 bdf:0000:84:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 eui64:8CE38EE2028DDB01 sn:70H0A00GTBU7 reason:unknown
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme93n1 bdf:0000:85:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 eui64:0025389401006B55 sn:S5URNE0N400075 reason:unknown
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme100n1 bdf:0000:c6:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 eui64:8CE38EE30087685F sn:YDF0A0380VY3 reason:unknown
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme99n1 was hot removed
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme93n1 was hot removed
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme100n1 was hot removed
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:ca:00.0 with driver=nvme
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:88:00.0 with driver=nvme
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:86:00.0 with driver=nvme
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:84:00.0] already in failed state
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:84:00.0)
9/11/2025 10:09:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:84:00.0: destruct=1
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134120260, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x2001393afe60, qid=0x04: Aborting trackers
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x20013840b4e0, qid=0x03: Aborting trackers
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0004
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0003
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0002
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139461c60 was disconnected and freed. reset controller.
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001393afe60 was disconnected and freed. reset controller.
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840b4e0 was disconnected and freed. reset controller.
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0004, state=6
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0003, state=6
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0002, state=6
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0001, state=6
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0000, state=5
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200139461c60, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x2001393afe60, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013840b4e0, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200134ccfe60, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x200139461c60, qid=0x02: Aborting trackers
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x200134ccfe60, qid=0x01: Aborting trackers
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0001
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x200139461c60, qid=0x0002: enter
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x2001393afe60, qid=0x0004: enter
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x20013840b4e0, qid=0x0003: enter
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x200134ccfe60, qid=0x0001: enter
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:84:00.0)
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:84:00.0: destruct=0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:84:00.0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200134ccfe60 was disconnected and freed. reset controller.
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f5c20 qid=0x0001 qp_destroy_elapsed=49693us res_destroy_elapsed=114us
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010c060 qid=0x0000 qp_destroy_elapsed=37923us res_destroy_elapsed=16us
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme100
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:c6:00
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0000, state=6
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0000
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 24 times ***
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139fa2060, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:c6:00.0] Prepare to destruct SSD: 0000:c6:00.0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c6:00.0] already in failed state
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:c6:00.0]: unregistering thread=app_thread
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:c6:00.0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c6:00.0)
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c6:00.0: destruct=1
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139fa2060, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52e860f3e0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00fa810, qp_create_elapsed=24730us, res_elapsed=8us
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00fa810 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139fa2060, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00f5c20 qid=0x0001 ctrlr=0x7f52cc39d770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f5c20 qid=0x0001
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00f5c20 qid=0x0001 ctrlr=0x7f52cc39d770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139fa2060, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 ctrlr=0x7f52cc39d770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d010c060 qid=0x0000 ctrlr=0x7f52cc39d770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010c060 qid=0x0000
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d010c060 qid=0x0000 ctrlr=0x7f52cc39d770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139fa2060, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139fa2060, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c6:00.0)
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c6:00.0: destruct=1
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 32 times ***
9/11/2025 10:09:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c6:00.0] already in failed state
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00fa810 qid=0x0001 qp_destroy_elapsed=49519us res_destroy_elapsed=114us
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139fa2060, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00eb520 qid=0x0000 qp_destroy_elapsed=38450us res_destroy_elapsed=15us
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139fa2060, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139fa2060, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x200139f7fe60, qid=0x03: Aborting trackers
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x20013840b2e0, qid=0x04: Aborting trackers
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x20013352fe60, qid=0x02: Aborting trackers
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0004
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0003
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0002
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0001
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200140b6fe60 was disconnected and freed. reset controller.
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139f7fe60 was disconnected and freed. reset controller.
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840b2e0 was disconnected and freed. reset controller.
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013352fe60 was disconnected and freed. reset controller.
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0004, state=6
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0003, state=6
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0002, state=6
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0001, state=6
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0000, state=5
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200140b6fe60, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200139f7fe60, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013840b2e0, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013352fe60, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x200140b6fe60, qid=0x01: Aborting trackers
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x200140b6fe60, qid=0x0001: enter
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x200139f7fe60, qid=0x0003: enter
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x20013840b2e0, qid=0x0004: enter
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x20013352fe60, qid=0x0002: enter
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c6:00.0)
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c6:00.0: destruct=0
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:c6:00.0
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme93
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:85:00
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0000, state=6
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0000
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 27 times ***
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:85:00.0] Prepare to destruct SSD: 0000:85:00.0
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:85:00.0] already in failed state
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:85:00.0]: unregistering thread=app_thread
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:85:00.0
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:85:00.0)
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:85:00.0: destruct=1
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d010f0b0 qid=0x0000 ctrlr=0x7f52cc187fa0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d010f0b0, qp_create_elapsed=24678us, res_elapsed=14us
9/11/2025 10:09:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010f0b0 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 ctrlr=0x7f52e83a0c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00fa810 qid=0x0001 ctrlr=0x7f52e83a0c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00fa810 qid=0x0001
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00fa810 qid=0x0001 ctrlr=0x7f52e83a0c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00eb520 qid=0x0000 ctrlr=0x7f52e83a0c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00eb520 qid=0x0000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00eb520 qid=0x0000 ctrlr=0x7f52e83a0c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 6 times ***
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 21 times ***
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:85:00.0] already in failed state
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:85:00.0)
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:85:00.0: destruct=1
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0038c80 qid=0x0001 qp_destroy_elapsed=49196us res_destroy_elapsed=112us
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ee5d0 qid=0x0000 qp_destroy_elapsed=38055us res_destroy_elapsed=15us
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 10 times ***
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x200132875ee0, qid=0x03: Aborting trackers
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0003
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0002
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013315d0e0 was disconnected and freed. reset controller.
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132875ee0 was disconnected and freed. reset controller.
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0004, state=6
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0003, state=6
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0002, state=6
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0000, state=5
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x20013315d0e0, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200132875ee0, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x200139bc9e60, qid=0x04: Aborting trackers
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x20013315d0e0, qid=0x02: Aborting trackers
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x2001325d90e0, qid=0x01: Aborting trackers
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0004
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0001
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x200139bc9e60, qid=0x0004: enter
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x20013315d0e0, qid=0x0002: enter
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x200132875ee0, qid=0x0003: enter
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:85:00.0)
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:85:00.0: destruct=0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:85:00.0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139bc9e60 was disconnected and freed. reset controller.
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001325d90e0 was disconnected and freed. reset controller.
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0001, state=6
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200139bc9e60, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325d90e0, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x2001325d90e0, qid=0x0001: enter
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme98n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme98n1 from latency reports
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme98n1
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme97n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme97n1 from latency reports
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme97n1
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme96n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme96n1 from latency reports
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme96n1
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 3
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:ca:00.0 is missing
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:88:00.0 is missing
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:86:00.0 is missing
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme98n1 bdf:0000:88:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 eui64:8CE38EE300B00E06 sn:1EE0A0T603R3 reason:unknown
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme97n1 bdf:0000:86:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 eui64:002538E7415006FE sn:S7Z9NG0X700006 reason:unknown
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme96n1 bdf:0000:ca:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 eui64:002538E141500853 sn:S7PNNG0X100108 reason:unknown
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme98n1 was hot removed
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme97n1 was hot removed
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme96n1 was hot removed
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:8b:00.0 with driver=nvme
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:8a:00.0 with driver=nvme
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme97
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:86:00
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0000, state=6
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0000
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 24 times ***
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:86:00.0] Prepare to destruct SSD: 0000:86:00.0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:86:00.0] already in failed state
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:86:00.0]: unregistering thread=app_thread
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:86:00.0
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:86:00.0)
9/11/2025 10:09:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:86:00.0: destruct=1
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00ebb60 qid=0x0000 ctrlr=0x7f52c8396010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00ebb60, qp_create_elapsed=23893us, res_elapsed=13us
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ebb60 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d0038c80 qid=0x0001 ctrlr=0x7f52d44bf640 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0038c80 qid=0x0001
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d0038c80 qid=0x0001 ctrlr=0x7f52d44bf640 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 ctrlr=0x7f52d44bf640 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00ee5d0 qid=0x0000 ctrlr=0x7f52d44bf640 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ee5d0 qid=0x0000
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00ee5d0 qid=0x0000 ctrlr=0x7f52d44bf640 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 9 times ***
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 10 times ***
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:86:00.0] already in failed state
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:86:00.0)
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:86:00.0: destruct=1
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ee990 qid=0x0001 qp_destroy_elapsed=50727us res_destroy_elapsed=111us
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ee210 qid=0x0000 qp_destroy_elapsed=38698us res_destroy_elapsed=15us
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 21 times ***
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x200131a77e60, qid=0x01: Aborting trackers
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0003
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0001
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013742fe60 was disconnected and freed. reset controller.
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200131a77e60 was disconnected and freed. reset controller.
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0004, state=6
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0003, state=6
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0002, state=6
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0001, state=6
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0000, state=5
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x200137c0fe60, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013742fe60, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013391fe60, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x200131a77e60, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x200137c0fe60, qid=0x02: Aborting trackers
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x20013742fe60, qid=0x03: Aborting trackers
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x20013391fe60, qid=0x04: Aborting trackers
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0004
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0002
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x200137c0fe60, qid=0x0002: enter
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x20013742fe60, qid=0x0003: enter
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x20013391fe60, qid=0x0004: enter
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x200131a77e60, qid=0x0001: enter
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:86:00.0)
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:86:00.0: destruct=0
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:86:00.0
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200137c0fe60 was disconnected and freed. reset controller.
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013391fe60 was disconnected and freed. reset controller.
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme98
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:88:00
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0000
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0000, state=6
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200139439e60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:88:00.0] Prepare to destruct SSD: 0000:88:00.0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:88:00.0] already in failed state
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:88:00.0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:88:00.0)
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:88:00.0: destruct=1
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 25 times ***
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:88:00.0]: unregistering thread=app_thread
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200139439e60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00e7140 qid=0x0000 ctrlr=0x7f52d4506010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00e7140, qp_create_elapsed=24377us, res_elapsed=12us
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00e7140 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200139439e60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 ctrlr=0x7f52e81660e0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00ee990 qid=0x0001 ctrlr=0x7f52e81660e0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ee990 qid=0x0001
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00ee990 qid=0x0001 ctrlr=0x7f52e81660e0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200139439e60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00ee210 qid=0x0000 ctrlr=0x7f52e81660e0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ee210 qid=0x0000
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00ee210 qid=0x0000 ctrlr=0x7f52e81660e0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 7 times ***
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200139439e60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200139439e60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:88:00.0] already in failed state
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:88:00.0)
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:88:00.0: destruct=1
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200139439e60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200139439e60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x20013fdffe60, qid=0x02: Aborting trackers
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x200138fbfe60, qid=0x01: Aborting trackers
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0004
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0002
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0001
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fdffe60 was disconnected and freed. reset controller.
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013979fe60 was disconnected and freed. reset controller.
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200138fbfe60 was disconnected and freed. reset controller.
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0004, state=6
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0003, state=6
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0002, state=6
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0001, state=6
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0000, state=5
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x20013fdffe60, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x20013979fe60, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200139461e60, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200138fbfe60, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x20013979fe60, qid=0x04: Aborting trackers
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x200139461e60, qid=0x03: Aborting trackers
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0003
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x20013fdffe60, qid=0x0002: enter
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x20013979fe60, qid=0x0004: enter
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x200139461e60, qid=0x0003: enter
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x200138fbfe60, qid=0x0001: enter
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:88:00.0)
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:88:00.0: destruct=0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:88:00.0
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139461e60 was disconnected and freed. reset controller.
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme96
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:ca:00
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0000, state=6
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0000
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 19 times ***
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:ca:00.0] Prepare to destruct SSD: 0000:ca:00.0
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:ca:00.0] already in failed state
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:ca:00.0]: unregistering thread=app_thread
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:ca:00.0
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:ca:00.0)
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:ca:00.0: destruct=1
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010c7e0 qid=0x0001 qp_destroy_elapsed=49442us res_destroy_elapsed=112us
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ebf70 qid=0x0000 qp_destroy_elapsed=35396us res_destroy_elapsed=17us
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00360b0 qid=0x0000 ctrlr=0x7f52e835ec20 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00360b0, qp_create_elapsed=25134us, res_elapsed=11us
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 1 time ***
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00360b0 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 ctrlr=0x7f52cc313b10 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d010c7e0 qid=0x0001 ctrlr=0x7f52cc313b10 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010c7e0 qid=0x0001
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d010c7e0 qid=0x0001 ctrlr=0x7f52cc313b10 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00ebf70 qid=0x0000 ctrlr=0x7f52cc313b10 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ebf70 qid=0x0000
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00ebf70 qid=0x0000 ctrlr=0x7f52cc313b10 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 9 times ***
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:ca:00.0: destruct=1
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:ca:00.0] already in failed state
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:ca:00.0)
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme94n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme94n1 from latency reports
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme94n1
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme90n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme90n1 from latency reports
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme90n1
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 6
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:8b:00.0 is missing
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:8a:00.0 is missing
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme94n1 bdf:0000:8a:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 eui64:002538E141500763 sn:S7PLNG0X100181 reason:unknown
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme90n1 bdf:0000:8b:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 eui64:002538E9410026BC sn:S7WBNE0X900403 reason:unknown
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme94n1 was hot removed
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme90n1 was hot removed
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:07:00.0 with driver=nvme
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:05:00.0 with driver=nvme
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:04:00.0 with driver=nvme
9/11/2025 10:09:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 31 times ***
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x2001405dfe60, qid=0x04: Aborting trackers
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x200138bcfe60, qid=0x02: Aborting trackers
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0004
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0003
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0002
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001405dfe60 was disconnected and freed. reset controller.
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200138bcfe60 was disconnected and freed. reset controller.
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200133d0fe60 was disconnected and freed. reset controller.
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0004, state=6
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0003, state=6
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0002, state=6
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0001, state=6
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0000, state=5
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001405dfe60, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013944dc60, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200138bcfe60, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200133d0fe60, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x20013944dc60, qid=0x01: Aborting trackers
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x200133d0fe60, qid=0x03: Aborting trackers
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0001
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x2001405dfe60, qid=0x0004: enter
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x20013944dc60, qid=0x0001: enter
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x200138bcfe60, qid=0x0002: enter
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x200133d0fe60, qid=0x0003: enter
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:ca:00.0)
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:ca:00.0: destruct=0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:ca:00.0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013944dc60 was disconnected and freed. reset controller.
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme94
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:8a:00
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0000, state=6
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 7 times ***
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:8a:00.0] Prepare to destruct SSD: 0000:8a:00.0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8a:00.0] already in failed state
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:8a:00.0]: unregistering thread=app_thread
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:8a:00.0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8a:00.0)
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8a:00.0: destruct=1
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010c420 qid=0x0001 qp_destroy_elapsed=49457us res_destroy_elapsed=114us
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00eb8e0 qid=0x0000 qp_destroy_elapsed=38113us res_destroy_elapsed=17us
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 11 times ***
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d010df10 qid=0x0000 ctrlr=0x7f52cc333010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d010df10, qp_create_elapsed=24389us, res_elapsed=13us
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010df10 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d010c420 qid=0x0001 ctrlr=0x7f52d4266450 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010c420 qid=0x0001
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d010c420 qid=0x0001 ctrlr=0x7f52d4266450 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 ctrlr=0x7f52d4266450 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00eb8e0 qid=0x0000 ctrlr=0x7f52d4266450 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00eb8e0 qid=0x0000
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00eb8e0 qid=0x0000 ctrlr=0x7f52d4266450 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8a:00.0] already in failed state
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8a:00.0)
9/11/2025 10:09:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8a:00.0: destruct=1
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0002
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139439c60 was disconnected and freed. reset controller.
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0002, state=6
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0001, state=6
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200139bc9c60, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200139439c60, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x200139bc9c60, qid=0x01: Aborting trackers
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x200139439c60, qid=0x02: Aborting trackers
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x200132853e60, qid=0x04: Aborting trackers
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x2001312dc360, qid=0x03: Aborting trackers
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0004
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0003
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0001
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x200139bc9c60, qid=0x0001: enter
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x200139439c60, qid=0x0002: enter
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8a:00.0)
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8a:00.0: destruct=0
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:8a:00.0
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139bc9c60 was disconnected and freed. reset controller.
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132853e60 was disconnected and freed. reset controller.
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001312dc360 was disconnected and freed. reset controller.
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0004, state=6
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0003, state=6
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0000, state=5
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200132853e60, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001312dc360, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x200132853e60, qid=0x0004: enter
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x2001312dc360, qid=0x0003: enter
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00e7140 qid=0x0001 qp_destroy_elapsed=48454us res_destroy_elapsed=117us
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme90
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:8b:00
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0000, state=6
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0000
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 1 time ***
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014082dc60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:8b:00.0] Prepare to destruct SSD: 0000:8b:00.0
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8b:00.0] already in failed state
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:8b:00.0]: unregistering thread=app_thread
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:8b:00.0
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8b:00.0)
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8b:00.0: destruct=1
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010a290 qid=0x0000 qp_destroy_elapsed=32994us res_destroy_elapsed=17us
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 22 times ***
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014082dc60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014082dc60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f6780 qid=0x0000 ctrlr=0x7f52c8206f60 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0012
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0012
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00f6780, qp_create_elapsed=24172us, res_elapsed=8us
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014082dc60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f6780 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014082dc60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00e7140 qid=0x0001 ctrlr=0x7f52d41c1f30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0011
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00e7140 qid=0x0001
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00e7140 qid=0x0001 ctrlr=0x7f52d41c1f30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0011
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 ctrlr=0x7f52d41c1f30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0011
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014082dc60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d010a290 qid=0x0000 ctrlr=0x7f52d41c1f30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0011
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010a290 qid=0x0000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d010a290 qid=0x0000 ctrlr=0x7f52d41c1f30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0011
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014082dc60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8b:00.0] already in failed state
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0011, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8b:00.0)
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8b:00.0: destruct=1
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014082dc60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014082dc60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20013840bce0, qid=0x02: Aborting trackers
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20013840b8e0, qid=0x04: Aborting trackers
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0004
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0003
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0002
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840bce0 was disconnected and freed. reset controller.
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840bae0 was disconnected and freed. reset controller.
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840b8e0 was disconnected and freed. reset controller.
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0004, state=6
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0003, state=6
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0002, state=6
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0001, state=6
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0000, state=5
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20013840bee0, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20013840bce0, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20013840bae0, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20013840b8e0, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20013840bee0, qid=0x01: Aborting trackers
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20013840bae0, qid=0x03: Aborting trackers
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0001
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20013840bee0, qid=0x0001: enter
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20013840bce0, qid=0x0002: enter
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20013840bae0, qid=0x0003: enter
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20013840b8e0, qid=0x0004: enter
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8b:00.0)
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8b:00.0: destruct=0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:8b:00.0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840bee0 was disconnected and freed. reset controller.
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00393f0 qid=0x0001 qp_destroy_elapsed=50288us res_destroy_elapsed=118us
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme92n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme92n1 from latency reports
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme92n1
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0038230 qid=0x0000 qp_destroy_elapsed=51861us res_destroy_elapsed=15us
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme91n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme91n1 from latency reports
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme91n1
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme89n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme89n1 from latency reports
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme92
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:07:00
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0000, state=6
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0000
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme89n1
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 8
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:07:00.0 is missing
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:05:00.0 is missing
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:04:00.0 is missing
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme92n1 bdf:0000:07:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 eui64:8CE38EE300FF11DF sn:2ET0A01D0AW4 reason:unknown
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme91n1 bdf:0000:05:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 eui64:8CE38EE202891501 sn:70G0A00FTBX7 reason:unknown
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme89n1 bdf:0000:04:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 eui64:0000000000000000 sn:S80DNG0Y200115 reason:unknown
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme92n1 was hot removed
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme91n1 was hot removed
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme89n1 was hot removed
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 25 times ***
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20014082de60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:07:00.0] Prepare to destruct SSD: 0000:07:00.0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:07:00.0] already in failed state
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:07:00.0]: unregistering thread=app_thread
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:07:00.0
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:07:00.0)
9/11/2025 10:09:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:07:00.0: destruct=1
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20014082de60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00f9af0 qid=0x0000 ctrlr=0x7f52d42efc40 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f9af0, qp_create_elapsed=24425us, res_elapsed=12us
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20014082de60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f9af0 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20014082de60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 ctrlr=0x7f52cc333010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00393f0 qid=0x0001 ctrlr=0x7f52cc333010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00393f0 qid=0x0001
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00393f0 qid=0x0001 ctrlr=0x7f52cc333010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20014082de60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d0038230 qid=0x0000 ctrlr=0x7f52cc333010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0038230 qid=0x0000
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d0038230 qid=0x0000 ctrlr=0x7f52cc333010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20014082de60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:07:00.0] already in failed state
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:07:00.0)
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:07:00.0: destruct=1
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20014082de60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20014082de60, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x20013ab682a0, qid=0x02: Aborting trackers
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x20013ab680a0, qid=0x03: Aborting trackers
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x20013840b6e0, qid=0x04: Aborting trackers
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0004
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0003
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0002
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0001
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013ab684a0 was disconnected and freed. reset controller.
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013ab682a0 was disconnected and freed. reset controller.
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013ab680a0 was disconnected and freed. reset controller.
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840b6e0 was disconnected and freed. reset controller.
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0004, state=6
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0003, state=6
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0002, state=6
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0001, state=6
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0000, state=5
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20013ab684a0, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20013ab682a0, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20013ab680a0, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20013840b6e0, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x20013ab684a0, qid=0x01: Aborting trackers
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x20013ab684a0, qid=0x0001: enter
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x20013ab682a0, qid=0x0002: enter
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x20013ab680a0, qid=0x0003: enter
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x20013840b6e0, qid=0x0004: enter
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:07:00.0)
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:07:00.0: destruct=0
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:07:00.0
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0039030 qid=0x0001 qp_destroy_elapsed=45003us res_destroy_elapsed=113us
9/11/2025 10:09:24 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010a650 qid=0x0000 qp_destroy_elapsed=37458us res_destroy_elapsed=16us
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme91
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:05:00
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0000, state=6
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0000
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 24 times ***
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139424d20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:05:00.0] Prepare to destruct SSD: 0000:05:00.0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:05:00.0] already in failed state
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:05:00.0]: unregistering thread=app_thread
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:05:00.0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:05:00.0)
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:05:00.0: destruct=1
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139424d20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52e819b010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000d
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d0033cb0, qp_create_elapsed=24751us, res_elapsed=5us
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0033cb0 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139424d20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d0039030 qid=0x0001 ctrlr=0x7f52c8249860 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0039030 qid=0x0001
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d0039030 qid=0x0001 ctrlr=0x7f52c8249860 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139424d20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 ctrlr=0x7f52c8249860 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d010a650 qid=0x0000 ctrlr=0x7f52c8249860 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010a650 qid=0x0000
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d010a650 qid=0x0000 ctrlr=0x7f52c8249860 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139424d20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139424d20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:05:00.0] already in failed state
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:05:00.0)
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:05:00.0: destruct=1
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139424d20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139424d20, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0001
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013ab68ca0 was disconnected and freed. reset controller.
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0004, state=6
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0003, state=6
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0002, state=6
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0001, state=6
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0000, state=5
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013ab68ca0, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013ab68aa0, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013ab688a0, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013ab686a0, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x20013ab68ca0, qid=0x01: Aborting trackers
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x20013ab68aa0, qid=0x02: Aborting trackers
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x20013ab688a0, qid=0x03: Aborting trackers
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x20013ab686a0, qid=0x04: Aborting trackers
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0004
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0003
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0002
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x20013ab68ca0, qid=0x0001: enter
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x20013ab68aa0, qid=0x0002: enter
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x20013ab688a0, qid=0x0003: enter
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x20013ab686a0, qid=0x0004: enter
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:05:00.0)
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:05:00.0: destruct=0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:05:00.0
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013ab68aa0 was disconnected and freed. reset controller.
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013ab688a0 was disconnected and freed. reset controller.
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013ab686a0 was disconnected and freed. reset controller.
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00385f0 qid=0x0001 qp_destroy_elapsed=49648us res_destroy_elapsed=113us
9/11/2025 10:09:23 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f71b0 qid=0x0000 qp_destroy_elapsed=38745us res_destroy_elapsed=16us
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme89
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:04:00
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0000, state=6
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0000
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 25 times ***
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013eb46360, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:04:00.0] Prepare to destruct SSD: 0000:04:00.0
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:04:00.0] already in failed state
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:04:00.0]: unregistering thread=app_thread
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:04:00.0
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:04:00.0)
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:04:00.0: destruct=1
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013eb46360, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52cc19d450 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00f3910, qp_create_elapsed=24085us, res_elapsed=9us
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013eb46360, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f3910 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013eb46360, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00385f0 qid=0x0001 ctrlr=0x7f52e819b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00385f0 qid=0x0001
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00385f0 qid=0x0001 ctrlr=0x7f52e819b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013eb46360, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 ctrlr=0x7f52e819b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f71b0 qid=0x0000 ctrlr=0x7f52e819b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f71b0 qid=0x0000
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f71b0 qid=0x0000 ctrlr=0x7f52e819b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013eb46360, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:04:00.0] already in failed state
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00, event: type=0002, info=0000, log page id=0004
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:04:00.0)
9/11/2025 10:09:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:04:00.0: destruct=1
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013eb46360, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013eb46360, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x200139bfec60, qid=0x04: Aborting trackers
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x20013450e1a0, qid=0x03: Aborting trackers
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x200132579ee0, qid=0x02: Aborting trackers
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0004
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0003
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0002
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0001
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139bfec60 was disconnected and freed. reset controller.
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013450e1a0 was disconnected and freed. reset controller.
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132579ee0 was disconnected and freed. reset controller.
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200131a4b120 was disconnected and freed. reset controller.
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0004, state=6
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0003, state=6
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0002, state=6
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0001, state=6
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0000, state=5
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x200139bfec60, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013450e1a0, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x200132579ee0, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x200131a4b120, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x200131a4b120, qid=0x01: Aborting trackers
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x200139bfec60, qid=0x0004: enter
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x20013450e1a0, qid=0x0003: enter
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x200132579ee0, qid=0x0002: enter
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x200131a4b120, qid=0x0001: enter
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:04:00.0)
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:04:00.0: destruct=0
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:04:00.0
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:09:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 10:09:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 10:08:30 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/11/2025 10:08:30 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/11/2025 10:08:27 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme101n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/11/2025 10:08:27 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme101n1 from latency reports
9/11/2025 10:08:27 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme101n1
9/11/2025 10:08:27 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 11
9/11/2025 10:08:27 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/11/2025 10:08:27 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme101n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199 reason:subsystem_reset
9/11/2025 10:08:27 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme101n1 was hot removed
9/11/2025 10:08:27 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010c990 qid=0x0001 qp_destroy_elapsed=50167us res_destroy_elapsed=112us
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme101
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f9af0 qid=0x0000 qp_destroy_elapsed=38597us res_destroy_elapsed=17us
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0000, state=6
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 27 times ***
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200142063ee0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:c8:00.0]: unregistering thread=app_thread
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:c8:00.0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200142063ee0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f1ba0 qid=0x0000 ctrlr=0x7f52c81a0920 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0033
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0033
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00f1ba0, qp_create_elapsed=24176us, res_elapsed=8us
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200142063ee0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f1ba0 Final Negotiated Queue Depth=32 R/W Depth=16
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200142063ee0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d010c990 qid=0x0001 ctrlr=0x7f52d4216af0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0032
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010c990 qid=0x0001
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d010c990 qid=0x0001 ctrlr=0x7f52d4216af0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0032
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200142063ee0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52d4216af0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0032
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f9af0 qid=0x0000 ctrlr=0x7f52d4216af0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0032
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f9af0 qid=0x0000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f9af0 qid=0x0000 ctrlr=0x7f52d4216af0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0032
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 6 times ***
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200142063ee0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0032, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200142063ee0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:373 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:373 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ABORT (08) qid:0 cid:382 nsid:0 cdw10:037f0002 cdw11:00000000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:693:nvme_pcie_qpair_complete_tracker !!!! [0000:c8:00.0]: Completing abort: qpair=0x200142063ee0, qpid=0, req=0x20013bf6c300
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app ctrlr.c:4675:_nvmf_request_complete !!! Abort response: subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, sqid=0x0001, cid=0x0000: sct/sc=0x00:0x00, cdw0=0x0000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200142063ee0, qid=0 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0002, state=6
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:2 cid:895 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa8fe60, qid=0x01: Aborting trackers
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa8fc60, qid=0x02: Aborting trackers
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x2001325d81a0, qid=0x04: Aborting trackers
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0004
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0003
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0002
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0001
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa8fc60, qid=0x0002: enter
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fa8fe60 was disconnected and freed. reset controller.
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fa8fc60 was disconnected and freed. reset controller.
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013944de60 was disconnected and freed. reset controller.
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001325d81a0 was disconnected and freed. reset controller.
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0002
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0004, state=6
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0003, state=6
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0001, state=6
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa8fe60, qid=1 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa8fc60, qid=2 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013944de60, qid=3 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x2001325d81a0, qid=4 ctrlr is removed - aborting reqs
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c264000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013944de60, qid=0x03: Aborting trackers
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa8fe60, qid=0x0001: enter
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013944de60, qid=0x0003: enter
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x2001325d81a0, qid=0x0004: enter
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:c8:00.0
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0004
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0003
9/11/2025 10:08:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0001
9/11/2025 10:08:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1959:spdk_nvme_ctrlr_reset_subsystem [0000:c8:00.0] !!! After nssr rc=0
9/11/2025 10:08:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1955:spdk_nvme_ctrlr_reset_subsystem [0000:c8:00.0] !!! Before nssr
9/11/2025 10:08:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1947:spdk_nvme_ctrlr_reset_subsystem [0000:c8:00.0] !!! Resetting subsystem for ctrlr=0000:c8:00.0, cntlid=0x0081
9/11/2025 10:08:23 PM 4070 I nvmf_tgt_app nvme.c:548:nvme_request_check_timeout !!! Timeout expired for nvme req=0x20013806ee40, opc=0x02, qpair=0x20013fa8fc60, type=IO
9/11/2025 10:08:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:2774:bdev_nvme_subsystem_reset !!! Initiating subsystem reset for: 0000:c8:00.0
9/11/2025 10:08:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:2764:_nvme_bdev_ctrlr_subsystem_reset !!! Subsystem reset rc=0
9/11/2025 10:08:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:2759:_nvme_bdev_ctrlr_subsystem_reset !!! [app_thread] Starting subsystem reset
9/11/2025 10:08:23 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 10:08:15 PM 4070 I nvmf_tgt_app nvme_ctrlr_cmd.c:597:_nvme_ctrlr_submit_abort_request !!! Submitting abort for: 0000:c8:00.0, req=0x20013bf6c300, outstanding_aborts=1
9/11/2025 10:08:15 PM 4070 I nvmf_tgt_app ctrlr.c:3532:nvmf_ctrlr_abort_request !!! Abort request: subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, sqid=0x0001, cid=0x0000 opc=0x02, lba=0, len=1, qid=0x0001, hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021
9/11/2025 10:08:15 PM 4070 I nvmf_tgt_app ctrlr.c:3500:nvmf_ctrlr_abort !!! [nvmf_tgt_poll_group_2] Abort request: hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021: subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, sqid=0x0001, cid=0x0000
9/11/2025 10:08:15 PM 4070 I nvmf_tgt_app bdev_nvme.c:8155:bdev_nvme_abort !!! [0000:c8:00.0]: qpair.id=2: bio=0x20025bd2bda0, bio_to_abort=0x20025bd2b860
9/11/2025 10:08:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 10:06:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 10:04:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 10:02:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 10:02:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 10:01:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 9:59:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 9:58:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:58:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 9:57:27 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 9:57:27 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":1038367574, "rx_prio4_bytes_delta":5249668, "tx_prio4_bytes":6207176896, "tx_prio4_bytes_delta":42140688, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":10136006, "rx_prio4_packets_delta":57853, "tx_prio4_packets":11237115, "tx_prio4_packets_delta":65711}
9/11/2025 9:57:27 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 9:57:27 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 9:57:27 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 9:56:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:56:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 9:55:27 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 9:54:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:54:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:51:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:51:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:49:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:49:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:46:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:46:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:46:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:46:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:44:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:44:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:42:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 9:42:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 9:41:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:41:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:39:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:39:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:36:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:36:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:34:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:34:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 9:32:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 9:32:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 9:31:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:31:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:30:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:30:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:29:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:29:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:28:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 9:28:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 9:26:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:26:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 9:26:26 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 9:26:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 9:24:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:24:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:24:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:24:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:21:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:21:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:20:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 9:20:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 9:19:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:19:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:18:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 9:18:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 9:16:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:16:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:14:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:14:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:12:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 9:12:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 9:11:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:11:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:09:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:09:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:06:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:06:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:04:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:04:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:02:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 9:02:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 9:01:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 9:01:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 9:00:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 9:00:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 8:59:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 8:59:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 8:58:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 8:58:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 8:57:26 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 8:57:26 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":1033117906, "rx_prio4_bytes_delta":5189298, "tx_prio4_bytes":6165036208, "tx_prio4_bytes_delta":39709208, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":10078153, "rx_prio4_packets_delta":57464, "tx_prio4_packets":11171404, "tx_prio4_packets_delta":64783}
9/11/2025 8:57:26 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 8:57:26 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 8:57:26 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 8:56:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 8:56:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 8:55:26 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 8:54:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:54:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:54:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 8:54:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 8:51:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:51:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 8:50:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 8:50:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 8:49:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 8:49:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 8:48:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:48:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:46:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:46:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:44:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 8:44:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 8:41:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:41:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:40:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 8:40:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 8:39:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:39:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 8:36:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 8:36:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 8:34:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:34:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:32:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 8:32:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 8:31:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 8:31:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 8:29:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:29:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:28:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 8:28:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 8:26:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:26:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:24:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:24:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:22:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:22:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:21:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:21:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:20:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:20:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:19:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:19:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:16:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:16:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:16:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 8:16:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 8:14:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 8:14:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:14:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:11:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:11:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:10:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 8:09:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:09:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 8:06:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:06:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 8:04:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:04:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 8:01:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:01:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 7:59:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:59:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 7:58:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 7:58:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 7:57:25 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 7:57:25 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":1027928608, "rx_prio4_bytes_delta":5152400, "tx_prio4_bytes":6125327000, "tx_prio4_bytes_delta":38782202, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":10020689, "rx_prio4_packets_delta":56911, "tx_prio4_packets":11106621, "tx_prio4_packets_delta":64126}
9/11/2025 7:57:25 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 7:57:25 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 7:57:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 7:56:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:56:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 7:55:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 37 times ***
9/11/2025 7:54:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 7:51:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:51:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 7:49:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:49:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 7:46:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:46:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 7:44:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:41:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 7:39:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:36:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 7:34:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 7:34:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 7:34:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:30:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 7:29:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:26:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 7:26:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 7:26:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 7:16:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 7:16:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:16:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 7:14:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 7:14:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 7:11:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 7:09:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:09:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 7:08:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:06:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 7:04:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:01:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 7:00:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 7:00:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 6:59:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 6:57:24 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 6:57:24 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":1022776208, "rx_prio4_bytes_delta":5194514, "tx_prio4_bytes":6086544798, "tx_prio4_bytes_delta":38745706, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9963778, "rx_prio4_packets_delta":57378, "tx_prio4_packets":11042495, "tx_prio4_packets_delta":64548}
9/11/2025 6:57:24 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 6:57:24 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 6:57:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 6:54:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:54:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 6:52:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 6:52:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:51:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:51:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:49:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:46:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:46:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:44:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:44:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:41:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:41:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:39:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:39:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:36:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:34:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:34:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 6:32:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 6:32:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:31:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:31:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:29:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:29:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 6:28:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:28:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 6:26:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:26:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:24:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:24:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:24:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:24:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:21:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:21:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 6:20:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 6:20:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:19:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:19:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 6:16:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:16:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:14:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:14:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 6:11:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:11:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 6:09:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:09:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 6:08:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 6:08:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:06:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:06:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 6:06:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 6:04:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:04:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 6:04:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 6:02:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 6:02:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 6:01:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:01:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 5:59:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:59:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 5:58:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 5:58:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 5:57:24 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 5:57:24 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":1017581694, "rx_prio4_bytes_delta":5205290, "tx_prio4_bytes":6047799092, "tx_prio4_bytes_delta":39730086, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9906400, "rx_prio4_packets_delta":57481, "tx_prio4_packets":10977947, "tx_prio4_packets_delta":64893}
9/11/2025 5:57:24 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 5:57:24 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 5:57:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 5:56:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 5:56:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 5:55:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 5:54:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:54:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 5:54:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 5:51:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:51:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 5:49:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:49:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 5:46:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:46:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 5:46:33 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 5:46:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 5:44:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:44:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 5:42:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:42:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:41:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:41:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 5:40:58 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 5:39:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:39:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 5:38:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 5:38:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:36:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:36:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 5:34:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 5:34:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:34:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 5:31:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:31:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 5:30:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 5:30:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:29:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:29:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 5:26:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:26:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 5:24:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:24:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 5:21:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:21:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 5:20:33 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 5:19:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:19:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 5:18:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:18:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 5:16:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:16:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 5:16:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 5:14:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:14:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 5:12:33 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 5:12:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 5:11:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:11:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 5:09:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:09:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 5:06:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:06:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 5:04:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 5:04:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 5:04:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 5:01:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:01:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 4:59:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:59:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:57:23 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 4:57:23 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":1012376404, "rx_prio4_bytes_delta":5236910, "tx_prio4_bytes":6008069006, "tx_prio4_bytes_delta":41825240, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9848919, "rx_prio4_packets_delta":57707, "tx_prio4_packets":10913054, "tx_prio4_packets_delta":65548}
9/11/2025 4:57:23 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 4:57:23 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 4:57:23 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 4:56:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 4:56:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 4:56:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:56:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:54:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:52:33 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 4:52:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 4:51:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:49:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:46:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:46:33 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:44:33 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 4:44:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 4:44:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:41:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:41:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 4:40:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:40:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 4:39:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:39:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 4:38:33 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 4:38:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 4:36:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:36:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 4:36:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:34:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 4:34:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 4:34:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 4:31:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:29:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:29:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 4:26:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:24:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:24:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 4:22:33 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:22:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 4:21:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:21:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 4:19:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:19:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 4:18:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:18:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 4:16:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:16:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 4:14:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 4:14:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 4:11:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:11:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 4:10:33 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 4:10:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 4:09:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 4:09:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 4:06:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:06:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 4:06:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:06:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 4:04:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 4:04:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 4:01:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 4:01:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 3:59:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 3:59:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 3:57:22 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 3:57:22 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":1007139494, "rx_prio4_bytes_delta":5224480, "tx_prio4_bytes":5966243766, "tx_prio4_bytes_delta":40887586, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9791212, "rx_prio4_packets_delta":57587, "tx_prio4_packets":10847506, "tx_prio4_packets_delta":65246}
9/11/2025 3:57:22 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 3:57:22 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 3:57:22 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 3:56:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 3:56:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 3:55:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 37 times ***
9/11/2025 3:54:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 3:54:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:54:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:51:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 3:51:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 3:50:27 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 3:49:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:49:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:46:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:46:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 3:44:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:44:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:41:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:41:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:39:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:39:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 3:36:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:36:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:34:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:34:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 3:31:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:31:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:29:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:29:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:28:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 3:26:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:26:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:24:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:24:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:21:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:21:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:19:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:19:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:16:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:16:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 3:16:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:16:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 3:14:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:14:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:12:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 3:12:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 3:11:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:11:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 3:09:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:09:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:06:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:06:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 3:04:27 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:04:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 3:04:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 3:02:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:02:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 3:01:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:01:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 2:59:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 2:59:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:57:22 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 2:57:22 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":1001915014, "rx_prio4_bytes_delta":5188088, "tx_prio4_bytes":5925356180, "tx_prio4_bytes_delta":40796296, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9733625, "rx_prio4_packets_delta":57196, "tx_prio4_packets":10782260, "tx_prio4_packets_delta":64824}
9/11/2025 2:57:22 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 2:57:22 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 2:57:22 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 2:56:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 2:56:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 2:55:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 37 times ***
9/11/2025 2:54:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:54:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:51:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 2:51:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 2:50:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 2:50:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 2:49:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:49:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:46:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:46:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 2:44:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:44:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:42:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 2:42:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 2:41:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:41:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:39:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:39:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:36:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:36:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:34:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 2:34:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:31:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:31:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:30:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 2:30:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 2:29:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 2:29:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:26:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:26:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 2:26:26 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:24:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 2:24:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 2:21:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:21:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:20:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 2:20:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 2:19:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 2:16:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 2:16:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 2:16:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 2:16:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 2:14:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 2:14:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 2:14:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 2:11:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 2:09:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 2:08:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 2:08:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 2:06:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 2:06:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 2:04:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:04:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:04:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 2:01:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 1:57:21 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 1:57:21 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":996726926, "rx_prio4_bytes_delta":5257068, "tx_prio4_bytes":5884559884, "tx_prio4_bytes_delta":42633030, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9676429, "rx_prio4_packets_delta":57869, "tx_prio4_packets":10717436, "tx_prio4_packets_delta":65848}
9/11/2025 1:57:21 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 1:57:21 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 1:56:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 1:56:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 1:56:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 1:55:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 1:54:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 1:52:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:52:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 1:46:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 1:46:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 1:46:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 1:41:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:41:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 1:40:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:40:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 1:31:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 1:30:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 1:30:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:29:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:26:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 1:25:56 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 1:24:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 1:24:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 1:24:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 1:22:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 1:22:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:19:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 1:18:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 1:18:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 1:16:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:16:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 1:16:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 1:16:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:14:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:14:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 1:12:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 1:12:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:11:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:11:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 1:09:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:09:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 1:08:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:08:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 1:06:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:06:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 1:06:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 1:06:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:04:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 1:04:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 1:02:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:02:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 1:01:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:01:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 1:00:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 1:00:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 12:59:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 12:59:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 12:57:20 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 12:57:20 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":991469858, "rx_prio4_bytes_delta":5261818, "tx_prio4_bytes":5841926854, "tx_prio4_bytes_delta":43152816, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9618560, "rx_prio4_packets_delta":57922, "tx_prio4_packets":10651588, "tx_prio4_packets_delta":66073}
9/11/2025 12:57:20 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 12:57:20 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 12:56:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 12:56:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 12:56:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 12:56:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 12:55:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 12:54:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 12:54:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 12:51:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:51:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 12:50:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:50:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:49:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 12:49:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 12:46:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:46:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 12:44:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 12:44:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 12:41:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:41:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 12:39:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 12:39:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 12:38:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 12:36:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:36:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 12:34:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:34:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:34:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 12:31:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:31:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 12:29:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:29:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 12:28:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 12:26:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 12:26:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 12:26:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 12:26:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 12:24:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:24:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:24:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 12:21:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:21:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 12:20:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 12:20:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 12:19:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:19:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 12:18:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 12:18:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 12:16:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:16:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 12:16:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 12:14:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:14:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 12:12:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 12:12:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:11:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:11:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 12:10:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:10:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 12:09:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:09:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 12:06:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:06:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 12:04:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:04:07 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 12:01:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:01:37 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 12:00:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 12:00:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 11:59:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 11:59:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 11:57:20 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 11:57:20 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":986208040, "rx_prio4_bytes_delta":5218318, "tx_prio4_bytes":5798774038, "tx_prio4_bytes_delta":43026262, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9560638, "rx_prio4_packets_delta":57783, "tx_prio4_packets":10585515, "tx_prio4_packets_delta":65873}
9/11/2025 11:57:20 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 11:57:20 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 11:56:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 11:56:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 11:56:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 11:56:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 11:55:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 11:54:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 11:54:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 11:52:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 11:52:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 11:51:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:51:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 11:49:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 11:49:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 11:48:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 11:48:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 11:46:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:46:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 11:44:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 11:44:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 11:44:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 11:42:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 11:41:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:41:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 11:39:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:39:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 11:38:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 11:38:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 11:36:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:34:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 11:31:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:29:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:29:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 11:26:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:26:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 11:26:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 11:24:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:21:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:21:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 11:20:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 11:19:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:18:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 11:18:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 11:16:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 11:16:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 11:16:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 11:16:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 11:14:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 11:14:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 11:11:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 11:09:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:09:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 11:08:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 11:08:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 11:06:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 11:06:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 11:04:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 11:01:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 11:01:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 10:59:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 10:57:19 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 10:57:19 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":980989722, "rx_prio4_bytes_delta":5225762, "tx_prio4_bytes":5755747776, "tx_prio4_bytes_delta":40666530, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9502855, "rx_prio4_packets_delta":57613, "tx_prio4_packets":10519642, "tx_prio4_packets_delta":65227}
9/11/2025 10:57:19 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 10:57:19 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 10:56:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 10:56:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 10:54:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:54:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 10:51:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:51:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 10:49:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:49:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 10:46:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:46:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 10:46:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 10:46:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 10:44:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:44:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 10:42:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 10:42:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 10:41:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:41:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 10:39:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:39:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 10:38:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 10:38:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:36:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:36:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 10:34:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 10:34:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 10:34:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:34:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 10:31:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:31:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 10:30:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 10:30:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 10:29:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:29:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 10:26:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:26:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 10:26:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 10:26:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 10:24:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 10:24:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 10:24:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:24:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 10:21:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:21:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 10:20:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:20:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 10:19:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:19:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 10:16:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:16:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 10:14:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:14:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 10:12:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 10:12:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 10:11:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:11:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 10:09:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:09:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 10:06:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:06:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 10:04:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:04:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 10:01:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 10:01:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 9:59:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 9:59:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 9:57:18 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 9:57:18 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":975763960, "rx_prio4_bytes_delta":5210398, "tx_prio4_bytes":5715081246, "tx_prio4_bytes_delta":43428500, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9445242, "rx_prio4_packets_delta":57435, "tx_prio4_packets":10454415, "tx_prio4_packets_delta":65618}
9/11/2025 9:57:18 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 9:57:18 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 9:56:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 9:56:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 9:56:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 9:56:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 9:54:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 9:54:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 9:51:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 9:51:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 9:49:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 9:49:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 9:48:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 9:48:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 9:46:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 9:46:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 9:44:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 9:44:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 9:44:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 9:44:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 9:42:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:42:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 9:41:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:41:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 9:40:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 9:39:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 9:39:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 9:38:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 9:38:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 9:36:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:36:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 9:34:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 9:34:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:34:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 9:31:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:31:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 9:29:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:29:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 9:28:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:28:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 9:26:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:26:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 9:24:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:24:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 9:22:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 9:22:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 9:21:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:21:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 9:20:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 9:19:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:19:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 9:18:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 9:18:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 9:16:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:16:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 9:16:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 9:14:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:14:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 9:11:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:11:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 9:10:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 9:10:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 9:09:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:09:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 9:06:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:06:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 9:06:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 9:06:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 9:04:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 9:04:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 9:01:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 9:01:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 8:59:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 8:59:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 8:57:18 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 8:57:18 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":970553562, "rx_prio4_bytes_delta":5266780, "tx_prio4_bytes":5671652746, "tx_prio4_bytes_delta":43155340, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9387807, "rx_prio4_packets_delta":57990, "tx_prio4_packets":10388797, "tx_prio4_packets_delta":66094}
9/11/2025 8:57:18 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 8:57:18 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 8:56:48 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:56:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 8:56:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 8:56:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 8:54:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 8:54:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 8:51:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 8:50:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 8:50:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 8:49:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 8:48:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 8:48:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:46:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 8:46:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 8:44:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 8:44:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 8:36:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 8:36:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 8:36:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 8:26:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 8:24:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 8:24:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 8:21:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 8:20:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 8:20:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 8:16:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 8:14:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 8:14:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 8:11:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 8:10:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 8:06:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 8:06:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 8:06:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 8:04:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 8:04:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 8:02:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 8:02:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 7:59:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 7:59:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 7:58:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 7:58:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 7:57:17 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 7:57:17 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":965286782, "rx_prio4_bytes_delta":5185060, "tx_prio4_bytes":5628497406, "tx_prio4_bytes_delta":38808372, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9329817, "rx_prio4_packets_delta":57275, "tx_prio4_packets":10322703, "tx_prio4_packets_delta":64539}
9/11/2025 7:57:17 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 7:57:17 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 7:57:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 7:56:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:56:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:56:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 7:55:12 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:54:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 7:54:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 7:54:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 7:51:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 7:51:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 7:50:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:49:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:49:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 7:46:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 7:46:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 7:44:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:44:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 7:41:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 7:41:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 7:39:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:39:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 7:36:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:36:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 7:36:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 7:36:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 7:34:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:34:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 7:32:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 7:32:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:31:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:31:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 7:29:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:29:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 7:26:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:26:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 7:24:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 7:24:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 7:24:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 7:21:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:21:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 7:20:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 7:19:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 7:19:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 7:16:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:16:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 7:14:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:14:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 7:12:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 7:12:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 7:11:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 7:11:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 7:09:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 7:09:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 7:08:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 7:08:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 7:06:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 7:06:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 7:04:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 7:04:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 7:01:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 7:01:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 6:59:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:59:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 6:57:16 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 6:57:16 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":960101722, "rx_prio4_bytes_delta":5265192, "tx_prio4_bytes":5589689034, "tx_prio4_bytes_delta":43045526, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9272542, "rx_prio4_packets_delta":57964, "tx_prio4_packets":10258164, "tx_prio4_packets_delta":66025}
9/11/2025 6:57:16 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 6:57:16 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 6:57:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 6:56:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 6:56:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 6:56:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 6:55:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 6:54:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 6:54:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 6:51:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:51:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 6:49:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 6:49:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/11/2025 6:48:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:48:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 6:46:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:46:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 6:46:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 6:46:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 6:44:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 6:44:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 6:42:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 6:42:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 6:41:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:41:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 6:40:26 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 6:40:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 6:39:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:39:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 6:38:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 6:38:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:36:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:36:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 6:34:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:34:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 6:34:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 6:31:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:31:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 6:29:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:29:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 6:26:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:26:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 6:26:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 6:26:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:24:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 6:24:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 6:21:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:19:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:19:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 6:18:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 6:18:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 6:16:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:16:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 6:16:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 6:16:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 6:14:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 6:14:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 6:14:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 6:12:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 6:12:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 6:11:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:11:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 6:10:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 6:09:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:06:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 6:06:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 6:04:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 6:04:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 6:01:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:59:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 5:59:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 5:57:16 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 5:57:16 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":954836530, "rx_prio4_bytes_delta":5238874, "tx_prio4_bytes":5546643508, "tx_prio4_bytes_delta":41825656, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9214578, "rx_prio4_packets_delta":57735, "tx_prio4_packets":10192139, "tx_prio4_packets_delta":65554}
9/11/2025 5:57:16 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 5:57:16 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 5:57:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 5:56:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 5:56:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 5:55:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 5:54:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:54:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 5:51:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:51:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 5:49:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:49:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 5:48:35 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 5:48:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 5:46:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:46:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 5:44:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:44:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 5:42:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:42:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:41:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:41:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 5:39:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:39:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 5:38:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 5:36:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:36:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 5:34:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:34:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 5:32:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:32:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 5:31:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:31:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 5:30:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 5:30:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 5:29:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:29:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 5:26:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 5:26:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 5:26:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 5:26:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 5:24:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:24:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 5:24:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 5:21:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:21:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 5:19:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:19:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 5:16:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:16:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 5:16:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 5:16:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 5:14:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 5:14:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 5:14:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 5:11:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:11:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 5:10:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 5:10:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 5:09:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 5:09:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:09:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 5:08:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 5:08:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 5:06:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:06:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 5:04:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 5:04:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/11/2025 5:04:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 5:01:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 5:01:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 5:00:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 5:00:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 4:59:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 4:59:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 4:57:15 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 4:57:15 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":949597656, "rx_prio4_bytes_delta":5198552, "tx_prio4_bytes":5504817852, "tx_prio4_bytes_delta":41888842, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9156843, "rx_prio4_packets_delta":57285, "tx_prio4_packets":10126585, "tx_prio4_packets_delta":65120}
9/11/2025 4:57:15 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 4:57:15 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 4:57:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 4:56:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 4:56:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 4:55:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 4:54:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 4:54:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 4:52:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 4:52:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 4:51:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 4:51:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 4:49:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:49:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 4:48:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 4:46:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 4:46:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 4:46:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 4:46:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 4:44:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:44:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 4:42:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 4:42:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 4:41:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 4:41:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 4:39:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:39:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 4:36:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:36:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/11/2025 4:34:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:34:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 4:31:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:31:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 4:30:30 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 4:30:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 4:29:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:29:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 4:26:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:26:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 4:26:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 4:26:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 4:24:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 4:24:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 4:21:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:21:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 4:19:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:19:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 4:18:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 4:18:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 4:16:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:16:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 4:14:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:14:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 4:11:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 4:11:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 4:09:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:09:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/11/2025 4:08:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 4:08:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 4:06:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 4:06:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 4:06:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 4:06:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 4:04:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 4:04:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 4:02:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 4:02:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 4:01:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 4:01:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 3:59:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:59:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 3:57:14 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 3:57:14 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":944399104, "rx_prio4_bytes_delta":5223622, "tx_prio4_bytes":5462929010, "tx_prio4_bytes_delta":40505012, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9099558, "rx_prio4_packets_delta":57614, "tx_prio4_packets":10061465, "tx_prio4_packets_delta":65191}
9/11/2025 3:57:14 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 3:57:14 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 3:57:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 3:56:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 3:56:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/11/2025 3:56:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 3:55:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 3:54:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 3:54:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 3:52:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 3:52:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 3:51:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 3:51:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 3:49:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 3:46:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 3:46:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 3:46:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 3:44:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 3:44:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 3:44:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 3:41:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 3:40:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 3:40:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 3:39:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 3:39:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 3:38:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 3:38:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 3:36:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 3:34:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 3:34:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 3:32:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 3:32:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 3:31:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 3:30:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 3:30:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 3:26:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 3:26:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 3:26:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:19:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/11/2025 3:18:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 3:18:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 3:11:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 3:10:29 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 3:10:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 3:09:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 2:59:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 2:57:14 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 2:57:14 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":939175482, "rx_prio4_bytes_delta":5208326, "tx_prio4_bytes":5422423998, "tx_prio4_bytes_delta":39748340, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":9041944, "rx_prio4_packets_delta":57485, "tx_prio4_packets":9996274, "tx_prio4_packets_delta":64872}
9/11/2025 2:57:14 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 2:57:14 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 2:57:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 2:56:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 2:56:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 2:56:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 2:55:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:54:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:54:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:51:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:51:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 2:50:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 2:50:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 2:49:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:49:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 2:48:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:48:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 2:44:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:44:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:42:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 2:42:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 2:41:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:41:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:40:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 2:40:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 2:39:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:39:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 2:36:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:36:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 2:36:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 2:34:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:34:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 2:31:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:31:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 2:29:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:29:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 2:26:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:26:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:26:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 2:26:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 2:24:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:24:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 2:21:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/11/2025 2:21:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 2:19:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:19:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 2:16:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:16:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 2:14:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:14:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 2:12:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 2:12:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 2:11:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:11:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 2:10:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 2:10:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 2:09:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:09:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 2:06:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:06:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 2:06:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 2:06:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 2:04:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 2:04:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 2:04:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/11/2025 2:01:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 2:01:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 2:00:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 2:00:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 1:59:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:59:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 1:58:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/11/2025 1:57:13 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 1:57:13 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":933967156, "rx_prio4_bytes_delta":5211048, "tx_prio4_bytes":5382675658, "tx_prio4_bytes_delta":39661024, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8984459, "rx_prio4_packets_delta":57496, "tx_prio4_packets":9931402, "tx_prio4_packets_delta":64892}
9/11/2025 1:57:13 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 1:57:13 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 1:57:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 1:56:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:56:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/11/2025 1:55:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 1:54:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 1:54:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/11/2025 1:54:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 1:52:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 1:52:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/11/2025 1:51:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 1:51:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:49:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:49:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:46:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 1:46:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:44:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 1:44:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 1:42:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 1:42:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 1:41:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 1:41:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:39:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 1:39:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:36:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:36:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:34:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 1:34:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:31:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:31:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 1:30:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 1:30:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 1:29:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 1:29:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:26:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:26:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 1:24:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/11/2025 1:24:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:21:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:21:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 1:19:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:19:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/11/2025 1:16:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:16:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 1:14:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/11/2025 1:14:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 1:14:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 1:11:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:11:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 1:10:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/11/2025 1:10:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 1:09:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:09:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 1:06:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:06:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 1:06:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/11/2025 1:06:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/11/2025 1:04:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:04:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/11/2025 1:02:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 1:01:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 1:01:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 12:59:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:59:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:57:12 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/11/2025 12:57:12 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":928756108, "rx_prio4_bytes_delta":5195358, "tx_prio4_bytes":5343014634, "tx_prio4_bytes_delta":41817528, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8926963, "rx_prio4_packets_delta":57592, "tx_prio4_packets":9866510, "tx_prio4_packets_delta":65441}
9/11/2025 12:57:12 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 12:57:12 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/11/2025 12:57:12 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/11/2025 12:56:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:56:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:291:DeleteOldReports Removed old latency report file /var/log/tpd/nvme_latency//BdevLatency_1749255033670196035_4210.report
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme99n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme99n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme98n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme98n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme97n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme97n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme96n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme96n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme94n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme94n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme93n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme93n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme92n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme92n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme91n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme91n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme90n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme90n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme89n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme89n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme101n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme101n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme100n1 ENABLED, Status NVG_SUCCESS
9/11/2025 12:55:07 AM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme100n1 DISABLED, Status NVG_SUCCESS
9/11/2025 12:54:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:54:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 12:51:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:49:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:49:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/11/2025 12:48:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:48:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 12:46:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:46:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:44:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/11/2025 12:44:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/11/2025 12:44:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 12:41:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:41:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 12:40:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:40:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:39:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:39:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 12:36:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:36:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 12:36:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:36:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:34:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:34:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 12:31:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:31:40 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 12:29:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:29:10 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 12:26:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:26:40 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 12:26:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:26:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:24:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:24:10 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 12:22:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:22:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:21:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:21:40 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 12:19:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:19:10 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 12:16:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:16:40 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 12:14:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/11/2025 12:14:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/11/2025 12:14:10 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 12:11:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:11:40 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/11/2025 12:09:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:09:10 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/11/2025 12:06:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:06:40 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 12:06:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/11/2025 12:06:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/11/2025 12:04:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:04:10 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/11/2025 12:02:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/11/2025 12:01:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/11/2025 12:01:40 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/10/2025 11:59:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 11:59:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 11:57:11 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 11:57:11 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":923560750, "rx_prio4_bytes_delta":5225564, "tx_prio4_bytes":5301197106, "tx_prio4_bytes_delta":40900210, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8869371, "rx_prio4_packets_delta":57638, "tx_prio4_packets":9801069, "tx_prio4_packets_delta":65293}
9/10/2025 11:57:11 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 11:57:11 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_error":31408, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 11:57:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/10/2025 11:56:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/10/2025 11:56:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 11:55:26 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 11:54:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/10/2025 11:54:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 11:54:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 11:51:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 11:51:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/10/2025 11:50:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 11:49:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:49:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 11:46:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 11:46:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/10/2025 11:44:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 11:44:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 11:44:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 11:41:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 11:41:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/10/2025 11:40:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 11:40:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/10/2025 11:39:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:39:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/10/2025 11:38:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 11:38:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:36:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 11:36:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 11:34:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:34:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/10/2025 11:32:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/10/2025 11:32:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 11:31:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:31:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 11:29:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:29:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 11:26:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:26:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/10/2025 11:24:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/10/2025 11:24:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 11:21:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:21:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 11:19:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 11:19:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 11:16:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:16:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/10/2025 11:16:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 11:14:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 11:14:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 11:12:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 11:12:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/10/2025 11:11:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:11:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 11:09:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:09:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 11:06:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 11:06:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 11:04:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 11:04:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/10/2025 11:02:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 11:01:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 11:01:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 11:00:26 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 11:00:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 10:59:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 10:59:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 10:57:11 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 10:57:11 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":918335186, "rx_prio4_bytes_delta":20330304, "tx_prio4_bytes":5260296896, "tx_prio4_bytes_delta":64628860, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8811733, "rx_prio4_packets_delta":156264, "tx_prio4_packets":9735776, "tx_prio4_packets_delta":164526}
9/10/2025 10:57:11 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 10:57:11 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":31100, "resp_cqe_flush_error_delta":4535, "resp_cqe_error":31408, "resp_cqe_error_delta":4580, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 10:57:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 10:56:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 10:56:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 10:55:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 10:54:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 10:54:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 10:51:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 10:51:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 10:50:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/10/2025 10:50:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 10:49:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 10:49:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 10:46:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 10:46:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 10:44:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 10:44:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 10:42:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/10/2025 10:42:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 19 times ***
9/10/2025 10:41:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 10:39:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 10:39:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 10:38:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 10:38:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 10:36:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 10:34:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 10:34:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 10:32:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 10:32:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/10/2025 10:31:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 10:30:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 10:30:23 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f1ba0 qid=0x0000 qp_destroy_elapsed=52428us res_destroy_elapsed=15us
9/10/2025 10:30:23 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_2]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d010c990 qid=0x0001 ctrlr=0x7f52d4216af0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0032
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d010c990, qp_create_elapsed=24535us, res_elapsed=47us
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010c990 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f9af0 qid=0x0000 ctrlr=0x7f52d4216af0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0032
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0032
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f9af0, qp_create_elapsed=24525us, res_elapsed=7us
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f9af0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52e826ca30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0031
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f1ba0 qid=0x0000 ctrlr=0x7f52e826ca30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0031
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f1ba0 qid=0x0000
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f1ba0 qid=0x0000 ctrlr=0x7f52e826ca30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0031
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme101n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme101n1 for latency reports
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 12
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme101n1 at 0000:c8:00.0 [002538E141500775] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 with nsid=1
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0031, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme101n1 eui64:002538E141500775 sn:S7PLNG0X100199 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme101n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 10:30:22 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:30:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:c8:00.0 (HotInNvme101)
9/10/2025 10:30:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:c8:00.0 name=HotInNvme101 ctrlr=0x20013230a500
9/10/2025 10:30:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme101
9/10/2025 10:30:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme101n1 sn:(S7PLNG0X100199 ) mn:(AELN15T4P5xnEQRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:30:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme95n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme95n1 from latency reports
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme95n1
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 11
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme95n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199 reason:unknown
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme95n1 was hot removed
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f3910 qid=0x0001 qp_destroy_elapsed=49159us res_destroy_elapsed=108us
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010b740 qid=0x0000 qp_destroy_elapsed=37572us res_destroy_elapsed=16us
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme95
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 31 times ***
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013ccd02e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:c8:00.0]: unregistering thread=app_thread
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:c8:00.0
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 10:30:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013ccd02e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f1ba0 qid=0x0000 ctrlr=0x7f52e826ca30 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0031
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0031
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00f1ba0, qp_create_elapsed=25123us, res_elapsed=9us
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013ccd02e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f1ba0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013ccd02e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f3910 qid=0x0001 ctrlr=0x7f52c82af770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0030
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f3910 qid=0x0001
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f3910 qid=0x0001 ctrlr=0x7f52c82af770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0030
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013ccd02e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52c82af770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0030
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d010b740 qid=0x0000 ctrlr=0x7f52c82af770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0030
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010b740 qid=0x0000
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d010b740 qid=0x0000 ctrlr=0x7f52c82af770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0030
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013ccd02e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0030, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013ccd02e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013ccd02e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa8fc60, qid=0x03: Aborting trackers
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0003
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0002
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fa8fe60 was disconnected and freed. reset controller.
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fa8fc60 was disconnected and freed. reset controller.
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200142063ee0, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa8fe60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa8fc60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013944de60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x200142063ee0, qid=0x01: Aborting trackers
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa8fe60, qid=0x02: Aborting trackers
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013944de60, qid=0x04: Aborting trackers
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0004
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0001
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x200142063ee0, qid=0x0001: enter
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa8fe60, qid=0x0002: enter
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa8fc60, qid=0x0003: enter
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013944de60, qid=0x0004: enter
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=0
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:c8:00.0
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200142063ee0 was disconnected and freed. reset controller.
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013944de60 was disconnected and freed. reset controller.
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:30:08 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 10:29:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f6780 qid=0x0000 qp_destroy_elapsed=48272us res_destroy_elapsed=18us
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0033cb0 qid=0x0000 qp_destroy_elapsed=49694us res_destroy_elapsed=17us
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_3]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d0106220 qid=0x0001 ctrlr=0x7f52c85f0e30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d0106220, qp_create_elapsed=24855us, res_elapsed=41us
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0106220 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_4]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00f5c20 qid=0x0001 ctrlr=0x7f52cc39d770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f1ba0 qid=0x0000 qp_destroy_elapsed=51816us res_destroy_elapsed=14us
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00f5c20, qp_create_elapsed=25505us, res_elapsed=45us
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f5c20 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0106220 qid=0x0000 qp_destroy_elapsed=39449us res_destroy_elapsed=17us
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_1]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00fa810 qid=0x0001 ctrlr=0x7f52e83a0c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:22 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00fa810, qp_create_elapsed=24410us, res_elapsed=45us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f5c20 qid=0x0000 qp_destroy_elapsed=63170us res_destroy_elapsed=17us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00fa810 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_2]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d0038c80 qid=0x0001 ctrlr=0x7f52d44bf640 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d0038c80, qp_create_elapsed=24127us, res_elapsed=38us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0038c80 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00fa810 qid=0x0000 qp_destroy_elapsed=57219us res_destroy_elapsed=13us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_3]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f3910 qid=0x0001 ctrlr=0x7f52c82af770 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0030
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f3910, qp_create_elapsed=24605us, res_elapsed=42us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f3910 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0038c80 qid=0x0000 qp_destroy_elapsed=37084us res_destroy_elapsed=17us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_1]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00ee990 qid=0x0001 ctrlr=0x7f52e81660e0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00ee990, qp_create_elapsed=24989us, res_elapsed=43us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f3910 qid=0x0000 qp_destroy_elapsed=38236us res_destroy_elapsed=16us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ee990 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_4]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d010c7e0 qid=0x0001 ctrlr=0x7f52cc313b10 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d010c7e0, qp_create_elapsed=23858us, res_elapsed=37us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010c7e0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_2]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d010c420 qid=0x0001 ctrlr=0x7f52d4266450 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d010c420, qp_create_elapsed=23801us, res_elapsed=37us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010c420 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d010c060 qid=0x0000 ctrlr=0x7f52cc39d770 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d010c060, qp_create_elapsed=23917us, res_elapsed=16us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010c060 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 ctrlr=0x7f52c8396010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00f6780 qid=0x0000 ctrlr=0x7f52c8396010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f6780 qid=0x0000
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00f6780 qid=0x0000 ctrlr=0x7f52c8396010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d010bca0 qid=0x0000 ctrlr=0x7f52c85f0e30 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d010bca0, qp_create_elapsed=23767us, res_elapsed=8us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010bca0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme99n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme99n1 for latency reports
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00ee5d0 qid=0x0000 ctrlr=0x7f52d44bf640 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00ee5d0, qp_create_elapsed=24088us, res_elapsed=14us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme98n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme98n1 for latency reports
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ee5d0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme97n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme97n1 for latency reports
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme96n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme96n1 for latency reports
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00ee210 qid=0x0000 ctrlr=0x7f52e81660e0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00ee210, qp_create_elapsed=25122us, res_elapsed=14us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme95n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme95n1 for latency reports
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ee210 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme94n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme94n1 for latency reports
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme93n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme93n1 for latency reports
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00ebf70 qid=0x0000 ctrlr=0x7f52cc313b10 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00ebf70, qp_create_elapsed=24039us, res_elapsed=11us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme100n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme100n1 for latency reports
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ebf70 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 ctrlr=0x7f52cc19c010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52cc19c010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0033cb0 qid=0x0000
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52cc19c010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 ctrlr=0x7f52e81f7010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 12
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme100n1 at 0000:c6:00.0 [8CE38EE30087685F] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 with nsid=1
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00f1ba0 qid=0x0000 ctrlr=0x7f52e81f7010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f1ba0 qid=0x0000
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00f1ba0 qid=0x0000 ctrlr=0x7f52e81f7010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 ctrlr=0x7f52cc1cc2f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d0106220 qid=0x0000 ctrlr=0x7f52cc1cc2f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0106220 qid=0x0000
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d0106220 qid=0x0000 ctrlr=0x7f52cc1cc2f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme100n1 eui64:8CE38EE30087685F sn:YDF0A0380VY3 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme100n1 bdf:0000:c6:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 eui64:8CE38EE30087685F sn:YDF0A0380VY3
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f5c20 qid=0x0000
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme99n1 at 0000:84:00.0 [8CE38EE2028DDB01] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 with nsid=1
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme99n1 eui64:8CE38EE2028DDB01 sn:70H0A00GTBU7 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme99n1 bdf:0000:84:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 eui64:8CE38EE2028DDB01 sn:70H0A00GTBU7
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d010b740 qid=0x0000 ctrlr=0x7f52c82af770 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0030
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0030
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d010b740, qp_create_elapsed=23774us, res_elapsed=9us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme98n1 at 0000:88:00.0 [8CE38EE300B00E06] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 with nsid=1
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010b740 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme98n1 eui64:8CE38EE300B00E06 sn:1EE0A0T603R3 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme98n1 bdf:0000:88:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 eui64:8CE38EE300B00E06 sn:1EE0A0T603R3
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme97n1 at 0000:86:00.0 [002538E7415006FE] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 with nsid=1
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00eb8e0 qid=0x0000 ctrlr=0x7f52d4266450 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00eb8e0, qp_create_elapsed=24098us, res_elapsed=12us
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme97n1 eui64:002538E7415006FE sn:S7Z9NG0X700006 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme97n1 bdf:0000:86:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 eui64:002538E7415006FE sn:S7Z9NG0X700006
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00eb8e0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52e82531d0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002f
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme96n1 at 0000:ca:00.0 [002538E141500853] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 with nsid=1
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52e82531d0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002f
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00fa810 qid=0x0000
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52e82531d0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002f
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 ctrlr=0x7f52d42ee800 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d0038c80 qid=0x0000 ctrlr=0x7f52d42ee800 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0038c80 qid=0x0000
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d0038c80 qid=0x0000 ctrlr=0x7f52d42ee800 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:21 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme96n1 eui64:002538E141500853 sn:S7PNNG0X100108 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme96n1 bdf:0000:ca:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 eui64:002538E141500853 sn:S7PNNG0X100108
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme95n1 at 0000:c8:00.0 [002538E141500775] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 with nsid=1
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x002f, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00eb520 qid=0x0000 ctrlr=0x7f52e83a0c40 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00eb520, qp_create_elapsed=24848us, res_elapsed=8us
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme95n1 eui64:002538E141500775 sn:S7PLNG0X100199 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme95n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00eb520 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 ctrlr=0x7f52d44dad70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme94n1 at 0000:8a:00.0 [002538E141500763] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 with nsid=1
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d44dad70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f3910 qid=0x0000
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d44dad70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme94n1 eui64:002538E141500763 sn:S7PLNG0X100181 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme94n1 bdf:0000:8a:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 eui64:002538E141500763 sn:S7PLNG0X100181
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme93n1 at 0000:85:00.0 [0025389401006B55] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 with nsid=1
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme93n1 eui64:0025389401006B55 sn:S5URNE0N400075 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme93n1 bdf:0000:85:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 eui64:0025389401006B55 sn:S5URNE0N400075
9/10/2025 10:29:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:c6:00.0 (HotInNvme100)
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:c6:00.0 name=HotInNvme100 ctrlr=0x200139fa2200
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme100
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme100n1 sn:(YDF0A0380VY3 ) mn:(KCM730T7P5xnETRI ) blocklen:512 blockcnt:60001615872 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:10 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:84:00.0 (HotInNvme99)
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:84:00.0 name=HotInNvme99 ctrlr=0x200134120400
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme99
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme99n1 sn:(70H0A00GTBU7 ) mn:(TCM61920P5xnFTRI ) blocklen:512 blockcnt:3750748848 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:7 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:13 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:c6:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:c6:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:84:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:84:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:c6:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:84:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:c6:00
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:c6:00.0
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:c6:00.0
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:84:00
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:84:00.0
9/10/2025 10:29:19 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:84:00.0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00370f0 qid=0x0000 qp_destroy_elapsed=48194us res_destroy_elapsed=17us
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0034070 qid=0x0000 qp_destroy_elapsed=49245us res_destroy_elapsed=16us
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0105760 qid=0x0000 qp_destroy_elapsed=49473us res_destroy_elapsed=14us
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f1790 qid=0x0000 qp_destroy_elapsed=37969us res_destroy_elapsed=16us
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:88:00.0 (HotInNvme98)
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:88:00.0 name=HotInNvme98 ctrlr=0x200132982500
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme98
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme98n1 sn:(1EE0A0T603R3 ) mn:(KCM71920P5xnETRI ) blocklen:512 blockcnt:3750748848 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:10 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:ca:00.0 (HotInNvme96)
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:86:00.0 (HotInNvme97)
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:ca:00.0 name=HotInNvme96 ctrlr=0x20013bf432c0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:86:00.0 name=HotInNvme97 ctrlr=0x20013bb14480
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme97
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme96
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme97n1 sn:(S7Z9NG0X700006 ) mn:(AELN61T4P5xnEQRI ) blocklen:512 blockcnt:120001134592 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:31 ns.npwa:31 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme96n1 sn:(S7PNNG0X100108 ) mn:(AELN30T7P5xnEQRI ) blocklen:512 blockcnt:60001615872 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:c8:00.0 (HotInNvme95)
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:8a:00.0 (HotInNvme94)
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:c8:00.0 name=HotInNvme95 ctrlr=0x20013ccd0480
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:8a:00.0 name=HotInNvme94 ctrlr=0x200132329480
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme95
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme94
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme95n1 sn:(S7PLNG0X100199 ) mn:(AELN15T4P5xnEQRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme94n1 sn:(S7PLNG0X100181 ) mn:(AELN15T4P5xnEQRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:88:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:85:00.0 (HotInNvme93)
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:85:00.0 name=HotInNvme93 ctrlr=0x200132577480
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme93
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme93n1 sn:(S5URNE0N400075 ) mn:(AEGL1920P5xnFTRI ) blocklen:512 blockcnt:3750748848 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:0 ns.npwg:0 ns.npwa:0 ns.nows:0 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:ca:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:8a:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:86:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:85:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:ca:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:8a:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:88:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:86:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:85:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:ca:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:8a:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:88:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:86:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:ca:00
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:ca:00.0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:ca:00.0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:85:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_2]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00e7140 qid=0x0001 ctrlr=0x7f52d41c1f30 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0011
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00e7140, qp_create_elapsed=24159us, res_elapsed=37us
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00e7140 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_4]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00393f0 qid=0x0001 ctrlr=0x7f52cc333010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00393f0, qp_create_elapsed=24823us, res_elapsed=39us
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:8a:00
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:8a:00.0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:8a:00.0
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00393f0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_3]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d0039030 qid=0x0001 ctrlr=0x7f52c8249860 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d0039030, qp_create_elapsed=23963us, res_elapsed=35us
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0039030 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_1]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00385f0 qid=0x0001 ctrlr=0x7f52e819b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:18 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00385f0, qp_create_elapsed=23823us, res_elapsed=35us
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00385f0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d0038230 qid=0x0000 ctrlr=0x7f52cc333010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d0038230, qp_create_elapsed=24137us, res_elapsed=14us
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0038230 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:88:00
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:88:00.0
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:88:00.0
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d010a650 qid=0x0000 ctrlr=0x7f52c8249860 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000c
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000c
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d010a650, qp_create_elapsed=23669us, res_elapsed=9us
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010a650 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 ctrlr=0x7f52cc2cfda0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00370f0 qid=0x0000 ctrlr=0x7f52cc2cfda0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00370f0 qid=0x0000
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00370f0 qid=0x0000 ctrlr=0x7f52cc2cfda0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 ctrlr=0x7f52c8316b00 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d0034070 qid=0x0000 ctrlr=0x7f52c8316b00 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0034070 qid=0x0000
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d0034070 qid=0x0000 ctrlr=0x7f52c8316b00 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme92n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme92n1 for latency reports
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0011
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d010a290, qp_create_elapsed=24314us, res_elapsed=11us
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d010a290 qid=0x0000 ctrlr=0x7f52d41c1f30 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0011
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme91n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme91n1 for latency reports
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010a290 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme90n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme90n1 for latency reports
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme89n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme89n1 for latency reports
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f71b0 qid=0x0000 ctrlr=0x7f52e819b010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00f71b0, qp_create_elapsed=24973us, res_elapsed=11us
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:86:00
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:86:00.0
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:86:00.0
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 4
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme92n1 at 0000:07:00.0 [8CE38EE300FF11DF] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 with nsid=1
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f71b0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 ctrlr=0x7f52e8186ea0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0010
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme92n1 eui64:8CE38EE300FF11DF sn:2ET0A01D0AW4 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme92n1 bdf:0000:07:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 eui64:8CE38EE300FF11DF sn:2ET0A01D0AW4
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d0105760 qid=0x0000 ctrlr=0x7f52e8186ea0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0010
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0105760 qid=0x0000
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d0105760 qid=0x0000 ctrlr=0x7f52e8186ea0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0010
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 ctrlr=0x7f52d4250310 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f1790 qid=0x0000 ctrlr=0x7f52d4250310 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f1790 qid=0x0000
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f1790 qid=0x0000 ctrlr=0x7f52d4250310 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme91n1 at 0000:05:00.0 [8CE38EE202891501] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 with nsid=1
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme91n1 eui64:8CE38EE202891501 sn:70G0A00FTBX7 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme91n1 bdf:0000:05:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 eui64:8CE38EE202891501 sn:70G0A00FTBX7
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme90n1 at 0000:8b:00.0 [002538E9410026BC] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 with nsid=1
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0010, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme90n1 eui64:002538E9410026BC sn:S7WBNE0X900403 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme90n1 bdf:0000:8b:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 eui64:002538E9410026BC sn:S7WBNE0X900403
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme89n1 at 0000:04:00.0 [0000000000000000] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 with nsid=1
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00, event: type=0002, info=0000, log page id=0004
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme89n1 eui64:0000000000000000 sn:S80DNG0Y200115 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme89n1 bdf:0000:04:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 eui64:0000000000000000 sn:S80DNG0Y200115
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:85:00.0
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:85:00
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:17 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:85:00.0
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:07:00.0 (HotInNvme92)
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:07:00.0 name=HotInNvme92 ctrlr=0x2001393ca500
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme92
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme92n1 sn:(2ET0A01D0AW4 ) mn:(KCM77680P5AxFTRI ) blocklen:512 blockcnt:15002931888 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:10 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:05:00.0 (HotInNvme91)
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:05:00.0 name=HotInNvme91 ctrlr=0x2001312dc500
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme91
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme91n1 sn:(70G0A00FTBX7 ) mn:(TCM615T4P5xnFTRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:7 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:13 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:8b:00.0 (HotInNvme90)
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:8b:00.0 name=HotInNvme90 ctrlr=0x200131a4d500
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme90
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme90n1 sn:(S7WBNE0X900403 ) mn:(AELN30T7P5xnFQRI ) blocklen:512 blockcnt:60001615872 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:07:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:8b:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:8b:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:07:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:05:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:05:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:8b:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:07:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:05:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:8b:00
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:8b:00.0
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:8b:00.0
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:07:00.0
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:07:00
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:07:00.0
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:05:00.0
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:05:00
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:16 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:05:00.0
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:04:00.0 (HotInNvme89)
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:04:00.0 name=HotInNvme89 ctrlr=0x20013eb46500
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme89
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme89n1 sn:(S80DNG0Y200115 ) mn:(SAMSUNG MZ3MO15THCLC-00AW7 ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:6 (262144) optimal_io_boundary:0 max_segment_size:262144 max_num_segments:256
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:04:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:04:00.0
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:04:00.0] !!! Setting CC.EN = 1
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:04:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:04:00
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:04:00.0
9/10/2025 10:29:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 10:29:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 10:27:27 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/10/2025 10:27:14 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 0
9/10/2025 10:27:14 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c6:00.0 with driver=nvme
9/10/2025 10:27:14 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:84:00.0 with driver=nvme
9/10/2025 10:27:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme86n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme86n1 from latency reports
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme86n1
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme85n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme85n1 from latency reports
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme85n1
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 0
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c6:00.0 is missing
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:84:00.0 is missing
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme86n1 bdf:0000:c6:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 eui64:8CE38EE30087685F sn:YDF0A0380VY3 reason:unknown
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme85n1 bdf:0000:84:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 eui64:8CE38EE2028DDB01 sn:70H0A00GTBU7 reason:unknown
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme86n1 was hot removed
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme85n1 was hot removed
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:86:00.0 with driver=nvme
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:85:00.0 with driver=nvme
9/10/2025 10:27:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010a900 qid=0x0001 qp_destroy_elapsed=49935us res_destroy_elapsed=110us
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ee470 qid=0x0000 qp_destroy_elapsed=38549us res_destroy_elapsed=15us
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme85
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:84:00
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0000, state=6
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0000
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 28 times ***
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:84:00.0] Prepare to destruct SSD: 0000:84:00.0
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:84:00.0] already in failed state
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:84:00.0]: unregistering thread=app_thread
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:84:00.0
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:84:00.0)
9/10/2025 10:27:10 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:84:00.0: destruct=1
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52cc19c010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d0033cb0, qp_create_elapsed=24199us, res_elapsed=12us
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0033cb0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d010a900 qid=0x0001 ctrlr=0x7f52cc435050 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010a900 qid=0x0001
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d010a900 qid=0x0001 ctrlr=0x7f52cc435050 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 ctrlr=0x7f52cc435050 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d00ee470 qid=0x0000 ctrlr=0x7f52cc435050 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ee470 qid=0x0000
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d00ee470 qid=0x0000 ctrlr=0x7f52cc435050 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 5 times ***
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 21 times ***
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:84:00.0] already in failed state
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23, event: type=0002, info=0000, log page id=0004
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:84:00.0)
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:84:00.0: destruct=1
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010a4f0 qid=0x0001 qp_destroy_elapsed=49630us res_destroy_elapsed=109us
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ed830 qid=0x0000 qp_destroy_elapsed=37738us res_destroy_elapsed=15us
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 11 times ***
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200136093e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x200138bcfe60, qid=0x03: Aborting trackers
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0003
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0002
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001405dfe60 was disconnected and freed. reset controller.
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200138bcfe60 was disconnected and freed. reset controller.
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0004, state=6
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0003, state=6
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0002, state=6
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0001, state=6
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0000, state=5
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x2001405dfe60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200138bcfe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013742fe60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200133d0fe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x2001405dfe60, qid=0x02: Aborting trackers
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x20013742fe60, qid=0x01: Aborting trackers
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x200133d0fe60, qid=0x04: Aborting trackers
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0004
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0001
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x2001405dfe60, qid=0x0002: enter
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x200138bcfe60, qid=0x0003: enter
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x20013742fe60, qid=0x0001: enter
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x200133d0fe60, qid=0x0004: enter
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:84:00.0)
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:84:00.0: destruct=0
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:84:00.0
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013742fe60 was disconnected and freed. reset controller.
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200133d0fe60 was disconnected and freed. reset controller.
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:27:09 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme86
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:c6:00
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0000
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0000, state=6
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 29 times ***
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:c6:00.0] Prepare to destruct SSD: 0000:c6:00.0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c6:00.0] already in failed state
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:c6:00.0]: unregistering thread=app_thread
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:c6:00.0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c6:00.0)
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c6:00.0: destruct=1
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00f6780 qid=0x0000 ctrlr=0x7f52c8396010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00f6780, qp_create_elapsed=23976us, res_elapsed=9us
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f6780 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d010a4f0 qid=0x0001 ctrlr=0x7f52c85e8010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010a4f0 qid=0x0001
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d010a4f0 qid=0x0001 ctrlr=0x7f52c85e8010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 ctrlr=0x7f52c85e8010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00ed830 qid=0x0000 ctrlr=0x7f52c85e8010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ed830 qid=0x0000
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00ed830 qid=0x0000 ctrlr=0x7f52c85e8010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 4 times ***
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 9 times ***
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c6:00.0] already in failed state
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22, event: type=0002, info=0000, log page id=0004
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c6:00.0)
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c6:00.0: destruct=1
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme83n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme83n1 from latency reports
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme83n1
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme79n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme79n1 from latency reports
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme79n1
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 22 times ***
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 2
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:86:00.0 is missing
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:85:00.0 is missing
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme83n1 bdf:0000:86:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 eui64:002538E7415006FE sn:S7Z9NG0X700006 reason:unknown
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme79n1 bdf:0000:85:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 eui64:0025389401006B55 sn:S5URNE0N400075 reason:unknown
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme83n1 was hot removed
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme79n1 was hot removed
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:ca:00.0 with driver=nvme
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:8a:00.0 with driver=nvme
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:88:00.0 with driver=nvme
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393c8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x20013e729e60, qid=0x02: Aborting trackers
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x20013391fe60, qid=0x04: Aborting trackers
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0004
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0003
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0002
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fdffe60 was disconnected and freed. reset controller.
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013e729e60 was disconnected and freed. reset controller.
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013391fe60 was disconnected and freed. reset controller.
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0004, state=6
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0003, state=6
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0002, state=6
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0001, state=6
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0000, state=5
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013fdffe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013e729e60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013391fe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013275fe60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x20013fdffe60, qid=0x03: Aborting trackers
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x20013275fe60, qid=0x01: Aborting trackers
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0001
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x20013fdffe60, qid=0x0003: enter
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x20013e729e60, qid=0x0002: enter
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x20013391fe60, qid=0x0004: enter
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x20013275fe60, qid=0x0001: enter
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c6:00.0)
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c6:00.0: destruct=0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:c6:00.0
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013275fe60 was disconnected and freed. reset controller.
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:27:08 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme79
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:85:00
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0000, state=6
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0000
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 29 times ***
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:85:00.0] Prepare to destruct SSD: 0000:85:00.0
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:85:00.0] already in failed state
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:85:00.0]: unregistering thread=app_thread
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:85:00.0
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:85:00.0)
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:85:00.0: destruct=1
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010a0e0 qid=0x0001 qp_destroy_elapsed=51170us res_destroy_elapsed=114us
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d44dad70 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f3910, qp_create_elapsed=24916us, res_elapsed=10us
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ecb40 qid=0x0000 qp_destroy_elapsed=38069us res_destroy_elapsed=15us
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f3910 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 ctrlr=0x7f52e835bdb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d010a0e0 qid=0x0001 ctrlr=0x7f52e835bdb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010a0e0 qid=0x0001
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d010a0e0 qid=0x0001 ctrlr=0x7f52e835bdb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00ecb40 qid=0x0000 ctrlr=0x7f52e835bdb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ecb40 qid=0x0000
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00ecb40 qid=0x0000 ctrlr=0x7f52e835bdb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 1 time ***
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:85:00.0] already in failed state
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21, event: type=0002, info=0000, log page id=0004
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:85:00.0)
9/10/2025 10:27:07 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:85:00.0: destruct=1
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x2001325772e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0003
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840bae0 was disconnected and freed. reset controller.
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x20013840bce0, qid=0x02: Aborting trackers
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x20013840bae0, qid=0x03: Aborting trackers
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x20013840b8e0, qid=0x04: Aborting trackers
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0004
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0002
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0001
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:85:00.0: destruct=0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:85:00.0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840bee0 was disconnected and freed. reset controller.
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840bce0 was disconnected and freed. reset controller.
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013840b8e0 was disconnected and freed. reset controller.
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0004, state=6
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0003, state=6
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0002, state=6
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0001, state=6
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0000, state=5
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x20013840bee0, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x20013840bce0, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x20013840bae0, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x20013840b8e0, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x20013840bee0, qid=0x01: Aborting trackers
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x20013840bee0, qid=0x0001: enter
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x20013840bce0, qid=0x0002: enter
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x20013840bae0, qid=0x0003: enter
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x20013840b8e0, qid=0x0004: enter
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:85:00.0)
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme83
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:86:00
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 18 times ***
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:86:00.0] Prepare to destruct SSD: 0000:86:00.0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:86:00.0] already in failed state
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:86:00.0]: unregistering thread=app_thread
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:86:00.0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:86:00.0)
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:86:00.0: destruct=1
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0109cd0 qid=0x0001 qp_destroy_elapsed=36183us res_destroy_elapsed=114us
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ed470 qid=0x0000 qp_destroy_elapsed=37828us res_destroy_elapsed=17us
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00f1ba0 qid=0x0000 ctrlr=0x7f52e81f7010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00f1ba0, qp_create_elapsed=25132us, res_elapsed=10us
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f1ba0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d0109cd0 qid=0x0001 ctrlr=0x7f52d44dad70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0109cd0 qid=0x0001
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d0109cd0 qid=0x0001 ctrlr=0x7f52d44dad70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 ctrlr=0x7f52d44dad70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00ed470 qid=0x0000 ctrlr=0x7f52d44dad70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ed470 qid=0x0000
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00ed470 qid=0x0000 ctrlr=0x7f52d44dad70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:86:00.0] already in failed state
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19, event: type=0002, info=0000, log page id=0004
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:86:00.0)
9/10/2025 10:27:06 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:86:00.0: destruct=1
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x20013979fe60, qid=0x02: Aborting trackers
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0002
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013979fe60 was disconnected and freed. reset controller.
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139404e60 was disconnected and freed. reset controller.
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013979fe60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x200139404e60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x2001393afe60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x200139404e60, qid=0x04: Aborting trackers
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x2001393afe60, qid=0x01: Aborting trackers
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x200134ccfe60, qid=0x03: Aborting trackers
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0004
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0003
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0001
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x20013979fe60, qid=0x0002: enter
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x200139404e60, qid=0x0004: enter
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x2001393afe60, qid=0x0001: enter
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x200134ccfe60, qid=0x0003: enter
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:86:00.0)
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:86:00.0: destruct=0
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:86:00.0
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001393afe60 was disconnected and freed. reset controller.
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200134ccfe60 was disconnected and freed. reset controller.
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x200134ccfe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme84n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme84n1 from latency reports
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme84
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:88:00
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0000, state=6
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0000
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme84n1
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:88:00.0] Prepare to destruct SSD: 0000:88:00.0
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:88:00.0] already in failed state
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:88:00.0]: unregistering thread=app_thread
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:88:00.0
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:88:00.0)
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:88:00.0: destruct=1
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme82n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme82n1 from latency reports
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme82n1
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme80n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme80n1 from latency reports
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme80n1
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 4
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:ca:00.0 is missing
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:8a:00.0 is missing
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:88:00.0 is missing
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme84n1 bdf:0000:88:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 eui64:8CE38EE300B00E06 sn:1EE0A0T603R3 reason:unknown
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme82n1 bdf:0000:ca:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 eui64:002538E141500853 sn:S7PNNG0X100108 reason:unknown
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme80n1 bdf:0000:8a:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 eui64:002538E141500763 sn:S7PLNG0X100181 reason:unknown
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme84n1 was hot removed
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme82n1 was hot removed
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme80n1 was hot removed
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:8b:00.0 with driver=nvme
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:07:00.0 with driver=nvme
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:05:00.0 with driver=nvme
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:04:00.0 with driver=nvme
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00eefb0 qid=0x0001 qp_destroy_elapsed=42026us res_destroy_elapsed=112us
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ed0b0 qid=0x0000 qp_destroy_elapsed=38533us res_destroy_elapsed=15us
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 17 times ***
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d0106220 qid=0x0000 ctrlr=0x7f52cc1cc2f0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:27:05 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d0106220, qp_create_elapsed=24485us, res_elapsed=14us
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0106220 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00eefb0 qid=0x0001 ctrlr=0x7f52e8158e70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00eefb0 qid=0x0001
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00eefb0 qid=0x0001 ctrlr=0x7f52e8158e70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 ctrlr=0x7f52e8158e70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00ed0b0 qid=0x0000 ctrlr=0x7f52e8158e70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ed0b0 qid=0x0000
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00ed0b0 qid=0x0000 ctrlr=0x7f52e8158e70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 39 times ***
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:88:00.0] already in failed state
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15, event: type=0002, info=0000, log page id=0004
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:88:00.0)
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:88:00.0: destruct=1
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140ecfd20, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x200140b6fe60, qid=0x03: Aborting trackers
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x200131a77e60, qid=0x02: Aborting trackers
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0004
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0003
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0002
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200140b6fe60 was disconnected and freed. reset controller.
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013352fe60 was disconnected and freed. reset controller.
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200131a77e60 was disconnected and freed. reset controller.
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0004, state=6
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0003, state=6
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0002, state=6
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0001, state=6
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0000, state=5
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200140b6fe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x20013eb46360, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x20013352fe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200131a77e60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x20013eb46360, qid=0x01: Aborting trackers
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x20013352fe60, qid=0x04: Aborting trackers
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0001
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x200140b6fe60, qid=0x0003: enter
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x20013eb46360, qid=0x0001: enter
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x20013352fe60, qid=0x0004: enter
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x200131a77e60, qid=0x0002: enter
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:88:00.0)
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:88:00.0: destruct=0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:88:00.0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013eb46360 was disconnected and freed. reset controller.
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00eebf0 qid=0x0001 qp_destroy_elapsed=49942us res_destroy_elapsed=109us
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme82
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:ca:00
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0000
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ecf50 qid=0x0000 qp_destroy_elapsed=31073us res_destroy_elapsed=17us
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:ca:00.0] Prepare to destruct SSD: 0000:ca:00.0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 29 times ***
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:ca:00.0] already in failed state
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:ca:00.0]: unregistering thread=app_thread
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:ca:00.0
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:ca:00.0)
9/10/2025 10:27:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:ca:00.0: destruct=1
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52c8205470 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00f5c20, qp_create_elapsed=24335us, res_elapsed=9us
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f5c20 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00eebf0 qid=0x0001 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00eebf0 qid=0x0001
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00eebf0 qid=0x0001 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00ecf50 qid=0x0000 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ecf50 qid=0x0000
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00ecf50 qid=0x0000 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 39 times ***
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:ca:00.0] already in failed state
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14, event: type=0002, info=0000, log page id=0004
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:ca:00.0)
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:ca:00.0: destruct=1
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x200139f7fe60, qid=0x04: Aborting trackers
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x20013884ee60, qid=0x03: Aborting trackers
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0004
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0003
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0002
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139f7fe60 was disconnected and freed. reset controller.
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013884ee60 was disconnected and freed. reset controller.
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013232e3e0 was disconnected and freed. reset controller.
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200139f7fe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013884ee60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013232e5e0, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013232e3e0, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x20013232e5e0, qid=0x01: Aborting trackers
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x20013232e3e0, qid=0x02: Aborting trackers
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0001
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x200139f7fe60, qid=0x0004: enter
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x20013884ee60, qid=0x0003: enter
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x20013232e5e0, qid=0x0001: enter
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x20013232e3e0, qid=0x0002: enter
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:ca:00.0)
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:ca:00.0: destruct=0
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:ca:00.0
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013232e5e0 was disconnected and freed. reset controller.
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ee830 qid=0x0001 qp_destroy_elapsed=48753us res_destroy_elapsed=113us
9/10/2025 10:27:03 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f3910 qid=0x0000 qp_destroy_elapsed=37773us res_destroy_elapsed=17us
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme80
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:8a:00
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 31 times ***
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:8a:00.0] Prepare to destruct SSD: 0000:8a:00.0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8a:00.0] already in failed state
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:8a:00.0]: unregistering thread=app_thread
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:8a:00.0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8a:00.0)
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8a:00.0: destruct=1
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d0038c80 qid=0x0000 ctrlr=0x7f52d42ee800 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d0038c80, qp_create_elapsed=24449us, res_elapsed=11us
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0038c80 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00ee830 qid=0x0001 ctrlr=0x7f52d4265010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ee830 qid=0x0001
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00ee830 qid=0x0001 ctrlr=0x7f52d4265010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 ctrlr=0x7f52d4265010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d4265010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f3910 qid=0x0000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d4265010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 24 times ***
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8a:00.0] already in failed state
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11, event: type=0002, info=0000, log page id=0004
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8a:00.0)
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8a:00.0: destruct=1
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme78n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme78n1 from latency reports
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme78n1
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme77n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme77n1 from latency reports
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme77n1
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme76n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme76n1 from latency reports
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme76n1
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme75n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme75n1 from latency reports
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme75n1
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 7
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:8b:00.0 is missing
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:07:00.0 is missing
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:05:00.0 is missing
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:04:00.0 is missing
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme78n1 bdf:0000:07:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 eui64:8CE38EE300FF11DF sn:2ET0A01D0AW4 reason:unknown
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme77n1 bdf:0000:05:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 eui64:8CE38EE202891501 sn:70G0A00FTBX7 reason:unknown
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme76n1 bdf:0000:8b:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 eui64:002538E9410026BC sn:S7WBNE0X900403 reason:unknown
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme75n1 bdf:0000:04:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 eui64:0000000000000000 sn:S80DNG0Y200115 reason:unknown
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme78n1 was hot removed
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme77n1 was hot removed
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme76n1 was hot removed
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme75n1 was hot removed
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397e9de0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x20025bc06320, qid=0x01: Aborting trackers
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x2001409bfe60, qid=0x04: Aborting trackers
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x200139425920, qid=0x02: Aborting trackers
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0004
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0003
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0002
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0001
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20025bc06320 was disconnected and freed. reset controller.
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001409bfe60 was disconnected and freed. reset controller.
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139425920 was disconnected and freed. reset controller.
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139425720 was disconnected and freed. reset controller.
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20025bc06320, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001409bfe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200139425920, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200139425720, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x200139425720, qid=0x03: Aborting trackers
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x20025bc06320, qid=0x0001: enter
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x2001409bfe60, qid=0x0004: enter
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x200139425920, qid=0x0002: enter
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x200139425720, qid=0x0003: enter
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8a:00.0)
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8a:00.0: destruct=0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:8a:00.0
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:27:02 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f2680 qid=0x0001 qp_destroy_elapsed=49506us res_destroy_elapsed=113us
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010f0b0 qid=0x0000 qp_destroy_elapsed=37967us res_destroy_elapsed=17us
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme76
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:8b:00
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0000
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 29 times ***
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200132432220, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:8b:00.0] Prepare to destruct SSD: 0000:8b:00.0
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8b:00.0] already in failed state
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:8b:00.0]: unregistering thread=app_thread
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:8b:00.0
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8b:00.0)
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8b:00.0: destruct=1
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200132432220, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0010
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d0105760, qp_create_elapsed=24686us, res_elapsed=7us
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d0105760 qid=0x0000 ctrlr=0x7f52e8186ea0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0010
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200132432220, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0105760 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200132432220, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f2680 qid=0x0001 ctrlr=0x7f52d41be0f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000f
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f2680 qid=0x0001
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f2680 qid=0x0001 ctrlr=0x7f52d41be0f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000f
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200132432220, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 ctrlr=0x7f52d41be0f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000f
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d010f0b0 qid=0x0000 ctrlr=0x7f52d41be0f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000f
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010f0b0 qid=0x0000
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d010f0b0 qid=0x0000 ctrlr=0x7f52d41be0f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000f
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 4 times ***
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200132432220, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8b:00.0] already in failed state
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000f, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09, event: type=0002, info=0000, log page id=0004
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8b:00.0)
9/10/2025 10:27:01 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8b:00.0: destruct=1
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200132432220, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d003a4a0 qid=0x0001 qp_destroy_elapsed=49912us res_destroy_elapsed=115us
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200132432220, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0003
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fc7eb20 was disconnected and freed. reset controller.
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20013fc7eb20, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20013fc7ed20, qid=0x02: Aborting trackers
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20013fc7eb20, qid=0x03: Aborting trackers
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20013fc7e920, qid=0x04: Aborting trackers
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x200132308160, qid=0x01: Aborting trackers
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0004
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0002
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0001
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20013fc7eb20, qid=0x0003: enter
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20013fc7e920, qid=0x0004: enter
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8b:00.0)
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8b:00.0: destruct=0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:8b:00.0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fc7ed20 was disconnected and freed. reset controller.
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fc7e920 was disconnected and freed. reset controller.
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132308160 was disconnected and freed. reset controller.
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20013fc7ed20, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20013fc7e920, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200132308160, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20013fc7ed20, qid=0x0002: enter
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x200132308160, qid=0x0001: enter
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0106220 qid=0x0000 qp_destroy_elapsed=37569us res_destroy_elapsed=15us
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme78
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:07:00
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0000, state=6
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 25 times ***
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:07:00.0] Prepare to destruct SSD: 0000:07:00.0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:07:00.0] already in failed state
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:07:00.0]: unregistering thread=app_thread
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:07:00.0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:07:00.0)
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:07:00.0: destruct=1
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00370f0 qid=0x0000 ctrlr=0x7f52cc2cfda0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00370f0, qp_create_elapsed=24522us, res_elapsed=8us
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00370f0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 ctrlr=0x7f52cc19c010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d003a4a0 qid=0x0001 ctrlr=0x7f52cc19c010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d003a4a0 qid=0x0001
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d003a4a0 qid=0x0001 ctrlr=0x7f52cc19c010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d0106220 qid=0x0000 ctrlr=0x7f52cc19c010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0106220 qid=0x0000
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d0106220 qid=0x0000 ctrlr=0x7f52cc19c010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 9 times ***
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 27 times ***
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:07:00.0] already in failed state
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03, event: type=0002, info=0000, log page id=0004
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:07:00.0)
9/10/2025 10:27:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:07:00.0: destruct=1
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d003abc0 qid=0x0001 qp_destroy_elapsed=49337us res_destroy_elapsed=112us
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f5c20 qid=0x0000 qp_destroy_elapsed=37690us res_destroy_elapsed=15us
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 4 times ***
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200132308ca0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x20013fc7e720, qid=0x01: Aborting trackers
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0004
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0001
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fc7e720 was disconnected and freed. reset controller.
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fc7e120 was disconnected and freed. reset controller.
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0004, state=6
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0003, state=6
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0002, state=6
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0001, state=6
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0000, state=5
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20013fc7e720, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20013fc7e520, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20013fc7e320, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20013fc7e120, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x20013fc7e520, qid=0x02: Aborting trackers
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x20013fc7e320, qid=0x03: Aborting trackers
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x20013fc7e120, qid=0x04: Aborting trackers
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0003
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0002
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x20013fc7e720, qid=0x0001: enter
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x20013fc7e520, qid=0x0002: enter
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x20013fc7e320, qid=0x0003: enter
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x20013fc7e120, qid=0x0004: enter
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:07:00.0)
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:07:00.0: destruct=0
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:07:00.0
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fc7e520 was disconnected and freed. reset controller.
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fc7e320 was disconnected and freed. reset controller.
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme77
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:05:00
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0000, state=6
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0000
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 27 times ***
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:05:00.0] Prepare to destruct SSD: 0000:05:00.0
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:05:00.0] already in failed state
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:05:00.0]: unregistering thread=app_thread
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:05:00.0
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:05:00.0)
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:05:00.0: destruct=1
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d0034070 qid=0x0000 ctrlr=0x7f52c8316b00 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000b
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000b
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d0034070, qp_create_elapsed=24124us, res_elapsed=8us
9/10/2025 10:26:59 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0034070 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 ctrlr=0x7f52c8316b00 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d003abc0 qid=0x0001 ctrlr=0x7f52c8316b00 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d003abc0 qid=0x0001
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d003abc0 qid=0x0001 ctrlr=0x7f52c8316b00 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52c8316b00 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f5c20 qid=0x0000
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52c8316b00 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 6 times ***
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 16 times ***
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:05:00.0] already in failed state
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01, event: type=0002, info=0000, log page id=0004
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:05:00.0)
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:05:00.0: destruct=1
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0035320 qid=0x0001 qp_destroy_elapsed=50027us res_destroy_elapsed=112us
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d01074c0 qid=0x0000 qp_destroy_elapsed=38684us res_destroy_elapsed=14us
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 15 times ***
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200139fc2260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x200136874e60, qid=0x01: Aborting trackers
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x200136874a60, qid=0x03: Aborting trackers
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0003
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0002
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0001
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200136874e60 was disconnected and freed. reset controller.
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200136874c60 was disconnected and freed. reset controller.
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200136874a60 was disconnected and freed. reset controller.
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0004, state=6
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0003, state=6
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0002, state=6
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0001, state=6
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0000, state=5
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200136874e60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200136874c60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200136874a60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200136874860, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x200136874c60, qid=0x02: Aborting trackers
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x200136874860, qid=0x04: Aborting trackers
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0004
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x200136874e60, qid=0x0001: enter
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x200136874c60, qid=0x0002: enter
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x200136874a60, qid=0x0003: enter
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x200136874860, qid=0x0004: enter
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:05:00.0)
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:05:00.0: destruct=0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:05:00.0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200136874860 was disconnected and freed. reset controller.
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme75
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:04:00
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0000
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 29 times ***
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013450e1a0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:04:00.0] Prepare to destruct SSD: 0000:04:00.0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:04:00.0] already in failed state
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:04:00.0]: unregistering thread=app_thread
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:04:00.0
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:04:00.0)
9/10/2025 10:26:58 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:04:00.0: destruct=1
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013450e1a0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f1790 qid=0x0000 ctrlr=0x7f52d4250310 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f1790, qp_create_elapsed=24420us, res_elapsed=9us
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013450e1a0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f1790 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013450e1a0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d0035320 qid=0x0001 ctrlr=0x7f52e8186ea0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0035320 qid=0x0001
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d0035320 qid=0x0001 ctrlr=0x7f52e8186ea0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 ctrlr=0x7f52e8186ea0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013450e1a0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d01074c0 qid=0x0000 ctrlr=0x7f52e8186ea0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d01074c0 qid=0x0000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d01074c0 qid=0x0000 ctrlr=0x7f52e8186ea0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 4 times ***
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013450e1a0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:04:00.0] already in failed state
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00, event: type=0002, info=0000, log page id=0004
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:04:00.0)
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:04:00.0: destruct=1
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013450e1a0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20013450e1a0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0003
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132308560 was disconnected and freed. reset controller.
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x200132308560, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x200132308760, qid=0x02: Aborting trackers
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x200132308560, qid=0x03: Aborting trackers
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x200132308360, qid=0x04: Aborting trackers
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0004
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0002
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0001
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x200132308560, qid=0x0003: enter
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:04:00.0)
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:04:00.0: destruct=0
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:04:00.0
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132308960 was disconnected and freed. reset controller.
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132308760 was disconnected and freed. reset controller.
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132308360 was disconnected and freed. reset controller.
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x200132308960, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x200132308760, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x200132308360, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x200132308960, qid=0x01: Aborting trackers
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x200132308960, qid=0x0001: enter
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x200132308760, qid=0x0002: enter
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x200132308360, qid=0x0004: enter
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:26:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:26:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:26:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:26:27 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 10:26:27 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 10:26:27 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:26:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:26:24 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme88n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:26:24 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme88n1 from latency reports
9/10/2025 10:26:24 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme88n1
9/10/2025 10:26:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 11
9/10/2025 10:26:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 10:26:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme88n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199 reason:subsystem_reset
9/10/2025 10:26:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme88n1 was hot removed
9/10/2025 10:26:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010d180 qid=0x0001 qp_destroy_elapsed=48932us res_destroy_elapsed=111us
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f5860 qid=0x0000 qp_destroy_elapsed=37706us res_destroy_elapsed=17us
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme88
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 25 times ***
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013b34a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:c8:00.0]: unregistering thread=app_thread
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:c8:00.0
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 10:26:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013b34a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52e82531d0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002f
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x002f
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00fa810, qp_create_elapsed=25139us, res_elapsed=7us
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013b34a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00fa810 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013b34a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d010d180 qid=0x0001 ctrlr=0x7f52cc398c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002e
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010d180 qid=0x0001
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d010d180 qid=0x0001 ctrlr=0x7f52cc398c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002e
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52cc398c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002e
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013b34a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f5860 qid=0x0000 ctrlr=0x7f52cc398c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002e
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f5860 qid=0x0000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f5860 qid=0x0000 ctrlr=0x7f52cc398c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002e
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013b34a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x002e, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013b34a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013b34a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:373 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:373 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ABORT (08) qid:0 cid:382 nsid:0 cdw10:037f0002 cdw11:00000000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:693:nvme_pcie_qpair_complete_tracker !!!! [0000:c8:00.0]: Completing abort: qpair=0x20013b34a360, qpid=0, req=0x20013988d300
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app ctrlr.c:4675:_nvmf_request_complete !!! Abort response: subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, sqid=0x0001, cid=0x0000: sct/sc=0x00:0x00, cdw0=0x0000
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:2 cid:895 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013eb5e0e0, qid=0x03: Aborting trackers
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x200134ce80e0, qid=0x04: Aborting trackers
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x2001325d81a0, qid=0x02: Aborting trackers
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0004
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0003
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0002
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0001
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x2001325d81a0, qid=0x0002: enter
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fa71e60 was disconnected and freed. reset controller.
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013eb5e0e0 was disconnected and freed. reset controller.
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200134ce80e0 was disconnected and freed. reset controller.
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001325d81a0 was disconnected and freed. reset controller.
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013eb5e0e0, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200134ce80e0, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x2001325d81a0, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c5cf000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa71e60, qid=0x01: Aborting trackers
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013fa71e60, qid=0x0001: enter
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013eb5e0e0, qid=0x0003: enter
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x200134ce80e0, qid=0x0004: enter
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=0
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:c8:00.0
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:26:22 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:26:21 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1959:spdk_nvme_ctrlr_reset_subsystem [0000:c8:00.0] !!! After nssr rc=0
9/10/2025 10:26:21 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1955:spdk_nvme_ctrlr_reset_subsystem [0000:c8:00.0] !!! Before nssr
9/10/2025 10:26:21 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1947:spdk_nvme_ctrlr_reset_subsystem [0000:c8:00.0] !!! Resetting subsystem for ctrlr=0000:c8:00.0, cntlid=0x0081
9/10/2025 10:26:21 PM 4070 I nvmf_tgt_app nvme.c:548:nvme_request_check_timeout !!! Timeout expired for nvme req=0x20013fe6ee40, opc=0x02, qpair=0x2001325d81a0, type=IO
9/10/2025 10:26:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:2774:bdev_nvme_subsystem_reset !!! Initiating subsystem reset for: 0000:c8:00.0
9/10/2025 10:26:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:2764:_nvme_bdev_ctrlr_subsystem_reset !!! Subsystem reset rc=0
9/10/2025 10:26:21 PM 4070 I nvmf_tgt_app bdev_nvme.c:2759:_nvme_bdev_ctrlr_subsystem_reset !!! [app_thread] Starting subsystem reset
9/10/2025 10:26:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/10/2025 10:26:13 PM 4070 I nvmf_tgt_app nvme_ctrlr_cmd.c:597:_nvme_ctrlr_submit_abort_request !!! Submitting abort for: 0000:c8:00.0, req=0x20013988d300, outstanding_aborts=1
9/10/2025 10:26:13 PM 4070 I nvmf_tgt_app ctrlr.c:3532:nvmf_ctrlr_abort_request !!! Abort request: subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, sqid=0x0001, cid=0x0000 opc=0x02, lba=0, len=1, qid=0x0001, hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021
9/10/2025 10:26:13 PM 4070 I nvmf_tgt_app ctrlr.c:3500:nvmf_ctrlr_abort !!! [nvmf_tgt_poll_group_4] Abort request: hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021: subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, sqid=0x0001, cid=0x0000
9/10/2025 10:26:13 PM 4070 I nvmf_tgt_app bdev_nvme.c:8155:bdev_nvme_abort !!! [0000:c8:00.0]: qpair.id=2: bio=0x20025bd2c2e0, bio_to_abort=0x20025bd2bda0
9/10/2025 10:26:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 10:24:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 10:24:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:21:54 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 10:21:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:19:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 10:19:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 10:18:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 10:18:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 10:16:54 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 10:16:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/10/2025 10:14:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 10:14:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 10:14:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/10/2025 10:12:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/10/2025 10:11:59 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:000f p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:000e p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:000d p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:000c p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:720:nvme_pcie_qpair_complete_tracker !!! retrying: opc=0x02
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:000b p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:720:nvme_pcie_qpair_complete_tracker !!! retrying: opc=0x02
9/10/2025 10:11:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 10:11:41 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:000a p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:41 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0009 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:41 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:11:41 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:720:nvme_pcie_qpair_complete_tracker !!! retrying: opc=0x02
9/10/2025 10:11:41 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0008 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:41 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0007 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:41 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0006 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:41 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:11:41 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:720:nvme_pcie_qpair_complete_tracker !!! retrying: opc=0x02
9/10/2025 10:11:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00fa810 qid=0x0000 qp_destroy_elapsed=37810us res_destroy_elapsed=22us
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0005 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0004 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0003 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0002 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:720:nvme_pcie_qpair_complete_tracker !!! retrying: opc=0x02
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0001 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:720:nvme_pcie_qpair_complete_tracker !!! retrying: opc=0x02
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_4]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d010d180 qid=0x0001 ctrlr=0x7f52cc398c40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002e
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d010d180, qp_create_elapsed=24603us, res_elapsed=52us
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010d180 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x002e
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f5860 qid=0x0000 ctrlr=0x7f52cc398c40 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002e
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00f5860, qp_create_elapsed=24301us, res_elapsed=9us
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f5860 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52c85f1460 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002d
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52c85f1460 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002d
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00fa810 qid=0x0000
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52c85f1460 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002d
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme88n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:11:36 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme88n1 for latency reports
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 12
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme88n1 at 0000:c8:00.0 [002538E141500775] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 with nsid=1
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x002d, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme88n1 eui64:002538E141500775 sn:S7PLNG0X100199 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme88n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:c8:00.0 (HotInNvme88)
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:c8:00.0 name=HotInNvme88 ctrlr=0x20013b34a500
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme88
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme88n1 sn:(S7PLNG0X100199 ) mn:(AELN15T4P5xnEQRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:11:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 10:11:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 10:11:09 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:11:07 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme87n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 10:11:07 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme87n1 from latency reports
9/10/2025 10:11:07 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme87n1
9/10/2025 10:11:07 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 11
9/10/2025 10:11:07 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 10:11:07 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme87n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199 reason:unknown
9/10/2025 10:11:07 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme87n1 was hot removed
9/10/2025 10:11:07 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f8b90 qid=0x0001 qp_destroy_elapsed=49805us res_destroy_elapsed=110us
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme87
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00e7140 qid=0x0000 qp_destroy_elapsed=38050us res_destroy_elapsed=17us
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:c8:00.0]: unregistering thread=app_thread
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:c8:00.0
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 26 times ***
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52c85f1460 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002d
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x002d
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00fa810, qp_create_elapsed=24121us, res_elapsed=8us
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00fa810 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f8b90 qid=0x0001 ctrlr=0x7f52d451faf0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002c
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f8b90 qid=0x0001
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f8b90 qid=0x0001 ctrlr=0x7f52d451faf0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002c
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52d451faf0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002c
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00e7140 qid=0x0000 ctrlr=0x7f52d451faf0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002c
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00e7140 qid=0x0000
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00e7140 qid=0x0000 ctrlr=0x7f52d451faf0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002c
9/10/2025 10:11:04 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 6 times ***
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x002c, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013315d0e0, qid=0x04: Aborting trackers
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0004
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0003
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013eb5e0e0 was disconnected and freed. reset controller.
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013315d0e0 was disconnected and freed. reset controller.
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013eb5e0e0, qid=3 ctrlr is removed - aborting reqs
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013705d0e0, qid=1 ctrlr is removed - aborting reqs
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x200134ce80e0, qid=2 ctrlr is removed - aborting reqs
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013315d0e0, qid=4 ctrlr is removed - aborting reqs
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013eb5e0e0, qid=0x03: Aborting trackers
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013705d0e0, qid=0x01: Aborting trackers
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x200134ce80e0, qid=0x02: Aborting trackers
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0002
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0001
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013eb5e0e0, qid=0x0003: enter
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013705d0e0, qid=0x0001: enter
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x200134ce80e0, qid=0x0002: enter
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013315d0e0, qid=0x0004: enter
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=0
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:c8:00.0
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013705d0e0 was disconnected and freed. reset controller.
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200134ce80e0 was disconnected and freed. reset controller.
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0004
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0003
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0002
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0001
9/10/2025 10:11:03 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 10:09:22 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 10:09:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/10/2025 10:08:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/10/2025 10:08:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 10:06:52 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 10:06:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/10/2025 10:04:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/10/2025 10:04:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/10/2025 10:01:52 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 10:01:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/10/2025 10:00:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:000a p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0009 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0008 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0007 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:720:nvme_pcie_qpair_complete_tracker !!! retrying: opc=0x02
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0006 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:720:nvme_pcie_qpair_complete_tracker !!! retrying: opc=0x02
9/10/2025 10:00:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00fa810 qid=0x0000 qp_destroy_elapsed=38752us res_destroy_elapsed=15us
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0005 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0004 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0003 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0002 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion NAMESPACE NOT READY (00/82) qid:2 cid:895 cdw0:0 sqhd:0001 p:1 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:256:nvme_io_qpair_print_command !!! READ sqid:2 cid:895 nsid:1 lba:0 len:1 SGL DATA BLOCK ADDRESS 0x25c309000 len:0x200 bdf:0000:c8:00.0 wwn:002538E141500775 sn:(S7PLNG0X100199 )
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:720:nvme_pcie_qpair_complete_tracker !!! retrying: opc=0x02
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_2]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f8b90 qid=0x0001 ctrlr=0x7f52d451faf0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002c
9/10/2025 10:00:25 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f8b90, qp_create_elapsed=24038us, res_elapsed=51us
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f8b90 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00e7140 qid=0x0000 ctrlr=0x7f52d451faf0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002c
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x002c
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00e7140, qp_create_elapsed=24341us, res_elapsed=9us
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00e7140 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52e82531d0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002b
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52e82531d0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002b
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00fa810 qid=0x0000
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52e82531d0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002b
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme87n1 ENABLED, Status NVG_SUCCESS
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme87n1 for latency reports
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 12
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme87n1 at 0000:c8:00.0 [002538E141500775] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 with nsid=1
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x002b, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme87n1 eui64:002538E141500775 sn:S7PLNG0X100199 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme87n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:c8:00.0 (HotInNvme87)
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme87
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme87n1 sn:(S7PLNG0X100199 ) mn:(AELN15T4P5xnEQRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:c8:00.0 name=HotInNvme87 ctrlr=0x2001325da500
9/10/2025 10:00:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:00:22 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/10/2025 10:00:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 10:00:00 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 9:59:56 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme81n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 9:59:56 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme81n1 from latency reports
9/10/2025 9:59:56 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme81n1
9/10/2025 9:59:56 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 11
9/10/2025 9:59:56 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 9:59:56 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme81n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199 reason:unknown
9/10/2025 9:59:56 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme81n1 was hot removed
9/10/2025 9:59:56 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme81
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 17 times ***
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:c8:00.0]: unregistering thread=app_thread
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:c8:00.0
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0109910 qid=0x0001 qp_destroy_elapsed=49574us res_destroy_elapsed=109us
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ef4a0 qid=0x0000 qp_destroy_elapsed=37817us res_destroy_elapsed=17us
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 5 times ***
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52e82531d0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002b
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x002b
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00fa810, qp_create_elapsed=25486us, res_elapsed=7us
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00fa810 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52cc188fb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002a
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d0109910 qid=0x0001 ctrlr=0x7f52cc188fb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002a
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0109910 qid=0x0001
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d0109910 qid=0x0001 ctrlr=0x7f52cc188fb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002a
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00ef4a0 qid=0x0000 ctrlr=0x7f52cc188fb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002a
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ef4a0 qid=0x0000
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00ef4a0 qid=0x0000 ctrlr=0x7f52cc188fb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002a
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c8:00.0] already in failed state
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x002a, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 9:59:54 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=1
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013fa71e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c8:00.0 wwn:002538E141500775
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013232e9e0, qid=0x03: Aborting trackers
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013232e7e0, qid=0x04: Aborting trackers
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0004
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0003
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0002
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013232ebe0 was disconnected and freed. reset controller.
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013232e9e0 was disconnected and freed. reset controller.
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013232e7e0 was disconnected and freed. reset controller.
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c8:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x2001358bd0e0, qid=1 ctrlr is removed - aborting reqs
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013232ebe0, qid=2 ctrlr is removed - aborting reqs
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013232e9e0, qid=3 ctrlr is removed - aborting reqs
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c8:00.0]: qpair=0x20013232e7e0, qid=4 ctrlr is removed - aborting reqs
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x2001358bd0e0, qid=0x01: Aborting trackers
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c8:00.0], qpair=0x20013232ebe0, qid=0x02: Aborting trackers
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c8:00.0]: Freeing qid=0x0001
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x2001358bd0e0, qid=0x0001: enter
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013232ebe0, qid=0x0002: enter
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013232e9e0, qid=0x0003: enter
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c8:00.0], qpair=0x20013232e7e0, qid=0x0004: enter
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c8:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c8:00.0)
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c8:00.0: destruct=0
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:c8:00.0
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001358bd0e0 was disconnected and freed. reset controller.
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0004
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0003
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0002
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c8:00.0] About to free IO qpair qid=0x0001
9/10/2025 9:59:53 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:59:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 9:59:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 9:57:06 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 9:57:06 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":898004882, "rx_prio4_bytes_delta":5216448, "tx_prio4_bytes":5195668036, "tx_prio4_bytes_delta":39868088, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8655469, "rx_prio4_packets_delta":57567, "tx_prio4_packets":9571250, "tx_prio4_packets_delta":64971}
9/10/2025 9:57:06 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 9:57:06 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":26565, "resp_cqe_error":26828, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 9:56:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 9:56:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 9:54:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:54:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 9:51:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:51:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 9:49:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:49:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 9:46:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:46:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 9:44:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 9:44:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/10/2025 9:44:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 9:41:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:41:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 9:39:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:39:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 9:36:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:36:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 9:34:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 9:34:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/10/2025 9:34:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 9:32:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 9:31:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:31:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 9:29:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 9:29:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 9:26:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:26:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 9:26:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 9:24:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 9:24:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 9:22:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 9:22:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/10/2025 9:21:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:19:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 9:18:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/10/2025 9:18:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 9:16:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 9:16:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 9:14:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/10/2025 9:14:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/10/2025 9:14:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 9:12:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 9:12:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/10/2025 9:11:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 9:11:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 9:09:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 9:06:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 9:04:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/10/2025 9:04:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 18 times ***
9/10/2025 9:01:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 9:00:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/10/2025 9:00:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 8:59:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 8:59:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 8:57:06 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 8:57:06 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":892788434, "rx_prio4_bytes_delta":12668084, "tx_prio4_bytes":5155799948, "tx_prio4_bytes_delta":51806886, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8597902, "rx_prio4_packets_delta":106196, "tx_prio4_packets":9506279, "tx_prio4_packets_delta":114063}
9/10/2025 8:57:06 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 8:57:06 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":26565, "resp_cqe_error":26828, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 8:57:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 8:54:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 8:52:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 8:52:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 22 times ***
9/10/2025 8:49:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 8:48:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/10/2025 8:48:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 14 times ***
9/10/2025 8:46:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 8:46:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 8:46:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 8:44:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 8:42:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/10/2025 8:42:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/10/2025 8:41:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 8:40:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 8:40:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/10/2025 8:36:55 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 8:36:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 8:34:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 30 times ***
9/10/2025 8:34:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 16 times ***
9/10/2025 8:31:55 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 8:31:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/10/2025 8:30:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 8:26:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/10/2025 8:26:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 32 times ***
9/10/2025 8:26:22 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 8:24:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 8:19:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 8:19:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 8:18:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 27 times ***
9/10/2025 8:18:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/10/2025 8:16:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 8:14:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 8:14:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:09:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 8:09:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:06:55 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 8:04:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 8:04:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:01:55 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 8:01:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:59:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 7:59:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/10/2025 7:57:05 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 7:57:05 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":880120350, "rx_prio4_bytes_delta":5354994, "tx_prio4_bytes":5103993062, "tx_prio4_bytes_delta":43573792, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8491706, "rx_prio4_packets_delta":58407, "tx_prio4_packets":9392216, "tx_prio4_packets_delta":66406}
9/10/2025 7:57:05 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 7:57:05 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":26565, "resp_cqe_flush_error_delta":3354, "resp_cqe_error":26828, "resp_cqe_error_delta":3388, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 7:57:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 7:56:55 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 7:56:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 13 times ***
9/10/2025 7:56:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 34 times ***
9/10/2025 7:56:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 7:55:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 25 times ***
9/10/2025 7:55:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 29 times ***
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f1790 qid=0x0000 qp_destroy_elapsed=48265us res_destroy_elapsed=16us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d003a7c0 qid=0x0000 qp_destroy_elapsed=49204us res_destroy_elapsed=17us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f79a0 qid=0x0000 qp_destroy_elapsed=50565us res_destroy_elapsed=13us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0034070 qid=0x0000 qp_destroy_elapsed=49136us res_destroy_elapsed=17us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f58a0 qid=0x0000 qp_destroy_elapsed=50163us res_destroy_elapsed=16us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00fa810 qid=0x0000 qp_destroy_elapsed=49506us res_destroy_elapsed=16us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f8c20 qid=0x0000 qp_destroy_elapsed=31550us res_destroy_elapsed=14us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_4]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d010a900 qid=0x0001 ctrlr=0x7f52cc435050 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d010a900, qp_create_elapsed=25017us, res_elapsed=39us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010a900 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_3]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d010a4f0 qid=0x0001 ctrlr=0x7f52c85e8010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d010a4f0, qp_create_elapsed=25158us, res_elapsed=41us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010a4f0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_1]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d010a0e0 qid=0x0001 ctrlr=0x7f52e835bdb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d010a0e0, qp_create_elapsed=23990us, res_elapsed=41us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010a0e0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_2]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d0109cd0 qid=0x0001 ctrlr=0x7f52d44dad70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d0109cd0, qp_create_elapsed=23765us, res_elapsed=36us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0109cd0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_4]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d0109910 qid=0x0001 ctrlr=0x7f52cc188fb0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d0109910, qp_create_elapsed=24116us, res_elapsed=40us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0109910 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_1]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00eefb0 qid=0x0001 ctrlr=0x7f52e8158e70 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00eefb0, qp_create_elapsed=24833us, res_elapsed=41us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00eefb0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_3]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00eebf0 qid=0x0001 ctrlr=0x7f52c8205470 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00eebf0, qp_create_elapsed=24029us, res_elapsed=39us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00eebf0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_2]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00ee830 qid=0x0001 ctrlr=0x7f52d4265010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00ee830, qp_create_elapsed=23706us, res_elapsed=38us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ee830 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d00ee470 qid=0x0000 ctrlr=0x7f52cc435050 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00ee470, qp_create_elapsed=23989us, res_elapsed=15us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ee470 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00ed830 qid=0x0000 ctrlr=0x7f52c85e8010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00ed830, qp_create_elapsed=23962us, res_elapsed=9us
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ed830 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 ctrlr=0x7f52cc377c90 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d00f1790 qid=0x0000 ctrlr=0x7f52cc377c90 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f1790 qid=0x0000
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d00f1790 qid=0x0000 ctrlr=0x7f52cc377c90 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 ctrlr=0x7f52c8206f60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme86n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme86n1 for latency reports
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d003a7c0 qid=0x0000 ctrlr=0x7f52c8206f60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d003a7c0 qid=0x0000
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d003a7c0 qid=0x0000 ctrlr=0x7f52c8206f60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme85n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:55:02 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme85n1 for latency reports
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00ed470 qid=0x0000 ctrlr=0x7f52d44dad70 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00ed470, qp_create_elapsed=24191us, res_elapsed=14us
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme84n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme84n1 for latency reports
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ed470 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme83n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme83n1 for latency reports
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme82n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme82n1 for latency reports
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00ed0b0 qid=0x0000 ctrlr=0x7f52e8158e70 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00ed0b0, qp_create_elapsed=24622us, res_elapsed=9us
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme81n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme81n1 for latency reports
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ed0b0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 ctrlr=0x7f52e81f7010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme80n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme80n1 for latency reports
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00f79a0 qid=0x0000 ctrlr=0x7f52e81f7010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f79a0 qid=0x0000
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00f79a0 qid=0x0000 ctrlr=0x7f52e81f7010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 ctrlr=0x7f52cc205010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d0034070 qid=0x0000 ctrlr=0x7f52cc205010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0034070 qid=0x0000
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d0034070 qid=0x0000 ctrlr=0x7f52cc205010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme79n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme79n1 for latency reports
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 12
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme86n1 at 0000:c6:00.0 [8CE38EE30087685F] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 with nsid=1
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22, event: type=0002, info=0000, log page id=0004
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00ef4a0 qid=0x0000 ctrlr=0x7f52cc188fb0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x002a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x002a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme86n1 eui64:8CE38EE30087685F sn:YDF0A0380VY3 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme86n1 bdf:0000:c6:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 eui64:8CE38EE30087685F sn:YDF0A0380VY3
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00ef4a0, qp_create_elapsed=24030us, res_elapsed=9us
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme85n1 at 0000:84:00.0 [8CE38EE2028DDB01] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 with nsid=1
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23, event: type=0002, info=0000, log page id=0004
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ef4a0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme85n1 eui64:8CE38EE2028DDB01 sn:70H0A00GTBU7 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme85n1 bdf:0000:84:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 eui64:8CE38EE2028DDB01 sn:70H0A00GTBU7
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme84n1 at 0000:88:00.0 [8CE38EE300B00E06] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 with nsid=1
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15, event: type=0002, info=0000, log page id=0004
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00ecf50 qid=0x0000 ctrlr=0x7f52c8205470 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00ecf50, qp_create_elapsed=23713us, res_elapsed=9us
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme84n1 eui64:8CE38EE300B00E06 sn:1EE0A0T603R3 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme84n1 bdf:0000:88:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 eui64:8CE38EE300B00E06 sn:1EE0A0T603R3
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ecf50 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 ctrlr=0x7f52c8414850 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0029
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme83n1 at 0000:86:00.0 [002538E7415006FE] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 with nsid=1
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f58a0 qid=0x0000 ctrlr=0x7f52c8414850 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0029
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f58a0 qid=0x0000
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 qpair=0x7f52d00f58a0 qid=0x0000 ctrlr=0x7f52c8414850 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0029
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 ctrlr=0x7f52c8186f90 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52c8186f90 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00fa810 qid=0x0000
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52c8186f90 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19, event: type=0002, info=0000, log page id=0004
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme83n1 eui64:002538E7415006FE sn:S7Z9NG0X700006 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme83n1 bdf:0000:86:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 eui64:002538E7415006FE sn:S7Z9NG0X700006
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme82n1 at 0000:ca:00.0 [002538E141500853] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 with nsid=1
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14, event: type=0002, info=0000, log page id=0004
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d4265010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f3910, qp_create_elapsed=24212us, res_elapsed=10us
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme82n1 eui64:002538E141500853 sn:S7PNNG0X100108 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme82n1 bdf:0000:ca:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 eui64:002538E141500853 sn:S7PNNG0X100108
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f3910 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 ctrlr=0x7f52d42ed3c0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00f8c20 qid=0x0000 ctrlr=0x7f52d42ed3c0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f8c20 qid=0x0000
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00f8c20 qid=0x0000 ctrlr=0x7f52d42ed3c0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme81n1 at 0000:c8:00.0 [002538E141500775] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 with nsid=1
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0029, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18, event: type=0002, info=0000, log page id=0004
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f3910 qid=0x0000 qp_destroy_elapsed=51628us res_destroy_elapsed=17us
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c8:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme81n1 eui64:002538E141500775 sn:S7PLNG0X100199 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme81n1 bdf:0000:c8:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay18 eui64:002538E141500775 sn:S7PLNG0X100199
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme80n1 at 0000:8a:00.0 [002538E141500763] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 with nsid=1
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11, event: type=0002, info=0000, log page id=0004
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00ecb40 qid=0x0000 ctrlr=0x7f52e835bdb0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00ecb40, qp_create_elapsed=24846us, res_elapsed=7us
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme80n1 eui64:002538E141500763 sn:S7PLNG0X100181 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme80n1 bdf:0000:8a:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 eui64:002538E141500763 sn:S7PLNG0X100181
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00ecb40 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 ctrlr=0x7f52d4265010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d4265010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f3910 qid=0x0000
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d4265010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme79n1 at 0000:85:00.0 [0025389401006B55] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 with nsid=1
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21, event: type=0002, info=0000, log page id=0004
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme79n1 eui64:0025389401006B55 sn:S5URNE0N400075 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme79n1 bdf:0000:85:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 eui64:0025389401006B55 sn:S5URNE0N400075
9/10/2025 7:55:01 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:c6:00.0 (HotInNvme86)
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme86
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme86n1 sn:(YDF0A0380VY3 ) mn:(KCM730T7P5xnETRI ) blocklen:512 blockcnt:60001615872 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:10 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:c6:00.0 name=HotInNvme86 ctrlr=0x2001393c8400
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:84:00.0 (HotInNvme85)
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:84:00.0 name=HotInNvme85 ctrlr=0x20013eb46500
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme85
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme85n1 sn:(70H0A00GTBU7 ) mn:(TCM61920P5xnFTRI ) blocklen:512 blockcnt:3750748848 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:7 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:13 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:c6:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:c6:00.0] !!! Setting CC.EN = 1
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:84:00.0] !!! Setting CC.EN = 1
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:84:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:c6:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:84:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:c6:00
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:c6:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:c6:00.0
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:c6:00.0
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:84:00
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:84:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:84:00.0
9/10/2025 7:55:00 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:84:00.0
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:88:00.0 (HotInNvme84)
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:88:00.0 name=HotInNvme84 ctrlr=0x200132874500
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme84
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme84n1 sn:(1EE0A0T603R3 ) mn:(KCM71920P5xnETRI ) blocklen:512 blockcnt:3750748848 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:10 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:ca:00.0 (HotInNvme82)
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:86:00.0 (HotInNvme83)
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:ca:00.0 name=HotInNvme82 ctrlr=0x2001393d8400
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:86:00.0 name=HotInNvme83 ctrlr=0x20013bb14480
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme83
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme82
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme83n1 sn:(S7Z9NG0X700006 ) mn:(AELN61T4P5xnEQRI ) blocklen:512 blockcnt:120001134592 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:31 ns.npwa:31 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme82n1 sn:(S7PNNG0X100108 ) mn:(AELN30T7P5xnEQRI ) blocklen:512 blockcnt:60001615872 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:c8:00.0 (HotInNvme81)
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:8a:00.0 (HotInNvme80)
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:c8:00.0 name=HotInNvme81 ctrlr=0x20013b34a500
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:8a:00.0 name=HotInNvme80 ctrlr=0x2001358b5500
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme81
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme80
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme81n1 sn:(S7PLNG0X100199 ) mn:(AELN15T4P5xnEQRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme80n1 sn:(S7PLNG0X100181 ) mn:(AELN15T4P5xnEQRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:88:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:85:00.0 (HotInNvme79)
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:85:00.0 name=HotInNvme79 ctrlr=0x200132577480
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme79
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme79n1 sn:(S5URNE0N400075 ) mn:(AEGL1920P5xnFTRI ) blocklen:512 blockcnt:3750748848 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:0 ns.npwg:0 ns.npwa:0 ns.nows:0 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:85:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:ca:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:8a:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:86:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:ca:00.0] !!! Setting CC.EN = 1
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:8a:00.0] !!! Setting CC.EN = 1
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:88:00.0] !!! Setting CC.EN = 1
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:86:00.0] !!! Setting CC.EN = 1
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:85:00.0] !!! Setting CC.EN = 1
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:ca:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:8a:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:88:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:86:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:85:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:ca:00
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:ca:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:ca:00.0
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:ca:00.0
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0033cb0 qid=0x0000 qp_destroy_elapsed=49107us res_destroy_elapsed=16us
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f6780 qid=0x0000 qp_destroy_elapsed=37517us res_destroy_elapsed=17us
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:8a:00
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8a:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:8a:00.0
9/10/2025 7:54:59 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:8a:00.0
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:88:00
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:88:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:88:00.0
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:88:00.0
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:86:00
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:86:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:86:00.0
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:86:00.0
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_2]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f2680 qid=0x0001 ctrlr=0x7f52d41be0f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000f
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f2680, qp_create_elapsed=24119us, res_elapsed=37us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f2680 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_4]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d003a4a0 qid=0x0001 ctrlr=0x7f52cc19c010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d003a4a0, qp_create_elapsed=24708us, res_elapsed=38us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d003a4a0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_3]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d003abc0 qid=0x0001 ctrlr=0x7f52c8316b00 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d003abc0, qp_create_elapsed=24053us, res_elapsed=37us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d003abc0 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:85:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:85:00.0
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:85:00
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:85:00.0
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:719:nvmf_ctrlr_add_io_qpair !!! [nvmf_tgt_poll_group_1]: Connected (IO). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d0035320 qid=0x0001 ctrlr=0x7f52e8186ea0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d0035320, qp_create_elapsed=23850us, res_elapsed=36us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0035320 Final Negotiated Queue Depth=256 R/W Depth=16
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d0106220 qid=0x0000 ctrlr=0x7f52cc19c010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d0106220, qp_create_elapsed=24093us, res_elapsed=10us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0106220 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52c8316b00 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000a
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000a
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00f5c20, qp_create_elapsed=23775us, res_elapsed=6us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f5c20 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52cc1cc2f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0033cb0 qid=0x0000
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52cc1cc2f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 ctrlr=0x7f52cc1cc2f0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00f6780 qid=0x0000 ctrlr=0x7f52c82b1010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f6780 qid=0x0000
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0106220 qid=0x0000 qp_destroy_elapsed=46424us res_destroy_elapsed=13us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00f6780 qid=0x0000 ctrlr=0x7f52c82b1010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 ctrlr=0x7f52c82b1010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme78n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme78n1 for latency reports
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f5c20 qid=0x0000 qp_destroy_elapsed=51655us res_destroy_elapsed=17us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme77n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme77n1 for latency reports
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d010f0b0 qid=0x0000 ctrlr=0x7f52d41be0f0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000f
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000f
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d010f0b0, qp_create_elapsed=24347us, res_elapsed=11us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme76n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme76n1 for latency reports
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d010f0b0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:33:Enable latency histogram for bdev HotInNvme75n1 ENABLED, Status NVG_SUCCESS
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:245:operator() Added bdev HotInNvme75n1 for latency reports
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 4
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme77n1 at 0000:05:00.0 [8CE38EE202891501] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 with nsid=1
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01, event: type=0002, info=0000, log page id=0004
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d01074c0 qid=0x0000 ctrlr=0x7f52e8186ea0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d01074c0, qp_create_elapsed=25086us, res_elapsed=6us
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme77n1 eui64:8CE38EE202891501 sn:70G0A00FTBX7 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme77n1 bdf:0000:05:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 eui64:8CE38EE202891501 sn:70G0A00FTBX7
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d01074c0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 ctrlr=0x7f52e82d3530 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000e
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d0106220 qid=0x0000 ctrlr=0x7f52e82d3530 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000e
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0106220 qid=0x0000
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d0106220 qid=0x0000 ctrlr=0x7f52e82d3530 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000e
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 ctrlr=0x7f52d42d4a60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52d42d4a60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f5c20 qid=0x0000
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52d42d4a60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme78n1 at 0000:07:00.0 [8CE38EE300FF11DF] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 with nsid=1
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0009, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03, event: type=0002, info=0000, log page id=0004
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme78n1 eui64:8CE38EE300FF11DF sn:2ET0A01D0AW4 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme78n1 bdf:0000:07:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 eui64:8CE38EE300FF11DF sn:2ET0A01D0AW4
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme76n1 at 0000:8b:00.0 [002538E9410026BC] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 with nsid=1
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000e, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09, event: type=0002, info=0000, log page id=0004
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme76n1 eui64:002538E9410026BC sn:S7WBNE0X900403 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09
9/10/2025 7:54:58 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme76n1 bdf:0000:8b:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 eui64:002538E9410026BC sn:S7WBNE0X900403
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:257:Scan Added HotInNvme75n1 at 0000:04:00.0 [0000000000000000] to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 with nsid=1
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 4 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 3 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 2 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00, event: type=0002, info=0000, log page id=0004
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 1 max_completions_cap = 128 num_trackers = 896 num_entries = 1024
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:242:Scan Adding HotInNvme75n1 eui64:0000000000000000 sn:S80DNG0Y200115 to subsystem nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:235:Scan Device added: bdev:HotInNvme75n1 bdf:0000:04:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 eui64:0000000000000000 sn:S80DNG0Y200115
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:07:00.0 (HotInNvme78)
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:05:00.0 (HotInNvme77)
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:07:00.0 name=HotInNvme78 ctrlr=0x200132432500
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:05:00.0 name=HotInNvme77 ctrlr=0x200139fc2400
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme78
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme77
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme78n1 sn:(2ET0A01D0AW4 ) mn:(KCM77680P5AxFTRI ) blocklen:512 blockcnt:15002931888 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:10 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme77n1 sn:(70G0A00FTBX7 ) mn:(TCM615T4P5xnFTRI ) blocklen:512 blockcnt:30001856512 phys_blocklen:4096 sector_size:512 atomic_bs:32768 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:7 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:63 ctrlr.mtds:13 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:8b:00.0 (HotInNvme76)
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:8b:00.0 name=HotInNvme76 ctrlr=0x200132329340
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme76
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme76n1 sn:(S7WBNE0X900403 ) mn:(AELN30T7P5xnFQRI ) blocklen:512 blockcnt:60001615872 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:9 (2060288) optimal_io_boundary:0 max_segment_size:2060288 max_num_segments:256
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:07:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:8b:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:8b:00.0] !!! Setting CC.EN = 1
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:07:00.0] !!! Setting CC.EN = 1
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:05:00.0] !!! Setting CC.EN = 1
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:05:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:8b:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:07:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:05:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:8b:00
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:8b:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:8b:00.0
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:8b:00.0
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:07:00
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:07:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:07:00.0
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:07:00.0
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:05:00
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:05:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:05:00.0
9/10/2025 7:54:57 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:05:00.0
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app bdev_nvme.c:5555:_attach_cb !!! [app_thread] Attached to 0000:04:00.0 (HotInNvme75)
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app bdev_nvme.c:5547:_attach_cb !!! [app_thread] Attaching to traddr=0000:04:00.0 name=HotInNvme75 ctrlr=0x200138019340
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app bdev_nvme.c:5434:nvme_ctrlr_create !!! [app_thread] Creating controller: HotInNvme75
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app bdev_nvme.c:4026:nvme_disk_create !!! bdev:HotInNvme75n1 sn:(S80DNG0Y200115 ) mn:(SAMSUNG MZ3MO15THCLC-00AW7 ) blocklen:512 blockcnt:30001856512 phys_blocklen:512 sector_size:512 atomic_bs:512 nsfeat.optperf:1 ns.npwg:7 ns.npwa:7 ns.nows:255 ns.niob:0 ns.lbads:9 (512) ns_atomic_write_unit:0 ns.nawun:0 ns.nawupf:0 ns.nabspf:0 ns.nabsn:0 ns.nabo:0 ctrlr.awupf:0 ctrlr.mtds:6 (262144) optimal_io_boundary:0 max_segment_size:262144 max_num_segments:256
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3870:nvme_ctrlr_process_init_enable_wait_for_ready_1 [0000:04:00.0] !!! Enabled: CC.EN = 1 && CSTS.RDY = 1 - controller is ready
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:136:nvme_pcie_qpair_construct [0000:04:00.0] qid = 0 max_completions_cap = 128 num_trackers = 384 num_entries = 512
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app nvme_pcie.c:996:nvme_pcie_ctrlr_construct !!! Constructed SSD: 0000:04:00.0
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4048:nvme_ctrlr_process_init [0000:04:00.0] !!! Setting CC.EN = 1
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:3828:nvme_ctrlr_process_init_wait_for_ready_0 [0000:04:00.0] !!! Disabled: CC.EN = 0 && CSTS.RDY = 0
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app pci.c:380:pci_device_init !!! init: 0000:04:00
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app nvme_pcie.c:911:nvme_pcie_ctrlr_construct !!! Constructing SSD: 0000:04:00.0
9/10/2025 7:54:56 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/10/2025 7:54:22 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/10/2025 7:54:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/10/2025 7:53:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/10/2025 7:53:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 0
9/10/2025 7:51:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:51:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:49:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:49:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:48:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:48:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:46:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:46:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:44:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:44:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:41:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:41:12 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 7:41:12 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 7:41:12 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 7:41:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:40:56 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:40:54 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 7:40:54 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 7:40:54 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:40:53 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:40:52 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/10/2025 7:40:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 7:40:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 7:40:25 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 7:40:25 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 7:40:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/10/2025 7:40:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/10/2025 7:39:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 33 times ***
9/10/2025 7:39:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:37:54 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/10/2025 7:37:41 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 0
9/10/2025 7:37:41 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:84:00.0 with driver=nvme
9/10/2025 7:37:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme74n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme74n1 from latency reports
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme74n1
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 0
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:84:00.0 is missing
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme74n1 bdf:0000:84:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 eui64:8CE38EE2028DDB01 sn:70H0A00GTBU7 reason:unknown
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme74n1 was hot removed
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c6:00.0 with driver=nvme
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:85:00.0 with driver=nvme
9/10/2025 7:37:38 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:37:37 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ebca0 qid=0x0001 qp_destroy_elapsed=49698us res_destroy_elapsed=111us
9/10/2025 7:37:37 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0105760 qid=0x0000 qp_destroy_elapsed=38326us res_destroy_elapsed=13us
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme74
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:84:00
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0000, state=6
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0000
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 27 times ***
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:84:00.0] Prepare to destruct SSD: 0000:84:00.0
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:84:00.0] already in failed state
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:84:00.0]: unregistering thread=app_thread
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:84:00.0
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:84:00.0)
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:84:00.0: destruct=1
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d00f1790 qid=0x0000 ctrlr=0x7f52cc377c90 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d00f1790, qp_create_elapsed=24194us, res_elapsed=10us
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f1790 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d00ebca0 qid=0x0001 ctrlr=0x7f52e833b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ebca0 qid=0x0001
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d00ebca0 qid=0x0001 ctrlr=0x7f52e833b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 ctrlr=0x7f52e833b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d0105760 qid=0x0000 ctrlr=0x7f52e833b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0105760 qid=0x0000
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23 qpair=0x7f52d0105760 qid=0x0000 ctrlr=0x7f52e833b010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 6 times ***
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 11 times ***
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:84:00.0] already in failed state
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay23, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:84:00.0)
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:84:00.0: destruct=1
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00eb8e0 qid=0x0001 qp_destroy_elapsed=40484us res_destroy_elapsed=109us
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00370f0 qid=0x0000 qp_destroy_elapsed=38001us res_destroy_elapsed=14us
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 20 times ***
9/10/2025 7:37:36 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013241bc60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:84:00.0 wwn:8CE38EE2028DDB01
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0003
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013d38e360 was disconnected and freed. reset controller.
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0003, state=6
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0002, state=6
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0000, state=5
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x20013d38e360, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x20013d38e360, qid=0x03: Aborting trackers
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x200139f7fe60, qid=0x04: Aborting trackers
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x200132362360, qid=0x01: Aborting trackers
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:84:00.0], qpair=0x200132362160, qid=0x02: Aborting trackers
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0004
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0002
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:84:00.0]: Freeing qid=0x0001
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x20013d38e360, qid=0x0003: enter
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:84:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:84:00.0)
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:84:00.0: destruct=0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:84:00.0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200139f7fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132362360 was disconnected and freed. reset controller.
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132362160 was disconnected and freed. reset controller.
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0004, state=6
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:84:00.0 cntlid=0x0001 qid=0x0001, state=6
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200139f7fe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200132362360, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:84:00.0]: qpair=0x200132362160, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x200139f7fe60, qid=0x0004: enter
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x200132362360, qid=0x0001: enter
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:84:00.0], qpair=0x200132362160, qid=0x0002: enter
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:84:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme73n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme73n1 from latency reports
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme73n1
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme68n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme68n1 from latency reports
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme68n1
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 1
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c6:00.0 is missing
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:85:00.0 is missing
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme73n1 bdf:0000:c6:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 eui64:8CE38EE30087685F sn:YDF0A0380VY3 reason:unknown
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme68n1 bdf:0000:85:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 eui64:0025389401006B55 sn:S5URNE0N400075 reason:unknown
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme73n1 was hot removed
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme68n1 was hot removed
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:ca:00.0 with driver=nvme
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:88:00.0 with driver=nvme
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:86:00.0 with driver=nvme
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:c6:00
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme73
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0000, state=6
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0000
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 24 times ***
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:c6:00.0] Prepare to destruct SSD: 0000:c6:00.0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c6:00.0] already in failed state
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:c6:00.0]: unregistering thread=app_thread
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:c6:00.0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c6:00.0)
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c6:00.0: destruct=1
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d003a7c0 qid=0x0000 ctrlr=0x7f52c8206f60 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d003a7c0, qp_create_elapsed=23987us, res_elapsed=8us
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d003a7c0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00eb8e0 qid=0x0001 ctrlr=0x7f52cc1b5f80 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00eb8e0 qid=0x0001
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00eb8e0 qid=0x0001 ctrlr=0x7f52cc1b5f80 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 ctrlr=0x7f52cc1b5f80 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00370f0 qid=0x0000 ctrlr=0x7f52cc1b5f80 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00370f0 qid=0x0000
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22 qpair=0x7f52d00370f0 qid=0x0000 ctrlr=0x7f52cc1b5f80 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:c6:00.0] already in failed state
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay22, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c6:00.0)
9/10/2025 7:37:35 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c6:00.0: destruct=1
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013bb142e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:c6:00.0 wwn:8CE38EE30087685F
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x2001393afe60, qid=0x04: Aborting trackers
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0004
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0002
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001409bfe60 was disconnected and freed. reset controller.
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001393afe60 was disconnected and freed. reset controller.
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0004, state=6
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0003, state=6
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0002, state=6
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0001, state=6
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:c6:00.0 cntlid=0x0001 qid=0x0000, state=5
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001409bfe60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x20013979fe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x2001393afe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:c6:00.0]: qpair=0x200134ccfe60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x2001409bfe60, qid=0x02: Aborting trackers
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x20013979fe60, qid=0x03: Aborting trackers
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:c6:00.0], qpair=0x200134ccfe60, qid=0x01: Aborting trackers
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0003
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:c6:00.0]: Freeing qid=0x0001
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x2001409bfe60, qid=0x0002: enter
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x20013979fe60, qid=0x0003: enter
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x2001393afe60, qid=0x0004: enter
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:c6:00.0], qpair=0x200134ccfe60, qid=0x0001: enter
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:c6:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:c6:00.0)
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:c6:00.0: destruct=0
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:c6:00.0
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013979fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200134ccfe60 was disconnected and freed. reset controller.
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:c6:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme68
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:85:00
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0000, state=6
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0000
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 19 times ***
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:85:00.0] Prepare to destruct SSD: 0000:85:00.0
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:85:00.0] already in failed state
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:85:00.0]: unregistering thread=app_thread
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:85:00.0
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:85:00.0)
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:85:00.0: destruct=1
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00eb520 qid=0x0001 qp_destroy_elapsed=37447us res_destroy_elapsed=113us
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 1 time ***
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:34 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d010d020 qid=0x0000 qp_destroy_elapsed=50179us res_destroy_elapsed=15us
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d4265010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f3910, qp_create_elapsed=24743us, res_elapsed=9us
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f3910 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 ctrlr=0x7f52e816be40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00eb520 qid=0x0001 ctrlr=0x7f52e816be40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00eb520 qid=0x0001
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d00eb520 qid=0x0001 ctrlr=0x7f52e816be40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d010d020 qid=0x0000 ctrlr=0x7f52e816be40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d010d020 qid=0x0000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21 qpair=0x7f52d010d020 qid=0x0000 ctrlr=0x7f52e816be40 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 6 times ***
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:85:00.0] already in failed state
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay21, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:85:00.0)
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:85:00.0: destruct=1
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200134ce80e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:85:00.0 wwn:0025389401006B55
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x200140b6fe60, qid=0x03: Aborting trackers
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x20013742fe60, qid=0x04: Aborting trackers
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0004
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0003
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0002
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200140b6fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013742fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013352fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0004, state=6
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0003, state=6
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0002, state=6
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0001, state=6
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:85:00.0 cntlid=0x0041 qid=0x0000, state=5
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x200140b6fe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x20013742fe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x20013352fe60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:85:00.0]: qpair=0x20013241be60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x20013352fe60, qid=0x02: Aborting trackers
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:85:00.0], qpair=0x20013241be60, qid=0x01: Aborting trackers
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:85:00.0]: Freeing qid=0x0001
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x200140b6fe60, qid=0x0003: enter
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x20013742fe60, qid=0x0004: enter
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x20013352fe60, qid=0x0002: enter
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:85:00.0], qpair=0x20013241be60, qid=0x0001: enter
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:85:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:85:00.0)
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:85:00.0: destruct=0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:85:00.0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013241be60 was disconnected and freed. reset controller.
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:85:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme71
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:86:00
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0000
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:86:00.0] Prepare to destruct SSD: 0000:86:00.0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:86:00.0] already in failed state
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:86:00.0]: unregistering thread=app_thread
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:86:00.0
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:86:00.0)
9/10/2025 7:37:33 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:86:00.0: destruct=1
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0039dd0 qid=0x0001 qp_destroy_elapsed=48133us res_destroy_elapsed=112us
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f1790 qid=0x0000 qp_destroy_elapsed=37561us res_destroy_elapsed=21us
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 12 times ***
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 1 time ***
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme72n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme72n1 from latency reports
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme72n1
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00f79a0 qid=0x0000 ctrlr=0x7f52e81f7010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme71n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme71n1 from latency reports
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d00f79a0, qp_create_elapsed=25025us, res_elapsed=7us
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme71n1
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme70n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme70n1 from latency reports
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f79a0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme70n1
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 3
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:ca:00.0 is missing
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:88:00.0 is missing
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:86:00.0 is missing
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme72n1 bdf:0000:88:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 eui64:8CE38EE300B00E06 sn:1EE0A0T603R3 reason:unknown
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme71n1 bdf:0000:86:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 eui64:002538E7415006FE sn:S7Z9NG0X700006 reason:unknown
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme70n1 bdf:0000:ca:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 eui64:002538E141500853 sn:S7PNNG0X100108 reason:unknown
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme72n1 was hot removed
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme71n1 was hot removed
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme70n1 was hot removed
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:8b:00.0 with driver=nvme
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:8a:00.0 with driver=nvme
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:07:00.0 with driver=nvme
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d0039dd0 qid=0x0001 ctrlr=0x7f52c82d2920 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0039dd0 qid=0x0001
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d0039dd0 qid=0x0001 ctrlr=0x7f52c82d2920 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 ctrlr=0x7f52c82d2920 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00f1790 qid=0x0000 ctrlr=0x7f52c82d2920 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f1790 qid=0x0000
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19 qpair=0x7f52d00f1790 qid=0x0000 ctrlr=0x7f52c82d2920 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:86:00.0] already in failed state
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay19, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:86:00.0)
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:86:00.0: destruct=1
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013e6f1260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:86:00.0 wwn:002538E7415006FE
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x2001405dfe60, qid=0x02: Aborting trackers
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0003
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0002
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001405dfe60 was disconnected and freed. reset controller.
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200138bcfe60 was disconnected and freed. reset controller.
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x200138bcfe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x20013fdffe60, qid=0x01: Aborting trackers
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x200138bcfe60, qid=0x03: Aborting trackers
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:86:00.0], qpair=0x200133d0fe60, qid=0x04: Aborting trackers
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0004
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:86:00.0]: Freeing qid=0x0001
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x2001405dfe60, qid=0x0002: enter
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x200138bcfe60, qid=0x0003: enter
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:86:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:86:00.0)
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:86:00.0: destruct=0
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:86:00.0
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013fdffe60 was disconnected and freed. reset controller.
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200133d0fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:86:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x2001405dfe60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x20013fdffe60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:86:00.0]: qpair=0x200133d0fe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x20013fdffe60, qid=0x0001: enter
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:86:00.0], qpair=0x200133d0fe60, qid=0x0004: enter
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:32 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:86:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme72
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0039a10 qid=0x0001 qp_destroy_elapsed=49969us res_destroy_elapsed=110us
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:88:00
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0000, state=6
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0000
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:88:00.0] Prepare to destruct SSD: 0000:88:00.0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:88:00.0]: unregistering thread=app_thread
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:88:00.0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 1 time ***
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:88:00.0] already in failed state
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:88:00.0)
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:88:00.0: destruct=1
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f3910 qid=0x0000 qp_destroy_elapsed=38216us res_destroy_elapsed=16us
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 23 times ***
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d0034070 qid=0x0000 ctrlr=0x7f52cc205010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d0034070, qp_create_elapsed=24517us, res_elapsed=10us
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0034070 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 ctrlr=0x7f52d4237010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d0039a10 qid=0x0001 ctrlr=0x7f52d4237010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0039a10 qid=0x0001
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d0039a10 qid=0x0001 ctrlr=0x7f52d4237010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d4237010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f3910 qid=0x0000
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15 qpair=0x7f52d00f3910 qid=0x0000 ctrlr=0x7f52d4237010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 7 times ***
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:88:00.0] already in failed state
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay15, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:88:00.0)
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:88:00.0: destruct=1
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001393d8260, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:88:00.0 wwn:8CE38EE300B00E06
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x200137c0fe60, qid=0x04: Aborting trackers
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0004
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0002
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200137c0fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132577e60 was disconnected and freed. reset controller.
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0004, state=6
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0003, state=6
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0002, state=6
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0001, state=6
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:88:00.0 cntlid=0x0001 qid=0x0000, state=5
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200137c0fe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x2001358b8e60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x20013391fe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:88:00.0]: qpair=0x200132577e60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x2001358b8e60, qid=0x01: Aborting trackers
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x20013391fe60, qid=0x03: Aborting trackers
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:88:00.0], qpair=0x200132577e60, qid=0x02: Aborting trackers
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0003
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:88:00.0]: Freeing qid=0x0001
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x200137c0fe60, qid=0x0004: enter
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x2001358b8e60, qid=0x0001: enter
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x20013391fe60, qid=0x0003: enter
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:88:00.0], qpair=0x200132577e60, qid=0x0002: enter
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:88:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:88:00.0)
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:88:00.0: destruct=0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:88:00.0
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001358b8e60 was disconnected and freed. reset controller.
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013391fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:31 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:88:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0039650 qid=0x0001 qp_destroy_elapsed=49569us res_destroy_elapsed=119us
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0039290 qid=0x0000 qp_destroy_elapsed=38051us res_destroy_elapsed=19us
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme70
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:ca:00
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0000
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 25 times ***
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013648a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:ca:00.0] Prepare to destruct SSD: 0000:ca:00.0
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:ca:00.0] already in failed state
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:ca:00.0]: unregistering thread=app_thread
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:ca:00.0
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:ca:00.0)
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:ca:00.0: destruct=1
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013648a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d00fa810 qid=0x0000 ctrlr=0x7f52c8186f90 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00fa810, qp_create_elapsed=24094us, res_elapsed=9us
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013648a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00fa810 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013648a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d0039650 qid=0x0001 ctrlr=0x7f52cc320ef0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0039650 qid=0x0001
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d0039650 qid=0x0001 ctrlr=0x7f52cc320ef0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 ctrlr=0x7f52cc320ef0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013648a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d0039290 qid=0x0000 ctrlr=0x7f52cc320ef0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0039290 qid=0x0000
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14 qpair=0x7f52d0039290 qid=0x0000 ctrlr=0x7f52cc320ef0 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 8 times ***
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013648a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:ca:00.0] already in failed state
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay14, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:ca:00.0)
9/10/2025 7:37:30 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:ca:00.0: destruct=1
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013648a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013648a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:ca:00.0 wwn:002538E141500853
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x200138fbfe60, qid=0x04: Aborting trackers
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0004
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0002
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200138fbfe60 was disconnected and freed. reset controller.
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001348dfe60 was disconnected and freed. reset controller.
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:ca:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x20013de7fe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200138fbfe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x200137055360, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:ca:00.0]: qpair=0x2001348dfe60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x20013de7fe60, qid=0x03: Aborting trackers
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x200137055360, qid=0x01: Aborting trackers
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:ca:00.0], qpair=0x2001348dfe60, qid=0x02: Aborting trackers
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0003
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:ca:00.0]: Freeing qid=0x0001
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x20013de7fe60, qid=0x0003: enter
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x200138fbfe60, qid=0x0004: enter
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x200137055360, qid=0x0001: enter
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:ca:00.0], qpair=0x2001348dfe60, qid=0x0002: enter
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:ca:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:ca:00.0)
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:ca:00.0: destruct=0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:ca:00.0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013de7fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200137055360 was disconnected and freed. reset controller.
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:ca:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d003ac60 qid=0x0001 qp_destroy_elapsed=72399us res_destroy_elapsed=111us
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme69n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme69n1 from latency reports
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme69n1
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme67n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme67n1 from latency reports
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme67n1
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ed920 qid=0x0000 qp_destroy_elapsed=33545us res_destroy_elapsed=16us
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme65n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme65n1 from latency reports
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme65n1
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 6
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:8b:00.0 is missing
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:8a:00.0 is missing
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:07:00.0 is missing
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme69n1 bdf:0000:8a:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 eui64:002538E141500763 sn:S7PLNG0X100181 reason:unknown
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme67n1 bdf:0000:07:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 eui64:8CE38EE300FF11DF sn:2ET0A01D0AW4 reason:unknown
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme65n1 bdf:0000:8b:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 eui64:002538E9410026BC sn:S7WBNE0X900403 reason:unknown
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme69n1 was hot removed
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme67n1 was hot removed
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme65n1 was hot removed
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:05:00.0 with driver=nvme
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:04:00.0 with driver=nvme
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme69
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:8a:00
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200141767e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:8a:00.0] Prepare to destruct SSD: 0000:8a:00.0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8a:00.0] already in failed state
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:8a:00.0]: unregistering thread=app_thread
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:8a:00.0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8a:00.0)
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8a:00.0: destruct=1
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 26 times ***
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200141767e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00f8c20 qid=0x0000 ctrlr=0x7f52d42ed3c0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f8c20, qp_create_elapsed=24520us, res_elapsed=8us
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200141767e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f8c20 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d003ac60 qid=0x0001 ctrlr=0x7f52c82cb130 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d003ac60 qid=0x0001 ctrlr=0x7f52c82cb130 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200141767e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 ctrlr=0x7f52c82cb130 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d003ac60 qid=0x0001
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00ed920 qid=0x0000 ctrlr=0x7f52c82cb130 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ed920 qid=0x0000
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11 qpair=0x7f52d00ed920 qid=0x0000 ctrlr=0x7f52c82cb130 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 6 times ***
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200141767e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200141767e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8a:00.0] already in failed state
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay11, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8a:00.0)
9/10/2025 7:37:29 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8a:00.0: destruct=1
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200141767e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200141767e60, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8a:00.0 wwn:002538E141500763
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x2001401efe60, qid=0x04: Aborting trackers
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x2001397b8ee0, qid=0x02: Aborting trackers
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0004
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0003
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0002
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001401efe60 was disconnected and freed. reset controller.
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001397b8ee0 was disconnected and freed. reset controller.
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013781fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8a:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001401efe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x2001397b8ee0, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x20013781fe60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8a:00.0]: qpair=0x200136c4fe60, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x20013781fe60, qid=0x03: Aborting trackers
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8a:00.0], qpair=0x200136c4fe60, qid=0x01: Aborting trackers
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8a:00.0]: Freeing qid=0x0001
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x2001401efe60, qid=0x0004: enter
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x2001397b8ee0, qid=0x0002: enter
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x20013781fe60, qid=0x0003: enter
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8a:00.0], qpair=0x200136c4fe60, qid=0x0001: enter
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8a:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8a:00.0)
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8a:00.0: destruct=0
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:8a:00.0
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200136c4fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8a:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f30b0 qid=0x0001 qp_destroy_elapsed=48894us res_destroy_elapsed=113us
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00f5240 qid=0x0000 qp_destroy_elapsed=37687us res_destroy_elapsed=16us
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme65
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:8b:00
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0000
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 26 times ***
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200140b60ee0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:8b:00.0] Prepare to destruct SSD: 0000:8b:00.0
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8b:00.0] already in failed state
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:8b:00.0]: unregistering thread=app_thread
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:8b:00.0
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8b:00.0)
9/10/2025 7:37:28 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8b:00.0: destruct=1
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200140b60ee0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_1]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d0106220 qid=0x0000 ctrlr=0x7f52e82d3530 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000e
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x000e
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_1] qpair=0x7f52d0106220, qp_create_elapsed=24659us, res_elapsed=4us
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200140b60ee0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0106220 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200140b60ee0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_4]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 ctrlr=0x7f52cc205010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f30b0 qid=0x0001 ctrlr=0x7f52cc205010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f30b0 qid=0x0001
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f30b0 qid=0x0001 ctrlr=0x7f52cc205010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200140b60ee0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f5240 qid=0x0000 ctrlr=0x7f52cc205010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00f5240 qid=0x0000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09 qpair=0x7f52d00f5240 qid=0x0000 ctrlr=0x7f52cc205010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x000d
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 7 times ***
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200140b60ee0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 33 times ***
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:8b:00.0] already in failed state
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x000d, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay09, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8b:00.0)
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8b:00.0: destruct=1
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00fa810 qid=0x0001 qp_destroy_elapsed=49079us res_destroy_elapsed=114us
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200140b60ee0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ed220 qid=0x0000 qp_destroy_elapsed=38186us res_destroy_elapsed=15us
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200140b60ee0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200140b60ee0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:374 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:8b:00.0 wwn:002538E9410026BC
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:374 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20014179ee60, qid=0x03: Aborting trackers
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20014179ec60, qid=0x04: Aborting trackers
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x20013cc91ee0, qid=0x01: Aborting trackers
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0004
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0003
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0002
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:8b:00.0]: Freeing qid=0x0001
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:8b:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:8b:00.0: destruct=0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:8b:00.0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20014179ee60 was disconnected and freed. reset controller.
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20014179ec60 was disconnected and freed. reset controller.
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013cc91ee0 was disconnected and freed. reset controller.
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200136093d20 was disconnected and freed. reset controller.
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:8b:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014179ee60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20014179ec60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x20013cc91ee0, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:8b:00.0]: qpair=0x200136093d20, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:8b:00.0], qpair=0x200136093d20, qid=0x02: Aborting trackers
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20014179ee60, qid=0x0003: enter
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20014179ec60, qid=0x0004: enter
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x20013cc91ee0, qid=0x0001: enter
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:8b:00.0], qpair=0x200136093d20, qid=0x0002: enter
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:8b:00.0)
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:8b:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme67
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:07:00
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0000, state=6
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0000
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 26 times ***
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:07:00.0] Prepare to destruct SSD: 0000:07:00.0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:07:00.0] already in failed state
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:07:00.0]: unregistering thread=app_thread
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:07:00.0
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:07:00.0)
9/10/2025 7:37:27 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:07:00.0: destruct=1
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_4]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52cc1cc2f0 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_4] qpair=0x7f52d0033cb0, qp_create_elapsed=24219us, res_elapsed=9us
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d0033cb0 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00fa810 qid=0x0001 ctrlr=0x7f52d4268060 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00fa810 qid=0x0001
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00fa810 qid=0x0001 ctrlr=0x7f52d4268060 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_2]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 ctrlr=0x7f52d4268060 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_2]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00ed220 qid=0x0000 ctrlr=0x7f52d4268060 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ed220 qid=0x0000
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_2]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03 qpair=0x7f52d00ed220 qid=0x0000 ctrlr=0x7f52d4268060 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 7 times ***
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 6 times ***
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:07:00.0] already in failed state
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay03, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:07:00.0)
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:07:00.0: destruct=1
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme66n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme66n1 from latency reports
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme66n1
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 1 time ***
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:38:Disable latency histogram for bdev HotInNvme64n1 DISABLED, Status NVG_CMD_FAILED: Unexpected response from server: {"code":-19,"message":"No such device"}}
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvg_histograms.cc:230:Run Removed bdev HotInNvme64n1 from latency reports
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev.c:7325:spdk_bdev_open_ext Currently unable to find bdev with name: HotInNvme64n1
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 9 times ***
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:347:Scan Number of available devices: 9
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:05:00.0 is missing
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:04:00.0 is missing
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme66n1 bdf:0000:05:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 eui64:8CE38EE202891501 sn:70G0A00FTBX7 reason:unknown
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:273:Scan Device removed: bdev:HotInNvme64n1 bdf:0000:04:00.0 subnqn:nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 eui64:0000000000000000 sn:S80DNG0Y200115 reason:unknown
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme66n1 was hot removed
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:267:Scan Namespace HotInNvme64n1 was hot removed
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00e7eb0 qid=0x0001 qp_destroy_elapsed=50908us res_destroy_elapsed=110us
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d00ece60 qid=0x0000 qp_destroy_elapsed=38612us res_destroy_elapsed=14us
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 9 times ***
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20025bc06320, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:07:00.0 wwn:8CE38EE300FF11DF
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x20013703fe60, qid=0x04: Aborting trackers
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0004
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0001
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200140b81ee0 was disconnected and freed. reset controller.
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20013703fe60 was disconnected and freed. reset controller.
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0004, state=6
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0003, state=6
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0002, state=6
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0001, state=6
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:07:00.0 cntlid=0x0002 qid=0x0000, state=5
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x200140b81ee0, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x2001393f6e60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x2001393f6c60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:07:00.0]: qpair=0x20013703fe60, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x200140b81ee0, qid=0x01: Aborting trackers
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x2001393f6e60, qid=0x02: Aborting trackers
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:07:00.0], qpair=0x2001393f6c60, qid=0x03: Aborting trackers
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0003
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:07:00.0]: Freeing qid=0x0002
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x200140b81ee0, qid=0x0001: enter
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x2001393f6e60, qid=0x0002: enter
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x2001393f6c60, qid=0x0003: enter
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:07:00.0], qpair=0x20013703fe60, qid=0x0004: enter
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:07:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:07:00.0)
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:07:00.0: destruct=0
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:07:00.0
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001393f6e60 was disconnected and freed. reset controller.
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001393f6c60 was disconnected and freed. reset controller.
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:26 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:07:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme66
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:05:00
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0000
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0000, state=6
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:05:00.0] Prepare to destruct SSD: 0000:05:00.0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:05:00.0] already in failed state
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:05:00.0)
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:05:00.0: destruct=1
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 32 times ***
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:05:00.0]: unregistering thread=app_thread
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:05:00.0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_3]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00f6780 qid=0x0000 ctrlr=0x7f52c82b1010 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0009
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0009
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_3] qpair=0x7f52d00f6780, qp_create_elapsed=24061us, res_elapsed=8us
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f6780 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00e7eb0 qid=0x0001 ctrlr=0x7f52e830af60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00e7eb0 qid=0x0001
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00e7eb0 qid=0x0001 ctrlr=0x7f52e830af60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_1]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 ctrlr=0x7f52e830af60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_1]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00ece60 qid=0x0000 ctrlr=0x7f52e830af60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d00ece60 qid=0x0000
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_1]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01 qpair=0x7f52d00ece60 qid=0x0000 ctrlr=0x7f52e830af60 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 10 times ***
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:05:00.0] already in failed state
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0008, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay01, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:05:00.0)
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:05:00.0: destruct=1
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0106220 qid=0x0001 qp_destroy_elapsed=49233us res_destroy_elapsed=110us
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 3 times ***
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app rdma.c:3061:nvg_destroy_qpair_thread !!! Freeing qpair: device=rocep197s0f0 qpair=0x7f52d0033cb0 qid=0x0000 qp_destroy_elapsed=38175us res_destroy_elapsed=15us
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 20 times ***
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x20013285a360, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:05:00.0 wwn:8CE38EE202891501
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x200132803ee0, qid=0x04: Aborting trackers
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x200132802360, qid=0x03: Aborting trackers
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0004
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0003
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0002
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x2001397beee0 was disconnected and freed. reset controller.
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132803ee0 was disconnected and freed. reset controller.
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132802360 was disconnected and freed. reset controller.
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0004, state=6
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0003, state=6
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0002, state=6
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0001, state=6
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:05:00.0 cntlid=0x0001 qid=0x0000, state=5
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x2001397beee0, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200138fd4ee0, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200132803ee0, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:05:00.0]: qpair=0x200132802360, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x2001397beee0, qid=0x02: Aborting trackers
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:05:00.0], qpair=0x200138fd4ee0, qid=0x01: Aborting trackers
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:05:00.0]: Freeing qid=0x0001
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x2001397beee0, qid=0x0002: enter
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x200138fd4ee0, qid=0x0001: enter
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x200132803ee0, qid=0x0004: enter
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:05:00.0], qpair=0x200132802360, qid=0x0003: enter
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:05:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:05:00.0)
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:05:00.0: destruct=0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:05:00.0
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200138fd4ee0 was disconnected and freed. reset controller.
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:25 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:05:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:440:nvme_bdev_ctrlr_delete !!! Freed: HotInNvme64
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app pci.c:463:pci_device_fini !!! fini: 0000:04:00
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0000, state=6
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0000
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 26 times ***
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x2001393c10e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:4331:nvme_ctrlr_destruct_async [0000:04:00.0] Prepare to destruct SSD: 0000:04:00.0
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:04:00.0] already in failed state
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=0 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:628:nvme_ctrlr_unregister [0000:04:00.0]: unregistering thread=app_thread
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:570:nvme_ctrlr_delete !!! [app_thread] Deleting 0000:04:00.0
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:04:00.0)
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:04:00.0: destruct=1
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x2001393c10e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app ctrlr.c:354:_nvmf_ctrlr_add_admin_qpair !!! [nvmf_tgt_poll_group_2]: Connected (ADM). subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d00f5c20 qid=0x0000 ctrlr=0x7f52d42d4a60 kato=10000 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0008
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app ctrlr.c:1296:nvmf_prop_set_cc !!! Property Set CC Enable=1 for cntlid=0x0008
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app rdma.c:1281:nvmf_rdma_qpair_initialize !!! [nvmf_tgt_poll_group_2] qpair=0x7f52d00f5c20, qp_create_elapsed=24394us, res_elapsed=8us
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 2 times ***
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x2001393c10e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app rdma.c:1649:nvmf_rdma_connect !!! [app_thread] device=rocep197s0f0 qpair=0x7f52d00f5c20 Final Negotiated Queue Depth=32 R/W Depth=16
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x2001393c10e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_4]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d0106220 qid=0x0001 ctrlr=0x7f52c82b1010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0007
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_4]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d0106220 qid=0x0001 ctrlr=0x7f52c82b1010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0007
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app ctrlr.c:623:_nvmf_ctrlr_destruct !!! [nvmf_tgt_poll_group_3]: Disconnected subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 ctrlr=0x7f52c82b1010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0007
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0106220 qid=0x0001
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x2001393c10e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app rdma.c:4583:nvmf_rdma_close_qpair !!! [nvmf_tgt_poll_group_3]: Closing qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52c82b1010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0007
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app rdma.c:3641:nvmf_rdma_disconnect !!! Disconnect event 10. qpair=0x7f52d0033cb0 qid=0x0000
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvmf.c:1224:_nvmf_qpair_destroy !!! [nvmf_tgt_poll_group_3]: Destroying qpair. subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00 qpair=0x7f52d0033cb0 qid=0x0000 ctrlr=0x7f52c82b1010 hostnqn=nqn.2020-19.com.hpe:51402ec001176210.021 cntlid=0x0007
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 7 times ***
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x2001393c10e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app subsystem.c:1380:_nvmf_ns_hot_remove !!! status=0
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions *** The above message repeated 38 times ***
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:1040:nvme_ctrlr_fail [0000:04:00.0] already in failed state
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app ctrlr.c:3992:nvmf_ctrlr_async_event_notification !!! AEN sent: cntlid=0x0007, subnqn=nqn.2019-20.com.hpe:51402EC018DE74E4.0.bay00, event: type=0002, info=0000, log page id=0004
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=4 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=3 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=2 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=1 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:04:00.0)
9/10/2025 7:37:24 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:04:00.0: destruct=1
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x2001393c10e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x2001393c10e0, qid=0 ctrlr is removed - aborting reqs
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:382 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:381 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:380 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:379 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:378 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:377 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:376 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:518:spdk_nvme_print_completion ABORTED - BY REQUEST (00/07) qid:0 cid:375 cdw0:0 sqhd:0000 p:0 m:0 dnr:0 bdf:0000:04:00.0 wwn:0000000000000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:382 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:381 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:380 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:379 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:378 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:377 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:376 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:230:nvme_admin_qpair_print_command ASYNC EVENT REQUEST (0c) qid:0 cid:375 nsid:0 cdw10:00000000 cdw11:00000000
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0003
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20014178ac60 was disconnected and freed. reset controller.
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0004, state=6
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0003, state=6
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0002, state=6
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0000, state=5
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20014178ac60, qid=3 ctrlr is removed - aborting reqs
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x20014178ae60, qid=0x02: Aborting trackers
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x20014178ac60, qid=0x03: Aborting trackers
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x200140800ee0, qid=0x04: Aborting trackers
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1279:nvme_pcie_ctrlr_delete_io_qpair !!! [0000:04:00.0], qpair=0x200132d51ee0, qid=0x01: Aborting trackers
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0004
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0002
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_pcie_common.c:1108:nvme_pcie_qpair_destroy !!! [0000:04:00.0]: Freeing qid=0x0001
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x20014178ae60, qid=0x0002: enter
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x20014178ac60, qid=0x0003: enter
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x200140800ee0, qid=0x0004: enter
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:670:nvme_ctrlr_release [0000:04:00.0]: ref=5 destruct=1 resetting=0 io_path_cache_clearing=0
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5666:remove_cb !!! [nvg_hotplug] Removing (0000:04:00.0)
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:5624:_bdev_nvme_delete !!! [nvg_hotplug] enter 0000:04:00.0: destruct=0
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:4780:nvme_ctrlr_depopulate_namespaces Removed: 0000:04:00.0
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x20014178ae60 was disconnected and freed. reset controller.
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200140800ee0 was disconnected and freed. reset controller.
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1629:bdev_nvme_disconnected_qpair_cb qpair 0x200132d51ee0 was disconnected and freed. reset controller.
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0004
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0003
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0002
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_transport.c:523:nvme_transport_ctrlr_disconnect_qpair Disconnecting ctrlr=0000:04:00.0 cntlid=0x0081 qid=0x0001, state=6
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x20014178ae60, qid=2 ctrlr is removed - aborting reqs
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x200140800ee0, qid=4 ctrlr is removed - aborting reqs
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_qpair.c:863:spdk_nvme_qpair_process_completions !!! [0000:04:00.0]: qpair=0x200132d51ee0, qid=1 ctrlr is removed - aborting reqs
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app nvme_ctrlr.c:587:spdk_nvme_ctrlr_free_io_qpair !!! [0000:04:00.0], qpair=0x200132d51ee0, qid=0x0001: enter
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app bdev_nvme.c:1607:bdev_nvme_disconnected_qpair_cb !!! [0000:04:00.0] About to free IO qpair qid=0x0001
9/10/2025 7:37:23 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:36:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:36:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:34:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:34:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:31:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:26:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:25:27 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 7:25:27 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 7:25:27 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 7:25:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:25:16 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 7:25:16 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 7:25:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:24:46 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 7:24:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:24:41 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 7:24:41 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 7:24:41 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 7:24:40 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 7:24:40 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 7:24:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:24:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:24:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:22:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:22:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:21:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:14:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:14:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:14:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:11:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:11:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:09:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:09:38 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 7:09:38 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 7:09:38 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 7:09:38 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:09:37 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:09:36 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 7:09:36 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 7:09:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:09:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/10/2025 7:09:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 7:09:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 11 times ***
9/10/2025 7:09:06 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 7:09:06 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 7:09:06 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 7:09:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:09:05 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 7:09:05 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 7:09:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:06:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:06:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:04:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:04:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:02:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:02:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:01:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:01:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:59:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:57:03 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:57:01 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 6:57:01 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":874765356, "rx_prio4_bytes_delta":5248614, "tx_prio4_bytes":5060419270, "tx_prio4_bytes_delta":42947312, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8433299, "rx_prio4_packets_delta":57745, "tx_prio4_packets":9325810, "tx_prio4_packets_delta":65754}
9/10/2025 6:57:01 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 6:57:01 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 6:54:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:54:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:54:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:54:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 6:54:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 6:54:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 6:54:00 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 6:54:00 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 6:54:00 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:53:40 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 6:53:39 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 6:53:39 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 6:53:37 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:53:35 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 6:53:35 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 6:53:35 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 6:53:35 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 6:49:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:48:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:46:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:46:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:44:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:44:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:41:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:41:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:38:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:38:44 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 6:38:44 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 6:38:44 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 6:38:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:38:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:38:30 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 6:38:30 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 6:38:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:38:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:37:55 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 6:37:52 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:37:50 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 6:37:50 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 6:37:49 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 6:37:49 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 6:37:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:37:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:37:45 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 6:37:45 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 6:37:45 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 6:37:45 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 6:37:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:36:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:31:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:29:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:29:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:26:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:26:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:24:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:24:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:22:57 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 6:22:57 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 6:22:57 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 6:22:57 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:22:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:22:40 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 6:22:40 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 6:22:15 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 6:22:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:22:10 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 6:22:10 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 6:22:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:22:06 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:22:05 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 6:22:05 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 6:22:05 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 6:22:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:22:04 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 6:22:04 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 6:19:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:18:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:16:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:16:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:14:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:14:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:12:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:12:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:11:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:11:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:09:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:07:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:07:10 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 6:07:10 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 6:07:10 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 6:07:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:07:00 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 6:07:00 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 6:07:00 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 6:06:58 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:06:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:06:30 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 6:06:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:06:25 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 6:06:25 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 6:06:25 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 6:06:25 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:06:24 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 6:06:24 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 6:06:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:06:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:06:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 6:04:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 6:04:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 6:01:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:57:01 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:57:00 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 5:57:00 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":869516742, "rx_prio4_bytes_delta":5210522, "tx_prio4_bytes":5017471958, "tx_prio4_bytes_delta":43222196, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8375554, "rx_prio4_packets_delta":57324, "tx_prio4_packets":9260056, "tx_prio4_packets_delta":65427}
9/10/2025 5:57:00 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 5:57:00 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 5:57:00 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:54:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:51:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:51:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:51:31 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 5:51:31 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 5:51:31 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 5:51:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 5:51:20 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 5:51:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:51:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:50:50 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 5:50:48 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 5:50:48 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 5:50:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:50:46 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:50:45 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 5:50:45 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 5:50:45 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 5:50:45 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 5:50:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:49:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:49:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:46:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:46:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:44:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:44:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:44:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:44:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:41:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:38:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:38:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:36:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:35:52 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 5:35:52 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 5:35:52 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 5:35:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:35:39 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 5:35:39 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 5:35:14 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 5:35:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:35:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 5:35:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 5:35:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 5:35:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 5:35:09 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:34:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:34:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:34:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:32:22 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:32:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:31:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:31:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:30:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:29:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:28:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:24:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:21:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:21:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:20:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:20:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 5:20:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 5:20:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 5:20:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:20:04 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 5:20:04 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 5:20:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:20:02 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:19:09 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 5:19:07 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:19:04 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 5:19:04 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 5:19:01 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:18:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 5:18:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 5:18:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 5:18:59 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:18:58 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 5:18:58 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 5:16:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:16:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:14:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:14:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:11:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:06:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:06:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:06:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:04:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 5:04:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:04:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:04:04 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:04:03 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 5:04:03 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 5:04:03 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 5:04:03 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:03:56 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:03:54 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 5:03:54 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 5:03:54 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 5:03:53 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/10/2025 5:02:59 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 5:02:59 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/10/2025 5:02:56 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 5:02:54 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 5:02:54 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 5:02:54 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 5:02:54 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 5:02:53 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 5:02:53 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 4:58:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:57:01 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:56:59 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 4:56:59 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":864306220, "rx_prio4_bytes_delta":5261218, "tx_prio4_bytes":4974249762, "tx_prio4_bytes_delta":44026480, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8318230, "rx_prio4_packets_delta":57853, "tx_prio4_packets":9194629, "tx_prio4_packets_delta":66132}
9/10/2025 4:56:59 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 4:56:59 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 4:56:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:56:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:54:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:54:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:51:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:51:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:49:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:49:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:47:58 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:47:57 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 4:47:57 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 4:47:57 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 4:47:57 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:47:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:47:49 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 4:47:49 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 4:47:49 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:47:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:47:19 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 4:47:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:47:14 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 4:47:14 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 4:47:13 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 4:47:13 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 4:47:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:47:10 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:47:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 4:47:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 4:47:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 4:47:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 4:47:09 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:46:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:46:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:44:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:44:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:42:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:42:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:41:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:38:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:38:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:36:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:34:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:34:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:34:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:32:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 4:32:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 4:32:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 4:32:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:32:05 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:32:04 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 4:32:04 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 4:31:39 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 4:31:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:31:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 4:31:34 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 4:31:34 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:31:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:31:29 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 4:31:29 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 4:31:29 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 4:31:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:31:28 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 4:31:28 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 4:24:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:24:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:21:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:21:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:19:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:19:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:16:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:16:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:16:30 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:16:28 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 4:16:28 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 4:16:28 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 4:16:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:16:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 4:16:24 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 4:16:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:16:04 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 4:15:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 4:15:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 4:15:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 4:15:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 4:14:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:14:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:12:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:12:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:11:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:06:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:04:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:04:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:04:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:04:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:02:22 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:01:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:01:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:01:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 4:01:11 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 4:01:11 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 4:01:11 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 4:01:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 4:00:53 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 4:00:53 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 4:00:53 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 4:00:23 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 4:00:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 4:00:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 4:00:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 4:00:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 4:00:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 4:00:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 4:00:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:56:58 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 3:56:58 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":859045002, "rx_prio4_bytes_delta":5208886, "tx_prio4_bytes":4930223282, "tx_prio4_bytes_delta":40967046, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8260377, "rx_prio4_packets_delta":57443, "tx_prio4_packets":9128497, "tx_prio4_packets_delta":65073}
9/10/2025 3:56:58 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 3:56:58 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 3:54:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:51:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:51:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:49:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:49:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:46:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:46:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:45:26 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:45:24 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 3:45:24 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 3:45:24 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 3:45:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:45:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:45:13 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 3:45:13 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 3:45:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:45:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 21 times ***
9/10/2025 3:44:48 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 3:44:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 3:44:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:44:43 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 3:44:43 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 3:44:43 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 3:44:43 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 3:44:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:44:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:44:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:36:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:34:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:34:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:31:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:31:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:30:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:30:01 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:29:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 3:29:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 3:29:59 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 3:29:59 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:29:38 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 3:29:38 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 3:29:38 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 3:29:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 26 times ***
9/10/2025 3:29:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 3:29:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 3:29:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 3:29:13 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 3:29:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:29:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/10/2025 3:29:09 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:29:08 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 3:29:08 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 3:29:08 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 3:29:08 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 3:29:08 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:26:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:26:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:26:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:24:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:24:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:24:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:24:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:21:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:21:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:19:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:19:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:16:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:14:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:14:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 3:14:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 3:14:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 3:14:03 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 3:14:03 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 3:14:03 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:14:01 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:13:33 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 3:13:29 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:13:28 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 3:13:28 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 3:13:28 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 3:13:28 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:13:27 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 3:13:27 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 3:12:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:12:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:11:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:06:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 3:04:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:04:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 3:01:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:01:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 3:00:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 3:00:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:59:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:59:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 2:58:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:58:30 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 2:58:30 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 2:58:30 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 2:58:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:58:23 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 2:58:23 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 2:58:23 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:58:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:57:53 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 2:57:52 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 2:57:52 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 2:57:48 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 2:57:48 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 2:57:48 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 2:57:48 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 2:56:58 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 2:56:58 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":853836116, "rx_prio4_bytes_delta":5247424, "tx_prio4_bytes":4889256236, "tx_prio4_bytes_delta":43194382, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8202934, "rx_prio4_packets_delta":57752, "tx_prio4_packets":9063424, "tx_prio4_packets_delta":65877}
9/10/2025 2:56:58 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 2:56:58 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 2:56:58 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:56:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:52:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:52:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 2:50:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:49:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 2:49:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:46:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:46:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:44:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:44:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 2:42:59 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:42:57 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 2:42:57 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 2:42:57 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 2:42:42 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 2:42:42 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 2:42:39 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:42:12 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 2:42:11 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 2:42:11 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 2:42:07 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 2:42:07 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 2:42:07 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 2:42:07 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 2:41:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:39:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 2:39:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:36:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:36:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:36:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:36:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:34:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:34:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 2:31:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:27:19 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 2:27:19 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 2:27:19 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 2:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 2:27:02 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 2:27:02 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:26:37 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 2:26:32 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 2:26:32 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 2:26:32 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 2:26:32 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 2:26:31 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 2:26:31 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 2:26:31 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:24:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:20:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:16:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:16:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:16:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 2:14:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 2:14:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:12:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:11:49 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 2:11:49 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 2:11:49 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 2:11:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:11:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:11:27 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 2:11:27 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 2:11:24 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:10:57 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 2:10:53 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:10:52 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 2:10:52 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 2:10:52 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 2:10:52 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:10:51 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 2:10:51 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 2:09:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:09:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 2:08:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:06:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 2:00:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 2:00:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:59:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:58:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:56:57 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 1:56:57 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":848588692, "rx_prio4_bytes_delta":5185344, "tx_prio4_bytes":4846061854, "tx_prio4_bytes_delta":42199154, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8145182, "rx_prio4_packets_delta":57439, "tx_prio4_packets":8997547, "tx_prio4_packets_delta":65374}
9/10/2025 1:56:57 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 1:56:57 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 1:56:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:56:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:55:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:55:50 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 1:55:50 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 1:55:50 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 1:55:50 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:55:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:55:47 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 1:55:47 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 1:55:47 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:55:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:55:22 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 1:55:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 1:55:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 1:55:17 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 1:55:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:55:16 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 1:55:16 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 1:54:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:54:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:51:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:46:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:44:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:41:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:41:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:40:14 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 1:40:14 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 1:40:14 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 1:40:12 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 1:40:12 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 1:40:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:40:09 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:39:47 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 1:39:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:39:42 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 1:39:42 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 1:39:42 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 1:39:42 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 1:39:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:39:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:36:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:36:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:34:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:34:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:31:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:30:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:30:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:26:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:24:41 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 1:24:41 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 1:24:41 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 1:24:38 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:24:36 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 1:24:36 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 1:24:36 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:24:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:24:06 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 1:24:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 1:24:01 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 1:24:01 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:24:00 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 1:24:00 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 1:24:00 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:23:56 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 1:23:56 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 1:23:56 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 1:23:56 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 1:22:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:22:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:21:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:21:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:19:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:19:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:18:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 1:18:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:16:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:14:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:10:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:09:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:09:02 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:09:01 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 1:09:01 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 1:09:01 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 1:09:01 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:08:51 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 1:08:51 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 1:08:51 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:08:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 1:08:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 1:08:20 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 1:08:20 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 1:08:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 1:08:16 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 1:08:16 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 1:08:16 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 1:08:16 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 1:06:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 1:01:42 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:59:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:59:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:56:58 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:56:56 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 12:56:56 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":843403348, "rx_prio4_bytes_delta":5170528, "tx_prio4_bytes":4803862700, "tx_prio4_bytes_delta":40550706, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8087743, "rx_prio4_packets_delta":56990, "tx_prio4_packets":8932173, "tx_prio4_packets_delta":64585}
9/10/2025 12:56:56 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 12:56:56 PM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 12:56:56 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:56:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:56:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:56:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:54:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:54:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:53:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 12:53:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 12:53:18 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 12:53:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:53:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 12:53:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 12:53:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:52:51 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 12:52:48 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:52:46 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 12:52:46 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 12:52:46 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 12:52:46 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 12:51:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:44:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:44:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:44:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:41:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:41:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:39:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:39:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:38:21 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:38:20 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:38:00 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 12:38:00 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 12:38:00 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 12:37:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:37:41 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 12:37:41 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 12:37:21 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 12:37:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:37:16 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 12:37:16 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 12:37:16 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 12:37:16 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:37:15 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 12:37:15 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 12:36:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:36:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:34:15 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:29:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:28:18 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:28:17 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:26:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:26:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:24:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:24:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:22:26 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 12:22:26 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 12:22:26 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 12:22:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:22:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 12:22:11 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 12:22:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:22:09 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:21:46 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 12:21:41 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 12:21:41 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 12:21:41 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 12:21:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:21:40 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 12:21:40 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 12:21:40 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:19:14 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:19:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:16:45 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:12:19 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:09:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:06:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:06:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:06:41 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:06:39 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 12:06:39 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 12:06:39 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 12:06:39 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:06:36 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 12:06:36 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 12:06:35 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 12:06:16 PM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 12:06:12 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 12:06:11 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 12:06:11 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 12:06:11 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 12:06:11 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:06:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 12:06:09 PM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 12:04:13 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 12:01:44 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 12:01:43 PM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:59:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:59:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:56:55 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 11:56:55 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":838232820, "rx_prio4_bytes_delta":5204712, "tx_prio4_bytes":4763311994, "tx_prio4_bytes_delta":41767962, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":8030753, "rx_prio4_packets_delta":57489, "tx_prio4_packets":8867588, "tx_prio4_packets_delta":65292}
9/10/2025 11:56:55 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 11:56:55 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 11:56:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:56:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:56:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:54:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:51:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:51:08 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 11:51:08 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 11:51:08 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 11:51:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:51:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:51:05 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 11:51:05 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 11:51:05 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:51:04 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 11:50:45 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 11:50:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 11:50:40 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 11:50:40 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 11:50:40 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 11:50:40 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:50:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 11:50:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 11:50:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:49:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:49:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:46:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:41:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:39:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:39:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:36:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:36:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:35:51 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:35:50 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 11:35:50 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 11:35:50 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 11:35:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:35:35 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 11:35:35 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 11:35:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 28 times ***
9/10/2025 11:35:20 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 11:35:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 11:35:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:35:15 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 11:35:15 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 11:35:15 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 11:35:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:35:14 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 11:35:14 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 11:34:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:34:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:30:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:30:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:29:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:26:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:26:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:24:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:24:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:22:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:21:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:20:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:20:13 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 11:20:13 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 11:20:13 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 11:20:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:20:11 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:20:10 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 11:20:10 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 11:20:10 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:20:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 31 times ***
9/10/2025 11:19:55 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 24 times ***
9/10/2025 11:19:45 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 11:19:42 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:19:40 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 11:19:40 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 11:19:40 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 11:19:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 11:19:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 11:19:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:19:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:19:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:18:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:16:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:16:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:14:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:09:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:06:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:06:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:06:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:06:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:04:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:04:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 11:04:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 11:04:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 11:04:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:04:36 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:04:35 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 11:04:35 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 11:04:35 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:04:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 11:04:15 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 11:04:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 11:04:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 6 times ***
9/10/2025 11:04:10 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 11:04:10 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 11:04:10 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 11:04:10 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 11:04:09 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 11:04:09 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 11:04:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 11:02:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 11:02:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:59:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:56:55 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 10:56:55 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":833028108, "rx_prio4_bytes_delta":5206204, "tx_prio4_bytes":4721544032, "tx_prio4_bytes_delta":40578642, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":7973264, "rx_prio4_packets_delta":57405, "tx_prio4_packets":8802296, "tx_prio4_packets_delta":64986}
9/10/2025 10:56:55 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 10:56:55 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 10:56:55 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:56:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:56:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:54:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:54:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:51:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:51:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:50:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:50:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:49:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:49:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:49:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:49:08 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 10:49:08 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 10:49:08 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 10:49:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:49:07 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:49:04 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 10:49:04 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 10:49:04 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:48:44 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 10:48:41 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:48:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 10:48:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 10:48:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 10:48:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:48:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 10:48:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 10:46:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:46:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:46:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:44:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:39:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:36:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:36:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:34:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:34:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:33:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:33:37 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 10:33:37 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 10:33:37 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 10:33:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:33:36 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:33:34 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 10:33:34 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 10:33:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:33:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 8 times ***
9/10/2025 10:33:14 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 10:33:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 9 times ***
9/10/2025 10:33:10 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:33:09 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 10:33:09 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 10:33:09 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 10:33:09 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 10:33:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:31:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:31:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:30:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:29:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:29:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:26:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:26:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:22:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:22:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:20:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:20:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:19:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:18:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:18:19 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 10:18:19 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 10:18:19 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 10:18:04 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 10:18:04 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 10:18:04 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:18:02 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:17:34 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 10:17:33 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 10:17:33 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 10:17:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:17:29 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 10:17:29 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 10:17:29 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 10:17:29 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 10:16:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:16:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:14:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:14:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:14:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:14:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:12:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:12:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:11:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:11:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:10:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:10:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:09:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:04:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 10:02:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 10:02:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 10:02:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 10:02:26 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 10:02:24 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 10:02:24 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 10:02:24 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:02:23 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:01:50 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:01:49 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 10:01:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 10:01:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:01:44 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 10:01:44 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 10:01:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:01:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:01:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 10:01:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 10:01:39 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 10:01:39 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 10:01:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 10:01:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 10:01:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:59:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:56:54 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 9:56:54 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":827821904, "rx_prio4_bytes_delta":5220312, "tx_prio4_bytes":4680965390, "tx_prio4_bytes_delta":41571100, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":7915859, "rx_prio4_packets_delta":57522, "tx_prio4_packets":8737310, "tx_prio4_packets_delta":65284}
9/10/2025 9:56:54 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 9:56:54 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 9:52:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:52:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:51:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:49:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:49:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:47:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:47:17 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 9:47:17 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 9:47:17 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 9:47:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:46:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:46:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:46:34 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 9:46:34 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 9:46:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:46:09 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 9:46:04 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:46:04 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 9:46:04 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 9:46:04 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:45:59 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 9:45:59 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 9:45:59 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 9:45:59 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:45:58 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 9:45:58 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 9:45:58 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:44:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:42:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:42:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:41:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:41:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:40:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:40:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:39:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:39:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:36:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:36:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:34:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:31:10 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 9:31:10 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 9:31:10 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 9:30:53 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 9:30:53 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 9:30:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:30:28 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 9:30:25 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:30:23 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 9:30:23 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 9:30:22 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 9:30:22 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 9:30:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:30:18 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 9:30:18 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 9:30:18 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 9:30:18 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 9:30:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:26:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:24:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:24:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:21:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:21:43 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:20:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:20:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:19:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:19:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:18:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:18:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:16:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:15:25 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 9:15:25 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 9:15:25 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 9:15:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:15:13 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 9:15:13 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 9:15:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 23 times ***
9/10/2025 9:14:48 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 9:14:46 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:14:43 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 9:14:43 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 9:14:40 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:14:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 9:14:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 9:14:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 9:14:38 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:14:37 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 9:14:37 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 9:14:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:14:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:14:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:14:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:11:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:10:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:10:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:08:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:08:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 9:04:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:04:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 9:04:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 9:01:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 9:01:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:59:45 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 8:59:45 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 8:59:45 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 8:59:34 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:59:33 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 8:59:33 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 8:59:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:59:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 7 times ***
9/10/2025 8:59:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 12 times ***
9/10/2025 8:59:08 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 8:59:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 8:59:03 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 8:59:03 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 8:59:03 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 8:59:03 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:59:02 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 8:59:02 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 8:59:02 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:56:53 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 8:56:53 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":822601592, "rx_prio4_bytes_delta":5247262, "tx_prio4_bytes":4639394290, "tx_prio4_bytes_delta":42724932, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":7858337, "rx_prio4_packets_delta":57732, "tx_prio4_packets":8672026, "tx_prio4_packets_delta":65731}
9/10/2025 8:56:53 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 8:56:53 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 8:56:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:56:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:56:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:54:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:52:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:52:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:48:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:48:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:46:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:44:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:44:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:44:01 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:44:00 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 8:44:00 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 8:44:00 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 8:44:00 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:43:58 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 8:43:58 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 8:43:55 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 17 times ***
9/10/2025 8:43:33 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 8:43:33 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:43:28 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 8:43:28 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 8:43:28 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 8:43:28 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:43:27 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 8:43:27 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 8:43:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:41:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:41:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:39:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:31:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:30:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:29:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:29:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:28:36 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:28:34 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 8:28:34 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 8:28:34 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 8:28:25 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:28:23 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 8:28:23 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 8:28:22 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:27:58 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 8:27:53 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 8:27:53 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 8:27:53 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 8:27:53 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 8:27:53 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:26:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:24:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:24:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:22:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:21:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:21:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:19:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:19:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:18:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:18:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:16:46 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:16:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:14:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:14:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:13:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:13:12 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 8:13:12 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 8:13:12 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 8:13:12 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:12:48 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 8:12:48 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 8:12:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:12:18 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 8:12:13 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 8:12:13 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 8:12:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:12:08 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 8:12:08 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 8:12:08 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 8:12:08 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:12:07 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 8:12:07 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 8:11:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:11:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:09:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:09:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 8:06:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:06:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:06:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 8:06:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 8:04:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 8:00:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:58:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:57:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:57:13 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 7:57:13 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 7:57:13 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 7:57:13 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:57:04 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:57:02 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 7:57:02 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 7:57:02 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:57:01 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/10/2025 7:56:52 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":0, "tx_prio4_bytes":0, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":0, "tx_prio4_packets":0}
9/10/2025 7:56:52 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.statistics", "type":"eth", "rx_prio4_marked":0, "rx_prio4_buf_discard":0, "rx_prio3_cong_discard":0, "rx_prio4_pause_transition":0, "rx_prio4_cong_discard":0, "tx_prio4_pause_duration":0, "rx_prio3_marked":0, "rx_prio4_pause_duration":0, "rx_prio3_packets":0, "rx_prio3_discards":0, "tx_prio3_packets":0, "rx_prio3_bytes":0, "rx_prio4_discards":0, "rx_prio4_bytes":817354330, "rx_prio4_bytes_delta":5182612, "tx_prio4_bytes":4596669358, "tx_prio4_bytes_delta":41409160, "tx_prio3_bytes":0, "tx_prio4_pause":0, "rx_prio3_buf_discard":0, "rx_prio4_pause":0, "rx_prio4_packets":7800605, "rx_prio4_packets_delta":57072, "tx_prio4_packets":8606295, "tx_prio4_packets_delta":64824}
9/10/2025 7:56:52 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-2.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":0, "resp_cqe_error":0, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 7:56:52 AM 4070 I nvmf_tgt_app collect_dp.cc:168:Collect {"id":"DP-1.hw_counters", "type":"rdma", "out_of_sequence":0, "resp_remote_access_errors":0, "resp_cqe_flush_error":23211, "resp_cqe_error":23440, "roce_adp_retrans":0, "duplicate_request":0, "implied_nak_seq_err":0, "roce_slow_restart_cnps":0, "out_of_buffer":0, "roce_adp_retrans_to":0, "np_ecn_marked_roce_packets":0, "roce_slow_restart":0, "rnr_nak_retry_err":0, "packet_seq_err":0, "roce_slow_restart_trans":0, "np_cnp_sent":0, "local_ack_timeout_err":0, "req_cqe_flush_error":0, "lifespan":10, "req_remote_access_errors":0, "rp_cnp_ignored":0, "resp_local_length_error":0, "req_cqe_error":0, "req_remote_invalid_request":0, "rp_cnp_handled":0}
9/10/2025 7:56:52 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 20 times ***
9/10/2025 7:56:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 15 times ***
9/10/2025 7:56:37 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 7:56:37 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 10 times ***
9/10/2025 7:56:32 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 7:56:32 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:119:rpc_power_cycle Dropping job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' due to drop constraint
9/10/2025 7:56:32 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 7:56:32 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:56:31 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=1
9/10/2025 7:56:31 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 7:56:31 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:56:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:56:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:54:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:51:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:51:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:50:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:50:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:49:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:49:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:46:46 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:46:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:46:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:46:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:44:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:42:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:42:18 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:41:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 7:41:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 7:41:38 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 7:41:27 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 7:41:27 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 7:41:27 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:40:57 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 7:40:54 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:40:52 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 7:40:52 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 7:40:52 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 7:40:52 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 7:39:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:36:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:36:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:36:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:34:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:34:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:31:46 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:26:11 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:26:09 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:226:RpcServerRun Job '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0' finished
9/10/2025 7:26:09 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:225:RpcServerRun Recovery script finished:
9/10/2025 7:26:09 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:212:RpcServerRun Command '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0: success
9/10/2025 7:26:09 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:25:49 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:25:47 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:325:Scan Device at 0000:c8:00.0 is present but user mode driver failure or subsystem reset timed out
9/10/2025 7:25:47 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:128:Scan Found device at 0000:c8:00.0 with driver=vfio-pci
9/10/2025 7:25:07 AM 4070 I nvmf_tgt_app nvme_device_discovery.cc:328:Scan Device at 0000:c8:00.0 is missing
9/10/2025 7:25:03 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:25:02 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:467:Recover Recovery on 127.127.10.10:55555 for 0000:c8:00.0 submitted
9/10/2025 7:25:02 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:93:rpc_power_cycle Received ps: bdf=0000:c8:00.0 drop_interval=60000ms origin=0
9/10/2025 7:25:02 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:251:RpcServerRun Starting '/usr/bin/timeout 240 /opt/tpd/bin/nvmf_tgt_device_recovery 0000:c8:00.0'
9/10/2025 7:25:02 AM 4070 I nvmf_tgt_app device_recovery_rpc_daemon.cc:438:Recover Recovery on 127.127.10.10:55555 for bdf 0000:c8:00.0
9/10/2025 7:25:02 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:24:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:24:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:22:17 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:22:16 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:21:46 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:20:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:20:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:16:19 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 4 times ***
9/10/2025 7:14:15 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:14:14 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:12:21 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 1 time ***
9/10/2025 7:12:20 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 3 times ***
9/10/2025 7:11:45 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***
9/10/2025 7:11:44 AM 4070 I nvmf_tgt_app DPDK:0: *** The above message repeated 2 times ***