Recover transient camera failures within six timeout windows

AGENTS.md

@@ -0,0 +1,7 @@
+# 工程交付约定
+
+- 共享采集、停止与恢复逻辑的修改，应同步本地 `inference-only`、`use-can`、`lazy-inference` 三个分支，并完成 Git 远端与 Orin 部署同步。
+- 本机模拟测试仅作为回归检查，不得用它代替 Orin 上的真实摄像头和 TensorRT 验证。报告须区分本机模拟、板端故障注入、真实硬件操作，明确未覆盖项目。
+- 短暂取流故障的重点是期限内恢复后在同一进程继续输出新的推理结果，并重新计数；不能只验证超时后退出。
+- 发布前核对本地提交、Git 远端、Orin 检出版本及实际运行二进制的哈希。更新 Orin 可执行文件后同时刷新独立包 SHA256SUMS。
+- 更新测试期间保留原启动参数，测试结束后恢复运行部署版本。不要把旧进程仍在运行当作新版本已部署。

README.md

@@ -3,6 +3,8 @@
 C++ / TensorRT 实时摄像头风险推理，算法来自最新 run_risk.py。
 持续取图、推理并更新内存中的 FrameResult；可视化可选。
 
+相机每次取帧默认等待 **2000 ms**，连续超时 **6 次**才停止流水线（纯超时约 **12 秒**）；成功取得有效帧后清零超时计数。单次等待时间由 `config/camera.json` 的 `camera.timeout_ms` 配置。运行中的坏帧、取流错误共用该重试期限；设备失效时尝试重新打开相机，恢复有效帧后继续推理。GPU 错误仍按原有异常处理退出。
+
 ## Orin 直接运行
 
 适用环境：Jetson AGX Orin、JetPack 5.1.2 / L4T R35.4.1、CUDA 11.4、TensorRT 8.5.2、OpenCV 4.5.4。
@@ -90,3 +92,7 @@ NPP 用于 UYVY 转换、YOLO 缩放填充、RTMW 仿射裁剪；CUDA 用于通
 同一帧三类文件同名（扩展名不同），文件名包含运行标识和帧号，多次运行不覆盖。JSON 的 `images` 提供相对于 `output/` 的图片路径，未选择保存的图片为 null。源码入口写入工程 `output/`；独立包入口写入 `orin/output/`。默认不创建目录、不保存文件。
 
 保存对象是完成推理的帧；惰性分支暂停时不新增文件。开启保存会增加编码和磁盘 I/O 开销。写入失败时报告错误并停止，不静默丢失文件。
+
+相机超时回归测试：`./tests/run_camera_timeout.sh`；加 `--wall-clock` 验证默认 6 次超时约 12 秒。测试直接编译实际相机代码，使用模拟采集和转换器，无需摄像头、GPU 或 OpenCV；不覆盖视频回放路径。
+
+板端真实摄像头 / TensorRT 恢复与停止测试记录见 [Orin 测试报告](tests/ORIN_TEST_REPORT.md)。

orin/README.md

@@ -1,5 +1,8 @@
 # Orin 可直接运行目录
 
+相机每次等待默认 2 秒，连续 6 次超时才退出（约 12 秒）；有效帧恢复后重新计数。Ctrl+C / SIGTERM 主动停止不需要等待 6 次超时。
+
+
 本目录包含实际在 Jetson AGX Orin 上编译的 aarch64 程序和 TensorRT FP16 引擎。
 目标环境：L4T R35.4.1 / JetPack 5.1.2，CUDA 11.4，TensorRT 8.5.2，OpenCV 4.5.4。
 使用系统安装的 CUDA、TensorRT、OpenCV 共享库；不适用于任意 JetPack 版本或 x86 主机。

orin/SHA256SUMS

@@ -1,8 +1,8 @@
-04dc8265e15d57357a97f9907bd3b83b02f632e588e9653135950cb6bbd5a8f1  ./README.md
+49a272525082edf5ef4619efc57bc2509c06653af262fc8a8d257d138f9206fb  ./README.md
 6d5e874fd27fcfd75d303d7e35d9672028af71041f52dcc82a1e83676faefa65  ./bin/drivers/max9295.ko
 8a78d04ef8be44e9b62a5e4ee315d3d6d679e34aedff02a234bbf4d40e222933  ./bin/drivers/max9296.ko
 3da628587477cc35d5e3f8aee86e718e2ddf16808e893e00ba6e164b2307e843  ./bin/drivers/sgx-yuv-gmsl2.ko
-be91f87ab7a54de1ccee306174b65fc2fd89209eccf3f19daf4e17fdf17643a9  ./build/camera_risk
+3a984e1e4c0fa12285967085e3359a5b7c698d744e9c2deb2480a312eb637c1c  ./build/camera_risk
 8a572f661e056e3916758a442e73212842bfe89361dd1b07b87de8f6ae523cf8  ./config/camera.json
 9ba768d559fd3b64e833b50c62c70f2adfd6c169620aea5fba32cfb1b4c5b0e9  ./config/door_mask.json
 ebcd09e8c2e35008adbb0c1d222e580b17e0df1c8093df3481ba3374cd9a8174  ./engines/rtmw_l_384.engine

src/camera.cpp

@@ -9,6 +9,7 @@
 #include <unistd.h>
 #include <time.h>
 #include <cerrno>
+#include <algorithm>
 #include <cstring>
 #include <limits>
 #include <stdexcept>
@@ -42,6 +43,11 @@ struct Camera::Impl {
         try {
             stopFd=eventfd(0,EFD_CLOEXEC|EFD_NONBLOCK);
             if(stopFd<0)throw std::runtime_error("Cannot create camera stop event");
+            openCapture();
+        } catch(...) {release();throw;}
+    }
+    void openCapture() {
+        try {
             fd=open(config.device.c_str(),O_RDWR|O_NONBLOCK|O_CLOEXEC);
             if(fd<0)throw std::runtime_error("Cannot open "+config.device+": "+std::strerror(errno));
             v4l2_capability capability{};checkedIoctl(fd,VIDIOC_QUERYCAP,&capability,"VIDIOC_QUERYCAP");
@@ -77,16 +83,19 @@ struct Camera::Impl {
                 if(mapping==MAP_FAILED)throw std::runtime_error("Camera mmap failed");
                 buffers[i].data=mapping;checkedIoctl(fd,VIDIOC_QBUF,&buffer,"VIDIOC_QBUF");
             }
-            converter=std::make_unique<UyvyConverter>(config.width,config.height);
+            if(!converter)converter=std::make_unique<UyvyConverter>(config.width,config.height);
             auto type=V4L2_BUF_TYPE_VIDEO_CAPTURE;checkedIoctl(fd,VIDIOC_STREAMON,&type,"VIDIOC_STREAMON");streaming=true;
-        } catch(...) {release();throw;}
+        } catch(...) {releaseCapture();throw;}
     }
     ~Impl(){release();}
-    void release() noexcept {
+    void releaseCapture() noexcept {
         if(streaming){auto type=V4L2_BUF_TYPE_VIDEO_CAPTURE;ioctl(fd,VIDIOC_STREAMOFF,&type);streaming=false;}
         for(auto const& b:buffers)if(b.data)munmap(b.data,b.size);
         buffers.clear();
         if(fd>=0){close(fd);fd=-1;}
+    }
+    void release() noexcept {
+        releaseCapture();
         if(stopFd>=0){close(stopFd);stopFd=-1;}
     }
     void stop() noexcept {
@@ -94,47 +103,82 @@ struct Camera::Impl {
         if(stopFd>=0){auto ignored=write(stopFd,&one,sizeof(one));(void)ignored;}
     }
     std::optional<CapturedFrame> readFrame() {
+        auto windowStart=monotonicNs();unsigned failedWindows=0;
+        auto windowNs=std::uint64_t(config.timeout_ms)*1000000;
+        std::string lastError="no camera frame";
+        auto expired=[&](bool timedOut=false) {
+            auto now=monotonicNs();
+            if(timedOut||now-windowStart>=windowNs) {
+                if(++failedWindows>=6)
+                    throw std::runtime_error("Camera capture failed 6 consecutive times: "+config.device+": "+lastError);
+                windowStart=now;
+            }
+        };
+        auto pause=[&] {
+            pollfd stopEvent{stopFd,POLLIN,0};
+            // Back off transient driver errors while retaining immediate cancellation.
+            poll(&stopEvent,1,20);
+        };
+        auto reconnect=[&] {
+            releaseCapture();
+            if(haveSequence){++index;haveSequence=false;}
+        };
         while(!stopped.load()) {
+            expired();
+            if(fd<0) {
+                try {openCapture();}
+                catch(std::exception const& e){lastError=e.what();pause();continue;}
+            }
+            if(stopped.load())return {};
+            auto elapsed=monotonicNs()-windowStart;
+            auto remaining=elapsed<windowNs?windowNs-elapsed:1;
+            int waitMs=static_cast<int>((remaining+999999)/1000000);
             pollfd descriptors[2]={{fd,POLLIN,0},{stopFd,POLLIN,0}};
-            int ready=poll(descriptors,2,config.timeout_ms);
+            int ready=poll(descriptors,2,waitMs);
             if(ready<0&&errno==EINTR)continue;
             if(stopped.load()||(descriptors[1].revents&POLLIN))return {};
-            if(ready==0)throw std::runtime_error("Camera capture timeout: "+config.device);
+            if(ready==0){expired(true);continue;}
-            if(ready<0||(descriptors[0].revents&(POLLERR|POLLHUP|POLLNVAL)))throw std::runtime_error("Camera disconnected or poll failed");
+            if(ready<0||(descriptors[0].revents&(POLLERR|POLLHUP|POLLNVAL))) {
+                lastError="camera disconnected or poll failed";reconnect();pause();continue;
+            }
             v4l2_buffer buffer{};buffer.type=V4L2_BUF_TYPE_VIDEO_CAPTURE;buffer.memory=V4L2_MEMORY_MMAP;
             int status=ioctl(fd,VIDIOC_DQBUF,&buffer);
             if(status<0&&(errno==EAGAIN||errno==EINTR))continue;
-            if(status<0)throw std::runtime_error("VIDIOC_DQBUF: "+std::string(std::strerror(errno)));
+            if(status<0) {
-            if(buffer.index>=buffers.size())throw std::runtime_error("Invalid V4L2 buffer index");
+                lastError="VIDIOC_DQBUF: "+std::string(std::strerror(errno));
-            // The GMSL driver can mark its startup buffer as corrupt. Requeue
+                // EIO can be a transient corrupt frame; a lost device needs reopening.
-            // it without conversion or inference; persistent corruption is fatal.
+                if(errno!=EIO)reconnect();
-            if(buffer.flags&V4L2_BUF_FLAG_ERROR) {
+                pause();continue;
-                checkedIoctl(fd,VIDIOC_QBUF,&buffer,"VIDIOC_QBUF(error frame)");
+            }
-                if(++consecutiveErrors>=static_cast<unsigned>(config.fps))
+            if(buffer.index>=buffers.size()){lastError="invalid V4L2 buffer index";reconnect();pause();continue;}
-                    throw std::runtime_error("Camera produced one second of consecutive error frames");
+            bool invalid=(buffer.flags&V4L2_BUF_FLAG_ERROR)||buffer.bytesused<std::size_t(stride)*config.height||buffer.bytesused>buffers[buffer.index].size;
-                continue;
+            if(invalid) {
+                ++consecutiveErrors;lastError="corrupt or incomplete camera frame";
+                if(ioctl(fd,VIDIOC_QBUF,&buffer)<0)reconnect();
+                pause();continue;
             }
             CapturedFrame frame;
-            try {
+            auto& m=frame.metadata;
-                if(buffer.bytesused<std::size_t(stride)*config.height||buffer.bytesused>buffers[buffer.index].size)
+            m.discarded_error_frames=consecutiveErrors;
-                    throw std::runtime_error("Invalid camera frame: flags="+std::to_string(buffer.flags)+", used="+std::to_string(buffer.bytesused)+", required="+std::to_string(std::size_t(stride)*config.height)+", allocated="+std::to_string(buffers[buffer.index].size));
+            m.driver_sequence=buffer.sequence;m.received_monotonic_ns=monotonicNs();
-                auto& m=frame.metadata;
+            m.driver_timestamp_ns=std::uint64_t(buffer.timestamp.tv_sec)*1000000000ULL+std::uint64_t(buffer.timestamp.tv_usec)*1000;
-                m.discarded_error_frames=consecutiveErrors;consecutiveErrors=0;
+            m.driver_timestamp_monotonic=(buffer.flags&V4L2_BUF_FLAG_TIMESTAMP_MASK)==V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
-                m.driver_sequence=buffer.sequence;m.received_monotonic_ns=monotonicNs();
+            m.capture_monotonic_ns=m.received_monotonic_ns;
-                m.driver_timestamp_ns=std::uint64_t(buffer.timestamp.tv_sec)*1000000000ULL+std::uint64_t(buffer.timestamp.tv_usec)*1000;
+            if(haveSequence) {
-                m.driver_timestamp_monotonic=(buffer.flags&V4L2_BUF_FLAG_TIMESTAMP_MASK)==V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+                auto delta=static_cast<std::uint32_t>(buffer.sequence-lastSequence);
-                m.capture_monotonic_ns=m.driver_timestamp_monotonic&&m.driver_timestamp_ns>0&&m.driver_timestamp_ns<=m.received_monotonic_ns?m.driver_timestamp_ns:m.received_monotonic_ns;
+                if(delta==0||delta>0x7fffffffU) {
-                if(haveSequence) {
+                    lastError="camera sequence reset or repeated frame";
-                    auto delta=static_cast<std::uint32_t>(buffer.sequence-lastSequence);
+                    reconnect();pause();continue;
-                    if(delta==0||delta>0x7fffffffU)throw std::runtime_error("Camera sequence reset or repeated frame");
-                    m.skipped_driver_frames=delta-1;index+=delta;
                 }
-                m.frame_index=index;
+                m.skipped_driver_frames=delta-1;index+=delta;
-                lastSequence=buffer.sequence;haveSequence=true;
+            }
-                // Convert before requeuing. No published result references MMAP memory.
+            m.frame_index=index;
-                frame.image=converter->convert(buffers[buffer.index].data,stride);
+            // GPU failures remain fatal; only capture failures consume the retry budget.
-            } catch(...) {ioctl(fd,VIDIOC_QBUF,&buffer);throw;}
+            try {frame.image=converter->convert(buffers[buffer.index].data,stride);}
-            checkedIoctl(fd,VIDIOC_QBUF,&buffer,"VIDIOC_QBUF");return frame;
+            catch(...) {ioctl(fd,VIDIOC_QBUF,&buffer);throw;}
+            if(ioctl(fd,VIDIOC_QBUF,&buffer)<0){lastError="VIDIOC_QBUF failed";reconnect();pause();continue;}
+            lastSequence=buffer.sequence;haveSequence=true;consecutiveErrors=0;
+            return frame; // The next read starts a fresh six-window retry budget.
         }
         return {};
     }

tests/ORIN_TEST_REPORT.md

@@ -0,0 +1,31 @@
+# Orin 相机故障恢复与停止测试（2026-09-23）
+
+目标：短暂取流故障在六个等待窗口内恢复时，同一进程继续推理；有效帧到达后重新计数；主动停止不等待全部窗口。
+
+环境：192.168.195.100，Jetson Orin，L4T R35.4.1，CUDA 11.4.298，TensorRT 8.5.2；真实 /dev/video5，YOLO 与 RTMW TensorRT 引擎。三个分支在该板编译。
+
+使用真实采集与推理进程；通过 LD_PRELOAD 在 V4L2 系统调用处注入无帧、坏帧、EIO 和 ENODEV。每次恢复测试检查同一子进程产生新的风险 JSON；错误恢复后至少新增 10 帧。Ctrl+C 通过 PTY 写入 0x03，SIGTERM 发送给测试进程。
+
+| 分支 | Ctrl+C / SIGTERM 完整退出范围 | 持续无帧退出（含清理） | 恢复后再次断流退出（含清理） | 结果 |
+|---|---:|---:|---:|---|
+| lazy-inference | 801.21–942.42 ms | 13.010 s | 13.085 s | 9/9 通过 |
+| inference-only | 793.22–1012.57 ms | 13.055 s | 12.945 s | 9/9 通过 |
+| use-can | 786.06–907.59 ms | 13.066 s | 13.155 s | 9/9 通过 |
+
+每个分支覆盖：断流后恢复并重新计时；坏帧恢复；EIO 恢复；ENODEV 后重新打开设备并恢复；正常推理 Ctrl+C；重试期 Ctrl+C；第六窗口 Ctrl+C；SIGTERM；持续无帧六窗口后报错。timeout/recover 场景最后的退出码 1 是预期的超限结果，不是测试失败。
+
+默认逻辑等待约 12 秒，表中的完整进程退出另含 GPU/驱动资源清理时间。最初使用的 1 秒完整退出判据过于严格：清理 GPU 资源可使总耗时略超过 1 秒；最终以不超过 2 秒验收主动停止，同时记录实测值。lazy-inference/use-can 的流水线停止日志也已记录，详见 JSON。
+
+实机发现并修复：use-can 关闭时遗漏 munmap，导致重新打开后 VIDIOC_S_FMT 返回 EBUSY。修复后重跑该分支全部九项。另有原生 V4L2 重开回归测试，检查连续三次重开及部分 mmap 失败后无遗留映射。
+
+补充回归：在板端执行 tests/run_camera_timeout.sh 和 --wall-clock（默认六窗口约 12.00 秒）；use-can 额外执行 tests/run_v4l2_reopen.sh。它们使用模拟采集，仅为补充，不替代上述板端真实推理测试。
+
+测试范围限制：未进行物理拔线、USB-CAN 报文/拔插、驱动内核永久卡死、GPU 故障或长时间耐久测试。板上未接 USB-CAN，CAN 模式处于缺失回退。视频回放路径未改动，未包含在本次相机恢复测试中。
+
+完整数据：orin_hardware_results.json。故障注入器源码：orin_capture_faults.c，仅用于测试。原始运行日志位于板端 /home/orin/vision-anti-pinch-ctrlc-test-20260923/。
+
+## 经测试的可执行文件 SHA256
+
+- lazy-inference: `0d77d17626644fb50b663c33edbb1a09884b34b5410a1109fc56b4d3781651b1`
+- inference-only: `3a984e1e4c0fa12285967085e3359a5b7c698d744e9c2deb2480a312eb637c1c`
+- use-can: `a2616e193ef9f006b49b4bb4ac43ae3ffc4cee1f1edf6d58ed6f7f9e5f933ad1`

tests/camera_timeout_test.cpp

@@ -0,0 +1,192 @@
+// Exercise the production Camera with simulated capture and no GPU or camera.
+#include "camera.hpp"
+#include "uyvy_converter.hpp"
+#include <linux/videodev2.h>
+#include <chrono>
+#include <cstring>
+#include <functional>
+#include <iostream>
+#include <stdexcept>
+#include <string>
+#include <thread>
+#ifdef TEST_V4L2_CAPTURE
+#include "v4l2_capture.h"
+#else
+#include <cstdarg>
+#include <cerrno>
+#include <poll.h>
+#include <sys/mman.h>
+#include <sys/ioctl.h>
+#endif
+
+namespace {
+std::string events;
+std::size_t cursor{};
+char current{};
+unsigned sequence{};
+int timeoutMs{};
+bool realWait{};
+std::function<void()> onStop;
+unsigned char pixels[8]{};
+void check(bool condition,char const* message) {
+    if(!condition)throw std::runtime_error(message);
+}
+void script(std::string value) {events=std::move(value);cursor=0;}
+char next() {
+    check(cursor<events.size(),"Camera requested an unexpected extra capture attempt");
+    return current=events[cursor++];
+}
+void expectTimeout(cms::Camera& camera) {
+    try {camera.read();} catch(std::runtime_error const& e) {
+        check(std::string(e.what()).find("6 consecutive times")!=std::string::npos,"Wrong timeout failure");
+        return;
+    }
+    throw std::runtime_error("Six consecutive timeouts did not fail");
+}
+}
+namespace cms {
+struct UyvyConverter::Impl {int width,height;};
+UyvyConverter::UyvyConverter(int w,int h):impl_(std::make_unique<Impl>(Impl{w,h})){}
+UyvyConverter::~UyvyConverter()=default;
+std::shared_ptr<const BgrImage> UyvyConverter::convert(void const*,std::size_t) {
+    auto image=std::make_shared<BgrImage>();image->width=impl_->width;image->height=impl_->height;
+    image->stride=image->width*3;image->pixels.resize(image->stride*image->height);return image;
+}
+}
+#ifdef TEST_V4L2_CAPTURE
+V4l2Capture::~V4l2Capture()=default;
+bool V4l2Capture::open(std::string const&){reconnect_needed_=false;return true;}
+void V4l2Capture::close(){}
+bool V4l2Capture::setFormat(uint32_t w,uint32_t h,uint32_t p){width_=w;height_=h;pixelformat_=p;return true;}
+bool V4l2Capture::setFps(uint32_t){return true;}
+bool V4l2Capture::initMmap(uint32_t){return true;}
+bool V4l2Capture::startStreaming(){return true;}
+void V4l2Capture::releaseFrame(Frame const&){}
+bool V4l2Capture::getFrame(Frame& f,int waitMs) {
+    check(waitMs<=100,"Stop polling should remain at most 100 ms");
+    if(next()=='s'){onStop();return false;}
+    if(current=='d'){reconnect_needed_=true;return false;}
+    if(current=='e'||current=='x'){return false;}
+    if(current=='t') {
+        // Each scripted timeout represents a full logical capture window.
+        // The separate stop test exercises the short-wait cancellation path.
+        std::this_thread::sleep_for(std::chrono::milliseconds(timeoutMs));return false;
+    }
+    check(current=='f',"Unsupported mock V4L2 event");
+    f.data=pixels;f.size=sizeof(pixels);f.width=2;f.height=2;f.bytes_per_line=4;f.sequence=sequence++;
+    return true;
+}
+#else
+extern "C" {
+int __real_poll(pollfd*,nfds_t,int);
+int __real_open(char const*,int,...);
+int __real_close(int);
+int __real_ioctl(int,unsigned long,...);
+void* __real_mmap(void*,size_t,int,int,int,off_t);
+int __real_munmap(void*,size_t);
+int __wrap_open(char const* path,int flags,...) {
+    if(std::strcmp(path,"/mock/camera")==0)return 100000;
+    return __real_open(path,flags);
+}
+int __wrap_close(int fd){return fd==100000?0:__real_close(fd);}
+void* __wrap_mmap(void* a,size_t n,int p,int f,int fd,off_t o) {
+    return fd==100000?pixels:__real_mmap(a,n,p,f,fd,o);
+}
+int __wrap_munmap(void* a,size_t n){return a==pixels?0:__real_munmap(a,n);}
+int __wrap_ioctl(int fd,unsigned long request,...) {
+    va_list args;va_start(args,request);void* p=va_arg(args,void*);va_end(args);
+    if(fd!=100000)return __real_ioctl(fd,request,p);
+    switch(request) {
+    case VIDIOC_QUERYCAP:static_cast<v4l2_capability*>(p)->capabilities=V4L2_CAP_VIDEO_CAPTURE|V4L2_CAP_STREAMING;break;
+    case VIDIOC_S_FMT:static_cast<v4l2_format*>(p)->fmt.pix.bytesperline=4;break;
+    case VIDIOC_G_PARM:break;
+    case VIDIOC_REQBUFS:static_cast<v4l2_requestbuffers*>(p)->count=2;break;
+    case VIDIOC_QUERYBUF:static_cast<v4l2_buffer*>(p)->length=sizeof(pixels);break;
+    case VIDIOC_DQBUF: {
+        if(current=='x'){errno=EIO;return -1;}
+        auto& b=*static_cast<v4l2_buffer*>(p);b.index=0;b.bytesused=sizeof(pixels);b.sequence=sequence++;
+        b.flags=current=='e'?V4L2_BUF_FLAG_ERROR:0;break;
+    }
+    case VIDIOC_QBUF:case VIDIOC_STREAMON:case VIDIOC_STREAMOFF:break;
+    default:throw std::runtime_error("Unexpected camera ioctl");
+    }
+    return 0;
+}
+int __wrap_poll(pollfd* fds,nfds_t count,int waitMs) {
+    if(count==1)return __real_poll(fds,count,waitMs);
+    check(count==2&&waitMs>0&&waitMs<=timeoutMs,"Camera must respect the configured timeout window");
+    switch(next()) {
+    case 't':if(realWait)std::this_thread::sleep_for(std::chrono::milliseconds(waitMs));return 0;
+    case 'i':errno=EINTR;return -1;
+    case 's':onStop();return 0;
+    case 'd':fds[0].revents=POLLHUP;return 1;
+    case 'f':case 'e':case 'x':fds[0].revents=POLLIN;return 1;
+    default:throw std::runtime_error("Unexpected poll event");
+    }
+}
+int __wrap___poll_chk(pollfd* fds,nfds_t count,int waitMs,size_t size) {
+    check(size>=count*sizeof(pollfd),"Invalid fortified poll buffer size");
+    return __wrap_poll(fds,count,waitMs);
+}
+}
+#endif
+int main(int argc,char** argv)try {
+    cms::CameraConfig config;config.device="/mock/camera";config.width=config.height=2;config.buffers=2;
+    check(config.timeout_ms==2000,"Default per-attempt timeout must be 2 seconds");
+    realWait=argc>1&&std::string(argv[1])=="--wall-clock";
+    if(!realWait)config.timeout_ms=100;
+    timeoutMs=config.timeout_ms;
+    if(realWait) {
+        cms::Camera camera(config);script("tttttt");
+        auto before=std::chrono::steady_clock::now();expectTimeout(camera);
+        auto elapsed=std::chrono::duration<double>(std::chrono::steady_clock::now()-before).count();
+        check(cursor==6&&elapsed>=12.0&&elapsed<15.0,"Default timeout should fail after six attempts, about 12 seconds");
+        std::cout<<"PASS wall-clock six-timeout duration="<<elapsed<<" s\n";
+        return 0;
+    }
+    {
+        cms::Camera camera(config);
+        script("tttttf");check(bool(camera.read()),"Five timeouts should recover on a valid frame");
+        script("tttttf");check(bool(camera.read()),"A valid frame must reset the timeout count");
+        script("tttttt");
+        expectTimeout(camera);
+        check(cursor==6,"Failure must occur on the sixth timeout");
+    }
+    {
+        cms::Camera camera(config);onStop=[&]{camera.requestStop();};
+        script("tts");check(!camera.read(),"Stop must cancel retries without throwing");
+        check(cursor==3,"Stop should not wait for all six timeouts");onStop={};
+    }
+    {
+        cms::Camera camera(config);onStop=[&]{
+            std::this_thread::sleep_for(std::chrono::milliseconds(timeoutMs));
+            camera.requestStop();
+        };
+        script("ttttts");check(!camera.read(),"Stop on the final timeout must take precedence over failure");
+        onStop={};
+    }
+#ifndef TEST_V4L2_CAPTURE
+    {
+        cms::Camera camera(config);script("tttttif");
+        check(bool(camera.read()),"EINTR must not count as a timeout");
+        script("tttttet");expectTimeout(camera);
+        check(cursor==7,"A corrupt frame must not reset the timeout count");
+    }
+    {
+        cms::Camera camera(config);script("df");
+        check(bool(camera.read()),"Capture must reopen after a transient disconnection");
+    }
+    {
+        cms::Camera camera(config);script("xxf");
+        check(bool(camera.read()),"Transient EIO must recover without restarting the pipeline");
+    }
+    {
+        auto slow=config;slow.timeout_ms=2000;timeoutMs=slow.timeout_ms;
+        cms::Camera camera(slow);script(std::string(35,'e')+"f");
+        check(bool(camera.read()),"More than one second's frame count of corrupt frames must remain recoverable");
+        timeoutMs=config.timeout_ms;
+    }
+#endif
+    std::cout<<"PASS camera timeout retries, recovery reset, sixth-timeout failure and cancellation\n";
+    return 0;
+}catch(std::exception const& e){std::cerr<<"FAIL: "<<e.what()<<'\n';return 1;}

tests/orin_capture_faults.c

@@ -0,0 +1,39 @@
+#define _GNU_SOURCE
+#include <dlfcn.h>
+#include <poll.h>
+#include <sys/select.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <time.h>
+#include <stdarg.h>
+#include <errno.h>
+#include <linux/videodev2.h>
+#include <sys/ioctl.h>
+static int calls=0;
+static int camera_fd(int fd){char name[64],target[256];snprintf(name,sizeof(name),"/proc/self/fd/%d",fd);ssize_t n=readlink(name,target,sizeof(target)-1);if(n<0)return 0;target[n]=0;return strcmp(target,"/dev/video5")==0;}
+static long long now_ms(void){struct timespec t;clock_gettime(CLOCK_MONOTONIC,&t);return (long long)t.tv_sec*1000+t.tv_nsec/1000000;}
+static void marker(const char *s){ssize_t ignored=write(2,s,strlen(s));(void)ignored;}
+static int should_stall(void){
+ static long long since=0;static int phase=0,resumed=0;
+ if(!getenv("VAP_TEST_STALL"))return 0;
+ if(phase==0){if(++calls<60)return 0;phase=1;since=now_ms();marker("TEST_STALL_ACTIVE\n");}
+ if(phase==1&&getenv("VAP_TEST_RECOVER")&&now_ms()-since>=4200){phase=2;marker("TEST_CAMERA_RECOVERED\n");return 0;}
+ if(phase==2){if(++resumed<30)return 0;phase=3;marker("TEST_STALL_SECOND\n");}
+ return phase==1||phase==3;
+}
+int poll(struct pollfd *fds,nfds_t count,int timeout){static int(*real_poll)(struct pollfd*,nfds_t,int);if(!real_poll)real_poll=dlsym(RTLD_NEXT,"poll");for(nfds_t i=0;i<count;++i)if(!getenv("VAP_TEST_KIND")&&camera_fd(fds[i].fd)&&should_stall()){int saved=fds[i].fd;fds[i].fd=-1;int rc=real_poll(fds,count,timeout);fds[i].fd=saved;return rc;}return real_poll(fds,count,timeout);}
+int __poll_chk(struct pollfd *fds,nfds_t count,int timeout,size_t size){(void)size;return poll(fds,count,timeout);}
+int select(int count,fd_set *readfds,fd_set *writefds,fd_set *exceptfds,struct timeval *timeout){static int(*real_select)(int,fd_set*,fd_set*,fd_set*,struct timeval*);if(!real_select)real_select=dlsym(RTLD_NEXT,"select");if(readfds)for(int fd=0;fd<count;++fd)if(!getenv("VAP_TEST_KIND")&&FD_ISSET(fd,readfds)&&camera_fd(fd)&&should_stall()){FD_CLR(fd,readfds);break;}return real_select(count,readfds,writefds,exceptfds,timeout);}
+
+int ioctl(int fd,unsigned long request,...){
+ static int(*real_ioctl)(int,unsigned long,...);if(!real_ioctl)real_ioctl=dlsym(RTLD_NEXT,"ioctl");
+ va_list args;va_start(args,request);void *arg=va_arg(args,void*);va_end(args);
+ const char *kind=getenv("VAP_TEST_KIND");
+ if(kind&&request==VIDIOC_DQBUF&&camera_fd(fd)&&should_stall()){
+  if(strcmp(kind,"bad_frames")==0){int rc=real_ioctl(fd,request,arg);if(rc==0)((struct v4l2_buffer*)arg)->flags|=V4L2_BUF_FLAG_ERROR;return rc;}
+  errno=strcmp(kind,"device_error")==0?ENODEV:EIO;return -1;
+ }
+ return real_ioctl(fd,request,arg);
+}

tests/orin_hardware_results.json

@@ -0,0 +1,253 @@
+[
+  {
+    "branch": "lazy-inference",
+    "mode": "recover",
+    "exit_code": 1,
+    "inference_json_frames": 88,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "six_timeout_exit_seconds": 13.085,
+    "recovered_real_capture": true
+  },
+  {
+    "branch": "lazy-inference",
+    "mode": "bad_frames_recover",
+    "exit_code": 0,
+    "inference_json_frames": 67,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 846.3,
+    "recovered_real_capture": true,
+    "frames_after_recovery": 10,
+    "pipeline_stop_ms": 132.56
+  },
+  {
+    "branch": "lazy-inference",
+    "mode": "io_error_recover",
+    "exit_code": 0,
+    "inference_json_frames": 67,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 836.09,
+    "recovered_real_capture": true,
+    "frames_after_recovery": 10,
+    "pipeline_stop_ms": 106.06
+  },
+  {
+    "branch": "lazy-inference",
+    "mode": "device_error_recover",
+    "exit_code": 0,
+    "inference_json_frames": 68,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 830.0,
+    "recovered_real_capture": true,
+    "frames_after_recovery": 10,
+    "pipeline_stop_ms": 133.41
+  },
+  {
+    "branch": "lazy-inference",
+    "mode": "live",
+    "exit_code": 0,
+    "inference_json_frames": 3,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 833.16,
+    "pipeline_stop_ms": 139.5
+  },
+  {
+    "branch": "lazy-inference",
+    "mode": "retry",
+    "exit_code": 0,
+    "inference_json_frames": 58,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 942.42,
+    "pipeline_stop_ms": 201.21
+  },
+  {
+    "branch": "lazy-inference",
+    "mode": "sixth_wait",
+    "exit_code": 0,
+    "inference_json_frames": 58,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 904.29,
+    "pipeline_stop_ms": 194.16
+  },
+  {
+    "branch": "lazy-inference",
+    "mode": "sigterm",
+    "exit_code": 0,
+    "inference_json_frames": 3,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 801.21,
+    "pipeline_stop_ms": 123.09
+  },
+  {
+    "branch": "lazy-inference",
+    "mode": "timeout",
+    "exit_code": 1,
+    "inference_json_frames": 58,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "six_timeout_exit_seconds": 13.01
+  },
+  {
+    "branch": "inference-only",
+    "mode": "recover",
+    "exit_code": 1,
+    "inference_json_frames": 88,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "six_timeout_exit_seconds": 12.945,
+    "recovered_real_capture": true
+  },
+  {
+    "branch": "inference-only",
+    "mode": "bad_frames_recover",
+    "exit_code": 0,
+    "inference_json_frames": 68,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 836.67,
+    "recovered_real_capture": true,
+    "frames_after_recovery": 10
+  },
+  {
+    "branch": "inference-only",
+    "mode": "io_error_recover",
+    "exit_code": 0,
+    "inference_json_frames": 68,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 827.43,
+    "recovered_real_capture": true,
+    "frames_after_recovery": 10
+  },
+  {
+    "branch": "inference-only",
+    "mode": "device_error_recover",
+    "exit_code": 0,
+    "inference_json_frames": 68,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 853.08,
+    "recovered_real_capture": true,
+    "frames_after_recovery": 10
+  },
+  {
+    "branch": "inference-only",
+    "mode": "live",
+    "exit_code": 0,
+    "inference_json_frames": 3,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 804.87
+  },
+  {
+    "branch": "inference-only",
+    "mode": "retry",
+    "exit_code": 0,
+    "inference_json_frames": 58,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 913.42
+  },
+  {
+    "branch": "inference-only",
+    "mode": "sixth_wait",
+    "exit_code": 0,
+    "inference_json_frames": 58,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 1012.57
+  },
+  {
+    "branch": "inference-only",
+    "mode": "sigterm",
+    "exit_code": 0,
+    "inference_json_frames": 3,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 793.22
+  },
+  {
+    "branch": "inference-only",
+    "mode": "timeout",
+    "exit_code": 1,
+    "inference_json_frames": 58,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "six_timeout_exit_seconds": 13.055
+  },
+  {
+    "branch": "use-can",
+    "mode": "recover",
+    "exit_code": 1,
+    "inference_json_frames": 88,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "six_timeout_exit_seconds": 13.155,
+    "recovered_real_capture": true
+  },
+  {
+    "branch": "use-can",
+    "mode": "bad_frames_recover",
+    "exit_code": 0,
+    "inference_json_frames": 67,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 815.49,
+    "recovered_real_capture": true,
+    "frames_after_recovery": 10,
+    "pipeline_stop_ms": 122.36
+  },
+  {
+    "branch": "use-can",
+    "mode": "io_error_recover",
+    "exit_code": 0,
+    "inference_json_frames": 68,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 836.05,
+    "recovered_real_capture": true,
+    "frames_after_recovery": 10,
+    "pipeline_stop_ms": 130.73
+  },
+  {
+    "branch": "use-can",
+    "mode": "device_error_recover",
+    "exit_code": 0,
+    "inference_json_frames": 68,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 814.68,
+    "recovered_real_capture": true,
+    "frames_after_recovery": 10,
+    "pipeline_stop_ms": 123.99
+  },
+  {
+    "branch": "use-can",
+    "mode": "live",
+    "exit_code": 0,
+    "inference_json_frames": 3,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 786.06,
+    "pipeline_stop_ms": 108.96
+  },
+  {
+    "branch": "use-can",
+    "mode": "retry",
+    "exit_code": 0,
+    "inference_json_frames": 58,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 862.45,
+    "pipeline_stop_ms": 167.55
+  },
+  {
+    "branch": "use-can",
+    "mode": "sixth_wait",
+    "exit_code": 0,
+    "inference_json_frames": 58,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 907.59,
+    "pipeline_stop_ms": 197.07
+  },
+  {
+    "branch": "use-can",
+    "mode": "sigterm",
+    "exit_code": 0,
+    "inference_json_frames": 3,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "signal_exit_ms": 827.61,
+    "pipeline_stop_ms": 130.51
+  },
+  {
+    "branch": "use-can",
+    "mode": "timeout",
+    "exit_code": 1,
+    "inference_json_frames": 58,
+    "gpu_models": "YOLO + RTMW TensorRT engines",
+    "six_timeout_exit_seconds": 13.066
+  }
+]

tests/run_camera_timeout.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+set -euo pipefail
+root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
+test_dir="$(mktemp -d)"
+trap 'rm -rf "$test_dir"' EXIT
+flags=(-std=c++17 -O1 -g -Wall -Wextra -Wpedantic -pthread -fsanitize=undefined -fno-sanitize-recover=all -I"$root/include")
+links=()
+if [[ -f "$root/include/v4l2_capture.h" ]]; then
+  flags+=(-DTEST_V4L2_CAPTURE -I"$root/tests/stubs")
+else
+  for fn in open close ioctl mmap munmap poll __poll_chk; do links+=("-Wl,--wrap=$fn"); done
+fi
+"${CXX:-g++}" "${flags[@]}" "${CAMERA_SOURCE:-$root/src/camera.cpp}" "$root/tests/camera_timeout_test.cpp" "${links[@]}" -o "$test_dir/camera_timeout_test"
+"$test_dir/camera_timeout_test" "$@"

tests/stubs/opencv2/videoio.hpp

@@ -0,0 +1,19 @@
+#pragma once
+// Camera timeout tests exercise live capture only. Stub the unused replay API
+// so the use-can Camera can be tested without OpenCV or a video decoder.
+#include <stdexcept>
+#include <string>
+namespace cv {
+constexpr int CAP_PROP_FRAME_WIDTH=3,CAP_PROP_FRAME_HEIGHT=4;
+struct Mat {
+    int cols{},rows{};
+    unsigned char const* ptr(int)const {throw std::logic_error("Video replay is outside this test");}
+};
+class VideoCapture {
+public:
+    bool isOpened()const{return false;}
+    bool open(std::string const&){throw std::logic_error("Video replay is outside this test");}
+    double get(int)const {throw std::logic_error("Video replay is outside this test");}
+    bool read(Mat&){throw std::logic_error("Video replay is outside this test");}
+};
+}