##Project Background In 2024, through a friend's recommendation, I came across a need for rehabilitation at a tertiary hospital in Shanxi. They face a common problem in many large hospitals in China: assessing the mobility of elderly hospitalized patients, relying on manual scales for a long time, low efficiency, strong subjectivity, and high requirements for doctor experience. Traditional tools such as Tinetti gait assessment and Berg balance scale require doctors or rehabilitation therapists to observe patients performing a series of movements at the bedside and score them item by item. For a rehabilitation department, this process needs to be repeated dozens of times a day, taking up a lot of time that could have been used for treatment. More importantly, there are subjective differences in the scoring criteria of different doctors, and the same patient may be evaluated by two people to arrive at different conclusions, which brings uncertainty to the subsequent development of rehabilitation plans. They hope to have a system that can automatically recognize the patient's movement posture through cameras, assist in generating scores, reduce the repetitive labor of medical staff, and make the evaluation results more objective and traceable. This requirement is both unfamiliar and familiar to me - familiar with visual recognition technology, unfamiliar with the dense implicit constraints in hospital scenes. I didn't refuse and took on this project.
##Requirement research: much more complex than I imagined Before the official development, I made a special trip to the hospital and followed up with the rehabilitation department for three days. These three days are one of the most valuable time investments for the entire project. I thought the requirements were very clear: taking videos, recognizing actions, and outputting scores. But after follow-up, it was found that reality is much more complex than this:
1、 The issue with camera position The hospital beds, treatment tables, and corridors in the rehabilitation room are intricate and complex, without a fixed "standard shooting angle". When patients perform different movements, their body orientation varies, and there are situations where they turn sideways or backwards. The system must be insensitive to angles, or at least provide reliable results within a limited range of angles.
2、 The Red Line of Data Privacy The hospital has explicitly stated that patient video data must not be transmitted to external servers, cannot be uploaded to the cloud, and must be processed locally within the hospital. This fundamentally determines the entire system architecture - we must take the edge computing route.
3、 The actual usage scenarios of doctors Doctors do not sit in front of a computer waiting for system output. They hope that the system can automatically record the patient's evaluation and directly see the conclusion on the screen next to the medical record system after completion, without the need for additional operations. The interface should be minimalist and the information density should be low.
These three constraints have completely reshaped my imagination of this project.
##Technical solution selection
Hardware Selection: Why Jetson Nano If data cannot be uploaded to the cloud, it means that inference must be completed locally. There are several options for local inference:
Ordinary industrial control computer+independent GPU: capable, but hospitals are unwilling to allocate a separate cabinet space for this system, which poses significant deployment resistance
Raspberry Pi and other ARM boards: low power consumption, small size, but without GPU, pure CPU running visual model frame rate is completely insufficient
NVIDIA Jetson Nano: 4GB memory, 128 core Maxw GPU, power consumption of about 5-10W, volume close to a palm, can be directly stuffed into the drawer of a nurse station trolley
Jetson Nano is the only reasonable choice in this scenario. Not because it has the strongest performance, but because it has found the best balance between "sufficient" performance and "floor to ceiling" volume power consumption.
Model selection: YOLOv11 Pose
The core of action capability assessment is to extract the key points of human bones, including the positions and confidence levels of 17 standard nodes such as shoulders, elbows, wrists, hips, knees, ankles, etc. YOLOv11 Pose was the best choice at the time in terms of both accuracy and speed
Single stage detection with low latency,built in keypoint regression head, no need to separately train posture estimation model,and the official provides multiple sizes (n/s/m/l), which can be flexibly selected according to hardware conditions
I ultimately chose YOLOv11m Pose, which has a native inference speed of about 14fps on Jetson Nano. After optimization with TensorRT, it has been increased to about 26-29fps, meeting the real-time evaluation requirements.
##Evaluation Algorithm: From Scale to Code This is the most technically advanced and easily underestimated part of the entire project. The Tinetti scale decomposes gait and balance into several sub items, each with a scoring criterion of 0/1/2. My task is to translate the standards described in these words into computable rules. Give a few examples:
Scale description code implementation: "Gait symmetry: two step lengths are basically the same". Extract the stride spacing between the left and right ankles, calculate the difference between consecutive frames, and record the deviation<15% as symmetry. "Trunk stability: no obvious swaying of the trunk during walking". Track the lateral displacement curve of the shoulder midpoint, calculate the standard deviation, and threshold the judgment of "getting up action: one successful standing up". Identify the transition from the distribution of key points in sitting posture to standing posture, and record the number of times to get up
This translation process heavily relies on domain knowledge. It took me nearly two weeks of follow-up and repeated communication with doctors to understand the actual meaning behind each text description in the scale. The doctor said 'stable gait', I want to ask: What does stability mean? What action is used to judge? How much is considered unstable?
##Overview of System Architecture
Camera (USB connected to Jetson Nano) ↓ Real time video frame capture (OpenCV) ↓ YOLOv11 Pose inference (TensorRT acceleration) ↓ Key point coordinate output (17 points x [x, y, conf]) ↓ Action capability assessment algorithm (angle calculation+trajectory analysis) ↓ Score generation+report rendering ↓ Lightweight Web Interface (Public Network Deployment Access)
The pitfalls encountered during the development process
Pit 1: Changes in lighting lead to a sudden drop in confidence of key points The color temperature and brightness of the fluorescent lamps in the rehabilitation room vary greatly at different time periods. The morning is cold with white light, and the afternoon sunlight is slanting in through the window, with an overall warm yellow color tone. This color temperature change in RGB space directly affects the model's recognition of skin and clothing edges, resulting in a significant decrease in keypoint confidence in the afternoon, especially for small nodes such as the wrist and ankle.
Solution: 1、During the preprocessing stage, CLAHE (Contrast Constrained Adaptive Histogram Equalization) processing was performed to improve lighting uniformity without sacrificing details 2、Convert the image from RGB to LAB space before inference, normalize the L channel, and reduce the impact of color temperature 3、Display a prompt on the UI stating 'current lighting is poor, suggest adjusting the camera' for key points with a relative reliability of less than 0.5, instead of directly outputting incorrect results
This question made me realize that models that run well in controlled laboratories may encounter various unexpected environmental disturbances in real hospital scenarios, and robust design is not a bonus, but a fundamental aspect.
Pit 2: Jetson Nano's Memory Wall The 4GB memory of Jetson Nano is shared between the CPU and GPU, and with the system occupying approximately 1.5GB, the space available for inference is actually quite limited. Running YOLOv11s Pose directly with PyTorch consumes nearly 2GB of memory and has a inference speed of only about 8fps, which is far from sufficient.
Solution: TensorRT quantization conversion 1、Export the PyTorch model to ONNX, perform FP16 quantization conversion using TensorRT, and generate an. engine file. After conversion: Inference speed: 14fps → 26-29fps Video memory usage: approximately 2.7GB → approximately 1.2GB Accuracy loss: Keypoint coordinate error<6 pixels, within acceptable range
The conversion of TensorRT itself is not complicated, but it has fallen into a pit: the TensorRT version (8. x) of Jetson Nano is incompatible with the PC version, and cannot generate. engine files on the PC for direct copying and use. The conversion must be completed locally on Jetson. This process takes about 15-20 minutes, but only needs to be done once.
Pit 3: The Semantic Gap in Scale Alignment This is the most difficult problem in the entire project, and also the least like a 'technical issue'.
One of the scoring criteria in the scale is "step continuity: no pauses or discontinuities in step". My initial understanding was: to detect the movement speed of ankle keypoints, frames with speeds close to zero are marked as "pauses", and if the number of pauses exceeds the threshold, points will be deducted.
But after seeing my implementation, the doctor said: it's not right. Older people walk slowly, and occasionally slowing down to adjust their center of gravity is normal, not considered a pause. The pause that truly needs to be recognized is the one that "hesitates for a moment and clearly loses its sense of rhythm".This sentence took me three whole days to rework. In the end, I changed to using the standard deviation of step frequency to measure continuity: people with stable step frequency have lower standard deviation; People with fluctuating step frequency have a high standard deviation. This indicator is closer to the actual judgment logic of doctors than 'whether the speed is zero'.
A similar alignment process is experienced for almost every sub item of the scale. This made me deeply understand one thing: in the medical field, there is always a gap between "technological implementation" and "clinical semantics", and the only way to cross this gap is through repeated communication, rather than guessing behind closed doors.
Pit 4: Key point deviation caused by differences in patient body shape
The body types of elderly patients vary greatly - some are less than 150cm tall, some weigh over 90kg, and hunchback and joint deformities are also common. These factors can lead to systematic bias in predicting key bone points: the performance of the model on training data (mainly for middle-aged and young people) significantly decreases when transferred to elderly people.
Temporary solution: We collected approximately 200 sets of posture video clips of elderly people and conducted small-scale domain adaptation fine-tuning in the hospital,Introduced bone length constraints for key point prediction of hip, knee, and ankle, triggering correction when predicted values violate body proportions.This problem has not been completely resolved, it has only been controlled within an acceptable range. If standardized products are to be developed in the future, the accumulation of specialized datasets for elderly posture is an indispensable task.
Final delivery
The system completed a two-week trial operation in the hospital and passed the acceptance test of the rehabilitation department. Main indicators:
Accuracy of key action recognition:~89% (matching with manual scoring by doctors) The average time for a single evaluation is about 6 minutes (traditional manual evaluation takes about 15-25 minutes) System stability: Continuous operation for 72 hours without crashing
I have completed the independent payment. This is the first project since I started my business that has truly gone through the business cycle - from receiving the demand to receiving the money, without any unfinished projects or disputes.