Torq has raised $70 million in a Series C funding round to expand its artificial intelligence (AI)-first cybersecurity hyperautomation solutions. The

Torq Raises $70 Million to Expand AI-First Cybersecurity Hyperautomation Solutions | PYMNTS.com

Torq, a startup developing a platform to automate specific cybersecurity tasks, has raised $70 million in a Series C round.

Torq, which automates cybersecurity workflows, raises $70M in new capital | TechCrunch

Torq

torq

In the era of widespread adoption of AI models, the data science community exhibits a rich diversity of backgrounds, fostering an environment conducive to innovation. Collaborative efforts stemming from this diversity continue to propel us toward the envisioned future of a decade or two ago. However, innovation often follows incremental patterns and may require decades before yielding tangible commercial benefits. Among industries adept at translating ideas into tangible products, automotive manufacturers stand out, boasting a deep familiarity with research and development processes and advanced algorithms. This proficiency is evident in the innovation pipeline for super and hypercars, drawing inspiration from advancements in rocket science. Leveraging extensive simulations and field tests, these manufacturers consistently push the boundaries of performance and safety in vehicle design. Major manufacturers have achieved breakthroughs with torque vectoring systems by capitalizing on established development pipelines and harnessing the capabilities of reinforcement learning. The late 2010s witnessed a surge in commercial vehicles featuring these systems, attributed partly to their association with electric and hybrid vehicles and partly to the appeal of enhanced safety. While the full extent of their influence on driver experience remains to be realized, these systems have already demonstrated remarkable efficacy, particularly in high-performance vehicles. So, what exactly is a torque vectoring system, and how is this innovation shaping the landscape of automotive manufacturing? What is Torque Vectoring In Torque Vectoring Systems (TVS), torque denotes a rotational force applied to a vehicle’s differentials or generated by independent electric motors on its wheels. These systems aim to regulate this rotational force to enhance speed, stability, handling, balance, and other performance aspects. A compelling illustration of the protracted nature of innovation is evident in the evolution of torque vectoring systems, with the first cars equipped with Torque Vectoring Systems (TVS) debuting in the 1990s, pioneered by Mitsubishi. Initially known as Active Yaw Control, Mitsubishi integrated this technology into vehicles like the 1996 Mitsubishi Lancer Evolution IV GSR models. The initial implementation relied on steering, throttle inputs, and g-force sensors to mitigate torque steer, a phenomenon where engine torque influences steering behavior. Initially geared towards racing applications, these systems have since evolved to cater to broader audiences, with companies like Rimac continuing to employ them for racing. In contrast, others emphasize safety and enhanced handling features for everyday drivers. Furthermore, advancements driven by the pursuit of eco-friendly transportation have led to the utilization of independent electric motors in torque vectoring systems, further expanding their applicability. Commercial Applications Ricardo (Honorable Mention) In 2006, Ricardo, a renowned engineering consultancy with over a century of experience, introduced the concept of torque vectoring. Ricardo rolled out its initial torque vectoring project centered around the Audi A6 4.2L, intending to license its technology to interested automakers. Unlike Mitsubishi’s Active Yaw Control system and similar counterparts, Ricardo’s torque vectoring approach prioritized safety and stability, showcasing a different aspect of the technology. Limited information is currently available on Ricardo’s involvement in assisting manufacturers in developing these systems for commercially available vehicles. However, they have publicly provided engineering support to Chinese students building cars for the global multi-event Formula SAE race in 2019. Rimac When discussing electric vehicles, Tesla typically takes the spotlight regarding market dominance. However, Rimac stands out as a company dedicated to crafting high-performance electric hypercars, where they excel. Just ask Elon… ( t) X pos Rimac burst onto the scene in January 2016 with their inaugural concept car, the Concept_One prototype. This marked the debut of their torque vectoring system (TVS) in an all-wheel-drive fully electric vehicle. Spearheaded by Tomislav Šimunić, Rimac’s TVS development pushed the boundaries by employing individual motors on each wheel. Moreover, they implemented reinforcement algorithms utilizing data from various sensors, including accelerometers, gyroscopes, steering angle, and wheel speed sensors. This data is then used to predict torque distribution to optimize vehicle stability. //www.youtube.com/shorts/i4jVW29gbDc?embedable=true Currently, Rimac boasts one of the fastest and safest hypercars on the market. Their innovation has led to unprecedented control, resulting in their car achieving the title of the fastest production car ever in 21 acceleration and deceleration categories. Furthermore, their TVS is so reliable that their drivers can cruise hands-free at speeds exceeding 230 mph while enjoying gelato! Toyota In 2018, Toyota introduced their Dynamic Torque Vectoring AWD and Electric 4WD systems, expanding the scope of this technology beyond on-road safety and control to encompass off-road safety features and enhanced fuel efficiency. //www.youtube.com/watch?v=LCqCa7PP1gE&embedable=true The video above showcases Toyota’s emphasis on “proactive torque transfer,” allowing for seamless adjustment of torque distribution when one or more wheels lose traction, enabling drivers to navigate various terrains more effectively. //youtu.be/d1UG1a38diw?embedable=true Furthermore, Toyota optimizes torque distribution to enhance fuel-efficiency, shifting from AWD to 2WD when conditions permit, directing torque predominantly to the front wheels and thus cutting fuel consumption. Their limited content suggests the employment of a Dynamic Control System designed for stability, taking into account challenging driving scenarios. Moreover, the system can evaluate tire traction levels. These functionalities resonate with the research findings discussed below, showcasing the seamless transition from research concepts to practical implementation in the automotive industry. Research Review Conceptual Models — Bicycle, Vehicle, Tire Model Following Occam’s razor principle, simplicity reigns supreme when initiating modeling endeavors, with complexity introduced only when its benefits are worthwhile. This philosophy holds for TVS as well. Early TVS adopted a bicycle model, allowing researchers to streamline vehicle dynamics assumptions by confining movements to two wheels. This simplified framework facilitated the study of lateral vehicle dynamics during various events like braking, acceleration, and cornering under typical conditions. Key inputs into the bicycle model include yaw rate, lateral acceleration, steering angle, and velocity. Yaw rate denotes the vehicle’s rotation speed around its vertical axis, aiding in determining its orientation during maneuvers like cornering. Lateral acceleration captures sideways movements perpendicular to the vehicle’s direction of travel. Steering angle refers to the front wheel angle, typically in relation to the rear wheel positioning, while velocity represents the vehicle’s speed. For more details, check out this article explaining a simple bicycle model. //dingyan89.medium.com/simple-understanding-of-kinematic-bicycle-model-81cac6420357?embedable=true Simplicity in modeling often comes with assumptions that may not hold, especially in challenging driving scenarios. Extreme conditions like rain, snow, or rough off-road terrain expose the limitations of the traditional bicycle model. The assumption of symmetry between the left and right wheels hinders the accurate analysis of lateral dynamics across all four wheels. To address this, researchers upgraded to a more comprehensive 4-wheel vehicle model. One such model, proposed in the 2023 journal article by Hui Deng, Youqun Zhao, Fen Lin, and Qiuwei Wang, boasts seven degrees of freedom. This expanded model captures all four wheels’ lateral, longitudinal, rotational, tilting, leaning, and spinning movements. However, it still falls short in capturing driving dynamics such as suspension behavior and tire-road interactions. Although suspension dynamics may be indirectly represented within the model, a separate tire model is necessary to capture tire-road interactions accurately. Deep Reinforcement Learning-Based Torque Vectoring Control Considering Economy & Safety² Tire models typically include parameters like tire force and slip angles. In the journal article by Michele Vignati, Edoardo Sabbioni, and Federico Cheli, researchers describe a simple tire model that incorporates lateral and longitudinal dynamics, slip angles, and slippage. Like the seven degrees of freedom model indirectly addresses suspension dynamics, the tire model considers slippage influenced by latent variables such as surface texture, conditions, and angles. This comprehensive approach allows researchers to analyze vehicle dynamics across various conditions and terrains, enhancing the torque vectoring system’s adaptability and performance. A Torque Vectoring Control for Enhancing Vehicle Performance in Drifting³ Control Schemes/Implementations In Torque Vectoring Systems (TVS) development, a significant portion of research revolves around employing Reinforcement Learning techniques, leveraging real-time data collected from various measurement devices. These solutions are predominantly implemented in all-wheel drive, four-wheel drive hybrid, and electric vehicles. Typically, the training of these systems takes place within simulated environments. However, in certain instances, researchers conduct road tests and utilize live test outcomes to validate simulation results. Torque Vectoring System Testing While specific details about testing TVS are often limited, researchers generally have three options: entirely computer-based simulations, hardware-in-the-loop testing combining simulation with real parts, and full-scale vehicle road tests. Computer-based simulations offer cost-effective evaluation of TVS with precise control over various driving conditions. Hardware-in-the-loop testing balances simulation and road tests, providing insights aligned with real-world performance. Although the most expensive, road tests offer the most reliable results by implementing TVS fully, enabling the most accurate evaluation of model performance. In the study by Shayan Taherian, Sampo Kuutti, Marco Visca, and Saber Fallah, computer-based simulations were utilized to test their RL algorithm across diverse environments. The researchers employed an adaptive algorithm based on Deep Deterministic Policy Gradient to update TVS control parameters as it learns from the environment. Testing across various velocity and friction environments allowed validation of their adaptive algorithm, particularly in high-speed/low-velocity scenarios. Self-adaptive Torque Vectoring Controller Using Reinforcement Learning⁴ Similarly, in the research by Wei Xu, Hong Chen, Haiyan Zhao, and Bingtao Ren, real-time Simulink simulations were leveraged to validate their model predictive controller. This controller predicts future system states and optimizes for multiple energy efficiency and safety objectives. Tests were conducted using an electric vehicle model with a 120 km/h velocity and an “adhesion coefficient” of .85, measuring tire friction between tire and ground. Torque Optimization Control for Electric Vehicles with Four In-Wheel Motors Equipped with Regenerative Braking System⁵ Powertrain & Fuel Types The specific problem addressed by torque vectoring systems heavily influences the powertrain choice and fuel types utilized in their implementations. Traditional gasoline and diesel vehicles necessitate mechanical implementations due to a single torque source, leading to complex mechanical differentials, challenges in power distribution, and overall less efficiency compared to hybrid and electric vehicle setups, which enable independent motor implementations. Similarly, All-Wheel Drive (AWD) and Four-Wheel Drive (4WD) powertrains offer superior advantages over other options. These configurations enhance stability, versatility, and energy efficiency. The work of Jeongmin Cho and Kunsoo Huh, in their study , underscores the stability and versatility benefits of AWD vehicles. By distributing power to all four wheels, AWD enhances the effectiveness of torque vectoring systems, granting increased control over each wheel. This translates to improved vehicle dynamics during cornering, acceleration, and deceleration. Torque Vectoring System Design for Hybrid Electric–All-Wheel Drive Vehicle⁶ Meta Control Algorithms In Reinforcement Learning (RL), an agent decides actions based on the current situation to maximize a reward function. It learns which actions yield the highest rewards by assessing the outcomes of its actions. In torque vectoring algorithms, auto manufacturers define different reward functions based on the specific goals of their TVS, which vary depending on the type of vehicles used, such as sports cars versus mini-vans. These goals include increased stability, controllability, fuel economy, cornering performance, and optimizing various vehicle dynamics behaviors. One example of a multi-objective implementation is outlined in the paper by Jae-Young Park, Seung-Jin Heo, and Daeoh Kang. Their algorithm features a supervisory controller that selects between an “agile” mode and a “safe” mode based on vehicle and driving conditions. Agile mode prioritizes driver controllability, while safe mode focuses on stability. Controllability refers to the synchronization between driver steering input and vehicle direction, while stability relates to the vehicle’s resistance to deviations or disturbances. Inputs such as vehicle speed, steering angle, yaw rate, lateral acceleration, and wheel slip guide the supervisory controller’s decision-making process. Development of Torque Vectoring Control Algorithm for Front Wheel Driven Dual Motor System and Evaluation of Vehicle Dynamics Performance⁷ Another example is presented in the paper by Hui Deng, Youqun Zhao, Fen Lin, and Qiuwei Wang. Here, the researchers optimize for safety, economy, and driver load. Economy involves assessing motor efficiency and energy savings compared to vehicles without torque vectoring systems. Safety focuses on vehicle stability, while driver load considers the physical and cognitive load based on the correlation between the driver’s steering wheel angle and the vehicle’s longitudinal acceleration. Deep Reinforcement Learning-Based Torque Vectoring Control Considering Economy and Safety⁸ The algorithm includes an active safety controller and a torque allocation layer. The active safety controller calculates total longitudinal acceleration and additional “yaw moment” values based on driving conditions, applying safety and driver load penalties. These values are then fed to the torque allocation layer, which distributes torque to all four wheels. Additionally, the researchers implement a tire model using a Fibonacci tree optimization algorithm and a vehicle model with 7 degrees of freedom to enhance system performance. The , research done by Park et al., aims to enhance the driver’s experience through improved smoothness and convergence. Their TVS, designed for commercial application, underwent rigorous validation through real vehicle experiments conducted under various conditions. Torque Vectoring Algorithm of Electronic-Four-Wheel Drive Vehicles for Enhancement of Cornering Performance⁹ The researchers implemented a smooth sliding controller, GPS, and sensor inputs to generate desired yaw moment values. These values are then utilized by the torque allocation algorithm, which employs a daisy-chaining allocation method to distribute torque efficiently. The goal is to optimize the convergence from current yaw moment values to target values, thereby reducing the instability during this transition. Conclusion The evolution of torque vectoring systems underscores the transformative power of innovation within the automotive industry. From pioneering beginnings to cutting-edge applications, these systems have revolutionized vehicle dynamics, speed, stability, and handling. As automotive manufacturers continue to push the boundaries of technology and leverage advancements in advanced algorithms, the future holds promise for further improvements in safety, performance, and driver experience. However, hurdles remain in integrating this technology into internal combustion engine vehicles, and ongoing research and development efforts are required to overcome them. With continued dedication to innovation, torque vectoring systems are poised to shape the future landscape of automotive manufacturing, driving us toward a new era of mobility. Research Reference Martin Dendaluce Jahnke et al., “Efficient Neural Network Implementations on Parallel Embedded Platforms Applied to Real-Time Torque-Vectoring Optimization Using Predictions for Multi-Motor Electric Vehicles,” 8, no. 2 (February 22, 2019): 250 Electronics Huifan Deng et al., “Deep Reinforcement Learning-Based Torque Vectoring Control Considering Economy and Safety,” 11, no. 4 (April 6, 2023): 459 Machines Michele Vignati, Edoardo Sabbioni, and Federico Cheli, “A Torque Vectoring Control for Enhancing Vehicle Performance in Drifting,” 7, no. 12 (December 5, 2018): 394 Electronics Shayan Taherian et al., “Self-Adaptive Torque Vectoring Controller Using Reinforcement Learning,” , September 19, 2021 2021 IEEE International Intelligent Transportation Systems Conference (ITSC) Wei Xu et al., “Torque Optimization Control for Electric Vehicles with Four In-Wheel Motors Equipped with Regenerative Braking System,” 57 (February 2019): 95–108 Mechatronics Jeongmin Cho and Kunsoo Huh, “Torque Vectoring System Design for Hybrid Electric–All Wheel Drive Vehicle,” 234, no. 10–11 (April 7, 2020): 2680–92 Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering Jae-Young Park, Seung-Jin Heo, and Dae-Oh Kang, “Development of Torque Vectoring Control Algorithm for Front Wheel Driven Dual Motor System and Evaluation of Vehicle Dynamics Performance,” 21, no. 5 (October 2020): 1283–91 International Journal of Automotive Technology Huifan Deng et al., “Deep Reinforcement Learning-Based Torque Vectoring Control Considering Economy and Safety,” 11, no. 4 (April 6, 2023): 459 Machines Giseo Park et al., “Torque Vectoring Algorithm of Electronic-Four-Wheel Drive Vehicles for Enhancement of Cornering Performance,” 69, no. 4 (April 2020): 3668–79 IEEE Transactions on Vehicular Technology

Driving Innovation: The Science of Torque Vectoring Systems

The New Private Cloud From the Eyes of an Architect

একজন স্থপতির চোখ থেকে নতুন ব্যক্তিগত মেঘ

Die neue Private Cloud aus der Sicht eines Architekten

La nueva nube privada desde la perspectiva de un arquitecto

Le nouveau cloud privé vu par un architecte

एक आर्किटेक्ट की नज़र से नया प्राइवेट क्लाउड

建築家の視点から見た新しいプライベートクラウド

건축가의 눈으로 본 새로운 프라이빗 클라우드

A nova nuvem privada vista por um arquiteto

Новое частное облако глазами архитектора

Bir Mimarın Gözünden Yeni Özel Bulut

Đám mây riêng mới từ góc nhìn của một kiến trúc sư

从建筑师的角度看新私有云

Code and the Art of Motorcycle Maintenance: Typical Rookie Faults

কোড এবং মোটরসাইকেল রক্ষণাবেক্ষণের শিল্প: সাধারণ রুকি ফল্টস

Code und die Kunst der Motorradwartung: Typische Anfängerfehler

Código y el arte del mantenimiento de motocicletas: errores típicos de los novatos

Code et art de l'entretien de la moto : les défauts typiques des débutants

कोड और मोटरसाइकिल रखरखाव की कला: आम तौर पर नौसिखियों की गलतियाँ

コードとオートバイ整備の技術: 初心者が陥りがちなミス

코드와 오토바이 유지 관리의 기술: 전형적인 초보자 오류

Código e a arte da manutenção de motocicletas: falhas típicas de novatos

Кодекс и искусство обслуживания мотоцикла: типичные ошибки новичков

Motosiklet Bakımının Kodu ve Sanatı: Tipik Acemi Hataları

Quy tắc và Nghệ thuật Bảo dưỡng Xe máy: Những Lỗi Thường gặp của Người mới bắt đầu

摩托车维修规范与艺术：新手常犯的典型错误

Authors:
(1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy;
(5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA;
(6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction II Methods III Results IV Discussion V. Conclusion, Acknowledgment, and References V. CONCLUSION In this study, we demonstrated the feasibility of employing deep learning in gait state detection for exoskeleton control, eliminating the necessity of ground reaction force sensors. Despite challenges such as a limited dataset, realtime constraints, and issues related to error propagation, these obstacles were successfully addressed. Evaluation of our model’s closed-loop performance versus typical sensors (i.e., force plates, FSR footplates) highlights the system’s applicability across new users and several walking speeds and conditions. To address our model limitations, future work could include training datasets encompassing various walking speeds, overground walking and other activities (e.g. ramps, stairs). Furthermore, this work could be used to implement a state machine that utilizes an estimated stance interpolation factor, selecting impedance parameters in the mid-level controller for exoskeleton control and tailored assistance during walking. The proposed framework will be validated in future on additional healthy individuals as well as individuals with lower-limb impairments (e.g., stroke, spinal cord injury). ACKNOWLEDGMENT This work was supported by the National Science Foundation / National Robotics Initiative (Grant No: 2024488). We would like to thank Tim Haswell for his technical support on the hardware improvements of the ExoMotus-X2 exoskeleton. REFERENCES [1] R. Baud, A. R. Manzoori, A. Ijspeert, and M. Bouri, “Review of control strategies for lower-limb exoskeletons to assist gait,” Journal of NeuroEngineering and Rehabilitation, vol. 18, no. 1, Jul. 2021. [Online]. Available: //doi.org/10.1186/s12984-021-00906-3 [2] J. de Miguel-Fernandez, J. Lobo-Prat, E. Prinsen, J. M. Font-Llagunes, and L. Marchal-Crespo, “Control strategies used in lower limb exoskeletons for gait rehabilitation after brain injury: a systematic review and analysis of clinical effectiveness,” Journal of NeuroEngineering and Rehabilitation, vol. 20, no. 1, Feb. 2023. [3] E. Basalp, P. Wolf, and L. Marchal-Crespo, “Haptic training: Which types facilitate (re)learning of which motor task and for whom? answers by a review,” IEEE Transactions on Haptics, vol. 14, no. 4, pp. 722–739, 2021. [4] M. Kim, A. M. Simon, and L. J. Hargrove, “Seamless and intuitive control of a powered prosthetic leg using deep neural network for transfemoral amputees,” Wearable Technologies, vol. 3, 2022. [5] W. Huo, S. Mohammed, Y. Amirat, and K. Kong, “Fast gait mode detection and assistive torque control of an exoskeletal robotic orthosis for walking assistance,” IEEE Transactions on Robotics, vol. 34, no. 4, pp. 1035–1052, 2018. [6] A. Ortlieb, R. Baud, T. Tracchia, B. Denkinger, Q. Herzig, H. Bleuler, and M. Bouri, “An active impedance controller to assist gait in people with neuromuscular diseases: Implementation to the hip joint of the autonomyo exoskeleton,” in 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), 2018. [7] E. B. Kuc¸¨ uktabak, Y. Wen, S. J. Kim, M. R. Short, D. Ludvig, L. Hargrove, E. J. Perreault, K. M. Lynch, and J. L. Pons, “Haptic transparency and interaction force control for a lower-limb exoskeleton,” IEEE Transactions on Robotics, pp. 1–19, 2024. [8] M. Mundt, A. Koeppe, S. David, F. Bamer, W. Potthast, and B. Markert, “Prediction of ground reaction force and joint moments based on optical motion capture data during gait,” Medical Engineering & Physics, vol. 86, pp. 29–34, 2020. [9] A. Karatsidis, G. Bellusci, H. Schepers, M. de Zee, M. Andersen, and P. Veltink, “Estimation of ground reaction forces and moments during gait using only inertial motion capture,” Sensors, vol. 17, no. 12, p. 75, Dec. 2016. [Online]. Available: //doi.org/10.3390/s17010075 [10] L. Mourot, L. Hoyet, F. L. Clerc, and P. Hellier, “Underpressure: Deep learning for foot contact detection, ground reaction force estimation and footskate cleanup,” Computer Graphics Forum, vol. 41, no. 8, pp. 195–206, 2022. [11] J.-Y. Jung, W. Heo, H. Yang, and H. Park, “A neural network-based gait phase classification method using sensors equipped on lower limb exoskeleton robots,” Sensors, vol. 15, no. 11, pp. 27 738–27 759, 2015. [12] D.-X. Liu, X. Wu, W. Du, C. Wang, and T. Xu, “Gait phase recognition for lower-limb exoskeleton with only joint angular sensors,” Sensors, vol. 16, no. 10, p. 1579, Sep. 2016. [13] C. Camardella, F. Porcini, A. Filippeschi, S. Marcheschi, M. Solazzi, and A. Frisoli, “Gait phases blended control for enhancing transparency on lower-limb exoskeletons,” IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 5453–5460, 2021. [14] V. Lippi, C. Camardella, A. Filippeschi, and F. Porcini, “Identification of gait phases with neural networks for smooth transparent control of a lower limb exoskeleton,” in Proceedings of the 18th International Conference on Informatics in Control, Automation and Robotics - Volume 1: ICINCO,, INSTICC. SciTePress, 2021, pp. 171–178. [15] J. Fong, E. B. Kuc¸uktabak, V. Crocher, Y. Tan, K. M. Lynch, J. L. Pons, and D. Oetomo, “Canopen robot controller (corc): An open software stack for human robot interaction development,” in Wearable Robotics: Challenges and Trends, J. C. Moreno, J. Masood, U. Schneider, C. Maufroy, and J. L. Pons, Eds. Cham: Springer International Publishing, 2022, pp. 287–292. [16] F. Labarriere, E. Thomas, L. Calistri, V. Optasanu, M. Gueugnon, ` P. Ornetti, and D. Laroche, “Machine learning approaches for activity recognition and/or activity prediction in locomotion assistive devices—a systematic review,” Sensors, vol. 20, no. 21, 2020. [17] M. Kim and L. J. Hargrove, “Deep-learning to map a benchmark dataset of non-amputee ambulation for controlling an open source bionic leg,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10 597–10 604, 2022. [18] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [19] E. B. Kuc¸uktabak, Y. Wen, M. Short, E. Demirbas¸, K. Lynch, and J. Pons, “Virtual physical coupling of two lower-limb exoskeletons,” in 2023 International Conference on Rehabilitation Robotics (ICORR), 2023, pp. 1–6. [20] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643, 2020. [21] R. A. Alamro, A. E. Chisholm, A. M. M. Williams, M. G. Carpenter, and T. Lam, “Overground walking with a robotic exoskeleton elicits trunk muscle activity in people with high-thoracic motor-complete spinal cord injury,” Journal of NeuroEngineering and Rehabilitation, vol. 15, no. 1, Nov. 2018. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Authors: Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction Abstract and I Introduction II Methods II Methods III Results III Results IV Discussion IV Discussion V. Conclusion, Acknowledgment, and References V. Conclusion, Acknowledgment, and References V. CONCLUSION In this study, we demonstrated the feasibility of employing deep learning in gait state detection for exoskeleton control, eliminating the necessity of ground reaction force sensors. Despite challenges such as a limited dataset, realtime constraints, and issues related to error propagation, these obstacles were successfully addressed. Evaluation of our model’s closed-loop performance versus typical sensors (i.e., force plates, FSR footplates) highlights the system’s applicability across new users and several walking speeds and conditions. To address our model limitations, future work could include training datasets encompassing various walking speeds, overground walking and other activities (e.g. ramps, stairs). Furthermore, this work could be used to implement a state machine that utilizes an estimated stance interpolation factor, selecting impedance parameters in the mid-level controller for exoskeleton control and tailored assistance during walking. The proposed framework will be validated in future on additional healthy individuals as well as individuals with lower-limb impairments (e.g., stroke, spinal cord injury). ACKNOWLEDGMENT This work was supported by the National Science Foundation / National Robotics Initiative (Grant No: 2024488). We would like to thank Tim Haswell for his technical support on the hardware improvements of the ExoMotus-X2 exoskeleton. REFERENCES [1] R. Baud, A. R. Manzoori, A. Ijspeert, and M. Bouri, “Review of control strategies for lower-limb exoskeletons to assist gait,” Journal of NeuroEngineering and Rehabilitation, vol. 18, no. 1, Jul. 2021. [Online]. Available: //doi.org/10.1186/s12984-021-00906-3 [2] J. de Miguel-Fernandez, J. Lobo-Prat, E. Prinsen, J. M. Font-Llagunes, and L. Marchal-Crespo, “Control strategies used in lower limb exoskeletons for gait rehabilitation after brain injury: a systematic review and analysis of clinical effectiveness,” Journal of NeuroEngineering and Rehabilitation, vol. 20, no. 1, Feb. 2023. [3] E. Basalp, P. Wolf, and L. Marchal-Crespo, “Haptic training: Which types facilitate (re)learning of which motor task and for whom? answers by a review,” IEEE Transactions on Haptics, vol. 14, no. 4, pp. 722–739, 2021. [4] M. Kim, A. M. Simon, and L. J. Hargrove, “Seamless and intuitive control of a powered prosthetic leg using deep neural network for transfemoral amputees,” Wearable Technologies, vol. 3, 2022. [5] W. Huo, S. Mohammed, Y. Amirat, and K. Kong, “Fast gait mode detection and assistive torque control of an exoskeletal robotic orthosis for walking assistance,” IEEE Transactions on Robotics, vol. 34, no. 4, pp. 1035–1052, 2018. [6] A. Ortlieb, R. Baud, T. Tracchia, B. Denkinger, Q. Herzig, H. Bleuler, and M. Bouri, “An active impedance controller to assist gait in people with neuromuscular diseases: Implementation to the hip joint of the autonomyo exoskeleton,” in 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), 2018. [7] E. B. Kuc¸¨ uktabak, Y. Wen, S. J. Kim, M. R. Short, D. Ludvig, L. Hargrove, E. J. Perreault, K. M. Lynch, and J. L. Pons, “Haptic transparency and interaction force control for a lower-limb exoskeleton,” IEEE Transactions on Robotics, pp. 1–19, 2024. [8] M. Mundt, A. Koeppe, S. David, F. Bamer, W. Potthast, and B. Markert, “Prediction of ground reaction force and joint moments based on optical motion capture data during gait,” Medical Engineering & Physics, vol. 86, pp. 29–34, 2020. [9] A. Karatsidis, G. Bellusci, H. Schepers, M. de Zee, M. Andersen, and P. Veltink, “Estimation of ground reaction forces and moments during gait using only inertial motion capture,” Sensors, vol. 17, no. 12, p. 75, Dec. 2016. [Online]. Available: //doi.org/10.3390/s17010075 [10] L. Mourot, L. Hoyet, F. L. Clerc, and P. Hellier, “Underpressure: Deep learning for foot contact detection, ground reaction force estimation and footskate cleanup,” Computer Graphics Forum, vol. 41, no. 8, pp. 195–206, 2022. [11] J.-Y. Jung, W. Heo, H. Yang, and H. Park, “A neural network-based gait phase classification method using sensors equipped on lower limb exoskeleton robots,” Sensors, vol. 15, no. 11, pp. 27 738–27 759, 2015. [12] D.-X. Liu, X. Wu, W. Du, C. Wang, and T. Xu, “Gait phase recognition for lower-limb exoskeleton with only joint angular sensors,” Sensors, vol. 16, no. 10, p. 1579, Sep. 2016. [13] C. Camardella, F. Porcini, A. Filippeschi, S. Marcheschi, M. Solazzi, and A. Frisoli, “Gait phases blended control for enhancing transparency on lower-limb exoskeletons,” IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 5453–5460, 2021. [14] V. Lippi, C. Camardella, A. Filippeschi, and F. Porcini, “Identification of gait phases with neural networks for smooth transparent control of a lower limb exoskeleton,” in Proceedings of the 18th International Conference on Informatics in Control, Automation and Robotics - Volume 1: ICINCO,, INSTICC. SciTePress, 2021, pp. 171–178. [15] J. Fong, E. B. Kuc¸uktabak, V. Crocher, Y. Tan, K. M. Lynch, J. L. Pons, and D. Oetomo, “Canopen robot controller (corc): An open software stack for human robot interaction development,” in Wearable Robotics: Challenges and Trends, J. C. Moreno, J. Masood, U. Schneider, C. Maufroy, and J. L. Pons, Eds. Cham: Springer International Publishing, 2022, pp. 287–292. [16] F. Labarriere, E. Thomas, L. Calistri, V. Optasanu, M. Gueugnon, ` P. Ornetti, and D. Laroche, “Machine learning approaches for activity recognition and/or activity prediction in locomotion assistive devices—a systematic review,” Sensors, vol. 20, no. 21, 2020. [17] M. Kim and L. J. Hargrove, “Deep-learning to map a benchmark dataset of non-amputee ambulation for controlling an open source bionic leg,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10 597–10 604, 2022. [18] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [19] E. B. Kuc¸uktabak, Y. Wen, M. Short, E. Demirbas¸, K. Lynch, and J. Pons, “Virtual physical coupling of two lower-limb exoskeletons,” in 2023 International Conference on Rehabilitation Robotics (ICORR), 2023, pp. 1–6. [20] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643, 2020. [21] R. A. Alamro, A. E. Chisholm, A. M. M. Williams, M. G. Carpenter, and T. Lam, “Overground walking with a robotic exoskeleton elicits trunk muscle activity in people with high-thoracic motor-complete spinal cord injury,” Journal of NeuroEngineering and Rehabilitation, vol. 15, no. 1, Nov. 2018. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Weight Distribution Estimation in Lower-Limb Exoskeletons via Deep Learning: Conclusion

Load Balancing For High Performance Computing
Using Quantum Annealing: Grid Based Application

Authors:
(1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy;
(5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA;
(6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction II Methods III Results IV Discussion V. Conclusion, Acknowledgment, and References IV. DISCUSSION In this paper, we evaluated a deep-learning model to predict the weight distribution (i.e., stance interpolation factor) of a user wearing a lower-limb exoskeleton during several walking conditions; we compared the exoskeleton controller performance when this stance interpolation factor was estimated with our deep-learning model (αˆ ) and measured with ground truth values (treadmill force plates or FSR sensor pads, α). This work highlights the viability and limitations of using deep-learning predictions to detect changes in the gait state using only joint kinematics information without ground reaction sensors. Utilizing a history of kinematic data was shown to enhance the model’s accuracy (Fig. 3 and Tab. I) in comparison to using instantaneous values, as seen in a similar study [14]. These findings show the advantage of utilizing additional temporal data from previous time steps, enhancing the network’s performance in both training and testing phases. Importantly, including additional data did not significantly alter the model’s prediction speed; we observed comparable prediction times when using a history of kinematic data with respect to the instantaneous value predictions. As a result, the model was implemented to achieve real-time performance. This is crucial for its use in the control of an exoskeleton, as predictions need to be faster than the main control loop. Achieving an average prediction time of 0.57 ms ensured the real-time usability of the system, as the main control loop runs usually significantly slower than 1 ms on lower-limb exoskeletons (e.g., 3 ms for the exoskeleton controller in our study). It is noteworthy that, without using TensorFlow Lite (i.e., employing the classic TensorFlow library), the average prediction time with a history of kinematic data is 67.1 ± 13.5 ms, which cannot be used in most real-time applications. Furthermore, the proposed method provides accurate predictions of the stance interpolation factor for previous and unseen users (Fig. 3 and Tab. I), demonstrating another aspect of its usability. Requiring training data for every exoskeleton user is not convenient or realistic, particularly in physical rehabilitation as a patient’s time spent receiving therapy must be prioritized. In the context of machine learning for robotic control, using the predictions in a closed loop can affect the performance due to error propagation in real time [20]. Therefore, it was critical to evaluate the closed-loop performance of our system, using several conditions. We demonstrated that the deep-learning approach produces a similar stance interpolation factor compared to treadmill force plates during walking with haptic transparent control; these findings generalize for walking speeds between 0.14 and 0.47 m/s (Fig. 4). To characterize the effect of the deep-learning predictions on the performance of the exoskeleton controller, we assessed both haptic rendering of nonzero impedance and haptic transparent modes during treadmill walking. Moreover, we evaluated the extent to which these results generalize to overground walking. During treadmill walking, mean interaction torque errors were similar between deep-learning predictions and treadmill force plate measurements. Evaluating the performance of the controller during more naturalistic, overground walking, the deep-learning predictions also facilitated similar interaction torque error profiles (Fig. 7A) and hip and knee kinematics (Fig. 7B). While the overall performance of the exoskeleton controller was comparable when implementing our deeplearning predictions and ground reaction sensor values, we did observe a delay-like effect in the stance interpolation factor estimation which likely resulted in some performance discrepancies at specific phases of the gait cycle. Specifically, we observed a longer stance duration when walking with the deep-learning predictions at slower speeds (Fig. 4); this was associated with a longer transition between left and right single stance (i.e., double stance to swing). During treadmill and overground walking, at the knee joint, we observed higher interaction torque error at the beginning of the swing phase (around 60-70% of the cycle, Fig. 6A) with the deep-learning predictions. This increase in interaction torque error was more noticeable during overground walking, and also observed at the hip joint. In this condition, the use of crutches may have promoted additional lateral movement which was not measured by the exoskeleton [21]. For both treadmill and overground walking, the discrepancies in interaction torque errors relative to their respective ground truths could be due to the deep-learning model requiring users to change their kinematics before the stance interpolation factor changes. This is different from using force sensors, which can detect changes in weight distribution independent of kinematics recorded by the exoskeleton. One way to improve the detection of changes in weight distribution in the deep-learning model is to incorporate the frontal plane IMU angle of the backpack in addition to the sagittal plane angles from the backpack and joint encoders. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Authors: Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction Abstract and I Introduction II Methods II Methods III Results III Results IV Discussion IV Discussion V. Conclusion, Acknowledgment, and References V. Conclusion, Acknowledgment, and References IV. DISCUSSION In this paper, we evaluated a deep-learning model to predict the weight distribution (i.e., stance interpolation factor) of a user wearing a lower-limb exoskeleton during several walking conditions; we compared the exoskeleton controller performance when this stance interpolation factor was estimated with our deep-learning model (αˆ ) and measured with ground truth values (treadmill force plates or FSR sensor pads, α). This work highlights the viability and limitations of using deep-learning predictions to detect changes in the gait state using only joint kinematics information without ground reaction sensors. Utilizing a history of kinematic data was shown to enhance the model’s accuracy (Fig. 3 and Tab. I) in comparison to using instantaneous values, as seen in a similar study [14]. These findings show the advantage of utilizing additional temporal data from previous time steps, enhancing the network’s performance in both training and testing phases. Importantly, including additional data did not significantly alter the model’s prediction speed; we observed comparable prediction times when using a history of kinematic data with respect to the instantaneous value predictions. As a result, the model was implemented to achieve real-time performance. This is crucial for its use in the control of an exoskeleton, as predictions need to be faster than the main control loop. Achieving an average prediction time of 0.57 ms ensured the real-time usability of the system, as the main control loop runs usually significantly slower than 1 ms on lower-limb exoskeletons (e.g., 3 ms for the exoskeleton controller in our study). It is noteworthy that, without using TensorFlow Lite (i.e., employing the classic TensorFlow library), the average prediction time with a history of kinematic data is 67.1 ± 13.5 ms, which cannot be used in most real-time applications. Furthermore, the proposed method provides accurate predictions of the stance interpolation factor for previous and unseen users (Fig. 3 and Tab. I), demonstrating another aspect of its usability. Requiring training data for every exoskeleton user is not convenient or realistic, particularly in physical rehabilitation as a patient’s time spent receiving therapy must be prioritized. In the context of machine learning for robotic control, using the predictions in a closed loop can affect the performance due to error propagation in real time [20]. Therefore, it was critical to evaluate the closed-loop performance of our system, using several conditions. We demonstrated that the deep-learning approach produces a similar stance interpolation factor compared to treadmill force plates during walking with haptic transparent control; these findings generalize for walking speeds between 0.14 and 0.47 m/s (Fig. 4). To characterize the effect of the deep-learning predictions on the performance of the exoskeleton controller, we assessed both haptic rendering of nonzero impedance and haptic transparent modes during treadmill walking. Moreover, we evaluated the extent to which these results generalize to overground walking. During treadmill walking, mean interaction torque errors were similar between deep-learning predictions and treadmill force plate measurements. Evaluating the performance of the controller during more naturalistic, overground walking, the deep-learning predictions also facilitated similar interaction torque error profiles (Fig. 7A) and hip and knee kinematics (Fig. 7B). While the overall performance of the exoskeleton controller was comparable when implementing our deeplearning predictions and ground reaction sensor values, we did observe a delay-like effect in the stance interpolation factor estimation which likely resulted in some performance discrepancies at specific phases of the gait cycle. Specifically, we observed a longer stance duration when walking with the deep-learning predictions at slower speeds (Fig. 4); this was associated with a longer transition between left and right single stance (i.e., double stance to swing). During treadmill and overground walking, at the knee joint, we observed higher interaction torque error at the beginning of the swing phase (around 60-70% of the cycle, Fig. 6A) with the deep-learning predictions. This increase in interaction torque error was more noticeable during overground walking, and also observed at the hip joint. In this condition, the use of crutches may have promoted additional lateral movement which was not measured by the exoskeleton [21]. For both treadmill and overground walking, the discrepancies in interaction torque errors relative to their respective ground truths could be due to the deep-learning model requiring users to change their kinematics before the stance interpolation factor changes. This is different from using force sensors, which can detect changes in weight distribution independent of kinematics recorded by the exoskeleton. One way to improve the detection of changes in weight distribution in the deep-learning model is to incorporate the frontal plane IMU angle of the backpack in addition to the sagittal plane angles from the backpack and joint encoders. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Weight Distribution Estimation in Lower-Limb Exoskeletons via Deep Learning: Discussion

Authors:
(1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy;
(5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA;
(6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction II Methods III Results IV Discussion V. Conclusion, Acknowledgment, and References III. RESULTS A. Validation of Deep-Learning Predictions The proposed LSTM model was evaluated using a leaveone-out cross-validation approach: in each iteration, the model was trained on five subjects and then tested on the remaining subject. In this test, we compared a model using instantaneous kinematic data (TW = 0 ms) and another using a history of kinematic data (TW = 300 ms). Fig. 3 shows the prediction for each model evaluated on the test set composed by Subject Four and the ground truth value. The model using instantaneous kinematic data resulted in a prediction time of 201±70.7 µs on a laptop (ThinkPad X1 Carbon 5th, Lenovo), while utilizing the history of kinematic data resulted in a prediction time of 573±308 µs. In terms of the prediction accuracy, the model incorporating a history of kinematic data produced a higher accuracy (TW = 0 ms: R2 = 0.84 ± 0.03, MSE = 1.8 × 10−2 ± 0.4; TW = 300 ms: R 2 = 0.90±0.02, MSE = 1.1×10−2 ±0.3). Tab. I provides a detailed report of the model’s accuracy for individual users. Specifically, it includes R 2 performances for both the training and test sets for TW = 0 ms and TW = 300 ms conditions. B. Closed-Loop Performance During Treadmill Walking After validation of the deep-learning predictions, we compared the closed-loop performance of the proposed method for αˆ estimation and the measured ground truth α from the treadmill force plates. Specifically, estimated or measured stance interpolation factor values were used as inputs to the WECC controller [7] and resulting kinematics and interaction torque errors were evaluated during walking. During the closed-loop tests, we focused on the model with TW = 300 ms due to its improved accuracy compared to TW = 0 ms, while still obtaining a prediction time suitable for real-time implementation (0.57 ms). In Fig. 4, we illustrate the stance interpolation factor with respect to normalized gait duration under different conditions. The stance interpolation factor values obtained from treadmill force plates and the deep-learning approach are displayed for three different speeds (0.14 m/s, 0.25 m/s, 0.47 m/s). Compared to using force plate values, the proposed method resulted in longer stance time during slower walking and shorter stance time during faster walking. At 0.14 m/s, stance duration was 56.0 ± 4.8% of the gait cycle using the treadmill force plates versus 67.1 ± 9.7% using deep learning. At 0.25 m/s, stance duration was similar between the two approaches: 59.6±6.99% of the gait cycle using the treadmill force plates, and 65.1±1.6% using deep learning. Finally, at 0.47 m/s, the treadmill force plates resulted in a stance duration of 69.3±10.2% versus 59.7±5.74% using deep learning. Interaction torque errors were investigated to evaluate the quality of the haptic rendering and transparency, using the force plates and the deep-learning predictions for the stance interpolation factor in real time. Fig. 5 presents the interaction torque tracking performance during the rendering mode while using the force plate measurements and deep-learning estimation in the dynamic robot model. It was observed that interaction torque error is higher with the deep-learning method at the start of the swing. Specifically, toe-off (60-70% of the gait cycle) led to an average hip interaction torque error of 0.068 Nm/kg using force plates and 0.109 Nm/kg using deep learning. However, this error was lower for the deep-learning approach in the stance phase. Within the 20- 50% range of the gait cycle, the average hip interaction error was 0.111 Nm/kg using force plates and 0.033 Nm/kg using deep learning. The mean of the interaction torque error over the whole cycle, across three users, was comparable between the two conditions. At the hip joint, mean absolute interaction torque errors of 0.101 ± 0.004 Nm/kg and 0.094 ± 0.005 Nm/kg were observed with the deep-learning and force plate conditions, respectively. Similarly, the mean absolute interaction torque error at the knee joint was 0.119±0.006 Nm/kg and 0.113±0.006 Nm/kg for the deep-learning and force plate conditions. Fig. 6 shows the results of the transparency performances on both conditions. The deep-learning approach resulted in a relatively smaller hip interaction torque error compared to the force-plate condition (0.063±0.006 Nm/kg vs. 0.073± 0.008 Nm/kg). On the other hand, the deep-learning approach resulted in a higher interaction torque error at the knee joints between 60% and 70% of the normalized gait, corresponding to the beginning of the swing phase. The mean knee interaction torque error in this period was −0.207 Nm/kg for the force plate condition and −0.312 Nm/kg for the deep-learning approach. Users also qualitatively reported that the exoskeleton felt heavier at this particular moment of the gait cycle. Across the whole gait cycle, the mean interaction torque error for the knee joint was similar for the deep-learning and the force plate conditions (0.078±0.007 Nm/kg vs. 0.071±0.007 Nm/kg). C. Closed-Loop Performance During Overground Walking During the overground tests, it was observed that the users preferred to walk at a slightly faster speed using the FSR footplates (0.241 ± 0.024 m/s) compared to the deep-learning estimation (0.198 ± 0.020 m/s). Results from Fig. 7B show similar kinematic patterns between the two conditions, particularly in the magnitudes of the joint angles. In the temporal domain, a longer stance period was observed with the deep-learning approach (75.3 ± 2.7%) compared to the FSR condition (63.1 ± 1.6%). Furthermore, Fig. 7 presents the interaction torque error over a gait cycle, for a representative user, using FSRs or deep learning. The interaction torque error, averaged over two users, was higher for the deep-learning condition (Hip: 0.094 ± 0.009 Nm/kg using FSR and 0.120 ± 0.014 Nm/kg using deep learning; Knee: 0.083 ± 0.007 Nm/kg using FSR and 0.116 ± 0.009 Nm/kg using deep learning). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Authors: Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction Abstract and I Introduction II Methods II Methods III Results III Results IV Discussion IV Discussion V. Conclusion, Acknowledgment, and References V. Conclusion, Acknowledgment, and References III. RESULTS A. Validation of Deep-Learning Predictions A. Validation of Deep-Learning Predictions The proposed LSTM model was evaluated using a leaveone-out cross-validation approach: in each iteration, the model was trained on five subjects and then tested on the remaining subject. In this test, we compared a model using instantaneous kinematic data (TW = 0 ms) and another using a history of kinematic data (TW = 300 ms). Fig. 3 shows the prediction for each model evaluated on the test set composed by Subject Four and the ground truth value. The model using instantaneous kinematic data resulted in a prediction time of 201±70.7 µs on a laptop (ThinkPad X1 Carbon 5th, Lenovo), while utilizing the history of kinematic data resulted in a prediction time of 573±308 µs. In terms of the prediction accuracy, the model incorporating a history of kinematic data produced a higher accuracy (TW = 0 ms: R2 = 0.84 ± 0.03, MSE = 1.8 × 10−2 ± 0.4; TW = 300 ms: R 2 = 0.90±0.02, MSE = 1.1×10−2 ±0.3). Tab. I provides a detailed report of the model’s accuracy for individual users. Specifically, it includes R 2 performances for both the training and test sets for TW = 0 ms and TW = 300 ms conditions. B. Closed-Loop Performance During Treadmill Walking B. Closed-Loop Performance During Treadmill Walking After validation of the deep-learning predictions, we compared the closed-loop performance of the proposed method for αˆ estimation and the measured ground truth α from the treadmill force plates. Specifically, estimated or measured stance interpolation factor values were used as inputs to the WECC controller [7] and resulting kinematics and interaction torque errors were evaluated during walking. During the closed-loop tests, we focused on the model with TW = 300 ms due to its improved accuracy compared to TW = 0 ms, while still obtaining a prediction time suitable for real-time implementation (0.57 ms). In Fig. 4, we illustrate the stance interpolation factor with respect to normalized gait duration under different conditions. The stance interpolation factor values obtained from treadmill force plates and the deep-learning approach are displayed for three different speeds (0.14 m/s, 0.25 m/s, 0.47 m/s). Compared to using force plate values, the proposed method resulted in longer stance time during slower walking and shorter stance time during faster walking. At 0.14 m/s, stance duration was 56.0 ± 4.8% of the gait cycle using the treadmill force plates versus 67.1 ± 9.7% using deep learning. At 0.25 m/s, stance duration was similar between the two approaches: 59.6±6.99% of the gait cycle using the treadmill force plates, and 65.1±1.6% using deep learning. Finally, at 0.47 m/s, the treadmill force plates resulted in a stance duration of 69.3±10.2% versus 59.7±5.74% using deep learning. Interaction torque errors were investigated to evaluate the quality of the haptic rendering and transparency, using the force plates and the deep-learning predictions for the stance interpolation factor in real time. Fig. 5 presents the interaction torque tracking performance during the rendering mode while using the force plate measurements and deep-learning estimation in the dynamic robot model. It was observed that interaction torque error is higher with the deep-learning method at the start of the swing. Specifically, toe-off (60-70% of the gait cycle) led to an average hip interaction torque error of 0.068 Nm/kg using force plates and 0.109 Nm/kg using deep learning. However, this error was lower for the deep-learning approach in the stance phase. Within the 20- 50% range of the gait cycle, the average hip interaction error was 0.111 Nm/kg using force plates and 0.033 Nm/kg using deep learning. The mean of the interaction torque error over the whole cycle, across three users, was comparable between the two conditions. At the hip joint, mean absolute interaction torque errors of 0.101 ± 0.004 Nm/kg and 0.094 ± 0.005 Nm/kg were observed with the deep-learning and force plate conditions, respectively. Similarly, the mean absolute interaction torque error at the knee joint was 0.119±0.006 Nm/kg and 0.113±0.006 Nm/kg for the deep-learning and force plate conditions. Fig. 6 shows the results of the transparency performances on both conditions. The deep-learning approach resulted in a relatively smaller hip interaction torque error compared to the force-plate condition (0.063±0.006 Nm/kg vs. 0.073± 0.008 Nm/kg). On the other hand, the deep-learning approach resulted in a higher interaction torque error at the knee joints between 60% and 70% of the normalized gait, corresponding to the beginning of the swing phase. The mean knee interaction torque error in this period was −0.207 Nm/kg for the force plate condition and −0.312 Nm/kg for the deep-learning approach. Users also qualitatively reported that the exoskeleton felt heavier at this particular moment of the gait cycle. Across the whole gait cycle, the mean interaction torque error for the knee joint was similar for the deep-learning and the force plate conditions (0.078±0.007 Nm/kg vs. 0.071±0.007 Nm/kg). C. Closed-Loop Performance During Overground Walking C. Closed-Loop Performance During Overground Walking During the overground tests, it was observed that the users preferred to walk at a slightly faster speed using the FSR footplates (0.241 ± 0.024 m/s) compared to the deep-learning estimation (0.198 ± 0.020 m/s). Results from Fig. 7B show similar kinematic patterns between the two conditions, particularly in the magnitudes of the joint angles. In the temporal domain, a longer stance period was observed with the deep-learning approach (75.3 ± 2.7%) compared to the FSR condition (63.1 ± 1.6%). Furthermore, Fig. 7 presents the interaction torque error over a gait cycle, for a representative user, using FSRs or deep learning. The interaction torque error, averaged over two users, was higher for the deep-learning condition (Hip: 0.094 ± 0.009 Nm/kg using FSR and 0.120 ± 0.014 Nm/kg using deep learning; Knee: 0.083 ± 0.007 Nm/kg using FSR and 0.116 ± 0.009 Nm/kg using deep learning). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Weight Distribution Estimation in Lower-Limb Exoskeletons via Deep Learning: Results

Authors:
(1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy;
(5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA;
(6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction II Methods III Results IV Discussion V. Conclusion, Acknowledgment, and References II. METHODS A. Exoskeleton We validated our proposed method on a commercially available lower-limb exoskeleton (ExoMotus-X2, Fourier Intelligence, Singapore). The exoskeleton system has four active degrees of freedom (DoF) at the hip and knee, and two passive DoFs at the ankle (Fig. 1B). The exoskeleton was modified to include strain gauges to estimate joint torques, and an Inertial Measurement Unit (IMU) on the backpack to measure the orientation of the trunk [7]. In this study, to measure GRFs, and therefore calculate weight distribution, two distinct devices were employed: a sensorized, split-belt treadmill and force-sensitive resistor (FSR) footplates. The treadmill is equipped with eight 3-DoF force plates (9047B, Kistler), offering highly accurate readings and a simple calibration process to zero the measured forces. However, its use restricts the practical applications of the exoskeleton and prevents its integration into overground scenarios. To address this limitation, FSR footplates were installed beneath the soles of the exoskeleton, facilitating the measurement of forces between the user-exoskeleton couple and the ground. Each footplate is equipped with 16 force-sensitive resistors, amounting to a total of 32 sensors across both footplates. These sensors interface with the ground through a rigid aluminum sole and rubber bearings, adding a degree of compliance. Although these sensors are more complex to calibrate and set up in comparison to the treadmill, they allow the exoskeleton to be used during overground walking. GRFs enable the calculation of weight distribution in the user-exoskeleton couple. This allows an accurate and smooth transition between the left and right stance dynamic models of the exoskeleton (Fig. 1A). In this work, we calculated the weight distribution as the ratio of the vertical GRF of each foot similar to [7], Communication between motors and sensors is achieved using the CANOpen protocol. The controller is implemented on a ROS and C++ based open-source platform called the CANOpen Robot Controller (CORC) [15]. B. Weight Distribution via Deep-Learning Regression The model was trained using the ADAM optimizer [18]. We utilized the MSE loss function to measure and minimize the discrepancy between the predicted (αˆ) and actual (α) values obtained from treadmill measurements. The training process was conducted over 10 epochs, processing the data in batches of 256 samples, to allow efficient weight adjustments while ensuring comprehensive coverage of the data. The LSTM model was trained using the TensorFlow library (v2.15.0, Google); we employed TensorFlow Lite (v.2.13.0, Google) to make predictions in real time. The converted lite model is called by a Python script that runs in parallel to CORC in C++. The communication between these two components is achieved via ROS. After the offline training and validation of prediction capability, we evaluated the capability of the LSTM network for closed-loop predictions of the stance interpolation factor αˆ in two experiments. In the first experiment, three users wore the exoskeleton and walked for one minute on a sensorized treadmill for each mode (i.e., haptic transparency and rendering). The stance interpolation factor for the exoskeleton controller was calculated either using the treadmill force plates as ground truth (α), or estimated using the LSTM model (αˆ ). The measured or estimated value was used as an input to the WECC controller for interaction torque estimation and closed-loop compensation control in real-time [7]. The performance of the proposed model was assessed by comparing the spatiotemporal gait features and resultant interaction torque errors. Reported and visualized interaction torque errors in the plots are calculated using the ground truth α obtained from the treadmill force plates, and divided by each user’s weight. We report all performance values as the mean ± standard deviation across users and steps unless otherwise specified. To assess the network’s ability to generalize to unobserved walking conditions and different controllers, we evaluated the model on three speeds (0.14 m/s, 0.25 m/s, and 0.47 m/s) and two exoskeleton control modes (haptic transparency and rendering). Moreover, two of the three test users were not in the training set. Inter-subject variability is removed from the presented box plots by demeaning the data of each condition/subject pair and subsequently adding the overall mean of each condition, including three subjects. In the second experiment, two users who were not featured in the training set performed trials of overground walking. The deep-learning approach was trained on a dataset of healthy individuals walking on a treadmill, thus we aim to demonstrate the model’s ability to generalize to more naturalistic walking behaviors. In this experiment, α was calculated using the FSR footplates, while αˆ was calculated using the deep-learning approach. The users walked for 10 meters at a self-selected speed, repeated three times for each condition; interaction torques and resultant joint angles were used to assess the performance. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Authors: Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction Abstract and I Introduction II Methods II Methods III Results III Results IV Discussion IV Discussion V. Conclusion, Acknowledgment, and References V. Conclusion, Acknowledgment, and References II. METHODS A. Exoskeleton A. Exoskeleton We validated our proposed method on a commercially available lower-limb exoskeleton (ExoMotus-X2, Fourier Intelligence, Singapore). The exoskeleton system has four active degrees of freedom (DoF) at the hip and knee, and two passive DoFs at the ankle (Fig. 1B). The exoskeleton was modified to include strain gauges to estimate joint torques, and an Inertial Measurement Unit (IMU) on the backpack to measure the orientation of the trunk [7]. In this study, to measure GRFs, and therefore calculate weight distribution, two distinct devices were employed: a sensorized, split-belt treadmill and force-sensitive resistor (FSR) footplates. The treadmill is equipped with eight 3-DoF force plates (9047B, Kistler), offering highly accurate readings and a simple calibration process to zero the measured forces. However, its use restricts the practical applications of the exoskeleton and prevents its integration into overground scenarios. To address this limitation, FSR footplates were installed beneath the soles of the exoskeleton, facilitating the measurement of forces between the user-exoskeleton couple and the ground. Each footplate is equipped with 16 force-sensitive resistors, amounting to a total of 32 sensors across both footplates. These sensors interface with the ground through a rigid aluminum sole and rubber bearings, adding a degree of compliance. Although these sensors are more complex to calibrate and set up in comparison to the treadmill, they allow the exoskeleton to be used during overground walking. GRFs enable the calculation of weight distribution in the user-exoskeleton couple. This allows an accurate and smooth transition between the left and right stance dynamic models of the exoskeleton (Fig. 1A). In this work, we calculated the weight distribution as the ratio of the vertical GRF of each foot similar to [7], Communication between motors and sensors is achieved using the CANOpen protocol. The controller is implemented on a ROS and C++ based open-source platform called the CANOpen Robot Controller (CORC) [15]. B. Weight Distribution via Deep-Learning Regression B. Weight Distribution via Deep-Learning Regression The model was trained using the ADAM optimizer [18]. We utilized the MSE loss function to measure and minimize the discrepancy between the predicted (αˆ) and actual (α) values obtained from treadmill measurements. The training process was conducted over 10 epochs, processing the data in batches of 256 samples, to allow efficient weight adjustments while ensuring comprehensive coverage of the data. The LSTM model was trained using the TensorFlow library (v2.15.0, Google); we employed TensorFlow Lite (v.2.13.0, Google) to make predictions in real time. The converted lite model is called by a Python script that runs in parallel to CORC in C++. The communication between these two components is achieved via ROS. After the offline training and validation of prediction capability, we evaluated the capability of the LSTM network for closed-loop predictions of the stance interpolation factor αˆ in two experiments. In the first experiment, three users wore the exoskeleton and walked for one minute on a sensorized treadmill for each mode (i.e., haptic transparency and rendering). The stance interpolation factor for the exoskeleton controller was calculated either using the treadmill force plates as ground truth (α), or estimated using the LSTM model (αˆ ). The measured or estimated value was used as an input to the WECC controller for interaction torque estimation and closed-loop compensation control in real-time [7]. The performance of the proposed model was assessed by comparing the spatiotemporal gait features and resultant interaction torque errors. Reported and visualized interaction torque errors in the plots are calculated using the ground truth α obtained from the treadmill force plates, and divided by each user’s weight. We report all performance values as the mean ± standard deviation across users and steps unless otherwise specified. To assess the network’s ability to generalize to unobserved walking conditions and different controllers, we evaluated the model on three speeds (0.14 m/s, 0.25 m/s, and 0.47 m/s) and two exoskeleton control modes (haptic transparency and rendering). Moreover, two of the three test users were not in the training set. Inter-subject variability is removed from the presented box plots by demeaning the data of each condition/subject pair and subsequently adding the overall mean of each condition, including three subjects. In the second experiment, two users who were not featured in the training set performed trials of overground walking. The deep-learning approach was trained on a dataset of healthy individuals walking on a treadmill, thus we aim to demonstrate the model’s ability to generalize to more naturalistic walking behaviors. In this experiment, α was calculated using the FSR footplates, while αˆ was calculated using the deep-learning approach. The users walked for 10 meters at a self-selected speed, repeated three times for each condition; interaction torques and resultant joint angles were used to assess the performance. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Weight Distribution Estimation in Lower-Limb Exoskeletons via Deep Learning: Methods

Authors:
(1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA;
(4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy;
(5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA;
(6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA;
(7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction II Methods III Results IV Discussion V. Conclusion, Acknowledgment, and References Abstract In the control of lower-limb exoskeletons with feet, the phase in the gait cycle can be identified by monitoring the weight distribution at the feet. This phase information can be used in the exoskeleton’s controller to compensate the dynamics of the exoskeleton and to assign impedance parameters. Typically the weight distribution is calculated using data from sensors such as treadmill force plates or insole force sensors. However, these solutions increase both the setup complexity and cost. For this reason, we propose a deep-learning approach that uses a short time window of joint kinematics to predict the weight distribution of an exoskeleton in real time. The model was trained on treadmill walking data from six users wearing a four-degree-of-freedom exoskeleton and tested in real time on three different users wearing the same device. This test set includes two users not present in the training set to demonstrate the model’s ability to generalize across individuals. Results show that the proposed method is able to fit the actual weight distribution with R 2 = 0.9 and is suitable for real-time control with prediction times less than 1 ms. Experiments in closed-loop exoskeleton control show that deep-learning-based weight distribution estimation can be used to replace force sensors in overground and treadmill walking. I. INTRODUCTION Lower-limb exoskeletons are increasingly being utilized in everyday tasks and rehabilitation for individuals with gait impairments. In clinical settings, these exoskeletons are typically classified into two main types: partial assistance and full mobilization. Full mobilization exoskeletons are designed for patients with severe motor control disorders, providing autonomous movement of the legs regardless of the patient’s input [1]. Conversely, partial assistance exoskeletons aid in the patient’s movement while still requiring their active input. In this paper, we focus on partial assistance exoskeletons, as they are commonly employed in rehabilitation. Depending on the functional level of the patient, these exoskeletons use techniques like haptic guidance/assistance [2] or error augmentation/resistance [3] to facilitate the (re)learning of walking behaviors. Implementing both haptic assistance and resistance strategies requires precise control over the interaction torques between the user and the exoskeleton, ensuring effective and safe physical interaction. Typically, the control of the interaction between a user and exoskeleton adopts a hierarchy of control levels: high-level, mid-level, and low-level [1], [4]. For lower-limb exoskeletons, the high-level controller calculates the desired interaction torques tailored to specific ambulatory activities, such as walking on flat terrain, stairs or ramps. The role of the mid-level controller is to estimate the various states within an activity, for example, identifying the gait states (e.g., swing and stance) during walking and setting a desired interaction torque behavior accordingly. Finally, the low-level controller is responsible for compensating the exoskeleton’s dynamics and generating motor commands, based on the desired interaction torque profile identified by the mid-level controller. In this three-level approach, an accurate, real-time estimation of the gait state in the mid-level and low-level is essential for properly controlling the exoskeleton. A common approach to gait state detection is to use gait events (e.g., heel strike, toe-off) to identify the switching between discrete gait states: left stance, right stance, and double stance [5], [6], [1]. However, achieving a smooth transition between these states is vital to ensure a control input that limits unexpected behaviors for the user. Kuc¸¨ uktabak et al. [7] proposed a ¨ whole-exoskeleton closed-loop compensation (WECC) controller that uses the ratio of ground reaction forces (GRF) (i.e., weight distribution) to approximate the double stance phase as a transition between the left and right stance. The weight distribution, quantified as the stance interpolation factor α, smoothly changes from zero (left stance) to one (right stance) during double stance, as visualized in Fig. 1. Nonetheless, the requirement of additional sensors for GRF measurements, such as treadmill force plates or insoles, increases the complexity and cost of these systems. Machine learning models have demonstrated their ability to estimate gait states using only joint kinematics of humans walking without exoskeletons, thus eliminating the need for GRF measurements [8], [9], [10]. Motivated by these results, recent studies have proposed real-time estimation of gait states with lower-limb exoskeletons. For instance, Jung et al. [11] and Liu et al. [12] used data from joint encoders and an inertial measurement unit (IMU) positioned on the trunk to classify the gait cycle during walking. These studies divided gait cycles into two (i.e., stance and swing) and eight gait states, respectively. However, discrete control states result in torque or modeling discontinuities. Camardella et al. [13] and Lippi et al. [14] address this issue by implementing a linear regression and a Neural Network (NN) model for regression with joint angles as inputs. The model’s output smoothly transitions between left and right stance, providing a continuous representation of the gait state. Despite this advancement, these approaches present some limitations. First, they only consider the present joint configuration, discarding past information. This approach might lead to inaccuracies in predictions as the weight distribution is influenced not only by the current posture, but also by the velocity and acceleration of the motion. Moreover, few considerations in these studies are given to the generalization across subjects and the prediction time performances. The latter is essential for real-time implementation of the exoskeleton’s whole-body controller. To address the aforementioned limitations, we present a deep-learning approach for estimating the weight distribution of a user wearing a lower-limb exoskeleton. This approach utilizes a short time window of kinematic data extracted from sensors integrated in the exoskeleton. For the validation of predictions, we compared the proposed approach with the method in [14] which only uses the instantaneous posture. Employing distinct users for training and testing resulted in a user-independent estimation, eliminating the need for a calibration phase. The proposed deep-learning method was trained on walking data from six users, and its closed-loop performance was evaluated on three users (two of which were not included in the training set) during treadmill and overground walking. The results showed that weight distribution can be accurately estimated using kinematic data with an R 2 of 0.9 averaged across users. In addition, this approach was compatible with real-time requirements, achieving a prediction time less than 1 ms. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Authors: Authors: (1) Clement Lhos, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (2) Emek Barıs¸ Kuc¸uktabak, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (3) Lorenzo Vianello, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA; (4) Lorenzo Amato, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and The Biorobotics Institute, Scuola Superiore Sant’Anna, 56025 Pontedera, Italy and Department of Excellence in Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy; (5) Matthew R. Short, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA; (6) Kevin Lynch2, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA; (7) Jose L. Pons, Legs and Walking Lab of Shirley Ryan AbilityLab, Chicago, IL, USA, Center for Robotics and Biosystems, Northwestern University, Evanston, IL, USA and Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA. Table of Links Abstract and I Introduction Abstract and I Introduction II Methods II Methods III Results III Results IV Discussion IV Discussion V. Conclusion, Acknowledgment, and References V. Conclusion, Acknowledgment, and References Abstract In the control of lower-limb exoskeletons with feet, the phase in the gait cycle can be identified by monitoring the weight distribution at the feet. This phase information can be used in the exoskeleton’s controller to compensate the dynamics of the exoskeleton and to assign impedance parameters. Typically the weight distribution is calculated using data from sensors such as treadmill force plates or insole force sensors. However, these solutions increase both the setup complexity and cost. For this reason, we propose a deep-learning approach that uses a short time window of joint kinematics to predict the weight distribution of an exoskeleton in real time. The model was trained on treadmill walking data from six users wearing a four-degree-of-freedom exoskeleton and tested in real time on three different users wearing the same device. This test set includes two users not present in the training set to demonstrate the model’s ability to generalize across individuals. Results show that the proposed method is able to fit the actual weight distribution with R 2 = 0.9 and is suitable for real-time control with prediction times less than 1 ms. Experiments in closed-loop exoskeleton control show that deep-learning-based weight distribution estimation can be used to replace force sensors in overground and treadmill walking. I. INTRODUCTION Lower-limb exoskeletons are increasingly being utilized in everyday tasks and rehabilitation for individuals with gait impairments. In clinical settings, these exoskeletons are typically classified into two main types: partial assistance and full mobilization. Full mobilization exoskeletons are designed for patients with severe motor control disorders, providing autonomous movement of the legs regardless of the patient’s input [1]. Conversely, partial assistance exoskeletons aid in the patient’s movement while still requiring their active input. In this paper, we focus on partial assistance exoskeletons, as they are commonly employed in rehabilitation. Depending on the functional level of the patient, these exoskeletons use techniques like haptic guidance/assistance [2] or error augmentation/resistance [3] to facilitate the (re)learning of walking behaviors. Implementing both haptic assistance and resistance strategies requires precise control over the interaction torques between the user and the exoskeleton, ensuring effective and safe physical interaction. Typically, the control of the interaction between a user and exoskeleton adopts a hierarchy of control levels: high-level, mid-level, and low-level [1], [4]. For lower-limb exoskeletons, the high-level controller calculates the desired interaction torques tailored to specific ambulatory activities, such as walking on flat terrain, stairs or ramps. The role of the mid-level controller is to estimate the various states within an activity, for example, identifying the gait states (e.g., swing and stance) during walking and setting a desired interaction torque behavior accordingly. Finally, the low-level controller is responsible for compensating the exoskeleton’s dynamics and generating motor commands, based on the desired interaction torque profile identified by the mid-level controller. In this three-level approach, an accurate, real-time estimation of the gait state in the mid-level and low-level is essential for properly controlling the exoskeleton. A common approach to gait state detection is to use gait events (e.g., heel strike, toe-off) to identify the switching between discrete gait states: left stance, right stance, and double stance [5], [6], [1]. However, achieving a smooth transition between these states is vital to ensure a control input that limits unexpected behaviors for the user. Kuc¸¨ uktabak et al. [7] proposed a ¨ whole-exoskeleton closed-loop compensation (WECC) controller that uses the ratio of ground reaction forces (GRF) (i.e., weight distribution) to approximate the double stance phase as a transition between the left and right stance. The weight distribution, quantified as the stance interpolation factor α, smoothly changes from zero (left stance) to one (right stance) during double stance, as visualized in Fig. 1. Nonetheless, the requirement of additional sensors for GRF measurements, such as treadmill force plates or insoles, increases the complexity and cost of these systems. Machine learning models have demonstrated their ability to estimate gait states using only joint kinematics of humans walking without exoskeletons, thus eliminating the need for GRF measurements [8], [9], [10]. Motivated by these results, recent studies have proposed real-time estimation of gait states with lower-limb exoskeletons. For instance, Jung et al. [11] and Liu et al. [12] used data from joint encoders and an inertial measurement unit (IMU) positioned on the trunk to classify the gait cycle during walking. These studies divided gait cycles into two (i.e., stance and swing) and eight gait states, respectively. However, discrete control states result in torque or modeling discontinuities. Camardella et al. [13] and Lippi et al. [14] address this issue by implementing a linear regression and a Neural Network (NN) model for regression with joint angles as inputs. The model’s output smoothly transitions between left and right stance, providing a continuous representation of the gait state. Despite this advancement, these approaches present some limitations. First, they only consider the present joint configuration, discarding past information. This approach might lead to inaccuracies in predictions as the weight distribution is influenced not only by the current posture, but also by the velocity and acceleration of the motion. Moreover, few considerations in these studies are given to the generalization across subjects and the prediction time performances. The latter is essential for real-time implementation of the exoskeleton’s whole-body controller. To address the aforementioned limitations, we present a deep-learning approach for estimating the weight distribution of a user wearing a lower-limb exoskeleton. This approach utilizes a short time window of kinematic data extracted from sensors integrated in the exoskeleton. For the validation of predictions, we compared the proposed approach with the method in [14] which only uses the instantaneous posture. Employing distinct users for training and testing resulted in a user-independent estimation, eliminating the need for a calibration phase. The proposed deep-learning method was trained on walking data from six users, and its closed-loop performance was evaluated on three users (two of which were not included in the training set) during treadmill and overground walking. The results showed that weight distribution can be accurately estimated using kinematic data with an R 2 of 0.9 averaged across users. In addition, this approach was compatible with real-time requirements, achieving a prediction time less than 1 ms. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Weight Distribution Estimation in Lower-Limb Exoskeletons via Deep Learning: Introduction

Authors:
(1) Hamed Alimohammadzadeh, University of Southern California, Los Angeles, USA;
(2) Rohit Bernard, University of Southern California, Los Angeles, USA;
(3) Yang Chen, University of Southern California, Los Angeles, USA;
(4) Trung Phan, University of Southern California, Los Angeles, USA;
(5) Prashant Singh, University of Southern California, Los Angeles, USA;
(6) Shuqin Zhu, University of Southern California, Los Angeles, USA;
(7) Heather Culbertson, University of Southern California, Los Angeles, USA;
(8) Shahram Ghandeharizadeh, University of Southern California, Los Angeles, USA. Table of Links Abstract and Introduction Flying Light Specks User Interaction Multimedia Systems Challenges Related Work Conclusions and Current Efforts, Acknowledgments, and References 3 USER INTERACTION FLSs will facilitate both kinesthetic (force) and tactile (skin-based) feedback to a user. While the former provides the shape and hardness of a virtual object, the latter renders an object’s surface properties. Both motivate the disaggregated FLS architecture of Figure 4 with diverse plug-in devices. To illustrate, textures for different surfaces may be rendered using different vibration actuators [19]. An actuator may be a plug-in device to the FLS system bus, enabling an experimenter to evaluate the actuator and its tactile feedback. An FLS acts as an encountered-type haptic device. Traditional kinesthetic feedback haptic rendering uses a vector-field approach [41] where the force to be displayed to the user is calculated based on their position in the 3D environment and geometric models of the virtual objects. Our FLS-based system will take a similar approach and calculate the desired force based on the user’s position and the 3D model of the object(s) to be rendered. For example, in displaying a virtual wall, the user is free to move their hand around the environment until their position penetrates inside the virtual wall, at which time the FLS would make contact with their hand to apply a force and prevent them from penetrating further into the virtual wall (Fig. 5). Using measurements of the FLS’s position and a 3D model of an object, we can determine how far the user has penetrated into the virtual object to calculate the amount of force to display to the user. With grounded impedance force-feedback haptic devices, the calculated force is then converted to motor torque using the device’s calibrated Jacobian matrix so that the desired force can be displayed to the user. Since the controllable output of the drones is thrust, displaying a set force to the user requires careful calibration between the commanded thrust and the resulting force. Previous researchers have completed this calibration for single axes of thrust/force for the Parrot AR.Drone [1]. Displaying more complex objects and interactions in 3-dimensions requires a more complete calibration of the system. Force Amplification: The size of the FLS plays an important role in its dynamics and the force that can be displayed to the user. The FLS’s motors determine the highest RPM that can be generated, and the propellers’ ratio of pitch:diameter affects the force per RPM generated from the air pushing downwards. A larger FLS usually means larger diameter propellers, leading to higher forces at a given RPM. However, the force that we will be able to output with an FLS will be significantly less than that of a traditional kinesthetic haptic device; for example, the common 3D Systems Touch device has a maximum force output of 3.3 N. The modality of human sensing also varies based on the level of force provided, with small forces (<1 N) being sensed as tactile information through the skin rather than kinesthetic information through the muscles [34]. Therefore, a single FLS will not be capable of providing sufficient force for a salient haptic interaction, especially if we want to generate the kinesthetic sensations necessary for displaying a rigid object. Therefore, the FLSs must coordinate together to combine their thrust so that an amplified force can be applied to the user. This coordination must be done in such a way that the formation remains stable during flight and that force can be passed between drones before eventually being felt by the user. This force amplification can be done either by FLSs in a parallel or series configuration (Fig. 6). In parallel, multiple FLSs will contact the user at the same time and will each provide a force proportional to that FLS’s disturbance from its own set-point. In series, only a single FLS will be in contact with the user, and both FLSs will have the same disturbance from their set-point; the user feels their combined force. User Safety: The goal of our system is to provide non-constrained encountered-type haptic feedback directly to users. To do this, the FLS must (1) be modified to allow users to safely have direct contact and (2) react to the user’s gesture input (e.g. push the FLS away from its desired position), and (3) provide meaningful haptic feedback. With the first, we use a cage to separate the user’s hand from the rotors. Cage shape, weight, and spatial density are key considerations that effect safety, lighting, and flying capability of the FLS. The added weight of the cage would affect the drone’s dynamics and induce imbalance issues when not calibrated. Spatial density of the cage’s mesh is important to prevent the user’s finger from accidentally penetrating the surfaces while not affecting the aerodynamics and lighting of the drone too significantly. However, too fine of a spatial density may significantly affect both the lighting and the flying ability of the drone. A balance must be made between choosing a mesh fine enough to protect the user’s fingers, but coarse enough to not disrupt the airflow and provide the desired textured lighting. Sensing Users: To create an effective touch-based interaction, each FLS should be capable of sensing touch from the user. At a minimum, the FLS should be able to detect the presence or absence of touch, and preferably it will be able to detect the location of touch as well. Capacitive sensors could be integrated into the FLS cage to easily detect the presence of user contact, and a more complex capacitive skin could be developed to determine point of contact. However, determining force of contact from this type of sensor would be challenging, and if this information is required more a more complex mesh of sensors must be developed. Here we propose to use open-loop control for the forces, so direct force sensing from the user is not required. This paper is available on arxiv under CC 4.0 license. Authors: (1) Hamed Alimohammadzadeh, University of Southern California, Los Angeles, USA; (2) Rohit Bernard, University of Southern California, Los Angeles, USA; (3) Yang Chen, University of Southern California, Los Angeles, USA; (4) Trung Phan, University of Southern California, Los Angeles, USA; (5) Prashant Singh, University of Southern California, Los Angeles, USA; (6) Shuqin Zhu, University of Southern California, Los Angeles, USA; (7) Heather Culbertson, University of Southern California, Los Angeles, USA; (8) Shahram Ghandeharizadeh, University of Southern California, Los Angeles, USA. Authors: Authors: (1) Hamed Alimohammadzadeh, University of Southern California, Los Angeles, USA; (2) Rohit Bernard, University of Southern California, Los Angeles, USA; (3) Yang Chen, University of Southern California, Los Angeles, USA; (4) Trung Phan, University of Southern California, Los Angeles, USA; (5) Prashant Singh, University of Southern California, Los Angeles, USA; (6) Shuqin Zhu, University of Southern California, Los Angeles, USA; (7) Heather Culbertson, University of Southern California, Los Angeles, USA; (8) Shahram Ghandeharizadeh, University of Southern California, Los Angeles, USA. Table of Links Abstract and Introduction Abstract and Introduction Flying Light Specks Flying Light Specks User Interaction User Interaction Multimedia Systems Challenges Multimedia Systems Challenges Related Work Related Work Conclusions and Current Efforts, Acknowledgments, and References Conclusions and Current Efforts, Acknowledgments, and References 3 USER INTERACTION FLSs will facilitate both kinesthetic (force) and tactile (skin-based) feedback to a user. While the former provides the shape and hardness of a virtual object, the latter renders an object’s surface properties. Both motivate the disaggregated FLS architecture of Figure 4 with diverse plug-in devices. To illustrate, textures for different surfaces may be rendered using different vibration actuators [19]. An actuator may be a plug-in device to the FLS system bus, enabling an experimenter to evaluate the actuator and its tactile feedback. An FLS acts as an encountered-type haptic device. Traditional kinesthetic feedback haptic rendering uses a vector-field approach [41] where the force to be displayed to the user is calculated based on their position in the 3D environment and geometric models of the virtual objects. Our FLS-based system will take a similar approach and calculate the desired force based on the user’s position and the 3D model of the object(s) to be rendered. For example, in displaying a virtual wall, the user is free to move their hand around the environment until their position penetrates inside the virtual wall, at which time the FLS would make contact with their hand to apply a force and prevent them from penetrating further into the virtual wall (Fig. 5). Using measurements of the FLS’s position and a 3D model of an object, we can determine how far the user has penetrated into the virtual object to calculate the amount of force to display to the user. With grounded impedance force-feedback haptic devices, the calculated force is then converted to motor torque using the device’s calibrated Jacobian matrix so that the desired force can be displayed to the user. Since the controllable output of the drones is thrust, displaying a set force to the user requires careful calibration between the commanded thrust and the resulting force. Previous researchers have completed this calibration for single axes of thrust/force for the Parrot AR.Drone [1]. Displaying more complex objects and interactions in 3-dimensions requires a more complete calibration of the system. Force Amplification: The size of the FLS plays an important role in its dynamics and the force that can be displayed to the user. The FLS’s motors determine the highest RPM that can be generated, and the propellers’ ratio of pitch:diameter affects the force per RPM generated from the air pushing downwards. A larger FLS usually means larger diameter propellers, leading to higher forces at a given RPM. However, the force that we will be able to output with an FLS will be significantly less than that of a traditional kinesthetic haptic device; for example, the common 3D Systems Touch device has a maximum force output of 3.3 N. The modality of human sensing also varies based on the level of force provided, with small forces (<1 N) being sensed as tactile information through the skin rather than kinesthetic information through the muscles [34]. Force Amplification: Therefore, a single FLS will not be capable of providing sufficient force for a salient haptic interaction, especially if we want to generate the kinesthetic sensations necessary for displaying a rigid object. Therefore, the FLSs must coordinate together to combine their thrust so that an amplified force can be applied to the user. This coordination must be done in such a way that the formation remains stable during flight and that force can be passed between drones before eventually being felt by the user. This force amplification can be done either by FLSs in a parallel or series configuration (Fig. 6). In parallel, multiple FLSs will contact the user at the same time and will each provide a force proportional to that FLS’s disturbance from its own set-point. In series, only a single FLS will be in contact with the user, and both FLSs will have the same disturbance from their set-point; the user feels their combined force. User Safety: The goal of our system is to provide non-constrained encountered-type haptic feedback directly to users. To do this, the FLS must (1) be modified to allow users to safely have direct contact and (2) react to the user’s gesture input (e.g. push the FLS away from its desired position), and (3) provide meaningful haptic feedback. With the first, we use a cage to separate the user’s hand from the rotors. Cage shape, weight, and spatial density are key considerations that effect safety, lighting, and flying capability of the FLS. The added weight of the cage would affect the drone’s dynamics and induce imbalance issues when not calibrated. Spatial density of the cage’s mesh is important to prevent the user’s finger from accidentally penetrating the surfaces while not affecting the aerodynamics and lighting of the drone too significantly. However, too fine of a spatial density may significantly affect both the lighting and the flying ability of the drone. A balance must be made between choosing a mesh fine enough to protect the user’s fingers, but coarse enough to not disrupt the airflow and provide the desired textured lighting. User Safety: Sensing Users: To create an effective touch-based interaction, each FLS should be capable of sensing touch from the user. At a minimum, the FLS should be able to detect the presence or absence of touch, and preferably it will be able to detect the location of touch as well. Capacitive sensors could be integrated into the FLS cage to easily detect the presence of user contact, and a more complex capacitive skin could be developed to determine point of contact. However, determining force of contact from this type of sensor would be challenging, and if this information is required more a more complex mesh of sensors must be developed. Here we propose to use open-loop control for the forces, so direct force sensing from the user is not required. Sensing Users: This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

How Flying Light Specks Enable Haptic Interactions

Stories, News & Mentions about TORQ

Torq’s Stories on HackerNoon

Torq's News & Mentions