面向机器人操作任务的视触觉传感技术综述

崔少伟; 王硕; 胡静怡; 张超凡

doi:10.11959/j.issn.2096-6652.202222

您当前的位置：

首页 >

文章列表页 >

面向机器人操作任务的视触觉传感技术综述

专题：自主智能体灵巧精准操作学习 | 更新时间：2024-06-05

- 面向机器人操作任务的视触觉传感技术综述
- A survey of visuotactile sensing technologies for robotic manipulation
- 智能科学与技术学报 2022年4卷第2期页码：186-199
- 作者机构：
  
  1. 中国科学院自动化研究所复杂系统管理与控制国家重点实验室，北京 100190
  2. 中国科学院大学未来技术学院，北京 100049
  3. 中国科学院大学人工智能学院，北京 100049
  4. 中国科学院脑科学与智能技术卓越创新中心，上海 200031
- 作者简介：
  
  [ "崔少伟（1996− ），男，中国科学院自动化研究所博士生，主要研究方向为机器人触觉感知与灵巧操作" ]
  [ "王硕（1973− ），男，博士，中国科学院自动化研究所复杂系统管理与控制国家重点实验室研究员，中国科学院大学人工智能学院首席教授，国家重点研发计划“智能机器人”重点专项总体组专家，国际标准化组织 ISO/TC 299 专家，中国机电一体化技术应用协会常务理事。主要从事机器人学习、视触融合灵巧操作、仿生机器鱼、水下机器人−作业臂系统等研究。在国内外期刊和会议上发表学术论文120余篇，合著图书4本，获授权发明专利21项、软件著作权12项，主持制定机器人国家标准2项。主持我国科学技术部科技创新2030—“新一代人工智能”重大项目“自主智能体灵巧精准操作学习方法研究与验证”、国家863计划项目、国家自然科学基金联合基金重点项目和面上项目、支撑计划项目、国际合作项目、北京市科学技术委员会项目等多项项目。曾获得国家自然科学奖二等奖1项，北京市科学技术奖一等奖1项，省部级二等奖 4 项，中国自动化学会技术发明奖一等奖1 项，中国自动化学会科普奖 1 项。中国科学院、中国自动化学会、中国人工智能学会优秀博士论文指导教师，曾获得中国科学院大学“领雁银奖·振翅奖”" ]
  [ "胡静怡（1997− ），女，中国科学院自动化研究所博士生，主要研究方向为机器人触觉主动感知" ]
  [ "张超凡（1998− ），女，中国科学院自动化研究所博士生，主要研究方向为机器人多模态感知与灵巧操作" ]
- 基金信息：
  
  科技创新2030—“新一代人工智能”重大项目;The National Key Research and Development Program of China(2018AAA0103003)
- DOI：10.11959/j.issn.2096-6652.202222
  中图分类号： TP242.6
- 网络出版日期：2022-06，
  
  纸质出版日期：2022-06-15
- 稿件说明：
移动端阅览
崔少伟, 王硕, 胡静怡, 等. 面向机器人操作任务的视触觉传感技术综述[J]. 智能科学与技术学报, 2022,4(2):186-199.

Shaowei CUI, Shuo WANG, Jingyi HU, et al. A survey of visuotactile sensing technologies for robotic manipulation[J]. Chinese journal of intelligent science and technology, 2022, 4(2): 186-199.
崔少伟, 王硕, 胡静怡, 等. 面向机器人操作任务的视触觉传感技术综述[J]. 智能科学与技术学报, 2022,4(2):186-199. DOI： 10.11959/j.issn.2096-6652.202222.

Shaowei CUI, Shuo WANG, Jingyi HU, et al. A survey of visuotactile sensing technologies for robotic manipulation[J]. Chinese journal of intelligent science and technology, 2022, 4(2): 186-199. DOI： 10.11959/j.issn.2096-6652.202222.

摘要

得益于高空间分辨率、多触觉模式的感知，视触觉传感技术已被广泛应用到机器人主动感知、位姿估计及掌内操作等各类机器人操作任务中。首先根据传感原理分类总结了目前主流的视触觉传感技术，主要可分为GelSight类视触觉传感器、双（多）目视触觉传感器以及其他类型视触觉传感器3类。在此基础上，进一步对主要触觉传感信息建模方法进行了介绍，包括接触表面三维几何形状、力/力矩、滑动等不同触觉模式。此外，聚焦机器人操作领域，对视触觉传感器的具体应用场景进行了探讨。最后，讨论了视触觉传感器的下一步发展方向以及如何将其进一步应用到机器人灵巧操作任务中。

Abstract

Thanks to the high spatial resolution and multi-mode tactile sensing

visuotactile sensing technology has been widely applied to various robotic manipulation tasks

such as robotic active perception

pose estimation

and in-hand manipulation.Firstly

the current mainstream visuotactile sensing technologies based on sensing principles were summarized

which could be mainly divided into three categories: GelSight-type visuotactile sensors

binocular (multi-view) visuotactile sensors

and other types.Meanwhile

the sensing methods of different tactile sensing modes were further summarized

including contact surface 3D geometry

force/torque

and sliding.Furthermore

focusing on the field of robot operation

the specific application scenarios of visuotactile sensors were discussed.Finally

future work of the visuotactile sensing technology and how they can be further applied to robotic dexterous manipulation tasks were given.

关键词

Keywords

references

MYERS D . Sensation and perception.Psychology [M ] . New York : Worth , 2010 .

ANSHEL J R . Visual ergonomics in the workplace [J ] . AAOHN Journal , 2007 , 55 ( 10 ): 414 - 420 .

RANASINGHE A , SORNKARN N , DASGUPTA P , et al . Salient feature of haptic-based guidance of people in low visibility environments using hard reins [J ] . IEEE Transactions on Cybernetics , 2016 , 46 ( 2 ): 568 - 579 .

AKHTAR N , MIAN A . Threat of adversarial attacks on deep learning in computer vision:a survey [J ] . IEEE Access , 2018 , 6 : 14410 - 14430 .

BILLARD A , KRAGIC D . Trends and challenges in robot manipulation [J ] . Science , 2019 , 364 ( 6446 ): eaat8414 .

LI Q , KROEMER O , SU Z , et al . A review of tactile information:perception and action through touch [J ] . IEEE Transactions on Robotics , 2020 , 36 ( 6 ): 1619 - 1634 .

CHI C , SUN X G , XUE N , et al . Recent progress in technologies for tactile sensors [J ] . Sensors (Basel,Switzerland) , 2018 , 18 ( 4 ): 948 .

ABAD A C , RANASINGHE A . Visuotactile sensors with emphasis on GelSight sensor:a review [J ] . IEEE Sensors Journal , 2020 , 20 ( 14 ): 7628 - 7638 .

YUAN W Z , DONG S Y , ADELSON E H . GelSight:high-resolution robot tactile sensors for estimating geometry and force [J ] . Sensors (Basel,Switzerland) , 2017 , 17 ( 12 ): 2762 .

DONG S Y , YUAN W Z , ADELSON E H . Improved GelSight tactile sensor for measuring geometry and slip [C ] // Proceedings of 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2017 : 137 - 144 .

DONLON E , DONG S Y , LIU M , et al . GelSlim:a high-resolution,compact,robust,and calibrated tactile-sensing finger [C ] // Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2018 : 1927 - 1934 .

TAYLOR I , DONG S Y , RODRIGUEZ A . GelSlim3.0:high-resolution measurement of shape,force and slip in a compact tactile-sensing finger [J ] . arXiv preprint,2021,arXiv:2103.12269 .

WARD-CHERRIER B , PESTELL N , CRAMPHORN L , et al . The TacTip family:soft optical tactile sensors with 3D-printed biomimetic morphologies [J ] . Soft Robotics , 2018 , 5 ( 2 ): 216 - 227 .

ALSPACH A , HASHIMOTO K , KUPPUSWAMY N , et al . Soft-bubble:a highly compliant dense geometry tactile sensor for robot manipulation [C ] // Proceedings of 2019 2nd IEEE International Conference on Soft Robotics . Piscataway:IEEE Press , 2019 : 597 - 604 .

YAMAGUCHI A , ATKESON C G . Combining finger vision and optical tactile sensing:reducing and handling errors while cutting vegetables [C ] // Proceedings of 2016 IEEE-RAS 16th International Conference on Humanoid Robots . Piscataway:IEEE Press , 2016 : 1045 - 1051 .

YAMAGUCHI A , ATKESON C G . Implementing tactile behaviors using FingerVision [C ] // Proceedings of 2017 IEEE-RAS 17th International Conference on Humanoid Robotics . Piscataway:IEEE Press , 2017 : 241 - 248 .

LAMBETA M , CHOU P W , TIAN S , et al . DIGIT:a novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation [J ] . IEEE Robotics and Automation Letters , 2020 , 5 ( 3 ): 3838 - 3845 .

WANG S X , SHE Y , ROMERO B , et al . GelSight Wedge:measuring high-resolution 3D contact geometry with a compact robot finger [C ] // Proceedings of 2021 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2021 : 6468 - 6475 .

DO W K , KENNEDY III M . DenseTact:optical tactile sensor for dense shape reconstruction [J ] . arXiv preprint,2022,arXiv:2201.01367 .

GOMES D F , LIN Z L , LUO S . GelTip:a finger-shaped optical tactile sensor for robotic manipulation [C ] // Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2021 : 9903 - 9909 .

SUN H , KUCHENBECKER K J , MARTIUS G . A soft thumb-sized vision-based sensor with accurate all-round force perception [J ] . Nature Machine Intelligence , 2022 , 4 ( 2 ): 135 - 145 .

KUPPUSWAMY N , ALSPACH A , UTTAMCHANDANI A , et al . Soft-bubble grippers for robust and perceptive manipulation [C ] // Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2020 : 9917 - 9924 .

PADMANABHA A , EBERT F , TIAN S , et al . OmniTact:a multi-directional high-resolution touch sensor [C ] // Proceedings of 2020 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2020 : 618 - 624 .

ZHANG T , CONG Y , LI X M , et al . Robot tactile sensing:vision based tactile sensor for force perception [C ] // Proceedings of 2018 IEEE 8th Annual International Conference on CYBER Technology in Automation,Control,and Intelligent Systems . Piscataway:IEEE Press , 2018 : 1360 - 1365 .

CUI S W , WANG R , HU J Y , et al . In-hand object localization using a novel high-resolution visuotactile sensor [J ] . IEEE Transactions on Industrial Electronics , 2022 , 69 ( 6 ): 6015 - 6025 .

KAKANI V , CUI X N , MA M J , et al . Vision-based tactile sensor mechanism for the estimation of contact position and force distribution using deep learning [J ] . Sensors (Basel,Switzerland) , 2021 , 21 ( 5 ): 1920 .

ZHANG L W , WANG Y , JIANG Y . Tac3D:a novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution [J ] . arXiv preprint,2022,arXiv:2202.06211 .

LEPORA N F . Soft biomimetic optical tactile sensing with the TacTip:a review [J ] . IEEE Sensors Journal , 2021 , 21 ( 19 ): 21131 - 21143 .

VLACK K , MIZOTA T , KAWAKAMI N , et al . GelForce:a vision-based traction field computer interface [C ] // Proceedings of CHI’05 Extended Abstracts on Human Factors in Computing Systems . New York:ACM Press , 2005 : 1154 - 1155 .

SATO K , KAMIYAMA K , KAWAKAMI N , et al . Finger-shaped GelForce:sensor for measuring surface traction fields for robotic hand [J ] . IEEE Transactions on Haptics , 2010 , 3 ( 1 ): 37 - 47 .

SFERRAZZA C , D'ANDREA R , . Design,motivation and evaluation of a full-resolution optical tactile sensor [J ] . Sensors (Basel,Switzerland) , 2019 , 19 ( 4 ): 928 .

SFERRAZZA C , WAHLSTEN A , TRUEEB C , et al . Ground truth force distribution for learning-based tactile sensing:a finite element approach [J ] . IEEE Access , 2019 , 7 : 173438 - 173449 .

SFERRAZZA C , D’ANDREA R , . Sim-to-real for high-resolution optical tactile sensing:from images to 3D contact force distributions [J ] . arXiv preprint,2020,arXiv:2012.11295 .

ZHANG Y Z , YUAN W H , KAN Z C , et al . Towards learning to detect and predict contact events on vision-based tactile sensors [J ] . arXiv preprint,2019,arXiv:1910.03973 .

DU Y P , ZHANG G L , ZHANG Y Z , et al . High-resolution 3-dimensional contact deformation tracking for FingerVision sensor with dense random color pattern [J ] . IEEE Robotics and Automation Letters , 2021 , 6 ( 2 ): 2147 - 2154 .

ZHANG G L , DU Y P , YU H Y , et al . DelTact:a vision-based tactile sensor using dense color pattern [J ] . arXiv preprint,2022,arXiv:2202.02179 .

JOHNSON M K , ADELSON E H . Retrographic sensing for the measurement of surface texture and shape [C ] // Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2009 : 1070 - 1077 .

ROMERO B , VEIGA F , ADELSON E . Soft,round,high resolution tactile fingertip sensors for dexterous robotic manipulation [C ] // Proceedings of 2020 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2020 : 4796 - 4802 .

CUI S W , WANG R , HU J Y , et al . Self-supervised contact geometry learning by GelStereo visuotactile sensing [J ] . IEEE Transactions on Instrumentation and Measurement , 2022 , 71 : 1 - 9 .

KAMIYAMA K , KAJIMOTO H , KAWAKAMI N , et al . Evaluation of a vision-based tactile sensor [C ] // Proceedings of IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2004 : 1542 - 1547 .

MA D L , DONLON E , DONG S Y , et al . Dense tactile force estimation using GelSlim and inverse FEM [C ] // Proceedings of 2019 International Conference on Robotics and Automation . Piscataway:IEEE Press , 2019 : 5418 - 5424 .

ZHANG Y Z , KAN Z C , YANG Y , et al . Effective estimation of contact force and torque for vision-based tactile sensors with Helmholtz-Hodge decomposition [J ] . IEEE Robotics and Automation Letters , 2019 , 4 ( 4 ): 4094 - 4101 .

DONG S Y , MA D L , DONLON E , et al . Maintaining grasps within slipping bounds by monitoring incipient slip [C ] // Proceedings of 2019 International Conference on Robotics and Automation . Piscataway:IEEE Press , 2019 : 3818 - 3824 .

SUI R M , ZHANG L W , LI T M , et al . Incipient slip detection method with vision-based tactile sensor based on distribution force and deformation [J ] . IEEE Sensors Journal , 2021 , 21 ( 22 ): 25973 - 25985 .

LI J H , DONG S Y , ADELSON E . Slip detection with combined tactile and visual information [C ] // Proceedings of 2018 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2018 : 7772 - 7777 .

ZHANG Y Z , KAN Z C , TSE Y A , et al . Fingervision tactile sensor design and slip detection using convolutional LSTM network [J ] . arXiv preprint,2018,arXiv:1810.02653 .

JAMES J W , LEPORA N F . Slip detection for grasp stabilization with a multifingered tactile robot hand [J ] . IEEE Transactions on Robotics , 2021 , 37 ( 2 ): 506 - 519 .

ABAD A C , REID D , RANASINGHE A . HaptiTemp:a next-generation thermosensitive GelSight-like visuotactile sensor [J ] . IEEE Sensors Journal , 2022 , 22 ( 3 ): 2722 - 2734 .

YUAN W Z , SRINIVASAN M A , ADELSON E H . Estimating object hardness with a GelSight touch sensor [C ] // Proceedings of 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2016 : 208 - 215 .

CALANDRA R , OWENS A , UPADHYAYA M , et al . The feeling of success:does touch sensing help predict grasp outcomes? [J ] . arXiv preprint,2017,arXiv:1710.05512 .

CALANDRA R , OWENS A , JAYARAMAN D , et al . More than a feeling:learning to grasp and regrasp using vision and touch [J ] . IEEE Robotics and Automation Letters , 2018 , 3 ( 4 ): 3300 - 3307 .

HOGAN F R , BAUZA M , CANAL O , et al . Tactile regrasp:grasp adjustments via simulated tactile transformations [C ] // Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2018 : 2963 - 2970 .

DONG S Y , JHA D K , ROMERES D , et al . Tactile-RL for insertion:generalization to objects of unknown geometry [C ] // Proceedings of 2021 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2021 : 6437 - 6443 .

WANG C , WANG S X , ROMERO B , et al . SwingBot:learning physical features from in-hand tactile exploration for dynamic swing-up manipulation [C ] // Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2021 : 5633 - 5640 .

ANZAI T , TAKAHASHI K . Deep gated multi-modal learning:in-hand object pose changes estimation using tactile and image data [C ] // Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2021 : 9361 - 9368 .

LI R , PLATT R , YUAN W Z , et al . Localization and manipulation of small parts using GelSight tactile sensing [C ] // Proceedings of 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2014 : 3988 - 3993 .

BAUZA M , VALLS E , LIM B , et al . Tactile object pose estimation from the first touch with geometric contact rendering [J ] . arXiv preprint,2020,arXiv:2012.05205 .

BAUZA M , CANAL O , RODRIGUEZ A . Tactile mapping and localization from high-resolution tactile imprints [C ] // Proceedings of 2019 International Conference on Robotics and Automation . Piscataway:IEEE Press , 2019 : 3811 - 3817 .

LEPORA N F , LLOYD J . Optimal deep learning for robot touch:training accurate pose models of 3D surfaces and edges [J ] . IEEE Robotics ＆ Automation Magazine , 2020 , 27 ( 2 ): 66 - 77 .

SHE Y , WANG S X , DONG S Y , et al . Cable manipulation with a tactile-reactive gripper [J ] . The International Journal of Robotics Research , 2021 , 40 ( 12/13/14 ): 1385 - 1401 .

KIM S , RODRIGUEZ A . Active extrinsic contact sensing:application to general peg-in-hole insertion [J ] . arXiv preprint,2021,arXiv:2110.03555 .

LAMBETA M , XU H Z , XU J W , et al . PyTouch:a machine learning library for touch processing [C ] // Proceedings of 2021 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2021 : 13208 - 13214 .

SORENSEN S , KOLAGUNDA A , SAPONARO P , et al . Refractive stereo ray tracing for reconstructing underwater structures [C ] // Proceedings of 2015 IEEE International Conference on Image Processing . Piscataway:IEEE Press , 2015 : 1712 - 1716 .

FISHEL J A , LOEB G E . Sensing tactile microvibrations with the BioTac—comparison with human sensitivity [C ] // Proceedings of 2012 4th IEEE RAS ＆ EMBS International Conference on Biomedical Robotics and Biomechatronics . Piscataway:IEEE Press , 2012 : 1122 - 1127 .

浏览量

899

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

暂无数据