A survey of visuotactile sensing technologies for robotic manipulation

Shaowei CUI; Shuo WANG; Jingyi HU; Chaofan ZHANG

doi:10.11959/j.issn.2096-6652.202222

您当前的位置：

首页 >

文章列表页 >

A survey of visuotactile sensing technologies for robotic manipulation

Special Topic: Autonomous Agent Learning for Dexterous and Accurate Manipulations | 更新时间：2024-06-05

- A survey of visuotactile sensing technologies for robotic manipulation
- Chinese Journal of Intelligent Science and Technology Vol. 4, Issue 2, Pages: 186-199(2022)
- 作者机构：
  
  1. 中国科学院自动化研究所复杂系统管理与控制国家重点实验室，北京 100190
  2. 中国科学院大学未来技术学院，北京 100049
  3. 中国科学院大学人工智能学院，北京 100049
  4. 中国科学院脑科学与智能技术卓越创新中心，上海 200031
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-6652.202222
  CLC： TP242.6
- Published Online：2022-06，
  
  Published：15 June 2022
- 稿件说明：
移动端阅览
Shaowei CUI, Shuo WANG, Jingyi HU, et al. A survey of visuotactile sensing technologies for robotic manipulation[J]. Chinese journal of intelligent science and technology, 2022, 4(2): 186-199.
DOI：

Shaowei CUI, Shuo WANG, Jingyi HU, et al. A survey of visuotactile sensing technologies for robotic manipulation[J]. Chinese journal of intelligent science and technology, 2022, 4(2): 186-199. DOI： 10.11959/j.issn.2096-6652.202222.

摘要

得益于高空间分辨率、多触觉模式的感知，视触觉传感技术已被广泛应用到机器人主动感知、位姿估计及掌内操作等各类机器人操作任务中。首先根据传感原理分类总结了目前主流的视触觉传感技术，主要可分为GelSight类视触觉传感器、双（多）目视触觉传感器以及其他类型视触觉传感器3类。在此基础上，进一步对主要触觉传感信息建模方法进行了介绍，包括接触表面三维几何形状、力/力矩、滑动等不同触觉模式。此外，聚焦机器人操作领域，对视触觉传感器的具体应用场景进行了探讨。最后，讨论了视触觉传感器的下一步发展方向以及如何将其进一步应用到机器人灵巧操作任务中。

Abstract

Thanks to the high spatial resolution and multi-mode tactile sensing

visuotactile sensing technology has been widely applied to various robotic manipulation tasks

such as robotic active perception

pose estimation

and in-hand manipulation.Firstly

the current mainstream visuotactile sensing technologies based on sensing principles were summarized

which could be mainly divided into three categories: GelSight-type visuotactile sensors

binocular (multi-view) visuotactile sensors

and other types.Meanwhile

the sensing methods of different tactile sensing modes were further summarized

including contact surface 3D geometry

force/torque

and sliding.Furthermore

focusing on the field of robot operation

the specific application scenarios of visuotactile sensors were discussed.Finally

future work of the visuotactile sensing technology and how they can be further applied to robotic dexterous manipulation tasks were given.

关键词

Keywords

references

MYERS D . Sensation and perception.Psychology [M ] . New York : Worth , 2010 .

ANSHEL J R . Visual ergonomics in the workplace [J ] . AAOHN Journal , 2007 , 55 ( 10 ): 414 - 420 .

RANASINGHE A , SORNKARN N , DASGUPTA P , et al . Salient feature of haptic-based guidance of people in low visibility environments using hard reins [J ] . IEEE Transactions on Cybernetics , 2016 , 46 ( 2 ): 568 - 579 .

AKHTAR N , MIAN A . Threat of adversarial attacks on deep learning in computer vision:a survey [J ] . IEEE Access , 2018 , 6 : 14410 - 14430 .

BILLARD A , KRAGIC D . Trends and challenges in robot manipulation [J ] . Science , 2019 , 364 ( 6446 ): eaat8414 .

LI Q , KROEMER O , SU Z , et al . A review of tactile information:perception and action through touch [J ] . IEEE Transactions on Robotics , 2020 , 36 ( 6 ): 1619 - 1634 .

CHI C , SUN X G , XUE N , et al . Recent progress in technologies for tactile sensors [J ] . Sensors (Basel,Switzerland) , 2018 , 18 ( 4 ): 948 .

ABAD A C , RANASINGHE A . Visuotactile sensors with emphasis on GelSight sensor:a review [J ] . IEEE Sensors Journal , 2020 , 20 ( 14 ): 7628 - 7638 .

YUAN W Z , DONG S Y , ADELSON E H . GelSight:high-resolution robot tactile sensors for estimating geometry and force [J ] . Sensors (Basel,Switzerland) , 2017 , 17 ( 12 ): 2762 .

DONG S Y , YUAN W Z , ADELSON E H . Improved GelSight tactile sensor for measuring geometry and slip [C ] // Proceedings of 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2017 : 137 - 144 .

DONLON E , DONG S Y , LIU M , et al . GelSlim:a high-resolution,compact,robust,and calibrated tactile-sensing finger [C ] // Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2018 : 1927 - 1934 .

TAYLOR I , DONG S Y , RODRIGUEZ A . GelSlim3.0:high-resolution measurement of shape,force and slip in a compact tactile-sensing finger [J ] . arXiv preprint,2021,arXiv:2103.12269 .

WARD-CHERRIER B , PESTELL N , CRAMPHORN L , et al . The TacTip family:soft optical tactile sensors with 3D-printed biomimetic morphologies [J ] . Soft Robotics , 2018 , 5 ( 2 ): 216 - 227 .

ALSPACH A , HASHIMOTO K , KUPPUSWAMY N , et al . Soft-bubble:a highly compliant dense geometry tactile sensor for robot manipulation [C ] // Proceedings of 2019 2nd IEEE International Conference on Soft Robotics . Piscataway:IEEE Press , 2019 : 597 - 604 .

YAMAGUCHI A , ATKESON C G . Combining finger vision and optical tactile sensing:reducing and handling errors while cutting vegetables [C ] // Proceedings of 2016 IEEE-RAS 16th International Conference on Humanoid Robots . Piscataway:IEEE Press , 2016 : 1045 - 1051 .

YAMAGUCHI A , ATKESON C G . Implementing tactile behaviors using FingerVision [C ] // Proceedings of 2017 IEEE-RAS 17th International Conference on Humanoid Robotics . Piscataway:IEEE Press , 2017 : 241 - 248 .

LAMBETA M , CHOU P W , TIAN S , et al . DIGIT:a novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation [J ] . IEEE Robotics and Automation Letters , 2020 , 5 ( 3 ): 3838 - 3845 .

WANG S X , SHE Y , ROMERO B , et al . GelSight Wedge:measuring high-resolution 3D contact geometry with a compact robot finger [C ] // Proceedings of 2021 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2021 : 6468 - 6475 .

DO W K , KENNEDY III M . DenseTact:optical tactile sensor for dense shape reconstruction [J ] . arXiv preprint,2022,arXiv:2201.01367 .

GOMES D F , LIN Z L , LUO S . GelTip:a finger-shaped optical tactile sensor for robotic manipulation [C ] // Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2021 : 9903 - 9909 .

SUN H , KUCHENBECKER K J , MARTIUS G . A soft thumb-sized vision-based sensor with accurate all-round force perception [J ] . Nature Machine Intelligence , 2022 , 4 ( 2 ): 135 - 145 .

KUPPUSWAMY N , ALSPACH A , UTTAMCHANDANI A , et al . Soft-bubble grippers for robust and perceptive manipulation [C ] // Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2020 : 9917 - 9924 .

PADMANABHA A , EBERT F , TIAN S , et al . OmniTact:a multi-directional high-resolution touch sensor [C ] // Proceedings of 2020 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2020 : 618 - 624 .

ZHANG T , CONG Y , LI X M , et al . Robot tactile sensing:vision based tactile sensor for force perception [C ] // Proceedings of 2018 IEEE 8th Annual International Conference on CYBER Technology in Automation,Control,and Intelligent Systems . Piscataway:IEEE Press , 2018 : 1360 - 1365 .

CUI S W , WANG R , HU J Y , et al . In-hand object localization using a novel high-resolution visuotactile sensor [J ] . IEEE Transactions on Industrial Electronics , 2022 , 69 ( 6 ): 6015 - 6025 .

KAKANI V , CUI X N , MA M J , et al . Vision-based tactile sensor mechanism for the estimation of contact position and force distribution using deep learning [J ] . Sensors (Basel,Switzerland) , 2021 , 21 ( 5 ): 1920 .

ZHANG L W , WANG Y , JIANG Y . Tac3D:a novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution [J ] . arXiv preprint,2022,arXiv:2202.06211 .

LEPORA N F . Soft biomimetic optical tactile sensing with the TacTip:a review [J ] . IEEE Sensors Journal , 2021 , 21 ( 19 ): 21131 - 21143 .

VLACK K , MIZOTA T , KAWAKAMI N , et al . GelForce:a vision-based traction field computer interface [C ] // Proceedings of CHI’05 Extended Abstracts on Human Factors in Computing Systems . New York:ACM Press , 2005 : 1154 - 1155 .

SATO K , KAMIYAMA K , KAWAKAMI N , et al . Finger-shaped GelForce:sensor for measuring surface traction fields for robotic hand [J ] . IEEE Transactions on Haptics , 2010 , 3 ( 1 ): 37 - 47 .

SFERRAZZA C , D'ANDREA R , . Design,motivation and evaluation of a full-resolution optical tactile sensor [J ] . Sensors (Basel,Switzerland) , 2019 , 19 ( 4 ): 928 .

SFERRAZZA C , WAHLSTEN A , TRUEEB C , et al . Ground truth force distribution for learning-based tactile sensing:a finite element approach [J ] . IEEE Access , 2019 , 7 : 173438 - 173449 .

SFERRAZZA C , D’ANDREA R , . Sim-to-real for high-resolution optical tactile sensing:from images to 3D contact force distributions [J ] . arXiv preprint,2020,arXiv:2012.11295 .

ZHANG Y Z , YUAN W H , KAN Z C , et al . Towards learning to detect and predict contact events on vision-based tactile sensors [J ] . arXiv preprint,2019,arXiv:1910.03973 .

DU Y P , ZHANG G L , ZHANG Y Z , et al . High-resolution 3-dimensional contact deformation tracking for FingerVision sensor with dense random color pattern [J ] . IEEE Robotics and Automation Letters , 2021 , 6 ( 2 ): 2147 - 2154 .

ZHANG G L , DU Y P , YU H Y , et al . DelTact:a vision-based tactile sensor using dense color pattern [J ] . arXiv preprint,2022,arXiv:2202.02179 .

JOHNSON M K , ADELSON E H . Retrographic sensing for the measurement of surface texture and shape [C ] // Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2009 : 1070 - 1077 .

ROMERO B , VEIGA F , ADELSON E . Soft,round,high resolution tactile fingertip sensors for dexterous robotic manipulation [C ] // Proceedings of 2020 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2020 : 4796 - 4802 .

CUI S W , WANG R , HU J Y , et al . Self-supervised contact geometry learning by GelStereo visuotactile sensing [J ] . IEEE Transactions on Instrumentation and Measurement , 2022 , 71 : 1 - 9 .

KAMIYAMA K , KAJIMOTO H , KAWAKAMI N , et al . Evaluation of a vision-based tactile sensor [C ] // Proceedings of IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2004 : 1542 - 1547 .

MA D L , DONLON E , DONG S Y , et al . Dense tactile force estimation using GelSlim and inverse FEM [C ] // Proceedings of 2019 International Conference on Robotics and Automation . Piscataway:IEEE Press , 2019 : 5418 - 5424 .

ZHANG Y Z , KAN Z C , YANG Y , et al . Effective estimation of contact force and torque for vision-based tactile sensors with Helmholtz-Hodge decomposition [J ] . IEEE Robotics and Automation Letters , 2019 , 4 ( 4 ): 4094 - 4101 .

DONG S Y , MA D L , DONLON E , et al . Maintaining grasps within slipping bounds by monitoring incipient slip [C ] // Proceedings of 2019 International Conference on Robotics and Automation . Piscataway:IEEE Press , 2019 : 3818 - 3824 .

SUI R M , ZHANG L W , LI T M , et al . Incipient slip detection method with vision-based tactile sensor based on distribution force and deformation [J ] . IEEE Sensors Journal , 2021 , 21 ( 22 ): 25973 - 25985 .

LI J H , DONG S Y , ADELSON E . Slip detection with combined tactile and visual information [C ] // Proceedings of 2018 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2018 : 7772 - 7777 .

ZHANG Y Z , KAN Z C , TSE Y A , et al . Fingervision tactile sensor design and slip detection using convolutional LSTM network [J ] . arXiv preprint,2018,arXiv:1810.02653 .

JAMES J W , LEPORA N F . Slip detection for grasp stabilization with a multifingered tactile robot hand [J ] . IEEE Transactions on Robotics , 2021 , 37 ( 2 ): 506 - 519 .

ABAD A C , REID D , RANASINGHE A . HaptiTemp:a next-generation thermosensitive GelSight-like visuotactile sensor [J ] . IEEE Sensors Journal , 2022 , 22 ( 3 ): 2722 - 2734 .

YUAN W Z , SRINIVASAN M A , ADELSON E H . Estimating object hardness with a GelSight touch sensor [C ] // Proceedings of 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2016 : 208 - 215 .

CALANDRA R , OWENS A , UPADHYAYA M , et al . The feeling of success:does touch sensing help predict grasp outcomes? [J ] . arXiv preprint,2017,arXiv:1710.05512 .

CALANDRA R , OWENS A , JAYARAMAN D , et al . More than a feeling:learning to grasp and regrasp using vision and touch [J ] . IEEE Robotics and Automation Letters , 2018 , 3 ( 4 ): 3300 - 3307 .

HOGAN F R , BAUZA M , CANAL O , et al . Tactile regrasp:grasp adjustments via simulated tactile transformations [C ] // Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2018 : 2963 - 2970 .

DONG S Y , JHA D K , ROMERES D , et al . Tactile-RL for insertion:generalization to objects of unknown geometry [C ] // Proceedings of 2021 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2021 : 6437 - 6443 .

WANG C , WANG S X , ROMERO B , et al . SwingBot:learning physical features from in-hand tactile exploration for dynamic swing-up manipulation [C ] // Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2021 : 5633 - 5640 .

ANZAI T , TAKAHASHI K . Deep gated multi-modal learning:in-hand object pose changes estimation using tactile and image data [C ] // Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2021 : 9361 - 9368 .

LI R , PLATT R , YUAN W Z , et al . Localization and manipulation of small parts using GelSight tactile sensing [C ] // Proceedings of 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway:IEEE Press , 2014 : 3988 - 3993 .

BAUZA M , VALLS E , LIM B , et al . Tactile object pose estimation from the first touch with geometric contact rendering [J ] . arXiv preprint,2020,arXiv:2012.05205 .

BAUZA M , CANAL O , RODRIGUEZ A . Tactile mapping and localization from high-resolution tactile imprints [C ] // Proceedings of 2019 International Conference on Robotics and Automation . Piscataway:IEEE Press , 2019 : 3811 - 3817 .

LEPORA N F , LLOYD J . Optimal deep learning for robot touch:training accurate pose models of 3D surfaces and edges [J ] . IEEE Robotics ＆ Automation Magazine , 2020 , 27 ( 2 ): 66 - 77 .

SHE Y , WANG S X , DONG S Y , et al . Cable manipulation with a tactile-reactive gripper [J ] . The International Journal of Robotics Research , 2021 , 40 ( 12/13/14 ): 1385 - 1401 .

KIM S , RODRIGUEZ A . Active extrinsic contact sensing:application to general peg-in-hole insertion [J ] . arXiv preprint,2021,arXiv:2110.03555 .

LAMBETA M , XU H Z , XU J W , et al . PyTouch:a machine learning library for touch processing [C ] // Proceedings of 2021 IEEE International Conference on Robotics and Automation . Piscataway:IEEE Press , 2021 : 13208 - 13214 .

SORENSEN S , KOLAGUNDA A , SAPONARO P , et al . Refractive stereo ray tracing for reconstructing underwater structures [C ] // Proceedings of 2015 IEEE International Conference on Image Processing . Piscataway:IEEE Press , 2015 : 1712 - 1716 .

FISHEL J A , LOEB G E . Sensing tactile microvibrations with the BioTac—comparison with human sensitivity [C ] // Proceedings of 2012 4th IEEE RAS ＆ EMBS International Conference on Biomedical Robotics and Biomechatronics . Piscataway:IEEE Press , 2012 : 1122 - 1127 .

Views

899

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：35685 Visits Today：160

⁰