Virtual reality environment for motion optimization using haptic devices. Application for Assembly Disassembly operation simulation by hand gestures recognition

Thesis Director : Peter MITROUCHEV (MCF-HDR, SIREP), Franck QUAINE (MCF-HDR, GIPSA-Lab)

PhD School : I-MEP2/ EDISCE

Start date : October 2018

 Proposed funding : Contrat doctoral. Demande I-MEP2 allocation MESR (Ministère de l'Enseignement Supérieure et de la Recherche) for 3 years

Brief Description: 

Context:

The existing platforms for assembly/disassembly (A/D) simulation by hand gestures recognition are often badly integrated in the Products Development Process (PDP). Some approaches for modelling of disassembly were proposed but they do not allow validating them because they do not take account of the physiological state of the operator for varied conditions of request (postures, efforts…). In the last decade, virtual reality (VR) technology has evolved to a new level of sophistication. Now it combines several human – computer interfaces (HCI) to provide various sensations such as: visual, auditory, haptic, which enables users to become immersed in a computer generated platform. Muscle-computer interaction (muCI), for instance, represents a useful tool to address hand gestures recognition and forearm muscle actions, particularly for characterizing the involved fingers and the exhaustion level during grasping. Adding such a technique in HCI to improve platform of immersion (VR) represents a scientific challenge for research today.

Partners: The proposed topic is a follow-up to WP9, Task 9.1. Interaction and Manipulation within Virtual Interactive Scenes of the European Infrastructure VISIONAIR (http://www.infra-visionair.eu/) (2011-2015) and currently working with the research themes of LABEX PERSYVAL Lab, Research Actions, AAR Authoring Augmented Reality, WP2, Real-time capture and simulation of the real world. Representation and editing of virtual prototypes. Natural interaction with the augmented world, (http://www.persyval-lab.org/index.html) in cooperation with GIPSA-Lab. In addition, it is part of a joint research theme, as part of a collaboration, begun seven years ago, with Shanghai Key Laboratory of Mechanical Automation and Robotics of the University of Shanghai, China.

Description of the subject:

Current simulation platforms do not offer the necessary information and versatility required for a complete A/D process simulation, including human/operator data management based on physiologic data (ElectroMyoGraphy-EMG signal). In order to improve the quality of a real-time A/D simulation environment, the present research subject is focused on two elements: a better haptic devices integration and mobility module evaluation including physiologic data about the musculo-skeletal state. The later should be able to generate the mobility of a component from an assembly with respect to its surroundings and the muscular capabilities of the operator. As a result, more transparent access behavior of the components in the simulation environment will be obtained. An immersive platform that integrates one or more haptic devices, using the model data, cognitive constraints, efforts in the muscles and their optimization, mobility information and the state of the musculo-skeletal system, should offer a more realistic simulation process through kinematic guided movements and human muscular control assessment. Nowadays, VR environments have significantly evolved towards the A/D simulation, highlighting new requirements for the preparation stages and their integration. Many of these platforms use haptic feedback and are facing difficulties while simulating the insertion/extraction operations realized by a human.

Objectives:

In this context, the main objective of this research is to improve the A/D process simulation through better haptic devices integration including physiologic data. To this end, a series of tests with 6 degrees of freedom (DOF) haptic device and EMG data are necessary. The aim is: i). to provide a robust acquisition technology associated to an appropriated EMG signal processing, based on the use of EMG network sensors (localization, tolerance in wrong positioning, optimization the number of electrodes) or the new technologies as MYO, for instance, in order to improve the time delay of detection for the separation and the classification of different hand gestures. ii). to propose a mobility module able to model contact relations between elementary components of a product and to determine the relative mobilities of the assembly components. A model including the physiologic state of the operator, including the quantification of the muscular fatigue for example, for planning and simulation of A/D operations will be proposed as well. It will be validated via its integration in a constrained virtual environment allowing the simulation of A/D operations within the framework of the existing data-processing environment, as its integration in the PDP. The main issues addressed in this proposal are: i). to perform A/D simulation tests with a 6 DOF haptic device; ii). to introduce muCI as supplementary data in A/D simulation; iii). to evaluate the possibility of implementing a mobility module in a real-time A/D simulation environment; iv). to interact naturally with the augmented world via human motion modelling and simulation

Note: No level of French is required

Skills and Profile:

The candidate should hold a Master Degree amongst: Mechanics, Robotics, Industrial Engineering, Artificial Intelligence. Fluency in English is a plus.

Desired knowledge Catia V5, C++, MatLab, CAD…

References

1.     Liu Z., Tan J., Constrained behaviour manipulation for interactive assembly in a virtual environment, International Journal of Advanced Manufacturing Technology, 2007, vol. 32, p. 797–810

2.     Cappelli, F., et al., 2007. Design for disassembly: a methodology for identifying the optimal disassembly sequence. J Eng Des 18(6):563–575

3.     Seliger, et al., 2000, Technological Conditions for Selling Use instead of Selling Products, Proceedings of the 7^th International ClRP Seminar on Life Cycle Engineering, Tokyo, Japan, 17-22

4.     Tang Y., et al.., 2002. Disassembly Modeling, Planning, and Application, Journal of Manufacturing Systems, 21(3), 200–217

5.     Lambert A.J.D., (2003) Disassembly sequencing: a survey. International Journal of Production Research. 41 (16), 3721-3759

6.     Kongar, E. and Gupta, S. M., (2006) Disassembly Sequencing using Genetic Algorithm, International Journal of Advanced Manufacturing Technology, Vol. 30, No. 5-6, 497-506

7.     Germanico G-B., et al., T., Development of a Haptic Virtual Reality System for Assembly Planning and Evaluation Procedia Technology, 7: pp. 265-272, 2013

8.     Kok A.J.F, Van Liere R., A Multimodal Virtual Reality Interface for 3D Interaction with VTK,Knowledge and Information Systems, Knowledge and Information Systems, 2007

9.     Gomez. D.A. et al.. (2014). Toward “Pseudo-Haptic Avatars”: Modifying the Visual Animation of Self-Avatar Can Simulate the Perception of Weight Lifting, IEEE Transactions on Visualization and Computer Graphics, April 2014, Vol. 20 issue 4: 654-661

10.   Brain R. U. et al., (2003) A Model of Human Muscle Energy Expenditure. Comput Methods Biomech Biomed Engin 6:99–111

11.   Lind A.R. Muscle fatigue and recovery from fatigue induced by sustained contractions. J Physiol. doi: 10.1113/jphysiol.1959.sp006231

12.   Popescu V.G et al., A virtual–reality–based telerehabilitation system with force feedback. IEEE Inf. Technol. Biomed., 4(1):45-51, 2000

13.     Salisbury J.K., Srinivasan M.A., Projects in VR: phantom based haptic interaction with virtual objects. IEEE Comp. Graph. Appl., Sept./Oct, 1997

14     Iacob R., Mitrouchev P., Léon J-C., Contact identification for assembly/disassembly simulation with a haptic device, 4-th Intuition Conference on Virtual Reality and Virtual Environments proceedings, 4-5 October 2007, Athens, Greece, pp. 111-249.

15     Iacob R., Mitrouchev P., Léon J-C., 2007, A simulation framework for assembly/disassembly process modelling, ASME’2007, IDETC/CIE Conferences, Computers and Information in Engineering Conference, September 4-7, 2007, Las Vegas, Nevada, USA.

16   Iacob R., Mitrouchev P., Léon J-C., Contact identification for assembly/disassembly simulation with a haptic device, The Visual Computer, ISSN: 0178-2789, Ed. Springer-Verlag, (2008) 24: 973-979.

17     Iacob R., Mitrouchev P., Léon J-C., Assembly simulation incorporating component mobility modelling based on functional surfaces, The International Journal of Advanced Manufacturing Technology, Ed. Springer, ISSN: 0268-3768, 2009, (en presse).

18     Iacob R., Popescu D., Mitrouchev P., Assembly/disassembly analysis and modelling techniques: a review, Strojniški vestnik, Journal of Mechanical Engineering, 58 (2012) 11, 653-664, DOI:10.5545/sv-jme.2011.183.

19    Wang Chenggang, “Generation of disassembly ranges. Integration in a Virtual Reality environment”, Thèse de Doctorat, INPGrenoble, novembre 2014.

20     Mitrouchev P., Wang C, LI G., Lu L., Selective Disassembly sequences generation based on lowest levels disassembly graph method. IJAMT-International journal of advanced manufacturing technology, Ed. Springer, Volume 80, Issue 1 (2015), pp. 141-159. (3 citations) Ed. Springer, DOI 10.1007/s00170-016-8827-6.

21     Chenggang WANG, Peter MITROUCHEV, Guiqin LI and Lixin LU, Disassembly operations’ efficiency evaluation in virtual environment, IJCIM-International Journal of Computer Integrated Manufacturing, Ed. Taylor & Francis, , 2015, (DOI: 10.1080/0951192X.2015.1033752.

22.   Peter Mitrouchev, Cheng-gang Wang, Jing-tao Chen, Disassembly Process Simulation in Virtual Reality Environment, in Advances on Mechanics, Design Engineering and Manufacturing, Lecture Notes in Mechanical Engineering, DOI 10.1007/978-3-319-45781-9_63

23  Chen Jingtao, “Biomechanical analysis of haptic interaction. Application”, Thèse de Doctorat, Grenoble INP, janvier 2017.

24   Jingtao Chen, Peter Mitrouchev, Sabine Coquillart, Franck Quaine, Disassembly task evaluation by muscle fatigue estimation in Virtual reality environment, International Journal of Advanced Manufacturing Technology, Ed. Springer, 2016, DOI 10.1007/s00170-016-8827-6

25   P. Mitrouchev, J. Chen, S. Coquillart and F. Quaine, Length perception in virtual reality environment, MIT- Management and Innovative Technologies 2016 Conference Proceedings, Fiesa, Slovenia, 05-07 September, 2016, pp. 56-62.

26   P. Mitrouchev, C. Wang, J. Chen, Virtual disassembly sequences generation and evaluation, 6th CIRP Conference on Assembly Technologies and Systems (CATS), Goteborg, Sweden, Ed. Elsevier, Procedia CIRP 44 (2016 ) 347 – 352.

27   P. Mitrouchev, C. Wang, J. Chen, Virtual reality environment for disassembly sequences generation and evaluation, EuroVR2015, Milano, October 15-16, 2015.

28   Wang C., Mitrouchev P., Li G. and Lu L., 3D Geometric Removability Analysis for Virtual Disassembly evaluation, Proceedings IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM 2014), July 8-11, 2014, Besançon, France, 6 pages (CD-ROM, paper N°DETC2014-34943).

29   Said H., Mitrouchev P., Tollenaere M., Disassembly sequencing for end-of-life products, Proceedings of Joint Conference on Mechanical, Design Engineering & Advanced Manufacturing, Toulouse : France (2014)

30   Wang C., Mitrouchev P., Li G., Lu L-X., Least levels disassembly graph method for selective disassembly planning, ASME 2014 International Design and Engineering Technical Conferences & Computers and Information in Engineering Conference, IDETC/CIE 2014, August 17-20, 2014 in Buffalo, NY., USA,  10 pages (CD-ROM, paper N°#264).   

31   R. Iacob, D. Popescu, F. Noel, T. Louis, P. Mitrouchev, A. Larcher, Assembly simulation using haptic devices, MIT 2014 Conference Proceedings, Fiesa, Slovenia, 27.09-01.10 2014, pp.19-25.