Implement a simple kalman filter. Then see if there are some implementations in python.
Add forward kinematics to all the fingers. Calculate velocity and possibly acceleration on them.
Add all calculated forces from all the fingers to the force controller and to the equivalent applied object excerted force.
Use the finger forward kinematics and vision information to adjust object position in the estimated position. and maybe do it later for velocity.
Check if the object is touched at all in the model. Don't exert force/torque if the position is outside of the box faces. The force can not be inside the faces, if this is the case, then reestimate object position. Fingers must be always touching object if there is some force measured on them. If there is force measured on them and there is no possible logical position for the object to be, then it means the object is deformable (or some error), in the case the object is deformable, then estimate the best possibly object position match, from here deformation can also be measured.
Adjust estimated position according to touch information.
Motion equations to the object. Estimate position and speed of object from vision, in every moment of time. Every time there is synchronized information of position, speed, and force. Then estimate final outcome, given some continuation of the current force for some extra time. Or while there is force don't estimate, until there is no force anymore, then estimate.
Inverse problem. Initially specify the action used (to select the right motion controller).
Try with different objects.
Constantly update the object model parameters. Update rule. Kalman filter?
Decidir hasta donde abarcar del systema total en el modelo dynamic (no lineal)
A partir de los diagramas de fase extraer automaticamente una funcion best fit (probando muchas funciones y métodos y escogiendo la funcion con menos parametros pero que se acerque más) que me diga cosas importantes cualitativas de systemas dynamics. Tal vez luego mathematicamente checkear que esas puedan ser posibles soluciones analiticas.