ISSN 2071-8594

Russian academy of sciences


Gennady Osipov

G. S. Osipov , A. I. Panov Rational Behaviour Planning of Cognitive Semiotic Agent in Dynamic Environment


The paper presents a general architecture of a cognitive semiotic agent acting in a dynamic environment. A new implementation and integration of the agent planning and learning subsystems are proposed to solve the symbol grounding problem. We suggest a new approach to the description of the semantic level of the sign-based world model component, which is used as a base for agent rational behavior synthesis. A formal definition of the behavior script and its use in generating a rational agent's action plan is proposed. In conclusion, we describe a model experiment that demonstrates the work of a semiotic agent in a game environment.


semiotic agent, sign-based world model, causal networks, semiotic network, planning, reinforcement learning.

PP. 80-100.

DOI 10.14357/20718594200408


1. Pospelov D.A. Desyat' «gorya hih to hek» v issledovaniyah po iskusstvennomu intellektu // Iskusstvennyj intellekt i prinyatie reshenij. 2019. № 4. P. 3-9.
2. Osipov G.S. Metody iskusstvennogo intellekta. M.: Fizmatlit. 2015. P. 297.
3. Schwarting W., Alonso-Mora J., Rus D. Planning and Decision-Making for Autonomous Vehicles // Annual Review of Control, Robotics, and Autonomous Systems. 2018. Vol. 1, № 1. P. 187–210.
4. Ghallab M., Nau D., Traverso P. Automated Planning and Acting // Automated Planning and Acting. 2016. P. 1–354.
5. Rankooh M. ITSAT: An Efficient SAT-Based Temporal Planner // Journal of Artificial Intelligence Research. 2015. Vol. 53. P.541-632.
6. Richter S., Westphal M. The LAMA planner: Guiding cost-based anytime planning with landmarks // Journal of Artificial Intelligence Research. 2010. Vol 39, pp. 127-177.
7. Alford R., Shivashankar V., Roberts M., Frank J., Aha D. Hi-erarchical planning: Relating task and goal decomposition with task sharing // IJCAI International Joint Conference on Artificial Intelligence. 2016. P.3022-3028.
8. Cardoso R., Bordini R. Decentralised Planning for Multi-Agent Programming Platforms AAMAS 2019. P.799-807.
9. Kiselev G.A., Panov A.I. Sign-based Approach to the Task of Role Distribution in the Coalition of Cognitive Agents // SPIIRAS Pro eedings. 2018. № 57. P. 161–187.
10. Borrajo D., Roubíčková A., Serina I. Progress in Case-Based Planning // ACM Computing Surveys. 2015 vol: 47 (2). P.1-39.
11. G.V. Rybina, YU.M. Blohin Metody i sredstva intellektual'nogo planirovaniya: primenenie dlya upravleniya processami postroeniya integrirovannyh ekspertnyh sistem // Iskusstvennyj intellekt i prinyatie reshenij. 2015. № 1. P.75-93.
12. B Wang Z Kaelbling L Lozano-Pérez T Learning to guide task and motion planning using score-space representationKim International Journal of Robotics Research 2019 vol: 38 (7). P.793-812.
13. Harnad S. Symbol Grounding Problem // Physica. 1990 vol: 42. P.335-346.
14. Besold T., Kuhnberger K. Towards integrated neural-symbolic systems for human-level AI: Two research programs helping to bridge the gaps // Biologically Inspired Cognitive Architectures 2015 vol: 14. P.97-110.
15. Kaelbling L Lozano-Pérez T Integrated task and motion planning in belief space // The International Journal of Robotics Research. 2013 vol: 32 (9-10). P.1194-1227.
16. Tarasov V. Ot mnogoagentnyh sistem k intellektual'nym organizaciyam: filosofiya, psihologiya, informatika // M.: Editorial URSS, 2002. P.352.
17. Karpov V.E., Tarasov V.B. Ot kollaborativnoj robototekhniki k social'nym robotam dlya podderzhki lyudej s ogranichennymi vozmozhnostyami: novye napravleniya razrabotki ispol'zovaniya intellektual'nyh agentov // Intellektual'nye tekhnologii i sredstva reabilitacii lyudej s ogranichennymi vozmozhnostyami (ITSR-2018). Trudy III mezhdunarodnoj konferencii. 2018. P.20-29.
18. Ali Dorri; Salil S. Kanhere ; Raja Jurdak Multi-Agent Systems: A Survey // IEEE Access. 2018. P.28573 – 28593.
19. Snaider J., Franklin S. Vector LIDA // Procedia Computer Science. 2014 vol: 41 pp: 188-203.
20. Leandro Carlos Fernandes etc. CaRINA Intelligent Robotic Car: Architectural Design and Applications // Journal of Systems Architecture 60(4).
21. Goertzel B., Pennachin C., Geisweiller N. (2014) The OpenCog Framework. In: Engineering General Intelligence, Part 2. Atlantis Thinking Machines, vol 6. Atlantis Press, Paris.
22. Laird J. The Soar Cognitive Architecture. MIT Press 2012. P.374.
23. Bothell D . ACT-R 7 Reference Manual. Carnegie Mellon University. 2015. P.516.
24. Hélie S., Sun R. Autonomous learning in psy hologi ally-oriented cognitive architectures: A survey // New Ideas in Psychology. 2014 vol: 34 (1). P.37-55.
25. Samsonovich A. Emotional biologically inspired cognitive architecture // Biologically Inspired Cognitive Architectures. 2013 vol: 6. P.109-125.
26. George D., Hawkins J. Towards a mathematical theory of cortical micro-circuits // PLoS computational biology. 2009 vol: 5 (10). P.1000532.
27. Hawkins J., Ahmad S., Cui Y. A Theory of How Columns in the Neocortex Enable Learning the Structure of the World // Frontiers in Neural Circuits. 2017. Vol. 11. P. 1–18.
28. Dileep George etc. A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs // Science 08 Dec 2017 Vol. 358, Issue 6368, eaag2612.
29. Schmidhuber J. Deep Learning in Neural Networks: An Overview // Neural Networks 2015 vol: 61. P.85-117.
30. Manhaeve R Kimmig A DeepProbLog : Neural Probabilistic Logic Programming arXiv:1805.10872v2.
31. Besold T. Etc Neural-Symbolic Learning and Reasoning: A Survey and Interpretation. 2017. P. 1-58.
32. Ghidini C Serafini L Distributed First Order Logic // Artificial Intelligence. 2017 vol: 253. P.1-39.
33. Schaul T Horgan D Gregor K Silver D. Universal Value Function Approximators // Proceedings of The 32nd International Conference on Machine Learning. 2015. P.1312-1320.
34. Mnih V., Kavukcuoglu K., Silver D., Graves A., Antonoglou I., Wierstra D., Riedmiller M. Playing Atari with Deep Reinforcement Learning arXiv: 1312.5602 2013.
35. Vinyals O., Babuschkin I., zarnecki W. M., Mathieu M., Dudzik A., Chung J., Choi D. H., Powell R., Ewalds T.,Georgiev P., et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning.Nature. 2019.
36. Silver D., Hubert T., Schrittwieser J., Antonoglou I., Lai M., Guez A., Lanctot M., Sifre L., Kumaran D., Graepel T., et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science . 362. 2018.
37. Julian Schrittwieser and etc. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. 2019 ArXiv 1911.08265.
38. Kuznecova Y.U., Osipov G., Panov A., Petrov A., Chudova N. Modelirovanie povedeniya, upravlyaemogo soznaniem // Sistemnyj analiz i informacionnye tekhnologii: tr. CHetvertoj Mezhdunar. konf. (Abzakovo, Rossiya, 17–23 avg. 2011 g.): v 2t. CHelyabinsk: Izd-vo CHelyab. Gos. un–ta. 2011. T. 1. P.6-13.
39. Osipov G.S., et al. Znakovaya kartina mira sub"ekta povedeniya. M.: Fizmatlit, 2018. P. 264.
40. Osipov G.S., Panov A.I., Chudova N. V. Behavior control as a function of consciousness. I. World model and goal setting // Journal of Computer and Systems Sciences International. 2014. Vol. 53. № 4. P. 517–529.
41. CHudova N.V. Konceptual'noe opisanie kartiny mira dlya zadachi modelirovaniya povedeniya, osnovannogo na soznanii // Iskusstvennyj intellekt i prinyatie reshenij. 2012. № 2. P. 51–62.
42. Paraense A Raizer K Gudwin R A machine consciousness approach to urban traffic control // Biologically Inspired Cognitive Architectures. 2016. vol: 15. P.61-73.
43. Madl T., Franklin S., Chen K., Trappl R. A computational cognitive framework ofspatial memory in brains and robots, Cognitive Systems Research. doi:
44. Osipov G. Dinamicheskie intellektual'nye sistemy // Iskusstvennyj intellekt i prinyatie reshenij 2008 (1). P.47-54.
45. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. Proximal Policy Optimization Algorithms 2017. P.1-12.
46. Soft Actor-Critic Algorithms and Applications, Haarnoja et al, 2018.
47. Choi D., Langley P. Evolution of the ICARUS Cognitive Architecture. Cognitive Systems Research.
48. Yi Wu. Learning andplanning with asemanticmodel. 2018. Arxiv 1809.10842.
49. Vin ent ran ois-Lavet Combined Reinforcement Learn-ing via Abstract Representations // The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19).
50. Minsky M. L. Frame-system theory //Thinking. 1977.
51. Pichotta K., Mooney R. J. Learning statistical scripts with LSTM recurrent neural networks //Thirtieth AAAI Conference on Artificial Intelligence. 2016.
52. Donadello I Serafini L Garcez A Logic Tensor Networks for Semantic Image Interpretation // Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI'17). 2017. P.1596-1602.
53. Kleyko D., Rahimi A., Rachkovskij D., Osipov E., Rabaey J. Classification and Recall With Binary Hyperdimensional Computing: Tradeoffs in Choice of Density and Mapping Characteristics // IEEE Transactions on Neural Networks and Learning Systems.. 2018 vol: 29 (12). P.5880-5898.
54. Leont'ev A.N. Deyatel'nost'. Soznanie. Lichnost'. M.: Politizdat, 1977. Vyp. Izd. 2-e. 304 s.
55. Vygotskij L.S. Myshlenie i rech' // Psihologiya razvitiya cheloveka / pod red. S. Bobko. : Eksmo, 2005. P.664–1019.
56. Chudova N.V. Aktual'nye problemy modelirovaniya celepolaganiya v znakovoj kartine mira. Vzglyad psihologa // Iskusstvennyj intellekt i prinyatie reshenij. 2020. № 1. P.70-79.
57. CHudova N.V. Psihologicheskie aspekty planirovaniya v znakovoj kartine mira // SHestnadcataya Nacional'naya konferenciya po iskusstvennomu intellektu s mezhdunarodnym uchastiem KII-2018 Trudy konferencii: v 2-h tomah. 2018. P. 88-95.
58. 1. Emel’yanov S. et al. Multilayer ognitive ar hite ture for UAV control // Cognitive Systems Research. 2016. Vol. 39. P. 58–72.
59. Kiselev G Panov A Hierarchical Psychologically Inspired Planning for Human-Robot Interaction Tasks // Interactive Collaborative Robotics. ICR 2019. Lecture Notes in Computer Science. 2019 vol: 11659. P.150-160.
60. Osipov G.S., Panov A.I., Chudova N. V. Behavior Control as a Function of Consciousness. II. Synthesis of a Behavior Plan // Journal of Computer and Systems Sciences International. 2015. Vol. 54, № 6. P. 882–896.
61. Panov A.I. Behavior Planning of Intelligent Agent with Sign World Model // Biologically Inspired Cognitive Architectures. 2017. Vol. 19. P. 21–31.
62. CHudova N.V., Kuznecova YU.M. Konceptual'naya model' samosoznaniya dlya znakovoj kartiny mira intellektual'nogo agenta // Iskusstvennyj intellekt i prinyatie reshenij. 2018. № 4. P. 86-94.
63. Osipov G.S., Pospelov D.A. Prikladnaya semiotika // Novosti iskusstvennogo intellekta. 1999. № 1. P.9–35.
64. Panov A.I. Formirovanie obraznoj komponenty znanij kognitivnogo agenta so znakovoj kartinoj mira // Informacionnye tekhnologii i vychislitel'nye sistemy. 2018. № 4. P. 84–96.
65. Osipov G.S. Sign-based representation and word model of actor // 2016 IEEE 8th International Conference on Intelligent Systems (IS) / ed. Yager R. et al. IEEE, 2016. P. 22–26.
66. Osipov G.S. Signs-Based vs. Symbolic Models // Advanc-es in Artificial Intelligence and Soft Computing / ed. Sidorov G., Galicia-Haro S.N. Springer International Publishing. 2015. P. 3–11.
67. Osipov G.S., Panov A.I. Relationships and Operations in a Sign-Based World Model of the Actor // Scientific and Techni al Information Pro essing. 2018. Vol. 45, № 5. P. 317–330.
68. George D. How the Brain Might Work: a Hierarchical and Temporal Model for Learning and Recognition// PhD Stanford University. 2008 (June).
69. Hengst B Hierarchical Approaches // Reinforcement Learning. 2012. P.293-323.
70. Levy A Platt R Saenko K Hierarchical Actor-Critic. 2018.
71. Pierre-Luc Bacon and Jean Harb and Doina Precup The Option-Critic Architecture.
72. Suvorova M.I., Kobozeva M.V., Sokolova E.G., Toldova S.YU.Izvlechenie scenarnoj informacii iz tekstov. CHast' 1. Postanovka zadachi i obzor metodov // Iskusstvennyj intellekt i prinyatie reshenij. 2020. № 1. P. 17-26.
73. Zolotova G. A., Onipenko N. K., Sidorova M. YU. Kommunikativnaya grammatika russkogo yazyka. M.: Institut russkogo yazyka im. V. V. Vinogradova RAN. 2004.
74. Gorodetskiy A., Shlychkova A., Panov A.I. Delta Schema Network in Model-based Reinforcement Learning // Artificial General Intelligence. AGI 2020. Lecture Notes in Computer Science / ed. Goertzel B. et al. Springer. 2020.
75. Albus, J. S. and Barbera, A. J. RCS: A cognitive architec-ture for intelligent multi-agent systems. Annual Reviews in Control 29 (1). 2005). P.87-99.
76. edunov B.E. «Elektronnyj let hik»: «to hka nevozvrata projdena ne budet». Bortovye operativno sovetuyushchie ekspertnye sistemy takticheskogo urovnya dlya pilotiruemyh letatel'nyh apparatov // Aviapanorama: Mezhdunarodnyj aviacionno-kosmicheskij zhurnal. 2016. № 1. P. 9.
77. Fedunov B.E. Intellektual'nye agenty v bazah znanij bortovyh operativno sovetuyushchih ekspertnyh sistemah tipovyh situacij funkcionirovaniya antropocentricheskogo ob"ekta // Izvestiya Rossijskoj akademii nauk. Teoriya i sistemy upravleniya. 2019. № 6. P. 90-102.