In my site design they explained that Google, Yelp and reputable sites will send out web crawlers (Known as Robots or Bots) index your web site for their search engines so you can be more easily found.
Now there are harmful Robots (bots) also. But if you block the big search engine robots your presence on them will be limited or minimized. So you need to be careful in blocking ALL ROBOTS.

So yes if you've been in IT (web related) you'd know this.

ie.
Google has many types of robots, including web crawlers, chatbots, and other robotics projects:
Googlebot
A web crawler that collects documents from the web to create a searchable index for Google Search. Googlebot has two types of crawlers: Googlebot Smartphone, which simulates a mobile user, and Googlebot Desktop, which simulates a desktop user. Google has 16 different bots for crawling and site rendering.

Facebook,Bing and Microsoft also use bots to create these searchable indexes.

They are NOT ALL BAD.