where are the robots is a Web Exploitation puzzle worth 100 points.

Description

Can you find the robots? https://jupiter.challenges.picoctf.org/problem/56830/ (link) or http://jupiter.challenges.picoctf.org:56830

Solution

I figured that this had to do with robots.txt.

First, I just tacked robots.txt to the end of the provided URL, which presented a robots.txt file:

User-agent: *
Disallow: /1bb4c.html

Next, I looked at 1bb4c.html, because this is a strange and non-standard file. The flag was in there: