> ## Documentation Index
> Fetch the complete documentation index at: https://docs.loadforge.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawling Real HTTP User

> Index a site and crawl all the pages with a random delay as a real user would.

## Overview

This documentation page provides an example of how to simulate a real user's browsing behavior. It demonstrates a test that replicates user actions on the [Locust documentation site](https://docs.locust.io/en/latest/). The test identifies all links in the site's table of contents and then accesses these links in a randomized manner, mimicking a genuine user's experience.

## Key Features

* **Real User Simulation**: This test accesses links with a random delay between 10 to 600 seconds, accurately simulating user load.
* **Versatility**: A prime example for those wanting to recreate real user actions on different websites.
* **Driven by Locust**: The power of locust.io backs LoadForge, an esteemed open-source load testing tool.

<Note title="Tried the Wizard?">
  LoadForge offers a comprehensive test generation wizard. This tool can automatically discover content on your domain and incorporate it into your test file, streamlining the process.
</Note>

## Test Script

Outlined below is the Python script underpinning this test:

```python theme={null}
import random
from locust import HttpUser, between, task
from pyquery import PyQuery

class AwesomeUser(HttpUser):
    # Simulated user behavior indicates a longer wait time (from 10 to 600 seconds) 
    # between pages due to the considerable content present on each page.
    wait_time = between(10, 600)

    def on_start(self):
        # Initial delay ensures users don't access the site all at once.
        self.wait()
        # The starting point for all users is assumed to be the index page.
        self.index_page()
        self.urls_on_current_page = self.toc_urls

    @task(10)
    def index_page(self):
        r = self.client.get("/")
        pq = PyQuery(r.content)
        link_elements = pq(".toctree-wrapper a.internal")
        self.toc_urls = [l.attrib["href"] for l in link_elements]

    @task(50)
    def load_page(self):
        url = random.choice(self.toc_urls)
        r = self.client.get(url)
        pq = PyQuery(r.content)
        link_elements = pq("a.internal")
        self.urls_on_current_page = [l.attrib["href"] for l in link_elements]

    @task(30)
    def load_sub_page(self):
        url = random.choice(self.urls_on_current_page)
        r = self.client.get(url)
```

<Note title="Locust Test Example">
  LoadForge operates on locust.io's foundation. Thus, if you're a current user of locust, this script is fully compatible. To amplify your testing strategies, consider importing your scripts into [LoadForge](https://loadforge.com).
</Note>

## Conclusion

LoadForge equips both novices and experts in load testing with an array of tools and examples, optimizing the efficacy of your tests. The real HTTP user crawling instance is a testament to the myriad ways you can gauge and hone the performance of your online platforms.