Request history endpoints

If you want to access the request history in your codebase, you can use the api endpoints that we provide in your codebase. There are 2 api's for accessing the request history, one is for getting all of your request history and another is for getting the response data of a request history.

Getting all of your request history

To get all of your request history, it is very simple. First you have to get your api key, to get your api key you just have to login to your account and you should see your api key in the builder page.

Once you have your api key, you can use your api key to send an request to the api. The api route is a get request, you should send your api key in the url query like this:

https://api.scrapeautomate.com/requests?apiKey=1cbf2a99-ef94-4255-aae9-fdbfdcff3b53

Now if you were to hit that url using an get request using curl, axios or anything you would like you should see an array of requests inside the response data. Here is an example response:

{
  "requests": [
    {
      "id": "cm4twcdii0002f9a6h9xr9mmi",
      "userId": "cm3f9vz6k0000l5b2iemru05l",
      "statusCode": "200",
      "message": "Request has been sent to http://localhost:8002/api/scraper?apiKey=1cbf2a99-ef94-4255-aae9-fdbfdcff3b53&render=true&url=https:%2F%2Fexample.com%2F&url=https://example.com/",
      "url": "http://localhost:8002/api/scraper?apiKey=1cbf2a99-ef94-4255-aae9-fdbfdcff3b53&render=true&url=https:%2F%2Fexample.com%2F&url=https://example.com/",
      "creditCost": 10,
      "createdAt": "2024-12-18T12:56:27.617Z",
      "headers": {
        "sa-age": "482739",
        "sa-date": "Wed, 18 Dec 2024 12:56:25 GMT",
        "sa-etag": "\"3147526947\"",
        "sa-vary": "Accept-Encoding",
        "sa-server": "ECAcc (sed/58AC)",
        "sa-expires": "Wed, 25 Dec 2024 12:56:25 GMT",
        "sa-x-cache": "HIT",
        "Content-Type": "text/html",
        "SA-Final-URL": "https://example.com/",
        "SA-Request-Cost": "10",
        "sa-content-type": "text/html; charset=UTF-8",
        "sa-accept-ranges": "bytes",
        "sa-cache-control": "max-age=604800",
        "sa-last-modified": "Thu, 17 Oct 2019 07:18:26 GMT",
        "sa-content-length": "648",
        "sa-content-encoding": "gzip",
        "SA-Remaining-Credits": "283",
        "sa-scraping-duration": "3947.904045"
      },
      "file_id": "e4da10e20953829e023b41b362d997729d105c55ef6b38e463dba97df64ad040_cm4twcdii0002f9a6h9xr9mmi",
      "workflow_id": null,
      "file_type": "html",
      "scraping_duration": 3947.904045
    }
  ],
  "pagination": {
    "currentPage": 1,
    "totalPages": 1,
    "totalCount": 4,
    "limit": 10
  }
}

If you notice there is some pagination, if you want to get the data from the next page add an page property to the query parameters and it will take you to the next page of data like this:

https://api.scrapeautomate.com/requests?apiKey=1cbf2a99-ef94-4255-aae9-fdbfdcff3b53&page=2

The above will take you to the next page of the requests, if you want to increase the limit of the data thats being returned per page you can set the limit property like this:

https://api.scrapeautomate.com/requests?apiKey=1cbf2a99-ef94-4255-aae9-fdbfdcff3b53&page=1&limit=100

This will make it so it returns 100 requests per page rather the only 10!

Getting the response data

Now that you can get all of the requests, you might be wondering how can I get the response data of a request? Its very simple, if you notice every single request from the all requests api has an "id" property, that id property allows you to get the history data very easily. To get the request history data you have to send an get request to this route:

https://api.scrapeautomate.com/requests/${requestId}?apiKey=1cbf2a99-ef94-4255-aae9-fdbfdcff3b53

You will need to replace the request id variable with the request id of the response you want. One thing to note is that the apiKey here is also required, to get the api key you just have to login to your account and you should see your api key in the builder page!

When you send an request to that api, that api might respond with many different types of content-type it could be an html, it could be an image, it could be even a markdown!

Here is an example response:

<!doctype html>
<html>
  <head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <style type="text/css">
      body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont, 'Segoe UI',
          'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;
      }
      div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0, 0, 0, 0.02);
      }
      a:link,
      a:visited {
        color: #38488f;
        text-decoration: none;
      }
      @media (max-width: 700px) {
        div {
          margin: 0 auto;
          width: auto;
        }
      }
    </style>
  </head>

  <body>
    <div>
      <h1>Example Domain</h1>
      <p>
        This domain is for use in illustrative examples in documents. You may
        use this domain in literature without prior coordination or asking for
        permission.
      </p>
      <p>
        <a href="https://www.iana.org/domains/example">More information...</a>
      </p>
    </div>
  </body>
</html>

Thats basically it! These 2 api's allow you to easily manage and access the request history!

Getting all of your request history​

Getting the response data​

Getting all of your request history

Getting the response data