cf

Dockerfile

@@ -0,0 +1,36 @@
+FROM node:latest
+
+# Install necessary packages including Chromium and XVFB
+RUN apt-get update && apt-get install -y \
+    wget \
+    gnupg \
+    ca-certificates \
+    apt-transport-https \
+    chromium \
+    chromium-driver \
+    xvfb \
+    && rm -rf /var/lib/apt/lists/*
+
+# Set the environment variable for Chromium binary
+ENV CHROME_BIN=/usr/bin/chromium
+
+# Set the working directory inside the container
+WORKDIR /app
+
+# Copy package.json and package-lock.json (if exists) to leverage Docker cache
+COPY package*.json ./
+
+# Install Node.js dependencies
+# Using `npm ci` is often preferred in Docker for reproducible builds if you have package-lock.json
+# Otherwise, `npm install` is fine. `npm update` is generally not needed here.
+RUN npm install
+
+# Copy the rest of your application code into the container
+COPY . .
+
+# Expose the port your Node.js application listens on (7860 as per your app.js)
+EXPOSE 7860
+
+# Command to run your Node.js application
+# This will keep the container running as long as app.js is active
+CMD ["node", "app.js"]

LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 - 2024 @zfcsoftware
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,199 @@
+---
+license: mit
+title: cf-token
+sdk: docker
+emoji: 😻
+colorFrom: indigo
+colorTo: gray
+short_description: cf-token
+---
+
+[![ScrapeDo](src/data/sdo.gif)](https://scrape.do/?utm_source=github&utm_medium=repo_ccs)
+
+> [!WARNING]
+> This repo will no longer receive updates. Thank you to everyone who supported it.
+
+# CF Clearance Scraper
+
+This library was created for testing and training purposes to retrieve the page source of websites, create Cloudflare Turnstile tokens and create Cloudflare WAF sessions.
+
+Cloudflare protection not only checks cookies in the request. It also checks variables in the header. For this reason, it is recommended to use it with the sample code in this readme file.
+
+Cookies with cf in the name belong to Cloudflare. You can find out what these cookies do and how long they are valid by **[Clicking Here](https://developers.cloudflare.com/fundamentals/reference/policies-compliances/cloudflare-cookies/)**.
+
+## Sponsor
+
+[![Capsolver](src/data/capsolver.png)](https://www.capsolver.com/?utm_source=github&utm_medium=repo&utm_campaign=scraping&utm_term=cf-clearance-scraper)
+
+## Installation
+
+Installation with Docker is recommended.
+
+**Docker**
+
+Please make sure you have installed the latest image. If you get an error, try downloading the latest version by going to Docker Hub.
+
+```bash
+sudo docker rmi zfcsoftware/cf-clearance-scraper:latest --force
+```
+
+```bash
+docker run -d -p 3000:3000 \
+-e PORT=3000 \
+-e browserLimit=20 \
+-e timeOut=60000 \
+zfcsoftware/cf-clearance-scraper:latest
+```
+
+**Github**
+
+```bash
+git clone https://github.com/zfcsoftware/cf-clearance-scraper
+cd cf-clearance-scraper
+npm install
+npm run start
+```
+
+## Create Cloudflare WAF Session
+
+By creating a session as in the example, you can send multiple requests to the same site without being blocked. Since sites may have TLS protection, it is recommended to send requests with the library in the example.
+
+```js
+const initCycleTLS = require('cycletls');
+async function test() {
+    const session = await fetch('http://localhost:3000/cf-clearance-scraper', {
+        method: 'POST',
+        headers: {
+            'Content-Type': 'application/json'
+        },
+        body: JSON.stringify({
+            url: 'https://nopecha.com/demo/cloudflare',
+            mode: "waf-session",
+            // proxy:{
+            //     host: '127.0.0.1',
+            //     port: 3000,
+            //     username: 'username',
+            //     password: 'password'
+            // }
+        })
+    }).then(res => res.json()).catch(err => { console.error(err); return null });
+
+    if (!session || session.code != 200) return console.error(session);
+
+    const cycleTLS = await initCycleTLS();
+    const response = await cycleTLS('https://nopecha.com/demo/cloudflare', {
+        body: '',
+        ja3: '772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,23-27-65037-43-51-45-16-11-13-17513-5-18-65281-0-10-35,25497-29-23-24,0', // https://scrapfly.io/web-scraping-tools/ja3-fingerprint
+        userAgent: session.headers["user-agent"],
+        // proxy: 'http://username:password@hostname.com:443',
+        headers: {
+            ...session.headers,
+            cookie: session.cookies.map(cookie => `${cookie.name}=${cookie.value}`).join('; ')
+        }
+    }, 'get');
+
+    console.log(response.status);
+    cycleTLS.exit().catch(err => { });
+}
+test()
+```
+
+## Create Turnstile Token with Little Resource Consumption
+
+This endpoint allows you to generate tokens for a Cloudflare Turnstile Captcha. It blocks the request that fetches the page resource and instead makes the page resource a simple Turnstile render page. This allows you to generate tokens without having to load any additional css or js files. 
+
+However, in this method, the siteKey variable must be sent to Turnstile along with the site to create the token. If this does not work, you can examine the token generation system by loading the full page resource described in the next section.
+
+```js
+fetch('http://localhost:3000/cf-clearance-scraper', {
+    method: 'POST',
+    headers: {
+        'Content-Type': 'application/json'
+    },
+    body: JSON.stringify({
+        url: 'https://turnstile.zeroclover.io/',
+        siteKey: "0x4AAAAAAAEwzhD6pyKkgXC0",
+        mode: "turnstile-min",
+        // proxy:{
+        //     host: '127.0.0.1',
+        //     port: 3000,
+        //     username: 'username',
+        //     password: 'password'
+        // }
+    })
+})
+    .then(res => res.json())
+    .then(console.log)
+    .catch(console.log);
+```
+
+## Creating Turnstile Token with Full Page Load
+
+This example request goes to the page at the given url address with a real browser, resolves the Turnstile and returns you the token.
+
+```js
+fetch('http://localhost:3000/cf-clearance-scraper', {
+    method: 'POST',
+    headers: {
+        'Content-Type': 'application/json'
+    },
+    body: JSON.stringify({
+        url: 'https://turnstile.zeroclover.io/',
+        mode: "turnstile-max",
+        // proxy:{
+        //     host: '127.0.0.1',
+        //     port: 3000,
+        //     username: 'username',
+        //     password: 'password'
+        // }
+    })
+})
+    .then(res => res.json())
+    .then(console.log)
+    .catch(console.log);
+```
+
+## Getting Page Source from a Site Protected with Cloudflare WAF
+
+With this request you can scrape the page source of a website protected with CF WAF.
+
+```js
+fetch('http://localhost:3000/cf-clearance-scraper', {
+    method: 'POST',
+    headers: {
+        'Content-Type': 'application/json'
+    },
+    body: JSON.stringify({
+        url: 'https://nopecha.com/demo/cloudflare',
+        mode: "source"
+        // proxy:{
+        //     host: '127.0.0.1',
+        //     port: 3000,
+        //     username: 'username',
+        //     password: 'password'
+        // }
+    })
+})
+    .then(res => res.json())
+    .then(console.log)
+    .catch(console.log);
+```
+
+## Quick Questions and Answers
+
+### Does It Open A New Browser On Every Request?
+No, a new context is started with each request and closed when the job is finished. Processes are executed with isolated contexts through a single browser.
+
+### How Do I Limit the Browser Context to Open?
+You can do this by changing the process.env.browserLimit value. The default is 20
+
+### How Do I Add Authentication to Api?
+You can add authorisation by changing the process.env.authToken variable. If this variable is added, it returns 401 if the authToken variable in the request body is not equal to the token you specify.
+
+### How Do I Set The Timeout Time?
+You can give the variable process.env.timeOut a value in milliseconds. The default is 60000.
+
+## Disclaimer of Liability
+This repository was created purely for testing and training purposes. The user is responsible for any prohibited liability that may arise from its use.
+The library is not intended to harm any site or company. The user is responsible for any damage that may arise. 
+Users of this repository are deemed to have accepted this disclaimer. 

app.js

@@ -0,0 +1,111 @@
+const express = require("express");
+const app = express();
+const bodyParser = require("body-parser");
+const cors = require("cors");
+const reqValidate = require("./src/module/reqValidate");
+
+const port = 7860;
+const authToken = null;
+global.browserLimit = 20;
+global.timeOut = 60000;
+
+global.browserLength = 0;
+// Add a new global variable to hold the browser instance directly
+global.browser = null;
+
+app.use(bodyParser.json());
+app.use(bodyParser.urlencoded({ extended: true }));
+app.use(cors());
+
+const server = app.listen(port, () => {
+  console.log(`Server running on port ${port}`);
+});
+
+if (server) {
+  try {
+    server.timeout = global.timeOut;
+    server.keepAliveTimeout = global.timeOut + 5000;
+    server.headersTimeout = global.timeOut + 6000;
+  } catch (e) {
+    console.error("Failed to set server timeouts:", e);
+  }
+}
+
+// Require createBrowser, but don't immediately call it here.
+// We'll manage its lifecycle from within the browser creation module itself.
+require("./src/module/createBrowser");
+
+const getSource = require("./src/endpoints/getSource");
+const solveTurnstileMin = require("./src/endpoints/solveTurnstile.min");
+const solveTurnstileMax = require("./src/endpoints/solveTurnstile.max");
+const wafSession = require("./src/endpoints/wafSession");
+
+async function handleRequest(fn, data) {
+  try {
+    return { ...(await fn(data)), code: 200 };
+  } catch (err) {
+    return { code: 500, message: String(err) };
+  }
+}
+
+app.post("/cf-clearance-scraper", async (req, res) => {
+  try {
+    const data = req.body;
+    const check = reqValidate(data);
+
+    if (check !== true) {
+      return res.status(400).json({ code: 400, message: "Bad Request", schema: check });
+    }
+
+    if (authToken && data.authToken !== authToken) {
+      return res.status(401).json({ code: 401, message: "Unauthorized" });
+    }
+
+    if (global.browserLength >= global.browserLimit) {
+      return res.status(429).json({ code: 429, message: "Too Many Requests" });
+    }
+
+    // Use global.browser instead of global.browser
+    if (!global.browser) {
+      return res.status(500).json({
+        code: 500,
+        message: "The scanner is not ready yet. Please try again a little later.",
+      });
+    }
+
+    global.browserLength++;
+
+    let result = { code: 500 };
+    try {
+      // Pass the active browser instance to the handler functions
+      const requestData = { ...data, browser: global.browser };
+      switch (data.mode) {
+        case "source":
+          result = await handleRequest(getSource, requestData);
+          break;
+        case "turnstile-min":
+          result = await handleRequest(solveTurnstileMin, requestData);
+          break;
+        case "turnstile-max":
+          result = await handleRequest(solveTurnstileMax, requestData);
+          break;
+        case "waf-session":
+          result = await handleRequest(wafSession, requestData);
+          break;
+        default:
+          result = { code: 400, message: "Invalid mode" };
+      }
+    } finally {
+      global.browserLength--;
+    }
+
+    res.status(result.code ?? 500).json(result);
+  } catch (e) {
+    console.error("Error in /cf-clearance-scraper:", e);
+    res.status(500).json({ code: 500, message: "Internal Server Error" });
+  }
+});
+
+app.use((req, res) => {
+  res.status(404).json({ code: 404, message: "Not Found" });
+});

package.json

@@ -0,0 +1,40 @@
+{
+  "name": "cf-clearance-scraper",
+  "version": "2.1.3",
+  "main": "app.js",
+  "scripts": {
+    "start": "node app.js",
+    "test": "node --experimental-vm-modules ./node_modules/.bin/jest --detectOpenHandles --verbose"
+  },
+  "jest": {
+    "testMatch": [
+      "**/tests/**/*.js"
+    ],
+    "verbose": true
+  },
+  "keywords": [
+    "cf-clearance",
+    "cloudflare",
+    "waf",
+    "scraper",
+    "puppeteer",
+    "xvfb",
+    "turnstile",
+    "bypass",
+    "undetected",
+    "stealth"
+  ],
+  "author": "zfcsoftware",
+  "license": "ISC",
+  "description": "This package is an experimental and educational package created for Cloudflare protections.",
+  "dependencies": {
+    "ajv": "^8.17.1",
+    "ajv-formats": "^3.0.1",
+    "body-parser": "^1.20.3",
+    "cors": "^2.8.5",
+    "express": "^4.21.0",
+    "jest": "^29.7.0",
+    "puppeteer-real-browser": "^1.4.0",
+    "supertest": "^7.0.0"
+  }
+}

src/data/fakePage.html

@@ -0,0 +1,30 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title></title>
+</head>
+
+<body>
+    <div class="turnstile"></div>
+    <script src="https://challenges.cloudflare.com/turnstile/v0/api.js?onload=onloadTurnstileCallback" defer></script>
+    <script>
+        window.onloadTurnstileCallback = function () {
+            turnstile.render('.turnstile', {
+                sitekey: '<site-key>',
+                callback: function (token) {
+                    var c = document.createElement('input');
+                    c.type = 'hidden';
+                    c.name = 'cf-response';
+                    c.value = token;
+                    document.body.appendChild(c);
+                },
+            });
+        };
+
+    </script>
+</body>
+
+</html>

src/endpoints/getSource.js

@@ -0,0 +1,45 @@
+async function getSource({ url, proxy }) {
+  return new Promise(async (resolve, reject) => {
+    if (!url) return reject("Missing url parameter");
+    let isResolved = false;
+    
+    const context = await global.browser.createBrowserContext().catch(() => null);
+    if (!context) return reject("Failed to create browser context");
+
+    const page = await context.newPage();
+
+    try {
+      await page.setRequestInterception(true);
+      page.on("request", (request) => {
+        try {
+          request.continue();
+        } catch (err) {
+          console.warn("Request interception error:", err);
+        }
+      });
+
+      page.on("response", async (res) => {
+        try {
+          if ([200, 302].includes(res.status()) && [url, url + "/"].includes(res.url())) {
+            await page.waitForNavigation({ waitUntil: "load", timeout: 5000 }).catch(() => {});
+            const html = await page.content();
+            isResolved = true;
+            await page.close();
+            resolve({ source: html });
+          }
+        } catch (e) {
+          console.error("Error processing response:", e);
+        }
+      });
+      await page.goto(url, { waitUntil: "domcontentloaded" });
+    } catch (e) {
+      console.error("Error in getSource:", e.message);
+      if (!isResolved) {
+        await page.close();
+        reject(e.message);
+      }
+    }
+  });
+}
+
+module.exports = getSource;

src/endpoints/solveTurnstile.max.js

@@ -0,0 +1,80 @@
+const fs = require("fs");
+function solveTurnstileMin({ url, proxy }) {
+  return new Promise(async (resolve, reject) => {
+    if (!url) return reject("Missing url parameter");
+
+    const context = await global.browser
+      .createBrowserContext({
+        proxyServer: proxy ? `http://${proxy.host}:${proxy.port}` : undefined, // https://pptr.dev/api/puppeteer.browsercontextoptions
+      })
+      .catch(() => null);
+
+    if (!context) return reject("Failed to create browser context");
+
+    let isResolved = false;
+
+    var cl = setTimeout(async () => {
+      if (!isResolved) {
+        await context.close();
+        reject("Timeout Error");
+      }
+    }, global.timeOut || 60000);
+
+    try {
+      const page = await context.newPage();
+
+      if (proxy?.username && proxy?.password)
+        await page.authenticate({
+          username: proxy.username,
+          password: proxy.password,
+        });
+        
+      await page.evaluateOnNewDocument(() => {
+        let token = null;
+        async function waitForToken() {
+          while (!token) {
+            try {
+              token = window.turnstile.getResponse();
+            } catch (e) {}
+            await new Promise((resolve) => setTimeout(resolve, 500));
+          }
+          var c = document.createElement("input");
+          c.type = "hidden";
+          c.name = "cf-response";
+          c.value = token;
+          document.body.appendChild(c);
+        }
+        waitForToken();
+      });
+
+      await page.goto(url, {
+        waitUntil: "domcontentloaded",
+      });
+
+      await page.waitForSelector('[name="cf-response"]', {
+        timeout: 60000,
+      });
+      const token = await page.evaluate(() => {
+        try {
+          return document.querySelector('[name="cf-response"]').value;
+        } catch (e) {
+          return null;
+        }
+      });
+      isResolved = true;
+      clearInterval(cl);
+      await context.close();
+      if (!token || token.length < 10) return reject("Failed to get token");
+      return resolve(token);
+    } catch (e) {
+      console.log(e);
+
+      if (!isResolved) {
+        await context.close();
+        clearInterval(cl);
+        reject(e.message);
+      }
+    }
+  });
+}
+module.exports = solveTurnstileMin;

src/endpoints/solveTurnstile.min.js

@@ -0,0 +1,67 @@
+async function solveTurnstileMin({ url, proxy, siteKey }) {
+  return new Promise(async (resolve, reject) => {
+    if (!url) return reject("Missing url parameter");
+    if (!siteKey) throw new Error("Missing siteKey parameter");
+    let isResolved = false;
+
+    const context = await global.browser.createBrowserContext().catch(() => null);
+    if (!context) return reject("Failed to create browser context");
+
+    const page = await context.newPage();
+    
+    try {
+      if (proxy?.username && proxy?.password) {
+        await page.authenticate({
+          username: proxy.username,
+          password: proxy.password,
+        });
+      }
+      await page.setRequestInterception(true);
+      page.on("request", async (request) => {
+        if (
+          [url, url + "/"].includes(request.url()) &&
+          request.resourceType() === "document"
+        ) {
+          const response = await request.respond({
+            status: 200,
+            contentType: "text/html",
+            body: String(
+              require("fs").readFileSync("./src/data/fakePage.html")
+            ).replace(/<site-key>/g, siteKey),
+          });
+        } else {
+          await request.continue();
+        }
+      });
+      await page.goto(url, {
+        waitUntil: "domcontentloaded",
+      });
+
+      await page.waitForSelector('[name="cf-response"]', {
+        timeout: 60000,
+      });
+
+      const token = await page.evaluate(() => {
+        try {
+          return document.querySelector('[name="cf-response"]').value;
+        } catch (e) {
+          return null;
+        }
+      });
+      
+      isResolved = true;
+      await page.close();
+
+      if (!token || token.length < 10) throw new Error("Failed to get token");
+      resolve({ token }); 
+    } catch (e) {
+      console.error("Error in getSource:", e.message);
+      if (!isResolved) {
+        await page.close();
+        reject(e.message);
+      }
+    }
+  });
+}
+
+module.exports = solveTurnstileMin;

src/endpoints/wafSession.js

@@ -0,0 +1,80 @@
+async function findAcceptLanguage(page) {
+  return await page.evaluate(async () => {
+    const result = await fetch("https://httpbin.org/get")
+      .then((res) => res.json())
+      .then(
+        (res) =>
+          res.headers["Accept-Language"] || res.headers["accept-language"]
+      )
+      .catch(() => null);
+    return result;
+  });
+}
+
+function getSource({ url, proxy }) {
+  return new Promise(async (resolve, reject) => {
+    if (!url) return reject("Missing url parameter");
+    const context = await global.browser
+      .createBrowserContext({
+        proxyServer: proxy ? `http://${proxy.host}:${proxy.port}` : undefined, // https://pptr.dev/api/puppeteer.browsercontextoptions
+      })
+      .catch(() => null);
+    if (!context) return reject("Failed to create browser context");
+
+    let isResolved = false;
+
+    var cl = setTimeout(async () => {
+      if (!isResolved) {
+        await context.close();
+        reject("Timeout Error");
+      }
+    }, global.timeOut || 60000);
+
+    try {
+      const page = await context.newPage();
+
+      if (proxy?.username && proxy?.password)
+        await page.authenticate({
+          username: proxy.username,
+          password: proxy.password,
+        });
+      let acceptLanguage = await findAcceptLanguage(page);
+      await page.setRequestInterception(true);
+      page.on("request", async (request) => request.continue());
+      page.on("response", async (res) => {
+        try {
+          if (
+            [200, 302].includes(res.status()) &&
+            [url, url + "/"].includes(res.url())
+          ) {
+            await page
+              .waitForNavigation({ waitUntil: "load", timeout: 5000 })
+              .catch(() => {});
+            const cookies = await page.cookies();
+            let headers = await res.request().headers();
+            delete headers["content-type"];
+            delete headers["accept-encoding"];
+            delete headers["accept"];
+            delete headers["content-length"];
+            headers["accept-language"] = acceptLanguage;
+            await context.close();
+            isResolved = true;
+            clearInterval(cl);
+            resolve({ cookies, headers });
+          }
+        } catch (e) {}
+      });
+
+      await page.goto(url, {
+        waitUntil: "domcontentloaded",
+      });
+    } catch (e) {
+      if (!isResolved) {
+        await context.close();
+        clearInterval(cl);
+        reject(e.message);
+      }
+    }
+  });
+}
+module.exports = getSource;

src/module/createBrowser.js

@@ -0,0 +1,89 @@
+// const puppeteer = require("puppeteer-core"); // Uncomment if using puppeteer-core
+const { connect } = require("puppeteer-real-browser");
+
+async function createBrowser(retry = 0) {
+  try {
+    // If a browser instance already exists, or if we're finished, return.
+    // global.finished should be set to true when the application is shutting down.
+    if (global.finished) return;
+
+    // If global.browser is not null, it means we already have a connected browser.
+    // This prevents launching multiple browsers if one is already active.
+    if (global.browser) {
+        console.log("Browser is already connected.");
+        return;
+    }
+
+    console.log("Launching new browser instance...");
+
+    /* If using puppeteer-core:
+    const browser = await puppeteer.launch({
+      headless: false,
+      args: ["--no-sandbox", "--disable-gpu"],
+    });
+    */
+
+    const { browser: _browser } = await connect({
+      headless: false,
+      turnstile: true,
+      connectOption: { defaultViewport: null },
+      disableXvfb: false,
+    });
+
+    // Store the new browser instance globally
+    global.browser = _browser;
+
+    // Set up a listener for when the browser disconnects
+    browser.on("disconnected", async () => {
+      if (global.finished) return; // If the app is shutting down, don't try to restart
+      console.log("Browser disconnected. Attempting to restart browser...");
+      global.browser = null; // Clear the disconnected instance
+      // Recursively call createBrowser to launch a new instance
+      await createBrowser();
+    });
+
+    console.log("Browser launched successfully and ready for use.");
+  } catch (e) {
+    console.error("Error launching browser:", e.stack);
+
+    // Only retry if the application is not finished and retry attempts are within limit
+    if (global.finished || retry >= 5) {
+      console.error("Max retries reached or application is shutting down. Could not launch browser.");
+      global.browser = null; // Ensure the instance is null if launch failed
+      return;
+    }
+
+    console.log(`Retrying browser launch (${retry + 1}/5)...`);
+    await new Promise((resolve) => setTimeout(resolve, 3000));
+    await createBrowser(retry + 1);
+  }
+}
+
+// Immediately call createBrowser when this module is required to ensure a browser is available from start
+createBrowser();
+
+// Optionally, export the createBrowser function if you need to manually trigger it from elsewhere
+module.exports = createBrowser;
+
+// Add a graceful shutdown handler
+process.on('SIGINT', async () => {
+    console.log('SIGINT signal received. Shutting down...');
+    global.finished = true; // Signal that the application is shutting down
+
+    if (global.browser) {
+        console.log('Closing browser...');
+        await global.browser.close().catch(e => console.error("Error closing browser:", e));
+    }
+    process.exit(0);
+});
+
+process.on('SIGTERM', async () => {
+    console.log('SIGTERM signal received. Shutting down...');
+    global.finished = true; // Signal that the application is shutting down
+
+    if (global.browser) {
+        console.log('Closing browser...');
+        await global.browser.close().catch(e => console.error("Error closing browser:", e));
+    }
+    process.exit(0);
+});

src/module/reqValidate.js

@@ -0,0 +1,58 @@
+const Ajv = require("ajv")
+const addFormats = require("ajv-formats")
+
+const ajv = new Ajv()
+addFormats(ajv)
+
+const schema = {
+    "type": "object",
+    "properties": {
+        "mode": {
+            "type": "string",
+            "enum": ["source", "turnstile-min", "turnstile-max", "waf-session"],
+        },
+        "proxy": {
+            "type": "object",
+            "properties": {
+                "host": { "type": "string" },
+                "port": { "type": "integer" },
+                "username": { "type": "string" },
+                "password": { "type": "string" }
+            },
+            "additionalProperties": false
+        },
+        "url": {
+            "type": "string",
+            "format": "uri",
+        },
+        "authToken": {
+            "type": "string"
+        },
+        "siteKey": {
+            "type": "string"
+        }
+    },
+    "required": ["mode", "url"],
+    "additionalProperties": false
+}
+
+// const data = {
+//     mode: "source",
+//     url: "https://example.com",
+//     proxy: {
+//         host: "localhost",
+//         port: 8080,
+//         username: "test",
+//         password: "test"
+//     },
+//     authToken: "123456"
+// }
+
+
+function validate(data) {
+    const valid = ajv.validate(schema, data)
+    if (!valid) return ajv.errors
+    else return true
+}
+
+module.exports = validate

tests/endpoints.test.js

@@ -0,0 +1,62 @@
+process.env.NODE_ENV = 'development'
+const server = require('../src/index')
+const request = require("supertest")
+
+beforeAll(async () => {
+    while (!global.browser) {
+        await new Promise(resolve => setTimeout(resolve, 1000));
+    }
+}, 30000);
+
+
+afterAll(async () => {
+    global.finished = true
+    await global.browser.close()
+})
+
+
+test('Scraping Page Source from Cloudflare Protection', async () => {
+    return request(server)
+        .post("/cf-clearance-scraper")
+        .send({
+            url: 'https://nopecha.com/demo/cloudflare',
+            mode: "source"
+        })
+        .expect(200)
+        .then(response => { expect(response.body.code).toEqual(200); })
+}, 60000)
+
+
+test('Creating a Turnstile Token With Site Key [min]', async () => {
+    return request(server)
+        .post("/cf-clearance-scraper")
+        .send({
+            url: 'https://turnstile.zeroclover.io/',
+            siteKey: "0x4AAAAAAAEwzhD6pyKkgXC0",
+            mode: "turnstile-min"
+        })
+        .expect(200)
+        .then(response => { expect(response.body.code).toEqual(200); })
+}, 60000)
+
+test('Creating a Turnstile Token With Site Key [max]', async () => {
+    return request(server)
+        .post("/cf-clearance-scraper")
+        .send({
+            url: 'https://turnstile.zeroclover.io/',
+            mode: "turnstile-max"
+        })
+        .expect(200)
+        .then(response => { expect(response.body.code).toEqual(200); })
+}, 60000)
+
+test('Create Cloudflare WAF Session', async () => {
+    return request(server)
+        .post("/cf-clearance-scraper")
+        .send({
+            url: 'https://nopecha.com/demo/cloudflare',
+            mode: "waf-session"
+        })
+        .expect(200)
+        .then(response => { expect(response.body.code).toEqual(200); })
+}, 60000)

tests/validate.test.js

@@ -0,0 +1,28 @@
+process.env.NODE_ENV = 'development'
+process.env.SKIP_LAUNCH = "true"
+process.env.authToken = "123456"
+process.env.browserLimit = -1
+
+const server = require('../src/index')
+const request = require("supertest")
+
+test('Request Authorisation Control Test', async () => {
+    return request(server)
+        .post("/cf-clearance-scraper")
+        .send({
+            url: 'https://nopecha.com/demo/cloudflare',
+            mode: "source"
+        })
+        .expect(401)
+}, 10000)
+
+test('Browser Context Limit Control Test', async () => {
+    return request(server)
+        .post("/cf-clearance-scraper")
+        .send({
+            url: 'https://nopecha.com/demo/cloudflare',
+            mode: "source",
+            authToken: "123456"
+        })
+        .expect(429)
+}, 10000)