opticparse-js/README.md

# Opticparse SDK & CLI Client

The official, strongly-typed TypeScript/JavaScript SDK and CLI tool for the **Opticparse AI Vision Web Scraper**. 

By capturing visual snapshots (screenshots) of target web pages and parsing them with Gemini's vision-language models, Opticparse allows you to scrape dynamic JS-heavy websites **without maintaining CSS or XPath selectors**. If the website redesigns its code, your scraper continues to work flawlessly.

---

## Features
* 🛡️ **Fully typed** interfaces with TypeScript autocomplete.
* ⚡ **Selector-free scraping** via AI multimodal visual extraction.
* 🌐 Works seamlessly with both **direct Render deployments** and the **opticparse.com Gateway**.
* 🐚 **Terminal CLI companion tool** to inspect and test scraping results locally.

---

## Installation

Install the package via npm:

```bash
npm install opticparse-js
```

---

## Authentication Configuration

The client requires an API Key. You can pass it explicitly in the code constructor or set it as environment variables:

### Direct Server Configuration
```env
OPTICPARSE_API_KEY=your_secure_opticparse_api_key_here
OPTICPARSE_API_URL=https://opticparse.onrender.com # defaults to http://localhost:8000
```

### opticparse.com Proxy Configuration
```env
opticparse.com_KEY=your_opticparse.com_developer_key_here
OPTICPARSE_USE_opticparse.com=true
```

---

## Library Usage (JS/TS)

### Direct Mode Example
Querying your direct/private Render or local backend server:

```typescript
import { OpticparseClient } from 'opticparse-js';

const client = new OpticparseClient({
  apiKey: 'your_secure_opticparse_api_key_here',
  apiUrl: 'https://opticparse.onrender.com'
});

async function run() {
  try {
    const result = await client.scrape({
      targetUrl: 'https://news.ycombinator.com',
      extractionQuery: 'Extract the top 5 article titles, their link URLs, and points as a JSON list.',
    });
    
    console.log(result);
  } catch (error) {
    console.error('Scrape error:', error);
  }
}

run();
```

### opticparse.com Mode Example
Querying the API hosted on the opticparse.com Hub:

```typescript
import { OpticparseClient } from 'opticparse-js';

const client = new OpticparseClient({
  apiKey: 'your_opticparse.com_developer_key_here',
  useopticparse.com: true
});

async function run() {
  const result = await client.scrape({
    targetUrl: 'https://example.com',
    extractionQuery: 'Extract the main heading text.'
  });
  console.log(result);
}

run();
```

---

## Command Line Interface (CLI)

You can run scrapers directly from your shell.

### Global Installation (Optional)
```bash
npm install -g opticparse-js
```

### Direct Server Query
```bash
# Set environment key
export OPTICPARSE_API_KEY="your_key"

# Scrape using CLI
npx opticparse scrape \
  --url "https://example.com" \
  --query "Extract the page header text" \
  --api-url "https://opticparse.onrender.com"
```

### opticparse.com Query
```bash
npx opticparse scrape \
  --url "https://example.com" \
  --query "Extract the page header text" \
  --key "your_opticparse.com_developer_key_here" \
  --opticparse.com
```

---

## API Reference Options

The `scrape()` method supports the following options:

| Option | Type | Default | Description |
|---|---|---|---|
| `targetUrl` | `string` | **Required** | The target website URL to scrape. |
| `extractionQuery` | `string` | **Required** | Instructions on what data structure and values to extract. |
| `viewportWidth` | `number` | `1280` | Browser viewport width in pixels. |
| `viewportHeight` | `number` | `800` | Browser viewport height in pixels. |
| `waitUntil` | `'networkidle' \| 'load' \| 'domcontentloaded'` | `'networkidle'` | Playwright page load synchronization state. |
| `timeout` | `number` | `30000` | Request and navigation timeout threshold in milliseconds. |

---

## License
MIT