Spaces:
Runtime error
Runtime error
ChartDB Admin commited on
Commit ·
0b5960f
1
Parent(s): a43516b
Deploy GDELT Engine v3.0.0 - Timestamp processing API with parallel workers
Browse files- .env.example +14 -0
- Dockerfile +37 -0
- README.md +57 -5
- cmd/main.go +135 -0
- go.mod +22 -0
- go.sum +62 -0
- internal/api/handlers.go +186 -0
- internal/api/server.go +66 -0
- internal/config/config.go +35 -0
- internal/console/logger.go +252 -0
- internal/constants/constants.go +101 -0
- internal/downloader/downloader.go +184 -0
- internal/lookups/lookups.go +148 -0
- internal/parser/parser.go +407 -0
- internal/processor/processor.go +686 -0
- internal/schema/schema.go +231 -0
- internal/storage/mongo.go +345 -0
- scripts/curl_commands.sh +105 -0
- scripts/normalize.py +191 -0
- scripts/stress_test.sh +373 -0
.env.example
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# GDELT Engine Configuration
|
| 2 |
+
# Copy this to .env and configure
|
| 3 |
+
|
| 4 |
+
# MongoDB connection URI (required for production)
|
| 5 |
+
MONGO_URI=mongodb://localhost:27017
|
| 6 |
+
|
| 7 |
+
# Database name (default: gdelt)
|
| 8 |
+
DATABASE_NAME=gdelt
|
| 9 |
+
|
| 10 |
+
# Server port (default: 7860)
|
| 11 |
+
PORT=7860
|
| 12 |
+
|
| 13 |
+
# Log level: debug, info, warn, error (default: info)
|
| 14 |
+
LOG_LEVEL=info
|
Dockerfile
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Build stage
|
| 2 |
+
FROM golang:1.21-alpine AS builder
|
| 3 |
+
|
| 4 |
+
WORKDIR /app
|
| 5 |
+
|
| 6 |
+
# Install git for go mod
|
| 7 |
+
RUN apk add --no-cache git ca-certificates
|
| 8 |
+
|
| 9 |
+
# Copy go mod files
|
| 10 |
+
COPY go.mod go.sum ./
|
| 11 |
+
RUN go mod download
|
| 12 |
+
|
| 13 |
+
# Copy source code
|
| 14 |
+
COPY . .
|
| 15 |
+
|
| 16 |
+
# Build binary
|
| 17 |
+
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o gdelt-engine ./cmd
|
| 18 |
+
|
| 19 |
+
# Final stage
|
| 20 |
+
FROM alpine:latest
|
| 21 |
+
|
| 22 |
+
RUN apk --no-cache add ca-certificates tzdata
|
| 23 |
+
|
| 24 |
+
WORKDIR /app
|
| 25 |
+
|
| 26 |
+
# Copy binary from builder
|
| 27 |
+
COPY --from=builder /app/gdelt-engine .
|
| 28 |
+
|
| 29 |
+
# Expose port
|
| 30 |
+
EXPOSE 7860
|
| 31 |
+
|
| 32 |
+
# Health check
|
| 33 |
+
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
| 34 |
+
CMD wget --no-verbose --tries=1 --spider http://localhost:7860/health || exit 1
|
| 35 |
+
|
| 36 |
+
# Run
|
| 37 |
+
CMD ["./gdelt-engine"]
|
README.md
CHANGED
|
@@ -1,11 +1,63 @@
|
|
| 1 |
---
|
| 2 |
-
title:
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
colorTo: green
|
| 6 |
sdk: docker
|
| 7 |
pinned: false
|
| 8 |
-
|
|
|
|
| 9 |
---
|
| 10 |
|
| 11 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: GDELT Engine
|
| 3 |
+
emoji: 🌍
|
| 4 |
+
colorFrom: blue
|
| 5 |
colorTo: green
|
| 6 |
sdk: docker
|
| 7 |
pinned: false
|
| 8 |
+
license: mit
|
| 9 |
+
app_port: 7860
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# GDELT Engine
|
| 13 |
+
|
| 14 |
+
High-performance Go API for processing GDELT (Global Database of Events, Language, and Tone) data.
|
| 15 |
+
|
| 16 |
+
## Features
|
| 17 |
+
|
| 18 |
+
- **Timestamp-triggered processing** - Submit timestamps via API
|
| 19 |
+
- **Parallel processing** - Up to 24 timestamps concurrently
|
| 20 |
+
- **Streaming downloads** - Memory-efficient file processing
|
| 21 |
+
- **Structural validation** - Validates GDELT timestamp format
|
| 22 |
+
|
| 23 |
+
## API Endpoints
|
| 24 |
+
|
| 25 |
+
| Endpoint | Method | Description |
|
| 26 |
+
|----------|--------|-------------|
|
| 27 |
+
| `/` | GET | API info |
|
| 28 |
+
| `/health` | GET | Health check |
|
| 29 |
+
| `/stats` | GET | Database statistics |
|
| 30 |
+
| `/process` | POST | Submit timestamps for processing |
|
| 31 |
+
| `/status/{ts}` | GET | Check timestamp status |
|
| 32 |
+
| `/timestamps` | GET | List all timestamps |
|
| 33 |
+
|
| 34 |
+
## Usage
|
| 35 |
+
|
| 36 |
+
### Submit Timestamps
|
| 37 |
+
|
| 38 |
+
```bash
|
| 39 |
+
curl -X POST https://subham9126-goworker.hf.space/process \
|
| 40 |
+
-H "Content-Type: application/json" \
|
| 41 |
+
-d '{"timestamps": ["20260128171500", "20260128173000"]}'
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
### Check Status
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
curl https://subham9126-goworker.hf.space/status/20260128171500
|
| 48 |
+
```
|
| 49 |
+
|
| 50 |
+
## Timestamp Format
|
| 51 |
+
|
| 52 |
+
- 14 characters: `YYYYMMDDHHmmss`
|
| 53 |
+
- Minutes must be: `00`, `15`, `30`, or `45`
|
| 54 |
+
- Cannot be in the future
|
| 55 |
+
|
| 56 |
+
## Environment Variables
|
| 57 |
+
|
| 58 |
+
Set these as **Secrets** in Space settings:
|
| 59 |
+
|
| 60 |
+
| Variable | Description |
|
| 61 |
+
|----------|-------------|
|
| 62 |
+
| `MONGO_URI` | MongoDB connection string (required) |
|
| 63 |
+
| `DATABASE_NAME` | Database name (default: gdelt) |
|
cmd/main.go
ADDED
|
@@ -0,0 +1,135 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package main
|
| 2 |
+
|
| 3 |
+
import (
|
| 4 |
+
"context"
|
| 5 |
+
"net/http"
|
| 6 |
+
"os"
|
| 7 |
+
"os/signal"
|
| 8 |
+
"strings"
|
| 9 |
+
"syscall"
|
| 10 |
+
"time"
|
| 11 |
+
|
| 12 |
+
"gdelt-engine/internal/api"
|
| 13 |
+
"gdelt-engine/internal/config"
|
| 14 |
+
"gdelt-engine/internal/console"
|
| 15 |
+
"gdelt-engine/internal/processor"
|
| 16 |
+
"gdelt-engine/internal/storage"
|
| 17 |
+
|
| 18 |
+
"go.uber.org/zap"
|
| 19 |
+
"go.uber.org/zap/zapcore"
|
| 20 |
+
)
|
| 21 |
+
|
| 22 |
+
const version = "3.0.0"
|
| 23 |
+
|
| 24 |
+
func main() {
|
| 25 |
+
// Load configuration
|
| 26 |
+
cfg := config.Load()
|
| 27 |
+
|
| 28 |
+
// Initialize loggers
|
| 29 |
+
zapLogger := initZapLogger(cfg.LogLevel)
|
| 30 |
+
defer zapLogger.Sync()
|
| 31 |
+
|
| 32 |
+
consoleLog := console.New(version)
|
| 33 |
+
|
| 34 |
+
// Print startup banner
|
| 35 |
+
consoleLog.PrintBanner(false)
|
| 36 |
+
consoleLog.Info("Mode: Timestamp-triggered processing with parallel workers")
|
| 37 |
+
|
| 38 |
+
// Create context with signal handling
|
| 39 |
+
ctx, cancel := context.WithCancel(context.Background())
|
| 40 |
+
defer cancel()
|
| 41 |
+
|
| 42 |
+
// Handle graceful shutdown
|
| 43 |
+
sigCh := make(chan os.Signal, 1)
|
| 44 |
+
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
|
| 45 |
+
|
| 46 |
+
// Connect to MongoDB
|
| 47 |
+
consoleLog.Info("Connecting to MongoDB...")
|
| 48 |
+
db, err := storage.NewMongoDB(ctx, cfg.MongoURI, cfg.DatabaseName, zapLogger)
|
| 49 |
+
if err != nil {
|
| 50 |
+
consoleLog.Fatal("MongoDB connection failed: %v", err)
|
| 51 |
+
}
|
| 52 |
+
defer func() {
|
| 53 |
+
closeCtx, closeCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
| 54 |
+
defer closeCancel()
|
| 55 |
+
db.Close(closeCtx)
|
| 56 |
+
}()
|
| 57 |
+
consoleLog.Success("MongoDB connected to %s", cfg.DatabaseName)
|
| 58 |
+
|
| 59 |
+
// Create processor with worker pool
|
| 60 |
+
proc := processor.NewProcessor(db, zapLogger,
|
| 61 |
+
processor.WithMaxParallelTimestamps(24),
|
| 62 |
+
processor.WithTimeout(5*time.Minute),
|
| 63 |
+
processor.WithBatchSize(1000),
|
| 64 |
+
)
|
| 65 |
+
consoleLog.Success("Processor initialized")
|
| 66 |
+
|
| 67 |
+
// Create API handlers
|
| 68 |
+
handlers := api.NewHandlers(proc, db.GetStats)
|
| 69 |
+
|
| 70 |
+
// Create and start API server
|
| 71 |
+
apiServer := api.NewServer(cfg.Port, handlers)
|
| 72 |
+
go func() {
|
| 73 |
+
consoleLog.Info("API server starting on :%s", cfg.Port)
|
| 74 |
+
consoleLog.Info("Endpoints:")
|
| 75 |
+
consoleLog.Info(" POST /process - Submit timestamps for processing")
|
| 76 |
+
consoleLog.Info(" GET /status/{ts} - Check timestamp status")
|
| 77 |
+
consoleLog.Info(" GET /timestamps - List all timestamps")
|
| 78 |
+
consoleLog.Info(" GET /stats - Database statistics")
|
| 79 |
+
consoleLog.Info(" GET /health - Health check")
|
| 80 |
+
if err := apiServer.Start(); err != nil && err != http.ErrServerClosed {
|
| 81 |
+
consoleLog.Error("API server error: %v", err)
|
| 82 |
+
}
|
| 83 |
+
}()
|
| 84 |
+
|
| 85 |
+
consoleLog.Success("Ready - waiting for requests")
|
| 86 |
+
|
| 87 |
+
// Wait for shutdown signal
|
| 88 |
+
sig := <-sigCh
|
| 89 |
+
consoleLog.Warn("Shutdown signal received: %s", sig.String())
|
| 90 |
+
|
| 91 |
+
// Graceful shutdown
|
| 92 |
+
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second)
|
| 93 |
+
defer shutdownCancel()
|
| 94 |
+
|
| 95 |
+
if err := apiServer.Shutdown(shutdownCtx); err != nil {
|
| 96 |
+
consoleLog.Error("API server shutdown error: %v", err)
|
| 97 |
+
}
|
| 98 |
+
|
| 99 |
+
cancel()
|
| 100 |
+
consoleLog.Success("Shutdown complete")
|
| 101 |
+
}
|
| 102 |
+
|
| 103 |
+
func initZapLogger(level string) *zap.Logger {
|
| 104 |
+
var zapLevel zapcore.Level
|
| 105 |
+
switch strings.ToLower(level) {
|
| 106 |
+
case "debug":
|
| 107 |
+
zapLevel = zapcore.DebugLevel
|
| 108 |
+
case "warn":
|
| 109 |
+
zapLevel = zapcore.WarnLevel
|
| 110 |
+
case "error":
|
| 111 |
+
zapLevel = zapcore.ErrorLevel
|
| 112 |
+
default:
|
| 113 |
+
zapLevel = zapcore.InfoLevel
|
| 114 |
+
}
|
| 115 |
+
|
| 116 |
+
encoderConfig := zap.NewProductionEncoderConfig()
|
| 117 |
+
encoderConfig.TimeKey = "ts"
|
| 118 |
+
encoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
|
| 119 |
+
|
| 120 |
+
config := zap.Config{
|
| 121 |
+
Level: zap.NewAtomicLevelAt(zapLevel),
|
| 122 |
+
Development: false,
|
| 123 |
+
Encoding: "json",
|
| 124 |
+
EncoderConfig: encoderConfig,
|
| 125 |
+
OutputPaths: []string{"stderr"},
|
| 126 |
+
ErrorOutputPaths: []string{"stderr"},
|
| 127 |
+
}
|
| 128 |
+
|
| 129 |
+
logger, err := config.Build()
|
| 130 |
+
if err != nil {
|
| 131 |
+
panic(err)
|
| 132 |
+
}
|
| 133 |
+
|
| 134 |
+
return logger
|
| 135 |
+
}
|
go.mod
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
module gdelt-engine
|
| 2 |
+
|
| 3 |
+
go 1.23
|
| 4 |
+
|
| 5 |
+
require (
|
| 6 |
+
go.mongodb.org/mongo-driver v1.17.2
|
| 7 |
+
go.uber.org/zap v1.27.0
|
| 8 |
+
)
|
| 9 |
+
|
| 10 |
+
require (
|
| 11 |
+
github.com/golang/snappy v0.0.4 // indirect
|
| 12 |
+
github.com/klauspost/compress v1.16.7 // indirect
|
| 13 |
+
github.com/montanaflynn/stats v0.7.1 // indirect
|
| 14 |
+
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
|
| 15 |
+
github.com/xdg-go/scram v1.1.2 // indirect
|
| 16 |
+
github.com/xdg-go/stringprep v1.0.4 // indirect
|
| 17 |
+
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
|
| 18 |
+
go.uber.org/multierr v1.10.0 // indirect
|
| 19 |
+
golang.org/x/crypto v0.26.0 // indirect
|
| 20 |
+
golang.org/x/sync v0.8.0 // indirect
|
| 21 |
+
golang.org/x/text v0.17.0 // indirect
|
| 22 |
+
)
|
go.sum
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
| 2 |
+
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
| 3 |
+
github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
|
| 4 |
+
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
| 5 |
+
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
| 6 |
+
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
| 7 |
+
github.com/klauspost/compress v1.16.7 h1:2mk3MPGNzKyxErAw8YaohYh69+pa4sIQSC0fPGCFR9I=
|
| 8 |
+
github.com/klauspost/compress v1.16.7/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
|
| 9 |
+
github.com/montanaflynn/stats v0.7.1 h1:etflOAAHORrCC44V+aR6Ftzort912ZU+YLiSTuV8eaE=
|
| 10 |
+
github.com/montanaflynn/stats v0.7.1/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow=
|
| 11 |
+
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
| 12 |
+
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
| 13 |
+
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
|
| 14 |
+
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
|
| 15 |
+
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
|
| 16 |
+
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
|
| 17 |
+
github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
|
| 18 |
+
github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
|
| 19 |
+
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
|
| 20 |
+
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
|
| 21 |
+
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 h1:ilQV1hzziu+LLM3zUTJ0trRztfwgjqKnBWNtSRkbmwM=
|
| 22 |
+
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfSfmXjznFBSZNN13rSJjlIOI1fUNAtF7rmI=
|
| 23 |
+
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
| 24 |
+
go.mongodb.org/mongo-driver v1.17.2 h1:gvZyk8352qSfzyZ2UMWcpDpMSGEr1eqE4T793SqyhzM=
|
| 25 |
+
go.mongodb.org/mongo-driver v1.17.2/go.mod h1:Hy04i7O2kC4RS06ZrhPRqj/u4DTYkFDAAccj+rVKqgQ=
|
| 26 |
+
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
| 27 |
+
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
| 28 |
+
go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
|
| 29 |
+
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
| 30 |
+
go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
|
| 31 |
+
go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
|
| 32 |
+
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
| 33 |
+
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
|
| 34 |
+
golang.org/x/crypto v0.26.0 h1:RrRspgV4mU+YwB4FYnuBoKsUapNIL5cohGAmSH3azsw=
|
| 35 |
+
golang.org/x/crypto v0.26.0/go.mod h1:GY7jblb9wI+FOo5y8/S2oY4zWP07AkOJ4+jxCqdqn54=
|
| 36 |
+
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
|
| 37 |
+
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
| 38 |
+
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
|
| 39 |
+
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
|
| 40 |
+
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
| 41 |
+
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
| 42 |
+
golang.org/x/sync v0.8.0 h1:3NFvSEYkUoMifnESzZl15y791HH1qU2xm6eCJU5ZPXQ=
|
| 43 |
+
golang.org/x/sync v0.8.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
| 44 |
+
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
| 45 |
+
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
| 46 |
+
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
| 47 |
+
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
| 48 |
+
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
| 49 |
+
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
| 50 |
+
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
| 51 |
+
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
| 52 |
+
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
| 53 |
+
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
| 54 |
+
golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
|
| 55 |
+
golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc=
|
| 56 |
+
golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
|
| 57 |
+
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
| 58 |
+
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
|
| 59 |
+
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
|
| 60 |
+
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
| 61 |
+
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
| 62 |
+
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
internal/api/handlers.go
ADDED
|
@@ -0,0 +1,186 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Package api provides HTTP handlers for the GDELT Engine API.
|
| 2 |
+
package api
|
| 3 |
+
|
| 4 |
+
import (
|
| 5 |
+
"context"
|
| 6 |
+
"encoding/json"
|
| 7 |
+
"fmt"
|
| 8 |
+
"net/http"
|
| 9 |
+
"time"
|
| 10 |
+
|
| 11 |
+
"gdelt-engine/internal/processor"
|
| 12 |
+
)
|
| 13 |
+
|
| 14 |
+
// Handlers contains all HTTP handler methods
|
| 15 |
+
type Handlers struct {
|
| 16 |
+
processor *processor.Processor
|
| 17 |
+
getStats func(ctx context.Context) (map[string]interface{}, error)
|
| 18 |
+
}
|
| 19 |
+
|
| 20 |
+
// NewHandlers creates new API handlers
|
| 21 |
+
func NewHandlers(proc *processor.Processor, getStats func(ctx context.Context) (map[string]interface{}, error)) *Handlers {
|
| 22 |
+
return &Handlers{
|
| 23 |
+
processor: proc,
|
| 24 |
+
getStats: getStats,
|
| 25 |
+
}
|
| 26 |
+
}
|
| 27 |
+
|
| 28 |
+
// ProcessRequest is the request body for POST /process
|
| 29 |
+
type ProcessRequest struct {
|
| 30 |
+
Timestamps []string `json:"timestamps"`
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
// TimestampStatus represents the status of a single timestamp in the response
|
| 34 |
+
type TimestampStatus struct {
|
| 35 |
+
Timestamp string `json:"timestamp"`
|
| 36 |
+
Status string `json:"status"`
|
| 37 |
+
Reason string `json:"reason,omitempty"`
|
| 38 |
+
}
|
| 39 |
+
|
| 40 |
+
// ProcessResponse is the immediate response for job acceptance
|
| 41 |
+
type ProcessResponse struct {
|
| 42 |
+
Accepted bool `json:"accepted"`
|
| 43 |
+
Message string `json:"message"`
|
| 44 |
+
TimestampsQueued int `json:"timestamps_queued"`
|
| 45 |
+
TimestampsRejected int `json:"timestamps_rejected"`
|
| 46 |
+
Queued []string `json:"queued,omitempty"`
|
| 47 |
+
Rejected []TimestampStatus `json:"rejected,omitempty"`
|
| 48 |
+
CheckStatusAt string `json:"check_status_at"`
|
| 49 |
+
}
|
| 50 |
+
|
| 51 |
+
// HandleProcess accepts timestamps and processes them in the background.
|
| 52 |
+
// Returns immediately with job acceptance status and per-timestamp validation results.
|
| 53 |
+
func (h *Handlers) HandleProcess(w http.ResponseWriter, r *http.Request) {
|
| 54 |
+
if r.Method != http.MethodPost {
|
| 55 |
+
h.respondError(w, http.StatusMethodNotAllowed, "POST required")
|
| 56 |
+
return
|
| 57 |
+
}
|
| 58 |
+
|
| 59 |
+
var req ProcessRequest
|
| 60 |
+
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
| 61 |
+
h.respondError(w, http.StatusBadRequest, fmt.Sprintf("Invalid JSON: %v", err))
|
| 62 |
+
return
|
| 63 |
+
}
|
| 64 |
+
|
| 65 |
+
if len(req.Timestamps) == 0 {
|
| 66 |
+
h.respondError(w, http.StatusBadRequest, "No timestamps provided")
|
| 67 |
+
return
|
| 68 |
+
}
|
| 69 |
+
|
| 70 |
+
// Queue timestamps - validation happens inside QueueTimestamps
|
| 71 |
+
queued, rejected := h.processor.QueueTimestamps(r.Context(), req.Timestamps)
|
| 72 |
+
|
| 73 |
+
// Convert rejected to response format
|
| 74 |
+
rejectedStatus := make([]TimestampStatus, len(rejected))
|
| 75 |
+
for i, r := range rejected {
|
| 76 |
+
rejectedStatus[i] = TimestampStatus{
|
| 77 |
+
Timestamp: r.Timestamp,
|
| 78 |
+
Status: string(r.Status),
|
| 79 |
+
Reason: r.Reason,
|
| 80 |
+
}
|
| 81 |
+
}
|
| 82 |
+
|
| 83 |
+
// Determine if we accepted any
|
| 84 |
+
accepted := len(queued) > 0
|
| 85 |
+
|
| 86 |
+
resp := ProcessResponse{
|
| 87 |
+
Accepted: accepted,
|
| 88 |
+
Message: fmt.Sprintf("Queued %d timestamps, rejected %d", len(queued), len(rejected)),
|
| 89 |
+
TimestampsQueued: len(queued),
|
| 90 |
+
TimestampsRejected: len(rejected),
|
| 91 |
+
Queued: queued,
|
| 92 |
+
Rejected: rejectedStatus,
|
| 93 |
+
CheckStatusAt: "/status/{timestamp}",
|
| 94 |
+
}
|
| 95 |
+
|
| 96 |
+
h.respondJSON(w, http.StatusAccepted, resp)
|
| 97 |
+
}
|
| 98 |
+
|
| 99 |
+
// HandleStatus returns the processing status for a specific timestamp.
|
| 100 |
+
func (h *Handlers) HandleStatus(w http.ResponseWriter, r *http.Request) {
|
| 101 |
+
if r.Method != http.MethodGet {
|
| 102 |
+
h.respondError(w, http.StatusMethodNotAllowed, "GET required")
|
| 103 |
+
return
|
| 104 |
+
}
|
| 105 |
+
|
| 106 |
+
// Extract timestamp from URL path
|
| 107 |
+
// Expects path like /status/20260128171500
|
| 108 |
+
timestamp := r.PathValue("timestamp")
|
| 109 |
+
if timestamp == "" {
|
| 110 |
+
// Fallback for older Go versions or manual parsing
|
| 111 |
+
path := r.URL.Path
|
| 112 |
+
if len(path) > 8 && path[:8] == "/status/" {
|
| 113 |
+
timestamp = path[8:]
|
| 114 |
+
}
|
| 115 |
+
}
|
| 116 |
+
|
| 117 |
+
if timestamp == "" {
|
| 118 |
+
h.respondError(w, http.StatusBadRequest, "Timestamp required in path")
|
| 119 |
+
return
|
| 120 |
+
}
|
| 121 |
+
|
| 122 |
+
status, err := h.processor.GetTimestampStatus(r.Context(), timestamp)
|
| 123 |
+
if err != nil {
|
| 124 |
+
h.respondError(w, http.StatusInternalServerError, err.Error())
|
| 125 |
+
return
|
| 126 |
+
}
|
| 127 |
+
|
| 128 |
+
h.respondJSON(w, http.StatusOK, status)
|
| 129 |
+
}
|
| 130 |
+
|
| 131 |
+
// HandleListTimestamps returns all completed and processing timestamps.
|
| 132 |
+
func (h *Handlers) HandleListTimestamps(w http.ResponseWriter, r *http.Request) {
|
| 133 |
+
if r.Method != http.MethodGet {
|
| 134 |
+
h.respondError(w, http.StatusMethodNotAllowed, "GET required")
|
| 135 |
+
return
|
| 136 |
+
}
|
| 137 |
+
|
| 138 |
+
list, err := h.processor.GetAllTimestamps(r.Context())
|
| 139 |
+
if err != nil {
|
| 140 |
+
h.respondError(w, http.StatusInternalServerError, err.Error())
|
| 141 |
+
return
|
| 142 |
+
}
|
| 143 |
+
|
| 144 |
+
h.respondJSON(w, http.StatusOK, list)
|
| 145 |
+
}
|
| 146 |
+
|
| 147 |
+
// HandleStats returns database statistics.
|
| 148 |
+
func (h *Handlers) HandleStats(w http.ResponseWriter, r *http.Request) {
|
| 149 |
+
if r.Method != http.MethodGet {
|
| 150 |
+
h.respondError(w, http.StatusMethodNotAllowed, "GET required")
|
| 151 |
+
return
|
| 152 |
+
}
|
| 153 |
+
|
| 154 |
+
stats, err := h.getStats(r.Context())
|
| 155 |
+
if err != nil {
|
| 156 |
+
h.respondError(w, http.StatusInternalServerError, err.Error())
|
| 157 |
+
return
|
| 158 |
+
}
|
| 159 |
+
|
| 160 |
+
h.respondJSON(w, http.StatusOK, stats)
|
| 161 |
+
}
|
| 162 |
+
|
| 163 |
+
// HandleHealth returns a simple health check response.
|
| 164 |
+
func (h *Handlers) HandleHealth(w http.ResponseWriter, r *http.Request) {
|
| 165 |
+
h.respondJSON(w, http.StatusOK, map[string]interface{}{
|
| 166 |
+
"status": "healthy",
|
| 167 |
+
"time": time.Now().Format(time.RFC3339),
|
| 168 |
+
})
|
| 169 |
+
}
|
| 170 |
+
|
| 171 |
+
// Helper response functions
|
| 172 |
+
|
| 173 |
+
func (h *Handlers) respondJSON(w http.ResponseWriter, status int, data interface{}) {
|
| 174 |
+
w.Header().Set("Content-Type", "application/json")
|
| 175 |
+
w.Header().Set("Access-Control-Allow-Origin", "*")
|
| 176 |
+
w.WriteHeader(status)
|
| 177 |
+
json.NewEncoder(w).Encode(data)
|
| 178 |
+
}
|
| 179 |
+
|
| 180 |
+
func (h *Handlers) respondError(w http.ResponseWriter, status int, message string) {
|
| 181 |
+
h.respondJSON(w, status, map[string]interface{}{
|
| 182 |
+
"success": false,
|
| 183 |
+
"error": message,
|
| 184 |
+
"timestamp": time.Now().Format(time.RFC3339),
|
| 185 |
+
})
|
| 186 |
+
}
|
internal/api/server.go
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Package api provides the HTTP server for the GDELT Engine.
|
| 2 |
+
package api
|
| 3 |
+
|
| 4 |
+
import (
|
| 5 |
+
"context"
|
| 6 |
+
"net/http"
|
| 7 |
+
"time"
|
| 8 |
+
)
|
| 9 |
+
|
| 10 |
+
// Server wraps the HTTP server.
|
| 11 |
+
type Server struct {
|
| 12 |
+
server *http.Server
|
| 13 |
+
handlers *Handlers
|
| 14 |
+
}
|
| 15 |
+
|
| 16 |
+
// NewServer creates a new API server.
|
| 17 |
+
func NewServer(port string, handlers *Handlers) *Server {
|
| 18 |
+
mux := http.NewServeMux()
|
| 19 |
+
|
| 20 |
+
// Register routes
|
| 21 |
+
mux.HandleFunc("/health", handlers.HandleHealth)
|
| 22 |
+
mux.HandleFunc("/stats", handlers.HandleStats)
|
| 23 |
+
mux.HandleFunc("/process", handlers.HandleProcess)
|
| 24 |
+
mux.HandleFunc("/timestamps", handlers.HandleListTimestamps)
|
| 25 |
+
mux.HandleFunc("/status/", handlers.HandleStatus)
|
| 26 |
+
|
| 27 |
+
// Root endpoint - API info
|
| 28 |
+
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
|
| 29 |
+
if r.URL.Path != "/" {
|
| 30 |
+
http.NotFound(w, r)
|
| 31 |
+
return
|
| 32 |
+
}
|
| 33 |
+
handlers.respondJSON(w, http.StatusOK, map[string]interface{}{
|
| 34 |
+
"name": "GDELT Engine",
|
| 35 |
+
"version": "3.0.0",
|
| 36 |
+
"endpoints": map[string]string{
|
| 37 |
+
"POST /process": "Submit timestamps for processing",
|
| 38 |
+
"GET /status/{ts}": "Check timestamp status",
|
| 39 |
+
"GET /timestamps": "List all timestamps",
|
| 40 |
+
"GET /stats": "Database statistics",
|
| 41 |
+
"GET /health": "Health check",
|
| 42 |
+
},
|
| 43 |
+
})
|
| 44 |
+
})
|
| 45 |
+
|
| 46 |
+
return &Server{
|
| 47 |
+
server: &http.Server{
|
| 48 |
+
Addr: ":" + port,
|
| 49 |
+
Handler: mux,
|
| 50 |
+
ReadTimeout: 30 * time.Second,
|
| 51 |
+
WriteTimeout: 30 * time.Second,
|
| 52 |
+
IdleTimeout: 60 * time.Second,
|
| 53 |
+
},
|
| 54 |
+
handlers: handlers,
|
| 55 |
+
}
|
| 56 |
+
}
|
| 57 |
+
|
| 58 |
+
// Start starts the HTTP server.
|
| 59 |
+
func (s *Server) Start() error {
|
| 60 |
+
return s.server.ListenAndServe()
|
| 61 |
+
}
|
| 62 |
+
|
| 63 |
+
// Shutdown gracefully shuts down the server.
|
| 64 |
+
func (s *Server) Shutdown(ctx context.Context) error {
|
| 65 |
+
return s.server.Shutdown(ctx)
|
| 66 |
+
}
|
internal/config/config.go
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package config
|
| 2 |
+
|
| 3 |
+
import "os"
|
| 4 |
+
|
| 5 |
+
// Config holds application configuration from environment variables.
|
| 6 |
+
type Config struct {
|
| 7 |
+
// MongoDB connection URI (required)
|
| 8 |
+
MongoURI string
|
| 9 |
+
|
| 10 |
+
// Database name
|
| 11 |
+
DatabaseName string
|
| 12 |
+
|
| 13 |
+
// API Server port
|
| 14 |
+
Port string
|
| 15 |
+
|
| 16 |
+
// Logging level (debug, info, warn, error)
|
| 17 |
+
LogLevel string
|
| 18 |
+
}
|
| 19 |
+
|
| 20 |
+
// Load reads configuration from environment variables with sensible defaults.
|
| 21 |
+
func Load() *Config {
|
| 22 |
+
return &Config{
|
| 23 |
+
MongoURI: getEnv("MONGO_URI", "mongodb://localhost:27017"),
|
| 24 |
+
DatabaseName: getEnv("DATABASE_NAME", "gdelt"),
|
| 25 |
+
Port: getEnv("PORT", "7860"),
|
| 26 |
+
LogLevel: getEnv("LOG_LEVEL", "info"),
|
| 27 |
+
}
|
| 28 |
+
}
|
| 29 |
+
|
| 30 |
+
func getEnv(key, defaultVal string) string {
|
| 31 |
+
if val := os.Getenv(key); val != "" {
|
| 32 |
+
return val
|
| 33 |
+
}
|
| 34 |
+
return defaultVal
|
| 35 |
+
}
|
internal/console/logger.go
ADDED
|
@@ -0,0 +1,252 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package console
|
| 2 |
+
|
| 3 |
+
import (
|
| 4 |
+
"fmt"
|
| 5 |
+
"os"
|
| 6 |
+
"strings"
|
| 7 |
+
"sync"
|
| 8 |
+
"time"
|
| 9 |
+
)
|
| 10 |
+
|
| 11 |
+
// ANSI color codes
|
| 12 |
+
const (
|
| 13 |
+
Reset = "\033[0m"
|
| 14 |
+
Bold = "\033[1m"
|
| 15 |
+
Dim = "\033[2m"
|
| 16 |
+
Red = "\033[31m"
|
| 17 |
+
Green = "\033[32m"
|
| 18 |
+
Yellow = "\033[33m"
|
| 19 |
+
Blue = "\033[34m"
|
| 20 |
+
Magenta = "\033[35m"
|
| 21 |
+
Cyan = "\033[36m"
|
| 22 |
+
White = "\033[37m"
|
| 23 |
+
BgBlue = "\033[44m"
|
| 24 |
+
)
|
| 25 |
+
|
| 26 |
+
// Logger provides optimistic console output with millisecond timestamps.
|
| 27 |
+
type Logger struct {
|
| 28 |
+
mu sync.Mutex
|
| 29 |
+
startUp time.Time
|
| 30 |
+
lastRun time.Time
|
| 31 |
+
nextRun time.Time
|
| 32 |
+
version string
|
| 33 |
+
mongoOK bool
|
| 34 |
+
}
|
| 35 |
+
|
| 36 |
+
// New creates a new console logger.
|
| 37 |
+
func New(version string) *Logger {
|
| 38 |
+
return &Logger{
|
| 39 |
+
startUp: time.Now(),
|
| 40 |
+
version: version,
|
| 41 |
+
}
|
| 42 |
+
}
|
| 43 |
+
|
| 44 |
+
// timestamp returns current time with millisecond precision.
|
| 45 |
+
func (l *Logger) timestamp() string {
|
| 46 |
+
return time.Now().Format("15:04:05.000")
|
| 47 |
+
}
|
| 48 |
+
|
| 49 |
+
// PrintBanner displays the startup banner.
|
| 50 |
+
func (l *Logger) PrintBanner(mongoConnected bool) {
|
| 51 |
+
l.mu.Lock()
|
| 52 |
+
defer l.mu.Unlock()
|
| 53 |
+
l.mongoOK = mongoConnected
|
| 54 |
+
|
| 55 |
+
mongoStatus := Green + "Connected" + Reset
|
| 56 |
+
if !mongoConnected {
|
| 57 |
+
mongoStatus = Red + "Disconnected" + Reset
|
| 58 |
+
}
|
| 59 |
+
|
| 60 |
+
fmt.Println()
|
| 61 |
+
fmt.Println(Cyan + "╭─────────────────────────────────────────────────────╮" + Reset)
|
| 62 |
+
fmt.Printf(Cyan+"│"+Reset+" %s🚀 GDELT Engine %s%-36s"+Cyan+"│"+Reset+"\n", Bold, l.version, Reset)
|
| 63 |
+
fmt.Printf(Cyan+"│"+Reset+" Status: %sRunning%s │ MongoDB: %-24s"+Cyan+"│"+Reset+"\n", Green, Reset, mongoStatus)
|
| 64 |
+
fmt.Println(Cyan + "╰─────────────────────────────────────────────────────╯" + Reset)
|
| 65 |
+
fmt.Println()
|
| 66 |
+
}
|
| 67 |
+
|
| 68 |
+
// Info logs an info message with timestamp.
|
| 69 |
+
func (l *Logger) Info(format string, args ...interface{}) {
|
| 70 |
+
l.mu.Lock()
|
| 71 |
+
defer l.mu.Unlock()
|
| 72 |
+
msg := fmt.Sprintf(format, args...)
|
| 73 |
+
fmt.Printf("[%s] %s⬤%s %s\n", l.timestamp(), Blue, Reset, msg)
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
// Success logs a success message with timestamp.
|
| 77 |
+
func (l *Logger) Success(format string, args ...interface{}) {
|
| 78 |
+
l.mu.Lock()
|
| 79 |
+
defer l.mu.Unlock()
|
| 80 |
+
msg := fmt.Sprintf(format, args...)
|
| 81 |
+
fmt.Printf("[%s] %s✅%s %s\n", l.timestamp(), Green, Reset, msg)
|
| 82 |
+
}
|
| 83 |
+
|
| 84 |
+
// Warn logs a warning message with timestamp.
|
| 85 |
+
func (l *Logger) Warn(format string, args ...interface{}) {
|
| 86 |
+
l.mu.Lock()
|
| 87 |
+
defer l.mu.Unlock()
|
| 88 |
+
msg := fmt.Sprintf(format, args...)
|
| 89 |
+
fmt.Printf("[%s] %s⚠️%s %s\n", l.timestamp(), Yellow, Reset, msg)
|
| 90 |
+
}
|
| 91 |
+
|
| 92 |
+
// Error logs an error message with timestamp.
|
| 93 |
+
func (l *Logger) Error(format string, args ...interface{}) {
|
| 94 |
+
l.mu.Lock()
|
| 95 |
+
defer l.mu.Unlock()
|
| 96 |
+
msg := fmt.Sprintf(format, args...)
|
| 97 |
+
fmt.Printf("[%s] %s❌%s %s\n", l.timestamp(), Red, Reset, msg)
|
| 98 |
+
}
|
| 99 |
+
|
| 100 |
+
// Debug logs a debug message with timestamp (dimmed).
|
| 101 |
+
func (l *Logger) Debug(format string, args ...interface{}) {
|
| 102 |
+
l.mu.Lock()
|
| 103 |
+
defer l.mu.Unlock()
|
| 104 |
+
msg := fmt.Sprintf(format, args...)
|
| 105 |
+
fmt.Printf("[%s] %s⋯ %s%s\n", l.timestamp(), Dim, msg, Reset)
|
| 106 |
+
}
|
| 107 |
+
|
| 108 |
+
// Progress prints a progress indicator.
|
| 109 |
+
func (l *Logger) Progress(label string, current, total int64) {
|
| 110 |
+
l.mu.Lock()
|
| 111 |
+
defer l.mu.Unlock()
|
| 112 |
+
|
| 113 |
+
pct := float64(current) / float64(total) * 100
|
| 114 |
+
barWidth := 40
|
| 115 |
+
filled := int(float64(barWidth) * pct / 100)
|
| 116 |
+
|
| 117 |
+
bar := strings.Repeat("█", filled) + strings.Repeat("░", barWidth-filled)
|
| 118 |
+
fmt.Printf("\r %s%s%s %s%.1f%%%s", Cyan, bar, Reset, Dim, pct, Reset)
|
| 119 |
+
|
| 120 |
+
if current >= total {
|
| 121 |
+
fmt.Println()
|
| 122 |
+
}
|
| 123 |
+
}
|
| 124 |
+
|
| 125 |
+
// StartPoll indicates polling has started.
|
| 126 |
+
func (l *Logger) StartPoll() {
|
| 127 |
+
l.mu.Lock()
|
| 128 |
+
defer l.mu.Unlock()
|
| 129 |
+
fmt.Printf("[%s] %s⏳%s Polling GDELT for updates...\n", l.timestamp(), Yellow, Reset)
|
| 130 |
+
}
|
| 131 |
+
|
| 132 |
+
// Download logs download progress.
|
| 133 |
+
func (l *Logger) Download(filename string, sizeKB int64) {
|
| 134 |
+
l.mu.Lock()
|
| 135 |
+
defer l.mu.Unlock()
|
| 136 |
+
fmt.Printf("[%s] %s📥%s Downloading %s%s%s (%dKB)\n",
|
| 137 |
+
l.timestamp(), Cyan, Reset, Bold, filename, Reset, sizeKB)
|
| 138 |
+
}
|
| 139 |
+
|
| 140 |
+
// Ingested logs successful ingestion.
|
| 141 |
+
func (l *Logger) Ingested(fileType string, rows int, durationMs int64) {
|
| 142 |
+
l.mu.Lock()
|
| 143 |
+
defer l.mu.Unlock()
|
| 144 |
+
|
| 145 |
+
icon := "📦"
|
| 146 |
+
color := Green
|
| 147 |
+
|
| 148 |
+
switch fileType {
|
| 149 |
+
case "export":
|
| 150 |
+
icon = "📰"
|
| 151 |
+
case "mentions":
|
| 152 |
+
icon = "💬"
|
| 153 |
+
case "gkg":
|
| 154 |
+
icon = "🌐"
|
| 155 |
+
}
|
| 156 |
+
|
| 157 |
+
fmt.Printf("[%s] %s%s%s %s: %s%s%s rows ingested (%s%.2fs%s)\n",
|
| 158 |
+
l.timestamp(), color, icon, Reset,
|
| 159 |
+
strings.Title(fileType),
|
| 160 |
+
Bold, formatNumber(rows), Reset,
|
| 161 |
+
Dim, float64(durationMs)/1000.0, Reset)
|
| 162 |
+
}
|
| 163 |
+
|
| 164 |
+
// PrintSummary displays ingestion summary box.
|
| 165 |
+
func (l *Logger) PrintSummary(files, rows int, durationMs int64, nextRun time.Time) {
|
| 166 |
+
l.mu.Lock()
|
| 167 |
+
defer l.mu.Unlock()
|
| 168 |
+
|
| 169 |
+
l.lastRun = time.Now()
|
| 170 |
+
l.nextRun = nextRun
|
| 171 |
+
|
| 172 |
+
untilNext := time.Until(nextRun)
|
| 173 |
+
nextStr := fmt.Sprintf("%s (in %s)", nextRun.Format("15:04:05"), formatDuration(untilNext))
|
| 174 |
+
|
| 175 |
+
fmt.Println()
|
| 176 |
+
fmt.Println(Green + "╭─────────────── Ingestion Summary ───────────────╮" + Reset)
|
| 177 |
+
fmt.Printf(Green+"│"+Reset+" Files: %s%-3d%s Rows: %s%-8s%s Time: %s%.2fs%s "+Green+"│"+Reset+"\n",
|
| 178 |
+
Bold, files, Reset,
|
| 179 |
+
Bold, formatNumber(rows), Reset,
|
| 180 |
+
Bold, float64(durationMs)/1000.0, Reset)
|
| 181 |
+
fmt.Printf(Green+"│"+Reset+" Next run: %-38s"+Green+"│"+Reset+"\n", nextStr)
|
| 182 |
+
fmt.Println(Green + "╰──────────────────────────────────────────────────╯" + Reset)
|
| 183 |
+
fmt.Println()
|
| 184 |
+
}
|
| 185 |
+
|
| 186 |
+
// PrintScheduleInfo displays schedule configuration.
|
| 187 |
+
func (l *Logger) PrintScheduleInfo(schedule []int, offsetSec int) {
|
| 188 |
+
l.mu.Lock()
|
| 189 |
+
defer l.mu.Unlock()
|
| 190 |
+
|
| 191 |
+
mins := make([]string, len(schedule))
|
| 192 |
+
for i, m := range schedule {
|
| 193 |
+
mins[i] = fmt.Sprintf(":%02d", m)
|
| 194 |
+
}
|
| 195 |
+
|
| 196 |
+
fmt.Printf("[%s] %s📅%s Schedule: %s%s%s + %ds offset\n",
|
| 197 |
+
l.timestamp(), Magenta, Reset,
|
| 198 |
+
Bold, strings.Join(mins, ", "), Reset, offsetSec)
|
| 199 |
+
}
|
| 200 |
+
|
| 201 |
+
// Fatal logs a fatal error and exits.
|
| 202 |
+
func (l *Logger) Fatal(format string, args ...interface{}) {
|
| 203 |
+
l.Error(format, args...)
|
| 204 |
+
os.Exit(1)
|
| 205 |
+
}
|
| 206 |
+
|
| 207 |
+
// formatNumber adds commas to numbers.
|
| 208 |
+
func formatNumber(n int) string {
|
| 209 |
+
if n < 1000 {
|
| 210 |
+
return fmt.Sprintf("%d", n)
|
| 211 |
+
}
|
| 212 |
+
return fmt.Sprintf("%d,%03d", n/1000, n%1000)
|
| 213 |
+
}
|
| 214 |
+
|
| 215 |
+
// formatDuration formats duration nicely.
|
| 216 |
+
func formatDuration(d time.Duration) string {
|
| 217 |
+
if d < time.Minute {
|
| 218 |
+
return fmt.Sprintf("%ds", int(d.Seconds()))
|
| 219 |
+
}
|
| 220 |
+
m := int(d.Minutes())
|
| 221 |
+
s := int(d.Seconds()) % 60
|
| 222 |
+
return fmt.Sprintf("%dm %ds", m, s)
|
| 223 |
+
}
|
| 224 |
+
|
| 225 |
+
// GetStatus returns current status for API.
|
| 226 |
+
func (l *Logger) GetStatus() map[string]interface{} {
|
| 227 |
+
l.mu.Lock()
|
| 228 |
+
defer l.mu.Unlock()
|
| 229 |
+
|
| 230 |
+
return map[string]interface{}{
|
| 231 |
+
"status": "running",
|
| 232 |
+
"uptime_seconds": int(time.Since(l.startUp).Seconds()),
|
| 233 |
+
"mongodb": l.mongoOK,
|
| 234 |
+
"last_run": l.lastRun.Format(time.RFC3339Nano),
|
| 235 |
+
"next_run": l.nextRun.Format(time.RFC3339Nano),
|
| 236 |
+
"version": l.version,
|
| 237 |
+
}
|
| 238 |
+
}
|
| 239 |
+
|
| 240 |
+
// SetNextRun updates the next run time.
|
| 241 |
+
func (l *Logger) SetNextRun(t time.Time) {
|
| 242 |
+
l.mu.Lock()
|
| 243 |
+
defer l.mu.Unlock()
|
| 244 |
+
l.nextRun = t
|
| 245 |
+
}
|
| 246 |
+
|
| 247 |
+
// SetLastRun updates the last run time.
|
| 248 |
+
func (l *Logger) SetLastRun(t time.Time) {
|
| 249 |
+
l.mu.Lock()
|
| 250 |
+
defer l.mu.Unlock()
|
| 251 |
+
l.lastRun = t
|
| 252 |
+
}
|
internal/constants/constants.go
ADDED
|
@@ -0,0 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Package constants provides configurable settings for the GDELT Engine.
|
| 2 |
+
// All values are exported as variables to allow runtime customization.
|
| 3 |
+
package constants
|
| 4 |
+
|
| 5 |
+
import "time"
|
| 6 |
+
|
| 7 |
+
// GDELT URL configuration
|
| 8 |
+
var (
|
| 9 |
+
// GDELTBaseURL is the base URL for GDELT data files
|
| 10 |
+
GDELTBaseURL = "http://data.gdeltproject.org/gdeltv2"
|
| 11 |
+
|
| 12 |
+
// ExportURLTemplate is the URL template for export CSV files
|
| 13 |
+
// Use %s placeholder for timestamp
|
| 14 |
+
ExportURLTemplate = GDELTBaseURL + "/%s.export.CSV.zip"
|
| 15 |
+
|
| 16 |
+
// MentionsURLTemplate is the URL template for mentions CSV files
|
| 17 |
+
MentionsURLTemplate = GDELTBaseURL + "/%s.mentions.CSV.zip"
|
| 18 |
+
|
| 19 |
+
// GKGURLTemplate is the URL template for GKG CSV files
|
| 20 |
+
GKGURLTemplate = GDELTBaseURL + "/%s.gkg.csv.zip"
|
| 21 |
+
)
|
| 22 |
+
|
| 23 |
+
// Processing configuration
|
| 24 |
+
var (
|
| 25 |
+
// MaxParallelTimestamps is the maximum timestamps to process concurrently
|
| 26 |
+
MaxParallelTimestamps = 24
|
| 27 |
+
|
| 28 |
+
// MaxParallelFiles is the maximum files to process in parallel per timestamp batch
|
| 29 |
+
MaxParallelFiles = 72
|
| 30 |
+
|
| 31 |
+
// DefaultWorkers is the default number of concurrent workers per timestamp
|
| 32 |
+
DefaultWorkers = 3
|
| 33 |
+
|
| 34 |
+
// MaxWorkers is the maximum allowed workers
|
| 35 |
+
MaxWorkers = 10
|
| 36 |
+
|
| 37 |
+
// DefaultTimeout is the default processing timeout per timestamp
|
| 38 |
+
DefaultTimeout = 5 * time.Minute
|
| 39 |
+
|
| 40 |
+
// DefaultHTTPTimeout is the HTTP client timeout for downloads
|
| 41 |
+
DefaultHTTPTimeout = 60 * time.Second
|
| 42 |
+
|
| 43 |
+
// HeadRequestTimeout is the timeout for HEAD validation requests
|
| 44 |
+
HeadRequestTimeout = 10 * time.Second
|
| 45 |
+
|
| 46 |
+
// DefaultBatchSize is the default batch size for database inserts
|
| 47 |
+
DefaultBatchSize = 1000
|
| 48 |
+
|
| 49 |
+
// ChannelBufferSize is the buffer size for processing channels
|
| 50 |
+
ChannelBufferSize = 100
|
| 51 |
+
|
| 52 |
+
// GraceWindowMinutes is the grace period for recent timestamps that might not be published yet
|
| 53 |
+
GraceWindowMinutes = 20
|
| 54 |
+
|
| 55 |
+
// MaxRetries is the maximum retry attempts for pending timestamps
|
| 56 |
+
MaxRetries = 3
|
| 57 |
+
)
|
| 58 |
+
|
| 59 |
+
// Timestamp format
|
| 60 |
+
const (
|
| 61 |
+
// TimestampFormat is the expected GDELT timestamp format: YYYYMMDDHHmmss
|
| 62 |
+
TimestampFormat = "20060102150405"
|
| 63 |
+
|
| 64 |
+
// TimestampLength is the expected length of a valid timestamp
|
| 65 |
+
TimestampLength = 14
|
| 66 |
+
)
|
| 67 |
+
|
| 68 |
+
// File types
|
| 69 |
+
const (
|
| 70 |
+
FileTypeExport = "export"
|
| 71 |
+
FileTypeMentions = "mentions"
|
| 72 |
+
FileTypeGKG = "gkg"
|
| 73 |
+
)
|
| 74 |
+
|
| 75 |
+
// FilesPerTimestamp is the number of GDELT files per timestamp
|
| 76 |
+
const FilesPerTimestamp = 3
|
| 77 |
+
|
| 78 |
+
// CSV parsing configuration
|
| 79 |
+
var (
|
| 80 |
+
// EventColumns is the expected minimum number of columns in export CSV
|
| 81 |
+
EventColumns = 61
|
| 82 |
+
|
| 83 |
+
// MentionColumns is the expected minimum number of columns in mentions CSV
|
| 84 |
+
MentionColumns = 15
|
| 85 |
+
|
| 86 |
+
// GKGColumns is the expected minimum number of columns in GKG CSV
|
| 87 |
+
GKGColumns = 15
|
| 88 |
+
|
| 89 |
+
// MaxLineSize is the maximum line size for CSV scanner (4MB for GKG)
|
| 90 |
+
MaxLineSize = 4 * 1024 * 1024
|
| 91 |
+
|
| 92 |
+
// InitialBufferSize is the initial buffer size for CSV scanner
|
| 93 |
+
InitialBufferSize = 256 * 1024
|
| 94 |
+
)
|
| 95 |
+
|
| 96 |
+
// URL normalization - tracking parameters to remove
|
| 97 |
+
var TrackingParams = []string{
|
| 98 |
+
"utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content",
|
| 99 |
+
"fbclid", "gclid", "dclid", "msclkid",
|
| 100 |
+
"ref", "source", "campaign",
|
| 101 |
+
}
|
internal/downloader/downloader.go
ADDED
|
@@ -0,0 +1,184 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Package downloader provides streaming HTTP download capabilities for GDELT files.
|
| 2 |
+
// It uses io.ReadCloser for true streaming without buffering entire files in memory.
|
| 3 |
+
package downloader
|
| 4 |
+
|
| 5 |
+
import (
|
| 6 |
+
"context"
|
| 7 |
+
"fmt"
|
| 8 |
+
"io"
|
| 9 |
+
"net/http"
|
| 10 |
+
"time"
|
| 11 |
+
|
| 12 |
+
"gdelt-engine/internal/constants"
|
| 13 |
+
|
| 14 |
+
"go.uber.org/zap"
|
| 15 |
+
)
|
| 16 |
+
|
| 17 |
+
// Downloader defines the interface for downloading files.
|
| 18 |
+
// This interface allows for easy mocking in tests.
|
| 19 |
+
type Downloader interface {
|
| 20 |
+
// StreamDownload returns an io.ReadCloser for streaming the file content.
|
| 21 |
+
// The caller is responsible for closing the reader.
|
| 22 |
+
StreamDownload(ctx context.Context, url string) (io.ReadCloser, int64, error)
|
| 23 |
+
}
|
| 24 |
+
|
| 25 |
+
// Compile-time interface verification
|
| 26 |
+
var _ Downloader = (*StreamingDownloader)(nil)
|
| 27 |
+
|
| 28 |
+
// Option is a functional option for configuring StreamingDownloader
|
| 29 |
+
type Option func(*StreamingDownloader)
|
| 30 |
+
|
| 31 |
+
// WithTimeout sets the HTTP client timeout
|
| 32 |
+
func WithTimeout(timeout time.Duration) Option {
|
| 33 |
+
return func(d *StreamingDownloader) {
|
| 34 |
+
d.client.Timeout = timeout
|
| 35 |
+
}
|
| 36 |
+
}
|
| 37 |
+
|
| 38 |
+
// WithLogger sets the logger
|
| 39 |
+
func WithLogger(logger *zap.Logger) Option {
|
| 40 |
+
return func(d *StreamingDownloader) {
|
| 41 |
+
d.logger = logger
|
| 42 |
+
}
|
| 43 |
+
}
|
| 44 |
+
|
| 45 |
+
// WithHTTPClient sets a custom HTTP client
|
| 46 |
+
func WithHTTPClient(client *http.Client) Option {
|
| 47 |
+
return func(d *StreamingDownloader) {
|
| 48 |
+
d.client = client
|
| 49 |
+
}
|
| 50 |
+
}
|
| 51 |
+
|
| 52 |
+
// StreamingDownloader implements Downloader with HTTP streaming.
|
| 53 |
+
// It streams response body directly without loading into memory.
|
| 54 |
+
type StreamingDownloader struct {
|
| 55 |
+
client *http.Client
|
| 56 |
+
logger *zap.Logger
|
| 57 |
+
}
|
| 58 |
+
|
| 59 |
+
// NewStreamingDownloader creates a new StreamingDownloader with functional options.
|
| 60 |
+
func NewStreamingDownloader(opts ...Option) *StreamingDownloader {
|
| 61 |
+
d := &StreamingDownloader{
|
| 62 |
+
client: &http.Client{
|
| 63 |
+
Timeout: constants.DefaultHTTPTimeout,
|
| 64 |
+
},
|
| 65 |
+
logger: zap.NewNop(), // No-op logger by default
|
| 66 |
+
}
|
| 67 |
+
|
| 68 |
+
for _, opt := range opts {
|
| 69 |
+
opt(d)
|
| 70 |
+
}
|
| 71 |
+
|
| 72 |
+
return d
|
| 73 |
+
}
|
| 74 |
+
|
| 75 |
+
// StreamDownload streams the HTTP response body directly.
|
| 76 |
+
// Returns io.ReadCloser for streaming, content length, and any error.
|
| 77 |
+
// The caller MUST close the reader when done.
|
| 78 |
+
func (d *StreamingDownloader) StreamDownload(ctx context.Context, url string) (io.ReadCloser, int64, error) {
|
| 79 |
+
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
| 80 |
+
if err != nil {
|
| 81 |
+
return nil, 0, fmt.Errorf("create request: %w", err)
|
| 82 |
+
}
|
| 83 |
+
|
| 84 |
+
// Set headers for better compatibility
|
| 85 |
+
req.Header.Set("User-Agent", "GDELT-Engine/3.0")
|
| 86 |
+
req.Header.Set("Accept", "*/*")
|
| 87 |
+
|
| 88 |
+
resp, err := d.client.Do(req)
|
| 89 |
+
if err != nil {
|
| 90 |
+
return nil, 0, fmt.Errorf("download failed: %w", err)
|
| 91 |
+
}
|
| 92 |
+
|
| 93 |
+
if resp.StatusCode != http.StatusOK {
|
| 94 |
+
resp.Body.Close()
|
| 95 |
+
return nil, 0, fmt.Errorf("HTTP %d: %s", resp.StatusCode, resp.Status)
|
| 96 |
+
}
|
| 97 |
+
|
| 98 |
+
d.logger.Debug("Download started",
|
| 99 |
+
zap.String("url", url),
|
| 100 |
+
zap.Int64("content_length", resp.ContentLength),
|
| 101 |
+
)
|
| 102 |
+
|
| 103 |
+
return resp.Body, resp.ContentLength, nil
|
| 104 |
+
}
|
| 105 |
+
|
| 106 |
+
// HeadCheckResult represents the result of a HEAD check
|
| 107 |
+
type HeadCheckResult int
|
| 108 |
+
|
| 109 |
+
const (
|
| 110 |
+
// HeadExists means the file exists (HTTP 200)
|
| 111 |
+
HeadExists HeadCheckResult = iota
|
| 112 |
+
// HeadNotFound means the file doesn't exist (HTTP 404)
|
| 113 |
+
HeadNotFound
|
| 114 |
+
// HeadError means there was a transient error
|
| 115 |
+
HeadError
|
| 116 |
+
)
|
| 117 |
+
|
| 118 |
+
// HeadCheck performs a HEAD request to verify file existence without downloading.
|
| 119 |
+
// This is the existential validation layer - verifies GDELT file actually exists.
|
| 120 |
+
func (d *StreamingDownloader) HeadCheck(ctx context.Context, url string) (HeadCheckResult, error) {
|
| 121 |
+
// Use shorter timeout for HEAD requests
|
| 122 |
+
ctx, cancel := context.WithTimeout(ctx, constants.HeadRequestTimeout)
|
| 123 |
+
defer cancel()
|
| 124 |
+
|
| 125 |
+
req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
|
| 126 |
+
if err != nil {
|
| 127 |
+
return HeadError, fmt.Errorf("create request: %w", err)
|
| 128 |
+
}
|
| 129 |
+
|
| 130 |
+
req.Header.Set("User-Agent", "GDELT-Engine/3.0")
|
| 131 |
+
|
| 132 |
+
resp, err := d.client.Do(req)
|
| 133 |
+
if err != nil {
|
| 134 |
+
return HeadError, fmt.Errorf("head request failed: %w", err)
|
| 135 |
+
}
|
| 136 |
+
defer resp.Body.Close()
|
| 137 |
+
|
| 138 |
+
switch resp.StatusCode {
|
| 139 |
+
case http.StatusOK:
|
| 140 |
+
d.logger.Debug("HEAD check passed", zap.String("url", url))
|
| 141 |
+
return HeadExists, nil
|
| 142 |
+
case http.StatusNotFound:
|
| 143 |
+
d.logger.Debug("HEAD check: not found", zap.String("url", url))
|
| 144 |
+
return HeadNotFound, nil
|
| 145 |
+
default:
|
| 146 |
+
return HeadError, fmt.Errorf("HTTP %d: %s", resp.StatusCode, resp.Status)
|
| 147 |
+
}
|
| 148 |
+
}
|
| 149 |
+
|
| 150 |
+
// DownloadResult holds the result of a download operation
|
| 151 |
+
type DownloadResult struct {
|
| 152 |
+
URL string
|
| 153 |
+
ContentLength int64
|
| 154 |
+
Reader io.ReadCloser
|
| 155 |
+
Error error
|
| 156 |
+
}
|
| 157 |
+
|
| 158 |
+
// DownloadAll downloads multiple URLs concurrently and returns results via channel.
|
| 159 |
+
// This is useful for downloading all 3 files for a timestamp in parallel.
|
| 160 |
+
func (d *StreamingDownloader) DownloadAll(ctx context.Context, urls []string) <-chan DownloadResult {
|
| 161 |
+
results := make(chan DownloadResult, len(urls))
|
| 162 |
+
|
| 163 |
+
go func() {
|
| 164 |
+
defer close(results)
|
| 165 |
+
|
| 166 |
+
for _, url := range urls {
|
| 167 |
+
select {
|
| 168 |
+
case <-ctx.Done():
|
| 169 |
+
results <- DownloadResult{URL: url, Error: ctx.Err()}
|
| 170 |
+
return
|
| 171 |
+
default:
|
| 172 |
+
reader, length, err := d.StreamDownload(ctx, url)
|
| 173 |
+
results <- DownloadResult{
|
| 174 |
+
URL: url,
|
| 175 |
+
ContentLength: length,
|
| 176 |
+
Reader: reader,
|
| 177 |
+
Error: err,
|
| 178 |
+
}
|
| 179 |
+
}
|
| 180 |
+
}
|
| 181 |
+
}()
|
| 182 |
+
|
| 183 |
+
return results
|
| 184 |
+
}
|
internal/lookups/lookups.go
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
package lookups
|
| 2 |
+
|
| 3 |
+
import (
|
| 4 |
+
"bufio"
|
| 5 |
+
"context"
|
| 6 |
+
"fmt"
|
| 7 |
+
"io"
|
| 8 |
+
"net/http"
|
| 9 |
+
"strings"
|
| 10 |
+
"sync"
|
| 11 |
+
"time"
|
| 12 |
+
|
| 13 |
+
"go.uber.org/zap"
|
| 14 |
+
)
|
| 15 |
+
|
| 16 |
+
const (
|
| 17 |
+
baseURL = "https://www.gdeltproject.org/data/lookups/"
|
| 18 |
+
eventCodesFile = "CAMEO.eventcodes.txt"
|
| 19 |
+
actorTypesFile = "CAMEO.type.txt"
|
| 20 |
+
countriesFile = "CAMEO.country.txt"
|
| 21 |
+
fetchTimeout = 30 * time.Second
|
| 22 |
+
)
|
| 23 |
+
|
| 24 |
+
// Cache holds in-memory lookup tables.
|
| 25 |
+
type Cache struct {
|
| 26 |
+
mu sync.RWMutex
|
| 27 |
+
eventCodes map[string]string // Event code -> description
|
| 28 |
+
actorTypes map[string]string // Actor type code -> label
|
| 29 |
+
countries map[string]string // Country code -> name
|
| 30 |
+
}
|
| 31 |
+
|
| 32 |
+
var cache = &Cache{
|
| 33 |
+
eventCodes: make(map[string]string),
|
| 34 |
+
actorTypes: make(map[string]string),
|
| 35 |
+
countries: make(map[string]string),
|
| 36 |
+
}
|
| 37 |
+
|
| 38 |
+
// FetchAll downloads and parses all CAMEO lookup tables.
|
| 39 |
+
func FetchAll(ctx context.Context, logger *zap.Logger) error {
|
| 40 |
+
start := time.Now()
|
| 41 |
+
|
| 42 |
+
// Fetch event codes
|
| 43 |
+
eventCodes, err := fetchAndParse(ctx, baseURL+eventCodesFile)
|
| 44 |
+
if err != nil {
|
| 45 |
+
return fmt.Errorf("failed to fetch event codes: %w", err)
|
| 46 |
+
}
|
| 47 |
+
|
| 48 |
+
// Fetch actor types
|
| 49 |
+
actorTypes, err := fetchAndParse(ctx, baseURL+actorTypesFile)
|
| 50 |
+
if err != nil {
|
| 51 |
+
return fmt.Errorf("failed to fetch actor types: %w", err)
|
| 52 |
+
}
|
| 53 |
+
|
| 54 |
+
// Fetch countries
|
| 55 |
+
countries, err := fetchAndParse(ctx, baseURL+countriesFile)
|
| 56 |
+
if err != nil {
|
| 57 |
+
return fmt.Errorf("failed to fetch countries: %w", err)
|
| 58 |
+
}
|
| 59 |
+
|
| 60 |
+
// Update cache atomically
|
| 61 |
+
cache.mu.Lock()
|
| 62 |
+
cache.eventCodes = eventCodes
|
| 63 |
+
cache.actorTypes = actorTypes
|
| 64 |
+
cache.countries = countries
|
| 65 |
+
cache.mu.Unlock()
|
| 66 |
+
|
| 67 |
+
logger.Info("Lookups fetched",
|
| 68 |
+
zap.Int("event_codes", len(eventCodes)),
|
| 69 |
+
zap.Int("actor_types", len(actorTypes)),
|
| 70 |
+
zap.Int("countries", len(countries)),
|
| 71 |
+
zap.Int64("duration_ms", time.Since(start).Milliseconds()),
|
| 72 |
+
)
|
| 73 |
+
|
| 74 |
+
return nil
|
| 75 |
+
}
|
| 76 |
+
|
| 77 |
+
// GetEventDescription returns the description for an event code.
|
| 78 |
+
func GetEventDescription(code string) string {
|
| 79 |
+
cache.mu.RLock()
|
| 80 |
+
defer cache.mu.RUnlock()
|
| 81 |
+
return cache.eventCodes[code]
|
| 82 |
+
}
|
| 83 |
+
|
| 84 |
+
// GetActorType returns the label for an actor type code.
|
| 85 |
+
func GetActorType(code string) string {
|
| 86 |
+
cache.mu.RLock()
|
| 87 |
+
defer cache.mu.RUnlock()
|
| 88 |
+
return cache.actorTypes[code]
|
| 89 |
+
}
|
| 90 |
+
|
| 91 |
+
// GetCountryName returns the country name for a country code.
|
| 92 |
+
func GetCountryName(code string) string {
|
| 93 |
+
cache.mu.RLock()
|
| 94 |
+
defer cache.mu.RUnlock()
|
| 95 |
+
return cache.countries[code]
|
| 96 |
+
}
|
| 97 |
+
|
| 98 |
+
// fetchAndParse downloads a lookup file and parses it into a map.
|
| 99 |
+
func fetchAndParse(ctx context.Context, url string) (map[string]string, error) {
|
| 100 |
+
ctx, cancel := context.WithTimeout(ctx, fetchTimeout)
|
| 101 |
+
defer cancel()
|
| 102 |
+
|
| 103 |
+
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
| 104 |
+
if err != nil {
|
| 105 |
+
return nil, err
|
| 106 |
+
}
|
| 107 |
+
|
| 108 |
+
resp, err := http.DefaultClient.Do(req)
|
| 109 |
+
if err != nil {
|
| 110 |
+
return nil, err
|
| 111 |
+
}
|
| 112 |
+
defer resp.Body.Close()
|
| 113 |
+
|
| 114 |
+
if resp.StatusCode != http.StatusOK {
|
| 115 |
+
return nil, fmt.Errorf("HTTP %d", resp.StatusCode)
|
| 116 |
+
}
|
| 117 |
+
|
| 118 |
+
return parseTabSeparated(resp.Body)
|
| 119 |
+
}
|
| 120 |
+
|
| 121 |
+
// parseTabSeparated parses tab-separated lookup files with header row.
|
| 122 |
+
func parseTabSeparated(r io.Reader) (map[string]string, error) {
|
| 123 |
+
result := make(map[string]string)
|
| 124 |
+
scanner := bufio.NewScanner(r)
|
| 125 |
+
|
| 126 |
+
// Skip header row
|
| 127 |
+
if scanner.Scan() {
|
| 128 |
+
// Header consumed
|
| 129 |
+
}
|
| 130 |
+
|
| 131 |
+
for scanner.Scan() {
|
| 132 |
+
line := strings.TrimSpace(scanner.Text())
|
| 133 |
+
if line == "" {
|
| 134 |
+
continue
|
| 135 |
+
}
|
| 136 |
+
|
| 137 |
+
parts := strings.SplitN(line, "\t", 2)
|
| 138 |
+
if len(parts) == 2 {
|
| 139 |
+
code := strings.TrimSpace(parts[0])
|
| 140 |
+
label := strings.TrimSpace(parts[1])
|
| 141 |
+
if code != "" {
|
| 142 |
+
result[code] = label
|
| 143 |
+
}
|
| 144 |
+
}
|
| 145 |
+
}
|
| 146 |
+
|
| 147 |
+
return result, scanner.Err()
|
| 148 |
+
}
|
internal/parser/parser.go
ADDED
|
@@ -0,0 +1,407 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Package parser provides streaming CSV parsing for GDELT files.
|
| 2 |
+
// It parses data line-by-line using channels for pipeline processing.
|
| 3 |
+
package parser
|
| 4 |
+
|
| 5 |
+
import (
|
| 6 |
+
"archive/zip"
|
| 7 |
+
"bufio"
|
| 8 |
+
"bytes"
|
| 9 |
+
"context"
|
| 10 |
+
"fmt"
|
| 11 |
+
"io"
|
| 12 |
+
"strconv"
|
| 13 |
+
"strings"
|
| 14 |
+
"time"
|
| 15 |
+
|
| 16 |
+
"gdelt-engine/internal/constants"
|
| 17 |
+
"gdelt-engine/internal/schema"
|
| 18 |
+
|
| 19 |
+
"go.uber.org/zap"
|
| 20 |
+
)
|
| 21 |
+
|
| 22 |
+
// ParseResult represents a parsed record sent through channel
|
| 23 |
+
type ParseResult struct {
|
| 24 |
+
// Record is the parsed data (Event, Mention, or GKG)
|
| 25 |
+
Record interface{}
|
| 26 |
+
|
| 27 |
+
// FileType indicates which type of record this is
|
| 28 |
+
FileType string
|
| 29 |
+
|
| 30 |
+
// LineNumber is the line number in the source file
|
| 31 |
+
LineNumber int
|
| 32 |
+
|
| 33 |
+
// Err is any parsing error (nil for successful parse)
|
| 34 |
+
Err error
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
// StreamParser defines the interface for streaming CSV parsers
|
| 38 |
+
type StreamParser interface {
|
| 39 |
+
// ParseStream reads from io.Reader and sends parsed records to channel
|
| 40 |
+
ParseStream(ctx context.Context, r io.Reader, fileType, timestamp string) <-chan ParseResult
|
| 41 |
+
|
| 42 |
+
// ParseZipStream extracts and parses a ZIP stream
|
| 43 |
+
ParseZipStream(ctx context.Context, zipData []byte, fileType, timestamp string) <-chan ParseResult
|
| 44 |
+
}
|
| 45 |
+
|
| 46 |
+
// Compile-time interface verification
|
| 47 |
+
var _ StreamParser = (*CSVStreamParser)(nil)
|
| 48 |
+
|
| 49 |
+
// CSVStreamParser implements StreamParser for GDELT CSV files
|
| 50 |
+
type CSVStreamParser struct {
|
| 51 |
+
logger *zap.Logger
|
| 52 |
+
bufferSize int
|
| 53 |
+
maxLine int
|
| 54 |
+
}
|
| 55 |
+
|
| 56 |
+
// ParserOption is a functional option for parser configuration
|
| 57 |
+
type ParserOption func(*CSVStreamParser)
|
| 58 |
+
|
| 59 |
+
// WithParserLogger sets the logger
|
| 60 |
+
func WithParserLogger(logger *zap.Logger) ParserOption {
|
| 61 |
+
return func(p *CSVStreamParser) {
|
| 62 |
+
p.logger = logger
|
| 63 |
+
}
|
| 64 |
+
}
|
| 65 |
+
|
| 66 |
+
// WithBufferSize sets the scanner buffer size
|
| 67 |
+
func WithBufferSize(size int) ParserOption {
|
| 68 |
+
return func(p *CSVStreamParser) {
|
| 69 |
+
p.bufferSize = size
|
| 70 |
+
}
|
| 71 |
+
}
|
| 72 |
+
|
| 73 |
+
// NewCSVStreamParser creates a new streaming CSV parser
|
| 74 |
+
func NewCSVStreamParser(opts ...ParserOption) *CSVStreamParser {
|
| 75 |
+
p := &CSVStreamParser{
|
| 76 |
+
logger: zap.NewNop(),
|
| 77 |
+
bufferSize: constants.InitialBufferSize,
|
| 78 |
+
maxLine: constants.MaxLineSize,
|
| 79 |
+
}
|
| 80 |
+
|
| 81 |
+
for _, opt := range opts {
|
| 82 |
+
opt(p)
|
| 83 |
+
}
|
| 84 |
+
|
| 85 |
+
return p
|
| 86 |
+
}
|
| 87 |
+
|
| 88 |
+
// ParseZipStream extracts the first file from ZIP data and parses it as a stream.
|
| 89 |
+
// Note: ZIP requires random access, so we need the full data here.
|
| 90 |
+
// The parsing itself is still streamed line-by-line.
|
| 91 |
+
func (p *CSVStreamParser) ParseZipStream(ctx context.Context, zipData []byte, fileType, timestamp string) <-chan ParseResult {
|
| 92 |
+
out := make(chan ParseResult, constants.ChannelBufferSize)
|
| 93 |
+
|
| 94 |
+
go func() {
|
| 95 |
+
defer close(out)
|
| 96 |
+
|
| 97 |
+
// Open ZIP from memory
|
| 98 |
+
zipReader, err := zip.NewReader(bytes.NewReader(zipData), int64(len(zipData)))
|
| 99 |
+
if err != nil {
|
| 100 |
+
out <- ParseResult{Err: fmt.Errorf("zip open failed: %w", err)}
|
| 101 |
+
return
|
| 102 |
+
}
|
| 103 |
+
|
| 104 |
+
if len(zipReader.File) == 0 {
|
| 105 |
+
out <- ParseResult{Err: fmt.Errorf("empty zip file")}
|
| 106 |
+
return
|
| 107 |
+
}
|
| 108 |
+
|
| 109 |
+
// Open first file in ZIP
|
| 110 |
+
csvFile, err := zipReader.File[0].Open()
|
| 111 |
+
if err != nil {
|
| 112 |
+
out <- ParseResult{Err: fmt.Errorf("csv extract failed: %w", err)}
|
| 113 |
+
return
|
| 114 |
+
}
|
| 115 |
+
defer csvFile.Close()
|
| 116 |
+
|
| 117 |
+
// Parse the CSV stream
|
| 118 |
+
for result := range p.ParseStream(ctx, csvFile, fileType, timestamp) {
|
| 119 |
+
select {
|
| 120 |
+
case <-ctx.Done():
|
| 121 |
+
out <- ParseResult{Err: ctx.Err()}
|
| 122 |
+
return
|
| 123 |
+
case out <- result:
|
| 124 |
+
}
|
| 125 |
+
}
|
| 126 |
+
}()
|
| 127 |
+
|
| 128 |
+
return out
|
| 129 |
+
}
|
| 130 |
+
|
| 131 |
+
// ParseStream creates a pipeline that parses CSV line-by-line using a goroutine.
|
| 132 |
+
// It sends parsed records through the returned channel.
|
| 133 |
+
func (p *CSVStreamParser) ParseStream(ctx context.Context, r io.Reader, fileType, timestamp string) <-chan ParseResult {
|
| 134 |
+
out := make(chan ParseResult, constants.ChannelBufferSize)
|
| 135 |
+
|
| 136 |
+
go func() {
|
| 137 |
+
defer close(out)
|
| 138 |
+
|
| 139 |
+
scanner := bufio.NewScanner(r)
|
| 140 |
+
|
| 141 |
+
// Set buffer for large lines (especially for GKG)
|
| 142 |
+
buf := make([]byte, 0, p.bufferSize)
|
| 143 |
+
scanner.Buffer(buf, p.maxLine)
|
| 144 |
+
|
| 145 |
+
lineNum := 0
|
| 146 |
+
skipped := 0
|
| 147 |
+
now := time.Now()
|
| 148 |
+
|
| 149 |
+
for scanner.Scan() {
|
| 150 |
+
lineNum++
|
| 151 |
+
|
| 152 |
+
select {
|
| 153 |
+
case <-ctx.Done():
|
| 154 |
+
out <- ParseResult{Err: ctx.Err()}
|
| 155 |
+
return
|
| 156 |
+
default:
|
| 157 |
+
}
|
| 158 |
+
|
| 159 |
+
line := scanner.Text()
|
| 160 |
+
if line == "" {
|
| 161 |
+
continue
|
| 162 |
+
}
|
| 163 |
+
|
| 164 |
+
record, err := p.parseLine(line, fileType, timestamp, now)
|
| 165 |
+
if err != nil {
|
| 166 |
+
skipped++
|
| 167 |
+
continue // Skip malformed lines
|
| 168 |
+
}
|
| 169 |
+
|
| 170 |
+
out <- ParseResult{
|
| 171 |
+
Record: record,
|
| 172 |
+
FileType: fileType,
|
| 173 |
+
LineNumber: lineNum,
|
| 174 |
+
}
|
| 175 |
+
}
|
| 176 |
+
|
| 177 |
+
if err := scanner.Err(); err != nil {
|
| 178 |
+
out <- ParseResult{Err: fmt.Errorf("scanner error: %w", err)}
|
| 179 |
+
}
|
| 180 |
+
|
| 181 |
+
if skipped > 0 {
|
| 182 |
+
p.logger.Debug("Skipped malformed rows",
|
| 183 |
+
zap.Int("count", skipped),
|
| 184 |
+
zap.String("type", fileType),
|
| 185 |
+
)
|
| 186 |
+
}
|
| 187 |
+
}()
|
| 188 |
+
|
| 189 |
+
return out
|
| 190 |
+
}
|
| 191 |
+
|
| 192 |
+
// parseLine parses a single CSV line based on file type
|
| 193 |
+
func (p *CSVStreamParser) parseLine(line, fileType, timestamp string, now time.Time) (interface{}, error) {
|
| 194 |
+
fields := strings.Split(line, "\t")
|
| 195 |
+
|
| 196 |
+
switch fileType {
|
| 197 |
+
case constants.FileTypeExport:
|
| 198 |
+
return p.parseEvent(fields, timestamp, now)
|
| 199 |
+
case constants.FileTypeMentions:
|
| 200 |
+
return p.parseMention(fields, timestamp, now)
|
| 201 |
+
case constants.FileTypeGKG:
|
| 202 |
+
return p.parseGKG(fields, timestamp, now)
|
| 203 |
+
default:
|
| 204 |
+
return nil, fmt.Errorf("unknown file type: %s", fileType)
|
| 205 |
+
}
|
| 206 |
+
}
|
| 207 |
+
|
| 208 |
+
// parseEvent parses an export CSV line into an Event
|
| 209 |
+
func (p *CSVStreamParser) parseEvent(fields []string, timestamp string, now time.Time) (*schema.Event, error) {
|
| 210 |
+
if len(fields) < constants.EventColumns {
|
| 211 |
+
return nil, fmt.Errorf("insufficient columns: %d < %d", len(fields), constants.EventColumns)
|
| 212 |
+
}
|
| 213 |
+
|
| 214 |
+
event := &schema.Event{
|
| 215 |
+
GlobalEventID: parseInt64(fields[0]),
|
| 216 |
+
Day: parseInt(fields[1]),
|
| 217 |
+
Actor1Name: fields[6],
|
| 218 |
+
Actor1CountryCode: fields[7],
|
| 219 |
+
Actor1Type1Code: fields[12],
|
| 220 |
+
Actor2Name: fields[16],
|
| 221 |
+
Actor2CountryCode: fields[17],
|
| 222 |
+
EventCode: fields[26],
|
| 223 |
+
EventBaseCode: fields[27],
|
| 224 |
+
EventRootCode: fields[28],
|
| 225 |
+
QuadClass: parseInt(fields[29]),
|
| 226 |
+
GoldsteinScale: parseFloat(fields[30]),
|
| 227 |
+
NumMentions: parseInt(fields[31]),
|
| 228 |
+
NumSources: parseInt(fields[32]),
|
| 229 |
+
NumArticles: parseInt(fields[33]),
|
| 230 |
+
AvgTone: parseFloat(fields[34]),
|
| 231 |
+
ActionGeoType: parseInt(fields[51]),
|
| 232 |
+
ActionGeoFullName: fields[52],
|
| 233 |
+
ActionGeoCountryCode: fields[53],
|
| 234 |
+
ActionGeoADM1Code: fields[54],
|
| 235 |
+
ActionGeoLat: parseFloat(fields[56]),
|
| 236 |
+
ActionGeoLong: parseFloat(fields[57]),
|
| 237 |
+
SourceURL: fields[60],
|
| 238 |
+
Timestamp: timestamp,
|
| 239 |
+
ProcessedAt: now,
|
| 240 |
+
}
|
| 241 |
+
|
| 242 |
+
return event, nil
|
| 243 |
+
}
|
| 244 |
+
|
| 245 |
+
// parseMention parses a mentions CSV line into a Mention
|
| 246 |
+
func (p *CSVStreamParser) parseMention(fields []string, timestamp string, now time.Time) (*schema.Mention, error) {
|
| 247 |
+
if len(fields) < constants.MentionColumns {
|
| 248 |
+
return nil, fmt.Errorf("insufficient columns: %d < %d", len(fields), constants.MentionColumns)
|
| 249 |
+
}
|
| 250 |
+
|
| 251 |
+
mention := &schema.Mention{
|
| 252 |
+
GlobalEventID: parseInt64(fields[0]),
|
| 253 |
+
EventTimeDate: parseInt64(fields[1]),
|
| 254 |
+
MentionTimeDate: parseInt64(fields[2]),
|
| 255 |
+
MentionType: parseInt(fields[3]),
|
| 256 |
+
MentionSourceName: fields[4],
|
| 257 |
+
MentionIdentifier: fields[5],
|
| 258 |
+
SentenceID: parseInt(fields[6]),
|
| 259 |
+
Actor1CharOffset: parseInt(fields[7]),
|
| 260 |
+
Actor2CharOffset: parseInt(fields[8]),
|
| 261 |
+
ActionCharOffset: parseInt(fields[9]),
|
| 262 |
+
InRawText: parseInt(fields[10]),
|
| 263 |
+
Confidence: parseInt(fields[11]),
|
| 264 |
+
MentionDocLen: parseInt(fields[12]),
|
| 265 |
+
MentionDocTone: parseFloat(fields[13]),
|
| 266 |
+
MentionDocTranslation: fields[14],
|
| 267 |
+
Timestamp: timestamp,
|
| 268 |
+
ProcessedAt: now,
|
| 269 |
+
}
|
| 270 |
+
|
| 271 |
+
return mention, nil
|
| 272 |
+
}
|
| 273 |
+
|
| 274 |
+
// parseGKG parses a GKG CSV line into a GKG record
|
| 275 |
+
func (p *CSVStreamParser) parseGKG(fields []string, timestamp string, now time.Time) (*schema.GKG, error) {
|
| 276 |
+
if len(fields) < constants.GKGColumns {
|
| 277 |
+
return nil, fmt.Errorf("insufficient columns: %d < %d", len(fields), constants.GKGColumns)
|
| 278 |
+
}
|
| 279 |
+
|
| 280 |
+
gkg := &schema.GKG{
|
| 281 |
+
GKGRECORDID: fields[0],
|
| 282 |
+
Date: parseInt64(fields[1]),
|
| 283 |
+
SourceCollectionID: parseInt(fields[2]),
|
| 284 |
+
SourceCommonName: fields[3],
|
| 285 |
+
DocumentIdentifier: fields[4],
|
| 286 |
+
Timestamp: timestamp,
|
| 287 |
+
ProcessedAt: now,
|
| 288 |
+
}
|
| 289 |
+
|
| 290 |
+
// Parse optional fields
|
| 291 |
+
if len(fields) > 5 {
|
| 292 |
+
gkg.Counts = fields[5]
|
| 293 |
+
}
|
| 294 |
+
if len(fields) > 6 {
|
| 295 |
+
gkg.V2Counts = fields[6]
|
| 296 |
+
}
|
| 297 |
+
if len(fields) > 7 {
|
| 298 |
+
gkg.Themes = splitSemicolon(fields[7])
|
| 299 |
+
}
|
| 300 |
+
if len(fields) > 8 {
|
| 301 |
+
gkg.V2Themes = splitSemicolon(fields[8])
|
| 302 |
+
}
|
| 303 |
+
if len(fields) > 9 {
|
| 304 |
+
gkg.Locations = splitSemicolon(fields[9])
|
| 305 |
+
}
|
| 306 |
+
if len(fields) > 10 {
|
| 307 |
+
gkg.V2Locations = splitSemicolon(fields[10])
|
| 308 |
+
}
|
| 309 |
+
if len(fields) > 11 {
|
| 310 |
+
gkg.Persons = splitSemicolon(fields[11])
|
| 311 |
+
}
|
| 312 |
+
if len(fields) > 12 {
|
| 313 |
+
gkg.V2Persons = splitSemicolon(fields[12])
|
| 314 |
+
}
|
| 315 |
+
if len(fields) > 13 {
|
| 316 |
+
gkg.Organizations = splitSemicolon(fields[13])
|
| 317 |
+
}
|
| 318 |
+
if len(fields) > 14 {
|
| 319 |
+
gkg.V2Organizations = splitSemicolon(fields[14])
|
| 320 |
+
}
|
| 321 |
+
if len(fields) > 15 {
|
| 322 |
+
gkg.V2Tone = fields[15]
|
| 323 |
+
}
|
| 324 |
+
if len(fields) > 16 {
|
| 325 |
+
gkg.Dates = fields[16]
|
| 326 |
+
}
|
| 327 |
+
if len(fields) > 17 {
|
| 328 |
+
gkg.GCAM = fields[17]
|
| 329 |
+
}
|
| 330 |
+
if len(fields) > 18 {
|
| 331 |
+
gkg.SharingImage = fields[18]
|
| 332 |
+
}
|
| 333 |
+
if len(fields) > 19 {
|
| 334 |
+
gkg.RelatedImages = splitSemicolon(fields[19])
|
| 335 |
+
}
|
| 336 |
+
if len(fields) > 20 {
|
| 337 |
+
gkg.SocialImageEmbeds = splitSemicolon(fields[20])
|
| 338 |
+
}
|
| 339 |
+
if len(fields) > 21 {
|
| 340 |
+
gkg.SocialVideoEmbeds = splitSemicolon(fields[21])
|
| 341 |
+
}
|
| 342 |
+
if len(fields) > 22 {
|
| 343 |
+
gkg.Quotations = splitSemicolon(fields[22])
|
| 344 |
+
}
|
| 345 |
+
if len(fields) > 23 {
|
| 346 |
+
gkg.AllNames = splitSemicolon(fields[23])
|
| 347 |
+
}
|
| 348 |
+
if len(fields) > 24 {
|
| 349 |
+
gkg.Amounts = splitSemicolon(fields[24])
|
| 350 |
+
}
|
| 351 |
+
if len(fields) > 25 {
|
| 352 |
+
gkg.TranslationInfo = fields[25]
|
| 353 |
+
}
|
| 354 |
+
if len(fields) > 26 {
|
| 355 |
+
gkg.Extras = fields[26]
|
| 356 |
+
}
|
| 357 |
+
|
| 358 |
+
return gkg, nil
|
| 359 |
+
}
|
| 360 |
+
|
| 361 |
+
// Helper functions
|
| 362 |
+
|
| 363 |
+
func parseInt(s string) int {
|
| 364 |
+
s = strings.TrimSpace(s)
|
| 365 |
+
if s == "" {
|
| 366 |
+
return 0
|
| 367 |
+
}
|
| 368 |
+
v, _ := strconv.Atoi(s)
|
| 369 |
+
return v
|
| 370 |
+
}
|
| 371 |
+
|
| 372 |
+
func parseInt64(s string) int64 {
|
| 373 |
+
s = strings.TrimSpace(s)
|
| 374 |
+
if s == "" {
|
| 375 |
+
return 0
|
| 376 |
+
}
|
| 377 |
+
v, _ := strconv.ParseInt(s, 10, 64)
|
| 378 |
+
return v
|
| 379 |
+
}
|
| 380 |
+
|
| 381 |
+
func parseFloat(s string) *float64 {
|
| 382 |
+
s = strings.TrimSpace(s)
|
| 383 |
+
if s == "" {
|
| 384 |
+
return nil
|
| 385 |
+
}
|
| 386 |
+
v, err := strconv.ParseFloat(s, 64)
|
| 387 |
+
if err != nil {
|
| 388 |
+
return nil
|
| 389 |
+
}
|
| 390 |
+
return &v
|
| 391 |
+
}
|
| 392 |
+
|
| 393 |
+
func splitSemicolon(s string) []string {
|
| 394 |
+
s = strings.TrimSpace(s)
|
| 395 |
+
if s == "" {
|
| 396 |
+
return nil
|
| 397 |
+
}
|
| 398 |
+
parts := strings.Split(s, ";")
|
| 399 |
+
var result []string
|
| 400 |
+
for _, p := range parts {
|
| 401 |
+
p = strings.TrimSpace(p)
|
| 402 |
+
if p != "" {
|
| 403 |
+
result = append(result, p)
|
| 404 |
+
}
|
| 405 |
+
}
|
| 406 |
+
return result
|
| 407 |
+
}
|
internal/processor/processor.go
ADDED
|
@@ -0,0 +1,686 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Package processor provides the main processing logic for GDELT timestamps.
|
| 2 |
+
// It uses parallel processing with bounded concurrency, structural validation,
|
| 3 |
+
// and existential validation via HEAD probing.
|
| 4 |
+
package processor
|
| 5 |
+
|
| 6 |
+
import (
|
| 7 |
+
"context"
|
| 8 |
+
"crypto/sha256"
|
| 9 |
+
"encoding/hex"
|
| 10 |
+
"fmt"
|
| 11 |
+
"io"
|
| 12 |
+
"net/url"
|
| 13 |
+
"strings"
|
| 14 |
+
"sync"
|
| 15 |
+
"time"
|
| 16 |
+
|
| 17 |
+
"gdelt-engine/internal/constants"
|
| 18 |
+
"gdelt-engine/internal/downloader"
|
| 19 |
+
"gdelt-engine/internal/parser"
|
| 20 |
+
"gdelt-engine/internal/schema"
|
| 21 |
+
|
| 22 |
+
"go.uber.org/zap"
|
| 23 |
+
)
|
| 24 |
+
|
| 25 |
+
// Storage defines the interface for database operations.
|
| 26 |
+
type Storage interface {
|
| 27 |
+
IsTimestampCompleted(ctx context.Context, timestamp string) (bool, error)
|
| 28 |
+
GetTimestampStatus(ctx context.Context, timestamp string) (*schema.CompletedTimestamp, error)
|
| 29 |
+
MarkTimestampStarted(ctx context.Context, timestamp string) error
|
| 30 |
+
MarkTimestampCompleted(ctx context.Context, ts *schema.CompletedTimestamp) error
|
| 31 |
+
GetAllTimestamps(ctx context.Context) (completed []schema.CompletedTimestamp, processing []string, err error)
|
| 32 |
+
MarkURLProcessed(ctx context.Context, url *schema.ProcessedURL) error
|
| 33 |
+
GetProcessedURLs(ctx context.Context, timestamp string) ([]schema.ProcessedURL, error)
|
| 34 |
+
BulkInsertArticles(ctx context.Context, articles []schema.Article) (int, error)
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
// ValidationResult represents the result of timestamp validation
|
| 38 |
+
type ValidationResult struct {
|
| 39 |
+
Timestamp string
|
| 40 |
+
Valid bool
|
| 41 |
+
Reason string
|
| 42 |
+
Status schema.ProcessingStatus
|
| 43 |
+
}
|
| 44 |
+
|
| 45 |
+
// Task represents a single file processing task
|
| 46 |
+
type Task struct {
|
| 47 |
+
Timestamp string
|
| 48 |
+
URL string
|
| 49 |
+
FileType string
|
| 50 |
+
}
|
| 51 |
+
|
| 52 |
+
// Result represents the result of processing a single file
|
| 53 |
+
type Result struct {
|
| 54 |
+
Task Task
|
| 55 |
+
RowCount int
|
| 56 |
+
ArticleCount int
|
| 57 |
+
Duration time.Duration
|
| 58 |
+
Err error
|
| 59 |
+
}
|
| 60 |
+
|
| 61 |
+
// ProcessorOption is a functional option for Processor
|
| 62 |
+
type ProcessorOption func(*Processor)
|
| 63 |
+
|
| 64 |
+
func WithWorkers(n int) ProcessorOption {
|
| 65 |
+
return func(p *Processor) { p.workers = n }
|
| 66 |
+
}
|
| 67 |
+
|
| 68 |
+
func WithTimeout(t time.Duration) ProcessorOption {
|
| 69 |
+
return func(p *Processor) { p.timeout = t }
|
| 70 |
+
}
|
| 71 |
+
|
| 72 |
+
func WithBatchSize(n int) ProcessorOption {
|
| 73 |
+
return func(p *Processor) { p.batchSize = n }
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
func WithMaxParallelTimestamps(n int) ProcessorOption {
|
| 77 |
+
return func(p *Processor) { p.maxParallelTimestamps = n }
|
| 78 |
+
}
|
| 79 |
+
|
| 80 |
+
// Processor handles concurrent timestamp processing
|
| 81 |
+
type Processor struct {
|
| 82 |
+
workers int
|
| 83 |
+
timeout time.Duration
|
| 84 |
+
batchSize int
|
| 85 |
+
maxParallelTimestamps int
|
| 86 |
+
downloader *downloader.StreamingDownloader
|
| 87 |
+
parser parser.StreamParser
|
| 88 |
+
storage Storage
|
| 89 |
+
logger *zap.Logger
|
| 90 |
+
|
| 91 |
+
// Semaphore for limiting parallel timestamps
|
| 92 |
+
semaphore chan struct{}
|
| 93 |
+
|
| 94 |
+
// Track processing state
|
| 95 |
+
mu sync.RWMutex
|
| 96 |
+
processing map[string]bool
|
| 97 |
+
}
|
| 98 |
+
|
| 99 |
+
// NewProcessor creates a new Processor with functional options
|
| 100 |
+
func NewProcessor(storage Storage, logger *zap.Logger, opts ...ProcessorOption) *Processor {
|
| 101 |
+
p := &Processor{
|
| 102 |
+
workers: constants.DefaultWorkers,
|
| 103 |
+
timeout: constants.DefaultTimeout,
|
| 104 |
+
batchSize: constants.DefaultBatchSize,
|
| 105 |
+
maxParallelTimestamps: constants.MaxParallelTimestamps,
|
| 106 |
+
storage: storage,
|
| 107 |
+
logger: logger,
|
| 108 |
+
processing: make(map[string]bool),
|
| 109 |
+
}
|
| 110 |
+
|
| 111 |
+
for _, opt := range opts {
|
| 112 |
+
opt(p)
|
| 113 |
+
}
|
| 114 |
+
|
| 115 |
+
// Initialize semaphore for parallel timestamp limit
|
| 116 |
+
p.semaphore = make(chan struct{}, p.maxParallelTimestamps)
|
| 117 |
+
|
| 118 |
+
// Initialize downloader and parser
|
| 119 |
+
p.downloader = downloader.NewStreamingDownloader(
|
| 120 |
+
downloader.WithLogger(logger),
|
| 121 |
+
)
|
| 122 |
+
p.parser = parser.NewCSVStreamParser(
|
| 123 |
+
parser.WithParserLogger(logger),
|
| 124 |
+
)
|
| 125 |
+
|
| 126 |
+
return p
|
| 127 |
+
}
|
| 128 |
+
|
| 129 |
+
// ValidateTimestampStructural performs structural validation (cheap, deterministic).
|
| 130 |
+
// Checks: length, parsable, minute ∈ {00,15,30,45}, not in future.
|
| 131 |
+
func ValidateTimestampStructural(ts string) *ValidationResult {
|
| 132 |
+
result := &ValidationResult{Timestamp: ts}
|
| 133 |
+
|
| 134 |
+
// Check length
|
| 135 |
+
if len(ts) != constants.TimestampLength {
|
| 136 |
+
result.Valid = false
|
| 137 |
+
result.Reason = fmt.Sprintf("invalid length: expected %d, got %d", constants.TimestampLength, len(ts))
|
| 138 |
+
result.Status = schema.StatusFailed
|
| 139 |
+
return result
|
| 140 |
+
}
|
| 141 |
+
|
| 142 |
+
// Parse as time
|
| 143 |
+
t, err := time.Parse(constants.TimestampFormat, ts)
|
| 144 |
+
if err != nil {
|
| 145 |
+
result.Valid = false
|
| 146 |
+
result.Reason = fmt.Sprintf("invalid format: %v", err)
|
| 147 |
+
result.Status = schema.StatusFailed
|
| 148 |
+
return result
|
| 149 |
+
}
|
| 150 |
+
|
| 151 |
+
// Check minute is valid GDELT interval (00, 15, 30, 45)
|
| 152 |
+
m := t.Minute()
|
| 153 |
+
if m%15 != 0 {
|
| 154 |
+
result.Valid = false
|
| 155 |
+
result.Reason = fmt.Sprintf("invalid minute %d: must be 00, 15, 30, or 45", m)
|
| 156 |
+
result.Status = schema.StatusFailed
|
| 157 |
+
return result
|
| 158 |
+
}
|
| 159 |
+
|
| 160 |
+
// Check not in future
|
| 161 |
+
if t.After(time.Now().UTC()) {
|
| 162 |
+
result.Valid = false
|
| 163 |
+
result.Reason = "timestamp is in the future"
|
| 164 |
+
result.Status = schema.StatusFailed
|
| 165 |
+
return result
|
| 166 |
+
}
|
| 167 |
+
|
| 168 |
+
result.Valid = true
|
| 169 |
+
return result
|
| 170 |
+
}
|
| 171 |
+
|
| 172 |
+
// isWithinGraceWindow checks if timestamp is within the grace window for recent data
|
| 173 |
+
func isWithinGraceWindow(ts string) bool {
|
| 174 |
+
t, err := time.Parse(constants.TimestampFormat, ts)
|
| 175 |
+
if err != nil {
|
| 176 |
+
return false
|
| 177 |
+
}
|
| 178 |
+
graceTime := time.Now().UTC().Add(-time.Duration(constants.GraceWindowMinutes) * time.Minute)
|
| 179 |
+
return t.After(graceTime)
|
| 180 |
+
}
|
| 181 |
+
|
| 182 |
+
// QueueTimestamps validates and queues timestamps for processing.
|
| 183 |
+
// Returns per-timestamp validation results.
|
| 184 |
+
func (p *Processor) QueueTimestamps(ctx context.Context, timestamps []string) (queued []string, rejected []ValidationResult) {
|
| 185 |
+
p.mu.Lock()
|
| 186 |
+
defer p.mu.Unlock()
|
| 187 |
+
|
| 188 |
+
for _, ts := range timestamps {
|
| 189 |
+
// Layer 1: Structural validation
|
| 190 |
+
validation := ValidateTimestampStructural(ts)
|
| 191 |
+
if !validation.Valid {
|
| 192 |
+
rejected = append(rejected, *validation)
|
| 193 |
+
p.logger.Debug("Rejected invalid timestamp",
|
| 194 |
+
zap.String("timestamp", ts),
|
| 195 |
+
zap.String("reason", validation.Reason),
|
| 196 |
+
)
|
| 197 |
+
continue
|
| 198 |
+
}
|
| 199 |
+
|
| 200 |
+
// Check if already processing
|
| 201 |
+
if p.processing[ts] {
|
| 202 |
+
rejected = append(rejected, ValidationResult{
|
| 203 |
+
Timestamp: ts,
|
| 204 |
+
Valid: false,
|
| 205 |
+
Reason: "already processing",
|
| 206 |
+
Status: schema.StatusProcessing,
|
| 207 |
+
})
|
| 208 |
+
continue
|
| 209 |
+
}
|
| 210 |
+
|
| 211 |
+
// Check if already completed
|
| 212 |
+
completed, err := p.storage.IsTimestampCompleted(ctx, ts)
|
| 213 |
+
if err != nil {
|
| 214 |
+
p.logger.Error("Failed to check timestamp status", zap.String("timestamp", ts), zap.Error(err))
|
| 215 |
+
continue
|
| 216 |
+
}
|
| 217 |
+
if completed {
|
| 218 |
+
rejected = append(rejected, ValidationResult{
|
| 219 |
+
Timestamp: ts,
|
| 220 |
+
Valid: false,
|
| 221 |
+
Reason: "already completed",
|
| 222 |
+
Status: schema.StatusCompleted,
|
| 223 |
+
})
|
| 224 |
+
continue
|
| 225 |
+
}
|
| 226 |
+
|
| 227 |
+
// Mark as processing
|
| 228 |
+
p.processing[ts] = true
|
| 229 |
+
queued = append(queued, ts)
|
| 230 |
+
}
|
| 231 |
+
|
| 232 |
+
// Start background processing for queued timestamps
|
| 233 |
+
if len(queued) > 0 {
|
| 234 |
+
go p.processTimestampsParallel(context.Background(), queued)
|
| 235 |
+
}
|
| 236 |
+
|
| 237 |
+
return queued, rejected
|
| 238 |
+
}
|
| 239 |
+
|
| 240 |
+
// processTimestampsParallel processes multiple timestamps in parallel with bounded concurrency.
|
| 241 |
+
func (p *Processor) processTimestampsParallel(ctx context.Context, timestamps []string) {
|
| 242 |
+
var wg sync.WaitGroup
|
| 243 |
+
|
| 244 |
+
for _, ts := range timestamps {
|
| 245 |
+
wg.Add(1)
|
| 246 |
+
go func(timestamp string) {
|
| 247 |
+
defer wg.Done()
|
| 248 |
+
|
| 249 |
+
// Acquire semaphore (limit parallel timestamps)
|
| 250 |
+
p.semaphore <- struct{}{}
|
| 251 |
+
defer func() { <-p.semaphore }()
|
| 252 |
+
|
| 253 |
+
p.processTimestamp(ctx, timestamp)
|
| 254 |
+
}(ts)
|
| 255 |
+
}
|
| 256 |
+
|
| 257 |
+
wg.Wait()
|
| 258 |
+
}
|
| 259 |
+
|
| 260 |
+
// processTimestamp processes a single timestamp with existential validation
|
| 261 |
+
func (p *Processor) processTimestamp(ctx context.Context, timestamp string) {
|
| 262 |
+
start := time.Now()
|
| 263 |
+
|
| 264 |
+
p.logger.Info("Processing timestamp", zap.String("timestamp", timestamp))
|
| 265 |
+
|
| 266 |
+
// Mark as started in database
|
| 267 |
+
if err := p.storage.MarkTimestampStarted(ctx, timestamp); err != nil {
|
| 268 |
+
p.logger.Error("Failed to mark timestamp started", zap.Error(err))
|
| 269 |
+
}
|
| 270 |
+
|
| 271 |
+
// Create timeout context
|
| 272 |
+
ctx, cancel := context.WithTimeout(ctx, p.timeout)
|
| 273 |
+
defer cancel()
|
| 274 |
+
|
| 275 |
+
// Generate tasks for all 3 files
|
| 276 |
+
tasks := p.generateTasks(timestamp)
|
| 277 |
+
|
| 278 |
+
// Layer 2: Existential validation via HEAD probing
|
| 279 |
+
// Check if at least the export file exists
|
| 280 |
+
exportURL := fmt.Sprintf(constants.ExportURLTemplate, timestamp)
|
| 281 |
+
headResult, err := p.downloader.HeadCheck(ctx, exportURL)
|
| 282 |
+
|
| 283 |
+
if headResult == downloader.HeadNotFound {
|
| 284 |
+
// Check if within grace window
|
| 285 |
+
if isWithinGraceWindow(timestamp) {
|
| 286 |
+
p.logger.Info("Timestamp not yet published, within grace window",
|
| 287 |
+
zap.String("timestamp", timestamp),
|
| 288 |
+
)
|
| 289 |
+
p.markTimestampResult(ctx, timestamp, start, schema.StatusPending,
|
| 290 |
+
"GDELT file not yet published (within grace window)", 0, 0, 0)
|
| 291 |
+
} else {
|
| 292 |
+
p.logger.Warn("Timestamp does not exist on GDELT",
|
| 293 |
+
zap.String("timestamp", timestamp),
|
| 294 |
+
)
|
| 295 |
+
p.markTimestampResult(ctx, timestamp, start, schema.StatusFailed,
|
| 296 |
+
"GDELT file does not exist", 0, 0, 0)
|
| 297 |
+
}
|
| 298 |
+
p.removeFromProcessing(timestamp)
|
| 299 |
+
return
|
| 300 |
+
}
|
| 301 |
+
|
| 302 |
+
if headResult == downloader.HeadError && err != nil {
|
| 303 |
+
p.logger.Error("HEAD check failed", zap.String("timestamp", timestamp), zap.Error(err))
|
| 304 |
+
p.markTimestampResult(ctx, timestamp, start, schema.StatusFailed,
|
| 305 |
+
fmt.Sprintf("HEAD check failed: %v", err), 0, 0, 0)
|
| 306 |
+
p.removeFromProcessing(timestamp)
|
| 307 |
+
return
|
| 308 |
+
}
|
| 309 |
+
|
| 310 |
+
// Process files in parallel with batching
|
| 311 |
+
results := p.processFilesParallel(ctx, tasks)
|
| 312 |
+
|
| 313 |
+
// Aggregate results
|
| 314 |
+
var totalRows, totalArticles, filesProcessed int
|
| 315 |
+
var lastError error
|
| 316 |
+
|
| 317 |
+
for result := range results {
|
| 318 |
+
if result.Err != nil {
|
| 319 |
+
p.logger.Error("Task failed",
|
| 320 |
+
zap.String("file_type", result.Task.FileType),
|
| 321 |
+
zap.Error(result.Err),
|
| 322 |
+
)
|
| 323 |
+
lastError = result.Err
|
| 324 |
+
continue
|
| 325 |
+
}
|
| 326 |
+
|
| 327 |
+
filesProcessed++
|
| 328 |
+
totalRows += result.RowCount
|
| 329 |
+
totalArticles += result.ArticleCount
|
| 330 |
+
|
| 331 |
+
p.logger.Info("File processed",
|
| 332 |
+
zap.String("file_type", result.Task.FileType),
|
| 333 |
+
zap.Int("rows", result.RowCount),
|
| 334 |
+
zap.Int("articles", result.ArticleCount),
|
| 335 |
+
zap.Duration("duration", result.Duration),
|
| 336 |
+
)
|
| 337 |
+
}
|
| 338 |
+
|
| 339 |
+
// Mark completion
|
| 340 |
+
status := schema.StatusCompleted
|
| 341 |
+
errMsg := ""
|
| 342 |
+
if lastError != nil {
|
| 343 |
+
status = schema.StatusFailed
|
| 344 |
+
errMsg = lastError.Error()
|
| 345 |
+
}
|
| 346 |
+
|
| 347 |
+
p.markTimestampResult(ctx, timestamp, start, status, errMsg, filesProcessed, totalRows, totalArticles)
|
| 348 |
+
p.removeFromProcessing(timestamp)
|
| 349 |
+
|
| 350 |
+
p.logger.Info("Timestamp completed",
|
| 351 |
+
zap.String("timestamp", timestamp),
|
| 352 |
+
zap.Int("files", filesProcessed),
|
| 353 |
+
zap.Int("rows", totalRows),
|
| 354 |
+
zap.Int("articles", totalArticles),
|
| 355 |
+
zap.Duration("duration", time.Since(start)),
|
| 356 |
+
)
|
| 357 |
+
}
|
| 358 |
+
|
| 359 |
+
// processFilesParallel processes multiple files in parallel with batching
|
| 360 |
+
func (p *Processor) processFilesParallel(ctx context.Context, tasks []Task) <-chan Result {
|
| 361 |
+
results := make(chan Result, len(tasks))
|
| 362 |
+
var wg sync.WaitGroup
|
| 363 |
+
|
| 364 |
+
// Process all files in parallel (limited by constants.MaxParallelFiles)
|
| 365 |
+
fileSemaphore := make(chan struct{}, constants.MaxParallelFiles)
|
| 366 |
+
|
| 367 |
+
for _, task := range tasks {
|
| 368 |
+
wg.Add(1)
|
| 369 |
+
go func(t Task) {
|
| 370 |
+
defer wg.Done()
|
| 371 |
+
|
| 372 |
+
// Acquire file semaphore
|
| 373 |
+
fileSemaphore <- struct{}{}
|
| 374 |
+
defer func() { <-fileSemaphore }()
|
| 375 |
+
|
| 376 |
+
result := p.processTask(ctx, t)
|
| 377 |
+
results <- result
|
| 378 |
+
}(task)
|
| 379 |
+
}
|
| 380 |
+
|
| 381 |
+
go func() {
|
| 382 |
+
wg.Wait()
|
| 383 |
+
close(results)
|
| 384 |
+
}()
|
| 385 |
+
|
| 386 |
+
return results
|
| 387 |
+
}
|
| 388 |
+
|
| 389 |
+
// markTimestampResult saves the timestamp processing result to database
|
| 390 |
+
func (p *Processor) markTimestampResult(ctx context.Context, timestamp string, start time.Time,
|
| 391 |
+
status schema.ProcessingStatus, errMsg string, filesProcessed, totalRows, totalArticles int) {
|
| 392 |
+
|
| 393 |
+
completedAt := time.Now()
|
| 394 |
+
ts := &schema.CompletedTimestamp{
|
| 395 |
+
Timestamp: timestamp,
|
| 396 |
+
Status: status,
|
| 397 |
+
StartedAt: start,
|
| 398 |
+
CompletedAt: &completedAt,
|
| 399 |
+
FilesTotal: constants.FilesPerTimestamp,
|
| 400 |
+
FilesProcessed: filesProcessed,
|
| 401 |
+
ArticlesCount: totalArticles,
|
| 402 |
+
TotalRows: totalRows,
|
| 403 |
+
DurationMs: time.Since(start).Milliseconds(),
|
| 404 |
+
Error: errMsg,
|
| 405 |
+
}
|
| 406 |
+
|
| 407 |
+
if err := p.storage.MarkTimestampCompleted(ctx, ts); err != nil {
|
| 408 |
+
p.logger.Error("Failed to mark timestamp result", zap.Error(err))
|
| 409 |
+
}
|
| 410 |
+
}
|
| 411 |
+
|
| 412 |
+
// removeFromProcessing removes a timestamp from the processing map
|
| 413 |
+
func (p *Processor) removeFromProcessing(timestamp string) {
|
| 414 |
+
p.mu.Lock()
|
| 415 |
+
delete(p.processing, timestamp)
|
| 416 |
+
p.mu.Unlock()
|
| 417 |
+
}
|
| 418 |
+
|
| 419 |
+
// processTask handles a single file download and processing
|
| 420 |
+
func (p *Processor) processTask(ctx context.Context, task Task) Result {
|
| 421 |
+
start := time.Now()
|
| 422 |
+
|
| 423 |
+
// Download stream
|
| 424 |
+
reader, _, err := p.downloader.StreamDownload(ctx, task.URL)
|
| 425 |
+
if err != nil {
|
| 426 |
+
return Result{Task: task, Err: fmt.Errorf("download: %w", err)}
|
| 427 |
+
}
|
| 428 |
+
defer reader.Close()
|
| 429 |
+
|
| 430 |
+
// Read ZIP content (required for zip.NewReader)
|
| 431 |
+
zipData, err := io.ReadAll(reader)
|
| 432 |
+
if err != nil {
|
| 433 |
+
return Result{Task: task, Err: fmt.Errorf("read zip: %w", err)}
|
| 434 |
+
}
|
| 435 |
+
|
| 436 |
+
// Parse ZIP stream
|
| 437 |
+
records := p.parser.ParseZipStream(ctx, zipData, task.FileType, task.Timestamp)
|
| 438 |
+
|
| 439 |
+
// Collect and extract articles
|
| 440 |
+
articles := make(map[string]*schema.Article)
|
| 441 |
+
var rowCount int
|
| 442 |
+
|
| 443 |
+
for result := range records {
|
| 444 |
+
if result.Err != nil {
|
| 445 |
+
if result.Err == context.Canceled || result.Err == context.DeadlineExceeded {
|
| 446 |
+
return Result{Task: task, Err: result.Err}
|
| 447 |
+
}
|
| 448 |
+
continue
|
| 449 |
+
}
|
| 450 |
+
|
| 451 |
+
rowCount++
|
| 452 |
+
|
| 453 |
+
// Extract URL based on record type
|
| 454 |
+
url := p.extractURL(result.Record, task.FileType)
|
| 455 |
+
if url == "" {
|
| 456 |
+
continue
|
| 457 |
+
}
|
| 458 |
+
|
| 459 |
+
// Create or update article
|
| 460 |
+
normalized := normalizeURL(url)
|
| 461 |
+
if normalized == "" {
|
| 462 |
+
continue
|
| 463 |
+
}
|
| 464 |
+
|
| 465 |
+
id := hashURL(normalized)
|
| 466 |
+
if existing, ok := articles[id]; ok {
|
| 467 |
+
existing.MentionCount++
|
| 468 |
+
if !contains(existing.Sources, task.FileType) {
|
| 469 |
+
existing.Sources = append(existing.Sources, task.FileType)
|
| 470 |
+
}
|
| 471 |
+
if task.FileType == constants.FileTypeGKG {
|
| 472 |
+
p.enrichArticle(existing, result.Record)
|
| 473 |
+
}
|
| 474 |
+
} else {
|
| 475 |
+
article := &schema.Article{
|
| 476 |
+
ID: id,
|
| 477 |
+
URL: url,
|
| 478 |
+
NormalizedURL: normalized,
|
| 479 |
+
Sources: []string{task.FileType},
|
| 480 |
+
Timestamp: task.Timestamp,
|
| 481 |
+
MentionCount: 1,
|
| 482 |
+
FirstSeen: time.Now(),
|
| 483 |
+
LastSeen: time.Now(),
|
| 484 |
+
}
|
| 485 |
+
if task.FileType == constants.FileTypeGKG {
|
| 486 |
+
p.enrichArticle(article, result.Record)
|
| 487 |
+
}
|
| 488 |
+
articles[id] = article
|
| 489 |
+
}
|
| 490 |
+
}
|
| 491 |
+
|
| 492 |
+
// Save articles in batches
|
| 493 |
+
articleSlice := make([]schema.Article, 0, len(articles))
|
| 494 |
+
for _, a := range articles {
|
| 495 |
+
articleSlice = append(articleSlice, *a)
|
| 496 |
+
}
|
| 497 |
+
|
| 498 |
+
insertedCount := 0
|
| 499 |
+
if len(articleSlice) > 0 {
|
| 500 |
+
inserted, err := p.storage.BulkInsertArticles(ctx, articleSlice)
|
| 501 |
+
if err != nil {
|
| 502 |
+
p.logger.Error("Failed to insert articles", zap.Error(err))
|
| 503 |
+
}
|
| 504 |
+
insertedCount = inserted
|
| 505 |
+
}
|
| 506 |
+
|
| 507 |
+
// Mark URL as processed
|
| 508 |
+
processedURL := &schema.ProcessedURL{
|
| 509 |
+
URL: task.URL,
|
| 510 |
+
Timestamp: task.Timestamp,
|
| 511 |
+
FileType: task.FileType,
|
| 512 |
+
Status: schema.StatusCompleted,
|
| 513 |
+
ProcessedAt: time.Now(),
|
| 514 |
+
RowCount: rowCount,
|
| 515 |
+
ArticlesExtracted: len(articles),
|
| 516 |
+
DurationMs: time.Since(start).Milliseconds(),
|
| 517 |
+
}
|
| 518 |
+
if err := p.storage.MarkURLProcessed(ctx, processedURL); err != nil {
|
| 519 |
+
p.logger.Error("Failed to mark URL processed", zap.Error(err))
|
| 520 |
+
}
|
| 521 |
+
|
| 522 |
+
return Result{
|
| 523 |
+
Task: task,
|
| 524 |
+
RowCount: rowCount,
|
| 525 |
+
ArticleCount: insertedCount,
|
| 526 |
+
Duration: time.Since(start),
|
| 527 |
+
}
|
| 528 |
+
}
|
| 529 |
+
|
| 530 |
+
// generateTasks creates Task objects for all 3 GDELT files for a timestamp
|
| 531 |
+
func (p *Processor) generateTasks(timestamp string) []Task {
|
| 532 |
+
return []Task{
|
| 533 |
+
{Timestamp: timestamp, URL: fmt.Sprintf(constants.ExportURLTemplate, timestamp), FileType: constants.FileTypeExport},
|
| 534 |
+
{Timestamp: timestamp, URL: fmt.Sprintf(constants.MentionsURLTemplate, timestamp), FileType: constants.FileTypeMentions},
|
| 535 |
+
{Timestamp: timestamp, URL: fmt.Sprintf(constants.GKGURLTemplate, timestamp), FileType: constants.FileTypeGKG},
|
| 536 |
+
}
|
| 537 |
+
}
|
| 538 |
+
|
| 539 |
+
// extractURL extracts the article URL from a parsed record
|
| 540 |
+
func (p *Processor) extractURL(record interface{}, fileType string) string {
|
| 541 |
+
switch fileType {
|
| 542 |
+
case constants.FileTypeExport:
|
| 543 |
+
if event, ok := record.(*schema.Event); ok {
|
| 544 |
+
return event.SourceURL
|
| 545 |
+
}
|
| 546 |
+
case constants.FileTypeMentions:
|
| 547 |
+
if mention, ok := record.(*schema.Mention); ok {
|
| 548 |
+
return mention.MentionIdentifier
|
| 549 |
+
}
|
| 550 |
+
case constants.FileTypeGKG:
|
| 551 |
+
if gkg, ok := record.(*schema.GKG); ok {
|
| 552 |
+
return gkg.DocumentIdentifier
|
| 553 |
+
}
|
| 554 |
+
}
|
| 555 |
+
return ""
|
| 556 |
+
}
|
| 557 |
+
|
| 558 |
+
// enrichArticle adds GKG metadata to an article
|
| 559 |
+
func (p *Processor) enrichArticle(article *schema.Article, record interface{}) {
|
| 560 |
+
gkg, ok := record.(*schema.GKG)
|
| 561 |
+
if !ok {
|
| 562 |
+
return
|
| 563 |
+
}
|
| 564 |
+
article.Themes = mergeUnique(article.Themes, gkg.Themes)
|
| 565 |
+
article.Persons = mergeUnique(article.Persons, gkg.Persons)
|
| 566 |
+
article.Organizations = mergeUnique(article.Organizations, gkg.Organizations)
|
| 567 |
+
article.Locations = mergeUnique(article.Locations, gkg.Locations)
|
| 568 |
+
}
|
| 569 |
+
|
| 570 |
+
// GetTimestampStatus returns the processing status for a timestamp
|
| 571 |
+
func (p *Processor) GetTimestampStatus(ctx context.Context, timestamp string) (interface{}, error) {
|
| 572 |
+
p.mu.RLock()
|
| 573 |
+
isProcessing := p.processing[timestamp]
|
| 574 |
+
p.mu.RUnlock()
|
| 575 |
+
|
| 576 |
+
if isProcessing {
|
| 577 |
+
return map[string]interface{}{
|
| 578 |
+
"timestamp": timestamp,
|
| 579 |
+
"status": "processing",
|
| 580 |
+
}, nil
|
| 581 |
+
}
|
| 582 |
+
|
| 583 |
+
ts, err := p.storage.GetTimestampStatus(ctx, timestamp)
|
| 584 |
+
if err != nil {
|
| 585 |
+
return nil, err
|
| 586 |
+
}
|
| 587 |
+
|
| 588 |
+
if ts == nil {
|
| 589 |
+
return map[string]interface{}{
|
| 590 |
+
"timestamp": timestamp,
|
| 591 |
+
"status": "not_found",
|
| 592 |
+
"message": "Timestamp not queued for processing",
|
| 593 |
+
}, nil
|
| 594 |
+
}
|
| 595 |
+
|
| 596 |
+
return ts, nil
|
| 597 |
+
}
|
| 598 |
+
|
| 599 |
+
// GetAllTimestamps returns all completed and processing timestamps
|
| 600 |
+
func (p *Processor) GetAllTimestamps(ctx context.Context) (interface{}, error) {
|
| 601 |
+
completed, processing, err := p.storage.GetAllTimestamps(ctx)
|
| 602 |
+
if err != nil {
|
| 603 |
+
return nil, err
|
| 604 |
+
}
|
| 605 |
+
|
| 606 |
+
p.mu.RLock()
|
| 607 |
+
for ts := range p.processing {
|
| 608 |
+
if !containsString(processing, ts) {
|
| 609 |
+
processing = append(processing, ts)
|
| 610 |
+
}
|
| 611 |
+
}
|
| 612 |
+
p.mu.RUnlock()
|
| 613 |
+
|
| 614 |
+
return map[string]interface{}{
|
| 615 |
+
"completed": completed,
|
| 616 |
+
"processing": processing,
|
| 617 |
+
"total_completed": len(completed),
|
| 618 |
+
"total_processing": len(processing),
|
| 619 |
+
}, nil
|
| 620 |
+
}
|
| 621 |
+
|
| 622 |
+
// Helper functions
|
| 623 |
+
|
| 624 |
+
func normalizeURL(rawURL string) string {
|
| 625 |
+
if rawURL == "" {
|
| 626 |
+
return ""
|
| 627 |
+
}
|
| 628 |
+
u, err := url.Parse(rawURL)
|
| 629 |
+
if err != nil {
|
| 630 |
+
return ""
|
| 631 |
+
}
|
| 632 |
+
if u.Scheme == "" {
|
| 633 |
+
u.Scheme = "http"
|
| 634 |
+
}
|
| 635 |
+
u.Host = strings.ToLower(u.Host)
|
| 636 |
+
u.Host = strings.TrimPrefix(u.Host, "www.")
|
| 637 |
+
q := u.Query()
|
| 638 |
+
for _, param := range constants.TrackingParams {
|
| 639 |
+
q.Del(param)
|
| 640 |
+
}
|
| 641 |
+
u.RawQuery = q.Encode()
|
| 642 |
+
u.Fragment = ""
|
| 643 |
+
u.Path = strings.TrimSuffix(u.Path, "/")
|
| 644 |
+
return u.String()
|
| 645 |
+
}
|
| 646 |
+
|
| 647 |
+
func hashURL(normalizedURL string) string {
|
| 648 |
+
hash := sha256.Sum256([]byte(normalizedURL))
|
| 649 |
+
return hex.EncodeToString(hash[:])
|
| 650 |
+
}
|
| 651 |
+
|
| 652 |
+
func contains(slice []string, item string) bool {
|
| 653 |
+
for _, s := range slice {
|
| 654 |
+
if s == item {
|
| 655 |
+
return true
|
| 656 |
+
}
|
| 657 |
+
}
|
| 658 |
+
return false
|
| 659 |
+
}
|
| 660 |
+
|
| 661 |
+
func containsString(slice []string, item string) bool {
|
| 662 |
+
return contains(slice, item)
|
| 663 |
+
}
|
| 664 |
+
|
| 665 |
+
func mergeUnique(a, b []string) []string {
|
| 666 |
+
seen := make(map[string]bool)
|
| 667 |
+
for _, s := range a {
|
| 668 |
+
seen[s] = true
|
| 669 |
+
}
|
| 670 |
+
for _, s := range b {
|
| 671 |
+
if !seen[s] {
|
| 672 |
+
a = append(a, s)
|
| 673 |
+
seen[s] = true
|
| 674 |
+
}
|
| 675 |
+
}
|
| 676 |
+
return a
|
| 677 |
+
}
|
| 678 |
+
|
| 679 |
+
// ValidateTimestamp is exported for use by API handlers
|
| 680 |
+
func ValidateTimestamp(timestamp string) error {
|
| 681 |
+
result := ValidateTimestampStructural(timestamp)
|
| 682 |
+
if !result.Valid {
|
| 683 |
+
return fmt.Errorf(result.Reason)
|
| 684 |
+
}
|
| 685 |
+
return nil
|
| 686 |
+
}
|
internal/schema/schema.go
ADDED
|
@@ -0,0 +1,231 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Package schema defines all MongoDB document structures for the GDELT Engine.
|
| 2 |
+
// This package provides isolated, configurable schema definitions for database models.
|
| 3 |
+
package schema
|
| 4 |
+
|
| 5 |
+
import "time"
|
| 6 |
+
|
| 7 |
+
// Collection names - configurable variables for easy customization
|
| 8 |
+
var (
|
| 9 |
+
// CollectionArticles stores unique extracted articles/URLs
|
| 10 |
+
CollectionArticles = "articles"
|
| 11 |
+
|
| 12 |
+
// CollectionCompletedTimestamps stores completed timestamp processing records
|
| 13 |
+
CollectionCompletedTimestamps = "completed_timestamps"
|
| 14 |
+
|
| 15 |
+
// CollectionProcessedURLs stores each processed GDELT file URL
|
| 16 |
+
CollectionProcessedURLs = "processed_urls"
|
| 17 |
+
|
| 18 |
+
// CollectionEvents stores GDELT events (optional, if storing raw data)
|
| 19 |
+
CollectionEvents = "events"
|
| 20 |
+
|
| 21 |
+
// CollectionMentions stores GDELT mentions (optional)
|
| 22 |
+
CollectionMentions = "mentions"
|
| 23 |
+
|
| 24 |
+
// CollectionGKG stores GDELT GKG records (optional)
|
| 25 |
+
CollectionGKG = "gkg"
|
| 26 |
+
)
|
| 27 |
+
|
| 28 |
+
// ProcessingStatus represents the status of a timestamp or URL processing
|
| 29 |
+
type ProcessingStatus string
|
| 30 |
+
|
| 31 |
+
const (
|
| 32 |
+
StatusPending ProcessingStatus = "pending"
|
| 33 |
+
StatusProcessing ProcessingStatus = "processing"
|
| 34 |
+
StatusCompleted ProcessingStatus = "completed"
|
| 35 |
+
StatusFailed ProcessingStatus = "failed"
|
| 36 |
+
)
|
| 37 |
+
|
| 38 |
+
// CompletedTimestamp tracks which GDELT timestamps have been fully processed.
|
| 39 |
+
// The timestamp string (e.g., "20260128171500") serves as the primary key.
|
| 40 |
+
type CompletedTimestamp struct {
|
| 41 |
+
// Timestamp is the GDELT timestamp in YYYYMMDDHHmmss format (primary key)
|
| 42 |
+
Timestamp string `bson:"_id" json:"timestamp"`
|
| 43 |
+
|
| 44 |
+
// Status of processing
|
| 45 |
+
Status ProcessingStatus `bson:"status" json:"status"`
|
| 46 |
+
|
| 47 |
+
// StartedAt is when processing began
|
| 48 |
+
StartedAt time.Time `bson:"started_at" json:"started_at"`
|
| 49 |
+
|
| 50 |
+
// CompletedAt is when processing finished (nil if still processing)
|
| 51 |
+
CompletedAt *time.Time `bson:"completed_at,omitempty" json:"completed_at,omitempty"`
|
| 52 |
+
|
| 53 |
+
// FilesTotal is the total number of files to process (usually 3)
|
| 54 |
+
FilesTotal int `bson:"files_total" json:"files_total"`
|
| 55 |
+
|
| 56 |
+
// FilesProcessed is how many files have been completed
|
| 57 |
+
FilesProcessed int `bson:"files_processed" json:"files_processed"`
|
| 58 |
+
|
| 59 |
+
// CurrentFile is the file currently being processed
|
| 60 |
+
CurrentFile string `bson:"current_file,omitempty" json:"current_file,omitempty"`
|
| 61 |
+
|
| 62 |
+
// ArticlesCount is the total number of unique articles extracted
|
| 63 |
+
ArticlesCount int `bson:"articles_count" json:"articles_count"`
|
| 64 |
+
|
| 65 |
+
// TotalRows is the total number of rows parsed across all files
|
| 66 |
+
TotalRows int `bson:"total_rows" json:"total_rows"`
|
| 67 |
+
|
| 68 |
+
// DurationMs is the total processing time in milliseconds
|
| 69 |
+
DurationMs int64 `bson:"duration_ms" json:"duration_ms"`
|
| 70 |
+
|
| 71 |
+
// Error message if processing failed
|
| 72 |
+
Error string `bson:"error,omitempty" json:"error,omitempty"`
|
| 73 |
+
}
|
| 74 |
+
|
| 75 |
+
// ProcessedURL tracks each individual GDELT file URL that has been processed.
|
| 76 |
+
// This provides granular tracking for resume capability and debugging.
|
| 77 |
+
type ProcessedURL struct {
|
| 78 |
+
// URL is the full GDELT file URL (primary key)
|
| 79 |
+
URL string `bson:"_id" json:"url"`
|
| 80 |
+
|
| 81 |
+
// Timestamp is the GDELT timestamp this URL belongs to
|
| 82 |
+
Timestamp string `bson:"timestamp" json:"timestamp"`
|
| 83 |
+
|
| 84 |
+
// FileType is "export", "mentions", or "gkg"
|
| 85 |
+
FileType string `bson:"file_type" json:"file_type"`
|
| 86 |
+
|
| 87 |
+
// Status of this URL processing
|
| 88 |
+
Status ProcessingStatus `bson:"status" json:"status"`
|
| 89 |
+
|
| 90 |
+
// ProcessedAt is when this URL was processed
|
| 91 |
+
ProcessedAt time.Time `bson:"processed_at" json:"processed_at"`
|
| 92 |
+
|
| 93 |
+
// RowCount is the number of rows parsed from this file
|
| 94 |
+
RowCount int `bson:"row_count" json:"row_count"`
|
| 95 |
+
|
| 96 |
+
// ArticlesExtracted is the number of unique articles extracted
|
| 97 |
+
ArticlesExtracted int `bson:"articles_extracted" json:"articles_extracted"`
|
| 98 |
+
|
| 99 |
+
// DurationMs is how long this file took to process
|
| 100 |
+
DurationMs int64 `bson:"duration_ms" json:"duration_ms"`
|
| 101 |
+
|
| 102 |
+
// Error message if processing failed
|
| 103 |
+
Error string `bson:"error,omitempty" json:"error,omitempty"`
|
| 104 |
+
}
|
| 105 |
+
|
| 106 |
+
// Article represents a unique news article extracted from GDELT data.
|
| 107 |
+
// Articles are deduplicated based on normalized URL hash.
|
| 108 |
+
type Article struct {
|
| 109 |
+
// ID is SHA256 hash of normalized URL (primary key)
|
| 110 |
+
ID string `bson:"_id" json:"id"`
|
| 111 |
+
|
| 112 |
+
// URL is the original article URL
|
| 113 |
+
URL string `bson:"url" json:"url"`
|
| 114 |
+
|
| 115 |
+
// NormalizedURL is the cleaned URL used for deduplication
|
| 116 |
+
NormalizedURL string `bson:"normalized_url" json:"normalized_url"`
|
| 117 |
+
|
| 118 |
+
// Sources lists which GDELT file types mentioned this URL
|
| 119 |
+
Sources []string `bson:"sources" json:"sources"`
|
| 120 |
+
|
| 121 |
+
// Themes from GKG data (if available)
|
| 122 |
+
Themes []string `bson:"themes,omitempty" json:"themes,omitempty"`
|
| 123 |
+
|
| 124 |
+
// Persons mentioned in the article (from GKG)
|
| 125 |
+
Persons []string `bson:"persons,omitempty" json:"persons,omitempty"`
|
| 126 |
+
|
| 127 |
+
// Organizations mentioned in the article (from GKG)
|
| 128 |
+
Organizations []string `bson:"organizations,omitempty" json:"organizations,omitempty"`
|
| 129 |
+
|
| 130 |
+
// Locations mentioned in the article (from GKG)
|
| 131 |
+
Locations []string `bson:"locations,omitempty" json:"locations,omitempty"`
|
| 132 |
+
|
| 133 |
+
// Timestamp is the GDELT batch timestamp this was extracted from
|
| 134 |
+
Timestamp string `bson:"timestamp" json:"timestamp"`
|
| 135 |
+
|
| 136 |
+
// MentionCount tracks how many times this URL was seen
|
| 137 |
+
MentionCount int `bson:"mention_count" json:"mention_count"`
|
| 138 |
+
|
| 139 |
+
// FirstSeen is when this article was first extracted
|
| 140 |
+
FirstSeen time.Time `bson:"first_seen" json:"first_seen"`
|
| 141 |
+
|
| 142 |
+
// LastSeen is the most recent time this article was seen
|
| 143 |
+
LastSeen time.Time `bson:"last_seen" json:"last_seen"`
|
| 144 |
+
}
|
| 145 |
+
|
| 146 |
+
// Event represents a GDELT event record with enriched fields.
|
| 147 |
+
type Event struct {
|
| 148 |
+
GlobalEventID int64 `bson:"GlobalEventID" json:"global_event_id"`
|
| 149 |
+
Day int `bson:"Day" json:"day"`
|
| 150 |
+
Actor1Name string `bson:"Actor1Name,omitempty" json:"actor1_name,omitempty"`
|
| 151 |
+
Actor1CountryCode string `bson:"Actor1CountryCode,omitempty" json:"actor1_country_code,omitempty"`
|
| 152 |
+
Actor1CountryName string `bson:"Actor1CountryName,omitempty" json:"actor1_country_name,omitempty"`
|
| 153 |
+
Actor1Type1Code string `bson:"Actor1Type1Code,omitempty" json:"actor1_type1_code,omitempty"`
|
| 154 |
+
Actor1Type1Name string `bson:"Actor1Type1Name,omitempty" json:"actor1_type1_name,omitempty"`
|
| 155 |
+
Actor2Name string `bson:"Actor2Name,omitempty" json:"actor2_name,omitempty"`
|
| 156 |
+
Actor2CountryCode string `bson:"Actor2CountryCode,omitempty" json:"actor2_country_code,omitempty"`
|
| 157 |
+
Actor2CountryName string `bson:"Actor2CountryName,omitempty" json:"actor2_country_name,omitempty"`
|
| 158 |
+
EventCode string `bson:"EventCode,omitempty" json:"event_code,omitempty"`
|
| 159 |
+
EventDescription string `bson:"EventDescription,omitempty" json:"event_description,omitempty"`
|
| 160 |
+
EventBaseCode string `bson:"EventBaseCode,omitempty" json:"event_base_code,omitempty"`
|
| 161 |
+
EventRootCode string `bson:"EventRootCode,omitempty" json:"event_root_code,omitempty"`
|
| 162 |
+
QuadClass int `bson:"QuadClass,omitempty" json:"quad_class,omitempty"`
|
| 163 |
+
GoldsteinScale *float64 `bson:"GoldsteinScale,omitempty" json:"goldstein_scale,omitempty"`
|
| 164 |
+
NumMentions int `bson:"NumMentions,omitempty" json:"num_mentions,omitempty"`
|
| 165 |
+
NumSources int `bson:"NumSources,omitempty" json:"num_sources,omitempty"`
|
| 166 |
+
NumArticles int `bson:"NumArticles,omitempty" json:"num_articles,omitempty"`
|
| 167 |
+
AvgTone *float64 `bson:"AvgTone,omitempty" json:"avg_tone,omitempty"`
|
| 168 |
+
ActionGeoType int `bson:"ActionGeo_Type,omitempty" json:"action_geo_type,omitempty"`
|
| 169 |
+
ActionGeoFullName string `bson:"ActionGeo_FullName,omitempty" json:"action_geo_full_name,omitempty"`
|
| 170 |
+
ActionGeoCountryCode string `bson:"ActionGeo_CountryCode,omitempty" json:"action_geo_country_code,omitempty"`
|
| 171 |
+
ActionGeoADM1Code string `bson:"ActionGeo_ADM1Code,omitempty" json:"action_geo_adm1_code,omitempty"`
|
| 172 |
+
ActionGeoLat *float64 `bson:"ActionGeo_Lat,omitempty" json:"action_geo_lat,omitempty"`
|
| 173 |
+
ActionGeoLong *float64 `bson:"ActionGeo_Long,omitempty" json:"action_geo_long,omitempty"`
|
| 174 |
+
SourceURL string `bson:"SourceURL,omitempty" json:"source_url,omitempty"`
|
| 175 |
+
Timestamp string `bson:"timestamp" json:"timestamp"`
|
| 176 |
+
ProcessedAt time.Time `bson:"processed_at" json:"processed_at"`
|
| 177 |
+
}
|
| 178 |
+
|
| 179 |
+
// Mention represents a GDELT mention record.
|
| 180 |
+
type Mention struct {
|
| 181 |
+
GlobalEventID int64 `bson:"GlobalEventID" json:"global_event_id"`
|
| 182 |
+
EventTimeDate int64 `bson:"EventTimeDate" json:"event_time_date"`
|
| 183 |
+
MentionTimeDate int64 `bson:"MentionTimeDate" json:"mention_time_date"`
|
| 184 |
+
MentionType int `bson:"MentionType" json:"mention_type"`
|
| 185 |
+
MentionSourceName string `bson:"MentionSourceName,omitempty" json:"mention_source_name,omitempty"`
|
| 186 |
+
MentionIdentifier string `bson:"MentionIdentifier" json:"mention_identifier"`
|
| 187 |
+
SentenceID int `bson:"SentenceID,omitempty" json:"sentence_id,omitempty"`
|
| 188 |
+
Actor1CharOffset int `bson:"Actor1CharOffset,omitempty" json:"actor1_char_offset,omitempty"`
|
| 189 |
+
Actor2CharOffset int `bson:"Actor2CharOffset,omitempty" json:"actor2_char_offset,omitempty"`
|
| 190 |
+
ActionCharOffset int `bson:"ActionCharOffset,omitempty" json:"action_char_offset,omitempty"`
|
| 191 |
+
InRawText int `bson:"InRawText,omitempty" json:"in_raw_text,omitempty"`
|
| 192 |
+
Confidence int `bson:"Confidence,omitempty" json:"confidence,omitempty"`
|
| 193 |
+
MentionDocLen int `bson:"MentionDocLen,omitempty" json:"mention_doc_len,omitempty"`
|
| 194 |
+
MentionDocTone *float64 `bson:"MentionDocTone,omitempty" json:"mention_doc_tone,omitempty"`
|
| 195 |
+
MentionDocTranslation string `bson:"MentionDocTranslation,omitempty" json:"mention_doc_translation,omitempty"`
|
| 196 |
+
Timestamp string `bson:"timestamp" json:"timestamp"`
|
| 197 |
+
ProcessedAt time.Time `bson:"processed_at" json:"processed_at"`
|
| 198 |
+
}
|
| 199 |
+
|
| 200 |
+
// GKG represents a GDELT Global Knowledge Graph record.
|
| 201 |
+
type GKG struct {
|
| 202 |
+
GKGRECORDID string `bson:"GKGRECORDID" json:"gkg_record_id"`
|
| 203 |
+
Date int64 `bson:"DATE" json:"date"`
|
| 204 |
+
SourceCollectionID int `bson:"SourceCollectionIdentifier" json:"source_collection_id"`
|
| 205 |
+
SourceCommonName string `bson:"SourceCommonName,omitempty" json:"source_common_name,omitempty"`
|
| 206 |
+
DocumentIdentifier string `bson:"DocumentIdentifier,omitempty" json:"document_identifier,omitempty"`
|
| 207 |
+
Counts string `bson:"Counts,omitempty" json:"counts,omitempty"`
|
| 208 |
+
V2Counts string `bson:"V2Counts,omitempty" json:"v2_counts,omitempty"`
|
| 209 |
+
Themes []string `bson:"Themes,omitempty" json:"themes,omitempty"`
|
| 210 |
+
V2Themes []string `bson:"V2Themes,omitempty" json:"v2_themes,omitempty"`
|
| 211 |
+
Locations []string `bson:"Locations,omitempty" json:"locations,omitempty"`
|
| 212 |
+
V2Locations []string `bson:"V2Locations,omitempty" json:"v2_locations,omitempty"`
|
| 213 |
+
Persons []string `bson:"Persons,omitempty" json:"persons,omitempty"`
|
| 214 |
+
V2Persons []string `bson:"V2Persons,omitempty" json:"v2_persons,omitempty"`
|
| 215 |
+
Organizations []string `bson:"Organizations,omitempty" json:"organizations,omitempty"`
|
| 216 |
+
V2Organizations []string `bson:"V2Organizations,omitempty" json:"v2_organizations,omitempty"`
|
| 217 |
+
V2Tone string `bson:"V2Tone,omitempty" json:"v2_tone,omitempty"`
|
| 218 |
+
Dates string `bson:"Dates,omitempty" json:"dates,omitempty"`
|
| 219 |
+
GCAM string `bson:"GCAM,omitempty" json:"gcam,omitempty"`
|
| 220 |
+
SharingImage string `bson:"SharingImage,omitempty" json:"sharing_image,omitempty"`
|
| 221 |
+
RelatedImages []string `bson:"RelatedImages,omitempty" json:"related_images,omitempty"`
|
| 222 |
+
SocialImageEmbeds []string `bson:"SocialImageEmbeds,omitempty" json:"social_image_embeds,omitempty"`
|
| 223 |
+
SocialVideoEmbeds []string `bson:"SocialVideoEmbeds,omitempty" json:"social_video_embeds,omitempty"`
|
| 224 |
+
Quotations []string `bson:"Quotations,omitempty" json:"quotations,omitempty"`
|
| 225 |
+
AllNames []string `bson:"AllNames,omitempty" json:"all_names,omitempty"`
|
| 226 |
+
Amounts []string `bson:"Amounts,omitempty" json:"amounts,omitempty"`
|
| 227 |
+
TranslationInfo string `bson:"TranslationInfo,omitempty" json:"translation_info,omitempty"`
|
| 228 |
+
Extras string `bson:"Extras,omitempty" json:"extras,omitempty"`
|
| 229 |
+
Timestamp string `bson:"timestamp" json:"timestamp"`
|
| 230 |
+
ProcessedAt time.Time `bson:"processed_at" json:"processed_at"`
|
| 231 |
+
}
|
internal/storage/mongo.go
ADDED
|
@@ -0,0 +1,345 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Package storage provides MongoDB operations for the GDELT Engine.
|
| 2 |
+
// It implements the Storage interface used by the processor.
|
| 3 |
+
package storage
|
| 4 |
+
|
| 5 |
+
import (
|
| 6 |
+
"context"
|
| 7 |
+
"fmt"
|
| 8 |
+
"time"
|
| 9 |
+
|
| 10 |
+
"gdelt-engine/internal/schema"
|
| 11 |
+
|
| 12 |
+
"go.mongodb.org/mongo-driver/bson"
|
| 13 |
+
"go.mongodb.org/mongo-driver/mongo"
|
| 14 |
+
"go.mongodb.org/mongo-driver/mongo/options"
|
| 15 |
+
"go.uber.org/zap"
|
| 16 |
+
)
|
| 17 |
+
|
| 18 |
+
const (
|
| 19 |
+
connectTimeout = 10 * time.Second
|
| 20 |
+
ttlExpireAfter = 24 * time.Hour
|
| 21 |
+
)
|
| 22 |
+
|
| 23 |
+
// MongoDB wraps the MongoDB client and handles database operations.
|
| 24 |
+
// It implements the processor.Storage interface.
|
| 25 |
+
type MongoDB struct {
|
| 26 |
+
client *mongo.Client
|
| 27 |
+
database *mongo.Database
|
| 28 |
+
logger *zap.Logger
|
| 29 |
+
}
|
| 30 |
+
|
| 31 |
+
// NewMongoDB creates a new MongoDB connection.
|
| 32 |
+
func NewMongoDB(ctx context.Context, uri, dbName string, logger *zap.Logger) (*MongoDB, error) {
|
| 33 |
+
ctx, cancel := context.WithTimeout(ctx, connectTimeout)
|
| 34 |
+
defer cancel()
|
| 35 |
+
|
| 36 |
+
clientOpts := options.Client().ApplyURI(uri)
|
| 37 |
+
client, err := mongo.Connect(ctx, clientOpts)
|
| 38 |
+
if err != nil {
|
| 39 |
+
return nil, fmt.Errorf("mongo connect failed: %w", err)
|
| 40 |
+
}
|
| 41 |
+
|
| 42 |
+
// Verify connection
|
| 43 |
+
if err := client.Ping(ctx, nil); err != nil {
|
| 44 |
+
return nil, fmt.Errorf("mongo ping failed: %w", err)
|
| 45 |
+
}
|
| 46 |
+
|
| 47 |
+
db := &MongoDB{
|
| 48 |
+
client: client,
|
| 49 |
+
database: client.Database(dbName),
|
| 50 |
+
logger: logger,
|
| 51 |
+
}
|
| 52 |
+
|
| 53 |
+
// Create indexes
|
| 54 |
+
if err := db.ensureIndexes(ctx); err != nil {
|
| 55 |
+
return nil, fmt.Errorf("index creation failed: %w", err)
|
| 56 |
+
}
|
| 57 |
+
|
| 58 |
+
logger.Info("MongoDB connected", zap.String("database", dbName))
|
| 59 |
+
return db, nil
|
| 60 |
+
}
|
| 61 |
+
|
| 62 |
+
// Close gracefully closes the MongoDB connection.
|
| 63 |
+
func (m *MongoDB) Close(ctx context.Context) error {
|
| 64 |
+
return m.client.Disconnect(ctx)
|
| 65 |
+
}
|
| 66 |
+
|
| 67 |
+
// ensureIndexes creates required indexes on all collections.
|
| 68 |
+
func (m *MongoDB) ensureIndexes(ctx context.Context) error {
|
| 69 |
+
// Articles indexes
|
| 70 |
+
articlesCol := m.database.Collection(schema.CollectionArticles)
|
| 71 |
+
articleIndexes := []mongo.IndexModel{
|
| 72 |
+
{Keys: bson.D{{Key: "normalized_url", Value: 1}}},
|
| 73 |
+
{Keys: bson.D{{Key: "timestamp", Value: 1}}},
|
| 74 |
+
{Keys: bson.D{{Key: "themes", Value: 1}}},
|
| 75 |
+
{Keys: bson.D{{Key: "first_seen", Value: -1}}},
|
| 76 |
+
{Keys: bson.D{{Key: "last_seen", Value: -1}}},
|
| 77 |
+
{Keys: bson.D{{Key: "mention_count", Value: -1}}},
|
| 78 |
+
}
|
| 79 |
+
if _, err := articlesCol.Indexes().CreateMany(ctx, articleIndexes); err != nil {
|
| 80 |
+
return fmt.Errorf("articles indexes: %w", err)
|
| 81 |
+
}
|
| 82 |
+
|
| 83 |
+
// Completed timestamps indexes
|
| 84 |
+
tsCol := m.database.Collection(schema.CollectionCompletedTimestamps)
|
| 85 |
+
tsIndexes := []mongo.IndexModel{
|
| 86 |
+
{Keys: bson.D{{Key: "status", Value: 1}}},
|
| 87 |
+
{Keys: bson.D{{Key: "completed_at", Value: -1}}},
|
| 88 |
+
}
|
| 89 |
+
if _, err := tsCol.Indexes().CreateMany(ctx, tsIndexes); err != nil {
|
| 90 |
+
return fmt.Errorf("timestamps indexes: %w", err)
|
| 91 |
+
}
|
| 92 |
+
|
| 93 |
+
// Processed URLs indexes
|
| 94 |
+
urlCol := m.database.Collection(schema.CollectionProcessedURLs)
|
| 95 |
+
urlIndexes := []mongo.IndexModel{
|
| 96 |
+
{Keys: bson.D{{Key: "timestamp", Value: 1}}},
|
| 97 |
+
{Keys: bson.D{{Key: "status", Value: 1}}},
|
| 98 |
+
}
|
| 99 |
+
if _, err := urlCol.Indexes().CreateMany(ctx, urlIndexes); err != nil {
|
| 100 |
+
return fmt.Errorf("urls indexes: %w", err)
|
| 101 |
+
}
|
| 102 |
+
|
| 103 |
+
m.logger.Info("Indexes ensured")
|
| 104 |
+
return nil
|
| 105 |
+
}
|
| 106 |
+
|
| 107 |
+
// IsTimestampCompleted checks if a timestamp has already been processed successfully.
|
| 108 |
+
func (m *MongoDB) IsTimestampCompleted(ctx context.Context, timestamp string) (bool, error) {
|
| 109 |
+
col := m.database.Collection(schema.CollectionCompletedTimestamps)
|
| 110 |
+
|
| 111 |
+
filter := bson.M{
|
| 112 |
+
"_id": timestamp,
|
| 113 |
+
"status": schema.StatusCompleted,
|
| 114 |
+
}
|
| 115 |
+
|
| 116 |
+
count, err := col.CountDocuments(ctx, filter)
|
| 117 |
+
if err != nil {
|
| 118 |
+
return false, fmt.Errorf("count timestamps: %w", err)
|
| 119 |
+
}
|
| 120 |
+
|
| 121 |
+
return count > 0, nil
|
| 122 |
+
}
|
| 123 |
+
|
| 124 |
+
// GetTimestampStatus returns the status of a specific timestamp.
|
| 125 |
+
func (m *MongoDB) GetTimestampStatus(ctx context.Context, timestamp string) (*schema.CompletedTimestamp, error) {
|
| 126 |
+
col := m.database.Collection(schema.CollectionCompletedTimestamps)
|
| 127 |
+
|
| 128 |
+
var ts schema.CompletedTimestamp
|
| 129 |
+
err := col.FindOne(ctx, bson.M{"_id": timestamp}).Decode(&ts)
|
| 130 |
+
if err != nil {
|
| 131 |
+
if err == mongo.ErrNoDocuments {
|
| 132 |
+
return nil, nil
|
| 133 |
+
}
|
| 134 |
+
return nil, fmt.Errorf("find timestamp: %w", err)
|
| 135 |
+
}
|
| 136 |
+
|
| 137 |
+
return &ts, nil
|
| 138 |
+
}
|
| 139 |
+
|
| 140 |
+
// MarkTimestampStarted marks a timestamp as started processing.
|
| 141 |
+
func (m *MongoDB) MarkTimestampStarted(ctx context.Context, timestamp string) error {
|
| 142 |
+
col := m.database.Collection(schema.CollectionCompletedTimestamps)
|
| 143 |
+
|
| 144 |
+
doc := schema.CompletedTimestamp{
|
| 145 |
+
Timestamp: timestamp,
|
| 146 |
+
Status: schema.StatusProcessing,
|
| 147 |
+
StartedAt: time.Now(),
|
| 148 |
+
FilesTotal: 3,
|
| 149 |
+
FilesProcessed: 0,
|
| 150 |
+
}
|
| 151 |
+
|
| 152 |
+
opts := options.Update().SetUpsert(true)
|
| 153 |
+
_, err := col.UpdateOne(
|
| 154 |
+
ctx,
|
| 155 |
+
bson.M{"_id": timestamp},
|
| 156 |
+
bson.M{"$set": doc},
|
| 157 |
+
opts,
|
| 158 |
+
)
|
| 159 |
+
|
| 160 |
+
return err
|
| 161 |
+
}
|
| 162 |
+
|
| 163 |
+
// MarkTimestampCompleted marks a timestamp as completed.
|
| 164 |
+
func (m *MongoDB) MarkTimestampCompleted(ctx context.Context, ts *schema.CompletedTimestamp) error {
|
| 165 |
+
col := m.database.Collection(schema.CollectionCompletedTimestamps)
|
| 166 |
+
|
| 167 |
+
opts := options.Update().SetUpsert(true)
|
| 168 |
+
_, err := col.UpdateOne(
|
| 169 |
+
ctx,
|
| 170 |
+
bson.M{"_id": ts.Timestamp},
|
| 171 |
+
bson.M{"$set": ts},
|
| 172 |
+
opts,
|
| 173 |
+
)
|
| 174 |
+
|
| 175 |
+
return err
|
| 176 |
+
}
|
| 177 |
+
|
| 178 |
+
// GetAllTimestamps returns all completed and processing timestamps.
|
| 179 |
+
func (m *MongoDB) GetAllTimestamps(ctx context.Context) ([]schema.CompletedTimestamp, []string, error) {
|
| 180 |
+
col := m.database.Collection(schema.CollectionCompletedTimestamps)
|
| 181 |
+
|
| 182 |
+
// Get completed
|
| 183 |
+
completedCursor, err := col.Find(ctx, bson.M{"status": schema.StatusCompleted},
|
| 184 |
+
options.Find().SetSort(bson.D{{Key: "completed_at", Value: -1}}).SetLimit(100))
|
| 185 |
+
if err != nil {
|
| 186 |
+
return nil, nil, fmt.Errorf("find completed: %w", err)
|
| 187 |
+
}
|
| 188 |
+
defer completedCursor.Close(ctx)
|
| 189 |
+
|
| 190 |
+
var completed []schema.CompletedTimestamp
|
| 191 |
+
if err := completedCursor.All(ctx, &completed); err != nil {
|
| 192 |
+
return nil, nil, fmt.Errorf("decode completed: %w", err)
|
| 193 |
+
}
|
| 194 |
+
|
| 195 |
+
// Get processing
|
| 196 |
+
processingCursor, err := col.Find(ctx, bson.M{"status": schema.StatusProcessing})
|
| 197 |
+
if err != nil {
|
| 198 |
+
return nil, nil, fmt.Errorf("find processing: %w", err)
|
| 199 |
+
}
|
| 200 |
+
defer processingCursor.Close(ctx)
|
| 201 |
+
|
| 202 |
+
var processingDocs []schema.CompletedTimestamp
|
| 203 |
+
if err := processingCursor.All(ctx, &processingDocs); err != nil {
|
| 204 |
+
return nil, nil, fmt.Errorf("decode processing: %w", err)
|
| 205 |
+
}
|
| 206 |
+
|
| 207 |
+
processing := make([]string, len(processingDocs))
|
| 208 |
+
for i, doc := range processingDocs {
|
| 209 |
+
processing[i] = doc.Timestamp
|
| 210 |
+
}
|
| 211 |
+
|
| 212 |
+
return completed, processing, nil
|
| 213 |
+
}
|
| 214 |
+
|
| 215 |
+
// MarkURLProcessed marks a URL as processed.
|
| 216 |
+
func (m *MongoDB) MarkURLProcessed(ctx context.Context, url *schema.ProcessedURL) error {
|
| 217 |
+
col := m.database.Collection(schema.CollectionProcessedURLs)
|
| 218 |
+
|
| 219 |
+
opts := options.Update().SetUpsert(true)
|
| 220 |
+
_, err := col.UpdateOne(
|
| 221 |
+
ctx,
|
| 222 |
+
bson.M{"_id": url.URL},
|
| 223 |
+
bson.M{"$set": url},
|
| 224 |
+
opts,
|
| 225 |
+
)
|
| 226 |
+
|
| 227 |
+
return err
|
| 228 |
+
}
|
| 229 |
+
|
| 230 |
+
// GetProcessedURLs returns all processed URLs for a timestamp.
|
| 231 |
+
func (m *MongoDB) GetProcessedURLs(ctx context.Context, timestamp string) ([]schema.ProcessedURL, error) {
|
| 232 |
+
col := m.database.Collection(schema.CollectionProcessedURLs)
|
| 233 |
+
|
| 234 |
+
cursor, err := col.Find(ctx, bson.M{"timestamp": timestamp})
|
| 235 |
+
if err != nil {
|
| 236 |
+
return nil, fmt.Errorf("find urls: %w", err)
|
| 237 |
+
}
|
| 238 |
+
defer cursor.Close(ctx)
|
| 239 |
+
|
| 240 |
+
var urls []schema.ProcessedURL
|
| 241 |
+
if err := cursor.All(ctx, &urls); err != nil {
|
| 242 |
+
return nil, fmt.Errorf("decode urls: %w", err)
|
| 243 |
+
}
|
| 244 |
+
|
| 245 |
+
return urls, nil
|
| 246 |
+
}
|
| 247 |
+
|
| 248 |
+
// BulkInsertArticles performs bulk upsert for unique articles.
|
| 249 |
+
func (m *MongoDB) BulkInsertArticles(ctx context.Context, articles []schema.Article) (int, error) {
|
| 250 |
+
if len(articles) == 0 {
|
| 251 |
+
return 0, nil
|
| 252 |
+
}
|
| 253 |
+
|
| 254 |
+
col := m.database.Collection(schema.CollectionArticles)
|
| 255 |
+
models := make([]mongo.WriteModel, 0, len(articles))
|
| 256 |
+
now := time.Now()
|
| 257 |
+
|
| 258 |
+
for _, article := range articles {
|
| 259 |
+
filter := bson.M{"_id": article.ID}
|
| 260 |
+
|
| 261 |
+
// Build $addToSet for arrays
|
| 262 |
+
addToSet := bson.M{}
|
| 263 |
+
if len(article.Sources) > 0 {
|
| 264 |
+
addToSet["sources"] = bson.M{"$each": article.Sources}
|
| 265 |
+
}
|
| 266 |
+
if len(article.Themes) > 0 {
|
| 267 |
+
addToSet["themes"] = bson.M{"$each": article.Themes}
|
| 268 |
+
}
|
| 269 |
+
if len(article.Persons) > 0 {
|
| 270 |
+
addToSet["persons"] = bson.M{"$each": article.Persons}
|
| 271 |
+
}
|
| 272 |
+
if len(article.Organizations) > 0 {
|
| 273 |
+
addToSet["organizations"] = bson.M{"$each": article.Organizations}
|
| 274 |
+
}
|
| 275 |
+
if len(article.Locations) > 0 {
|
| 276 |
+
addToSet["locations"] = bson.M{"$each": article.Locations}
|
| 277 |
+
}
|
| 278 |
+
|
| 279 |
+
update := bson.M{
|
| 280 |
+
"$set": bson.M{
|
| 281 |
+
"url": article.URL,
|
| 282 |
+
"normalized_url": article.NormalizedURL,
|
| 283 |
+
"timestamp": article.Timestamp,
|
| 284 |
+
"last_seen": now,
|
| 285 |
+
},
|
| 286 |
+
"$inc": bson.M{
|
| 287 |
+
"mention_count": article.MentionCount,
|
| 288 |
+
},
|
| 289 |
+
"$setOnInsert": bson.M{
|
| 290 |
+
"first_seen": now,
|
| 291 |
+
},
|
| 292 |
+
}
|
| 293 |
+
|
| 294 |
+
if len(addToSet) > 0 {
|
| 295 |
+
update["$addToSet"] = addToSet
|
| 296 |
+
}
|
| 297 |
+
|
| 298 |
+
model := mongo.NewUpdateOneModel().
|
| 299 |
+
SetFilter(filter).
|
| 300 |
+
SetUpdate(update).
|
| 301 |
+
SetUpsert(true)
|
| 302 |
+
models = append(models, model)
|
| 303 |
+
}
|
| 304 |
+
|
| 305 |
+
opts := options.BulkWrite().SetOrdered(false)
|
| 306 |
+
result, err := col.BulkWrite(ctx, models, opts)
|
| 307 |
+
if err != nil {
|
| 308 |
+
if mongo.IsDuplicateKeyError(err) {
|
| 309 |
+
m.logger.Debug("Some duplicates in articles batch, continuing")
|
| 310 |
+
} else {
|
| 311 |
+
return 0, fmt.Errorf("articles bulk write: %w", err)
|
| 312 |
+
}
|
| 313 |
+
}
|
| 314 |
+
|
| 315 |
+
inserted := 0
|
| 316 |
+
if result != nil {
|
| 317 |
+
inserted = int(result.UpsertedCount + result.ModifiedCount)
|
| 318 |
+
}
|
| 319 |
+
|
| 320 |
+
return inserted, nil
|
| 321 |
+
}
|
| 322 |
+
|
| 323 |
+
// GetStats returns collection statistics.
|
| 324 |
+
func (m *MongoDB) GetStats(ctx context.Context) (map[string]interface{}, error) {
|
| 325 |
+
articlesCount, err := m.database.Collection(schema.CollectionArticles).CountDocuments(ctx, bson.M{})
|
| 326 |
+
if err != nil {
|
| 327 |
+
return nil, err
|
| 328 |
+
}
|
| 329 |
+
|
| 330 |
+
tsCount, err := m.database.Collection(schema.CollectionCompletedTimestamps).CountDocuments(ctx, bson.M{"status": schema.StatusCompleted})
|
| 331 |
+
if err != nil {
|
| 332 |
+
return nil, err
|
| 333 |
+
}
|
| 334 |
+
|
| 335 |
+
urlsCount, err := m.database.Collection(schema.CollectionProcessedURLs).CountDocuments(ctx, bson.M{})
|
| 336 |
+
if err != nil {
|
| 337 |
+
return nil, err
|
| 338 |
+
}
|
| 339 |
+
|
| 340 |
+
return map[string]interface{}{
|
| 341 |
+
"articles": articlesCount,
|
| 342 |
+
"completed_timestamps": tsCount,
|
| 343 |
+
"processed_urls": urlsCount,
|
| 344 |
+
}, nil
|
| 345 |
+
}
|
scripts/curl_commands.sh
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
# Quick curl commands for testing GDELT Engine API
|
| 3 |
+
# Copy and paste these directly into terminal
|
| 4 |
+
|
| 5 |
+
API="http://localhost:8080"
|
| 6 |
+
|
| 7 |
+
echo "═══════════════════════════════════════════════════════════"
|
| 8 |
+
echo " GDELT Engine - Quick Test Commands"
|
| 9 |
+
echo "═══════════════════════════════════════════════════════════"
|
| 10 |
+
|
| 11 |
+
cat << 'EOF'
|
| 12 |
+
|
| 13 |
+
# ─────────────────────────────────────────────────────────────
|
| 14 |
+
# BASIC TESTS
|
| 15 |
+
# ─────────────────────────────────────────────────────────────
|
| 16 |
+
|
| 17 |
+
# Health check
|
| 18 |
+
curl -s http://localhost:8080/health | jq .
|
| 19 |
+
|
| 20 |
+
# Get stats
|
| 21 |
+
curl -s http://localhost:8080/stats | jq .
|
| 22 |
+
|
| 23 |
+
# List timestamps
|
| 24 |
+
curl -s http://localhost:8080/timestamps | jq .
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
# ─────────────────────────────────────────────────────────────
|
| 28 |
+
# VALIDATION TESTS
|
| 29 |
+
# ─────────────────────────────────────────────────────────────
|
| 30 |
+
|
| 31 |
+
# Invalid: wrong length
|
| 32 |
+
curl -s http://localhost:8080/process -X POST \
|
| 33 |
+
-H "Content-Type: application/json" \
|
| 34 |
+
-d '{"timestamps": ["202601281715"]}' | jq .
|
| 35 |
+
|
| 36 |
+
# Invalid: minute not 00/15/30/45
|
| 37 |
+
curl -s http://localhost:8080/process -X POST \
|
| 38 |
+
-H "Content-Type: application/json" \
|
| 39 |
+
-d '{"timestamps": ["20260128171700"]}' | jq .
|
| 40 |
+
|
| 41 |
+
# Invalid: future date
|
| 42 |
+
curl -s http://localhost:8080/process -X POST \
|
| 43 |
+
-H "Content-Type: application/json" \
|
| 44 |
+
-d '{"timestamps": ["20990101120000"]}' | jq .
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
# ─────────────────────────────────────────────────────────────
|
| 48 |
+
# PROCESSING TESTS
|
| 49 |
+
# ─────────────────────────────────────────────────────────────
|
| 50 |
+
|
| 51 |
+
# Process 1 timestamp
|
| 52 |
+
curl -s http://localhost:8080/process -X POST \
|
| 53 |
+
-H "Content-Type: application/json" \
|
| 54 |
+
-d '{"timestamps": ["20260128171500"]}' | jq .
|
| 55 |
+
|
| 56 |
+
# Check status
|
| 57 |
+
curl -s http://localhost:8080/status/20260128171500 | jq .
|
| 58 |
+
|
| 59 |
+
# Process 4 timestamps (parallel)
|
| 60 |
+
curl -s http://localhost:8080/process -X POST \
|
| 61 |
+
-H "Content-Type: application/json" \
|
| 62 |
+
-d '{"timestamps": ["20260128180000", "20260128181500", "20260128183000", "20260128184500"]}' | jq .
|
| 63 |
+
|
| 64 |
+
# Process 8 timestamps
|
| 65 |
+
curl -s http://localhost:8080/process -X POST \
|
| 66 |
+
-H "Content-Type: application/json" \
|
| 67 |
+
-d '{"timestamps": ["20260127120000", "20260127121500", "20260127123000", "20260127124500", "20260127130000", "20260127131500", "20260127133000", "20260127134500"]}' | jq .
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
# ─────────────────────────────────────────────────────────────
|
| 71 |
+
# STRESS TESTS
|
| 72 |
+
# ─────────────────────────────────────────────────────────────
|
| 73 |
+
|
| 74 |
+
# 24 timestamps (1 day worth at 15-min intervals in 6 hours)
|
| 75 |
+
curl -s http://localhost:8080/process -X POST \
|
| 76 |
+
-H "Content-Type: application/json" \
|
| 77 |
+
-d '{"timestamps": [
|
| 78 |
+
"20260126120000", "20260126121500", "20260126123000", "20260126124500",
|
| 79 |
+
"20260126130000", "20260126131500", "20260126133000", "20260126134500",
|
| 80 |
+
"20260126140000", "20260126141500", "20260126143000", "20260126144500",
|
| 81 |
+
"20260126150000", "20260126151500", "20260126153000", "20260126154500",
|
| 82 |
+
"20260126160000", "20260126161500", "20260126163000", "20260126164500",
|
| 83 |
+
"20260126170000", "20260126171500", "20260126173000", "20260126174500"
|
| 84 |
+
]}' | jq .
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
# ─────────────────────────────────────────────────────────────
|
| 88 |
+
# MONITOR WITH SYSTEM STATS
|
| 89 |
+
# ─────────────────────────────────────────────────────────────
|
| 90 |
+
|
| 91 |
+
# Watch progress with CPU/RAM
|
| 92 |
+
watch -n 2 'echo "=== Progress ===" && curl -s http://localhost:8080/timestamps | jq "{completed: .total_completed, processing: .total_processing}" && echo "\n=== System ===" && free -h | head -2 && echo "" && top -bn1 | head -5'
|
| 93 |
+
|
| 94 |
+
# Monitor single timestamp until done
|
| 95 |
+
while true; do
|
| 96 |
+
status=$(curl -s http://localhost:8080/status/20260128171500 | jq -r '.status')
|
| 97 |
+
echo "Status: $status"
|
| 98 |
+
[ "$status" = "completed" ] || [ "$status" = "failed" ] && break
|
| 99 |
+
sleep 2
|
| 100 |
+
done
|
| 101 |
+
|
| 102 |
+
EOF
|
| 103 |
+
|
| 104 |
+
echo ""
|
| 105 |
+
echo "Copy any command above and run in terminal!"
|
scripts/normalize.py
ADDED
|
@@ -0,0 +1,191 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""
|
| 3 |
+
URL Normalization Script for GDELT Engine
|
| 4 |
+
|
| 5 |
+
This isolated Python script normalizes URLs for deduplication.
|
| 6 |
+
It can be called from Go via exec.Command or used standalone.
|
| 7 |
+
|
| 8 |
+
Usage:
|
| 9 |
+
python normalize.py <url>
|
| 10 |
+
echo "url1\nurl2" | python normalize.py --stdin
|
| 11 |
+
"""
|
| 12 |
+
|
| 13 |
+
import sys
|
| 14 |
+
import re
|
| 15 |
+
from urllib.parse import urlparse, urlunparse, parse_qs, urlencode
|
| 16 |
+
|
| 17 |
+
# Tracking parameters to remove
|
| 18 |
+
TRACKING_PARAMS = {
|
| 19 |
+
# UTM parameters
|
| 20 |
+
'utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
|
| 21 |
+
'utm_id', 'utm_cid',
|
| 22 |
+
# Social media tracking
|
| 23 |
+
'fbclid', 'gclid', 'dclid', 'msclkid', 'twclid', 'igshid',
|
| 24 |
+
# Analytics
|
| 25 |
+
'ref', 'source', 'campaign', 'mc_cid', 'mc_eid',
|
| 26 |
+
'_ga', '_gl', '_hsenc', '_hsmi',
|
| 27 |
+
# News specific
|
| 28 |
+
'ncid', 'ocid', 'cmpid', 'ns_mchannel', 'ns_source', 'ns_campaign',
|
| 29 |
+
# General
|
| 30 |
+
'share', 'action', 'module', 'feed', 'rss',
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
# URL shortener domains that should not be normalized
|
| 34 |
+
SHORTENERS = {
|
| 35 |
+
'bit.ly', 't.co', 'goo.gl', 'ow.ly', 'tinyurl.com', 'is.gd',
|
| 36 |
+
'buff.ly', 'dlvr.it', 'j.mp', 'spr.ly', 'ht.ly',
|
| 37 |
+
}
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def normalize_url(url: str) -> str:
|
| 41 |
+
"""
|
| 42 |
+
Normalize a URL for deduplication.
|
| 43 |
+
|
| 44 |
+
Steps:
|
| 45 |
+
1. Parse URL
|
| 46 |
+
2. Ensure scheme (default http)
|
| 47 |
+
3. Lowercase host
|
| 48 |
+
4. Remove www prefix
|
| 49 |
+
5. Remove tracking parameters
|
| 50 |
+
6. Remove fragment
|
| 51 |
+
7. Clean path (remove trailing slash, normalize)
|
| 52 |
+
"""
|
| 53 |
+
if not url:
|
| 54 |
+
return ''
|
| 55 |
+
|
| 56 |
+
# Ensure scheme
|
| 57 |
+
if not url.startswith(('http://', 'https://')):
|
| 58 |
+
url = 'http://' + url
|
| 59 |
+
|
| 60 |
+
try:
|
| 61 |
+
parsed = urlparse(url)
|
| 62 |
+
except Exception:
|
| 63 |
+
return ''
|
| 64 |
+
|
| 65 |
+
# Skip shorteners - return as-is
|
| 66 |
+
host = parsed.netloc.lower()
|
| 67 |
+
if any(short in host for short in SHORTENERS):
|
| 68 |
+
return url
|
| 69 |
+
|
| 70 |
+
# Lowercase and clean host
|
| 71 |
+
host = host.lower()
|
| 72 |
+
if host.startswith('www.'):
|
| 73 |
+
host = host[4:]
|
| 74 |
+
|
| 75 |
+
# Remove tracking parameters
|
| 76 |
+
query_params = parse_qs(parsed.query, keep_blank_values=True)
|
| 77 |
+
clean_params = {
|
| 78 |
+
k: v for k, v in query_params.items()
|
| 79 |
+
if k.lower() not in TRACKING_PARAMS
|
| 80 |
+
}
|
| 81 |
+
clean_query = urlencode(clean_params, doseq=True) if clean_params else ''
|
| 82 |
+
|
| 83 |
+
# Clean path
|
| 84 |
+
path = parsed.path
|
| 85 |
+
# Remove trailing slash (except for root)
|
| 86 |
+
if path != '/' and path.endswith('/'):
|
| 87 |
+
path = path.rstrip('/')
|
| 88 |
+
# Normalize double slashes
|
| 89 |
+
path = re.sub(r'/+', '/', path)
|
| 90 |
+
|
| 91 |
+
# Reconstruct URL without fragment
|
| 92 |
+
normalized = urlunparse((
|
| 93 |
+
parsed.scheme,
|
| 94 |
+
host,
|
| 95 |
+
path,
|
| 96 |
+
parsed.params,
|
| 97 |
+
clean_query,
|
| 98 |
+
'' # No fragment
|
| 99 |
+
))
|
| 100 |
+
|
| 101 |
+
return normalized
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def extract_domain(url: str) -> str:
|
| 105 |
+
"""Extract the domain from a URL."""
|
| 106 |
+
try:
|
| 107 |
+
parsed = urlparse(url)
|
| 108 |
+
host = parsed.netloc.lower()
|
| 109 |
+
if host.startswith('www.'):
|
| 110 |
+
host = host[4:]
|
| 111 |
+
return host
|
| 112 |
+
except Exception:
|
| 113 |
+
return ''
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
def is_valid_news_url(url: str) -> bool:
|
| 117 |
+
"""Check if URL is likely a news article (not homepage, category, etc)."""
|
| 118 |
+
if not url:
|
| 119 |
+
return False
|
| 120 |
+
|
| 121 |
+
try:
|
| 122 |
+
parsed = urlparse(url)
|
| 123 |
+
path = parsed.path
|
| 124 |
+
|
| 125 |
+
# Must have a path beyond just /
|
| 126 |
+
if not path or path == '/':
|
| 127 |
+
return False
|
| 128 |
+
|
| 129 |
+
# Common non-article patterns
|
| 130 |
+
non_article_patterns = [
|
| 131 |
+
r'^/category/',
|
| 132 |
+
r'^/tag/',
|
| 133 |
+
r'^/author/',
|
| 134 |
+
r'^/page/',
|
| 135 |
+
r'^/search',
|
| 136 |
+
r'^/about',
|
| 137 |
+
r'^/contact',
|
| 138 |
+
r'^/privacy',
|
| 139 |
+
r'^/terms',
|
| 140 |
+
r'/feed/?$',
|
| 141 |
+
r'/rss/?$',
|
| 142 |
+
]
|
| 143 |
+
|
| 144 |
+
for pattern in non_article_patterns:
|
| 145 |
+
if re.match(pattern, path, re.IGNORECASE):
|
| 146 |
+
return False
|
| 147 |
+
|
| 148 |
+
# Likely an article if has date-like pattern or enough path depth
|
| 149 |
+
if re.search(r'/\d{4}/\d{2}/', path): # Date in URL
|
| 150 |
+
return True
|
| 151 |
+
if path.count('/') >= 2: # Multiple path segments
|
| 152 |
+
return True
|
| 153 |
+
if re.search(r'\.\w{3,4}$', path): # Has file extension
|
| 154 |
+
return True
|
| 155 |
+
if len(path) > 20: # Long path likely an article
|
| 156 |
+
return True
|
| 157 |
+
|
| 158 |
+
return True # Default to true
|
| 159 |
+
|
| 160 |
+
except Exception:
|
| 161 |
+
return False
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def main():
|
| 165 |
+
"""Main entry point."""
|
| 166 |
+
if len(sys.argv) > 1:
|
| 167 |
+
if sys.argv[1] == '--stdin':
|
| 168 |
+
# Read URLs from stdin
|
| 169 |
+
for line in sys.stdin:
|
| 170 |
+
url = line.strip()
|
| 171 |
+
if url:
|
| 172 |
+
normalized = normalize_url(url)
|
| 173 |
+
print(normalized)
|
| 174 |
+
elif sys.argv[1] == '--validate':
|
| 175 |
+
# Validate URLs from stdin
|
| 176 |
+
for line in sys.stdin:
|
| 177 |
+
url = line.strip()
|
| 178 |
+
if url and is_valid_news_url(url):
|
| 179 |
+
print(normalize_url(url))
|
| 180 |
+
else:
|
| 181 |
+
# Single URL argument
|
| 182 |
+
print(normalize_url(sys.argv[1]))
|
| 183 |
+
else:
|
| 184 |
+
print("Usage: normalize.py <url>", file=sys.stderr)
|
| 185 |
+
print(" normalize.py --stdin", file=sys.stderr)
|
| 186 |
+
print(" normalize.py --validate", file=sys.stderr)
|
| 187 |
+
sys.exit(1)
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
if __name__ == '__main__':
|
| 191 |
+
main()
|
scripts/stress_test.sh
ADDED
|
@@ -0,0 +1,373 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
# GDELT Engine Stress Test Suite
|
| 3 |
+
# Tests API endpoints with various load scenarios and monitors system resources
|
| 4 |
+
|
| 5 |
+
set -e
|
| 6 |
+
|
| 7 |
+
API_URL="${API_URL:-http://localhost:8080}"
|
| 8 |
+
MONITOR_INTERVAL=1
|
| 9 |
+
|
| 10 |
+
# Colors for output
|
| 11 |
+
RED='\033[0;31m'
|
| 12 |
+
GREEN='\033[0;32m'
|
| 13 |
+
YELLOW='\033[1;33m'
|
| 14 |
+
BLUE='\033[0;34m'
|
| 15 |
+
NC='\033[0m' # No Color
|
| 16 |
+
|
| 17 |
+
# Generate valid GDELT timestamps (must be 00, 15, 30, or 45 minutes)
|
| 18 |
+
generate_timestamps() {
|
| 19 |
+
local count=$1
|
| 20 |
+
local base_date="${2:-20260128}"
|
| 21 |
+
local timestamps=()
|
| 22 |
+
|
| 23 |
+
hours=(17 18 19 20 21 22 23 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16)
|
| 24 |
+
minutes=(00 15 30 45)
|
| 25 |
+
|
| 26 |
+
for ((i=0; i<count; i++)); do
|
| 27 |
+
hour_idx=$((i / 4 % 24))
|
| 28 |
+
min_idx=$((i % 4))
|
| 29 |
+
hour=$(printf "%02d" ${hours[$hour_idx]})
|
| 30 |
+
min=${minutes[$min_idx]}
|
| 31 |
+
timestamps+=("${base_date}${hour}${min}00")
|
| 32 |
+
done
|
| 33 |
+
|
| 34 |
+
echo "${timestamps[@]}"
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
# Monitor system resources
|
| 38 |
+
monitor_resources() {
|
| 39 |
+
local pid=$1
|
| 40 |
+
local duration=$2
|
| 41 |
+
local output_file=$3
|
| 42 |
+
|
| 43 |
+
echo "timestamp,cpu_percent,mem_mb,goroutines" > "$output_file"
|
| 44 |
+
|
| 45 |
+
for ((i=0; i<duration; i++)); do
|
| 46 |
+
if kill -0 $pid 2>/dev/null; then
|
| 47 |
+
# Get CPU and memory for the process
|
| 48 |
+
stats=$(ps -p $pid -o %cpu,rss --no-headers 2>/dev/null || echo "0 0")
|
| 49 |
+
cpu=$(echo $stats | awk '{print $1}')
|
| 50 |
+
mem_kb=$(echo $stats | awk '{print $2}')
|
| 51 |
+
mem_mb=$((mem_kb / 1024))
|
| 52 |
+
|
| 53 |
+
# Get goroutine count if pprof is available
|
| 54 |
+
goroutines=$(curl -s "${API_URL}/debug/pprof/goroutine?debug=0" 2>/dev/null | wc -l || echo "N/A")
|
| 55 |
+
|
| 56 |
+
echo "$(date +%s),$cpu,$mem_mb,$goroutines" >> "$output_file"
|
| 57 |
+
fi
|
| 58 |
+
sleep $MONITOR_INTERVAL
|
| 59 |
+
done
|
| 60 |
+
}
|
| 61 |
+
|
| 62 |
+
print_header() {
|
| 63 |
+
echo ""
|
| 64 |
+
echo -e "${BLUE}═══════════════════════════════════════════════════════════${NC}"
|
| 65 |
+
echo -e "${BLUE} $1${NC}"
|
| 66 |
+
echo -e "${BLUE}═══════════════════════════════════════════════════════════${NC}"
|
| 67 |
+
}
|
| 68 |
+
|
| 69 |
+
print_result() {
|
| 70 |
+
local name=$1
|
| 71 |
+
local status=$2
|
| 72 |
+
local duration=$3
|
| 73 |
+
|
| 74 |
+
if [ "$status" = "PASS" ]; then
|
| 75 |
+
echo -e "${GREEN}✓ $name${NC} - ${duration}ms"
|
| 76 |
+
else
|
| 77 |
+
echo -e "${RED}✗ $name${NC} - $status"
|
| 78 |
+
fi
|
| 79 |
+
}
|
| 80 |
+
|
| 81 |
+
# ═══════════════════════════════════════════════════════════
|
| 82 |
+
# TEST 1: Health Check
|
| 83 |
+
# ═══════════════════════════════════════════════════════════
|
| 84 |
+
test_health() {
|
| 85 |
+
print_header "Test 1: Health Check"
|
| 86 |
+
|
| 87 |
+
start=$(date +%s%3N)
|
| 88 |
+
response=$(curl -s -w "\n%{http_code}" "${API_URL}/health")
|
| 89 |
+
http_code=$(echo "$response" | tail -1)
|
| 90 |
+
body=$(echo "$response" | head -n -1)
|
| 91 |
+
end=$(date +%s%3N)
|
| 92 |
+
duration=$((end - start))
|
| 93 |
+
|
| 94 |
+
if [ "$http_code" = "200" ]; then
|
| 95 |
+
print_result "Health endpoint" "PASS" "$duration"
|
| 96 |
+
echo "$body" | jq .
|
| 97 |
+
else
|
| 98 |
+
print_result "Health endpoint" "FAIL: HTTP $http_code" "$duration"
|
| 99 |
+
fi
|
| 100 |
+
}
|
| 101 |
+
|
| 102 |
+
# ═══════════════════════════════════════════════════════════
|
| 103 |
+
# TEST 2: Stats Check
|
| 104 |
+
# ═══════════════════════════════════════════════════════════
|
| 105 |
+
test_stats() {
|
| 106 |
+
print_header "Test 2: Database Stats"
|
| 107 |
+
|
| 108 |
+
start=$(date +%s%3N)
|
| 109 |
+
response=$(curl -s -w "\n%{http_code}" "${API_URL}/stats")
|
| 110 |
+
http_code=$(echo "$response" | tail -1)
|
| 111 |
+
body=$(echo "$response" | head -n -1)
|
| 112 |
+
end=$(date +%s%3N)
|
| 113 |
+
duration=$((end - start))
|
| 114 |
+
|
| 115 |
+
if [ "$http_code" = "200" ]; then
|
| 116 |
+
print_result "Stats endpoint" "PASS" "$duration"
|
| 117 |
+
echo "$body" | jq .
|
| 118 |
+
else
|
| 119 |
+
print_result "Stats endpoint" "FAIL: HTTP $http_code" "$duration"
|
| 120 |
+
fi
|
| 121 |
+
}
|
| 122 |
+
|
| 123 |
+
# ═══════════════════════════════════════════════════════════
|
| 124 |
+
# TEST 3: Validation Tests
|
| 125 |
+
# ═══════════════════════════════════════════════════════════
|
| 126 |
+
test_validation() {
|
| 127 |
+
print_header "Test 3: Timestamp Validation"
|
| 128 |
+
|
| 129 |
+
echo -e "\n${YELLOW}Testing invalid timestamps:${NC}"
|
| 130 |
+
|
| 131 |
+
# Invalid length
|
| 132 |
+
echo -n " Invalid length (12 chars): "
|
| 133 |
+
response=$(curl -s "${API_URL}/process" -X POST -H "Content-Type: application/json" \
|
| 134 |
+
-d '{"timestamps": ["202601281715"]}')
|
| 135 |
+
echo "$response" | jq -c '.rejected[0].reason // .error'
|
| 136 |
+
|
| 137 |
+
# Invalid minute
|
| 138 |
+
echo -n " Invalid minute (17): "
|
| 139 |
+
response=$(curl -s "${API_URL}/process" -X POST -H "Content-Type: application/json" \
|
| 140 |
+
-d '{"timestamps": ["20260128171700"]}')
|
| 141 |
+
echo "$response" | jq -c '.rejected[0].reason // .error'
|
| 142 |
+
|
| 143 |
+
# Future timestamp
|
| 144 |
+
echo -n " Future timestamp: "
|
| 145 |
+
response=$(curl -s "${API_URL}/process" -X POST -H "Content-Type: application/json" \
|
| 146 |
+
-d '{"timestamps": ["20990128171500"]}')
|
| 147 |
+
echo "$response" | jq -c '.rejected[0].reason // .error'
|
| 148 |
+
|
| 149 |
+
# Valid timestamp
|
| 150 |
+
echo -e "\n${YELLOW}Testing valid timestamp:${NC}"
|
| 151 |
+
echo -n " Valid format: "
|
| 152 |
+
response=$(curl -s "${API_URL}/process" -X POST -H "Content-Type: application/json" \
|
| 153 |
+
-d '{"timestamps": ["20260128171500"]}')
|
| 154 |
+
echo "$response" | jq -c '{accepted, queued: .timestamps_queued, rejected: .timestamps_rejected}'
|
| 155 |
+
}
|
| 156 |
+
|
| 157 |
+
# ═══════════════════════════════════════════════════════════
|
| 158 |
+
# TEST 4: Single Timestamp Processing
|
| 159 |
+
# ═══════════════════════════════════════════════════════════
|
| 160 |
+
test_single() {
|
| 161 |
+
print_header "Test 4: Single Timestamp Processing"
|
| 162 |
+
|
| 163 |
+
local ts="20260128180000"
|
| 164 |
+
|
| 165 |
+
echo -e "${YELLOW}Submitting 1 timestamp: $ts${NC}"
|
| 166 |
+
|
| 167 |
+
start=$(date +%s%3N)
|
| 168 |
+
response=$(curl -s "${API_URL}/process" -X POST -H "Content-Type: application/json" \
|
| 169 |
+
-d "{\"timestamps\": [\"$ts\"]}")
|
| 170 |
+
end=$(date +%s%3N)
|
| 171 |
+
|
| 172 |
+
echo "$response" | jq .
|
| 173 |
+
|
| 174 |
+
echo -e "\n${YELLOW}Waiting for completion...${NC}"
|
| 175 |
+
|
| 176 |
+
for i in {1..60}; do
|
| 177 |
+
sleep 2
|
| 178 |
+
status=$(curl -s "${API_URL}/status/$ts")
|
| 179 |
+
current_status=$(echo "$status" | jq -r '.status')
|
| 180 |
+
|
| 181 |
+
if [ "$current_status" = "completed" ]; then
|
| 182 |
+
echo -e "${GREEN}✓ Completed!${NC}"
|
| 183 |
+
echo "$status" | jq .
|
| 184 |
+
break
|
| 185 |
+
elif [ "$current_status" = "failed" ] || [ "$current_status" = "invalid" ]; then
|
| 186 |
+
echo -e "${RED}✗ Failed!${NC}"
|
| 187 |
+
echo "$status" | jq .
|
| 188 |
+
break
|
| 189 |
+
else
|
| 190 |
+
echo -n "."
|
| 191 |
+
fi
|
| 192 |
+
done
|
| 193 |
+
}
|
| 194 |
+
|
| 195 |
+
# ═══════════════════════════════════════════════════════════
|
| 196 |
+
# TEST 5: Batch Processing (4 timestamps)
|
| 197 |
+
# ═══════════════════════════════════════════════════════════
|
| 198 |
+
test_batch_small() {
|
| 199 |
+
print_header "Test 5: Small Batch (4 timestamps)"
|
| 200 |
+
|
| 201 |
+
timestamps=$(generate_timestamps 4 "20260127")
|
| 202 |
+
ts_array=$(echo $timestamps | tr ' ' ',' | sed 's/^/["/;s/,/","/g;s/$/"]/')
|
| 203 |
+
|
| 204 |
+
echo -e "${YELLOW}Submitting: $ts_array${NC}"
|
| 205 |
+
|
| 206 |
+
start=$(date +%s)
|
| 207 |
+
response=$(curl -s "${API_URL}/process" -X POST -H "Content-Type: application/json" \
|
| 208 |
+
-d "{\"timestamps\": $ts_array}")
|
| 209 |
+
|
| 210 |
+
echo "$response" | jq -c '{accepted, queued: .timestamps_queued}'
|
| 211 |
+
|
| 212 |
+
echo -e "\n${YELLOW}Monitoring progress...${NC}"
|
| 213 |
+
|
| 214 |
+
for i in {1..120}; do
|
| 215 |
+
sleep 2
|
| 216 |
+
list=$(curl -s "${API_URL}/timestamps")
|
| 217 |
+
completed=$(echo "$list" | jq '.total_completed')
|
| 218 |
+
processing=$(echo "$list" | jq '.total_processing')
|
| 219 |
+
|
| 220 |
+
echo -ne "\r Completed: $completed | Processing: $processing "
|
| 221 |
+
|
| 222 |
+
if [ "$processing" = "0" ]; then
|
| 223 |
+
break
|
| 224 |
+
fi
|
| 225 |
+
done
|
| 226 |
+
|
| 227 |
+
end=$(date +%s)
|
| 228 |
+
echo -e "\n${GREEN}✓ Batch completed in $((end - start)) seconds${NC}"
|
| 229 |
+
}
|
| 230 |
+
|
| 231 |
+
# ═══════════════════════════════════════════════════════════
|
| 232 |
+
# TEST 6: Large Batch Stress Test
|
| 233 |
+
# ═══════════════════════════════════════════════════════════
|
| 234 |
+
test_batch_large() {
|
| 235 |
+
print_header "Test 6: Large Batch Stress Test (24 timestamps)"
|
| 236 |
+
|
| 237 |
+
timestamps=$(generate_timestamps 24 "20260126")
|
| 238 |
+
ts_array=$(echo $timestamps | tr ' ' ',' | sed 's/^/["/;s/,/","/g;s/$/"]/')
|
| 239 |
+
|
| 240 |
+
echo -e "${YELLOW}Submitting 24 timestamps...${NC}"
|
| 241 |
+
|
| 242 |
+
# Start resource monitoring in background
|
| 243 |
+
if [ -n "$ENGINE_PID" ]; then
|
| 244 |
+
monitor_resources $ENGINE_PID 300 "/tmp/gdelt_stress_resources.csv" &
|
| 245 |
+
MONITOR_PID=$!
|
| 246 |
+
fi
|
| 247 |
+
|
| 248 |
+
start=$(date +%s)
|
| 249 |
+
response=$(curl -s "${API_URL}/process" -X POST -H "Content-Type: application/json" \
|
| 250 |
+
-d "{\"timestamps\": $ts_array}")
|
| 251 |
+
|
| 252 |
+
echo "$response" | jq -c '{accepted, queued: .timestamps_queued}'
|
| 253 |
+
|
| 254 |
+
echo -e "\n${YELLOW}Monitoring progress (this may take 2-3 minutes)...${NC}"
|
| 255 |
+
|
| 256 |
+
for i in {1..180}; do
|
| 257 |
+
sleep 2
|
| 258 |
+
list=$(curl -s "${API_URL}/timestamps")
|
| 259 |
+
completed=$(echo "$list" | jq '.total_completed')
|
| 260 |
+
processing=$(echo "$list" | jq '.total_processing')
|
| 261 |
+
|
| 262 |
+
# Get current system stats
|
| 263 |
+
cpu=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
|
| 264 |
+
mem=$(free -m | awk 'NR==2{printf "%.1f", $3/$2*100}')
|
| 265 |
+
|
| 266 |
+
echo -ne "\r Completed: $completed | Processing: $processing | CPU: ${cpu}% | RAM: ${mem}% "
|
| 267 |
+
|
| 268 |
+
if [ "$processing" = "0" ]; then
|
| 269 |
+
break
|
| 270 |
+
fi
|
| 271 |
+
done
|
| 272 |
+
|
| 273 |
+
end=$(date +%s)
|
| 274 |
+
|
| 275 |
+
# Stop monitoring
|
| 276 |
+
if [ -n "$MONITOR_PID" ]; then
|
| 277 |
+
kill $MONITOR_PID 2>/dev/null || true
|
| 278 |
+
fi
|
| 279 |
+
|
| 280 |
+
echo -e "\n${GREEN}✓ Stress test completed in $((end - start)) seconds${NC}"
|
| 281 |
+
|
| 282 |
+
# Show final stats
|
| 283 |
+
echo -e "\n${YELLOW}Final Statistics:${NC}"
|
| 284 |
+
curl -s "${API_URL}/stats" | jq .
|
| 285 |
+
}
|
| 286 |
+
|
| 287 |
+
# ═══════════════════════════════════════════════════════════
|
| 288 |
+
# TEST 7: Duplicate/Idempotency Test
|
| 289 |
+
# ═══════════════════════════════════════════════════════════
|
| 290 |
+
test_idempotency() {
|
| 291 |
+
print_header "Test 7: Idempotency Test"
|
| 292 |
+
|
| 293 |
+
local ts="20260128183000"
|
| 294 |
+
|
| 295 |
+
echo -e "${YELLOW}Submitting same timestamp twice:${NC}"
|
| 296 |
+
|
| 297 |
+
echo -n " First request: "
|
| 298 |
+
response1=$(curl -s "${API_URL}/process" -X POST -H "Content-Type: application/json" \
|
| 299 |
+
-d "{\"timestamps\": [\"$ts\"]}")
|
| 300 |
+
echo "$response1" | jq -c '{queued: .timestamps_queued, rejected: .timestamps_rejected}'
|
| 301 |
+
|
| 302 |
+
sleep 1
|
| 303 |
+
|
| 304 |
+
echo -n " Second request (should reject): "
|
| 305 |
+
response2=$(curl -s "${API_URL}/process" -X POST -H "Content-Type: application/json" \
|
| 306 |
+
-d "{\"timestamps\": [\"$ts\"]}")
|
| 307 |
+
echo "$response2" | jq -c '{queued: .timestamps_queued, rejected: .timestamps_rejected}'
|
| 308 |
+
}
|
| 309 |
+
|
| 310 |
+
# ═══════════════════════════════════════════════════════════
|
| 311 |
+
# MAIN
|
| 312 |
+
# ═══════════════════════════════════════════════════════════
|
| 313 |
+
|
| 314 |
+
echo ""
|
| 315 |
+
echo -e "${GREEN}╔═══════════════════════════════════════════════════════════╗${NC}"
|
| 316 |
+
echo -e "${GREEN}║ GDELT Engine Stress Test Suite ║${NC}"
|
| 317 |
+
echo -e "${GREEN}╚═══════════════════════════════════════════════════════════╝${NC}"
|
| 318 |
+
echo ""
|
| 319 |
+
echo -e "API URL: ${BLUE}${API_URL}${NC}"
|
| 320 |
+
echo ""
|
| 321 |
+
|
| 322 |
+
# Find engine PID if running
|
| 323 |
+
ENGINE_PID=$(pgrep -f "go run cmd/main.go" 2>/dev/null || pgrep -f "./engine" 2>/dev/null || echo "")
|
| 324 |
+
if [ -n "$ENGINE_PID" ]; then
|
| 325 |
+
echo -e "Engine PID: ${BLUE}${ENGINE_PID}${NC}"
|
| 326 |
+
fi
|
| 327 |
+
|
| 328 |
+
# Parse arguments
|
| 329 |
+
case "${1:-all}" in
|
| 330 |
+
health)
|
| 331 |
+
test_health
|
| 332 |
+
;;
|
| 333 |
+
stats)
|
| 334 |
+
test_stats
|
| 335 |
+
;;
|
| 336 |
+
validate)
|
| 337 |
+
test_validation
|
| 338 |
+
;;
|
| 339 |
+
single)
|
| 340 |
+
test_single
|
| 341 |
+
;;
|
| 342 |
+
batch)
|
| 343 |
+
test_batch_small
|
| 344 |
+
;;
|
| 345 |
+
stress)
|
| 346 |
+
test_batch_large
|
| 347 |
+
;;
|
| 348 |
+
idempotency)
|
| 349 |
+
test_idempotency
|
| 350 |
+
;;
|
| 351 |
+
all)
|
| 352 |
+
test_health
|
| 353 |
+
test_stats
|
| 354 |
+
test_validation
|
| 355 |
+
# Comment out heavy tests by default
|
| 356 |
+
# test_single
|
| 357 |
+
# test_batch_small
|
| 358 |
+
# test_batch_large
|
| 359 |
+
# test_idempotency
|
| 360 |
+
echo -e "\n${YELLOW}Quick tests complete. Run individual tests for processing:${NC}"
|
| 361 |
+
echo " ./scripts/stress_test.sh single # Test 1 timestamp"
|
| 362 |
+
echo " ./scripts/stress_test.sh batch # Test 4 timestamps"
|
| 363 |
+
echo " ./scripts/stress_test.sh stress # Test 24 timestamps"
|
| 364 |
+
echo " ./scripts/stress_test.sh idempotency # Test duplicate handling"
|
| 365 |
+
;;
|
| 366 |
+
*)
|
| 367 |
+
echo "Usage: $0 {health|stats|validate|single|batch|stress|idempotency|all}"
|
| 368 |
+
exit 1
|
| 369 |
+
;;
|
| 370 |
+
esac
|
| 371 |
+
|
| 372 |
+
echo ""
|
| 373 |
+
echo -e "${GREEN}Done!${NC}"
|