Spaces:
Sleeping
Sleeping
| Dec 11 20:12:11 INFO: Started server process [2] | |
| Dec 11 20:12:11 INFO: Waiting for application startup. | |
| Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: ============================================================ | |
| Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: 🚀 SwiftOps API v1.0.0 | PRODUCTION | |
| Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: 📊 Dashboard: Enabled | |
| Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: ============================================================ | |
| Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: 📦 Database: | |
| Dec 11 20:12:12 INFO: 2025-12-11T20:12:12 - app.main: ✓ Connected | 47 tables | 6 users | |
| Dec 11 20:12:12 INFO: 2025-12-11T20:12:12 - app.main: 💾 Cache & Sessions: | |
| Dec 11 20:12:13 INFO: 2025-12-11T20:12:13 - app.services.otp_service: ✅ OTP Service initialized with Redis storage | |
| Dec 11 20:12:13 INFO: 2025-12-11T20:12:13 - app.main: ✓ Redis: Connected | |
| Dec 11 20:12:13 INFO: 2025-12-11T20:12:13 - app.main: 🔌 External Services: | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✓ Cloudinary: Connected | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✓ Resend: Configured | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ○ WASender: Disconnected | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✓ Supabase: Connected | 6 buckets | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ⏰ Scheduler: | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - apscheduler.scheduler: Adding job tentatively -- it will be properly scheduled when the scheduler starts | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - apscheduler.scheduler: Added job "Daily Field Agent Reconciliation" to job store "default" | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - apscheduler.scheduler: Scheduler started | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.tasks.scheduler: Reconciliation scheduler started (runs at 10 PM Africa/Nairobi) | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✓ Daily reconciliation scheduler started (runs at midnight) | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ============================================================ | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✅ Startup complete | Ready to serve requests | |
| Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ============================================================ | |
| Dec 11 20:12:14 INFO: Application startup complete. | |
| Dec 11 20:12:14 INFO: Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit) | |
| Dec 11 20:12:20 INFO: 10.244.45.245:40870 - "GET /health HTTP/1.1" 200 OK | |
| Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - apscheduler.executors.default: Running job "Daily Field Agent Reconciliation (trigger: cron[hour='22', minute='0'], next run at: 2025-12-11 22:00:00 UTC)" (scheduled at 2025-12-11 22:00:00+00:00) | |
| Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - app.tasks.scheduler: Starting scheduled validation for 2025-12-11 | |
| Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - app.tasks.scheduler: Validating 1 projects for 2025-12-11 | |
| Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - app.services.reconciliation.reconciliation_service: Starting reconciliation: project=0ade6bd1-e492-4e25-b681-59f42058d29a, date=2025-12-11, type=scheduled | |
| Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - app.services.reconciliation.reconciliation_service: Aggregated 0 agents in 210ms | |
| Dec 11 22:00:00 ERROR: 2025-12-11T22:00:00 - app.services.reconciliation.reconciliation_service: Reconciliation failed: (psycopg2.ProgrammingError) can't adapt type 'dict' | |
| Dec 11 22:00:00 [SQL: | |
| Dec 11 22:00:00 UPDATE reconciliation_runs | |
| Dec 11 22:00:00 SET | |
| Dec 11 22:00:00 status = 'completed', | |
| Dec 11 22:00:00 completed_at = NOW(), | |
| Dec 11 22:00:00 agents_processed = %(agents_processed)s, | |
| Dec 11 22:00:00 timesheets_created = %(timesheets_created)s, | |
| Dec 11 22:00:00 timesheets_updated = %(timesheets_updated)s, | |
| Dec 11 22:00:00 assignments_processed = %(assignments_processed)s, | |
| Dec 11 22:00:00 expenses_processed = %(expenses_processed)s, | |
| Dec 11 22:00:00 summary_stats = %(summary_stats)s, | |
| Dec 11 22:00:00 anomalies_detected = %(anomalies)s, | |
| Dec 11 22:00:00 execution_time_ms = %(execution_time_ms)s, | |
| Dec 11 22:00:00 query_time_ms = %(query_time_ms)s, | |
| Dec 11 22:00:00 updated_at = NOW() | |
| Dec 11 22:00:00 WHERE id = %(run_id)s | |
| Dec 11 22:00:00 ] | |
| Dec 11 22:00:00 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}] | |
| Dec 11 22:00:00 (Background on this error at: https://sqlalche.me/e/20/f405) | |
| Dec 11 22:00:00 Traceback (most recent call last): | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context | |
| Dec 11 22:00:00 self.dialect.do_execute( | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute | |
| Dec 11 22:00:00 cursor.execute(statement, parameters) | |
| Dec 11 22:00:00 psycopg2.ProgrammingError: can't adapt type 'dict' | |
| Dec 11 22:00:00 | |
| Dec 11 22:00:00 The above exception was the direct cause of the following exception: | |
| Dec 11 22:00:00 | |
| Dec 11 22:00:00 Traceback (most recent call last): | |
| Dec 11 22:00:00 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 117, in reconcile_project_day | |
| Dec 11 22:00:00 self._complete_run( | |
| Dec 11 22:00:00 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 482, in _complete_run | |
| Dec 11 22:00:00 self.db.execute(query, { | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2308, in execute | |
| Dec 11 22:00:00 return self._execute_internal( | |
| Dec 11 22:00:00 ^^^^^^^^^^^^^^^^^^^^^^^ | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2199, in _execute_internal | |
| Dec 11 22:00:00 result = conn.execute( | |
| Dec 11 22:00:00 ^^^^^^^^^^^^^ | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute | |
| Dec 11 22:00:00 return meth( | |
| Dec 11 22:00:00 ^^^^^ | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection | |
| Dec 11 22:00:00 return connection._execute_clauseelement( | |
| Dec 11 22:00:00 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement | |
| Dec 11 22:00:00 ret = self._execute_context( | |
| Dec 11 22:00:00 ^^^^^^^^^^^^^^^^^^^^^^ | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context | |
| Dec 11 22:00:00 return self._exec_single_context( | |
| Dec 11 22:00:00 ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context | |
| Dec 11 22:00:00 self._handle_dbapi_exception( | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception | |
| Dec 11 22:00:00 raise sqlalchemy_exception.with_traceback(exc_info[2]) from e | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context | |
| Dec 11 22:00:00 self.dialect.do_execute( | |
| Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute | |
| Dec 11 22:00:00 cursor.execute(statement, parameters) | |
| Dec 11 22:00:00 sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'dict' | |
| Dec 11 22:00:00 [SQL: | |
| Dec 11 22:00:00 UPDATE reconciliation_runs | |
| Dec 11 22:00:00 SET | |
| Dec 11 22:00:00 status = 'completed', | |
| Dec 11 22:00:00 completed_at = NOW(), | |
| Dec 11 22:00:00 agents_processed = %(agents_processed)s, | |
| Dec 11 22:00:00 timesheets_created = %(timesheets_created)s, | |
| Dec 11 22:00:00 timesheets_updated = %(timesheets_updated)s, | |
| Dec 11 22:00:00 assignments_processed = %(assignments_processed)s, | |
| Dec 11 22:00:00 expenses_processed = %(expenses_processed)s, | |
| Dec 11 22:00:00 summary_stats = %(summary_stats)s, | |
| Dec 11 22:00:00 anomalies_detected = %(anomalies)s, | |
| Dec 11 22:00:00 execution_time_ms = %(execution_time_ms)s, | |
| Dec 11 22:00:00 query_time_ms = %(query_time_ms)s, | |
| Dec 11 22:00:00 updated_at = NOW() | |
| Dec 11 22:00:00 WHERE id = %(run_id)s | |
| Dec 11 22:00:00 ] | |
| Dec 11 22:00:00 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}] | |
| Dec 11 22:00:00 (Background on this error at: https://sqlalche.me/e/20/f405) | |
| Dec 11 22:00:01 ERROR: 2025-12-11T22:00:01 - app.tasks.scheduler: Failed to validate project 0ade6bd1-e492-4e25-b681-59f42058d29a (Atomio Fttx): Reconciliation failed: (psycopg2.ProgrammingError) can't adapt type 'dict' | |
| Dec 11 22:00:01 [SQL: | |
| Dec 11 22:00:01 UPDATE reconciliation_runs | |
| Dec 11 22:00:01 SET | |
| Dec 11 22:00:01 status = 'completed', | |
| Dec 11 22:00:01 completed_at = NOW(), | |
| Dec 11 22:00:01 agents_processed = %(agents_processed)s, | |
| Dec 11 22:00:01 timesheets_created = %(timesheets_created)s, | |
| Dec 11 22:00:01 timesheets_updated = %(timesheets_updated)s, | |
| Dec 11 22:00:01 assignments_processed = %(assignments_processed)s, | |
| Dec 11 22:00:01 expenses_processed = %(expenses_processed)s, | |
| Dec 11 22:00:01 summary_stats = %(summary_stats)s, | |
| Dec 11 22:00:01 anomalies_detected = %(anomalies)s, | |
| Dec 11 22:00:01 execution_time_ms = %(execution_time_ms)s, | |
| Dec 11 22:00:01 query_time_ms = %(query_time_ms)s, | |
| Dec 11 22:00:01 updated_at = NOW() | |
| Dec 11 22:00:01 WHERE id = %(run_id)s | |
| Dec 11 22:00:01 ] | |
| Dec 11 22:00:01 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}] | |
| Dec 11 22:00:01 (Background on this error at: https://sqlalche.me/e/20/f405) | |
| Dec 11 22:00:01 Traceback (most recent call last): | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context | |
| Dec 11 22:00:01 self.dialect.do_execute( | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute | |
| Dec 11 22:00:01 cursor.execute(statement, parameters) | |
| Dec 11 22:00:01 psycopg2.ProgrammingError: can't adapt type 'dict' | |
| Dec 11 22:00:01 | |
| Dec 11 22:00:01 The above exception was the direct cause of the following exception: | |
| Dec 11 22:00:01 | |
| Dec 11 22:00:01 Traceback (most recent call last): | |
| Dec 11 22:00:01 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 117, in reconcile_project_day | |
| Dec 11 22:00:01 self._complete_run( | |
| Dec 11 22:00:01 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 482, in _complete_run | |
| Dec 11 22:00:01 self.db.execute(query, { | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2308, in execute | |
| Dec 11 22:00:01 return self._execute_internal( | |
| Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^^ | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2199, in _execute_internal | |
| Dec 11 22:00:01 result = conn.execute( | |
| Dec 11 22:00:01 ^^^^^^^^^^^^^ | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute | |
| Dec 11 22:00:01 return meth( | |
| Dec 11 22:00:01 ^^^^^ | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection | |
| Dec 11 22:00:01 return connection._execute_clauseelement( | |
| Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement | |
| Dec 11 22:00:01 ret = self._execute_context( | |
| Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^ | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context | |
| Dec 11 22:00:01 return self._exec_single_context( | |
| Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context | |
| Dec 11 22:00:01 self._handle_dbapi_exception( | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception | |
| Dec 11 22:00:01 raise sqlalchemy_exception.with_traceback(exc_info[2]) from e | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context | |
| Dec 11 22:00:01 self.dialect.do_execute( | |
| Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute | |
| Dec 11 22:00:01 cursor.execute(statement, parameters) | |
| Dec 11 22:00:01 sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'dict' | |
| Dec 11 22:00:01 [SQL: | |
| Dec 11 22:00:01 UPDATE reconciliation_runs | |
| Dec 11 22:00:01 SET | |
| Dec 11 22:00:01 status = 'completed', | |
| Dec 11 22:00:01 completed_at = NOW(), | |
| Dec 11 22:00:01 agents_processed = %(agents_processed)s, | |
| Dec 11 22:00:01 timesheets_created = %(timesheets_created)s, | |
| Dec 11 22:00:01 timesheets_updated = %(timesheets_updated)s, | |
| Dec 11 22:00:01 assignments_processed = %(assignments_processed)s, | |
| Dec 11 22:00:01 expenses_processed = %(expenses_processed)s, | |
| Dec 11 22:00:01 summary_stats = %(summary_stats)s, | |
| Dec 11 22:00:01 anomalies_detected = %(anomalies)s, | |
| Dec 11 22:00:01 execution_time_ms = %(execution_time_ms)s, | |
| Dec 11 22:00:01 query_time_ms = %(query_time_ms)s, | |
| Dec 11 22:00:01 updated_at = NOW() | |
| Dec 11 22:00:01 WHERE id = %(run_id)s | |
| Dec 11 22:00:01 ] | |
| Dec 11 22:00:01 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}] | |
| Dec 11 22:00:01 (Background on this error at: https://sqlalche.me/e/20/f405) | |
| Dec 11 22:00:01 | |
| Dec 11 22:00:01 The above exception was the direct cause of the following exception: | |
| Dec 11 22:00:01 | |
| Dec 11 22:00:01 Traceback (most recent call last): | |
| Dec 11 22:00:01 File "/app/src/app/tasks/scheduler.py", line 132, in validate_all_projects | |
| Dec 11 22:00:01 run_id = service.reconcile_project_day( | |
| Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| Dec 11 22:00:01 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 149, in reconcile_project_day | |
| Dec 11 22:00:01 raise ReconciliationError(f"Reconciliation failed: {str(e)}") from e | |
| Dec 11 22:00:01 app.services.reconciliation.reconciliation_service.ReconciliationError: Reconciliation failed: (psycopg2.ProgrammingError) can't adapt type 'dict' | |
| Dec 11 22:00:01 [SQL: | |
| Dec 11 22:00:01 UPDATE reconciliation_runs | |
| Dec 11 22:00:01 SET | |
| Dec 11 22:00:01 status = 'completed', | |
| Dec 11 22:00:01 completed_at = NOW(), | |
| Dec 11 22:00:01 agents_processed = %(agents_processed)s, | |
| Dec 11 22:00:01 timesheets_created = %(timesheets_created)s, | |
| Dec 11 22:00:01 timesheets_updated = %(timesheets_updated)s, | |
| Dec 11 22:00:01 assignments_processed = %(assignments_processed)s, | |
| Dec 11 22:00:01 expenses_processed = %(expenses_processed)s, | |
| Dec 11 22:00:01 summary_stats = %(summary_stats)s, | |
| Dec 11 22:00:01 anomalies_detected = %(anomalies)s, | |
| Dec 11 22:00:01 execution_time_ms = %(execution_time_ms)s, | |
| Dec 11 22:00:01 query_time_ms = %(query_time_ms)s, | |
| Dec 11 22:00:01 updated_at = NOW() | |
| Dec 11 22:00:01 WHERE id = %(run_id)s | |
| Dec 11 22:00:01 ] | |
| Dec 11 22:00:01 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}] | |
| Dec 11 22:00:01 (Background on this error at: https://sqlalche.me/e/20/f405) | |
| Dec 11 22:00:01 INFO: 2025-12-11T22:00:01 - app.tasks.scheduler: Validation summary: 0/1 projects succeeded | |
| Dec 11 22:00:01 WARNING: 2025-12-11T22:00:01 - app.tasks.scheduler: Failed projects: ['Atomio Fttx'] | |
| Dec 11 22:00:01 INFO: 2025-12-11T22:00:01 - app.tasks.scheduler: Scheduled validation completed for 2025-12-11 | |
| Dec 11 22:00:01 INFO: 2025-12-11T22:00:01 - apscheduler.executors.default: Job "Daily Field Agent Reconciliation (trigger: cron[hour='22', minute='0'], next run at: 2025-12-12 22:00:00 UTC)" executed successfully | |
| Dec 11 22:00:01 INFO: 2025-12-11T22:00:01 - app.tasks.scheduler: Job daily_validation completed successfully | |
| Dec 12 06:27:33 INFO: 10.244.16.189:59402 - "GET /health HTTP/1.1" 200 OK |