Spaces:
Sleeping
Sleeping
File size: 18,613 Bytes
d12a170 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 |
Dec 11 20:12:11 INFO: Started server process [2]
Dec 11 20:12:11 INFO: Waiting for application startup.
Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: ============================================================
Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: 🚀 SwiftOps API v1.0.0 | PRODUCTION
Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: 📊 Dashboard: Enabled
Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: ============================================================
Dec 11 20:12:11 INFO: 2025-12-11T20:12:11 - app.main: 📦 Database:
Dec 11 20:12:12 INFO: 2025-12-11T20:12:12 - app.main: ✓ Connected | 47 tables | 6 users
Dec 11 20:12:12 INFO: 2025-12-11T20:12:12 - app.main: 💾 Cache & Sessions:
Dec 11 20:12:13 INFO: 2025-12-11T20:12:13 - app.services.otp_service: ✅ OTP Service initialized with Redis storage
Dec 11 20:12:13 INFO: 2025-12-11T20:12:13 - app.main: ✓ Redis: Connected
Dec 11 20:12:13 INFO: 2025-12-11T20:12:13 - app.main: 🔌 External Services:
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✓ Cloudinary: Connected
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✓ Resend: Configured
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ○ WASender: Disconnected
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✓ Supabase: Connected | 6 buckets
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ⏰ Scheduler:
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - apscheduler.scheduler: Adding job tentatively -- it will be properly scheduled when the scheduler starts
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - apscheduler.scheduler: Added job "Daily Field Agent Reconciliation" to job store "default"
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - apscheduler.scheduler: Scheduler started
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.tasks.scheduler: Reconciliation scheduler started (runs at 10 PM Africa/Nairobi)
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✓ Daily reconciliation scheduler started (runs at midnight)
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ============================================================
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ✅ Startup complete | Ready to serve requests
Dec 11 20:12:14 INFO: 2025-12-11T20:12:14 - app.main: ============================================================
Dec 11 20:12:14 INFO: Application startup complete.
Dec 11 20:12:14 INFO: Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)
Dec 11 20:12:20 INFO: 10.244.45.245:40870 - "GET /health HTTP/1.1" 200 OK
Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - apscheduler.executors.default: Running job "Daily Field Agent Reconciliation (trigger: cron[hour='22', minute='0'], next run at: 2025-12-11 22:00:00 UTC)" (scheduled at 2025-12-11 22:00:00+00:00)
Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - app.tasks.scheduler: Starting scheduled validation for 2025-12-11
Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - app.tasks.scheduler: Validating 1 projects for 2025-12-11
Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - app.services.reconciliation.reconciliation_service: Starting reconciliation: project=0ade6bd1-e492-4e25-b681-59f42058d29a, date=2025-12-11, type=scheduled
Dec 11 22:00:00 INFO: 2025-12-11T22:00:00 - app.services.reconciliation.reconciliation_service: Aggregated 0 agents in 210ms
Dec 11 22:00:00 ERROR: 2025-12-11T22:00:00 - app.services.reconciliation.reconciliation_service: Reconciliation failed: (psycopg2.ProgrammingError) can't adapt type 'dict'
Dec 11 22:00:00 [SQL:
Dec 11 22:00:00 UPDATE reconciliation_runs
Dec 11 22:00:00 SET
Dec 11 22:00:00 status = 'completed',
Dec 11 22:00:00 completed_at = NOW(),
Dec 11 22:00:00 agents_processed = %(agents_processed)s,
Dec 11 22:00:00 timesheets_created = %(timesheets_created)s,
Dec 11 22:00:00 timesheets_updated = %(timesheets_updated)s,
Dec 11 22:00:00 assignments_processed = %(assignments_processed)s,
Dec 11 22:00:00 expenses_processed = %(expenses_processed)s,
Dec 11 22:00:00 summary_stats = %(summary_stats)s,
Dec 11 22:00:00 anomalies_detected = %(anomalies)s,
Dec 11 22:00:00 execution_time_ms = %(execution_time_ms)s,
Dec 11 22:00:00 query_time_ms = %(query_time_ms)s,
Dec 11 22:00:00 updated_at = NOW()
Dec 11 22:00:00 WHERE id = %(run_id)s
Dec 11 22:00:00 ]
Dec 11 22:00:00 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}]
Dec 11 22:00:00 (Background on this error at: https://sqlalche.me/e/20/f405)
Dec 11 22:00:00 Traceback (most recent call last):
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
Dec 11 22:00:00 self.dialect.do_execute(
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
Dec 11 22:00:00 cursor.execute(statement, parameters)
Dec 11 22:00:00 psycopg2.ProgrammingError: can't adapt type 'dict'
Dec 11 22:00:00
Dec 11 22:00:00 The above exception was the direct cause of the following exception:
Dec 11 22:00:00
Dec 11 22:00:00 Traceback (most recent call last):
Dec 11 22:00:00 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 117, in reconcile_project_day
Dec 11 22:00:00 self._complete_run(
Dec 11 22:00:00 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 482, in _complete_run
Dec 11 22:00:00 self.db.execute(query, {
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2308, in execute
Dec 11 22:00:00 return self._execute_internal(
Dec 11 22:00:00 ^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2199, in _execute_internal
Dec 11 22:00:00 result = conn.execute(
Dec 11 22:00:00 ^^^^^^^^^^^^^
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
Dec 11 22:00:00 return meth(
Dec 11 22:00:00 ^^^^^
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
Dec 11 22:00:00 return connection._execute_clauseelement(
Dec 11 22:00:00 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
Dec 11 22:00:00 ret = self._execute_context(
Dec 11 22:00:00 ^^^^^^^^^^^^^^^^^^^^^^
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
Dec 11 22:00:00 return self._exec_single_context(
Dec 11 22:00:00 ^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
Dec 11 22:00:00 self._handle_dbapi_exception(
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
Dec 11 22:00:00 raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
Dec 11 22:00:00 self.dialect.do_execute(
Dec 11 22:00:00 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
Dec 11 22:00:00 cursor.execute(statement, parameters)
Dec 11 22:00:00 sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'dict'
Dec 11 22:00:00 [SQL:
Dec 11 22:00:00 UPDATE reconciliation_runs
Dec 11 22:00:00 SET
Dec 11 22:00:00 status = 'completed',
Dec 11 22:00:00 completed_at = NOW(),
Dec 11 22:00:00 agents_processed = %(agents_processed)s,
Dec 11 22:00:00 timesheets_created = %(timesheets_created)s,
Dec 11 22:00:00 timesheets_updated = %(timesheets_updated)s,
Dec 11 22:00:00 assignments_processed = %(assignments_processed)s,
Dec 11 22:00:00 expenses_processed = %(expenses_processed)s,
Dec 11 22:00:00 summary_stats = %(summary_stats)s,
Dec 11 22:00:00 anomalies_detected = %(anomalies)s,
Dec 11 22:00:00 execution_time_ms = %(execution_time_ms)s,
Dec 11 22:00:00 query_time_ms = %(query_time_ms)s,
Dec 11 22:00:00 updated_at = NOW()
Dec 11 22:00:00 WHERE id = %(run_id)s
Dec 11 22:00:00 ]
Dec 11 22:00:00 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}]
Dec 11 22:00:00 (Background on this error at: https://sqlalche.me/e/20/f405)
Dec 11 22:00:01 ERROR: 2025-12-11T22:00:01 - app.tasks.scheduler: Failed to validate project 0ade6bd1-e492-4e25-b681-59f42058d29a (Atomio Fttx): Reconciliation failed: (psycopg2.ProgrammingError) can't adapt type 'dict'
Dec 11 22:00:01 [SQL:
Dec 11 22:00:01 UPDATE reconciliation_runs
Dec 11 22:00:01 SET
Dec 11 22:00:01 status = 'completed',
Dec 11 22:00:01 completed_at = NOW(),
Dec 11 22:00:01 agents_processed = %(agents_processed)s,
Dec 11 22:00:01 timesheets_created = %(timesheets_created)s,
Dec 11 22:00:01 timesheets_updated = %(timesheets_updated)s,
Dec 11 22:00:01 assignments_processed = %(assignments_processed)s,
Dec 11 22:00:01 expenses_processed = %(expenses_processed)s,
Dec 11 22:00:01 summary_stats = %(summary_stats)s,
Dec 11 22:00:01 anomalies_detected = %(anomalies)s,
Dec 11 22:00:01 execution_time_ms = %(execution_time_ms)s,
Dec 11 22:00:01 query_time_ms = %(query_time_ms)s,
Dec 11 22:00:01 updated_at = NOW()
Dec 11 22:00:01 WHERE id = %(run_id)s
Dec 11 22:00:01 ]
Dec 11 22:00:01 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}]
Dec 11 22:00:01 (Background on this error at: https://sqlalche.me/e/20/f405)
Dec 11 22:00:01 Traceback (most recent call last):
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
Dec 11 22:00:01 self.dialect.do_execute(
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
Dec 11 22:00:01 cursor.execute(statement, parameters)
Dec 11 22:00:01 psycopg2.ProgrammingError: can't adapt type 'dict'
Dec 11 22:00:01
Dec 11 22:00:01 The above exception was the direct cause of the following exception:
Dec 11 22:00:01
Dec 11 22:00:01 Traceback (most recent call last):
Dec 11 22:00:01 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 117, in reconcile_project_day
Dec 11 22:00:01 self._complete_run(
Dec 11 22:00:01 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 482, in _complete_run
Dec 11 22:00:01 self.db.execute(query, {
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2308, in execute
Dec 11 22:00:01 return self._execute_internal(
Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2199, in _execute_internal
Dec 11 22:00:01 result = conn.execute(
Dec 11 22:00:01 ^^^^^^^^^^^^^
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
Dec 11 22:00:01 return meth(
Dec 11 22:00:01 ^^^^^
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
Dec 11 22:00:01 return connection._execute_clauseelement(
Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
Dec 11 22:00:01 ret = self._execute_context(
Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
Dec 11 22:00:01 return self._exec_single_context(
Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
Dec 11 22:00:01 self._handle_dbapi_exception(
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
Dec 11 22:00:01 raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
Dec 11 22:00:01 self.dialect.do_execute(
Dec 11 22:00:01 File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
Dec 11 22:00:01 cursor.execute(statement, parameters)
Dec 11 22:00:01 sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'dict'
Dec 11 22:00:01 [SQL:
Dec 11 22:00:01 UPDATE reconciliation_runs
Dec 11 22:00:01 SET
Dec 11 22:00:01 status = 'completed',
Dec 11 22:00:01 completed_at = NOW(),
Dec 11 22:00:01 agents_processed = %(agents_processed)s,
Dec 11 22:00:01 timesheets_created = %(timesheets_created)s,
Dec 11 22:00:01 timesheets_updated = %(timesheets_updated)s,
Dec 11 22:00:01 assignments_processed = %(assignments_processed)s,
Dec 11 22:00:01 expenses_processed = %(expenses_processed)s,
Dec 11 22:00:01 summary_stats = %(summary_stats)s,
Dec 11 22:00:01 anomalies_detected = %(anomalies)s,
Dec 11 22:00:01 execution_time_ms = %(execution_time_ms)s,
Dec 11 22:00:01 query_time_ms = %(query_time_ms)s,
Dec 11 22:00:01 updated_at = NOW()
Dec 11 22:00:01 WHERE id = %(run_id)s
Dec 11 22:00:01 ]
Dec 11 22:00:01 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}]
Dec 11 22:00:01 (Background on this error at: https://sqlalche.me/e/20/f405)
Dec 11 22:00:01
Dec 11 22:00:01 The above exception was the direct cause of the following exception:
Dec 11 22:00:01
Dec 11 22:00:01 Traceback (most recent call last):
Dec 11 22:00:01 File "/app/src/app/tasks/scheduler.py", line 132, in validate_all_projects
Dec 11 22:00:01 run_id = service.reconcile_project_day(
Dec 11 22:00:01 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 22:00:01 File "/app/src/app/services/reconciliation/reconciliation_service.py", line 149, in reconcile_project_day
Dec 11 22:00:01 raise ReconciliationError(f"Reconciliation failed: {str(e)}") from e
Dec 11 22:00:01 app.services.reconciliation.reconciliation_service.ReconciliationError: Reconciliation failed: (psycopg2.ProgrammingError) can't adapt type 'dict'
Dec 11 22:00:01 [SQL:
Dec 11 22:00:01 UPDATE reconciliation_runs
Dec 11 22:00:01 SET
Dec 11 22:00:01 status = 'completed',
Dec 11 22:00:01 completed_at = NOW(),
Dec 11 22:00:01 agents_processed = %(agents_processed)s,
Dec 11 22:00:01 timesheets_created = %(timesheets_created)s,
Dec 11 22:00:01 timesheets_updated = %(timesheets_updated)s,
Dec 11 22:00:01 assignments_processed = %(assignments_processed)s,
Dec 11 22:00:01 expenses_processed = %(expenses_processed)s,
Dec 11 22:00:01 summary_stats = %(summary_stats)s,
Dec 11 22:00:01 anomalies_detected = %(anomalies)s,
Dec 11 22:00:01 execution_time_ms = %(execution_time_ms)s,
Dec 11 22:00:01 query_time_ms = %(query_time_ms)s,
Dec 11 22:00:01 updated_at = NOW()
Dec 11 22:00:01 WHERE id = %(run_id)s
Dec 11 22:00:01 ]
Dec 11 22:00:01 [parameters: {'agents_processed': 0, 'timesheets_created': 0, 'timesheets_updated': 0, 'assignments_processed': 0, 'expenses_processed': 0, 'summary_stats': {}, 'anomalies': [], 'execution_time_ms': 572, 'query_time_ms': 209, 'run_id': '2746d0ae-b21c-4161-bb5b-95e96a114142'}]
Dec 11 22:00:01 (Background on this error at: https://sqlalche.me/e/20/f405)
Dec 11 22:00:01 INFO: 2025-12-11T22:00:01 - app.tasks.scheduler: Validation summary: 0/1 projects succeeded
Dec 11 22:00:01 WARNING: 2025-12-11T22:00:01 - app.tasks.scheduler: Failed projects: ['Atomio Fttx']
Dec 11 22:00:01 INFO: 2025-12-11T22:00:01 - app.tasks.scheduler: Scheduled validation completed for 2025-12-11
Dec 11 22:00:01 INFO: 2025-12-11T22:00:01 - apscheduler.executors.default: Job "Daily Field Agent Reconciliation (trigger: cron[hour='22', minute='0'], next run at: 2025-12-12 22:00:00 UTC)" executed successfully
Dec 11 22:00:01 INFO: 2025-12-11T22:00:01 - app.tasks.scheduler: Job daily_validation completed successfully
Dec 12 06:27:33 INFO: 10.244.16.189:59402 - "GET /health HTTP/1.1" 200 OK |