Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+airflow/airflow.db filter=lfs diff=lfs merge=lfs -text
+duckdb_databases/financial_data.db filter=lfs diff=lfs merge=lfs -text
+minio filter=lfs diff=lfs merge=lfs -text
+temp/BTCUSDT-1s-2025-08.csv filter=lfs diff=lfs merge=lfs -text
+temp/BTCUSDT-1s-2025-09.csv filter=lfs diff=lfs merge=lfs -text
+temp/temp_parquet_chunks/.part-00000-5d1f072f-086f-4a7a-8c65-5cbc6839e5b5-c000.snappy.parquet.crc filter=lfs diff=lfs merge=lfs -text
+temp/temp_parquet_chunks/.part-00002-5d1f072f-086f-4a7a-8c65-5cbc6839e5b5-c000.snappy.parquet.crc filter=lfs diff=lfs merge=lfs -text
+temp/temp_parquet_chunks/.part-00003-5d1f072f-086f-4a7a-8c65-5cbc6839e5b5-c000.snappy.parquet.crc filter=lfs diff=lfs merge=lfs -text
+zrok filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,208 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+*.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/

.python_version

@@ -0,0 +1 @@
+3.12

CHANGELOG.md

@@ -0,0 +1,789 @@
+# CHANGELOG
+
+## v1.1.9
+
+CHANGE: The `publicProxy` now supports "striped session cookies" to support larger authentication payloads when working with OIDC providers that use larger tokens/payloads. (https://github.com/openziti/zrok/issues/1101)
+
+FIX: Fix for icon/favicon in HTML for the api console. (https://github.com/openziti/zrok/pull/1094)
+
+## v1.1.8
+
+CHANGE: The `ContextDialer` in `agent.Controller.NewClient` now utilizes the "two-phase" approach to invoking `Dial` on the remote agent service. First, it attempts to dial the service with the current service list state. If that fails, it will call `RefreshService` to update the service list and does a second `Dial`. If the second `Dial` fails, then the connection fails. This works around service list staleness issues. (https://github.com/openziti/zrok/issues/1090)
+
+CHANGE: `github.com/openziti/sdk-golang` updated to `v1.2.8`.
+
+CHANGE: CI pull requests now trigger a native-architecture Windows build.
+
+## v1.1.7
+
+FIX: Missing import for windows-specific build.
+
+## v1.1.6
+
+FEATURE: The `agent.Agent` now can optionally enforce that agent remoting starts successfully when creating a new instance. The `agent.Config` struct has a new `RequireRemoting` boolean to control this behavior. (https://github.com/openziti/zrok/issues/1085)
+
+CHANGE: Additional diagnostic logging in the zrok Agent; details around sub-process executions, etc. (https://github.com/openziti/zrok/issues/1084)
+
+## v1.1.5
+
+CHANGE: Upgraded go toolchain to `v1.24.6`. (https://github.com/openziti/zrok/issues/1072)
+
+## v1.1.4
+
+CHANGE: Update `github.com/caddyserver/caddy/v2` to `v2.9.1`; fixes CVE-2024-53259 (would only potentially effect users using the QUIC protocol, very atypical) (https://github.com/openziti/zrok/issues/1047)
+
+## v1.1.3
+
+FEATURE: A new `compatibility` > `version_patterns` array is included in the controller configuration, allowing for dynamic adjustment of allowed client version strings (https://github.com/openziti/zrok/issues/1030)
+
+FEATURE: A new `compatibility` > `log_version` boolean is included in the controller configuration. When this boolean is set to `true`, the controller will log all client versions provided for compatibility checking.
+
+CHANGE: Update `github.com/openziti/sdk-golang` to `v1.2.3`
+
+CHANGE: Minor vulnerability packages updated in `ui` and `agent/agentUi`
+
+FIX: The `scope` field of the metrics returned from `/metrics/environment/...` is now properly set as `environment` and the from `/metrics/share/...` is now properly set as `share` (https://github.com/openziti/zrok/issues/1031)
+
+## v1.1.2
+
+FIX: A panic happened in the `publicProxy` implementation when no `oauth` config block is present (https://github.com/openziti/zrok/issues/1032)
+
+## v1.1.1
+
+FIX: Masquerade as `v1.0-v1.1.1 [gitHash]` when performing client version checks. Will be replaced with the usual client identifier in `v1.1.2` when the regular expressions for controlling client compatibility are externalized in the controller config (https://github.com/openziti/zrok/issues/1028)
+
+## v1.1.0
+
+FEATURE: Rewritten and improved `publicProxy` package (`zrok access public`), with support for extensible OAuth-compliant identity providers. The `publicProxy` configuration now supports any number of configured OAuth-compliant providers (rather than just a single `google` provider and/or a single `github` provider). Also includes a new OIDC-compliant generic IDP provider integration. Improvements to authentication flows and security all around. See the [updated guide](https://docs.zrok.io/docs/guides/self-hosting/oauth/configuring-oauth/) on using OAuth-based identity providers with the zrok public frontend (https://github.com/openziti/zrok/issues/968)
+
+FEATURE: Templatized and improved static pages (not found/404, unauthorized/401, health check, etc.) used by the public frontend. Consolidated variable data using golang `text/template` so that static `proxyUi` package can display additional error information and provide extension points for replacing all of the templated content with external files. See the [error pages guide](https://docs.zrok.io/docs/guides/self-hosting/error-pages/) for more information on customizing the built-in template (https://github.com/openziti/zrok/issues/1012)
+
+FEATURE: `zrok access private` now includes a `--template-path` allowing the embedded `proxyUi` template to be replaced with an external HTML file (https://github.com/openziti/zrok/issues/1012)
+
+FIX: Invoking `/agent/*` endpoints to remotely manage agents with remoting was causing a new API session to be allocated in the ziti controller for each request. A slightly different strategy was employed for embedding the ziti SDK into the zrok controller that should mitigate this (https://github.com/openziti/zrok/issues/1023)
+
+## v1.0.8
+
+FEATURE: New opt-in configuration item `superNetwork` which enables multiple data plane connections to the OpenZiti underlay, a separate control plane connection, enabling SDK-based flow control. To opt-in use `zrok config set superNetwork true` in each environment, or set the `ZROK_SUPER_NETWORK` environment variable to `true` (https://github.com/openziti/zrok/issues/1010)
+
+CHANGE: Updated `github.com/openziti/sdk-golang` to `v1.2.1` (https://github.com/openziti/zrok/issues/1010)
+
+## v1.0.7
+
+FEATURE: zrok Agent now supports health checks (against the target endpoint) for `proxy` backend shares using the `zrok agent share http-healthcheck` command. The zrok API now includes an `/agent/share/http-healthcheck` endpoint for remotely performing these checks against remoted Agents. See the guide for using the feature at https://docs.zrok.io/guides/agent/http-healthcheck/ (https://github.com/openziti/zrok/issues/1002)
+
+FEATURE: `/overview`, `/detail/share`, `/detail/environment`, and `/overview/{organizationToken}/{accountEmail}` all adjusted to include `envZId` in share detail output (https://github.com/openziti/zrok/issues/998)
+
+FEATURE: New add and delete API endpoints for frontend grants. New `zrok admin create frontend-grant` and `zrok admin delete frontend-grant` CLI for invoking these API endpoints from the command line (https://github.com/openziti/zrok/issues/992)
+
+FEATURE: New admin endpoint for deleting accounts. New `zrok admin delete account` CLI for invoking the API endpoint from the command line (https://github.com/openziti/zrok/issues/993)
+
+FEATURE: New admin endpoint for deleting identities. New `zrok admin delete identity` CLI for invoking the API endpoint from the command line (https://github.com/openziti/zrok/issues/800)
+
+FEATURE: New API endpoint (`/overview/public-frontends`) that returns the public frontends available to authenticated account. The public frontends include those marked with the `open` permission mode, and those marked `closed` where the user has a frontend grant allowing them to access the frontend. New CLI command `zrok overview public-frontends` to allow end users to list the public frontends their account can use (https://github.com/openziti/zrok/issues/996)
+
+CHANGE: Updated `openapi-generator-cli` from `7.12.0` to `7.14.0`
+
+## v1.0.6
+
+CHANGE: The `/overview` endpoint has been adjusted to include a new `remoteAgent` `boolean` on the `environment` instances, indicating whether or not the environment has an enrolled remote agent (https://github.com/openziti/zrok/issues/977)
+
+CHANGE: Adjusted core framework entry points to support changing zrokdir, and host interrogation functions to better support embedded zrok functionality (https://github.com/openziti/zrok/issues/976)
+
+## v1.0.5
+
+FEATURE: Initial support for zrok Agent remoting; new `zrok agent enroll` and `zrok agent unenroll` commands that establish opt-in remote Agent management facilities on a per-environment basis. The central API has been augmented to allow for remote control (creating shares and private access instances) of these agents; see the [remoting guide](https://docs.zrok.io/docs/guides/agent/remoting) for details (https://github.com/openziti/zrok/issues/967)
+
+CHANGE: `zrok share public`, `zrok share private`, and `zrok reserve` all default to the "closed" permission mode (they previously defaulted to the "open" permission mode). The `--closed` flag has been replaced with a new `--open` flag. See the [Permission Modes](https://docs.zrok.io/docs/guides/permission-modes/) docs for details (https://github.com/openziti/zrok/issues/971)
+
+FIX: `zrok enable` now handles the case where the user ID does not resolve to a username when generating the default environment description (https://github.com/openziti/zrok/issues/959)
+
+FIX: Linux packages were optimized to avoid manage file revision conflicts (https://github.com/openziti/zrok/issues/817)
+
+## v1.0.4
+
+FIX: `zrok admin bootstrap` and `zrok enable` functionality were broken in `v1.0.3`. A bad combination of dependencies caused issues with marshalling data from the associated controller endpoints
+
+CHANGE: `github.com/openziti/sdk-golang` has been updated to `v1.1.0`, `github.com/openziti/ziti` has been updated to `v1.6.0`. Related dependencies and indirects also updated
+
+CHANGE: Updated to `golang` `v1.24` as the official build toolchain
+
+## v1.0.3
+
+FEATURE: `zrok agent console` now outputs the URL it is attempting to open. New `zrok agent console --headless` option to only emit the agent console URL (https://github.com/openziti/zrok/issues/944)
+
+FEATURE: New `zrok admin unbootstrap` to remove zrok resources from the underlying OpenZiti instance (https://github.com/openziti/zrok/issues/935)
+
+FEATURE: New InfluxDB metrics capture infrastructure for `zrok test canary` framework (https://github.com/openziti/zrok/issues/948)
+
+FEATURE: New `zrok test canary enabler` to validate `enable`/`disable` operations and gather performance metrics around how those paths are operating (https://github.com/openziti/zrok/issues/771)
+
+FEATURE: New `zrok test canary` infrastructure capable of supporting more complex testing scenarios; now capable of streaming canary metrics into an InfluxDB repository; new programming framework for developing additional types of streaming canary metrics (https://github.com/openziti/zrok/issues/948 https://github.com/openziti/zrok/issues/954)
+
+FEATURE: All `zrok test canary` commands that have "min" and "max" values (`--min-pacing` and `--max-pacing` for example) now include a singular version of that flag for setting both "min" and "max" to the same value (`--pacing` for example). The singular version of the flag always overrides any `--min-*` or `--max-*` values that might be set
+
+CHANGE: New _guard_ to prevent users from running potentially dangerous `zrok test canary` commands inadvertently without understanding what they do (https://github.com/openziti/zrok/issues/947)
+
+CHANGE: Updated `npm` dependencies for `ui`, `agent/agentUi` and `website`. Updated `github.com/openziti/sdk-golang` to `v0.24.0`
+
+## v1.0.2
+
+FEATURE: "Auto-rebase" for enabled environments where the `apiEndpoint` is set to `https://api.zrok.io`. This will automatically migrate existing environments to the new `apiEndpoint` for the `v1.0.x` series (https://github.com/openziti/zrok/issues/936)
+
+FEATURE: New `admin/new_account_link` configuration option to allow the insertion of "how do I register for an account?" links into the login form (https://github.com/openziti/zrok/issues/552)
+
+CHANGE: The release environment, share, and access modals in the API console now have a better message letting the user know they will still need to clean up their `zrok` processes (https://github.com/openziti/zrok/issues/910)
+
+CHANGE: The openziti/zrok Docker image has been updated to use the latest version of the ziti CLI, 1.4.3 (https://github.com/openziti/zrok/pull/917)
+
+## v1.0.1
+
+FEATURE: The zrok Agent now persists private accesses and reserved shares between executions. Any `zrok access private` instances or `zrok share reserved` instances created using the agent are now persisted to a registry stored in `${HOME}/.zrok`. When restarting the agent these accesses and reserved shares are re-created from the data in this registry (https://github.com/openziti/zrok/pull/922)
+
+FEATURE: zrok-agent Linux package runs the agent as a user service (https://github.com/openziti/zrok/issues/883)
+
+CHANGE: Updated the "Getting Started" guide to be slightly more streamlined and reflect the `v1.0` changes (https://github.com/openziti/zrok/issues/877)
+
+CHANGE: let the Docker instance set the Caddy HTTPS port (https://github.com/openziti/zrok/pull/920)
+
+CHANGE: Add Traefik option for TLS termination in the Docker instance (https://github.com/openziti/zrok/issues/808)
+
+## v1.0.0
+
+MAJOR RELEASE: zrok reaches version 1.0.0!
+
+FEATURE: Completely redesigned web interface ("API Console"). New implementation provides a dual-mode interface supporting an improved visual network navigator and also a "tabular" view, which provides a more traditional "data" view. New stack built using vite, React, and TypeScript (https://github.com/openziti/zrok/issues/724)
+
+FEATURE: New "zrok Agent", a background manager process for your zrok environments, which allows you to easily manage and work with multiple `zrok share` and `zrok access` processes. New `--subordinate` flag added to `zrok share [public|private|reserved]` and `zrok access private` to operate in a mode that allows an Agent to manage shares and accesses (https://github.com/openziti/zrok/issues/463)
+
+FEATURE: New "zrok Agent UI" a web-based user interface for the zrok Agent, which allows creating and releasing shares and accesses through a web browser. This is just an initial chunk of the new Agent UI, and is considered a "minimum viable" version of this interface (https://github.com/openziti/zrok/issues/221)
+
+FEATURE: `zrok share [public|private|reserved]` and `zrok access private` now auto-detect if the zrok Agent is running in an environment and will automatically service share and access requests through the Agent, rather than in-process if the Agent is running. If the Agent is not running, operation remains as it was in `v0.4.x` and the share or access is handled in-process. New `--force-agent` and `--force-local` flags exist to skip Agent detection and manually select an operating mode (https://github.com/openziti/zrok/issues/751)
+
+FEATURE: `zrok access private` supports a new `--auto` mode, which can automatically find an available open address/port to bind the frontend listener on. Also includes `--auto-address`, `--auto-start-port`, and `--auto-end-port` features with sensible defaults. Supported by both the agent and local operating modes (https://github.com/openziti/zrok/issues/780)
+
+FEATURE: `zrok rebase` commands (`zrok rebase apiEndpoint` and `zrok rebase accountToken`) allows "rebasing" an enabled environment onto a different API endpoint or a different account token. This is useful for migrating already-enabled environments between endpoints supporting different zrok versions, and is also useful when regenerating an account token (https://github.com/openziti/zrok/issues/869, https://github.com/openziti/zrok/issues/897)
+
+FEATURE: `zrok test canary` CLI tree replaces the old `zrok test loop` tree; new `zrok test canary public-proxy` and `zrok test canary private-proxy` provide modernized, updated versions of what the `zrok test loop` commands used to do. This new approach will serve as the foundation for all future zrok testing infrastructure (https://github.com/openziti/zrok/issues/771)
+
+FEATURE: New `/api/v1/versions` endpoint to return comprehensive, full stack version information about the deployed service instance. Currently only returns a single `controllerVersion` property (https://github.com/openziti/zrok/issues/881) 
+
+CHANGE: The default API URL for `v1.0.x` zrok clients is now `https://api-v1.zrok.io` (instead of the older `https://api.zrok.io`). The zrok.io deployment will now be maintaining version-specific DNS for versioned API endpoints.
+
+CHANGE: Refactored API implementation. Cleanup, lint removal, additional data elements added, unused data removed (https://github.com/openziti/zrok/issues/834)
+
+CHANGE: Deprecated the `passwords` configuration stanza. The zrok controller and API console now use a hard-coded set of (what we believe to be) reasonable assumptions about password quality (https://github.com/openziti/zrok/issues/834)
+
+CHANGE: The protocol for determining valid client versions has been changed. Previously a zrok client would do a `GET` against the `/api/v1/version` endpoint and do a local version string comparison (as a normal precondition to any API call) to see if the controller version matched. The protocol has been amended so that any out-of-date client using the old protocol will receive a version string indicating that they need to uprade their client. New clients will do a `POST` against the `/api/v1/clientVersionCheck` endpoint, posting their client version, and the server will check for compatibility. Does not change the security posture in any significant way, but gives more flexibility on the server side for managing client compatibility. Provides a better, cleared out-of-date error message for old clients when accessing `v1.0.0`+ (https://github.com/openziti/zrok/issues/859)
+
+CHANGE: The Node.js SDK is now generated by `openapi-generator` using the `typescript-fetch` template. Examples and SDK components updated to use the `v1.0.0` API and generated client (https://github.com/openziti/zrok/issues/893)
+
+CHANGE: The Python SDK is now generated by `openapi-generator` and requires a newer `urllib3` version 2.1.0. The published Python module, `zrok`, inherits the dependencies of the generated packages (https://github.com/openziti/zrok/issues/894)
+
+## v0.4.49
+
+FIX: Release artifacts now include a reproducible source archive. The archive's download URL is now used by the Homebrew formula when building from source instead of the archive generated on-demand by GitHub (https://github.com/openziti/zrok/issues/858).
+
+FIX: Pre-releases are no longer uploaded to the stable Linux package repo, and workflows that promote stable release artifacts to downstream distribution channels enforce semver stable release tags, i.e., not having a semver hyphenated prerelease suffix.
+
+CHANGE: The release `checksums.txt` has been renamed `checksums.sha256.txt` to reflect the use of a collision-resistant algorithm instead of `shasum`'s default algorithm, SHA-1.
+
+CHANGE: The dependency graph is now published as a release artifact named `sbom-{version}.spdx.json` (https://github.com/openziti/zrok/issues/888).
+
+CHANGE: Pre-releases are uploaded to the pre-release Linux package repo and Docker Hub for testing. [RELEASING.md](./RELEASING.md) describes releaser steps and the events they trigger.
+
+CHANGE: Linux release binaries are now built on the ziti-builder container image based on Ubuntu Focal 20.04 to preserve backward compatibility as the ubuntu-20.04 GitHub runner is end of life.
+
+CHANGE: Container images now include SLSA and SBOM attestations, and these are also published to the Docker Hub registry (https://github.com/openziti/zrok/issues/890).
+
+CHANGE: Release binary and text artifacts are now accompanied by provenance attestations (https://github.com/openziti/zrok/issues/889).
+
+## v0.4.48
+
+FEATURE: The controller configuration now supports a `disable_auto_migration` boolean in the `store` stanza. When set to `true`, the controller will not attempt to auto-migrate (or otherwise validate the migration state) of the underlying database. Leaving `disable_auto_migration` out, or setting it to false will retain the default behavior of auto-migrating when starting the zrok controller. The `zrok admin migrate` command will still perform a migration regardless of how this setting is configured in the controller configuration (https://github.com/openziti/zrok/issues/866)
+
+FIX: the Python SDK erroneously assumed the enabled zrok environment contained a config.json file, and was changed to only load it if the file was present (https://github.com/openziti/zrok/pull/853/).
+
+## v0.4.47
+
+CHANGE: the Docker instance will wait for the ziti container healthy status (contribution from Ben Wong @bwong365 - https://github.com/openziti/zrok/pull/790)
+
+CHANGE: Document solving the DNS propagation timeout for Docker instances that are using Caddy to manage the wildcard certificate.
+
+CHANGE: Add usage hint in `zrok config get --help` to clarify how to list all valid `configName` and their current values by running `zrok status`.
+
+CHANGE: The Python SDK's `Overview()` function was refactored as a class method (https://github.com/openziti/zrok/pull/846).
+
+FEATURE: The Python SDK now includes a `ProxyShare` class providing an HTTP proxy for public and private shares and a
+Jupyter notebook example (https://github.com/openziti/zrok/pull/847).
+
+FIX: PyPi publishing was failing due to a CI issue (https://github.com/openziti/zrok/issues/849)
+
+## v0.4.46
+
+FEATURE: Linux service template for systemd user units (https://github.com/openziti/zrok/pull/818)
+
+FIX: Docker share examples had incorrect default path for zrok environment mountpoint
+
+FIX: Clarify how to use DNS providers like Route53 with the zrok Docker instance sample.
+
+CHANGE: Use port 80 for the default Ziti API endpoint in the zrok Docker instance sample (https://github.com/openziti/zrok/issues/793).
+
+CHANGE: Clarify OS requirements for zrok VPN
+
+CHANGE: Set the Windows executable search path in the Windows install guide.
+
+CHANGE: bump macOS runner for Python module from macos-12 to macos-13
+
+## v0.4.45
+
+FEATURE: Minimal support for "organizations". Site admin API endpoints provided to create, list, and delete "organizations". Site admin API endpoints provided to add, list, and remove "organization members" (zrok accounts) with the ability to mark accounts as a "organization admin". API endpoints provided for organization admins to list the members of their organizations, and to also see the overview (environments, shares, and accesses) for any account in their organization. API endpoint for end users to see which organizations their account is a member of (https://github.com/openziti/zrok/issues/537)
+
+CHANGE: briefly mention the backend modes that apply to public and private share concepts
+
+FIX: Update indirect dependency `github.com/golang-jwt/jwt/v4` to version `v4.5.1` (https://github.com/openziti/zrok/issues/794)
+
+FIX: Document unique names
+
+FIX: reduce Docker image sizes (https://github.com/openziti/zrok/pull/783)
+
+FIX: Docker reserved private share startup error (https://github.com/openziti/zrok/pull/801)
+
+FIX: Correct the download URL for the armv7 Linux release (https://github.com/openziti/zrok/issues/782)
+
+## v0.4.44
+
+FIX: Fix for goreleaser build action to align with changed ARM64 build path.
+
+## v0.4.43
+
+CHANGE: Update `github.com/openziti/sdk-golang` to version `v0.23.44`. Remove old `github.com/openziti/fabric` dependency, instead pulling in the modern `github.com/openziti/ziti` dependency.
+
+FIX: Bypass interstitial page for HTTP `OPTIONS` method (https://github.com/openziti/zrok/issues/777)
+
+## v0.4.42
+
+CHANGE: Switch all `Dial` operations made into the OpenZiti overlay to use `DialWithOptions(..., &ziti.DialOptions{ConnectTimeout: 30 * time.Second})`, switching to a 30 second timeout from a 5 second default (https://github.com/openziti/zrok/issues/772)
+
+FIX: Removed the `--basic-auth` flag from `zrok share private` as this was ignored... even if `zrok access private` honored the `ziti.proxy.v1` config to ask for basic auth, it would still be easy to write a custom SDK client that ignored the basic auth and accessed the share directly; better to remove the option than to allow confusing usage (https://github.com/openziti/zrok/issues/770)
+
+FIX: always append common options like `--headless` and conditionally append `--verbose --insecure` if their respective env vars are set to when running in a service manager like systemd or Docker and wrapping the `zrok` command with the `zrok-share.bash` shell script (https://openziti.discourse.group/t/question-about-reserved-public-vs-temp-public-shares/3169)
+
+FIX: Correct registration page CSS to ensure that the entire form is visible
+
+## v0.4.41
+
+FIX: Fixed crash when invoking `zrok share reserved` with no arguments (https://github.com/openziti/zrok/issues/740)
+
+FIX: zrok-share.service on Linux failed to start with a private share in closed permission mode
+
+FIX: Update `gopkg.in/go-jose/go-jose.v2` to `v2.6.3` to fix vulnerability around compressed data (https://github.com/openziti/zrok/issues/761)
+
+## v0.4.40
+
+FEATURE: New endpoint for synchronizing grants for an account (https://github.com/openziti/zrok/pull/744). Useful for updating the `zrok.proxy.v1` config objects containing interstitial setting when the `skip_interstitial_grants` table has been updated.
+
+FIX: prune incorrect troubleshooting advice about listing Caddy's certificates
+
+## v0.4.39
+
+FEATURE: New API endpoint allowing direct creation of accounts in the zrok database. Requires an admin token (specified in the controller configuration yaml) for authentication. See the OpenAPI spec for details of the API endpoint. The `zrok admin create account` CLI was also updated to call the API endpoint, rather than directly operating on the underlying database (https://github.com/openziti/zrok/issues/734). The [Docker](https://github.com/openziti/zrok/pull/736) and [Kubernetes](https://github.com/openziti/helm-charts/pull/249) zrok instance deployments were adapted to the new CLI parameter shape.
+
+FEATURE: Support `html_path` directive in `interstitial` stanza of public frontend configuration to support using an external HTML file for the interstitial page (https://github.com/openziti/zrok/issues/716)
+
+FEATURE: `zrok access private` now includes a `--response-header` flag to add headers to the response for HTTP-based backends. Add flag multiple times to add multiple headers to the response. Expects `key:value` header definitions in this format: `--response-header "Access-Control-Allow-Origin: *"` (https://github.com/openziti/zrok/issues/522)
+
+CHANGE: Update `github.com/openziti/sdk-golang` (and related dependencies) to version `v0.23.40`.
+
+CHANGE: upgrade to ziti v1.1.7 CLI in zrok container image
+
+## v0.4.38
+
+FEATURE: Conditionally enable interstitial page based on `User-Agent` prefix list. See the [frontend configuration template](etc/frontend.yml) for details on the new configuration structure (https://github.com/openziti/zrok/issues/715) 
+
+CHANGE: The interstitial configuration has been modified from a simple `interstitial: <bool>` to a richer structure, but the config version has not been incremented; this feature has not been widely adopted yet. See the [frontend configuration template](etc/frontend.yml) for details on the new structure.
+
+CHANGE: The registration page where a new user's password is set now includes a required checkbox, asking them to acknowledge the terms and conditions presented above the checkbox (https://github.com/openziti/zrok/issues/669)
+
+FIX: The registration page where a new user's password is set now includes better styling of the error message `<div/>` to prevent the entire page from jumping when the message changes.
+
+## v0.4.37
+
+FIX: Fix for setting the `zrok_interstitial` cookie on Chrome-based browsers.
+
+FIX: Fix for `store.IsAccountGrantedSkipInterstitial` to respect the `deleted` flag.
+
+FIX: When an error occurs connecting to the proxied endpoint, the `proxy` backend should return HTTP status `502` (https://github.com/openziti/zrok/issues/703)
+
+## v0.4.36
+
+FEATURE: New interstitial pages that can be enabled per-frontend, and disabled per-account (https://github.com/openziti/zrok/issues/704)
+
+CHANGE: Enable `"declaration": true` in `tsconfig.json` for Node SDK.
+
+FIX: build 32bit build for armhf to fix [the FPE issue](https://github.com/openziti/zrok/issues/654) and [the missing link issue](https://github.com/openziti/zrok/issues/642)
+
+CHANGE: add [cross-build instructions](./BUILD.md) (includes new snapshot build target `armel`)
+
+## v0.4.35
+
+FEATURE: Added import for `github.com/greenpau/caddy-security` to include that Caddy plugin to enable authentication, authorization, and credentials extensions for the `caddy` backend (https://github.com/openziti/zrok/issues/506)
+
+FEATURE: Closed permission mode for Docker and Linux private shares
+
+CHANGE: add example in ./etc/caddy to set X-Real-IP header to public share client IP
+
+CHANGE: auto-update the ziti CLI version that is built in to the openziti/zrok container image
+
+CHANGE: Docker examples set HOME to enable running CLI commands in the container
+
+FIX: Fix for environment count inheritance when using a resource count class to override global environment count (https://github.com/openziti/zrok/issues/695)
+
+## v0.4.34
+
+FEATURE: Linux service support for all private share modes (contribution from Stefan Adelbert @stefanadelbert)
+
+FIX: Fix for mixing limited and unlimited (-1) resource counts in the limits system (https://github.com/openziti/zrok/issues/680)
+
+FIX: Fix for sending multiple warning emails when a warning is applied to an account (https://github.com/openziti/zrok/issues/685)
+
+CHANGE: add Docker compose example for multiple share containers using the same enabled environment in [compose.override.yml](./docker/compose/zrok-public-reserved/compose.override.yml)
+
+CHANGE: bump many GitHub Actions that were using deprecated distributions of Node.js
+
+CHANGE: bump macOS runner for Node SDK from macos-11 to macos-12
+
+## v0.4.33
+
+FIX: Fix for log message in `Agent.CanAccessShare` (`"account '#%d' over frontends per share limit '%d'"`), which was not returning the correct limit value.
+
+FIX: Properly set `permission_mode` in `frontends` when createing a private frontend using `zrok access private` (https://github.com/openziti/zrok/issues/677)
+
+CHANGE: Updated `react-bootstrap` to version `2.10.2` (web console).
+
+CHANGE: Updated `@mui/material` to version `5.15.18` (web console).
+
+CHANGE: Updated `react` and `react-dom` to version `18.3.1` (web console).
+
+CHANGE: Updated `recharts` to version `2.12.7` (web console).
+
+CHANGE: Updated `react-router-dom` to version `6.23.1` (web console).
+
+CHANGE: Updated `axios` to version `1.7.2` for (node SDK).
+
+CHANGE: Updated `@openziti/ziti-sdk-nodejs` to version `0.17.0` (node SDK).
+
+## v0.4.32
+
+FEATURE: New permission mode support for public frontends. Open permission mode frontends are available to all users in the service instance. Closed permission mode frontends reference the new `frontend_grants` table that can be used to control which accounts are allowed to create shares using that frontend. `zrok admin create frontend` now supports `--closed` flag to create closed permission mode frontends (https://github.com/openziti/zrok/issues/539)
+
+FEATURE: New config `defaultFrontend` that specifies the default frontend to be used for an environment. Provides the default `--frontend` for `zrok share public` and `zrok reserve public` (https://github.com/openziti/zrok/issues/663)
+
+FEATURE: Resource count limits now include `share_frontends` to limit the number of frontends that are allowed to make connections to a share (https://github.com/openziti/zrok/issues/650)
+
+CHANGE: The frontend selection flag used by `zrok share public` and `zrok reserve public` has been changed from `--frontends` to `--frontend`
+
+FIX: use controller config spec v4 in the Docker instance
+
+## v0.4.31
+
+FEATURE: New "limits classes" limits implementation (https://github.com/openziti/zrok/issues/606). This new feature allows for extensive limits customization on a per-user basis, with fallback to the global defaults in the controller configuration.
+
+CHANGE: The controller configuration version has been updated to version `4` (`v: 4`) to support the new limits global configuration changes (https://github.com/openziti/zrok/issues/606).
+
+CHANGE: A new `ZROK_CTRL_CONFIG_VERSION` environment variable now exists to temporarily force the controller to assume a specific controller configuration version, regardless of what version exists in the file. This allows two different config versions to potentially be co-mingled in the same controller configuration file. Use with care (https://github.com/openziti/zrok/issues/648)
+
+CHANGE: Log messages that said `backend proxy endpoint` were clarified to say `backend target`.
+
+FIX: Correct the syntax for the Docker and Linux zrok-share "frontdoor" service that broke OAuth email address pattern matching.
+
+## v0.4.30
+
+FIX: Fix to the Node.js release process to properly support releasing on a tag.
+
+## v0.4.29
+
+FIX: Backed out an incorrect change to support a FreeBSD port in progress.
+
+## v0.4.28
+
+FEATURE: Node.js support for the zrok SDK (https://github.com/openziti/zrok/issues/400)
+
+FEATURE: A Docker Compose project for self-hosting a zrok instance and [accompanying Docker guide](https://docs.zrok.io/docs/guides/self-hosting/docker) for more information.
+
+CHANGE: the container images run as "ziggy" (UID 2171) instead of the generic restricted user "nobody" (UID 65534). This reduces the risk of unexpected file permissions when binding the Docker host's filesystem to a zrok container.
+
+CHANGE: the Docker sharing guides were simplified and expanded
+
+## v0.4.27
+
+FEATURE: New `vpn` backend mode. Use `sudo zrok share private --backend-mode vpn` on the _VPN server_ host, then `sudo zrok access private <token>` on _VPN client_ machine. Works with reserved shares using `zrok reserve private --backend-mode vpn`. Use `<target>` parameter to override default VPN network settings `zrok share private -b vpn 192.168.255.42/24` -- server IP is `192.168.255.42` and VPN netmask will be `192.168.255.0/24`. Client IPs are assigned automatically from netmask range.
+
+CHANGE: Update to OpenZiti SDK (`github.com/openziti/sdk-golang`) at `v0.23.22`.
+
+CHANGE: Added indexes to `environments`, `shares`, and `frontends` tables to improve overall query performance on both PostgreSQL and Sqlite.
+
+FIX: Also update the Python SDK to include the permission mode and access grants fields on the `ShareRequest` (https://github.com/openziti/zrok/issues/432)
+
+FIX: Add a way to find the username on Linux when /etc/passwd and stdlib can't resolve the UID (https://github.com/openziti/zrok/issues/454)
+
+## v0.4.26
+
+FEATURE: New _permission modes_ available for shares. _Open permission mode_ retains the behavior of previous zrok releases and is the default setting. _Closed permission mode_ (`--closed`) only allows a share to be accessed (`zrok access`) by users who have been granted access with the `--access-grant` flag. See the documentation at (https://docs.zrok.io/docs/guides/permission-modes/) (https://github.com/openziti/zrok/issues/432)
+
+CHANGE: The target for a `socks` share is automatically set to `socks` to improve web console display.
+
+CHANGE: Enhancements to the look and feel of the account actions tab in the web console. Textual improvements.
+
+FIX: The regenerate account token dialog incorrectly specified the path `${HOME}/.zrok/environments.yml`. This, was corrected to be `${HOME}/.zrok/environments.json`.
+
+FIX: Align zrok frontdoor examples and Linux package (`zrok-share`) with the new OAuth email flag `--oauth-email-address-patterns` introduced in v0.4.25.
+
+FIX: Reloading the web console when logged in no longer provokes the user to the login page.
+
+## v0.4.25
+
+FEATURE: New action in the web console that allows changing the password of the logged-in account (https://github.com/openziti/zrok/issues/148)
+
+FEATURE: The web console now supports revoking your current account token and generating a new one (https://github.com/openziti/zrok/issues/191)
+
+CHANGE: When specifying OAuth configuration for public shares from the `zrok share public` or `zrok reserve` public commands, the flags and functionality for restricting the allowed email addresses of the authenticating users has changed. The old flag was `--oauth-email-domains`, which took a string value that needed to be contained in the user's email address. The new flag is `--oauth-email-address-patterns`, which accepts a glob-style filter, using https://github.com/gobwas/glob (https://github.com/openziti/zrok/issues/413)
+
+CHANGE: Creating a reserved share checks for token collision and returns a more appropriate error message (https://github.com/openziti/zrok/issues/531)
+
+CHANGE: Update UI to add a 'true' value on `reserved` boolean (https://github.com/openziti/zrok/issues/443)
+
+CHANGE: OpenZiti SDK (github.com/openziti/sdk-golang) updated to version `v0.22.29`, which introduces changes to OpenZiti API session handling
+
+FIX: Fixed bug where a second password reset request would for any account would fail (https://github.com/openziti/zrok/issues/452)
+
+## v0.4.24
+
+FEATURE: New `socks` backend mode for use with private sharing. Use `zrok share private --backend-mode socks` and then `zrok access private` that share from somewhere else... very lightweight VPN-like functionality (https://github.com/openziti/zrok/issues/558)
+
+FEATURE: New `zrok admin create account` command that allows populating accounts directly into the underlying controller database (https://github.com/openziti/zrok/issues/551)
+
+CHANGE: The `zrok test loopback public` utility to report non-`200` errors and also ensure that the listening side of the test is fully established before starting loopback testing.
+
+CHANGE: The OpenZiti SDK for golang (https://github.com/openziti/sdk-golang) has been updated to version `v0.22.28`
+
+## v0.4.23
+
+FEATURE: New CLI commands have been implemented for working with the `drive` share backend mode (part of the "zrok Drives" functionality). These commands include `zrok cp`, `zrok mkdir` `zrok mv`, `zrok ls`, and `zrok rm`. These are initial, minimal versions of these commands and very likely contain bugs and ergonomic annoyances. There is a guide available at (`docs/guides/drives.mdx`) that explains how to work with these tools in detail (https://github.com/openziti/zrok/issues/438)
+
+FEATURE: Python SDK now has a decorator for integrating with various server side frameworks. See the `http-server` example.
+
+FEATURE: Python SDK share and access handling now supports context management.
+
+FEATURE: TLS for `zrok` controller and frontends. Add the `tls:` stanza to your controller configuration (see `etc/ctrl.yml`) to enable TLS support for the controller API. Add the `tls:` stanza to your frontend configuration (see `etc/frontend.yml`) to enable TLS support for frontends (be sure to check your `public` frontend template) (#24)(https://github.com/openziti/zrok/issues/24)
+
+CHANGE: Improved OpenZiti resource cleanup resilience. Previous resource cleanup would stop when an error was encountered at any stage of the cleanup process (serps, sps, config, service). New cleanup implementation logs errors but continues to clean up anything that it can (https://github.com/openziti/zrok/issues/533)
+
+CHANGE: Instead of setting the `ListenOptions.MaxConnections` property to `64`, use the default value of `3`. This property actually controls the number of terminators created on the underlying OpenZiti network. This property is actually getting renamed to `ListenOptions.MaxTerminators` in an upcoming release of `github.com/openziti/sdk-golang` (https://github.com/openziti/zrok/issues/535)
+
+CHANGE: Versioning for the Python SDK has been updated to use versioneer for management.
+
+CHANGE: Python SDK package name has been renamed to `zrok`, dropping the `-sdk` postfix. [pypi](https://pypi.org/project/zrok).
+
+## v0.4.22
+
+FIX: The goreleaser action is not updated to work with the latest golang build. Modifed `go.mod` to comply with what goreleaser expects
+
+## v0.4.21
+
+FEATURE: The web console now supports deleting `zrok access` frontends (https://github.com/openziti/zrok/issues/504)
+
+CHANGE: The web console now displays the frontend token as the label for any `zrok access` frontends throughout the user interface (https://github.com/openziti/zrok/issues/504)
+
+CHANGE: Updated `github.com/rubenv/sql-migrate` to `v1.6.0`
+
+CHANGE: Updated `github.com/openziti/sdk-golang` to `v0.22.6`
+
+FIX: The migration `sqlite3/015_v0_4_19_share_unique_name_constraint.sql` has been adjusted to delete the old `shares_old` table as the last step of the migration process. Not sure exactly why, but SQLite is unhappy otherwise (https://github.com/openziti/zrok/issues/504)
+
+FIX: Email addresses have been made case-insensitive. Please note that there is a migration included in this release (`016_v0_4_21_lowercase_email.sql`) which will attempt to ensure that all email addresses in your existing database are stored in lowercase; **if this migration fails you will need to manually remediate the duplicate account entries** (https://github.com/openziti/zrok/issues/517)
+
+FIX: Stop sending authentication cookies to non-authenticated shares (https://github.com/openziti/zrok/issues/512)
+
+## v0.4.20
+
+CHANGE: OpenZiti SDK updated to `v0.21.2`. All `ziti.ListenOptions` listener options configured to use `WaitForNEstablishedListeners: 1`. When a `zrok share` client or an `sdk.Share` client are connected to an OpenZiti router that supports "listener established" events, then listen calls will not return until the listener is fully established on the OpenZiti network. Previously a `zrok share` client could report that it is fully operational and listening before the listener is fully established on the OpenZiti network; in practice this produced a very small window of time when the share would not be ready to accept requests. This change eliminates this window of time (https://github.com/openziti/zrok/issues/490)
+
+FIX: Require the JWT in a zrok OAuth cookie to have an audience claim that matches the public share hostname. This prevents a cookie from one share from being use to log in to another share.
+
+## v0.4.19
+
+FEATURE: Reserved shares now support unique names ("vanity tokens"). This allows for the creation of reserved shares with identifiable names rather than generated share tokens. Includes basic support for profanity checking (https://github.com/openziti/zrok/issues/401)
+
+CHANGE: The `publicProxy` endpoint implementation used in the `zrok access public` frontend has been updated to use the new `RefreshService(serviceName)` call instead of `RefreshServices()`. This should greatly improve the performance of requests against missing or non-responsive zrok shares (https://github.com/openziti/zrok/issues/487)
+
+CHANGE: The Python SDK has been updated to properly support the "reserved" flag on the `ShareRequest` passed to `CreateShare`
+
+CHANGE: Dependency updates; `github.com/openziti/sdk-golang@v0.20.145`; `github.com/caddyserver/caddy/v2@2.7.6`; indirect dependencies
+
+## v0.4.18
+
+FEATURE: Python SDK added. Can be found on [pypi](https://test.pypi.org/project/zrok-sdk). `pastebin` example illustrates basic SDK usage (see `sdk/python/examples/README.md` for details) (https://github.com/openziti/zrok/issues/401)
+
+CHANGE: Moved the golang zrok sdk into `sdk/golang/sdk` to normalize location for future SDK's. 
+
+CHANGE: add restart policies to docker compose samples used by the guide docs, e.g., reserved public share should auto-start on boot, temp public share should not.
+
+## v0.4.17
+
+CHANGE: Replaced most in-line shell scripts in Docker Compose projects with installed scripts that are shared between the Docker and Linux service. This normalizes the operational configuration of both Docker shares and Linux service, i.e., to use the same env vars.
+
+CHANGE: Upgrade to Docusaurus v3 for documentation.
+
+FIX: Some Docker shares had broken env mountpoints
+
+## v0.4.16
+
+FEATURE: Publish Linux packages for `zrok` CLI and a systemd service for running a reserved public share (`zrok-share`).
+
+## v0.4.15
+
+CHANGE: Updated the code signing and notarization process for macos binaries. The previous release process used the `gon` utility to handle both code signing and notarization. Apple changed the requirements and the `gon` utility no longer properly functions as of 2023-11-01. The `goreleaser` process has been adjusted to use the `notarytool` utility that ships with XCode to sign and notarize the binary (https://github.com/openziti/zrok/issues/435)
+
+## v0.4.14
+
+FEATURE: `zrok` Drives "Phase 1" (`p1`) functionality included in this release. This includes new `--backend-mode drive`, which accepts a folder path as a target. A `drive` share can be mounted as a network drive on Windows, macOS, and Linux, allowing full read/write access from all applications on those systems (https://github.com/openziti/zrok/issues/218) Subsequent releases will address CLI use cases and provide further refinements to the overall approach.
+
+FEATURE: Docker Compose project for a reserved public share in docker/compose/zrok-public-reserved/compose.yml is described in the [public share guide](https://docs.zrok.io/docs/guides/docker-share/docker_public_share_guide/).
+
+## v0.4.13
+
+FIX: Update to Homebrew automation to properly integrate with the latest version of the Homebrew release process.
+
+## v0.4.12
+
+FIX: The `zrok reserve` command was not properly recording the reserved share status of the shares that it created, preventing the `zrok release` command from properly releasing them (https://github.com/openziti/zrok/issues/427) If a user encounters reserved shares that cannot be released with the `zrok release` command, they can be deleted through the web console.
+
+## v0.4.11
+
+FEATURE: The `zrok reserve` command now incorporates the `--json-output|-j` flag, which outputs the reservation details as JSON, rather than as human-consumable log messages. Other commands will produce similar output in the future (https://github.com/openziti/zrok/issues/422)
+
+FIX: Include `--oauth-provider` and associated flags for the `zrok reserve` command, allowing reserved shares to specify OAuth authentication (https://github.com/openziti/zrok/issues/421)
+
+## v0.4.10
+
+CHANGE: The public frontend configuration has been bumped from `v: 2` to `v: 3`. The `redirect_host`, `redirect_port` and `redirect_http_only` parameters have been removed. These three configuration options have been replaced with `bind_address`, `redirect_url` and `cookie_domain`. See the OAuth configuration guide at `docs/guides/self-hosting/oauth/configuring-oauth.md` for more details (https://github.com/openziti/zrok/issues/411)
+
+## v0.4.9
+
+FIX: Remove extraneous share token prepended to OAuth frontend redirect.
+
+## v0.4.8
+
+FEATURE: The `sdk` package now includes a `sdk.Overview` function, which returns a complete description of the account attached to the enabled environment. Useful for inventorying the deployed shares and environments (https://github.com/openziti/zrok/issues/407)
+
+CHANGE: The `zrok access public` frontend configuration format has changed and now requires that the configuration document include a `v: 2` declaration. This frontend configuration format is now versioned and when the code updates the configuration structure, you will receive an error message at startup, provoking you to look into updating your configuration (https://github.com/openziti/zrok/issues/406)
+
+CHANGE: The title color of the header was changed from white to flourescent green, to better match the overall branding
+
+CHANGE: Tweaks to build and release process for logging and deprecations. Pin golang version at 1.21.3+ and node version at 18.x across all platforms
+
+CHANGE: Improvements to email invitation sent in response to `zrok invite` to correct broken links, some minor HTML issues and improve overall deliverability (https://github.com/openziti/zrok/issues/405)
+
+CHANGE: Added warning message after `zrok invite` submit directing the user to check their "spam" folder if they do not receive the invite message.
+
+## v0.4.7
+
+FEATURE: OAuth authentication with the ability to restrict authenticated users to specified domains for `zrok share public`. Supports both Google and GitHub authentication in this version. More authentication providers, and extensibility to come in future `zrok` releases. See the OAuth configuration guide at `docs/guides/self-hosting/oauth/configuring-oauth.md` for details (https://github.com/openziti/zrok/issues/45, https://github.com/openziti/zrok/issues/404)
+
+CHANGE: `--basic-auth` realm now presented as the share token rather than as `zrok` in `publicProxy` frontend implementation
+
+## v0.4.6
+
+FEATURE: New `--backend-mode caddy`, which pre-processes a `Caddyfile` allowing a `bind` statement to work like this: `bind {{ .ZrokBindAddress }}`. Allows development of complicated API gateways and multi-backend shares, while maintaining the simple, ephemeral sharing model provided by `zrok` (https://github.com/openziti/zrok/issues/391)
+
+CHANGE: `--backend-mode web` has been refactored to utilize Caddy as the integrated web server. This provides for a much nicer web-based file browsing experience, while maintaining the existing web server facilities (https://github.com/openziti/zrok/issues/392)
+
+CHANGE: Updated the golang version for release builds to `1.21.0` and the node version to `18.x`
+
+CHANGE: Added `FrontendEndponts` to `sdk.Share`, returning selected frontend URLs to callers of `sdk.CreateShare`
+
+CHANGE: Added a short alias `-b` for `--backend-mode` to improve CLI ergonomics (https://github.com/openziti/zrok/issues/397)
+
+## v0.4.5
+
+FEATURE: New health check endpoint (`/health`), which verifies that the underlying SQL store and metrics repository (InfluxDB, if configured) are operating correctly (https://github.com/openziti/zrok/issues/372)
+
+CHANGE: Updated to golang v1.21.0 and node v18.x
+
+FIX: `zrok admin bootstrap` and `zrok enable` both broken with latest OpenZiti releases (tested with `v0.30.0`); updated to latest OpenZiti golang SDK (https://github.com/openziti/zrok/issues/389)
+
+## v0.4.4
+
+FIX: `zrok status`, `zrok enable`, `zrok config`, etc. were all causing a panic when used on systems that had no previous `~/.zrok` directory (https://github.com/openziti/zrok/issues/383)
+
+## v0.4.3
+
+FEATURE: New `zrok overview` command, which returns all of the account details as a single JSON structure. See the OpenAPI spec at `specs/zrok.yml` for more details of the `/api/v1/overview` endpoint (https://github.com/openziti/zrok/issues/374)
+
+FEATURE: New `zrok` SDK (https://github.com/openziti/zrok/issues/34). `pastebin` example illustrates basic SDK usage (see `sdk/examples/pastebin/README.md` for details) ((https://github.com/openziti/zrok/issues/379)
+
+## v0.4.2
+
+Some days are just like this. `v0.4.2` is a re-do of `v0.4.1`. Trying to get Homebrew working and had a bad release. Hopefully this is the one.
+
+## v0.4.1
+
+FEATURE: New `zrok console` command to open the currently configured web console in the local web browser (https://github.com/openziti/zrok/issues/170)
+
+CHANGE: Further tweaks to the release process to automatically get the latest release into Homebrew (https://github.com/openziti/zrok/issues/264)
+
+## v0.4.0
+
+FEATURE: New `tcpTunnel` backend mode allowing for private sharing of local TCP sockets with other `zrok` users (https://github.com/openziti/zrok/issues/170)
+
+FEATURE: New `udpTunnel` backend mode allowing for private sharing of local UDP sockets with other `zrok` users (https://github.com/openziti/zrok/issues/306)
+
+FEATURE: New metrics infrastructure based on OpenZiti usage events (https://github.com/openziti/zrok/issues/128). See the [v0.4 Metrics Guide](docs/guides/metrics-and-limits/configuring-metrics.md) for more information.
+
+FEATURE: New limits implementation based on the new metrics infrastructure (https://github.com/openziti/zrok/issues/235). See the [v0.4 Limits Guide](docs/guides/metrics-and-limits/configuring-limits.md) for more information.
+
+FEATURE: The invite mechanism has been reworked to improve user experience. The configuration has been updated to include a new `invite` stanza, and now includes a boolean flag indicating whether or not the instance allows new invitations to be created, and also includes contact details for requesting a new invite. These values are used by the `zrok invite` command to provide a smoother end-user invite experience https://github.com/openziti/zrok/issues/229)
+
+FEATURE: New password strength checking rules and configuration. See the example configuration file (`etc/ctrl.yml`) for details about how to configure the strength checking rules (https://github.com/openziti/zrok/issues/167)
+
+FEATURE: A new `admin/profile_endpoint` configuration option is available to start a `net/http/pprof` listener. See `etc/ctrl.yml` for details.
+
+CHANGE: The controller configuration version bumps from `v: 2` to `v: 3` to support all of the new `v0.4` functionality. See the [example ctrl.yml](etc/ctrl.yml) for details on the new configuration.
+
+CHANGE: The underlying database store now utilizes a `deleted` flag on all tables to implement "soft deletes". This was necessary for the new metrics infrastructure, where we need to account for metrics data that arrived after the lifetime of a share or environment; and also we're going to need this for limits, where we need to see historical information about activity in the past (https://github.com/openziti/zrok/issues/262)
+
+CHANGE: Updated to latest `github.com/openziti/sdk-golang` (https://github.com/openziti/zrok/issues/335)
+
+FIX: `zrok share reserved --override-endpoint` now works correctly; `--override-endpoint` was being incorrectly ignore previously (https://github.com/openziti/zrok/pull/348)
+
+## v0.3.7
+
+FIX: Improved TUI word-wrapping (https://github.com/openziti/zrok/issues/180)
+
+## v0.3.6
+
+CHANGE: Additional change to support branch builds (for CI purposes) and additional containerization efforts around k8s.
+
+## v0.3.5
+
+CHANGE: `zrok config set apiEndpoint` now validates that the new API endpoint correctly starts with `http://` or `https://` (https://github.com/openziti/zrok/issues/258)
+
+CHANGE: Additional linting to support homebrew (https://github.com/openziti/zrok/issues/264)
+
+## v0.3.4
+
+CHANGE: `zrok test endpoint` incorporates `--ziti` mode (and related flags) to allow direct endpoint listening on a Ziti service
+
+CHANGE: `zrok test websocket` command to test websockets, whether over TCP or over Ziti
+
+FIX: Websocket support now functional
+
+## v0.3.3
+
+CHANGE: `zrok test loop` has been moved to `zrok test loop public`, making way for additional types of loopback testing tools. The `zrok test endpoint` server now includes an `/echo` endpoint, which provides a simple echo websocket (https://github.com/openziti/zrok/issues/237)
+
+## v0.3.2
+
+FEATURE: New docker infrastructure, including `docker-compose.yml` examples (and documentation) illustrating how to deploy `zrok` in `docker`-based environments
+
+CHANGE: Include missing `--headless` flag for `zrok enable` and `zrok access private` (https://github.com/openziti/zrok/issues/246)
+
+CHANGE: Fix for `zrok enable` error path handling (https://github.com/openziti/zrok/issues/244)
+
+FEATURE: `zrok controller validate` and `zrok access public validate` will both perform a quick syntax validation on controller and public frontend configuration documents (https://github.com/openziti/zrok/issues/238)
+
+	$ zrok controller validate etc/dev.yml 
+	[ERROR]: controller config validation failed (error loading controller config 'etc/dev.yml': field 'maintenance': field 'registration': field 'expiration_timeout': got [bool], expected [time.Duration])
+
+CHANGE: `zrok status` no longer shows secrets (secret token, ziti identity) unless the `--secrets` flag is passed (https://github.com/openziti/zrok/issues/243)
+
+## v0.3.1
+
+CHANGE: Incorporate initial docker image build (https://github.com/openziti/zrok/issues/217)
+
+CHANGE: Improve target URL parsing for `zrok share` when using `--backend-mode` proxy (https://github.com/openziti/zrok/issues/211)
+
+	New and improved URL handling for proxy backends:
+
+	9090 -> http://127.0.0.1:9090
+	localhost:9090 -> http://127.0.0.1:9090
+	https://localhost:9090 -> https://localhost:9090
+
+CHANGE: Improve usability of `zrok invite` TUI in low-color environments (https://github.com/openziti/zrok/issues/206)
+
+CHANGE: Better error responses when `zrok invite` fails due to missing token (https://github.com/openziti/zrok/issues/207)
+
+## v0.3.0
+
+CHANGE: Removed some minor web console lint and warnings (https://github.com/openziti/zrok/issues/205)
+
+## v0.3.0-rc6
+
+CHANGE: Better error message when `zrok admin create frontend` runs into a duplicate name collision (https://github.com/openziti/zrok/issues/168)
+
+CHANGE: Gentler CLI error messages by default (https://github.com/openziti/zrok/issues/203)
+
+CHANGE: Add favicon to web console (https://github.com/openziti/zrok/issues/198)
+
+CHANGE: Add configurable "terms of use" link in the controller configuration, and optionally display the link on the login form and registration forms (https://github.com/openziti/zrok/issues/184)
+
+CHANGE: Prevent multiple `zrok enable` commands from succeeding (https://github.com/openziti/zrok/issues/190)
+
+CHANGE: New `--insecure` flag for `share <public|private|reserved>` commands (https://github.com/openziti/zrok/issues/195)
+
+## v0.3.0-rc5
+
+CHANGE: Improvements to controller log messages to assist in operations (https://github.com/openziti/zrok/issues/186)
+
+CHANGE: `armv7` builds for Linux are now shipped with releases; these builds were tested against a Raspberry Pi 4 (https://github.com/openziti/zrok/issues/93)
+
+CHANGE: `zrok config set` now includes a warning when the `apiEndpoint` config is changed and an environment is already enabled; the user will not see the change until `zrok disable` is run. The CLI now includes a `zrok config unset` command (https://github.com/openziti/zrok/issues/188)
+
+## v0.3.0-rc4
+
+CHANGE: Enable notarization for macos binaries (https://github.com/openziti/zrok/issues/92)
+
+## v0.3.0-rc3
+
+> This release increments the configuration version from `1` to `2`. See the note below.
+
+CHANGE: The email "from" configuration moved from `registration/email_from` to `email/from`. **NOTE: This change increments the configuration `V` from `1` to `2`.**
+
+CHANGE: Replaced un-salted sha512 password hashing with salted hashing based on Argon2 **NOTE: This version will _invalidate_ all account passwords, and will require all users to use the 'Forgot Password?' function to reset their password.** (https://github.com/openziti/zrok/issues/156)
+
+CHANGE: Switched from `ubuntu-latest` (`22.04`) for the Linux builds to `ubuntu-20.04`. Should improve `glibc` compatibility with older Linux distributions (https://github.com/openziti/zrok/issues/179)
+
+CHANGE: `zrok admin generate` now outputs the generated tokens to `stdout` after successfully provisioning the tokens (https://github.com/openziti/zrok/issues/181)
+
+FIX: Fixed log message in `resetPasswordRequest.go` (https://github.com/openziti/zrok/issues/175)
+
+FIX: Fixed `-v` (verbose mode) on in TUI-based `zrok share` and `zrok access` (https://github.com/openziti/zrok/issues/174)
+
+## v0.3.0-rc2
+
+FEATURE: Allow users to reset their password (https://github.com/openziti/zrok/issues/65)
+
+CHANGE: Improved email styling for new user invite emails (https://github.com/openziti/zrok/issues/157)
+
+CHANGE: Migrated from `openziti-test-kitchen` to `openziti` (https://github.com/openziti/zrok/issues/158).
+
+CHANGE: Show a hint when `zrok invite` fails, indicating that the user should check to see if they need to be using the `--token` flag and token-based invites (https://github.com/openziti/zrok/issues/172).
+
+FIX: Fixed PostgreSQL migration issue where sequences got reset and resulted in primary key collisions on a couple of tables (https://github.com/openziti/zrok/issues/160).
+
+FIX: Remove `frontend` instances when `zrok disable`-ing an environment containing them (https://github.com/openziti/zrok/issues/171)
+
+## v0.3.x Series
+
+The `v0.2` series was a _proof-of-concept_ implementation for the overall `zrok` architecture and the concept.
+
+`v0.3` is a massive elaboration of the concept, pivoting it from being a simple ephemeral reverse proxy solution, to being the beginnings of a comprehensive sharing platform, complete with public and private sharing (built on top of OpenZiti).
+
+`v0.3.0` includes the minimal functionality required to produce an early, preview version of the elaborated `zrok` concept, suitable for both production use at `zrok.io`, and also suitable for private self-hosting.
+
+From `v0.3.0` forward, we will begin tracking notable changes in this document.
+
+## v0.2.18
+
+* DEFECT: Token generation has been improved to use an alphabet consisting of `[a-zA-Z0-9]`. Service token generation continues to use a case-insensitive alphabet consisting of `[a-z0-9]` to be DNS-safe.

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2019 NetFoundry, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -0,0 +1,108 @@
+# zrok - Secure Internet Sharing Made Simple
+
+![zrok logo](docs/images/zrok_cover.png)
+
+**Share anything, anywhere, instantly. Enterprise reliability. No firewall changes. No port forwarding. No hassle.**
+
+`zrok` lets you securely share web services, files, and network resources with anyone—whether they're across the internet or your private network. Built on zero-trust networking, it works through firewalls and NAT without requiring any network configuration changes.
+
+## Quick Start
+
+Get sharing in under 2 minutes:
+
+1. **[Install zrok](https://docs.zrok.io/docs/guides/install/)** for your platform
+2. **Get an account**: `zrok invite` (use the free [zrok.io service](https://docs.zrok.io/docs/getting-started/))
+3. **Enable sharing**: `zrok enable`
+
+That's it! Now you can share anything:
+
+```bash
+# Share a web service publicly
+$ zrok share public localhost:8080
+
+# Share files as a network drive  
+$ zrok share public --backend-mode drive ~/Documents
+
+# Share privately with other zrok users
+$ zrok share private localhost:3000
+```
+
+![zrok Web Console](docs/images/zrok_web_console.png)
+
+## What You Can Share
+
+### Web Services
+Instantly make local web apps accessible over the internet:
+
+```bash
+$ zrok share public localhost:8080
+```
+![zrok share public](docs/images/zrok_share_public.png)
+
+### Files & Directories  
+Turn any folder into a shareable network drive:
+
+```bash
+$ zrok share public --backend-mode drive ~/Repos/zrok
+```
+![zrok share public -b drive](docs/images/zrok_share_public_drive.png)
+![mounted zrok drive](docs/images/zrok_share_public_drive_explorer.png)
+
+### Private Resources
+Share TCP/UDP services securely with other zrok users—no public internet exposure.
+
+## Key Features
+
+- **Zero Configuration**: Works through firewalls, NAT, and corporate networks
+- **Secure by Default**: End-to-end encryption with zero-trust architecture  
+- **Public & Private Sharing**: Share with anyone or just specific users
+- **Multiple Protocols**: HTTP/HTTPS, TCP, UDP, and file sharing
+- **Cross-Platform**: Windows, macOS, Linux, and Raspberry Pi
+- **Self-Hostable**: Run your own zrok service instance
+
+## How It Works
+
+`zrok` is built on [OpenZiti](https://docs.openziti.io/docs/learn/introduction/), a programmable zero-trust network overlay. This means:
+
+- **No inbound connectivity required**: Works from behind firewalls and NAT
+- **End-to-end encryption**: All traffic is encrypted, even from zrok servers
+- **Peer-to-peer connections**: Direct connections between users when possible
+- **Identity-based access**: Share with specific users, not IP addresses
+
+## Developer SDK
+
+Embed `zrok` sharing into your applications with our Go SDK:
+
+```go
+// Create a share
+shr, err := sdk.CreateShare(root, &sdk.ShareRequest{
+    BackendMode: sdk.TcpTunnelBackendMode,
+    ShareMode:   sdk.PrivateShareMode,
+})
+
+// Accept connections
+listener, err := sdk.NewListener(shr.Token, root)
+```
+
+[Read the SDK guide](https://blog.openziti.io/the-zrok-sdk) for complete examples.
+
+## Self-Hosting
+
+Run your own `zrok` service—from Raspberry Pi to enterprise scale:
+
+- Single binary contains everything you need
+- Scales from small personal instances to large public services  
+- Built on the same codebase as the public `zrok.io` service
+
+[Self-Hosting Guide](https://docs.zrok.io/docs/guides/self-hosting/self_hosting_guide/)
+
+## Resources
+
+- **[Documentation](https://docs.zrok.io/)**
+- **[Office Hours Videos](https://www.youtube.com/watch?v=Edqv7yRmXb0&list=PLMUj_5fklasLuM6XiCNqwAFBuZD1t2lO2)**
+- **[Building from Source](./BUILD.md)**
+- **[Contributing](./CONTRIBUTING.md)**
+
+---
+
+*Ready to start sharing? [Get started with zrok →](https://docs.zrok.io/docs/getting-started)*

airflow/airflow-webserver.pid

@@ -0,0 +1 @@
+257

airflow/airflow.cfg

@@ -0,0 +1,2498 @@
+[core]
+# The folder where your airflow pipelines live, most likely a
+# subfolder in a code repository. This path must be absolute.
+#
+# Variable: AIRFLOW__CORE__DAGS_FOLDER
+#
+dags_folder = /kaggle/working/BTC-USDT-ETL-Pipeline/airflow/dags
+
+# Hostname by providing a path to a callable, which will resolve the hostname.
+# The format is "package.function".
+# 
+# For example, default value ``airflow.utils.net.getfqdn`` means that result from patched
+# version of `socket.getfqdn() <https://docs.python.org/3/library/socket.html#socket.getfqdn>`__,
+# see related `CPython Issue <https://github.com/python/cpython/issues/49254>`__.
+# 
+# No argument should be required in the function specified.
+# If using IP address as hostname is preferred, use value ``airflow.utils.net.get_host_ip_address``
+#
+# Variable: AIRFLOW__CORE__HOSTNAME_CALLABLE
+#
+hostname_callable = airflow.utils.net.getfqdn
+
+# A callable to check if a python file has airflow dags defined or not and should
+# return ``True`` if it has dags otherwise ``False``.
+# If this is not provided, Airflow uses its own heuristic rules.
+# 
+# The function should have the following signature
+# 
+# .. code-block:: python
+# 
+#     def func_name(file_path: str, zip_file: zipfile.ZipFile | None = None) -> bool: ...
+#
+# Variable: AIRFLOW__CORE__MIGHT_CONTAIN_DAG_CALLABLE
+#
+might_contain_dag_callable = airflow.utils.file.might_contain_dag_via_default_heuristic
+
+# Default timezone in case supplied date times are naive
+# can be `UTC` (default), `system`, or any `IANA <https://www.iana.org/time-zones>`
+# timezone string (e.g. Europe/Amsterdam)
+#
+# Variable: AIRFLOW__CORE__DEFAULT_TIMEZONE
+#
+default_timezone = utc
+
+# The executor class that airflow should use. Choices include
+# ``SequentialExecutor``, ``LocalExecutor``, ``CeleryExecutor``,
+# ``KubernetesExecutor``, ``CeleryKubernetesExecutor``, ``LocalKubernetesExecutor`` or the
+# full import path to the class when using a custom executor.
+#
+# Variable: AIRFLOW__CORE__EXECUTOR
+#
+executor = SequentialExecutor
+
+# The auth manager class that airflow should use. Full import path to the auth manager class.
+#
+# Variable: AIRFLOW__CORE__AUTH_MANAGER
+#
+auth_manager = airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
+
+# This defines the maximum number of task instances that can run concurrently per scheduler in
+# Airflow, regardless of the worker count. Generally this value, multiplied by the number of
+# schedulers in your cluster, is the maximum number of task instances with the running
+# state in the metadata database. Setting this value to zero allows unlimited parallelism.
+#
+# Variable: AIRFLOW__CORE__PARALLELISM
+#
+parallelism = 32
+
+# The maximum number of task instances allowed to run concurrently in each DAG. To calculate
+# the number of tasks that is running concurrently for a DAG, add up the number of running
+# tasks for all DAG runs of the DAG. This is configurable at the DAG level with ``max_active_tasks``,
+# which is defaulted as ``[core] max_active_tasks_per_dag``.
+# 
+# An example scenario when this would be useful is when you want to stop a new dag with an early
+# start date from stealing all the executor slots in a cluster.
+#
+# Variable: AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG
+#
+max_active_tasks_per_dag = 16
+
+# Are DAGs paused by default at creation
+#
+# Variable: AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION
+#
+dags_are_paused_at_creation = True
+
+# The maximum number of active DAG runs per DAG. The scheduler will not create more DAG runs
+# if it reaches the limit. This is configurable at the DAG level with ``max_active_runs``,
+# which is defaulted as ``[core] max_active_runs_per_dag``.
+#
+# Variable: AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG
+#
+max_active_runs_per_dag = 16
+
+# (experimental) The maximum number of consecutive DAG failures before DAG is automatically paused.
+# This is also configurable per DAG level with ``max_consecutive_failed_dag_runs``,
+# which is defaulted as ``[core] max_consecutive_failed_dag_runs_per_dag``.
+# If not specified, then the value is considered as 0,
+# meaning that the dags are never paused out by default.
+#
+# Variable: AIRFLOW__CORE__MAX_CONSECUTIVE_FAILED_DAG_RUNS_PER_DAG
+#
+max_consecutive_failed_dag_runs_per_dag = 0
+
+# The name of the method used in order to start Python processes via the multiprocessing module.
+# This corresponds directly with the options available in the Python docs:
+# `multiprocessing.set_start_method
+# <https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method>`__
+# must be one of the values returned by `multiprocessing.get_all_start_methods()
+# <https://docs.python.org/3/library/multiprocessing.html#multiprocessing.get_all_start_methods>`__.
+#
+# Example: mp_start_method = fork
+#
+# Variable: AIRFLOW__CORE__MP_START_METHOD
+#
+# mp_start_method = 
+
+# Whether to load the DAG examples that ship with Airflow. It's good to
+# get started, but you probably want to set this to ``False`` in a production
+# environment
+#
+# Variable: AIRFLOW__CORE__LOAD_EXAMPLES
+#
+load_examples = True
+
+# Path to the folder containing Airflow plugins
+#
+# Variable: AIRFLOW__CORE__PLUGINS_FOLDER
+#
+plugins_folder = /kaggle/working/BTC-USDT-ETL-Pipeline/airflow/plugins
+
+# Should tasks be executed via forking of the parent process
+# 
+# * ``False``: Execute via forking of the parent process
+# * ``True``: Spawning a new python process, slower than fork, but means plugin changes picked
+#   up by tasks straight away
+#
+# Variable: AIRFLOW__CORE__EXECUTE_TASKS_NEW_PYTHON_INTERPRETER
+#
+execute_tasks_new_python_interpreter = False
+
+# Secret key to save connection passwords in the db
+#
+# Variable: AIRFLOW__CORE__FERNET_KEY
+#
+fernet_key = 
+
+# Whether to disable pickling dags
+#
+# Variable: AIRFLOW__CORE__DONOT_PICKLE
+#
+donot_pickle = True
+
+# How long before timing out a python file import
+#
+# Variable: AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT
+#
+dagbag_import_timeout = 30.0
+
+# Should a traceback be shown in the UI for dagbag import errors,
+# instead of just the exception message
+#
+# Variable: AIRFLOW__CORE__DAGBAG_IMPORT_ERROR_TRACEBACKS
+#
+dagbag_import_error_tracebacks = True
+
+# If tracebacks are shown, how many entries from the traceback should be shown
+#
+# Variable: AIRFLOW__CORE__DAGBAG_IMPORT_ERROR_TRACEBACK_DEPTH
+#
+dagbag_import_error_traceback_depth = 2
+
+# How long before timing out a DagFileProcessor, which processes a dag file
+#
+# Variable: AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT
+#
+dag_file_processor_timeout = 50
+
+# The class to use for running task instances in a subprocess.
+# Choices include StandardTaskRunner, CgroupTaskRunner or the full import path to the class
+# when using a custom task runner.
+#
+# Variable: AIRFLOW__CORE__TASK_RUNNER
+#
+task_runner = StandardTaskRunner
+
+# If set, tasks without a ``run_as_user`` argument will be run with this user
+# Can be used to de-elevate a sudo user running Airflow when executing tasks
+#
+# Variable: AIRFLOW__CORE__DEFAULT_IMPERSONATION
+#
+default_impersonation = 
+
+# What security module to use (for example kerberos)
+#
+# Variable: AIRFLOW__CORE__SECURITY
+#
+security = 
+
+# Turn unit test mode on (overwrites many configuration options with test
+# values at runtime)
+#
+# Variable: AIRFLOW__CORE__UNIT_TEST_MODE
+#
+unit_test_mode = False
+
+# Whether to enable pickling for xcom (note that this is insecure and allows for
+# RCE exploits).
+#
+# Variable: AIRFLOW__CORE__ENABLE_XCOM_PICKLING
+#
+enable_xcom_pickling = False
+
+# What classes can be imported during deserialization. This is a multi line value.
+# The individual items will be parsed as a pattern to a glob function.
+# Python built-in classes (like dict) are always allowed.
+#
+# Variable: AIRFLOW__CORE__ALLOWED_DESERIALIZATION_CLASSES
+#
+allowed_deserialization_classes = airflow.*
+
+# What classes can be imported during deserialization. This is a multi line value.
+# The individual items will be parsed as regexp patterns.
+# This is a secondary option to ``[core] allowed_deserialization_classes``.
+#
+# Variable: AIRFLOW__CORE__ALLOWED_DESERIALIZATION_CLASSES_REGEXP
+#
+allowed_deserialization_classes_regexp = 
+
+# When a task is killed forcefully, this is the amount of time in seconds that
+# it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED
+#
+# Variable: AIRFLOW__CORE__KILLED_TASK_CLEANUP_TIME
+#
+killed_task_cleanup_time = 60
+
+# Whether to override params with dag_run.conf. If you pass some key-value pairs
+# through ``airflow dags backfill -c`` or
+# ``airflow dags trigger -c``, the key-value pairs will override the existing ones in params.
+#
+# Variable: AIRFLOW__CORE__DAG_RUN_CONF_OVERRIDES_PARAMS
+#
+dag_run_conf_overrides_params = True
+
+# If enabled, Airflow will only scan files containing both ``DAG`` and ``airflow`` (case-insensitive).
+#
+# Variable: AIRFLOW__CORE__DAG_DISCOVERY_SAFE_MODE
+#
+dag_discovery_safe_mode = True
+
+# The pattern syntax used in the
+# `.airflowignore
+# <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#airflowignore>`__
+# files in the DAG directories. Valid values are ``regexp`` or ``glob``.
+#
+# Variable: AIRFLOW__CORE__DAG_IGNORE_FILE_SYNTAX
+#
+dag_ignore_file_syntax = regexp
+
+# The number of retries each task is going to have by default. Can be overridden at dag or task level.
+#
+# Variable: AIRFLOW__CORE__DEFAULT_TASK_RETRIES
+#
+default_task_retries = 0
+
+# The number of seconds each task is going to wait by default between retries. Can be overridden at
+# dag or task level.
+#
+# Variable: AIRFLOW__CORE__DEFAULT_TASK_RETRY_DELAY
+#
+default_task_retry_delay = 300
+
+# The maximum delay (in seconds) each task is going to wait by default between retries.
+# This is a global setting and cannot be overridden at task or DAG level.
+#
+# Variable: AIRFLOW__CORE__MAX_TASK_RETRY_DELAY
+#
+max_task_retry_delay = 86400
+
+# The weighting method used for the effective total priority weight of the task
+#
+# Variable: AIRFLOW__CORE__DEFAULT_TASK_WEIGHT_RULE
+#
+default_task_weight_rule = downstream
+
+# Maximum possible time (in seconds) that task will have for execution of auxiliary processes
+# (like listeners, mini scheduler...) after task is marked as success..
+#
+# Variable: AIRFLOW__CORE__TASK_SUCCESS_OVERTIME
+#
+task_success_overtime = 20
+
+# The default task execution_timeout value for the operators. Expected an integer value to
+# be passed into timedelta as seconds. If not specified, then the value is considered as None,
+# meaning that the operators are never timed out by default.
+#
+# Variable: AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT
+#
+default_task_execution_timeout = 
+
+# Updating serialized DAG can not be faster than a minimum interval to reduce database write rate.
+#
+# Variable: AIRFLOW__CORE__MIN_SERIALIZED_DAG_UPDATE_INTERVAL
+#
+min_serialized_dag_update_interval = 30
+
+# If ``True``, serialized DAGs are compressed before writing to DB.
+# 
+# .. note::
+# 
+#     This will disable the DAG dependencies view
+#
+# Variable: AIRFLOW__CORE__COMPRESS_SERIALIZED_DAGS
+#
+compress_serialized_dags = False
+
+# Fetching serialized DAG can not be faster than a minimum interval to reduce database
+# read rate. This config controls when your DAGs are updated in the Webserver
+#
+# Variable: AIRFLOW__CORE__MIN_SERIALIZED_DAG_FETCH_INTERVAL
+#
+min_serialized_dag_fetch_interval = 10
+
+# Maximum number of Rendered Task Instance Fields (Template Fields) per task to store
+# in the Database.
+# All the template_fields for each of Task Instance are stored in the Database.
+# Keeping this number small may cause an error when you try to view ``Rendered`` tab in
+# TaskInstance view for older tasks.
+#
+# Variable: AIRFLOW__CORE__MAX_NUM_RENDERED_TI_FIELDS_PER_TASK
+#
+max_num_rendered_ti_fields_per_task = 30
+
+# On each dagrun check against defined SLAs
+#
+# Variable: AIRFLOW__CORE__CHECK_SLAS
+#
+check_slas = True
+
+# Path to custom XCom class that will be used to store and resolve operators results
+#
+# Example: xcom_backend = path.to.CustomXCom
+#
+# Variable: AIRFLOW__CORE__XCOM_BACKEND
+#
+xcom_backend = airflow.models.xcom.BaseXCom
+
+# By default Airflow plugins are lazily-loaded (only loaded when required). Set it to ``False``,
+# if you want to load plugins whenever 'airflow' is invoked via cli or loaded from module.
+#
+# Variable: AIRFLOW__CORE__LAZY_LOAD_PLUGINS
+#
+lazy_load_plugins = True
+
+# By default Airflow providers are lazily-discovered (discovery and imports happen only when required).
+# Set it to ``False``, if you want to discover providers whenever 'airflow' is invoked via cli or
+# loaded from module.
+#
+# Variable: AIRFLOW__CORE__LAZY_DISCOVER_PROVIDERS
+#
+lazy_discover_providers = True
+
+# Hide sensitive **Variables** or **Connection extra json keys** from UI
+# and task logs when set to ``True``
+# 
+# .. note::
+# 
+#     Connection passwords are always hidden in logs
+#
+# Variable: AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS
+#
+hide_sensitive_var_conn_fields = True
+
+# A comma-separated list of extra sensitive keywords to look for in variables names or connection's
+# extra JSON.
+#
+# Variable: AIRFLOW__CORE__SENSITIVE_VAR_CONN_NAMES
+#
+sensitive_var_conn_names = 
+
+# Task Slot counts for ``default_pool``. This setting would not have any effect in an existing
+# deployment where the ``default_pool`` is already created. For existing deployments, users can
+# change the number of slots using Webserver, API or the CLI
+#
+# Variable: AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT
+#
+default_pool_task_slot_count = 128
+
+# The maximum list/dict length an XCom can push to trigger task mapping. If the pushed list/dict has a
+# length exceeding this value, the task pushing the XCom will be failed automatically to prevent the
+# mapped tasks from clogging the scheduler.
+#
+# Variable: AIRFLOW__CORE__MAX_MAP_LENGTH
+#
+max_map_length = 1024
+
+# The default umask to use for process when run in daemon mode (scheduler, worker,  etc.)
+# 
+# This controls the file-creation mode mask which determines the initial value of file permission bits
+# for newly created files.
+# 
+# This value is treated as an octal-integer.
+#
+# Variable: AIRFLOW__CORE__DAEMON_UMASK
+#
+daemon_umask = 0o077
+
+# Class to use as dataset manager.
+#
+# Example: dataset_manager_class = airflow.datasets.manager.DatasetManager
+#
+# Variable: AIRFLOW__CORE__DATASET_MANAGER_CLASS
+#
+# dataset_manager_class = 
+
+# Kwargs to supply to dataset manager.
+#
+# Example: dataset_manager_kwargs = {"some_param": "some_value"}
+#
+# Variable: AIRFLOW__CORE__DATASET_MANAGER_KWARGS
+#
+# dataset_manager_kwargs = 
+
+# Dataset URI validation should raise an exception if it is not compliant with AIP-60.
+# By default this configuration is false, meaning that Airflow 2.x only warns the user.
+# In Airflow 3, this configuration will be enabled by default.
+#
+# Variable: AIRFLOW__CORE__STRICT_DATASET_URI_VALIDATION
+#
+strict_dataset_uri_validation = False
+
+# (experimental) Whether components should use Airflow Internal API for DB connectivity.
+#
+# Variable: AIRFLOW__CORE__DATABASE_ACCESS_ISOLATION
+#
+database_access_isolation = False
+
+# (experimental) Airflow Internal API url.
+# Only used if ``[core] database_access_isolation`` is ``True``.
+#
+# Example: internal_api_url = http://localhost:8080
+#
+# Variable: AIRFLOW__CORE__INTERNAL_API_URL
+#
+# internal_api_url = 
+
+# Secret key used to authenticate internal API clients to core. It should be as random as possible.
+# However, when running more than 1 instances of webserver / internal API services, make sure all
+# of them use the same ``secret_key`` otherwise calls will fail on authentication.
+# The authentication token generated using the secret key has a short expiry time though - make
+# sure that time on ALL the machines that you run airflow components on is synchronized
+# (for example using ntpd) otherwise you might get "forbidden" errors when the logs are accessed.
+#
+# Variable: AIRFLOW__CORE__INTERNAL_API_SECRET_KEY
+#
+internal_api_secret_key = JRNP2IC4kIaVxisy9+AW4A==
+
+# The ability to allow testing connections across Airflow UI, API and CLI.
+# Supported options: ``Disabled``, ``Enabled``, ``Hidden``. Default: Disabled
+# Disabled - Disables the test connection functionality and disables the Test Connection button in UI.
+# Enabled - Enables the test connection functionality and shows the Test Connection button in UI.
+# Hidden - Disables the test connection functionality and hides the Test Connection button in UI.
+# Before setting this to Enabled, make sure that you review the users who are able to add/edit
+# connections and ensure they are trusted. Connection testing can be done maliciously leading to
+# undesired and insecure outcomes.
+# See `Airflow Security Model: Capabilities of authenticated UI users
+# <https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html#capabilities-of-authenticated-ui-users>`__
+# for more details.
+#
+# Variable: AIRFLOW__CORE__TEST_CONNECTION
+#
+test_connection = Disabled
+
+# The maximum length of the rendered template field. If the value to be stored in the
+# rendered template field exceeds this size, it's redacted.
+#
+# Variable: AIRFLOW__CORE__MAX_TEMPLATED_FIELD_LENGTH
+#
+max_templated_field_length = 4096
+
+[database]
+# Path to the ``alembic.ini`` file. You can either provide the file path relative
+# to the Airflow home directory or the absolute path if it is located elsewhere.
+#
+# Variable: AIRFLOW__DATABASE__ALEMBIC_INI_FILE_PATH
+#
+alembic_ini_file_path = alembic.ini
+
+# The SQLAlchemy connection string to the metadata database.
+# SQLAlchemy supports many different database engines.
+# See: `Set up a Database Backend: Database URI
+# <https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#database-uri>`__
+# for more details.
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
+#
+sql_alchemy_conn = sqlite:////kaggle/working/BTC-USDT-ETL-Pipeline/airflow/airflow.db
+
+# Extra engine specific keyword args passed to SQLAlchemy's create_engine, as a JSON-encoded value
+#
+# Example: sql_alchemy_engine_args = {"arg1": true}
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_ARGS
+#
+# sql_alchemy_engine_args = 
+
+# The encoding for the databases
+#
+# Variable: AIRFLOW__DATABASE__SQL_ENGINE_ENCODING
+#
+sql_engine_encoding = utf-8
+
+# Collation for ``dag_id``, ``task_id``, ``key``, ``external_executor_id`` columns
+# in case they have different encoding.
+# By default this collation is the same as the database collation, however for ``mysql`` and ``mariadb``
+# the default is ``utf8mb3_bin`` so that the index sizes of our index keys will not exceed
+# the maximum size of allowed index when collation is set to ``utf8mb4`` variant, see
+# `GitHub Issue Comment <https://github.com/apache/airflow/pull/17603#issuecomment-901121618>`__
+# for more details.
+#
+# Variable: AIRFLOW__DATABASE__SQL_ENGINE_COLLATION_FOR_IDS
+#
+# sql_engine_collation_for_ids = 
+
+# If SQLAlchemy should pool database connections.
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED
+#
+sql_alchemy_pool_enabled = True
+
+# The SQLAlchemy pool size is the maximum number of database connections
+# in the pool. 0 indicates no limit.
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_SIZE
+#
+sql_alchemy_pool_size = 5
+
+# The maximum overflow size of the pool.
+# When the number of checked-out connections reaches the size set in pool_size,
+# additional connections will be returned up to this limit.
+# When those additional connections are returned to the pool, they are disconnected and discarded.
+# It follows then that the total number of simultaneous connections the pool will allow
+# is **pool_size** + **max_overflow**,
+# and the total number of "sleeping" connections the pool will allow is pool_size.
+# max_overflow can be set to ``-1`` to indicate no overflow limit;
+# no limit will be placed on the total number of concurrent connections. Defaults to ``10``.
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_MAX_OVERFLOW
+#
+sql_alchemy_max_overflow = 10
+
+# The SQLAlchemy pool recycle is the number of seconds a connection
+# can be idle in the pool before it is invalidated. This config does
+# not apply to sqlite. If the number of DB connections is ever exceeded,
+# a lower config value will allow the system to recover faster.
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE
+#
+sql_alchemy_pool_recycle = 1800
+
+# Check connection at the start of each connection pool checkout.
+# Typically, this is a simple statement like "SELECT 1".
+# See `SQLAlchemy Pooling: Disconnect Handling - Pessimistic
+# <https://docs.sqlalchemy.org/en/14/core/pooling.html#disconnect-handling-pessimistic>`__
+# for more details.
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING
+#
+sql_alchemy_pool_pre_ping = True
+
+# The schema to use for the metadata database.
+# SQLAlchemy supports databases with the concept of multiple schemas.
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_SCHEMA
+#
+sql_alchemy_schema = 
+
+# Import path for connect args in SQLAlchemy. Defaults to an empty dict.
+# This is useful when you want to configure db engine args that SQLAlchemy won't parse
+# in connection string. This can be set by passing a dictionary containing the create engine parameters.
+# For more details about passing create engine parameters (keepalives variables, timeout etc)
+# in Postgres DB Backend see `Setting up a PostgreSQL Database
+# <https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#setting-up-a-postgresql-database>`__
+# e.g ``connect_args={"timeout":30}`` can be defined in ``airflow_local_settings.py`` and
+# can be imported as shown below
+#
+# Example: sql_alchemy_connect_args = airflow_local_settings.connect_args
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_CONNECT_ARGS
+#
+# sql_alchemy_connect_args = 
+
+# Important Warning: Use of sql_alchemy_session_maker Highly Discouraged
+# Import path for function which returns 'sqlalchemy.orm.sessionmaker'.
+# Improper configuration of sql_alchemy_session_maker can lead to serious issues,
+# including data corruption, unrecoverable application crashes. Please review the SQLAlchemy
+# documentation for detailed guidance on proper configuration and best practices.
+#
+# Example: sql_alchemy_session_maker = airflow_local_settings._sessionmaker
+#
+# Variable: AIRFLOW__DATABASE__SQL_ALCHEMY_SESSION_MAKER
+#
+# sql_alchemy_session_maker = 
+
+# Whether to load the default connections that ship with Airflow when ``airflow db init`` is called.
+# It's good to get started, but you probably want to set this to ``False`` in a production environment.
+#
+# Variable: AIRFLOW__DATABASE__LOAD_DEFAULT_CONNECTIONS
+#
+load_default_connections = True
+
+# Number of times the code should be retried in case of DB Operational Errors.
+# Not all transactions will be retried as it can cause undesired state.
+# Currently it is only used in ``DagFileProcessor.process_file`` to retry ``dagbag.sync_to_db``.
+#
+# Variable: AIRFLOW__DATABASE__MAX_DB_RETRIES
+#
+max_db_retries = 3
+
+# Whether to run alembic migrations during Airflow start up. Sometimes this operation can be expensive,
+# and the users can assert the correct version through other means (e.g. through a Helm chart).
+# Accepts ``True`` or ``False``.
+#
+# Variable: AIRFLOW__DATABASE__CHECK_MIGRATIONS
+#
+check_migrations = True
+
+[logging]
+# The folder where airflow should store its log files.
+# This path must be absolute.
+# There are a few existing configurations that assume this is set to the default.
+# If you choose to override this you may need to update the
+# ``[logging] dag_processor_manager_log_location`` and
+# ``[logging] child_process_log_directory settings`` as well.
+#
+# Variable: AIRFLOW__LOGGING__BASE_LOG_FOLDER
+#
+base_log_folder = /kaggle/working/BTC-USDT-ETL-Pipeline/airflow/logs
+
+# Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic Search.
+# Set this to ``True`` if you want to enable remote logging.
+#
+# Variable: AIRFLOW__LOGGING__REMOTE_LOGGING
+#
+remote_logging = False
+
+# Users must supply an Airflow connection id that provides access to the storage
+# location. Depending on your remote logging service, this may only be used for
+# reading logs, not writing them.
+#
+# Variable: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
+#
+remote_log_conn_id = 
+
+# Whether the local log files for GCS, S3, WASB and OSS remote logging should be deleted after
+# they are uploaded to the remote location.
+#
+# Variable: AIRFLOW__LOGGING__DELETE_LOCAL_LOGS
+#
+delete_local_logs = False
+
+# Path to Google Credential JSON file. If omitted, authorization based on `the Application Default
+# Credentials
+# <https://cloud.google.com/docs/authentication/application-default-credentials>`__ will
+# be used.
+#
+# Variable: AIRFLOW__LOGGING__GOOGLE_KEY_PATH
+#
+google_key_path = 
+
+# Storage bucket URL for remote logging
+# S3 buckets should start with **s3://**
+# Cloudwatch log groups should start with **cloudwatch://**
+# GCS buckets should start with **gs://**
+# WASB buckets should start with **wasb** just to help Airflow select correct handler
+# Stackdriver logs should start with **stackdriver://**
+#
+# Variable: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
+#
+remote_base_log_folder = 
+
+# The remote_task_handler_kwargs param is loaded into a dictionary and passed to the ``__init__``
+# of remote task handler and it overrides the values provided by Airflow config. For example if you set
+# ``delete_local_logs=False`` and you provide ``{"delete_local_copy": true}``, then the local
+# log files will be deleted after they are uploaded to remote location.
+#
+# Example: remote_task_handler_kwargs = {"delete_local_copy": true}
+#
+# Variable: AIRFLOW__LOGGING__REMOTE_TASK_HANDLER_KWARGS
+#
+remote_task_handler_kwargs = 
+
+# Use server-side encryption for logs stored in S3
+#
+# Variable: AIRFLOW__LOGGING__ENCRYPT_S3_LOGS
+#
+encrypt_s3_logs = False
+
+# Logging level.
+# 
+# Supported values: ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
+#
+# Variable: AIRFLOW__LOGGING__LOGGING_LEVEL
+#
+logging_level = INFO
+
+# Logging level for celery. If not set, it uses the value of logging_level
+# 
+# Supported values: ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
+#
+# Variable: AIRFLOW__LOGGING__CELERY_LOGGING_LEVEL
+#
+celery_logging_level = 
+
+# Logging level for Flask-appbuilder UI.
+# 
+# Supported values: ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
+#
+# Variable: AIRFLOW__LOGGING__FAB_LOGGING_LEVEL
+#
+fab_logging_level = WARNING
+
+# Logging class
+# Specify the class that will specify the logging configuration
+# This class has to be on the python classpath
+#
+# Example: logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
+#
+# Variable: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
+#
+logging_config_class = 
+
+# Flag to enable/disable Colored logs in Console
+# Colour the logs when the controlling terminal is a TTY.
+#
+# Variable: AIRFLOW__LOGGING__COLORED_CONSOLE_LOG
+#
+colored_console_log = True
+
+# Log format for when Colored logs is enabled
+#
+# Variable: AIRFLOW__LOGGING__COLORED_LOG_FORMAT
+#
+colored_log_format = [%%(blue)s%%(asctime)s%%(reset)s] {%%(blue)s%%(filename)s:%%(reset)s%%(lineno)d} %%(log_color)s%%(levelname)s%%(reset)s - %%(log_color)s%%(message)s%%(reset)s
+
+# Specifies the class utilized by Airflow to implement colored logging
+#
+# Variable: AIRFLOW__LOGGING__COLORED_FORMATTER_CLASS
+#
+colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter
+
+# Format of Log line
+#
+# Variable: AIRFLOW__LOGGING__LOG_FORMAT
+#
+log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
+
+# Defines the format of log messages for simple logging configuration
+#
+# Variable: AIRFLOW__LOGGING__SIMPLE_LOG_FORMAT
+#
+simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
+
+# Where to send dag parser logs. If "file", logs are sent to log files defined by child_process_log_directory.
+#
+# Variable: AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET
+#
+dag_processor_log_target = file
+
+# Format of Dag Processor Log line
+#
+# Variable: AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_FORMAT
+#
+dag_processor_log_format = [%%(asctime)s] [SOURCE:DAG_PROCESSOR] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
+
+# Determines the formatter class used by Airflow for structuring its log messages
+# The default formatter class is timezone-aware, which means that timestamps attached to log entries
+# will be adjusted to reflect the local timezone of the Airflow instance
+#
+# Variable: AIRFLOW__LOGGING__LOG_FORMATTER_CLASS
+#
+log_formatter_class = airflow.utils.log.timezone_aware.TimezoneAware
+
+# An import path to a function to add adaptations of each secret added with
+# ``airflow.utils.log.secrets_masker.mask_secret`` to be masked in log messages. The given function
+# is expected to require a single parameter: the secret to be adapted. It may return a
+# single adaptation of the secret or an iterable of adaptations to each be masked as secrets.
+# The original secret will be masked as well as any adaptations returned.
+#
+# Example: secret_mask_adapter = urllib.parse.quote
+#
+# Variable: AIRFLOW__LOGGING__SECRET_MASK_ADAPTER
+#
+secret_mask_adapter = 
+
+# Specify prefix pattern like mentioned below with stream handler ``TaskHandlerWithCustomFormatter``
+#
+# Example: task_log_prefix_template = {{ti.dag_id}}-{{ti.task_id}}-{{execution_date}}-{{ti.try_number}}
+#
+# Variable: AIRFLOW__LOGGING__TASK_LOG_PREFIX_TEMPLATE
+#
+task_log_prefix_template = 
+
+# Formatting for how airflow generates file names/paths for each task run.
+#
+# Variable: AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE
+#
+log_filename_template = dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{%% if ti.map_index >= 0 %%}map_index={{ ti.map_index }}/{%% endif %%}attempt={{ try_number }}.log
+
+# Formatting for how airflow generates file names for log
+#
+# Variable: AIRFLOW__LOGGING__LOG_PROCESSOR_FILENAME_TEMPLATE
+#
+log_processor_filename_template = {{ filename }}.log
+
+# Full path of dag_processor_manager logfile.
+#
+# Variable: AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_LOCATION
+#
+dag_processor_manager_log_location = /kaggle/working/BTC-USDT-ETL-Pipeline/airflow/logs/dag_processor_manager/dag_processor_manager.log
+
+# Whether DAG processor manager will write logs to stdout
+#
+# Variable: AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_STDOUT
+#
+dag_processor_manager_log_stdout = False
+
+# Name of handler to read task instance logs.
+# Defaults to use ``task`` handler.
+#
+# Variable: AIRFLOW__LOGGING__TASK_LOG_READER
+#
+task_log_reader = task
+
+# A comma\-separated list of third-party logger names that will be configured to print messages to
+# consoles\.
+#
+# Example: extra_logger_names = connexion,sqlalchemy
+#
+# Variable: AIRFLOW__LOGGING__EXTRA_LOGGER_NAMES
+#
+extra_logger_names = 
+
+# When you start an Airflow worker, Airflow starts a tiny web server
+# subprocess to serve the workers local log files to the airflow main
+# web server, who then builds pages and sends them to users. This defines
+# the port on which the logs are served. It needs to be unused, and open
+# visible from the main web server to connect into the workers.
+#
+# Variable: AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT
+#
+worker_log_server_port = 8793
+
+# Port to serve logs from for triggerer.
+# See ``[logging] worker_log_server_port`` description for more info.
+#
+# Variable: AIRFLOW__LOGGING__TRIGGER_LOG_SERVER_PORT
+#
+trigger_log_server_port = 8794
+
+# We must parse timestamps to interleave logs between trigger and task.  To do so,
+# we need to parse timestamps in log files. In case your log format is non-standard,
+# you may provide import path to callable which takes a string log line and returns
+# the timestamp (datetime.datetime compatible).
+#
+# Example: interleave_timestamp_parser = path.to.my_func
+#
+# Variable: AIRFLOW__LOGGING__INTERLEAVE_TIMESTAMP_PARSER
+#
+# interleave_timestamp_parser = 
+
+# Permissions in the form or of octal string as understood by chmod. The permissions are important
+# when you use impersonation, when logs are written by a different user than airflow. The most secure
+# way of configuring it in this case is to add both users to the same group and make it the default
+# group of both users. Group-writeable logs are default in airflow, but you might decide that you are
+# OK with having the logs other-writeable, in which case you should set it to ``0o777``. You might
+# decide to add more security if you do not use impersonation and change it to ``0o755`` to make it
+# only owner-writeable. You can also make it just readable only for owner by changing it to ``0o700``
+# if all the access (read/write) for your logs happens from the same user.
+#
+# Example: file_task_handler_new_folder_permissions = 0o775
+#
+# Variable: AIRFLOW__LOGGING__FILE_TASK_HANDLER_NEW_FOLDER_PERMISSIONS
+#
+file_task_handler_new_folder_permissions = 0o775
+
+# Permissions in the form or of octal string as understood by chmod. The permissions are important
+# when you use impersonation, when logs are written by a different user than airflow. The most secure
+# way of configuring it in this case is to add both users to the same group and make it the default
+# group of both users. Group-writeable logs are default in airflow, but you might decide that you are
+# OK with having the logs other-writeable, in which case you should set it to ``0o666``. You might
+# decide to add more security if you do not use impersonation and change it to ``0o644`` to make it
+# only owner-writeable. You can also make it just readable only for owner by changing it to ``0o600``
+# if all the access (read/write) for your logs happens from the same user.
+#
+# Example: file_task_handler_new_file_permissions = 0o664
+#
+# Variable: AIRFLOW__LOGGING__FILE_TASK_HANDLER_NEW_FILE_PERMISSIONS
+#
+file_task_handler_new_file_permissions = 0o664
+
+# By default Celery sends all logs into stderr.
+# If enabled any previous logging handlers will get *removed*.
+# With this option AirFlow will create new handlers
+# and send low level logs like INFO and WARNING to stdout,
+# while sending higher severity logs to stderr.
+#
+# Variable: AIRFLOW__LOGGING__CELERY_STDOUT_STDERR_SEPARATION
+#
+celery_stdout_stderr_separation = False
+
+# If enabled, Airflow may ship messages to task logs from outside the task run context, e.g. from
+# the scheduler, executor, or callback execution context. This can help in circumstances such as
+# when there's something blocking the execution of the task and ordinarily there may be no task
+# logs at all.
+# This is set to ``True`` by default. If you encounter issues with this feature
+# (e.g. scheduler performance issues) it can be disabled.
+#
+# Variable: AIRFLOW__LOGGING__ENABLE_TASK_CONTEXT_LOGGER
+#
+enable_task_context_logger = True
+
+# A comma separated list of keywords related to errors whose presence should display the line in red
+# color in UI
+#
+# Variable: AIRFLOW__LOGGING__COLOR_LOG_ERROR_KEYWORDS
+#
+color_log_error_keywords = error,exception
+
+# A comma separated list of keywords related to warning whose presence should display the line in yellow
+# color in UI
+#
+# Variable: AIRFLOW__LOGGING__COLOR_LOG_WARNING_KEYWORDS
+#
+color_log_warning_keywords = warn
+
+[metrics]
+# `StatsD <https://github.com/statsd/statsd>`__ integration settings.
+
+# If true, ``[metrics] metrics_allow_list`` and ``[metrics] metrics_block_list`` will use
+# regex pattern matching anywhere within the metric name instead of only prefix matching
+# at the start of the name.
+#
+# Variable: AIRFLOW__METRICS__METRICS_USE_PATTERN_MATCH
+#
+metrics_use_pattern_match = False
+
+# Configure an allow list (comma separated string) to send only certain metrics.
+# If ``[metrics] metrics_use_pattern_match`` is ``false``, match only the exact metric name prefix.
+# If ``[metrics] metrics_use_pattern_match`` is ``true``, provide regex patterns to match.
+#
+# Example: metrics_allow_list = "scheduler,executor,dagrun,pool,triggerer,celery" or "^scheduler,^executor,heartbeat|timeout"
+#
+# Variable: AIRFLOW__METRICS__METRICS_ALLOW_LIST
+#
+metrics_allow_list = 
+
+# Configure a block list (comma separated string) to block certain metrics from being emitted.
+# If ``[metrics] metrics_allow_list`` and ``[metrics] metrics_block_list`` are both configured,
+# ``[metrics] metrics_block_list`` is ignored.
+# 
+# If ``[metrics] metrics_use_pattern_match`` is ``false``, match only the exact metric name prefix.
+# 
+# If ``[metrics] metrics_use_pattern_match`` is ``true``, provide regex patterns to match.
+#
+# Example: metrics_block_list = "scheduler,executor,dagrun,pool,triggerer,celery" or "^scheduler,^executor,heartbeat|timeout"
+#
+# Variable: AIRFLOW__METRICS__METRICS_BLOCK_LIST
+#
+metrics_block_list = 
+
+# Enables sending metrics to StatsD.
+#
+# Variable: AIRFLOW__METRICS__STATSD_ON
+#
+statsd_on = False
+
+# Specifies the host address where the StatsD daemon (or server) is running
+#
+# Variable: AIRFLOW__METRICS__STATSD_HOST
+#
+statsd_host = localhost
+
+# Specifies the port on which the StatsD daemon (or server) is listening to
+#
+# Variable: AIRFLOW__METRICS__STATSD_PORT
+#
+statsd_port = 8125
+
+# Defines the namespace for all metrics sent from Airflow to StatsD
+#
+# Variable: AIRFLOW__METRICS__STATSD_PREFIX
+#
+statsd_prefix = airflow
+
+# A function that validate the StatsD stat name, apply changes to the stat name if necessary and return
+# the transformed stat name.
+# 
+# The function should have the following signature
+# 
+# .. code-block:: python
+# 
+#     def func_name(stat_name: str) -> str: ...
+#
+# Variable: AIRFLOW__METRICS__STAT_NAME_HANDLER
+#
+stat_name_handler = 
+
+# To enable datadog integration to send airflow metrics.
+#
+# Variable: AIRFLOW__METRICS__STATSD_DATADOG_ENABLED
+#
+statsd_datadog_enabled = False
+
+# List of datadog tags attached to all metrics(e.g: ``key1:value1,key2:value2``)
+#
+# Variable: AIRFLOW__METRICS__STATSD_DATADOG_TAGS
+#
+statsd_datadog_tags = 
+
+# Set to ``False`` to disable metadata tags for some of the emitted metrics
+#
+# Variable: AIRFLOW__METRICS__STATSD_DATADOG_METRICS_TAGS
+#
+statsd_datadog_metrics_tags = True
+
+# If you want to utilise your own custom StatsD client set the relevant
+# module path below.
+# Note: The module path must exist on your
+# `PYTHONPATH <https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH>`
+# for Airflow to pick it up
+#
+# Variable: AIRFLOW__METRICS__STATSD_CUSTOM_CLIENT_PATH
+#
+# statsd_custom_client_path = 
+
+# If you want to avoid sending all the available metrics tags to StatsD,
+# you can configure a block list of prefixes (comma separated) to filter out metric tags
+# that start with the elements of the list (e.g: ``job_id,run_id``)
+#
+# Example: statsd_disabled_tags = job_id,run_id,dag_id,task_id
+#
+# Variable: AIRFLOW__METRICS__STATSD_DISABLED_TAGS
+#
+statsd_disabled_tags = job_id,run_id
+
+# To enable sending Airflow metrics with StatsD-Influxdb tagging convention.
+#
+# Variable: AIRFLOW__METRICS__STATSD_INFLUXDB_ENABLED
+#
+statsd_influxdb_enabled = False
+
+# Enables sending metrics to OpenTelemetry.
+#
+# Variable: AIRFLOW__METRICS__OTEL_ON
+#
+otel_on = False
+
+# Specifies the hostname or IP address of the OpenTelemetry Collector to which Airflow sends
+# metrics and traces.
+#
+# Variable: AIRFLOW__METRICS__OTEL_HOST
+#
+otel_host = localhost
+
+# Specifies the port of the OpenTelemetry Collector that is listening to.
+#
+# Variable: AIRFLOW__METRICS__OTEL_PORT
+#
+otel_port = 8889
+
+# The prefix for the Airflow metrics.
+#
+# Variable: AIRFLOW__METRICS__OTEL_PREFIX
+#
+otel_prefix = airflow
+
+# Defines the interval, in milliseconds, at which Airflow sends batches of metrics and traces
+# to the configured OpenTelemetry Collector.
+#
+# Variable: AIRFLOW__METRICS__OTEL_INTERVAL_MILLISECONDS
+#
+otel_interval_milliseconds = 60000
+
+# If ``True``, all metrics are also emitted to the console. Defaults to ``False``.
+#
+# Variable: AIRFLOW__METRICS__OTEL_DEBUGGING_ON
+#
+otel_debugging_on = False
+
+# The default service name of traces.
+#
+# Variable: AIRFLOW__METRICS__OTEL_SERVICE
+#
+otel_service = Airflow
+
+# If ``True``, SSL will be enabled. Defaults to ``False``.
+# To establish an HTTPS connection to the OpenTelemetry collector,
+# you need to configure the SSL certificate and key within the OpenTelemetry collector's
+# ``config.yml`` file.
+#
+# Variable: AIRFLOW__METRICS__OTEL_SSL_ACTIVE
+#
+otel_ssl_active = False
+
+[traces]
+# Distributed traces integration settings.
+
+# Enables sending traces to OpenTelemetry.
+#
+# Variable: AIRFLOW__TRACES__OTEL_ON
+#
+otel_on = False
+
+# Specifies the hostname or IP address of the OpenTelemetry Collector to which Airflow sends
+# traces.
+#
+# Variable: AIRFLOW__TRACES__OTEL_HOST
+#
+otel_host = localhost
+
+# Specifies the port of the OpenTelemetry Collector that is listening to.
+#
+# Variable: AIRFLOW__TRACES__OTEL_PORT
+#
+otel_port = 8889
+
+# The default service name of traces.
+#
+# Variable: AIRFLOW__TRACES__OTEL_SERVICE
+#
+otel_service = Airflow
+
+# If True, all traces are also emitted to the console. Defaults to False.
+#
+# Variable: AIRFLOW__TRACES__OTEL_DEBUGGING_ON
+#
+otel_debugging_on = False
+
+# If True, SSL will be enabled.  Defaults to False.
+# To establish an HTTPS connection to the OpenTelemetry collector,
+# you need to configure the SSL certificate and key within the OpenTelemetry collector's
+# config.yml file.
+#
+# Variable: AIRFLOW__TRACES__OTEL_SSL_ACTIVE
+#
+otel_ssl_active = False
+
+# If True, after the task is complete, the full task log messages will be added as the
+# span events, chunked by 64k size. defaults to False.
+#
+# Variable: AIRFLOW__TRACES__OTEL_TASK_LOG_EVENT
+#
+otel_task_log_event = False
+
+[secrets]
+# Full class name of secrets backend to enable (will precede env vars and metastore in search path)
+#
+# Example: backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
+#
+# Variable: AIRFLOW__SECRETS__BACKEND
+#
+backend = 
+
+# The backend_kwargs param is loaded into a dictionary and passed to ``__init__``
+# of secrets backend class. See documentation for the secrets backend you are using.
+# JSON is expected.
+# 
+# Example for AWS Systems Manager ParameterStore:
+# ``{"connections_prefix": "/airflow/connections", "profile_name": "default"}``
+#
+# Variable: AIRFLOW__SECRETS__BACKEND_KWARGS
+#
+backend_kwargs = 
+
+# .. note:: |experimental|
+# 
+# Enables local caching of Variables, when parsing DAGs only.
+# Using this option can make dag parsing faster if Variables are used in top level code, at the expense
+# of longer propagation time for changes.
+# Please note that this cache concerns only the DAG parsing step. There is no caching in place when DAG
+# tasks are run.
+#
+# Variable: AIRFLOW__SECRETS__USE_CACHE
+#
+use_cache = False
+
+# .. note:: |experimental|
+# 
+# When the cache is enabled, this is the duration for which we consider an entry in the cache to be
+# valid. Entries are refreshed if they are older than this many seconds.
+# It means that when the cache is enabled, this is the maximum amount of time you need to wait to see a
+# Variable change take effect.
+#
+# Variable: AIRFLOW__SECRETS__CACHE_TTL_SECONDS
+#
+cache_ttl_seconds = 900
+
+[cli]
+# In what way should the cli access the API. The LocalClient will use the
+# database directly, while the json_client will use the api running on the
+# webserver
+#
+# Variable: AIRFLOW__CLI__API_CLIENT
+#
+api_client = airflow.api.client.local_client
+
+# If you set web_server_url_prefix, do NOT forget to append it here, ex:
+# ``endpoint_url = http://localhost:8080/myroot``
+# So api will look like: ``http://localhost:8080/myroot/api/experimental/...``
+#
+# Variable: AIRFLOW__CLI__ENDPOINT_URL
+#
+endpoint_url = http://localhost:8080
+
+[debug]
+# Used only with ``DebugExecutor``. If set to ``True`` DAG will fail with first
+# failed task. Helpful for debugging purposes.
+#
+# Variable: AIRFLOW__DEBUG__FAIL_FAST
+#
+fail_fast = False
+
+[api]
+# Enables the deprecated experimental API. Please note that these API endpoints do not have
+# access control. An authenticated user has full access.
+# 
+# .. warning::
+# 
+#   This `Experimental REST API
+#   <https://airflow.apache.org/docs/apache-airflow/stable/deprecated-rest-api-ref.html>`__ is
+#   deprecated since version 2.0. Please consider using
+#   `the Stable REST API
+#   <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html>`__.
+#   For more information on migration, see
+#   `RELEASE_NOTES.rst <https://github.com/apache/airflow/blob/main/RELEASE_NOTES.rst>`_
+#
+# Variable: AIRFLOW__API__ENABLE_EXPERIMENTAL_API
+#
+enable_experimental_api = False
+
+# Comma separated list of auth backends to authenticate users of the API. See
+# `Security: API
+# <https://airflow.apache.org/docs/apache-airflow/stable/security/api.html>`__ for possible values.
+# ("airflow.api.auth.backend.default" allows all requests for historic reasons)
+#
+# Variable: AIRFLOW__API__AUTH_BACKENDS
+#
+auth_backends = airflow.api.auth.backend.session
+
+# Used to set the maximum page limit for API requests. If limit passed as param
+# is greater than maximum page limit, it will be ignored and maximum page limit value
+# will be set as the limit
+#
+# Variable: AIRFLOW__API__MAXIMUM_PAGE_LIMIT
+#
+maximum_page_limit = 100
+
+# Used to set the default page limit when limit param is zero or not provided in API
+# requests. Otherwise if positive integer is passed in the API requests as limit, the
+# smallest number of user given limit or maximum page limit is taken as limit.
+#
+# Variable: AIRFLOW__API__FALLBACK_PAGE_LIMIT
+#
+fallback_page_limit = 100
+
+# The intended audience for JWT token credentials used for authorization. This value must match on the client and server sides. If empty, audience will not be tested.
+#
+# Example: google_oauth2_audience = project-id-random-value.apps.googleusercontent.com
+#
+# Variable: AIRFLOW__API__GOOGLE_OAUTH2_AUDIENCE
+#
+google_oauth2_audience = 
+
+# Path to Google Cloud Service Account key file (JSON). If omitted, authorization based on
+# `the Application Default Credentials
+# <https://cloud.google.com/docs/authentication/production#finding_credentials_automatically>`__ will
+# be used.
+#
+# Example: google_key_path = /files/service-account-json
+#
+# Variable: AIRFLOW__API__GOOGLE_KEY_PATH
+#
+google_key_path = 
+
+# Used in response to a preflight request to indicate which HTTP
+# headers can be used when making the actual request. This header is
+# the server side response to the browser's
+# Access-Control-Request-Headers header.
+#
+# Variable: AIRFLOW__API__ACCESS_CONTROL_ALLOW_HEADERS
+#
+access_control_allow_headers = 
+
+# Specifies the method or methods allowed when accessing the resource.
+#
+# Variable: AIRFLOW__API__ACCESS_CONTROL_ALLOW_METHODS
+#
+access_control_allow_methods = 
+
+# Indicates whether the response can be shared with requesting code from the given origins.
+# Separate URLs with space.
+#
+# Variable: AIRFLOW__API__ACCESS_CONTROL_ALLOW_ORIGINS
+#
+access_control_allow_origins = 
+
+# Indicates whether the **xcomEntries** endpoint supports the **deserialize**
+# flag. If set to ``False``, setting this flag in a request would result in a
+# 400 Bad Request error.
+#
+# Variable: AIRFLOW__API__ENABLE_XCOM_DESERIALIZE_SUPPORT
+#
+enable_xcom_deserialize_support = False
+
+[lineage]
+# what lineage backend to use
+#
+# Variable: AIRFLOW__LINEAGE__BACKEND
+#
+backend = 
+
+[operators]
+# The default owner assigned to each new operator, unless
+# provided explicitly or passed via ``default_args``
+#
+# Variable: AIRFLOW__OPERATORS__DEFAULT_OWNER
+#
+default_owner = airflow
+
+# The default value of attribute "deferrable" in operators and sensors.
+#
+# Variable: AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE
+#
+default_deferrable = false
+
+# Indicates the default number of CPU units allocated to each operator when no specific CPU request
+# is specified in the operator's configuration
+#
+# Variable: AIRFLOW__OPERATORS__DEFAULT_CPUS
+#
+default_cpus = 1
+
+# Indicates the default number of RAM allocated to each operator when no specific RAM request
+# is specified in the operator's configuration
+#
+# Variable: AIRFLOW__OPERATORS__DEFAULT_RAM
+#
+default_ram = 512
+
+# Indicates the default number of disk storage allocated to each operator when no specific disk request
+# is specified in the operator's configuration
+#
+# Variable: AIRFLOW__OPERATORS__DEFAULT_DISK
+#
+default_disk = 512
+
+# Indicates the default number of GPUs allocated to each operator when no specific GPUs request
+# is specified in the operator's configuration
+#
+# Variable: AIRFLOW__OPERATORS__DEFAULT_GPUS
+#
+default_gpus = 0
+
+# Default queue that tasks get assigned to and that worker listen on.
+#
+# Variable: AIRFLOW__OPERATORS__DEFAULT_QUEUE
+#
+default_queue = default
+
+# Is allowed to pass additional/unused arguments (args, kwargs) to the BaseOperator operator.
+# If set to ``False``, an exception will be thrown,
+# otherwise only the console message will be displayed.
+#
+# Variable: AIRFLOW__OPERATORS__ALLOW_ILLEGAL_ARGUMENTS
+#
+allow_illegal_arguments = False
+
+[webserver]
+# The message displayed when a user attempts to execute actions beyond their authorised privileges.
+#
+# Variable: AIRFLOW__WEBSERVER__ACCESS_DENIED_MESSAGE
+#
+access_denied_message = Access is Denied
+
+# Path of webserver config file used for configuring the webserver parameters
+#
+# Variable: AIRFLOW__WEBSERVER__CONFIG_FILE
+#
+config_file = /kaggle/working/BTC-USDT-ETL-Pipeline/airflow/webserver_config.py
+
+# The base url of your website: Airflow cannot guess what domain or CNAME you are using.
+# This is used to create links in the Log Url column in the Browse - Task Instances menu,
+# as well as in any automated emails sent by Airflow that contain links to your webserver.
+#
+# Variable: AIRFLOW__WEBSERVER__BASE_URL
+#
+base_url = http://localhost:8080
+
+# Default timezone to display all dates in the UI, can be UTC, system, or
+# any IANA timezone string (e.g. **Europe/Amsterdam**). If left empty the
+# default value of core/default_timezone will be used
+#
+# Example: default_ui_timezone = America/New_York
+#
+# Variable: AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE
+#
+default_ui_timezone = UTC
+
+# The ip specified when starting the web server
+#
+# Variable: AIRFLOW__WEBSERVER__WEB_SERVER_HOST
+#
+web_server_host = 0.0.0.0
+
+# The port on which to run the web server
+#
+# Variable: AIRFLOW__WEBSERVER__WEB_SERVER_PORT
+#
+web_server_port = 8080
+
+# Paths to the SSL certificate and key for the web server. When both are
+# provided SSL will be enabled. This does not change the web server port.
+#
+# Variable: AIRFLOW__WEBSERVER__WEB_SERVER_SSL_CERT
+#
+web_server_ssl_cert = 
+
+# Paths to the SSL certificate and key for the web server. When both are
+# provided SSL will be enabled. This does not change the web server port.
+#
+# Variable: AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY
+#
+web_server_ssl_key = 
+
+# The type of backend used to store web session data, can be ``database`` or ``securecookie``. For the
+# ``database`` backend, sessions are store in the database and they can be
+# managed there (for example when you reset password of the user, all sessions for that user are
+# deleted). For the ``securecookie`` backend, sessions are stored in encrypted cookies on the client
+# side. The ``securecookie`` mechanism is 'lighter' than database backend, but sessions are not deleted
+# when you reset password of the user, which means that other than waiting for expiry time, the only
+# way to invalidate all sessions for a user is to change secret_key and restart webserver (which
+# also invalidates and logs out all other user's sessions).
+# 
+# When you are using ``database`` backend, make sure to keep your database session table small
+# by periodically running ``airflow db clean --table session`` command, especially if you have
+# automated API calls that will create a new session for each call rather than reuse the sessions
+# stored in browser cookies.
+#
+# Example: session_backend = securecookie
+#
+# Variable: AIRFLOW__WEBSERVER__SESSION_BACKEND
+#
+session_backend = database
+
+# Number of seconds the webserver waits before killing gunicorn master that doesn't respond
+#
+# Variable: AIRFLOW__WEBSERVER__WEB_SERVER_MASTER_TIMEOUT
+#
+web_server_master_timeout = 120
+
+# Number of seconds the gunicorn webserver waits before timing out on a worker
+#
+# Variable: AIRFLOW__WEBSERVER__WEB_SERVER_WORKER_TIMEOUT
+#
+web_server_worker_timeout = 120
+
+# Number of workers to refresh at a time. When set to 0, worker refresh is
+# disabled. When nonzero, airflow periodically refreshes webserver workers by
+# bringing up new ones and killing old ones.
+#
+# Variable: AIRFLOW__WEBSERVER__WORKER_REFRESH_BATCH_SIZE
+#
+worker_refresh_batch_size = 1
+
+# Number of seconds to wait before refreshing a batch of workers.
+#
+# Variable: AIRFLOW__WEBSERVER__WORKER_REFRESH_INTERVAL
+#
+worker_refresh_interval = 6000
+
+# If set to ``True``, Airflow will track files in plugins_folder directory. When it detects changes,
+# then reload the gunicorn. If set to ``True``, gunicorn starts without preloading, which is slower,
+# uses more memory, and may cause race conditions. Avoid setting this to ``True`` in production.
+#
+# Variable: AIRFLOW__WEBSERVER__RELOAD_ON_PLUGIN_CHANGE
+#
+reload_on_plugin_change = False
+
+# Secret key used to run your flask app. It should be as random as possible. However, when running
+# more than 1 instances of webserver, make sure all of them use the same ``secret_key`` otherwise
+# one of them will error with "CSRF session token is missing".
+# The webserver key is also used to authorize requests to Celery workers when logs are retrieved.
+# The token generated using the secret key has a short expiry time though - make sure that time on
+# ALL the machines that you run airflow components on is synchronized (for example using ntpd)
+# otherwise you might get "forbidden" errors when the logs are accessed.
+#
+# Variable: AIRFLOW__WEBSERVER__SECRET_KEY
+#
+secret_key = JRNP2IC4kIaVxisy9+AW4A==
+
+# Number of workers to run the Gunicorn web server
+#
+# Variable: AIRFLOW__WEBSERVER__WORKERS
+#
+workers = 4
+
+# The worker class gunicorn should use. Choices include
+# ``sync`` (default), ``eventlet``, ``gevent``.
+# 
+# .. warning::
+# 
+#     When using ``gevent`` you might also want to set the ``_AIRFLOW_PATCH_GEVENT``
+#     environment variable to ``"1"`` to make sure gevent patching is done as early as possible.
+# 
+#     Be careful to set ``_AIRFLOW_PATCH_GEVENT`` only on the web server as gevent patching may
+#     affect the scheduler behavior via the ``multiprocessing`` sockets module and cause crash.
+# 
+#     See related Issues / PRs for more details:
+# 
+#     * https://github.com/benoitc/gunicorn/issues/2796
+#     * https://github.com/apache/airflow/issues/8212
+#     * https://github.com/apache/airflow/pull/28283
+#
+# Variable: AIRFLOW__WEBSERVER__WORKER_CLASS
+#
+worker_class = sync
+
+# Log files for the gunicorn webserver. '-' means log to stderr.
+#
+# Variable: AIRFLOW__WEBSERVER__ACCESS_LOGFILE
+#
+access_logfile = -
+
+# Log files for the gunicorn webserver. '-' means log to stderr.
+#
+# Variable: AIRFLOW__WEBSERVER__ERROR_LOGFILE
+#
+error_logfile = -
+
+# Access log format for gunicorn webserver.
+# default format is ``%%(h)s %%(l)s %%(u)s %%(t)s "%%(r)s" %%(s)s %%(b)s "%%(f)s" "%%(a)s"``
+# See `Gunicorn Settings: 'access_log_format' Reference
+# <https://docs.gunicorn.org/en/stable/settings.html#access-log-format>`__ for more details
+#
+# Variable: AIRFLOW__WEBSERVER__ACCESS_LOGFORMAT
+#
+access_logformat = 
+
+# Expose the configuration file in the web server. Set to ``non-sensitive-only`` to show all values
+# except those that have security implications. ``True`` shows all values. ``False`` hides the
+# configuration completely.
+#
+# Variable: AIRFLOW__WEBSERVER__EXPOSE_CONFIG
+#
+expose_config = False
+
+# Expose hostname in the web server
+#
+# Variable: AIRFLOW__WEBSERVER__EXPOSE_HOSTNAME
+#
+expose_hostname = False
+
+# Expose stacktrace in the web server
+#
+# Variable: AIRFLOW__WEBSERVER__EXPOSE_STACKTRACE
+#
+expose_stacktrace = False
+
+# Default DAG view. Valid values are: ``grid``, ``graph``, ``duration``, ``gantt``, ``landing_times``
+#
+# Variable: AIRFLOW__WEBSERVER__DAG_DEFAULT_VIEW
+#
+dag_default_view = grid
+
+# Default DAG orientation. Valid values are:
+# ``LR`` (Left->Right), ``TB`` (Top->Bottom), ``RL`` (Right->Left), ``BT`` (Bottom->Top)
+#
+# Variable: AIRFLOW__WEBSERVER__DAG_ORIENTATION
+#
+dag_orientation = LR
+
+# Sorting order in grid view. Valid values are: ``topological``, ``hierarchical_alphabetical``
+#
+# Variable: AIRFLOW__WEBSERVER__GRID_VIEW_SORTING_ORDER
+#
+grid_view_sorting_order = topological
+
+# The amount of time (in secs) webserver will wait for initial handshake
+# while fetching logs from other worker machine
+#
+# Variable: AIRFLOW__WEBSERVER__LOG_FETCH_TIMEOUT_SEC
+#
+log_fetch_timeout_sec = 5
+
+# Time interval (in secs) to wait before next log fetching.
+#
+# Variable: AIRFLOW__WEBSERVER__LOG_FETCH_DELAY_SEC
+#
+log_fetch_delay_sec = 2
+
+# Distance away from page bottom to enable auto tailing.
+#
+# Variable: AIRFLOW__WEBSERVER__LOG_AUTO_TAILING_OFFSET
+#
+log_auto_tailing_offset = 30
+
+# Animation speed for auto tailing log display.
+#
+# Variable: AIRFLOW__WEBSERVER__LOG_ANIMATION_SPEED
+#
+log_animation_speed = 1000
+
+# By default, the webserver shows paused DAGs. Flip this to hide paused
+# DAGs by default
+#
+# Variable: AIRFLOW__WEBSERVER__HIDE_PAUSED_DAGS_BY_DEFAULT
+#
+hide_paused_dags_by_default = False
+
+# Consistent page size across all listing views in the UI
+#
+# Variable: AIRFLOW__WEBSERVER__PAGE_SIZE
+#
+page_size = 100
+
+# Define the color of navigation bar
+#
+# Variable: AIRFLOW__WEBSERVER__NAVBAR_COLOR
+#
+navbar_color = #fff
+
+# Define the color of text in the navigation bar
+#
+# Variable: AIRFLOW__WEBSERVER__NAVBAR_TEXT_COLOR
+#
+navbar_text_color = #51504f
+
+# Define the color of navigation bar links when hovered
+#
+# Variable: AIRFLOW__WEBSERVER__NAVBAR_HOVER_COLOR
+#
+navbar_hover_color = #eee
+
+# Define the color of text in the navigation bar when hovered
+#
+# Variable: AIRFLOW__WEBSERVER__NAVBAR_TEXT_HOVER_COLOR
+#
+navbar_text_hover_color = #51504f
+
+# Define the color of the logo text
+#
+# Variable: AIRFLOW__WEBSERVER__NAVBAR_LOGO_TEXT_COLOR
+#
+navbar_logo_text_color = #51504f
+
+# Default dagrun to show in UI
+#
+# Variable: AIRFLOW__WEBSERVER__DEFAULT_DAG_RUN_DISPLAY_NUMBER
+#
+default_dag_run_display_number = 25
+
+# Enable werkzeug ``ProxyFix`` middleware for reverse proxy
+#
+# Variable: AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX
+#
+enable_proxy_fix = False
+
+# Number of values to trust for ``X-Forwarded-For``.
+# See `Werkzeug: X-Forwarded-For Proxy Fix
+# <https://werkzeug.palletsprojects.com/en/2.3.x/middleware/proxy_fix/>`__ for more details.
+#
+# Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_FOR
+#
+proxy_fix_x_for = 1
+
+# Number of values to trust for ``X-Forwarded-Proto``.
+# See `Werkzeug: X-Forwarded-For Proxy Fix
+# <https://werkzeug.palletsprojects.com/en/2.3.x/middleware/proxy_fix/>`__ for more details.
+#
+# Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_PROTO
+#
+proxy_fix_x_proto = 1
+
+# Number of values to trust for ``X-Forwarded-Host``.
+# See `Werkzeug: X-Forwarded-For Proxy Fix
+# <https://werkzeug.palletsprojects.com/en/2.3.x/middleware/proxy_fix/>`__ for more details.
+#
+# Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_HOST
+#
+proxy_fix_x_host = 1
+
+# Number of values to trust for ``X-Forwarded-Port``.
+# See `Werkzeug: X-Forwarded-For Proxy Fix
+# <https://werkzeug.palletsprojects.com/en/2.3.x/middleware/proxy_fix/>`__ for more details.
+#
+# Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_PORT
+#
+proxy_fix_x_port = 1
+
+# Number of values to trust for ``X-Forwarded-Prefix``.
+# See `Werkzeug: X-Forwarded-For Proxy Fix
+# <https://werkzeug.palletsprojects.com/en/2.3.x/middleware/proxy_fix/>`__ for more details.
+#
+# Variable: AIRFLOW__WEBSERVER__PROXY_FIX_X_PREFIX
+#
+proxy_fix_x_prefix = 1
+
+# Set secure flag on session cookie
+#
+# Variable: AIRFLOW__WEBSERVER__COOKIE_SECURE
+#
+cookie_secure = False
+
+# Set samesite policy on session cookie
+#
+# Variable: AIRFLOW__WEBSERVER__COOKIE_SAMESITE
+#
+cookie_samesite = Lax
+
+# Default setting for wrap toggle on DAG code and TI log views.
+#
+# Variable: AIRFLOW__WEBSERVER__DEFAULT_WRAP
+#
+default_wrap = False
+
+# Allow the UI to be rendered in a frame
+#
+# Variable: AIRFLOW__WEBSERVER__X_FRAME_ENABLED
+#
+x_frame_enabled = True
+
+# Send anonymous user activity to your analytics tool
+# choose from ``google_analytics``, ``segment``, ``metarouter``, or ``matomo``
+#
+# Variable: AIRFLOW__WEBSERVER__ANALYTICS_TOOL
+#
+# analytics_tool = 
+
+# Unique ID of your account in the analytics tool
+#
+# Variable: AIRFLOW__WEBSERVER__ANALYTICS_ID
+#
+# analytics_id = 
+
+# Your instances url, only applicable to Matomo.
+#
+# Example: analytics_url = https://your.matomo.instance.com/
+#
+# Variable: AIRFLOW__WEBSERVER__ANALYTICS_URL
+#
+# analytics_url = 
+
+# 'Recent Tasks' stats will show for old DagRuns if set
+#
+# Variable: AIRFLOW__WEBSERVER__SHOW_RECENT_STATS_FOR_COMPLETED_RUNS
+#
+show_recent_stats_for_completed_runs = True
+
+# The UI cookie lifetime in minutes. User will be logged out from UI after
+# ``[webserver] session_lifetime_minutes`` of non-activity
+#
+# Variable: AIRFLOW__WEBSERVER__SESSION_LIFETIME_MINUTES
+#
+session_lifetime_minutes = 43200
+
+# Sets a custom page title for the DAGs overview page and site title for all pages
+#
+# Variable: AIRFLOW__WEBSERVER__INSTANCE_NAME
+#
+# instance_name = 
+
+# Whether the custom page title for the DAGs overview page contains any Markup language
+#
+# Variable: AIRFLOW__WEBSERVER__INSTANCE_NAME_HAS_MARKUP
+#
+instance_name_has_markup = False
+
+# How frequently, in seconds, the DAG data will auto-refresh in graph or grid view
+# when auto-refresh is turned on
+#
+# Variable: AIRFLOW__WEBSERVER__AUTO_REFRESH_INTERVAL
+#
+auto_refresh_interval = 3
+
+# Boolean for displaying warning for publicly viewable deployment
+#
+# Variable: AIRFLOW__WEBSERVER__WARN_DEPLOYMENT_EXPOSURE
+#
+warn_deployment_exposure = True
+
+# Comma separated string of view events to exclude from dag audit view.
+# All other events will be added minus the ones passed here.
+# The audit logs in the db will not be affected by this parameter.
+#
+# Example: audit_view_excluded_events = cli_task_run,running,success
+#
+# Variable: AIRFLOW__WEBSERVER__AUDIT_VIEW_EXCLUDED_EVENTS
+#
+# audit_view_excluded_events = 
+
+# Comma separated string of view events to include in dag audit view.
+# If passed, only these events will populate the dag audit view.
+# The audit logs in the db will not be affected by this parameter.
+#
+# Example: audit_view_included_events = dagrun_cleared,failed
+#
+# Variable: AIRFLOW__WEBSERVER__AUDIT_VIEW_INCLUDED_EVENTS
+#
+# audit_view_included_events = 
+
+# Boolean for running SwaggerUI in the webserver.
+#
+# Variable: AIRFLOW__WEBSERVER__ENABLE_SWAGGER_UI
+#
+enable_swagger_ui = True
+
+# Boolean for running Internal API in the webserver.
+#
+# Variable: AIRFLOW__WEBSERVER__RUN_INTERNAL_API
+#
+run_internal_api = False
+
+# The caching algorithm used by the webserver. Must be a valid hashlib function name.
+#
+# Example: caching_hash_method = sha256
+#
+# Variable: AIRFLOW__WEBSERVER__CACHING_HASH_METHOD
+#
+caching_hash_method = md5
+
+# Behavior of the trigger DAG run button for DAGs without params. ``False`` to skip and trigger
+# without displaying a form to add a **dag_run.conf**, ``True`` to always display the form.
+# The form is displayed always if parameters are defined.
+#
+# Variable: AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAMS
+#
+show_trigger_form_if_no_params = False
+
+# Number of recent DAG run configurations in the selector on the trigger web form.
+#
+# Example: num_recent_configurations_for_trigger = 10
+#
+# Variable: AIRFLOW__WEBSERVER__NUM_RECENT_CONFIGURATIONS_FOR_TRIGGER
+#
+num_recent_configurations_for_trigger = 5
+
+# A DAG author is able to provide any raw HTML into ``doc_md`` or params description in
+# ``description_md`` for text formatting. This is including potentially unsafe javascript.
+# Displaying the DAG or trigger form in web UI provides the DAG author the potential to
+# inject malicious code into clients browsers. To ensure the web UI is safe by default,
+# raw HTML is disabled by default. If you trust your DAG authors, you can enable HTML
+# support in markdown by setting this option to ``True``.
+# 
+# This parameter also enables the deprecated fields ``description_html`` and
+# ``custom_html_form`` in DAG params until the feature is removed in a future version.
+#
+# Example: allow_raw_html_descriptions = False
+#
+# Variable: AIRFLOW__WEBSERVER__ALLOW_RAW_HTML_DESCRIPTIONS
+#
+allow_raw_html_descriptions = False
+
+# The maximum size of the request payload (in MB) that can be sent.
+#
+# Variable: AIRFLOW__WEBSERVER__ALLOWED_PAYLOAD_SIZE
+#
+allowed_payload_size = 1.0
+
+# Require confirmation when changing a DAG in the web UI. This is to prevent accidental changes
+# to a DAG that may be running on sensitive environments like production.
+# When set to ``True``, confirmation dialog will be shown when a user tries to Pause/Unpause,
+# Trigger a DAG
+#
+# Variable: AIRFLOW__WEBSERVER__REQUIRE_CONFIRMATION_DAG_CHANGE
+#
+require_confirmation_dag_change = False
+
+[email]
+# Configuration email backend and whether to
+# send email alerts on retry or failure
+
+# Email backend to use
+#
+# Variable: AIRFLOW__EMAIL__EMAIL_BACKEND
+#
+email_backend = airflow.utils.email.send_email_smtp
+
+# Email connection to use
+#
+# Variable: AIRFLOW__EMAIL__EMAIL_CONN_ID
+#
+email_conn_id = smtp_default
+
+# Whether email alerts should be sent when a task is retried
+#
+# Variable: AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_RETRY
+#
+default_email_on_retry = True
+
+# Whether email alerts should be sent when a task failed
+#
+# Variable: AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_FAILURE
+#
+default_email_on_failure = True
+
+# File that will be used as the template for Email subject (which will be rendered using Jinja2).
+# If not set, Airflow uses a base template.
+#
+# Example: subject_template = /path/to/my_subject_template_file
+#
+# Variable: AIRFLOW__EMAIL__SUBJECT_TEMPLATE
+#
+# subject_template = 
+
+# File that will be used as the template for Email content (which will be rendered using Jinja2).
+# If not set, Airflow uses a base template.
+#
+# Example: html_content_template = /path/to/my_html_content_template_file
+#
+# Variable: AIRFLOW__EMAIL__HTML_CONTENT_TEMPLATE
+#
+# html_content_template = 
+
+# Email address that will be used as sender address.
+# It can either be raw email or the complete address in a format ``Sender Name <sender@email.com>``
+#
+# Example: from_email = Airflow <airflow@example.com>
+#
+# Variable: AIRFLOW__EMAIL__FROM_EMAIL
+#
+# from_email = 
+
+# ssl context to use when using SMTP and IMAP SSL connections. By default, the context is "default"
+# which sets it to ``ssl.create_default_context()`` which provides the right balance between
+# compatibility and security, it however requires that certificates in your operating system are
+# updated and that SMTP/IMAP servers of yours have valid certificates that have corresponding public
+# keys installed on your machines. You can switch it to "none" if you want to disable checking
+# of the certificates, but it is not recommended as it allows MITM (man-in-the-middle) attacks
+# if your infrastructure is not sufficiently secured. It should only be set temporarily while you
+# are fixing your certificate configuration. This can be typically done by upgrading to newer
+# version of the operating system you run Airflow components on,by upgrading/refreshing proper
+# certificates in the OS or by updating certificates for your mail servers.
+#
+# Example: ssl_context = default
+#
+# Variable: AIRFLOW__EMAIL__SSL_CONTEXT
+#
+ssl_context = default
+
+[smtp]
+# If you want airflow to send emails on retries, failure, and you want to use
+# the airflow.utils.email.send_email_smtp function, you have to configure an
+# smtp server here
+
+# Specifies the host server address used by Airflow when sending out email notifications via SMTP.
+#
+# Variable: AIRFLOW__SMTP__SMTP_HOST
+#
+smtp_host = localhost
+
+# Determines whether to use the STARTTLS command when connecting to the SMTP server.
+#
+# Variable: AIRFLOW__SMTP__SMTP_STARTTLS
+#
+smtp_starttls = True
+
+# Determines whether to use an SSL connection when talking to the SMTP server.
+#
+# Variable: AIRFLOW__SMTP__SMTP_SSL
+#
+smtp_ssl = False
+
+# Username to authenticate when connecting to smtp server.
+#
+# Example: smtp_user = airflow
+#
+# Variable: AIRFLOW__SMTP__SMTP_USER
+#
+# smtp_user = 
+
+# Password to authenticate when connecting to smtp server.
+#
+# Example: smtp_password = airflow
+#
+# Variable: AIRFLOW__SMTP__SMTP_PASSWORD
+#
+# smtp_password = 
+
+# Defines the port number on which Airflow connects to the SMTP server to send email notifications.
+#
+# Variable: AIRFLOW__SMTP__SMTP_PORT
+#
+smtp_port = 25
+
+# Specifies the default **from** email address used when Airflow sends email notifications.
+#
+# Variable: AIRFLOW__SMTP__SMTP_MAIL_FROM
+#
+smtp_mail_from = airflow@example.com
+
+# Determines the maximum time (in seconds) the Apache Airflow system will wait for a
+# connection to the SMTP server to be established.
+#
+# Variable: AIRFLOW__SMTP__SMTP_TIMEOUT
+#
+smtp_timeout = 30
+
+# Defines the maximum number of times Airflow will attempt to connect to the SMTP server.
+#
+# Variable: AIRFLOW__SMTP__SMTP_RETRY_LIMIT
+#
+smtp_retry_limit = 5
+
+[sentry]
+# `Sentry <https://docs.sentry.io>`__ integration. Here you can supply
+# additional configuration options based on the Python platform.
+# See `Python / Configuration / Basic Options
+# <https://docs.sentry.io/platforms/python/configuration/options/>`__ for more details.
+# Unsupported options: ``integrations``, ``in_app_include``, ``in_app_exclude``,
+# ``ignore_errors``, ``before_breadcrumb``, ``transport``.
+
+# Enable error reporting to Sentry
+#
+# Variable: AIRFLOW__SENTRY__SENTRY_ON
+#
+sentry_on = false
+
+#
+# Variable: AIRFLOW__SENTRY__SENTRY_DSN
+#
+sentry_dsn = 
+
+# Dotted path to a before_send function that the sentry SDK should be configured to use.
+#
+# Variable: AIRFLOW__SENTRY__BEFORE_SEND
+#
+# before_send = 
+
+[scheduler]
+# Task instances listen for external kill signal (when you clear tasks
+# from the CLI or the UI), this defines the frequency at which they should
+# listen (in seconds).
+#
+# Variable: AIRFLOW__SCHEDULER__JOB_HEARTBEAT_SEC
+#
+job_heartbeat_sec = 5
+
+# The scheduler constantly tries to trigger new tasks (look at the
+# scheduler section in the docs for more information). This defines
+# how often the scheduler should run (in seconds).
+#
+# Variable: AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC
+#
+scheduler_heartbeat_sec = 5
+
+# The frequency (in seconds) at which the LocalTaskJob should send heartbeat signals to the
+# scheduler to notify it's still alive. If this value is set to 0, the heartbeat interval will default
+# to the value of ``[scheduler] scheduler_zombie_task_threshold``.
+#
+# Variable: AIRFLOW__SCHEDULER__LOCAL_TASK_JOB_HEARTBEAT_SEC
+#
+local_task_job_heartbeat_sec = 0
+
+# The number of times to try to schedule each DAG file
+# -1 indicates unlimited number
+#
+# Variable: AIRFLOW__SCHEDULER__NUM_RUNS
+#
+num_runs = -1
+
+# Controls how long the scheduler will sleep between loops, but if there was nothing to do
+# in the loop. i.e. if it scheduled something then it will start the next loop
+# iteration straight away.
+#
+# Variable: AIRFLOW__SCHEDULER__SCHEDULER_IDLE_SLEEP_TIME
+#
+scheduler_idle_sleep_time = 1
+
+# Number of seconds after which a DAG file is parsed. The DAG file is parsed every
+# ``[scheduler] min_file_process_interval`` number of seconds. Updates to DAGs are reflected after
+# this interval. Keeping this number low will increase CPU usage.
+#
+# Variable: AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL
+#
+min_file_process_interval = 30
+
+# How often (in seconds) to check for stale DAGs (DAGs which are no longer present in
+# the expected files) which should be deactivated, as well as datasets that are no longer
+# referenced and should be marked as orphaned.
+#
+# Variable: AIRFLOW__SCHEDULER__PARSING_CLEANUP_INTERVAL
+#
+parsing_cleanup_interval = 60
+
+# How long (in seconds) to wait after we have re-parsed a DAG file before deactivating stale
+# DAGs (DAGs which are no longer present in the expected files). The reason why we need
+# this threshold is to account for the time between when the file is parsed and when the
+# DAG is loaded. The absolute maximum that this could take is ``[core] dag_file_processor_timeout``,
+# but when you have a long timeout configured, it results in a significant delay in the
+# deactivation of stale dags.
+#
+# Variable: AIRFLOW__SCHEDULER__STALE_DAG_THRESHOLD
+#
+stale_dag_threshold = 50
+
+# How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
+#
+# Variable: AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL
+#
+dag_dir_list_interval = 300
+
+# How often should stats be printed to the logs. Setting to 0 will disable printing stats
+#
+# Variable: AIRFLOW__SCHEDULER__PRINT_STATS_INTERVAL
+#
+print_stats_interval = 30
+
+# How often (in seconds) should pool usage stats be sent to StatsD (if statsd_on is enabled)
+#
+# Variable: AIRFLOW__SCHEDULER__POOL_METRICS_INTERVAL
+#
+pool_metrics_interval = 5.0
+
+# If the last scheduler heartbeat happened more than ``[scheduler] scheduler_health_check_threshold``
+# ago (in seconds), scheduler is considered unhealthy.
+# This is used by the health check in the **/health** endpoint and in ``airflow jobs check`` CLI
+# for SchedulerJob.
+#
+# Variable: AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_THRESHOLD
+#
+scheduler_health_check_threshold = 30
+
+# When you start a scheduler, airflow starts a tiny web server
+# subprocess to serve a health check if this is set to ``True``
+#
+# Variable: AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK
+#
+enable_health_check = False
+
+# When you start a scheduler, airflow starts a tiny web server
+# subprocess to serve a health check on this host
+#
+# Variable: AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_SERVER_HOST
+#
+scheduler_health_check_server_host = 0.0.0.0
+
+# When you start a scheduler, airflow starts a tiny web server
+# subprocess to serve a health check on this port
+#
+# Variable: AIRFLOW__SCHEDULER__SCHEDULER_HEALTH_CHECK_SERVER_PORT
+#
+scheduler_health_check_server_port = 8974
+
+# How often (in seconds) should the scheduler check for orphaned tasks and SchedulerJobs
+#
+# Variable: AIRFLOW__SCHEDULER__ORPHANED_TASKS_CHECK_INTERVAL
+#
+orphaned_tasks_check_interval = 300.0
+
+# Determines the directory where logs for the child processes of the scheduler will be stored
+#
+# Variable: AIRFLOW__SCHEDULER__CHILD_PROCESS_LOG_DIRECTORY
+#
+child_process_log_directory = /kaggle/working/BTC-USDT-ETL-Pipeline/airflow/logs/scheduler
+
+# Local task jobs periodically heartbeat to the DB. If the job has
+# not heartbeat in this many seconds, the scheduler will mark the
+# associated task instance as failed and will re-schedule the task.
+#
+# Variable: AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD
+#
+scheduler_zombie_task_threshold = 300
+
+# How often (in seconds) should the scheduler check for zombie tasks.
+#
+# Variable: AIRFLOW__SCHEDULER__ZOMBIE_DETECTION_INTERVAL
+#
+zombie_detection_interval = 10.0
+
+# Turn off scheduler catchup by setting this to ``False``.
+# Default behavior is unchanged and
+# Command Line Backfills still work, but the scheduler
+# will not do scheduler catchup if this is ``False``,
+# however it can be set on a per DAG basis in the
+# DAG definition (catchup)
+#
+# Variable: AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT
+#
+catchup_by_default = True
+
+# Setting this to ``True`` will make first task instance of a task
+# ignore depends_on_past setting. A task instance will be considered
+# as the first task instance of a task when there is no task instance
+# in the DB with an execution_date earlier than it., i.e. no manual marking
+# success will be needed for a newly added task to be scheduled.
+#
+# Variable: AIRFLOW__SCHEDULER__IGNORE_FIRST_DEPENDS_ON_PAST_BY_DEFAULT
+#
+ignore_first_depends_on_past_by_default = True
+
+# This changes the batch size of queries in the scheduling main loop.
+# This should not be greater than ``[core] parallelism``.
+# If this is too high, SQL query performance may be impacted by
+# complexity of query predicate, and/or excessive locking.
+# Additionally, you may hit the maximum allowable query length for your db.
+# Set this to 0 to use the value of ``[core] parallelism``
+#
+# Variable: AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY
+#
+max_tis_per_query = 16
+
+# Should the scheduler issue ``SELECT ... FOR UPDATE`` in relevant queries.
+# If this is set to ``False`` then you should not run more than a single
+# scheduler at once
+#
+# Variable: AIRFLOW__SCHEDULER__USE_ROW_LEVEL_LOCKING
+#
+use_row_level_locking = True
+
+# Max number of DAGs to create DagRuns for per scheduler loop.
+#
+# Variable: AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP
+#
+max_dagruns_to_create_per_loop = 10
+
+# How many DagRuns should a scheduler examine (and lock) when scheduling
+# and queuing tasks.
+#
+# Variable: AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE
+#
+max_dagruns_per_loop_to_schedule = 20
+
+# Should the Task supervisor process perform a "mini scheduler" to attempt to schedule more tasks of the
+# same DAG. Leaving this on will mean tasks in the same DAG execute quicker, but might starve out other
+# dags in some circumstances
+#
+# Variable: AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION
+#
+schedule_after_task_execution = True
+
+# The scheduler reads dag files to extract the airflow modules that are going to be used,
+# and imports them ahead of time to avoid having to re-do it for each parsing process.
+# This flag can be set to ``False`` to disable this behavior in case an airflow module needs
+# to be freshly imported each time (at the cost of increased DAG parsing time).
+#
+# Variable: AIRFLOW__SCHEDULER__PARSING_PRE_IMPORT_MODULES
+#
+parsing_pre_import_modules = True
+
+# The scheduler can run multiple processes in parallel to parse dags.
+# This defines how many processes will run.
+#
+# Variable: AIRFLOW__SCHEDULER__PARSING_PROCESSES
+#
+parsing_processes = 2
+
+# One of ``modified_time``, ``random_seeded_by_host`` and ``alphabetical``.
+# The scheduler will list and sort the dag files to decide the parsing order.
+# 
+# * ``modified_time``: Sort by modified time of the files. This is useful on large scale to parse the
+#   recently modified DAGs first.
+# * ``random_seeded_by_host``: Sort randomly across multiple Schedulers but with same order on the
+#   same host. This is useful when running with Scheduler in HA mode where each scheduler can
+#   parse different DAG files.
+# * ``alphabetical``: Sort by filename
+#
+# Variable: AIRFLOW__SCHEDULER__FILE_PARSING_SORT_MODE
+#
+file_parsing_sort_mode = modified_time
+
+# Whether the dag processor is running as a standalone process or it is a subprocess of a scheduler
+# job.
+#
+# Variable: AIRFLOW__SCHEDULER__STANDALONE_DAG_PROCESSOR
+#
+standalone_dag_processor = False
+
+# Only applicable if ``[scheduler] standalone_dag_processor`` is true and  callbacks are stored
+# in database. Contains maximum number of callbacks that are fetched during a single loop.
+#
+# Variable: AIRFLOW__SCHEDULER__MAX_CALLBACKS_PER_LOOP
+#
+max_callbacks_per_loop = 20
+
+# Only applicable if ``[scheduler] standalone_dag_processor`` is true.
+# Time in seconds after which dags, which were not updated by Dag Processor are deactivated.
+#
+# Variable: AIRFLOW__SCHEDULER__DAG_STALE_NOT_SEEN_DURATION
+#
+dag_stale_not_seen_duration = 600
+
+# Turn off scheduler use of cron intervals by setting this to ``False``.
+# DAGs submitted manually in the web UI or with trigger_dag will still run.
+#
+# Variable: AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE
+#
+use_job_schedule = True
+
+# Allow externally triggered DagRuns for Execution Dates in the future
+# Only has effect if schedule_interval is set to None in DAG
+#
+# Variable: AIRFLOW__SCHEDULER__ALLOW_TRIGGER_IN_FUTURE
+#
+allow_trigger_in_future = False
+
+# How often to check for expired trigger requests that have not run yet.
+#
+# Variable: AIRFLOW__SCHEDULER__TRIGGER_TIMEOUT_CHECK_INTERVAL
+#
+trigger_timeout_check_interval = 15
+
+# Amount of time a task can be in the queued state before being retried or set to failed.
+#
+# Variable: AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT
+#
+task_queued_timeout = 600.0
+
+# How often to check for tasks that have been in the queued state for
+# longer than ``[scheduler] task_queued_timeout``.
+#
+# Variable: AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT_CHECK_INTERVAL
+#
+task_queued_timeout_check_interval = 120.0
+
+# The run_id pattern used to verify the validity of user input to the run_id parameter when
+# triggering a DAG. This pattern cannot change the pattern used by scheduler to generate run_id
+# for scheduled DAG runs or DAG runs triggered without changing the run_id parameter.
+#
+# Variable: AIRFLOW__SCHEDULER__ALLOWED_RUN_ID_PATTERN
+#
+allowed_run_id_pattern = ^[A-Za-z0-9_.~:+-]+$
+
+# Whether to create DAG runs that span an interval or one single point in time for cron schedules, when
+# a cron string is provided to ``schedule`` argument of a DAG.
+# 
+# * ``True``: **CronDataIntervalTimetable** is used, which is suitable
+#   for DAGs with well-defined data interval. You get contiguous intervals from the end of the previous
+#   interval up to the scheduled datetime.
+# * ``False``: **CronTriggerTimetable** is used, which is closer to the behavior of cron itself.
+# 
+# Notably, for **CronTriggerTimetable**, the logical date is the same as the time the DAG Run will
+# try to schedule, while for **CronDataIntervalTimetable**, the logical date is the beginning of
+# the data interval, but the DAG Run will try to schedule at the end of the data interval.
+#
+# Variable: AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS
+#
+create_cron_data_intervals = True
+
+[triggerer]
+# How many triggers a single Triggerer will run at once, by default.
+#
+# Variable: AIRFLOW__TRIGGERER__DEFAULT_CAPACITY
+#
+default_capacity = 1000
+
+# How often to heartbeat the Triggerer job to ensure it hasn't been killed.
+#
+# Variable: AIRFLOW__TRIGGERER__JOB_HEARTBEAT_SEC
+#
+job_heartbeat_sec = 5
+
+# If the last triggerer heartbeat happened more than ``[triggerer] triggerer_health_check_threshold``
+# ago (in seconds), triggerer is considered unhealthy.
+# This is used by the health check in the **/health** endpoint and in ``airflow jobs check`` CLI
+# for TriggererJob.
+#
+# Variable: AIRFLOW__TRIGGERER__TRIGGERER_HEALTH_CHECK_THRESHOLD
+#
+triggerer_health_check_threshold = 30
+
+[kerberos]
+# Location of your ccache file once kinit has been performed.
+#
+# Variable: AIRFLOW__KERBEROS__CCACHE
+#
+ccache = /tmp/airflow_krb5_ccache
+
+# gets augmented with fqdn
+#
+# Variable: AIRFLOW__KERBEROS__PRINCIPAL
+#
+principal = airflow
+
+# Determines the frequency at which initialization or re-initialization processes occur.
+#
+# Variable: AIRFLOW__KERBEROS__REINIT_FREQUENCY
+#
+reinit_frequency = 3600
+
+# Path to the kinit executable
+#
+# Variable: AIRFLOW__KERBEROS__KINIT_PATH
+#
+kinit_path = kinit
+
+# Designates the path to the Kerberos keytab file for the Airflow user
+#
+# Variable: AIRFLOW__KERBEROS__KEYTAB
+#
+keytab = airflow.keytab
+
+# Allow to disable ticket forwardability.
+#
+# Variable: AIRFLOW__KERBEROS__FORWARDABLE
+#
+forwardable = True
+
+# Allow to remove source IP from token, useful when using token behind NATted Docker host.
+#
+# Variable: AIRFLOW__KERBEROS__INCLUDE_IP
+#
+include_ip = True
+
+[sensors]
+# Sensor default timeout, 7 days by default (7 * 24 * 60 * 60).
+#
+# Variable: AIRFLOW__SENSORS__DEFAULT_TIMEOUT
+#
+default_timeout = 604800
+
+[usage_data_collection]
+# Airflow integrates `Scarf <https://about.scarf.sh/>`__ to collect basic platform and usage data
+# during operation. This data assists Airflow maintainers in better understanding how Airflow is used.
+# Insights gained from this telemetry are critical for prioritizing patches, minor releases, and
+# security fixes. Additionally, this information supports key decisions related to the development road map.
+# Check the FAQ doc for more information on what data is collected.
+# 
+# Deployments can opt-out of analytics by setting the ``enabled`` option
+# to ``False``, or the ``SCARF_ANALYTICS=false`` environment variable.
+# Individual users can easily opt-out of analytics in various ways documented in the
+# `Scarf Do Not Track docs <https://docs.scarf.sh/gateway/#do-not-track>`__.
+
+# Enable or disable usage data collection and sending.
+#
+# Variable: AIRFLOW__USAGE_DATA_COLLECTION__ENABLED
+#
+enabled = True
+
+[common.io]
+# Common IO configuration section
+
+# Path to a location on object storage where XComs can be stored in url format.
+#
+# Example: xcom_objectstorage_path = s3://conn_id@bucket/path
+#
+# Variable: AIRFLOW__COMMON.IO__XCOM_OBJECTSTORAGE_PATH
+#
+xcom_objectstorage_path = 
+
+# Threshold in bytes for storing XComs in object storage. -1 means always store in the
+# database. 0 means always store in object storage. Any positive number means
+# it will be stored in object storage if the size of the value is greater than the threshold.
+#
+# Example: xcom_objectstorage_threshold = 1000000
+#
+# Variable: AIRFLOW__COMMON.IO__XCOM_OBJECTSTORAGE_THRESHOLD
+#
+xcom_objectstorage_threshold = -1
+
+# Compression algorithm to use when storing XComs in object storage. Supported algorithms
+# are a.o.: snappy, zip, gzip, bz2, and lzma. If not specified, no compression will be used.
+# Note that the compression algorithm must be available in the Python installation (e.g.
+# python-snappy for snappy). Zip, gz, bz2 are available by default.
+#
+# Example: xcom_objectstorage_compression = gz
+#
+# Variable: AIRFLOW__COMMON.IO__XCOM_OBJECTSTORAGE_COMPRESSION
+#
+xcom_objectstorage_compression = 
+
+[fab]
+# This section contains configs specific to FAB provider.
+
+# Boolean for enabling rate limiting on authentication endpoints.
+#
+# Variable: AIRFLOW__FAB__AUTH_RATE_LIMITED
+#
+auth_rate_limited = True
+
+# Rate limit for authentication endpoints.
+#
+# Variable: AIRFLOW__FAB__AUTH_RATE_LIMIT
+#
+auth_rate_limit = 5 per 40 second
+
+# Update FAB permissions and sync security manager roles
+# on webserver startup
+#
+# Variable: AIRFLOW__FAB__UPDATE_FAB_PERMS
+#
+update_fab_perms = True
+
+[imap]
+# Options for IMAP provider.
+
+# ssl_context = 
+

airflow/airflow.db

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33e2a2d0de019f573198b2bd423a7d642c3f01f321c8527c3e8f04b9ee73d60a
+size 1282048

airflow/dags/new6.py

@@ -0,0 +1,147 @@
+from airflow import DAG
+from airflow.operators.python import PythonOperator
+from datetime import datetime, timedelta, timezone
+import os
+import sys
+
+# Add project root to path
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
+
+from components.btcusdt_ingest_data import crawl_data_from_sources
+from components.datalake_cr import up_to_minio
+from components.process_data import extract_from_minio, transform_financial_data
+from components.duckdb_api import push_to_duckdb
+from components.duckdb2csv import duckdb_to_csv
+from components.model.training import train_lstm_model
+from components.model.evaluation import metric_and_predict_lstm_model
+from components.utils.file_utils import (
+    load_extract_config, 
+    define_server_filenames, 
+    load_pipeline_config
+)
+
+# DAG default arguments
+default_args = {
+    'owner': 'airflow',
+    'start_date': datetime(2025, 10, 7, 20, 0),
+    'retries': 1,
+    'retry_delay': timedelta(minutes=5),
+}
+
+# Define DAGs
+dag_1 = DAG('crawl_to_minio', default_args=default_args, 
+            schedule_interval='@monthly', max_active_runs=1, catchup=False)
+dag_2 = DAG('etl_to_duckdb', default_args=default_args, 
+            schedule_interval='@monthly', max_active_runs=1, catchup=False)
+dag_3 = DAG('lstm_forecast', default_args=default_args, 
+            schedule_interval='@monthly', max_active_runs=1, catchup=False)
+dag_4 = DAG('duckdb_to_csv_export', default_args=default_args, 
+            schedule_interval='@monthly', max_active_runs=1, catchup=False)
+
+# Load pipeline configuration
+pipeline_config = load_pipeline_config()
+
+# DAG 1: Crawl to MinIO
+download_binance_csv = PythonOperator(
+    task_id='download_binance_csv',
+    python_callable=crawl_data_from_sources,
+    dag=dag_1
+)
+
+extract_filenames_task = PythonOperator(
+    task_id='extract_filenames',
+    python_callable=define_server_filenames,
+    dag=dag_1
+)
+
+upload_to_minio_storage = PythonOperator(
+    task_id='upload_to_minio',
+    python_callable=up_to_minio,
+    op_kwargs={
+        'client_files': '{{ ti.xcom_pull(task_ids="download_binance_csv") }}',
+        'server_files': '{{ ti.xcom_pull(task_ids="extract_filenames") }}',
+        'bucket_name': pipeline_config['minio']['bucket_name']
+    },
+    dag=dag_1
+)
+
+# DAG 2: MinIO to DuckDB
+extract_data = PythonOperator(
+    task_id='extract_data',
+    python_callable=extract_from_minio,
+    op_kwargs={
+        'bucket_name': pipeline_config['minio']['bucket_name'],
+        'file_names': load_extract_config("extract_data.yml")["files"]
+    },
+    dag=dag_2
+)
+
+transform_data = PythonOperator(
+    task_id='transform_data',
+    python_callable=transform_financial_data,
+    op_kwargs={
+        'parquet_file_paths': '{{ ti.xcom_pull(task_ids="extract_data") }}',
+        'temp_parquet_path': pipeline_config['paths']['temp_parquet_path'],
+        'output_parquet_path': pipeline_config['paths']['output_parquet_path']
+    },
+    dag=dag_2
+)
+
+push_to_warehouse = PythonOperator(
+    task_id='export_duckdb',
+    python_callable=push_to_duckdb,
+    op_kwargs={
+        'duckdb_path': pipeline_config['paths']['duckdb_path'],
+        'parquet_path': '{{ ti.xcom_pull(task_ids="transform_data") }}'
+    },
+    dag=dag_2
+)
+
+# DAG 3: LSTM Forecasting
+train_lstm = PythonOperator(
+    task_id='train_lstm_model',
+    python_callable=train_lstm_model,
+    dag=dag_3
+)
+
+# metric_and_predict_lstm = PythonOperator(
+#     task_id='metric_and_predict_lstm',
+#     python_callable=metric_and_predict_lstm_model,
+#     provide_context=True,
+#     dag=dag_3
+# )
+
+metric_and_predict_lstm = PythonOperator(
+    task_id='metric_and_predict_lstm',
+    python_callable=metric_and_predict_lstm_model,
+    op_kwargs={
+        'train_result': '{{ ti.xcom_pull(task_ids="train_lstm_model") }}'
+    },
+    provide_context=True,  # Still needed for Jinja templating in op_kwargs
+    dag=dag_3
+)
+
+# metric_and_predict_lstm = PythonOperator(
+#     task_id='metric_and_predict_lstm',
+#     python_callable=metric_and_predict_lstm_model,
+#     op_kwargs={'ti': '{{ ti }}'},  # Explicitly pass task instance
+#     provide_context=True,  # Ensure context is provided for XCom
+#     dag=dag_3
+# )
+
+# DAG 4: DuckDB to CSV
+export_duckdb_to_csv = PythonOperator(
+    task_id='export_duckdb_to_csv',
+    python_callable=duckdb_to_csv,
+    op_kwargs={
+        'duckdb_path': pipeline_config['paths']['duckdb_path'],
+        'output_csv_path': pipeline_config['paths']['output_csv_path']
+    },
+    dag=dag_4
+)
+
+# Dependencies
+download_binance_csv >> extract_filenames_task >> upload_to_minio_storage
+extract_data >> transform_data >> push_to_warehouse
+train_lstm >> metric_and_predict_lstm
+export_duckdb_to_csv

airflow/webserver_config.py

@@ -0,0 +1,132 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Default configuration for the Airflow webserver."""
+
+from __future__ import annotations
+
+import os
+
+from flask_appbuilder.const import AUTH_DB
+
+# from airflow.www.fab_security.manager import AUTH_LDAP
+# from airflow.www.fab_security.manager import AUTH_OAUTH
+# from airflow.www.fab_security.manager import AUTH_OID
+# from airflow.www.fab_security.manager import AUTH_REMOTE_USER
+
+
+basedir = os.path.abspath(os.path.dirname(__file__))
+
+# Flask-WTF flag for CSRF
+WTF_CSRF_ENABLED = True
+WTF_CSRF_TIME_LIMIT = None
+
+# ----------------------------------------------------
+# AUTHENTICATION CONFIG
+# ----------------------------------------------------
+# For details on how to set up each of the following authentication, see
+# http://flask-appbuilder.readthedocs.io/en/latest/security.html# authentication-methods
+# for details.
+
+# The authentication type
+# AUTH_OID : Is for OpenID
+# AUTH_DB : Is for database
+# AUTH_LDAP : Is for LDAP
+# AUTH_REMOTE_USER : Is for using REMOTE_USER from web server
+# AUTH_OAUTH : Is for OAuth
+AUTH_TYPE = AUTH_DB
+
+# Uncomment to setup Full admin role name
+# AUTH_ROLE_ADMIN = 'Admin'
+
+# Uncomment and set to desired role to enable access without authentication
+# AUTH_ROLE_PUBLIC = 'Viewer'
+
+# Will allow user self registration
+# AUTH_USER_REGISTRATION = True
+
+# The recaptcha it's automatically enabled for user self registration is active and the keys are necessary
+# RECAPTCHA_PRIVATE_KEY = PRIVATE_KEY
+# RECAPTCHA_PUBLIC_KEY = PUBLIC_KEY
+
+# Config for Flask-Mail necessary for user self registration
+# MAIL_SERVER = 'smtp.gmail.com'
+# MAIL_USE_TLS = True
+# MAIL_USERNAME = 'yourappemail@gmail.com'
+# MAIL_PASSWORD = 'passwordformail'
+# MAIL_DEFAULT_SENDER = 'sender@gmail.com'
+
+# The default user self registration role
+# AUTH_USER_REGISTRATION_ROLE = "Public"
+
+# When using OAuth Auth, uncomment to setup provider(s) info
+# Google OAuth example:
+# OAUTH_PROVIDERS = [{
+#   'name':'google',
+#     'token_key':'access_token',
+#     'icon':'fa-google',
+#         'remote_app': {
+#             'api_base_url':'https://www.googleapis.com/oauth2/v2/',
+#             'client_kwargs':{
+#                 'scope': 'email profile'
+#             },
+#             'access_token_url':'https://accounts.google.com/o/oauth2/token',
+#             'authorize_url':'https://accounts.google.com/o/oauth2/auth',
+#             'request_token_url': None,
+#             'client_id': GOOGLE_KEY,
+#             'client_secret': GOOGLE_SECRET_KEY,
+#         }
+# }]
+
+# When using LDAP Auth, setup the ldap server
+# AUTH_LDAP_SERVER = "ldap://ldapserver.new"
+
+# When using OpenID Auth, uncomment to setup OpenID providers.
+# example for OpenID authentication
+# OPENID_PROVIDERS = [
+#    { 'name': 'Yahoo', 'url': 'https://me.yahoo.com' },
+#    { 'name': 'AOL', 'url': 'http://openid.aol.com/<username>' },
+#    { 'name': 'Flickr', 'url': 'http://www.flickr.com/<username>' },
+#    { 'name': 'MyOpenID', 'url': 'https://www.myopenid.com' }]
+
+# ----------------------------------------------------
+# Theme CONFIG
+# ----------------------------------------------------
+# Flask App Builder comes up with a number of predefined themes
+# that you can use for Apache Airflow.
+# http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes
+# Please make sure to remove "navbar_color" configuration from airflow.cfg
+# in order to fully utilize the theme. (or use that property in conjunction with theme)
+# APP_THEME = "bootstrap-theme.css"  # default bootstrap
+# APP_THEME = "amelia.css"
+# APP_THEME = "cerulean.css"
+# APP_THEME = "cosmo.css"
+# APP_THEME = "cyborg.css"
+# APP_THEME = "darkly.css"
+# APP_THEME = "flatly.css"
+# APP_THEME = "journal.css"
+# APP_THEME = "lumen.css"
+# APP_THEME = "paper.css"
+# APP_THEME = "readable.css"
+# APP_THEME = "sandstone.css"
+# APP_THEME = "simplex.css"
+# APP_THEME = "slate.css"
+# APP_THEME = "solar.css"
+# APP_THEME = "spacelab.css"
+# APP_THEME = "superhero.css"
+# APP_THEME = "united.css"
+# APP_THEME = "yeti.css"

ckpts/.gitignore

@@ -0,0 +1,2 @@
+*.keras
+*.pkl

ckpts/model_2025-10-28-11-33-51-(+07).h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86e23253135898780db99f96e8ff9625297d3d3389bd5f928a024f49c59e547c
+size 2626736

ckpts/scaler_2025-10-28-11-33-51-(+07).pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e9ec47a4549e291a30cbac7225656f56d65069b62ead6d139323ec28a3933f3
+size 523

components/btcusdt_ingest_data.py

@@ -0,0 +1,157 @@
+import requests
+import zipfile
+import io
+import os
+import yaml
+import pandas as pd
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+def add_column_headers(csv_data: bytes, expected_columns: list) -> bytes:
+
+    try:
+        # Read CSV data into a DataFrame without headers
+        df = pd.read_csv(io.BytesIO(csv_data), header=None)
+        
+        # Validate column count
+        if len(df.columns) != len(expected_columns):
+            raise ValueError(
+                f"CSV has {len(df.columns)} columns, expected {len(expected_columns)}"
+            )
+        
+        # Assign column names
+        df.columns = expected_columns
+        
+        # Convert DataFrame back to CSV bytes with headers
+        csv_buffer = io.StringIO()
+        df.to_csv(csv_buffer, index=False)
+        return csv_buffer.getvalue().encode('utf-8')
+    
+    except pd.errors.ParserError as e:
+        raise Exception(f"Failed to parse CSV data: {e}")
+    except Exception as e:
+        raise Exception(f"Failed to process CSV with headers: {e}")
+
+def download_and_extract_binance_data(url: str, output_path: str = "temp/input.csv") -> pd.DataFrame:
+    expected_columns = [
+        "Open time", "Open", "High", "Low", "Close", "Volume",
+        "Close time", "Quote asset volume", "Number of trades",
+        "Taker buy base asset volume", "Taker buy quote asset volume", "Ignore"
+    ]
+    
+    try:
+        response = requests.get(url, timeout=10)
+        response.raise_for_status()
+
+        with io.BytesIO(response.content) as zip_file:
+            with zipfile.ZipFile(zip_file, 'r') as zip_ref:
+                csv_file_name = zip_ref.namelist()[0]
+                csv_data = zip_ref.read(csv_file_name)
+                # csv_data = add_column_headers(csv_data, expected_columns)
+                with open(output_path, 'wb') as output_file:
+                    output_file.write(csv_data)
+
+        print(f"Successfully downloaded and extracted data to {output_path}")
+
+        # Load the saved CSV into a DataFrame
+        df = pd.read_csv(output_path)
+        
+        # Print the first few rows to inspect the data
+        # print("First few rows of the CSV:")
+        # print(df.head())
+        # print("Open time and Close time dtypes:", df["Open time"].dtype, df["Close time"].dtype)
+
+        # Check for invalid or out-of-range timestamps
+        # try:
+        #     df["Open time"] = pd.to_datetime(df["Open time"], unit="ms", errors="coerce")
+        #     df["Close time"] = pd.to_datetime(df["Close time"], unit="ms", errors="coerce")
+        # except Exception as e:
+        #     print(f"Timestamp conversion error: {e}")
+        #     print("Invalid Open time values:", df[df["Open time"].isna()])
+        #     print("Invalid Close time values:", df[df["Close time"].isna()])
+        #     raise
+
+        print("CSV structure validated and headers added successfully")
+        return df
+
+    except requests.RequestException as e:
+        raise Exception(f"Failed to download file from {url}: {e}")
+    except zipfile.BadZipFile as e:
+        raise Exception(f"Invalid ZIP file: {e}")
+    except IOError as e:
+        raise Exception(f"Failed to write to {output_path}: {e}")
+    except Exception as e:
+        raise Exception(f"Failed during processing: {e}")
+
+def crawl_data_from_sources():
+    try:
+        # Load data sources configuration
+        sources_path = Path("configs/data_sources.yml")
+        with open(sources_path, 'r') as file:
+            data_sources = yaml.safe_load(file)
+            if not data_sources or not isinstance(data_sources, list):
+                raise ValueError("Invalid or empty data_sources configuration file")
+
+        # Load data limit configuration
+        limits_path = Path("configs/data_limit.yml")
+        with open(limits_path, 'r') as file:
+            data_limits = yaml.safe_load(file)
+            if not data_limits or not isinstance(data_limits, list):
+                raise ValueError("Invalid or empty data_limit configuration file")
+
+        # Create a dictionary of limits for each data source
+        limits_dict = {limit['name']: limit['limit'] for limit in data_limits if isinstance(limit, dict) and 'name' in limit and 'limit' in limit}
+        
+        output_paths = []
+        # Process each data source
+        for data_source in data_sources:
+            try:
+                if not isinstance(data_source, dict) or 'name' not in data_source or 'url' not in data_source:
+                    print(f"Skipping invalid data source: {data_source}")
+                    continue
+
+                # Get allowed periods for this data source
+                allowed_periods = limits_dict.get(data_source['name'], [])
+
+                # Process each allowed period
+                for period in allowed_periods:
+                    try:
+                        # Validate period format
+                        try:
+                            datetime.strptime(period, '%Y-%m')
+                        except ValueError:
+                            print(f"Invalid period format for {data_source['name']}: {period}")
+                            continue
+
+                        # Construct unique output path
+                        output_path = f"temp/{data_source['name']}-{period}.csv"
+
+                        # Create the directory if it doesn't exist
+                        os.makedirs(os.path.dirname(output_path), exist_ok=True)
+
+                        # Construct URL
+                        url = f"{data_source['url']}{data_source['name']}-{period}.zip"
+
+                        # Download and extract data
+                        download_and_extract_binance_data(url, output_path)
+
+                    except Exception as e:
+                        print(f"Failed to process period {period} for {data_source['name']}: {e}")
+                        continue
+                    output_paths.append(output_path)
+                
+            except Exception as e:
+                print(f"Failed to process data source {data_source.get('name', 'unknown')}: {e}")
+                continue
+            
+        return output_paths
+    
+    except (yaml.YAMLError, FileNotFoundError) as e:
+        raise Exception(f"Failed to load configuration: {e}")
+    except Exception as e:
+        raise Exception(f"Script execution failed: {e}")
+
+if __name__ == "__main__":
+    out_paths = crawl_data_from_sources()
+    print("Downloaded files:", out_paths)

components/datalake_cr.py

@@ -0,0 +1,44 @@
+import os
+import sys
+import ast
+
+# Add the project root directory to the Python path
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+from minio_api.client import sign_in, upload_file, download_file, create_bucket, list_objects
+
+def up_to_minio(client_files, server_files, bucket_name="minio-ngrok-bucket"):
+    """Upload the local CSV file to MinIO."""
+    # Convert stringified lists to actual lists if necessary
+    if isinstance(client_files, str):
+        try:
+            client_files = ast.literal_eval(client_files)
+        except (ValueError, SyntaxError) as e:
+            raise ValueError(f"Failed to parse client_files as a list: {client_files}, error: {e}")
+    if isinstance(server_files, str):
+        try:
+            server_files = ast.literal_eval(server_files)
+        except (ValueError, SyntaxError) as e:
+            raise ValueError(f"Failed to parse server_files as a list: {server_files}, error: {e}")
+             
+    for client_file, server_file in zip(client_files, server_files):
+        # Check if local file exists
+        if not os.path.exists(client_file):
+            raise FileNotFoundError(f"Local file {client_file} does not exist")
+        
+        minio_client = sign_in()
+        # Create bucket
+        create_bucket(minio_client, bucket_name)
+
+        # Upload file
+        print("Uploading file to MinIO:", client_file, "as", server_file)
+        upload_file(minio_client, bucket_name, client_file, server_file)
+
+if __name__ == "__main__":
+    # Example usage
+    try:
+        up_to_minio(["temp/BTCUSDT-1s-2025-09.csv"], 
+                    ["BTCUSDT-1s-2025-09.csv"], 
+                    "minio-ngrok-bucket")
+        print("File uploaded successfully.")
+    except Exception as e:
+        print(f"Error uploading file: {e}")

components/delete_lstm_predict.py

@@ -0,0 +1,56 @@
+import pandas as pd
+import numpy as np
+from components.model.model_utils import create_sequences
+from model import build_lstm_model
+from sklearn.preprocessing import MinMaxScaler
+from tensorflow import keras
+
+df = pd.read_csv('/tmp/BTCUSDT-1s-2024-05.csv', header=None)
+df.columns = [
+    "open_time", "open", "high", "low", "close", "volume",
+    "close_time", "quote_asset_volume", "number_of_trades",
+    "taker_buy_base_asset_volume", "taker_buy_quote_asset_volume", "ignore"
+]
+
+prices = df['close'].astype(float).values.reshape(-1, 1)
+scaler = MinMaxScaler()
+prices_scaled = scaler.fit_transform(prices)
+
+split_idx = int(len(prices_scaled) * 0.8)
+test_data = prices_scaled[split_idx - 60:]
+
+seq_length = 60
+X_test, y_test = create_sequences(test_data, seq_length)
+
+model = build_lstm_model(seq_length)
+model.load_weights('./ckpts/lstm_checkpoint.keras')
+
+loss = model.evaluate(X_test, y_test, verbose=0)
+print(f"Test loss: {loss}")
+
+last_seq = prices_scaled[-seq_length:]
+next_pred = model.predict(last_seq.reshape(1, seq_length, 1))
+next_price = scaler.inverse_transform(next_pred)
+print(f"Predicted next close price: {next_price[0][0]}")
+checkpoint_cb = keras.callbacks.ModelCheckpoint(
+    './ckpts/lstm_checkpoint.keras', save_best_only=True, monitor='val_loss'
+)
+
+# Train
+model.fit(
+    X_train, y_train,
+    epochs=5,
+    batch_size=64,
+    validation_data=(X_test, y_test),
+    callbacks=[checkpoint_cb],
+    verbose=2
+)
+
+# Evaluate and predict
+loss = model.evaluate(X_test, y_test, verbose=0)
+print(f"Test loss: {loss}")
+
+last_seq = prices_scaled[-seq_length:]
+next_pred = model.predict(last_seq.reshape(1, seq_length, 1))
+next_price = scaler.inverse_transform(next_pred)
+print(f"Predicted next close price: {next_price[0][0]}")

components/delete_lstm_train.py

@@ -0,0 +1,42 @@
+import pandas as pd
+import numpy as np
+from components.model.model_utils import create_sequences
+from model import build_lstm_model
+from sklearn.preprocessing import MinMaxScaler
+from tensorflow import keras
+import os
+
+df = pd.read_csv('/tmp/BTCUSDT-1s-2024-05.csv', header=None)
+df.columns = [
+    "open_time", "open", "high", "low", "close", "volume",
+    "close_time", "quote_asset_volume", "number_of_trades",
+    "taker_buy_base_asset_volume", "taker_buy_quote_asset_volume", "ignore"
+]
+
+prices = df['close'].astype(float).values.reshape(-1, 1)
+scaler = MinMaxScaler()
+prices_scaled = scaler.fit_transform(prices)
+
+split_idx = int(len(prices_scaled) * 0.8)
+train_data = prices_scaled[:split_idx]
+test_data = prices_scaled[split_idx - 60:]
+
+seq_length = 60
+X_train, y_train = create_sequences(train_data, seq_length)
+X_test, y_test = create_sequences(test_data, seq_length)
+
+model = build_lstm_model(seq_length)
+
+os.makedirs('./ckpts', exist_ok=True)
+checkpoint_cb = keras.callbacks.ModelCheckpoint(
+    './ckpts/lstm_checkpoint.keras', save_best_only=True, monitor='val_loss'
+)
+
+model.fit(
+    X_train, y_train,
+    epochs=5,
+    batch_size=64,
+    validation_data=(X_test, y_test),
+    callbacks=[checkpoint_cb],
+    verbose=2
+)

components/delete_model.py

@@ -0,0 +1,10 @@
+from tensorflow import keras
+
+def build_lstm_model(seq_length):
+    model = keras.Sequential([
+        keras.layers.LSTM(50, return_sequences=True, input_shape=(seq_length, 1)),
+        keras.layers.LSTM(50),
+        keras.layers.Dense(1)
+    ])
+    model.compile(optimizer='adam', loss='mse')
+    return model

components/duckdb2csv.py

@@ -0,0 +1,22 @@
+import duckdb
+import pandas as pd
+import logging
+
+def duckdb_to_csv(duckdb_path, output_csv_path):
+    try:
+        # Connect to DuckDB
+        con = duckdb.connect(duckdb_path)
+        # Query data
+        df = con.execute("SELECT * FROM aggregated_financial_data").fetchdf()
+        if df.empty:
+            raise ValueError("No data found in table 'aggregated_financial_data'")
+        # Save to CSV
+        df.to_csv(output_csv_path, index=False)
+        logging.info(f"Successfully exported data to {output_csv_path}")
+    except Exception as e:
+        logging.error(f"Error in duckdb_to_csv: {str(e)}")
+        raise
+
+if __name__ == "__main__":
+    duckdb_to_csv("duckdb_databases/financial_data.db",
+                      "analytics/financial_data.csv")

components/duckdb_api.py

@@ -0,0 +1,68 @@
+import os
+import duckdb
+import shutil
+import pandas as pd
+
+def push_to_duckdb(duckdb_path, parquet_path, temp_parquet_path="temp/duckdb_temp_parquet"):
+    """
+    Push the aggregated data from a Parquet directory to DuckDB.
+    
+    Args:
+        duckdb_path (str): Path to the DuckDB database file
+        parquet_path (str): Path to the Parquet directory containing the aggregated data
+        temp_parquet_path (str): Temporary path for storing Parquet files
+    """
+    # Validate input parquet_path
+    if not isinstance(parquet_path, str):
+        raise ValueError(f"parquet_path must be a string, got {type(parquet_path)}: {parquet_path}")
+    if not os.path.exists(parquet_path):
+        raise FileNotFoundError(f"Parquet directory does not exist at {parquet_path}")
+    if not os.path.isdir(parquet_path):
+        raise ValueError(f"parquet_path must be a directory, got a file at {parquet_path}")
+
+    # Ensure the temporary directory is clean before copying
+    if os.path.exists(temp_parquet_path):
+        shutil.rmtree(temp_parquet_path)
+    os.makedirs(temp_parquet_path, exist_ok=True)
+
+    # Copy the input Parquet directory to the temporary directory
+    try:
+        shutil.copytree(parquet_path, temp_parquet_path, dirs_exist_ok=True)
+        print(f"Copied Parquet directory from {parquet_path} to {temp_parquet_path}")
+    except Exception as e:
+        raise RuntimeError(f"Failed to copy Parquet directory from {parquet_path} to {temp_parquet_path}: {e}")
+
+    # Connect to DuckDB
+    directory = os.path.dirname(duckdb_path)
+    if directory and not os.path.exists(directory):
+        os.makedirs(directory)
+    
+    con = duckdb.connect(duckdb_path)
+  
+    # Create or replace the table in DuckDB by reading the Parquet files
+    try:
+        con.execute(f"""
+            CREATE OR REPLACE TABLE aggregated_financial_data AS
+            SELECT * FROM parquet_scan('{temp_parquet_path}/*.parquet')
+        """)
+        print(f"Successfully loaded data into DuckDB table from {temp_parquet_path}")
+    except Exception as e:
+        raise RuntimeError(f"Failed to load Parquet files into DuckDB: {e}")
+    finally:
+        con.close()
+    
+    # Clean up temporary Parquet directory
+    if os.path.exists(temp_parquet_path):
+        shutil.rmtree(temp_parquet_path)
+        print(f"Cleaned up temporary directory {temp_parquet_path}")
+
+if __name__ == "__main__":
+    from process_data import process_financial_data
+    duckdb_path = "duckdb_databases/financial_data.db"
+    parquet_path = process_financial_data()
+    
+    try:
+        push_to_duckdb(duckdb_path, parquet_path)
+    except Exception as e:
+        print(f"Error pushing to DuckDB: {e}")
+    

components/model/data_utils.py

@@ -0,0 +1,90 @@
+import os
+import logging
+import tensorflow as tf
+import pandas as pd
+import pyarrow.parquet as pq
+from sklearn.preprocessing import MinMaxScaler
+
+# Configure logging
+logger = logging.getLogger(__name__)
+
+def create_data_loader(parquet_paths: list, scaler: MinMaxScaler, seq_length: int, batch_size: int) -> tf.data.Dataset:
+    """Create a tf.data.Dataset from Parquet files for LSTM training or evaluation.
+
+    Args:
+        parquet_paths (list): List of paths to Parquet files.
+        scaler (MinMaxScaler): Scaler fitted on the data.
+        seq_length (int): Length of input sequences.
+        batch_size (int): Batch size for the dataset.
+
+    Returns:
+        tf.data.Dataset: Dataset yielding (sequence, target) pairs with shapes (batch_size, seq_length, 1) and (batch_size, 1).
+
+    Raises:
+        ValueError: If inputs are invalid or no valid data is found.
+    """
+    if not parquet_paths:
+        logger.error("No parquet paths provided")
+        raise ValueError("parquet_paths cannot be empty")
+    if not isinstance(scaler, MinMaxScaler):
+        logger.error("Invalid scaler provided")
+        raise ValueError("scaler must be an instance of MinMaxScaler")
+    if not isinstance(seq_length, int) or seq_length <= 0:
+        logger.error(f"Invalid seq_length: {seq_length}")
+        raise ValueError("seq_length must be a positive integer")
+    if not isinstance(batch_size, int) or batch_size <= 0:
+        logger.error(f"Invalid batch_size: {batch_size}")
+        raise ValueError("batch_size must be a positive integer")
+
+    total_sequences = 0
+    def _scaled_generator():
+        nonlocal total_sequences
+        for path in parquet_paths:
+            if not os.path.exists(path):
+                logger.warning(f"Parquet file not found, skipping: {path}")
+                continue
+            try:
+                file_size = os.path.getsize(path) / (1024 * 1024)  # Size in MB
+                if file_size < 100:  # Load small files into memory
+                    df = pd.read_parquet(path, columns=['Close'])
+                    logger.debug(f"Loaded {path} into memory, size: {file_size:.2f} MB")
+                    if 'Close' not in df.columns or df['Close'].isna().any():
+                        logger.warning(f"Invalid or missing 'Close' column in {path}")
+                        continue
+                    prices = df['Close'].astype('float32').values.reshape(-1, 1)
+                    if prices.size <= seq_length:
+                        logger.warning(f"File {path} has {prices.size} rows, insufficient for seq_length {seq_length}")
+                        continue
+                    scaled = scaler.transform(prices)
+                    for j in range(len(scaled) - seq_length):
+                        total_sequences += 1
+                        yield scaled[j:j + seq_length], scaled[j + seq_length]
+                else:
+                    parquet_file = pq.ParquetFile(path)
+                    for batch in parquet_file.iter_batches(batch_size=10_000, columns=['Close']):
+                        chunk = batch.to_pandas()
+                        if 'Close' not in chunk.columns or chunk['Close'].isna().any():
+                            logger.warning(f"Invalid or missing 'Close' column in {path}")
+                            continue
+                        prices = chunk['Close'].astype('float32').values.reshape(-1, 1)
+                        scaled = scaler.transform(prices)
+                        logger.debug(f"Processing batch from {path}, scaled shape: {scaled.shape}")
+                        for j in range(len(scaled) - seq_length):
+                            total_sequences += 1
+                            yield scaled[j:j + seq_length], scaled[j + seq_length]
+            except Exception as e:
+                logger.error(f"Error processing parquet file {path}: {e}")
+                continue
+
+        if total_sequences == 0:
+            logger.error("No valid sequences generated from any Parquet file")
+            raise ValueError("No valid sequences generated from any Parquet file")
+
+    dataset = tf.data.Dataset.from_generator(
+        _scaled_generator,
+        output_types=(tf.float32, tf.float32),
+        output_shapes=((seq_length, 1), (1,))
+    ).batch(batch_size).prefetch(tf.data.AUTOTUNE)
+    
+    logger.info(f"Created data loader with seq_length={seq_length}, batch_size={batch_size}, total_sequences={total_sequences}")
+    return dataset

components/model/evaluation.py

@@ -0,0 +1,239 @@
+import os
+import logging
+import pandas as pd
+import numpy as np
+from sklearn.preprocessing import MinMaxScaler
+import pickle
+from sklearn.metrics import (
+    mean_squared_error, 
+    mean_absolute_error, 
+    mean_absolute_percentage_error
+)
+from typing import Dict, List, Tuple
+from datetime import datetime, timezone
+import tensorflow as tf
+import sys
+import ast
+
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
+from components.utils.file_utils import load_extract_config, get_parquet_file_names
+from components.model.model_utils import build_model_from_config
+from components.model.data_utils import create_data_loader
+from components.utils.utils import parse_timezone
+
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s %(levelname)s: %(message)s',
+    datefmt='%Y-%m-%d %H:%M:%S %Z'
+)
+logger = logging.getLogger(__name__)
+
+# def model_evaluate(model, scaler: MinMaxScaler, ds: tf.data.Dataset) -> Tuple[float, float]:
+#     """Evaluate a model on a dataset and return RMSE and MAE.
+
+#     Args:
+#         model: Trained Keras model.
+#         scaler (MinMaxScaler): Scaler used for data normalization.
+#         ds (tf.data.Dataset): Dataset to evaluate on.
+
+#     Returns:
+#         Tuple[float, float]: RMSE and MAE metrics.
+#     """
+#     y_true, y_pred = [], []
+#     for X, y in ds:
+#         pred = model.predict(X, verbose=2)
+#         y_true.append(y.numpy())
+#         y_pred.append(pred)
+#     y_true = np.concatenate(y_true)
+#     y_pred = np.concatenate(y_pred)
+#     y_true_orig = scaler.inverse_transform(y_true)
+#     y_pred_orig = scaler.inverse_transform(y_pred)
+#     return (np.sqrt(mean_squared_error(y_true_orig, y_pred_orig)), 
+#             mean_absolute_error(y_true_orig, y_pred_orig))
+
+def model_evaluate(model, scaler: MinMaxScaler, ds: tf.data.Dataset) -> Tuple[float, float]:
+    """Evaluate a model on a dataset and return RMSE and MAE.
+
+    Args:
+        model: Trained Keras model.
+        scaler (MinMaxScaler): Scaler used for data normalization.
+        ds (tf.data.Dataset): Dataset to evaluate on.
+
+    Returns:
+        Tuple[float, float]: RMSE and MAE metrics.
+    """
+    # Collect true labels (y) from dataset
+    y_true = []
+    for _, y in ds:
+        y_true.append(y.numpy())
+    y_true = np.concatenate(y_true)
+
+    # Predict the entire dataset
+    y_pred = model.predict(ds, verbose=0)  # Silent predictions
+
+    # Inverse transform to original scale
+    y_true_orig = scaler.inverse_transform(y_true)
+    y_pred_orig = scaler.inverse_transform(y_pred)
+
+    # Calculate metrics
+    return (np.sqrt(mean_squared_error(y_true_orig, y_pred_orig)), 
+            mean_absolute_error(y_true_orig, y_pred_orig),
+            mean_absolute_percentage_error(y_true_orig, y_pred_orig))
+
+def metric_and_predict_lstm_model(train_result: Dict) -> Dict:
+    """Evaluate the trained LSTM model and predict the next price.
+
+    Args:
+        train_result (Dict): Training result dictionary from train_lstm_model task.
+
+    Returns:
+        Dict: Evaluation metrics and prediction metadata.
+    """
+    # Access ti directly from kwargs
+    if not train_result:
+        raise ValueError("No training result provided.")
+    
+    # Convert string representation to dictionary if necessary
+    train_result = ast.literal_eval(train_result)
+
+    cfg = load_extract_config('model_config.yml')
+    parquet_folder = load_extract_config('pipeline_config.yml')['paths']['parquet_folder']
+    os.makedirs(parquet_folder, exist_ok=True)
+
+    model_cfg = cfg['model']
+    data_cfg = cfg['data']
+    out_cfg = cfg['output']
+    dt_str = train_result['datetime']
+    model_filename = train_result['model_filename']
+    dataset_merge = train_result['dataset_merge']
+
+    model_path = train_result['model_path']
+    scaler_path = train_result['scaler_path']
+    seq_length = data_cfg['seq_length']
+    batch_size = cfg['evaluation'].get('eval_batch_size', 64)
+
+    # Load scaler and model
+    with open(scaler_path, 'rb') as f:
+        scaler = pickle.load(f)
+    model = build_model_from_config(seq_length, cfg)
+    model.load_weights(model_path)
+
+    # Create dataset
+    parquet_paths = [parquet_folder for el in get_parquet_file_names()]
+    dataset = create_data_loader(parquet_paths, scaler, seq_length, batch_size)
+
+    # Calculate splits
+    total_seqs = sum(max(0, len(pd.read_parquet(path, columns=['Close'])) - seq_length)
+                     for path in parquet_paths if os.path.exists(path))
+    if total_seqs == 0:
+        raise ValueError("Not enough sequences for evaluation.")
+
+    steps_total = (total_seqs + batch_size - 1) // batch_size
+    steps_train = int(steps_total * data_cfg['train_ratio'])
+    steps_val = int(steps_total * data_cfg['val_ratio'])
+    steps_test = steps_total - steps_train - steps_val
+
+    train_ds = dataset.take(steps_train)
+    val_ds = dataset.skip(steps_train).take(steps_val)
+    test_ds = dataset.skip(steps_train + steps_val)
+
+    # Evaluate model
+    # train_rmse, train_mae = model_evaluate(model, scaler, train_ds)
+    # val_rmse, val_mae = model_evaluate(model, scaler, val_ds)
+    train_rmse, train_mae, train_mape = model_evaluate(model, scaler, train_ds)
+    val_rmse, val_mae, val_mape = model_evaluate(model, scaler, val_ds)
+    test_rmse, test_mae, test_mape = model_evaluate(model, scaler, test_ds)
+
+    # Save metrics
+    metrics_path = os.path.join(out_cfg['metrics']['metrics_dir'], f"metrics_{dt_str}.csv")
+    os.makedirs(out_cfg['metrics']['metrics_dir'], exist_ok=True)
+
+    metrics_data = [
+        [model_filename, dataset_merge, "Train", "RMSE", train_rmse],
+        [model_filename, dataset_merge, "Train", "MAE", train_mae],
+        [model_filename, dataset_merge, "Train", "MAPE", train_mape],
+        [model_filename, dataset_merge, "Val", "RMSE", val_rmse],
+        [model_filename, dataset_merge, "Val", "MAE", val_mae],
+        [model_filename, dataset_merge, "Val", "MAPE", val_mape],
+        [model_filename, dataset_merge, "Test", "RMSE", test_rmse],
+        [model_filename, dataset_merge, "Test", "MAE", test_mae],
+        [model_filename, dataset_merge, "Test", "MAPE", test_mape],
+    ]
+
+    metrics_df = pd.DataFrame(
+        metrics_data,
+        columns=['model_path', 'dataset_merge', 'Split', 'Metric', 'Value']
+    )
+    metrics_df.to_csv(metrics_path, index=False)
+
+    # Predict next price
+    last_chunk = None
+    for path in reversed(parquet_paths):
+        if os.path.exists(path):
+            df_tail = pd.read_parquet(path).tail(seq_length)
+            if len(df_tail) >= seq_length:
+                last_chunk = df_tail['Close'].values.astype('float32').reshape(-1, 1)
+                break
+    if last_chunk is None:
+        raise ValueError("Not enough recent data for prediction.")
+
+    last_scaled = scaler.transform(last_chunk)
+    next_scaled = model.predict(last_scaled.reshape(1, seq_length, 1), verbose=2)
+    next_price = scaler.inverse_transform(next_scaled)[0][0]
+
+    # Save prediction
+    pred_path = os.path.join(out_cfg['predictions']['pred_dir'], f"prediction_{dt_str}.txt")
+    os.makedirs(os.path.dirname(pred_path), exist_ok=True)
+
+    with open(pred_path, 'w') as f:
+        f.write(f"Model Run: {dt_str}\n")
+        f.write(f"Model File: {model_filename}\n")
+        f.write(f"Dataset Merged: {dataset_merge}\n")
+        f.write(f"Architecture: {model_cfg['architecture'].upper()}\n")
+        f.write(f"Predicted Next Close: {next_price:.6f}\n")
+        f.write(f"Based on last {seq_length} timesteps.\n\n")
+        f.write("Evaluation Metrics:\n")
+        f.write(f"  Train  -> RMSE: {train_rmse:8.6f} | MAE: {train_mae:8.6f} | MAPE: {train_mape:8.6f}\n")
+        f.write(f"  Val    -> RMSE: {val_rmse:8.6f} | MAE: {val_mae:8.6f} | MAPE: {val_mape:8.6f}\n")
+        f.write(f"  Test   -> RMSE: {test_rmse:8.6f} | MAE: {test_mae:8.6f} | MAPE: {test_mape:8.6f}\n")
+
+    logging.info(f"Next price: {next_price:.4f} | Test RMSE: {test_rmse:.6f} | Dataset: {dataset_merge}")
+
+    return {
+        'metrics_path': metrics_path,
+        'prediction_path': pred_path,
+        'next_price': float(next_price)
+    }
+
+if __name__ == "__main__":
+    logger.info("Running standalone evaluation test")
+    # Simulate training result for testing
+    cfg = load_extract_config('model_config.yml')
+    out_cfg = cfg['output']
+    data_cfg = cfg['data']
+    
+    # Mock training result (adjust paths to match an actual trained model and scaler)
+    mock_train_result = {
+        'model_path': os.path.join(out_cfg['checkpoints']['model_dir'], 
+                                   'model_2025-10-24-21-59-42-(+07).h5'),
+        'model_filename': 'model_2025-10-24-18-40-00-(+07).h5',
+        'scaler_path': os.path.join(out_cfg['checkpoints']['scaler_dir'], 
+                                    'scaler_2025-10-24-21-59-42-(+07).pkl'),
+        'datetime': '2025-10-24-21-59-42-(+07',
+        'dataset_merge': 'BTCUSDT-1s-2025-08 + BTCUSDT-1s-2025-09'
+    }
+
+    # Simulate Airflow task instance
+    class MockTaskInstance:
+        def xcom_pull(self, task_ids):
+            return mock_train_result
+    
+    mock_ti = MockTaskInstance()
+    
+    try:
+        result = metric_and_predict_lstm_model(ti=mock_ti)
+        logger.info("Evaluation completed successfully!")
+        logger.info(f"Result: {result}")
+    except Exception as e:
+        logger.error(f"Evaluation failed: {str(e)}")
+    logger.info("Standalone evaluation run completed")

components/model/model_utils.py

@@ -0,0 +1,294 @@
+import os
+import logging
+import numpy as np
+import tensorflow as tf
+import pyarrow.parquet as pq
+import pandas as pd
+from sklearn.preprocessing import MinMaxScaler
+from tensorflow import keras
+from typing import Tuple
+from datetime import datetime, timezone, timedelta
+import sys
+
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
+from components.model.data_utils import create_data_loader
+
+# Configure logging with +07:00 timezone
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s %(levelname)s: %(message)s',
+    datefmt='%Y-%m-%d %H:%M:%S %Z'
+)
+logger = logging.getLogger(__name__)
+
+def create_sequences(data: np.ndarray, seq_length: int) -> Tuple[np.ndarray, np.ndarray]:
+    """Create sequences of data for LSTM model training and prediction.
+
+    Args:
+        data (np.ndarray): Input time series data (scaled), shape (n_samples, n_features).
+        seq_length (int): Length of each sequence.
+
+    Returns:
+        Tuple[np.ndarray, np.ndarray]: (X, y) where X is input sequences (n_samples, seq_length, n_features)
+                                      and y is target values (n_samples, n_features).
+
+    Raises:
+        ValueError: If data is empty, seq_length is invalid, or data has insufficient length.
+    """
+    if not isinstance(data, np.ndarray):
+        logger.error("Input data must be a numpy array")
+        raise ValueError("Input data must be a numpy array")
+    if data.size == 0:
+        logger.error("Input data is empty")
+        raise ValueError("Input data is empty")
+    if not isinstance(seq_length, int) or seq_length <= 0:
+        logger.error(f"Invalid seq_length: {seq_length}")
+        raise ValueError("seq_length must be a positive integer")
+    if len(data) <= seq_length:
+        logger.error(f"Data length {len(data)} is insufficient for seq_length {seq_length}")
+        raise ValueError(f"Data length {len(data)} is insufficient for seq_length {seq_length}")
+
+    X, y = [], []
+    for i in range(len(data) - seq_length):
+        sequence = data[i:i + seq_length]
+        target = data[i + seq_length]
+        X.append(sequence)
+        y.append(target)
+
+    X = np.array(X)
+    y = np.array(y)
+
+    if len(X.shape) == 2:
+        X = X.reshape(X.shape[0], X.shape[1], 1)
+
+    logger.info(f"Created {X.shape[0]} sequences: X shape {X.shape}, y shape {y.shape}")
+    return X, y
+
+def build_model_from_config(seq_length: int, cfg: dict) -> keras.Model:
+    """Build an LSTM-based model based on configuration.
+
+    Args:
+        seq_length (int): Length of input sequences.
+        cfg (dict): Model configuration dictionary with 'model' key containing architecture, units, etc.
+
+    Returns:
+        keras.Model: Compiled Keras model.
+
+    Raises:
+        ValueError: If configuration is invalid or architecture is unsupported.
+    """
+    if not isinstance(cfg, dict) or 'model' not in cfg:
+        logger.error("Invalid configuration: 'model' key missing")
+        raise ValueError("Configuration must be a dictionary with a 'model' key")
+
+    model_cfg = cfg['model']
+    arch = model_cfg.get('architecture')
+    units = model_cfg.get('units')
+    layers = model_cfg.get('layers', 1)
+    dropout = model_cfg.get('dropout', 0.2)
+    activation = model_cfg.get('activation', 'tanh')
+    learning_rate = model_cfg.get('learning_rate', 0.001)
+
+    if not isinstance(units, int) or units <= 0:
+        logger.error(f"Invalid units: {units}")
+        raise ValueError("units must be a positive integer")
+    if not isinstance(layers, int) or layers <= 0:
+        logger.error(f"Invalid layers: {layers}")
+        raise ValueError("layers must be a positive integer")
+    if not isinstance(dropout, float) or not 0 <= dropout < 1:
+        logger.error(f"Invalid dropout: {dropout}")
+        raise ValueError("dropout must be a float between 0 and 1")
+    if arch not in ['lstm', 'bilstm', 'gru', 'custom']:
+        logger.error(f"Unsupported architecture: {arch}")
+        raise ValueError(f"Unsupported architecture: {arch}")
+    if not isinstance(seq_length, int) or seq_length <= 0:
+        logger.error(f"Invalid seq_length: {seq_length}")
+        raise ValueError("seq_length must be a positive integer")
+    if not isinstance(learning_rate, (int, float)) or learning_rate <= 0:
+        logger.error(f"Invalid learning_rate: {learning_rate}")
+        raise ValueError("learning_rate must be a positive number")
+
+    inputs = keras.layers.Input(shape=(seq_length, 1))
+    x = inputs
+
+    if arch == 'lstm':
+        # Improved LSTM Layer 1 (100 units, return_sequences=True)
+        x = keras.layers.LSTM(
+            units, return_sequences=True, activation=activation,
+            dropout=dropout, recurrent_dropout=0.1, kernel_regularizer=tf.keras.regularizers.l2(0.01)
+        )(x)
+        
+        # Attention mechanism
+        attention = keras.layers.Attention()([x, x])
+        x = keras.layers.Add()([x, attention])  # Residual connection
+        
+        # Improved LSTM Layer 2 (50 units, return_sequences=True)
+        x = keras.layers.LSTM(
+            units // 2, return_sequences=True, activation=activation,
+            dropout=dropout, recurrent_dropout=0.1, kernel_regularizer=tf.keras.regularizers.l2(0.01)
+        )(x)
+        
+        # Improved LSTM Layer 3 (25 units, return_sequences=False)
+        x = keras.layers.LSTM(
+            units // 4, return_sequences=False, activation=activation,
+            dropout=dropout, recurrent_dropout=0.1
+        )(x)
+        
+        # Dense layers
+        x = keras.layers.Dense(50, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01))(x)
+        x = keras.layers.Dropout(dropout)(x)
+        x = keras.layers.Dense(25, activation='relu')(x)
+        x = keras.layers.Dropout(dropout)(x)
+        
+        # Output layer
+        x = keras.layers.Dense(1)(x)
+    elif arch == 'gru':
+        # Improved GRU Layer 1 (100 units, return_sequences=True)
+        x = keras.layers.GRU(
+            units, return_sequences=True, activation=activation,
+            dropout=dropout, recurrent_dropout=0.1, kernel_regularizer=tf.keras.regularizers.l2(0.01)
+        )(x)
+        
+        # Attention mechanism
+        attention = keras.layers.Attention()([x, x])
+        x = keras.layers.Add()([x, attention])  # Residual connection
+        
+        # Improved GRU Layer 2 (50 units, return_sequences=True)
+        x = keras.layers.GRU(
+            units // 2, return_sequences=True, activation=activation,
+            dropout=dropout, recurrent_dropout=0.1, kernel_regularizer=tf.keras.regularizers.l2(0.01)
+        )(x)
+        
+        # Improved GRU Layer 3 (25 units, return_sequences=False)
+        x = keras.layers.GRU(
+            units // 4, return_sequences=False, activation=activation,
+            dropout=dropout, recurrent_dropout=0.1
+        )(x)
+        
+        # Dense layers
+        x = keras.layers.Dense(50, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01))(x)
+        x = keras.layers.Dropout(dropout)(x)
+        x = keras.layers.Dense(25, activation='relu')(x)
+        x = keras.layers.Dropout(dropout)(x)
+        
+        # Output layer
+        x = keras.layers.Dense(1)(x)
+    elif arch == 'bilstm':
+        # Improved BiLSTM Layer 1 (100 units, return_sequences=True)
+        x = keras.layers.Bidirectional(
+            keras.layers.LSTM(
+                units, return_sequences=True, activation=activation,
+                dropout=dropout, recurrent_dropout=0.1, kernel_regularizer=tf.keras.regularizers.l2(0.01)
+            )
+        )(x)
+        
+        # Attention mechanism
+        attention = keras.layers.Attention()([x, x])
+        x = keras.layers.Add()([x, attention])  # Residual connection
+        
+        # Improved BiLSTM Layer 2 (50 units, return_sequences=True)
+        x = keras.layers.Bidirectional(
+            keras.layers.LSTM(
+                units // 2, return_sequences=True, activation=activation,
+                dropout=dropout, recurrent_dropout=0.1, kernel_regularizer=tf.keras.regularizers.l2(0.01)
+            )
+        )(x)
+        
+        # Improved BiLSTM Layer 3 (25 units, return_sequences=False)
+        x = keras.layers.Bidirectional(
+            keras.layers.LSTM(
+                units // 4, return_sequences=False, activation=activation,
+                dropout=dropout, recurrent_dropout=0.1
+            )
+        )(x)
+        
+        # Dense layers
+        x = keras.layers.Dense(50, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01))(x)
+        x = keras.layers.Dropout(dropout)(x)
+        x = keras.layers.Dense(25, activation='relu')(x)
+        x = keras.layers.Dropout(dropout)(x)
+        
+        # Output layer
+        x = keras.layers.Dense(1)(x)
+    elif arch == 'custom':
+        # Improved Custom Layer 1 (LSTM, 100 units, return_sequences=True)
+        x = keras.layers.LSTM(
+            units, return_sequences=True, activation=activation,
+            dropout=dropout, recurrent_dropout=0.1, kernel_regularizer=tf.keras.regularizers.l2(0.01)
+        )(x)
+        
+        # Attention mechanism
+        attention = keras.layers.Attention()([x, x])
+        x = keras.layers.Add()([x, attention])  # Residual connection
+        
+        # Improved Custom Layer 2 (LSTM, 50 units, return_sequences=True)
+        x = keras.layers.LSTM(
+            units // 2, return_sequences=True, activation=activation,
+            dropout=dropout, recurrent_dropout=0.1, kernel_regularizer=tf.keras.regularizers.l2(0.01)
+        )(x)
+        
+        # Improved Custom Layer 3 (LSTM, 25 units, return_sequences=False)
+        x = keras.layers.LSTM(
+            units // 4, return_sequences=False, activation=activation,
+            dropout=dropout, recurrent_dropout=0.1
+        )(x)
+        
+        # Dense layers
+        x = keras.layers.Dense(50, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01))(x)
+        x = keras.layers.Dropout(dropout)(x)
+        x = keras.layers.Dense(25, activation='relu')(x)
+        x = keras.layers.Dropout(dropout)(x)
+        
+        # Output layer
+        x = keras.layers.Dense(1)(x)
+
+    model = keras.Model(inputs, x)
+
+    optimizer_name = model_cfg.get('optimizer', 'adam').lower()
+    if optimizer_name == 'adam':
+        optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
+    else:
+        logger.warning(f"Optimizer {optimizer_name} not explicitly handled, using default parameters")
+        optimizer = optimizer_name
+
+    model.compile(
+        optimizer=optimizer,
+        loss=model_cfg.get('loss', 'mse'),
+        metrics=['mae']
+    )
+    
+    logger.info(f"Built model: architecture={arch}, units={units}, layers={layers}, learning_rate={learning_rate}")
+    return model
+
+if __name__ == "__main__":
+    import pandas as pd
+    from components.utils.file_utils import load_config
+
+    logger.info("Running standalone tests for model_utils.py")
+    # Test create_sequences
+    data = np.array([[10000], [10050], [10100], [10150], [10200]])
+    seq_length = 3
+    X, y = create_sequences(data, seq_length)
+    print(f"create_sequences: X shape {X.shape}, y shape {y.shape}")
+    print(f"Sample sequence: {X[0]}, target: {y[0]}")
+
+    # Test create_data_loader
+    scaler = MinMaxScaler()
+    scaler.fit(data)
+    parquet_paths = ['temp/extracted_from_minio/btcusdt_1h.parquet']
+    if not os.path.exists(parquet_paths[0]):
+        os.makedirs(os.path.dirname(parquet_paths[0]), exist_ok=True)
+        pd.DataFrame({'Close': [10000, 10050, 10100, 10150, 10200]}).to_parquet(parquet_paths[0])
+    
+    dataset = create_data_loader(parquet_paths, scaler, seq_length=3, batch_size=2)
+    for x, y in dataset.take(1):
+        print(f"create_data_loader: x shape {x.shape}, y shape {y.shape}")
+
+    # Test build_model_from_config for all architectures
+    config = load_config('configs/model_config.yml')
+    for arch in ['lstm', 'gru', 'bilstm', 'custom']:
+        config['model']['architecture'] = arch
+        model = build_model_from_config(seq_length=3, cfg=config)
+        print(f"\nModel summary for {arch}:")
+        model.summary()
+    logger.info("Standalone tests completed successfully.")

components/model/old_model_utils.py

@@ -0,0 +1,189 @@
+import os
+import logging
+import numpy as np
+import tensorflow as tf
+import pyarrow.parquet as pq
+import pandas as pd
+from sklearn.preprocessing import MinMaxScaler
+from tensorflow import keras
+from typing import Tuple
+from datetime import datetime, timezone, timedelta
+import sys
+
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
+from components.model.data_utils import create_data_loader
+
+# Configure logging with +07:00 timezone
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s %(levelname)s: %(message)s',
+    datefmt='%Y-%m-%d %H:%M:%S %Z'
+)
+logger = logging.getLogger(__name__)
+
+def create_sequences(data: np.ndarray, seq_length: int) -> Tuple[np.ndarray, np.ndarray]:
+    """Create sequences of data for LSTM model training and prediction.
+
+    Args:
+        data (np.ndarray): Input time series data (scaled), shape (n_samples, n_features).
+        seq_length (int): Length of each sequence.
+
+    Returns:
+        Tuple[np.ndarray, np.ndarray]: (X, y) where X is input sequences (n_samples, seq_length, n_features)
+                                      and y is target values (n_samples, n_features).
+
+    Raises:
+        ValueError: If data is empty, seq_length is invalid, or data has insufficient length.
+    """
+    if not isinstance(data, np.ndarray):
+        logger.error("Input data must be a numpy array")
+        raise ValueError("Input data must be a numpy array")
+    if data.size == 0:
+        logger.error("Input data is empty")
+        raise ValueError("Input data is empty")
+    if not isinstance(seq_length, int) or seq_length <= 0:
+        logger.error(f"Invalid seq_length: {seq_length}")
+        raise ValueError("seq_length must be a positive integer")
+    if len(data) <= seq_length:
+        logger.error(f"Data length {len(data)} is insufficient for seq_length {seq_length}")
+        raise ValueError(f"Data length {len(data)} is insufficient for seq_length {seq_length}")
+
+    X, y = [], []
+    for i in range(len(data) - seq_length):
+        sequence = data[i:i + seq_length]
+        target = data[i + seq_length]
+        X.append(sequence)
+        y.append(target)
+
+    X = np.array(X)
+    y = np.array(y)
+
+    if len(X.shape) == 2:
+        X = X.reshape(X.shape[0], X.shape[1], 1)
+
+    logger.info(f"Created {X.shape[0]} sequences: X shape {X.shape}, y shape {y.shape}")
+    return X, y
+
+def build_model_from_config(seq_length: int, cfg: dict) -> keras.Model:
+    """Build an LSTM-based model based on configuration.
+
+    Args:
+        seq_length (int): Length of input sequences.
+        cfg (dict): Model configuration dictionary with 'model' key containing architecture, units, etc.
+
+    Returns:
+        keras.Model: Compiled Keras model.
+
+    Raises:
+        ValueError: If configuration is invalid or architecture is unsupported.
+    """
+    if not isinstance(cfg, dict) or 'model' not in cfg:
+        logger.error("Invalid configuration: 'model' key missing")
+        raise ValueError("Configuration must be a dictionary with a 'model' key")
+
+    model_cfg = cfg['model']
+    arch = model_cfg.get('architecture')
+    units = model_cfg.get('units')
+    layers = model_cfg.get('layers', 1)
+    dropout = model_cfg.get('dropout', 0.2)
+    activation = model_cfg.get('activation', 'tanh')
+    learning_rate = model_cfg.get('learning_rate', 0.001)  # Default learning rate if not specified
+
+    if not isinstance(units, int) or units <= 0:
+        logger.error(f"Invalid units: {units}")
+        raise ValueError("units must be a positive integer")
+    if not isinstance(layers, int) or layers <= 0:
+        logger.error(f"Invalid layers: {layers}")
+        raise ValueError("layers must be a positive integer")
+    if not isinstance(dropout, float) or not 0 <= dropout < 1:
+        logger.error(f"Invalid dropout: {dropout}")
+        raise ValueError("dropout must be a float between 0 and 1")
+    if arch not in ['lstm', 'bilstm', 'gru', 'custom']:
+        logger.error(f"Unsupported architecture: {arch}")
+        raise ValueError(f"Unsupported architecture: {arch}")
+    if not isinstance(seq_length, int) or seq_length <= 0:
+        logger.error(f"Invalid seq_length: {seq_length}")
+        raise ValueError("seq_length must be a positive integer")
+    if not isinstance(learning_rate, (int, float)) or learning_rate <= 0:
+        logger.error(f"Invalid learning_rate: {learning_rate}")
+        raise ValueError("learning_rate must be a positive number")
+
+    inputs = keras.layers.Input(shape=(seq_length, 1))
+    x = inputs
+
+    for i in range(layers):
+        return_seq = i < layers - 1
+        if arch == 'lstm':
+            x = keras.layers.LSTM(
+                units, return_sequences=return_seq, activation=activation,
+                dropout=dropout, recurrent_dropout=0.1
+            )(x)
+        elif arch == 'bilstm':
+            x = keras.layers.Bidirectional(
+                keras.layers.LSTM(
+                    units, return_sequences=return_seq, activation=activation,
+                    dropout=dropout, recurrent_dropout=0.1
+                )
+            )(x)
+        elif arch == 'gru':
+            x = keras.layers.GRU(
+                units, return_sequences=return_seq, activation=activation,
+                dropout=dropout, recurrent_dropout=0.1
+            )(x)
+        elif arch == 'custom':
+            x = keras.layers.LSTM(units, return_sequences=True)(x)
+            x = keras.layers.LSTM(units // 2, return_sequences=False)(x)
+            x = keras.layers.Dense(50, activation='relu')(x)
+            x = keras.layers.Dropout(dropout)(x)
+
+    if arch != 'custom':
+        x = keras.layers.Dense(1)(x)
+
+    model = keras.Model(inputs, x)
+
+    # Configure optimizer with specified learning rate
+    optimizer_name = model_cfg.get('optimizer', 'adam').lower()
+    if optimizer_name == 'adam':
+        optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
+    else:
+        logger.warning(f"Optimizer {optimizer_name} not explicitly handled, using default parameters")
+        optimizer = optimizer_name  # Fallback to string, Keras will handle it
+
+    model.compile(
+        optimizer=optimizer,
+        loss=model_cfg.get('loss', 'mse'),
+        metrics=['mae']
+    )
+    
+    logger.info(f"Built model: architecture={arch}, units={units}, layers={layers}, learning_rate={learning_rate}")
+    return model
+
+if __name__ == "__main__":
+    import pandas as pd
+    from components.utils.file_utils import load_config
+
+    logger.info("Running standalone tests for lstm_utils.py")
+    # Test create_sequences
+    data = np.array([[10000], [10050], [10100], [10150], [10200]])
+    seq_length = 3
+    X, y = create_sequences(data, seq_length)
+    print(f"create_sequences: X shape {X.shape}, y shape {y.shape}")
+    print(f"Sample sequence: {X[0]}, target: {y[0]}")
+
+    # Test create_data_loader
+    scaler = MinMaxScaler()
+    scaler.fit(data)
+    parquet_paths = ['temp/extracted_from_minio/btcusdt_1h.parquet']
+    if not os.path.exists(parquet_paths[0]):
+        os.makedirs(os.path.dirname(parquet_paths[0]), exist_ok=True)
+        pd.DataFrame({'Close': [10000, 10050, 10100, 10150, 10200]}).to_parquet(parquet_paths[0])
+    
+    dataset = create_data_loader(parquet_paths, scaler, seq_length=3, batch_size=2)
+    for x, y in dataset.take(1):
+        print(f"create_data_loader: x shape {x.shape}, y shape {y.shape}")
+
+    # Test build_model_from_config
+    config = load_config('model_config.yml')
+    model = build_model_from_config(seq_length=3, cfg=config)
+    model.summary()
+    logger.info("Standalone tests completed successfully.")

components/model/training.py

@@ -0,0 +1,194 @@
+import os
+import logging
+import pandas as pd
+import numpy as np
+from sklearn.preprocessing import MinMaxScaler
+import pickle
+from datetime import datetime, timezone, timedelta
+import tensorflow as tf
+from tensorflow import keras
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from typing import Dict, List
+import sys
+
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
+from components.utils.file_utils import load_extract_config, get_parquet_file_names
+from components.model.model_utils import build_model_from_config
+from components.model.data_utils import create_data_loader
+from components.utils.utils import parse_timezone
+
+# Configure logging with +07:00 timezone
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s %(levelname)s: %(message)s',
+    datefmt='%Y-%m-%d %H:%M:%S %Z'
+)
+logger = logging.getLogger(__name__)
+
+def train_lstm_model(**kwargs) -> Dict:
+    """Train an LSTM model for BTC/USDT forecasting and save model and scaler.
+
+    Args:
+        kwargs: Airflow task instance arguments.
+
+    Returns:
+        Dict: Training metadata including model path, scaler path, metrics, and dataset info.
+    """
+    # Verify GPU availability
+    gpus = tf.config.list_physical_devices('GPU')
+    if not gpus:
+        logger.warning("No GPU detected. Training on CPU, which may be slower.")
+    else:
+        logger.info(f"GPUs detected: {len(gpus)}. Using CUDA for training.")
+        for gpu in gpus:
+            tf.config.experimental.set_memory_growth(gpu, True)
+
+    cfg = load_extract_config('model_config.yml')
+    model_cfg = cfg['model']
+    train_cfg = cfg['training']
+    data_cfg = cfg['data']
+    out_cfg = cfg['output']
+    ver_cfg = cfg['versioning']
+
+    # Parse timezone from YAML
+    tz_offset_str = ver_cfg['timezone']  # '+07:00'
+    tz = parse_timezone(tz_offset_str)
+
+    # Get current time in the specified timezone
+    dt = datetime.now(tz)
+    dt_str = dt.strftime(ver_cfg['datetime_format']) + f"-({dt.strftime('%z')[:3]})"
+    model_path = os.path.join(out_cfg['checkpoints']['model_dir'], f"model_{dt_str}.h5")
+    scaler_path = os.path.join(out_cfg['checkpoints']['scaler_dir'], f"scaler_{dt_str}.pkl")
+    parquet_folder = load_extract_config('pipeline_config.yml')['paths']['parquet_folder']
+
+    # Ensure output directories exist
+    os.makedirs(os.path.dirname(model_path), exist_ok=True)
+    os.makedirs(os.path.dirname(scaler_path), exist_ok=True)
+
+    
+    # Load data
+    file_names = get_parquet_file_names()
+    parquet_paths = [os.path.join(parquet_folder, el) for el in file_names]
+    all_df = pd.DataFrame()
+    used_files = []
+
+    for path, name in zip(parquet_paths, file_names):
+        if os.path.exists(path):
+            df = pd.read_parquet(path)
+            logger.info(f"Loaded {path} with {len(df)} rows")
+            all_df = pd.concat([all_df, df], ignore_index=True)
+            clean_name = name.replace(".parquet", "").replace(".csv", "")
+            used_files.append(clean_name)
+        else:
+            logger.warning(f"File not found: {path}")
+
+    if all_df.empty:
+        logger.error("No data loaded from Parquet files")
+        raise ValueError("No data loaded from Parquet files")
+
+    dataset_merge = " + ".join(used_files) if used_files else "none"
+    logger.info(f"Dataset merged: {dataset_merge}, total rows: {len(all_df)}")
+
+    # Scale data
+    scaler = MinMaxScaler()
+    prices = all_df['Close'].astype(float).values.reshape(-1, 1)
+    if prices.size <= data_cfg['seq_length']:
+        logger.error(f"Total data size {prices.size} is insufficient for seq_length {data_cfg['seq_length']}")
+        raise ValueError(f"Total data size {prices.size} is insufficient for seq_length {data_cfg['seq_length']}")
+    prices_scaled = scaler.fit_transform(prices)
+
+    # Create dataset with smaller batch size
+    seq_length = data_cfg['seq_length']
+    batch_size = train_cfg.get('batch_size', 64)  # Default to 64 if not specified
+    if batch_size > 8192:
+        logger.warning(f"Batch size {batch_size} is large; reducing to 64 to avoid memory issues")
+        batch_size = 64
+
+    dataset = create_data_loader(parquet_paths, scaler, seq_length, batch_size)
+
+    # Calculate exact number of sequences
+    total_seqs = 0
+    for path in parquet_paths:
+        if os.path.exists(path):
+            df = pd.read_parquet(path, columns=['Close'])
+            seqs = max(0, len(df) - seq_length)
+            total_seqs += seqs
+            logger.info(f"File {path}: {len(df)} rows, {seqs} sequences")
+    
+    if total_seqs == 0:
+        logger.error("Not enough sequences for training")
+        raise ValueError("Not enough sequences for training")
+
+    # Calculate steps for training, validation, and test
+    steps_total = (total_seqs + batch_size - 1) // batch_size
+    train_ratio = data_cfg.get('train_ratio', 0.7)
+    val_ratio = data_cfg.get('val_ratio', 0.2)
+    steps_train = max(1, int(steps_total * train_ratio))
+    steps_val = max(1, int(steps_total * val_ratio))
+    steps_test = max(1, steps_total - steps_train - steps_val)
+    logger.info(f"Dataset splits: total_steps={steps_total}, train={steps_train}, val={steps_val}, test={steps_test}")
+
+    # Save scaler
+    with open(scaler_path, 'wb') as f:
+        pickle.dump(scaler, f)
+
+    train_ds = dataset.take(steps_train)
+    val_ds = dataset.skip(steps_train).take(steps_val)
+    test_ds = dataset.skip(steps_train + steps_val)
+
+    # Build and train model
+    model = build_model_from_config(seq_length, cfg)
+    checkpoint_cb = keras.callbacks.ModelCheckpoint(
+        model_path, save_best_only=True, monitor='val_loss', verbose=0
+    )
+    early_stop = keras.callbacks.EarlyStopping(
+        monitor='val_loss', patience=train_cfg['patience'], restore_best_weights=True
+    )
+
+    # Log model summary
+    model.summary(print_fn=lambda x: logger.info(x))
+
+    # Train with exact steps_per_epoch
+    model.fit(
+        train_ds,
+        epochs=train_cfg['epochs'],
+        steps_per_epoch=steps_train,
+        validation_data=val_ds,
+        validation_steps=steps_val,
+        callbacks=[checkpoint_cb, early_stop],
+        verbose=2
+    )
+
+    
+
+    # # Test evaluation
+    # y_true, y_pred = [], []
+    # for X, y in test_ds:
+    #     pred = model.predict(X, verbose=0)
+    #     y_true.append(y.numpy())
+    #     y_pred.append(pred)
+    # y_true = np.concatenate(y_true)
+    # y_pred = np.concatenate(y_pred)
+    # y_true_orig = scaler.inverse_transform(y_true)
+    # y_pred_orig = scaler.inverse_transform(y_pred)
+    # test_rmse = np.sqrt(mean_squared_error(y_true_orig, y_pred_orig))
+    # test_mae = mean_absolute_error(y_true_orig, y_pred_orig)
+
+    # logger.info(f"Test RMSE: {test_rmse:.4f}, MAE: {test_mae:.4f}")
+
+    return {
+        'model_path': model_path,
+        'model_filename': os.path.basename(model_path),
+        'scaler_path': scaler_path,
+        'datetime': dt_str,
+        'dataset_merge': dataset_merge,
+        # 'test_rmse': float(test_rmse),
+        # 'test_mae': float(test_mae)
+    }
+
+if __name__ == "__main__":
+    logger.info("Running standalone training test")
+    result = train_lstm_model()
+    print("Training completed successfully!")
+    print(result)
+    logger.info("Standalone training run completed")

components/old-process_data.py

@@ -0,0 +1,111 @@
+# scripts/process_data.py
+from pyspark.sql import SparkSession
+from pyspark.sql.types import StructType, StructField, LongType, DoubleType, IntegerType
+from pyspark.sql.functions import col, row_number, floor, first, max, min, last, sum
+from pyspark.sql.window import Window
+import os
+import sys
+import shutil
+import pandas as pd
+
+# Add the project root directory to the Python path
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+from minio_api.client import sign_in
+from minio_api.minio_utils import get_minio_data
+
+def initialize_spark_session(app_name="MinIO to Spark DataFrame", 
+                             driver_memory="4g", executor_memory="4g"):
+    return SparkSession.builder \
+        .appName(app_name) \
+        .config("spark.driver.memory", driver_memory) \
+        .config("spark.executor.memory", executor_memory) \
+        .getOrCreate()
+
+def create_dataframe_from_csv(spark, csv_lines, temp_parquet_path="temp/temp_parquet_chunks", 
+                              chunk_size=int(3e+6)):
+    os.makedirs(temp_parquet_path, exist_ok=True)
+    schema = StructType([
+        StructField("Open time", LongType(), True),
+        StructField("Open", DoubleType(), True),
+        StructField("High", DoubleType(), True),
+        StructField("Low", DoubleType(), True),
+        StructField("Close", DoubleType(), True),
+        StructField("Volume", DoubleType(), True),
+        StructField("Close time", LongType(), True),
+        StructField("Quote asset volume", DoubleType(), True),
+        StructField("Number of trades", IntegerType(), True),
+        StructField("Taker buy base asset volume", DoubleType(), True),
+        StructField("Taker buy quote asset volume", DoubleType(), True),
+        StructField("Ignore", IntegerType(), True)
+    ])
+
+    if csv_lines and csv_lines[0].startswith("Open time,"):
+        data_lines = csv_lines[1:]
+    else:
+        data_lines = csv_lines
+
+    if os.path.exists(temp_parquet_path):
+        shutil.rmtree(temp_parquet_path)
+
+    for i in range(0, len(data_lines), chunk_size):
+        chunk = data_lines[i:i + chunk_size]
+        rdd_chunk = spark.sparkContext.parallelize(chunk).repartition(8)
+        df_chunk = spark.read.schema(schema).csv(rdd_chunk, header=False)
+        df_chunk.write.mode("append").parquet(temp_parquet_path)
+
+    return spark.read.parquet(temp_parquet_path)
+
+def resample_dataframe(df, track_each=3600):
+    keep_cols = ["Open time", "Open", "High", "Low", "Close", "Number of trades"]
+    df = df.select(keep_cols)
+    window_spec = Window.orderBy("Open time")
+    df = df.withColumn("row_number", row_number().over(window_spec))
+    df = df.withColumn("group_id", floor((col("row_number") - 1) / track_each))
+    aggregations = [
+        first("Open time").alias("Open time"),
+        first("Open").alias("Open"),
+        max("High").alias("High"),
+        min("Low").alias("Low"),
+        last("Close").alias("Close"),
+        sum("Number of trades").alias("Number of trades")
+    ]
+    aggregated_df = df.groupBy("group_id").agg(*aggregations)
+    return aggregated_df.select("Open time", "Open", "High", "Low", "Close", "Number of trades")
+
+def process_financial_data(bucket_name="minio-ngrok-bucket", file_name="BTCUSDT-1s-2025-09.csv", 
+                           temp_parquet_path="temp/temp_parquet_chunks", 
+                           output_parquet_path="temp/aggregated_output"):
+    minio_client = sign_in()
+    spark = initialize_spark_session()
+    
+    try:
+        csv_lines = get_minio_data(minio_client, bucket_name, file_name)
+        print(f"Fetched CSV data from MinIO: {len(csv_lines)} lines")
+        df = create_dataframe_from_csv(spark, csv_lines, temp_parquet_path)
+        print("Created Spark DataFrame from CSV data.")
+        aggregated_df = resample_dataframe(df)
+        print("Resampled DataFrame with OHLC aggregations.")
+        
+        # Save aggregated DataFrame to a temporary Parquet directory
+        os.makedirs(os.path.dirname(output_parquet_path), exist_ok=True)
+        aggregated_df.write.mode("overwrite").parquet(output_parquet_path)
+        print(f"Saved aggregated DataFrame to {output_parquet_path}")
+        
+        # Verify that the Parquet directory exists
+        if not os.path.exists(output_parquet_path) or not os.path.isdir(output_parquet_path):
+            raise FileNotFoundError(f"Parquet directory {output_parquet_path} was not created or is not a directory.")
+        else:
+            print(f"Verified: Parquet directory exists at {output_parquet_path}")
+        
+        return output_parquet_path
+    
+    except Exception as e:
+        print(f"Error in process_financial_data: {e}")
+        raise
+    finally:
+        spark.stop()
+
+if __name__ == "__main__":
+    # Example usage
+    output_parquet_path = process_financial_data()
+    print(output_parquet_path)

components/process_data.py

@@ -0,0 +1,150 @@
+from pyspark.sql import SparkSession
+from pyspark.sql.types import StructType, StructField, LongType, DoubleType, IntegerType
+from pyspark.sql.functions import col, row_number, floor, first, max, min, last, sum
+from pyspark.sql.window import Window
+
+import os
+import sys
+import shutil
+import pandas as pd
+import ast
+import io
+
+# Add the project root directory to the Python path
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+from minio_api.minio_utils import get_minio_data
+from minio_api.client import sign_in
+
+
+def initialize_spark_session(app_name="MinIO to Spark DataFrame", 
+                             driver_memory="4g", executor_memory="4g"):
+    return SparkSession.builder \
+        .appName(app_name) \
+        .config("spark.driver.memory", driver_memory) \
+        .config("spark.executor.memory", executor_memory) \
+        .getOrCreate()
+
+def create_dataframe_from_csv(spark, parquet_file_path, schema, temp_parquet_path="temp/temp_parquet_chunks", 
+                              chunk_size=int(3e+6)):
+    os.makedirs(temp_parquet_path, exist_ok=True)
+    
+    # Clear the temporary Parquet path if it exists
+    if os.path.exists(temp_parquet_path):
+        shutil.rmtree(temp_parquet_path)
+    
+    # Read Parquet file directly with Spark, applying the schema
+    df = spark.read.schema(schema).parquet(parquet_file_path)
+    
+    # Write the DataFrame to the temporary Parquet path (for consistency with original logic)
+    df.write.mode("append").parquet(temp_parquet_path)
+    
+    # Read back the Parquet data from the temporary path
+    return spark.read.parquet(temp_parquet_path)
+
+def resample_dataframe(df, track_each=3600):
+    keep_cols = ["Open time", "Open", "High", "Low", "Close", "Number of trades"]
+    df = df.select(keep_cols)
+    window_spec = Window.orderBy("Open time")
+    df = df.withColumn("row_number", row_number().over(window_spec))
+    df = df.withColumn("group_id", floor((col("row_number") - 1) / track_each))
+    aggregations = [
+        first("Open time").alias("Open time"),
+        first("Open").alias("Open"),
+        max("High").alias("High"),
+        min("Low").alias("Low"),
+        last("Close").alias("Close"),
+        sum("Number of trades").alias("Number of trades")
+    ]
+    aggregated_df = df.groupBy("group_id").agg(*aggregations)
+    return aggregated_df.select("Open time", "Open", "High", "Low", "Close", "Number of trades")
+
+def extract_from_minio(bucket_name="minio-ngrok-bucket", 
+                       file_names=["BTCUSDT-1s-2025-09.csv"]):
+    minio_client = sign_in()
+    out_parquet_file_paths = []
+    headers = [
+        "Open time", "Open", "High", "Low", "Close", "Volume",
+        "Close time", "Quote asset volume", "Number of trades",
+        "Taker buy base asset volume", "Taker buy quote asset volume", "Ignore"
+    ]
+    
+    for file_name in file_names:
+        csv_lines = get_minio_data(minio_client, bucket_name, file_name)
+        if not csv_lines:
+            raise ValueError(f"No data retrieved from MinIO for bucket {bucket_name}, file {file_name}")
+        temp_parquet_path = f"temp/extracted_from_minio/{os.path.splitext(os.path.basename(file_name))[0]}.parquet"
+        os.makedirs(os.path.dirname(temp_parquet_path), exist_ok=True)
+        
+        # Convert CSV lines to DataFrame with specified headers
+        df = pd.read_csv(io.StringIO('\n'.join(csv_lines)), names=headers)
+        df.to_parquet(temp_parquet_path, index=False)  
+        
+        out_parquet_file_paths.append(temp_parquet_path)
+    
+    return out_parquet_file_paths
+
+def transform_financial_data(parquet_file_paths, 
+                            temp_parquet_path="temp/temp_parquet_chunks", 
+                            output_parquet_path="temp/aggregated_output"):
+    spark = initialize_spark_session()
+    
+    try:
+        # Define the schema
+        schema = StructType([
+            StructField("Open time", LongType(), True),
+            StructField("Open", DoubleType(), True),
+            StructField("High", DoubleType(), True),
+            StructField("Low", DoubleType(), True),
+            StructField("Close", DoubleType(), True),
+            StructField("Volume", DoubleType(), True),
+            StructField("Close time", LongType(), True),
+            StructField("Quote asset volume", DoubleType(), True),
+            StructField("Number of trades", LongType(), True),
+            StructField("Taker buy base asset volume", DoubleType(), True),
+            StructField("Taker buy quote asset volume", DoubleType(), True),
+            StructField("Ignore", LongType(), True)
+        ])
+
+        # output_parquet_paths = []
+        if isinstance(parquet_file_paths, str):
+            try:
+                parquet_file_paths = ast.literal_eval(parquet_file_paths)
+            except (ValueError, SyntaxError) as e:
+                raise ValueError(f"Failed to parse server_files as a list: {parquet_file_paths}, error: {e}")
+
+        for parquet_file_path in parquet_file_paths:
+            # Create DataFrame using create_dataframe_from_csv
+            df = create_dataframe_from_csv(spark, parquet_file_path, schema, temp_parquet_path)
+            print("Created Spark DataFrame from CSV file.")
+            aggregated_df = resample_dataframe(df)
+            print("Resampled DataFrame with OHLC aggregations.")
+            
+            # Save aggregated DataFrame to a temporary Parquet directory
+            os.makedirs(os.path.dirname(output_parquet_path), exist_ok=True)
+            # aggregated_df.write.mode("overwrite").parquet(output_parquet_path)
+            aggregated_df.write.mode("append").parquet(output_parquet_path)
+            print(f"Saved aggregated DataFrame to {output_parquet_path}")
+            
+            # Verify that the Parquet directory exists
+            if not os.path.exists(output_parquet_path) or not os.path.isdir(output_parquet_path):
+                raise FileNotFoundError(f"Parquet directory {output_parquet_path} was not created or is not a directory.")
+            else:
+                print(f"Verified: Parquet directory exists at {output_parquet_path}")
+            # output_parquet_paths.append(output_parquet_path)
+
+        # name_output_parquet_paths = [os.path.basename(path) for path in output_parquet_paths]
+        
+        return output_parquet_path#, name_output_parquet_paths
+    
+    except Exception as e:
+        print(f"Error in transform_financial_data: {e}")
+        raise
+    finally:
+        spark.stop()
+
+if __name__ == "__main__":
+    # Example usage
+    extracted_parquet_path = extract_from_minio()
+    output_parquet_path, name_output_parquet_paths = transform_financial_data(extracted_parquet_path)
+    print(output_parquet_path)
+    print(name_output_parquet_paths)

components/utils/file_utils.py

@@ -0,0 +1,105 @@
+import os
+import yaml
+import logging
+from typing import Dict, List
+
+# Configure logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+
+def load_extract_config(config_name: str) -> Dict:
+    """Load a YAML configuration file from the configs directory.
+
+    Args:
+        config_name (str): Name of the config file (e.g., 'model_config.yml').
+
+    Returns:
+        Dict: Parsed configuration dictionary.
+
+    Raises:
+        FileNotFoundError: If the configuration file does not exist.
+        ValueError: If config_name is empty or not a string.
+    """
+    if not isinstance(config_name, str) or not config_name.strip():
+        raise ValueError("config_name must be a non-empty string")
+
+    config_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', 'configs', config_name))
+    logging.debug(f"Attempting to load config from: {config_path}")
+
+    if not os.path.exists(config_path):
+        logging.error(f"Configuration file not found: {config_path}")
+        raise FileNotFoundError(f"Configuration file not found: {config_path}")
+
+    try:
+        with open(config_path, 'r') as f:
+            config = yaml.safe_load(f)
+        if config is None:
+            logging.warning(f"Configuration file {config_name} is empty")
+            return {}
+        logging.info(f"Successfully loaded config: {config_name}")
+        return config
+    except yaml.YAMLError as e:
+        logging.error(f"Failed to parse YAML in {config_name}: {e}")
+        raise
+
+def get_parquet_file_names() -> List[str]:
+    """Retrieve Parquet file names from the extract_data.yml configuration.
+
+    Returns:
+        List[str]: List of Parquet file names derived from CSV file names.
+
+    Raises:
+        FileNotFoundError: If extract_data.yml is missing.
+        ValueError: If no files are specified in the configuration.
+    """
+    config = load_extract_config('extract_data.yml')
+    files = config.get('files', [])
+    if not files:
+        logging.error("No files specified in extract_data.yml")
+        raise ValueError("No files specified in extract_data.yml")
+    
+    parquet_files = [f.replace(".csv", ".parquet") for f in files]
+    logging.debug(f"Derived Parquet file names: {parquet_files}")
+    return parquet_files
+
+def load_pipeline_config() -> Dict:
+    """Load pipeline configuration from pipeline_config.yml.
+
+    Returns:
+        Dict: Pipeline configuration dictionary.
+
+    Raises:
+        FileNotFoundError: If pipeline_config.yml is missing.
+    """
+    config = load_extract_config('pipeline_config.yml')
+    logging.debug(f"Pipeline config loaded: {config}")
+    return config
+
+def define_server_filenames(**kwargs) -> List[str]:
+    """Extract base filenames from client file paths using Airflow XCom.
+
+    Args:
+        kwargs: Airflow task instance arguments containing 'ti' for XCom.
+
+    Returns:
+        List[str]: List of base filenames.
+
+    Raises:
+        KeyError: If 'ti' is not provided in kwargs.
+        ValueError: If no files are retrieved from XCom.
+    """
+    if 'ti' not in kwargs:
+        logging.error("Task instance 'ti' not provided in kwargs")
+        raise KeyError("Task instance 'ti' not provided in kwargs")
+
+    ti = kwargs['ti']
+    client_files = ti.xcom_pull(task_ids='download_binance_csv')
+    if client_files is None:
+        logging.error("No files retrieved from XCom for task 'download_binance_csv'")
+        raise ValueError("No files retrieved from XCom for task 'download_binance_csv'")
+
+    if not isinstance(client_files, list):
+        client_files = [client_files]
+    
+    server_files = [os.path.basename(p) for p in client_files]
+    logging.debug(f"Extracted server filenames: {server_files}")
+    return server_files

components/utils/utils.py

@@ -0,0 +1,26 @@
+from datetime import timezone, timedelta
+import re
+
+def parse_timezone(tz_offset_str):
+    """
+    Parse a timezone offset string (e.g., '+07:00') and return a timezone object.
+    
+    Args:
+        tz_offset_str (str): Timezone offset in format '[+-]HH:MM'
+    
+    Returns:
+        timezone: A datetime.timezone object with the specified offset
+    
+    Raises:
+        ValueError: If the timezone format is invalid
+    """
+    match = re.match(r'([+-])(\d{2}):(\d{2})', tz_offset_str)
+    if not match:
+        raise ValueError(f"Invalid timezone format: {tz_offset_str}")
+    
+    sign, hours, minutes = match.groups()
+    hours, minutes = int(hours), int(minutes)
+    if sign == '-':
+        hours, minutes = -hours, -minutes
+    
+    return timezone(timedelta(hours=hours, minutes=minutes))

configs/data_limit.yml

@@ -0,0 +1,4 @@
+- name: BTCUSDT-1s
+  limit:
+    - "2025-08"
+    - "2025-09"

configs/data_sources.yml

@@ -0,0 +1,2 @@
+- name: BTCUSDT-1s
+  url: https://data.binance.vision/data/spot/monthly/klines/BTCUSDT/1s/

configs/delete_lstm_hyperparams.yml

@@ -0,0 +1,17 @@
+# LSTM / BiLSTM / GRU / Custom Model Hyperparameters
+architecture: bilstm  # Options: lstm, bilstm, gru, custom
+seq_length: 60 # Number of time steps to look back
+units: 100
+layers: 2
+dropout: 0.2
+activation: tanh
+
+optimizer: adam
+loss: mse
+epochs: 50
+batch_size: 64
+patience: 10
+
+train_ratio: 0.8
+val_ratio: 0.1
+test_ratio: 0.1

configs/extract_data.yml

@@ -0,0 +1,4 @@
+files:
+  - BTCUSDT-1s-2025-08.csv
+  - BTCUSDT-1s-2025-09.csv
+storage_folder: temp/extracted_from_minio

configs/model_config.yml

@@ -0,0 +1,41 @@
+# =============================================================================
+# Unified Model & Output Configuration
+# =============================================================================
+
+model:
+  architecture: bilstm        # lstm | bilstm | gru | custom
+  units: 100 #100
+  layers: 3 #2
+  dropout: 0.2
+  activation: tanh
+  optimizer: adam
+  learning_rate: 0.0005
+  loss: mse
+
+training:
+  epochs: 20
+  batch_size: 6144
+  patience: 5
+
+evaluation:
+  eval_batch_size: 6144
+
+data:
+  seq_length: 60
+  train_ratio: 0.8
+  val_ratio: 0.1
+
+output:
+  checkpoints:
+    model_dir: ckpts
+    scaler_dir: ckpts
+
+  metrics:
+    metrics_dir: evaluation
+
+  predictions:
+    pred_dir: evaluation
+
+versioning:
+  datetime_format: '%Y-%m-%d-%H-%M-%S'
+  timezone: '+07:00'

configs/pipeline_config.yml

@@ -0,0 +1,8 @@
+minio:
+  bucket_name: minio-ngrok-bucket
+paths:
+  temp_parquet_path: temp/temp_parquet_chunks
+  output_parquet_path: temp/aggregated_output
+  duckdb_path: duckdb_databases/financial_data.db
+  output_csv_path: analytics/financial_data.csv
+  parquet_folder: temp/extracted_from_minio

docs/data_sources.md

@@ -0,0 +1,26 @@
+# Data Source: Binance BTCUSDT 1s Kline
+
+- **Source:** [Binance Data Public](https://data.binance.vision/?prefix=data/spot/monthly/klines/BTCUSDT/1s/)
+- **Example file:** `BTCUSDT-1s-2024-05.csv.gz`
+- **Columns:**
+  1. Open time
+  2. Open
+  3. High
+  4. Low
+  5. Close
+  6. Volume
+  7. Close time
+  8. Quote asset volume
+  9. Number of trades
+  10. Taker buy base asset volume
+  11. Taker buy quote asset volume
+  12. Ignore
+
+## Usage
+
+- The pipeline downloads and processes this data for deep learning time series forecasting.
+- The DAG can be modified to use a different month by changing the filename in the download step.
+
+## Acknowledgment
+You can refer to https://github.com/binance/binance-public-data?tab=readme-ov-file#klines to get more infromation
+

docs/dependencies.md

@@ -0,0 +1,22 @@
+# Dependencies
+
+## System
+
+- Hadoop/HDFS
+- Spark
+- Airflow
+- Python 3.8+
+
+## Python
+
+Install with pip:
+
+```bash
+pip install pandas numpy scikit-learn tensorflow
+```
+
+## Notes
+
+- Ensure Java is installed for Hadoop/Spark.
+- Airflow and Hadoop should be configured and running before triggering the DAG.
+- If using a dev container, dependencies may already be installed.

docs/frameworks_installation.md

@@ -0,0 +1,72 @@
+## Spark
+### Install Java 8 (required for Spark)
+!apt-get update -qq
+!apt-get install openjdk-8-jdk-headless -qq > /dev/null
+
+### Download and extract Spark (use the latest version; this is 3.5.6 with Hadoop 3)
+!wget -q https://downloads.apache.org/spark/spark-3.5.6/spark-3.5.6-bin-hadoop3.tgz
+!tar xf spark-3.5.6-bin-hadoop3.tgz
+
+### Install PySpark and findspark (helps locate Spark)
+!pip install -q pyspark findspark duckdb  # duckdb for your script
+
+### Set environment variables
+import os
+os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
+os.environ["SPARK_HOME"] = "/content/spark-3.5.6-bin-hadoop3"
+
+### Initialize findspark
+import findspark
+findspark.init()
+
+
+## Hadoop
+
+!wget https://downloads.apache.org/hadoop/common/hadoop-3.4.2/hadoop-3.4.2.tar.gz
+!tar -xzvf hadoop-3.4.2.tar.gz && cp -r hadoop-3.4.2/ /usr/local/
+
+
+JAVA_HOME = !readlink -f /usr/bin/java | sed "s:bin/java::"
+java_home_text = JAVA_HOME[0]
+java_home_text_command = f"$ {JAVA_HOME[0]} "
+!echo export JAVA_HOME=$java_home_text >>/usr/local/hadoop-3.4.2/etc/hadoop/hadoop-env.sh
+
+# Set environment variables
+import os
+os.environ['HADOOP_HOME']="/usr/local/hadoop-3.4.2"
+os.environ['JAVA_HOME']=java_home_text
+
+!alias hadoop="/usr/local/hadoop-3.4.2/bin/hadoop"
+!alias hdfs="/usr/local/hadoop-3.4.2/bin/hdfs"
+!source ~/.bashrc   # or source ~/.zshrc
+!sudo ln -s /usr/local/hadoop-3.4.2/bin/hadoop /usr/local/bin/hadoop
+!sudo ln -s /usr/local/hadoop-3.4.2/bin/hdfs /usr/local/bin/hdfs
+!hadoop
+!hdfs
+## Airflow
+
+pip install apache-airflow
+
+airflow db init
+
+airflow webserver -p 8080 &
+airflow scheduler &
+
+## Ngrok
+
+## MinIO
+### Client
+```bash
+pip install minio
+```
+### Server
+# Install MinIO binary
+!wget https://dl.min.io/server/minio/release/linux-amd64/minio
+!chmod +x minio
+!mkdir -p ~/minio-data
+
+import os
+os.environ['MINIO_ROOT_USER'] = 'username'
+os.environ['MINIO_ROOT_PASSWORD'] = 'username_password'
+
+!./minio server ~/minio-data --address ":12390" --console-address ":12391" &

docs/install_airflow.md

@@ -0,0 +1,121 @@
+# Installing and Setting Up Apache Airflow
+
+This guide provides detailed instructions for installing and configuring Apache Airflow with support for asynchronous tasks, Celery, PostgreSQL, and Kubernetes. The steps below ensure a proper setup for running Airflow, initializing its database, creating an admin user, and starting the scheduler and webserver. This setup is suitable for a local development environment or a scalable production setup with the specified backends.
+
+## Prerequisites
+Before proceeding, ensure you have the following:
+- **Python 3.12**: Airflow 2.10.3 is compatible with Python 3.12, as specified in the constraint file.
+- **pip**: The Python package manager to install Airflow and its dependencies.
+- **PostgreSQL**: If using PostgreSQL as the metadata database (recommended for production).
+- **Celery**: For distributed task execution (optional, included in the installation).
+- **Kubernetes**: For running Airflow in a Kubernetes cluster (optional, included in the installation).
+- **Sufficient permissions**: To create directories and run background processes.
+- **Virtual environment** (recommended): To isolate dependencies. Create one with:
+  ```bash
+  python -m venv venv
+  source venv/bin/activate  # On Windows: venv\Scripts\activate
+  ```
+
+## Installation Steps
+
+### 1. Install Apache Airflow
+Install Airflow for version 2.10.3.
+
+```bash
+pip install apache-airflow==2.10.3
+```
+
+### 2. Set Up the Airflow Home Directory
+Airflow requires a home directory to store its configuration, logs, and DAGs. The following Python script sets the `AIRFLOW_HOME` environment variable and creates the directory if it doesn't exist.
+
+```python
+import os
+import time
+
+# Ensure environment
+os.environ['AIRFLOW_HOME'] = '<your_project_path>/airflow'
+os.makedirs('airflow', exist_ok=True)
+```
+
+Replace `<your_project_path>` with the absolute path to your project directory (e.g., `/home/user/BTC-USDT-ETL-Pipeline`). For example:
+```python
+os.environ['AIRFLOW_HOME'] = '/home/user/BTC-USDT-ETL-Pipeline/airflow'
+```
+
+This script ensures the `airflow` directory is created in your project path to store Airflow's configuration files, logs, and SQLite database (if not using PostgreSQL).
+
+### 3. Initialize the Airflow Database
+Initialize the Airflow metadata database, which stores DAG runs, task instances, and other metadata. This step is required before starting the scheduler or webserver.
+
+```bash
+# Re-init the database (resets metadata but keeps DAGs if any)
+airflow db init
+```
+
+**Note**:
+- This command creates a default `airflow.cfg` configuration file in `AIRFLOW_HOME`.
+- If using PostgreSQL, ensure the database is running and update the `sql_alchemy_conn` in `airflow.cfg` to point to your PostgreSQL instance (e.g., `postgresql+psycopg2://user:password@localhost:5432/airflow`).
+- Running `airflow db init` resets metadata but preserves any DAGs in the `dags` folder.
+
+### 4. Create an Admin User
+The Airflow webserver requires at least one admin user for login. Create an admin user with the following command:
+
+```bash
+# Create admin user (critical—webserver needs this for login)
+airflow users create \
+    --username admin \
+    --firstname Admin \
+    --lastname User \
+    --role Admin \
+    --email admin@example.com \
+    --password admin
+```
+
+This command creates a user with:
+- Username: `admin`
+- Password: `admin` (change this in production for security)
+- Role: `Admin` (grants full access to the Airflow UI)
+
+To verify the user was created successfully, list all users:
+
+```bash
+# Verify user creation
+airflow users list
+```
+
+### 5. Start the Airflow Scheduler
+The scheduler is responsible for scheduling and executing DAGs. Start it in the background using `nohup` to ensure it continues running.
+
+```bash
+# Start scheduler first (it needs DB)
+nohup airflow scheduler > airflow/scheduler.log 2>&1 &
+```
+
+**Notes**:
+- The scheduler requires the database to be initialized first.
+- Logs are redirected to `scheduler.log` in the specified directory.
+- Replace `airflow` with your `AIRFLOW_HOME` path if different.
+
+### 6. Start the Airflow Webserver
+The webserver provides the Airflow UI for managing DAGs, viewing task logs, and monitoring runs. Start it on port 8081 (or another port if needed).
+
+```bash
+airflow webserver --port 8081 > airflow/airflow.log 2>&1 &
+```
+
+**Notes**:
+- The webserver runs on `http://localhost:8081` by default.
+- Logs are redirected to `airflow.log` in the `AIRFLOW_HOME` directory.
+- Access the UI by navigating to `http://localhost:8081` in your browser and logging in with the admin credentials (username: `admin`, password: `admin`).
+
+## Additional Notes
+- **Configuration**: After running `airflow db init`, review and modify `airflow.cfg` in the `AIRFLOW_HOME` directory to customize settings (e.g., executor type, database connection, or Celery broker).
+- **Celery Setup**: If using the Celery executor, ensure a message broker (e.g., Redis or RabbitMQ) is running and configured in `airflow.cfg`.
+- **Kubernetes Executor**: For Kubernetes, configure the Kubernetes executor in `airflow.cfg` and ensure your Kubernetes cluster is accessible.
+- **Security**: Change the default admin password and secure the database connection in production environments.
+- **Logs**: Check `scheduler.log` and `airflow.log` for troubleshooting.
+
+## Next Steps
+- Place your DAGs in the `AIRFLOW_HOME/dags` folder to start defining workflows.
+- Explore the Airflow UI to monitor and manage your DAGs.
+- Refer to the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/) for advanced configurations.

docs/install_minio_server.md

@@ -0,0 +1,59 @@
+# MinIO Server Setup Guide
+
+This guide provides step-by-step instructions to set up and run a MinIO server on a Linux system.
+
+## Prerequisites
+- Python 3.x installed
+- Required Python packages: `python-dotenv`, `wget` (install using `pip install python-dotenv wget`)
+- A `minio.env` environment file
+- Administrative privileges for file permissions and port usage
+- Free ports for MinIO API (default: 9000) and WebUI (default: 9001)
+
+## Setup Instructions
+
+### 1. Download and Prepare MinIO Binary
+Run the following commands to download the MinIO server binary and set up the data directory:
+
+```bash
+wget https://dl.min.io/server/minio/release/linux-amd64/minio
+chmod +x minio
+mkdir -p ~/minio-data
+mkdir -p ~/minio-logs
+```
+
+### 2. Configure Environment Variables
+Create a `minio.env` file in the same directory as the MinIO binary with the following content, replacing placeholders with your desired values:
+
+```
+MINIO_ROOT_USER=<your_username>
+MINIO_ROOT_PASSWORD=<your_password>
+MINIO_HOST=localhost:<minio_port>
+MINIO_CONSOLE_ADDRESS=localhost:<minio_web_port>
+```
+
+- `<your_username>`: Choose a secure username for the MinIO admin account.
+- `<your_password>`: Use a strong password (at least 8 characters).
+- Ensure ports `<minio_port>` (API) and `<minio_web_port>` (WebUI) are free, or update them to available ports.
+
+### 3. Start the MinIO Server
+Run the following command to start the MinIO server in the background, using the environment variables from `minio.env`:
+
+```bash
+export MINIO_ROOT_USER=<your_username>
+export MINIO_ROOT_PASSWORD=<your_password>
+MINIO_ROOT_USER=$MINIO_ROOT_USER MINIO_ROOT_PASSWORD=$MINIO_ROOT_PASSWORD \
+./minio server ~/minio-data --address :<minio_port> --console-address :<minio_web_port> > ~/minio-logs/minio_server.log 2>&1 &
+```
+
+- This command exports environment variables and starts the MinIO server.
+- Logs are saved to `~/minio-logs/minio_server.log` for troubleshooting.
+
+### 4. Access MinIO
+- **API Access**: Connect to `http://localhost:<minio_port>` for programmatic access.
+- **WebUI Access**: Open `http://localhost:<minio_web_port>` in a browser and log in with `<your_username>` and `<your_password>`.
+
+## Notes
+- **Stopping the Server**: To stop the MinIO server, find its process ID using `ps aux | grep minio` and terminate it with `kill <pid>`.
+- **Port Conflicts**: If ports `<minio_port>` or `<minio_web_port>` are in use, modify `MINIO_ADDRESS` and `MINIO_CONSOLE_ADDRESS` in `minio.env` to use different ports.
+- **Security**: Store `minio.env` securely and avoid exposing sensitive credentials.
+- **Data Directory**: The `~/minio-data` directory stores MinIO buckets and objects. Ensure it has sufficient disk space.

docs/install_spark.md

@@ -0,0 +1,27 @@
+# Install Apache Spark
+
+This guide provides step-by-step instructions to download, install, and configure Apache Spark 3.5.6 with Hadoop 3 support on a Linux-based system. Apache Spark is a powerful open-source data processing engine designed for big data and machine learning workloads. The following commands will help you set up Spark and configure the environment variables to run Spark applications, including PySpark with Python 3.
+
+## Prerequisites
+- A Linux-based operating system (e.g., Ubuntu, CentOS).
+- `wget` and `tar` utilities installed.
+- `sudo` privileges for moving files to system directories.
+- Python 3 installed (for PySpark).
+
+## Installation Steps
+
+```bash
+# Download Apache Spark 3.5.6 with Hadoop 3 support
+wget https://downloads.apache.org/spark/spark-3.5.6/spark-3.5.6-bin-hadoop3.tgz
+
+# Extract the downloaded tarball
+tar -xzf spark-3.5.6-bin-hadoop3.tgz
+
+# Move the extracted folder to /opt/spark
+sudo mv spark-3.5.6-bin-hadoop3 /opt/spark
+
+# Set environment variables for Spark
+export SPARK_HOME=/opt/spark
+export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
+export PYSPARK_PYTHON=python3
+```

docs/visualize_data.md

@@ -0,0 +1,16 @@
+# Analytics & Visualization
+
+## DuckDB
+
+Run ad-hoc queries using python to get `output_csv_path`:
+
+```bash
+python components/duckdb2csv.py
+```
+
+## LockerStudio
+
+Upload `analytics/financial_data.csv` to Google Looker and create Report.
+
+## Looker
+You can see report at: https://lookerstudio.google.com/reporting/d12e8138-ffdb-40ac-a7fd-9fa986464f54/page/YtFbF/edit

duckdb_databases/financial_data.db

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20b333763c7dd6bdddd4eb0e88b7eb427faa464a213bc024fcd6cb5001e92bdb
+size 536576

evaluation/.gitignore

@@ -0,0 +1 @@
+*.csv