topguy commited on
Commit
f9b4d61
·
1 Parent(s): 6e44555

feat: Add README and CLI application

Browse files

Introduces a comprehensive README.md with installation and usage instructions for both the web and CLI interfaces.
Adds cli_app.py for command-line log processing, interacting with the Gradio app's API.
Updates app.py to expose API endpoints for filter loading and application, and includes filter reordering functionality in the UI.
Updates design.md to reflect the new CLI application and API endpoints.

Files changed (4) hide show
  1. README.md +120 -0
  2. app.py +63 -3
  3. cli_app.py +61 -0
  4. design.md +24 -4
README.md ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # LogViewer
2
+
3
+ LogViewer is a versatile tool designed for efficient viewing and filtering of log files. It offers both a user-friendly web interface built with Gradio and a powerful command-line interface (CLI) for automated processing.
4
+
5
+ ## Features
6
+
7
+ * **Web Interface (Gradio):**
8
+ * Upload and display log files.
9
+ * Apply multiple filters (include/exclude text, include/exclude regex).
10
+ * Case-sensitive filtering option.
11
+ * Reorder applied filters to control processing order.
12
+ * Save and load filter configurations (JSON format).
13
+ * Download filtered log content.
14
+ * **Command-Line Interface (CLI):**
15
+ * Process log files using pre-defined filter configurations.
16
+ * Automate log analysis workflows.
17
+
18
+ ## Installation
19
+
20
+ To set up LogViewer on your local machine, follow these steps:
21
+
22
+ ### Prerequisites
23
+
24
+ * Python 3.8 or higher
25
+ * pip (Python package installer)
26
+ * `venv` (Python's built-in virtual environment module)
27
+
28
+ ### Steps
29
+
30
+ 1. **Clone the repository:**
31
+ ```bash
32
+ git clone https://github.com/your-username/LogViewer.git
33
+ cd LogViewer
34
+ ```
35
+ *(Note: Replace `your-username` with the actual GitHub username once the repository is created.)*
36
+
37
+ 2. **Create and activate a virtual environment:**
38
+ It's highly recommended to use a virtual environment to manage project dependencies.
39
+ ```bash
40
+ python -m venv venv
41
+ # On Linux/macOS:
42
+ source venv/bin/activate
43
+ # On Windows:
44
+ .\venv\Scripts\activate
45
+ ```
46
+
47
+ 3. **Install dependencies:**
48
+ ```bash
49
+ pip install -r requirements.txt
50
+ ```
51
+
52
+ ## Usage
53
+
54
+ ### Web Interface (Gradio App)
55
+
56
+ To launch the interactive web interface:
57
+
58
+ 1. **Start the application:**
59
+ Ensure your virtual environment is activated.
60
+ ```bash
61
+ python app.py
62
+ ```
63
+ 2. **Access the UI:**
64
+ Open your web browser and navigate to the URL displayed in your terminal (usually `http://127.0.0.1:7860`).
65
+
66
+ **How to use:**
67
+ * **Upload Log File:** Click the "Upload Log File" button to select your log file.
68
+ * **Add Filters:** Choose a "Filter Type" (e.g., "Include Text", "Exclude Regex"), enter a "Filter Value", and check "Case Sensitive" if needed. Click "Add Filter".
69
+ * **Manage Filters:** Applied filters will appear in the "Applied Filters" list. You can select a filter and use "Remove Selected Filter", "Move Up", or "Move Down" to adjust them.
70
+ * **Save/Load Filters:** Use "Save Filters" to download your current filter configuration as a JSON file, or "Load Filters (.json)" to upload a previously saved configuration.
71
+ * **Save Filtered Log:** After applying filters, click "Save Filtered Log" to download the processed log content.
72
+
73
+ ### Command-Line Interface (CLI)
74
+
75
+ To process log files using the CLI, the Gradio web application (`app.py`) **must be running** in the background, as the CLI interacts with its API.
76
+
77
+ 1. **Ensure the Gradio app is running:**
78
+ Open a separate terminal, activate your virtual environment, and run:
79
+ ```bash
80
+ python app.py
81
+ ```
82
+ Keep this terminal open.
83
+
84
+ 2. **Run the CLI tool:**
85
+ In a new terminal, activate your virtual environment and use `cli_app.py`:
86
+ ```bash
87
+ python cli_app.py <log_file_path> <filter_file_path> [-o <output_file_path>]
88
+ ```
89
+ * `<log_file_path>`: The path to the input log file you want to process.
90
+ * `<filter_file_path>`: The path to a JSON file containing your filter configurations (e.g., `filters.json` saved from the web UI).
91
+ * `-o <output_file_path>` (optional): The path where the filtered log content will be saved. If not provided, a default name will be generated (e.g., `your_log_filters_filtered.txt`).
92
+
93
+ **Example:**
94
+
95
+ ```bash
96
+ python cli_app.py my_application.log my_filters.json -o processed_log.txt
97
+ ```
98
+
99
+ ## Filter File Format
100
+
101
+ Filter configurations are saved and loaded as JSON files. Each filter is an object within a list, with the following structure:
102
+
103
+ ```json
104
+ [
105
+ {
106
+ "type": "Include Text",
107
+ "value": "ERROR",
108
+ "case_sensitive": true
109
+ },
110
+ {
111
+ "type": "Exclude Regex",
112
+ "value": "DEBUG|INFO",
113
+ "case_sensitive": false
114
+ }
115
+ ]
116
+ ```
117
+
118
+ * `type`: Can be "Include Text", "Exclude Text", "Include Regex", or "Exclude Regex".
119
+ * `value`: The string or regex pattern for the filter.
120
+ * `case_sensitive`: A boolean (`true` or `false`) indicating whether the filter should be case-sensitive.
app.py CHANGED
@@ -13,12 +13,38 @@ def remove_filter(filters, selected_filter):
13
  if len(parts) == 3:
14
  filter_type, filter_value, case_str = parts
15
  case_sensitive = case_str == "Case Sensitive: True"
16
- for f in filters:
17
  if f["type"] == filter_type and f["value"] == filter_value and f["case_sensitive"] == case_sensitive:
18
- filters.remove(f)
19
  break
20
  return filters, None
21
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
  def apply_filters(file, filters):
23
  if file is not None:
24
  with open(file.name) as f:
@@ -67,7 +93,7 @@ def save_filtered_log(log_content):
67
  return "filtered_log.txt"
68
  return None
69
 
70
- with gr.Blocks(theme=gr.themes.Soft()) as demo:
71
  filters_state = gr.State([])
72
 
73
  gr.Markdown("## Log File Viewer")
@@ -93,6 +119,8 @@ with gr.Blocks(theme=gr.themes.Soft()) as demo:
93
  with gr.Row():
94
  applied_filters_list = gr.Radio(label="Applied Filters", interactive=True)
95
  remove_filter_button = gr.Button("Remove Selected Filter")
 
 
96
 
97
  with gr.Row():
98
  save_filtered_log_button = gr.Button("Save Filtered Log")
@@ -127,6 +155,34 @@ with gr.Blocks(theme=gr.themes.Soft()) as demo:
127
  outputs=text_output
128
  )
129
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  save_filters_button.click(
131
  save_filters,
132
  inputs=filters_state,
@@ -159,6 +215,10 @@ with gr.Blocks(theme=gr.themes.Soft()) as demo:
159
  outputs=gr.File(label="Download Filtered Log")
160
  )
161
 
 
 
 
 
162
  if __name__ == "__main__":
163
  demo.launch()
164
  demo.launch()
 
13
  if len(parts) == 3:
14
  filter_type, filter_value, case_str = parts
15
  case_sensitive = case_str == "Case Sensitive: True"
16
+ for i, f in enumerate(filters):
17
  if f["type"] == filter_type and f["value"] == filter_value and f["case_sensitive"] == case_sensitive:
18
+ filters.pop(i)
19
  break
20
  return filters, None
21
 
22
+ def move_filter_up(filters, selected_filter):
23
+ if selected_filter:
24
+ parts = selected_filter.split(" - ")
25
+ if len(parts) == 3:
26
+ filter_type, filter_value, case_str = parts
27
+ case_sensitive = case_str == "Case Sensitive: True"
28
+ for i, f in enumerate(filters):
29
+ if f["type"] == filter_type and f["value"] == filter_value and f["case_sensitive"] == case_sensitive:
30
+ if i > 0:
31
+ filters[i], filters[i-1] = filters[i-1], filters[i]
32
+ break
33
+ return filters, selected_filter
34
+
35
+ def move_filter_down(filters, selected_filter):
36
+ if selected_filter:
37
+ parts = selected_filter.split(" - ")
38
+ if len(parts) == 3:
39
+ filter_type, filter_value, case_str = parts
40
+ case_sensitive = case_str == "Case Sensitive: True"
41
+ for i, f in enumerate(filters):
42
+ if f["type"] == filter_type and f["value"] == filter_value and f["case_sensitive"] == case_sensitive:
43
+ if i < len(filters) - 1:
44
+ filters[i], filters[i+1] = filters[i+1], filters[i]
45
+ break
46
+ return filters, selected_filter
47
+
48
  def apply_filters(file, filters):
49
  if file is not None:
50
  with open(file.name) as f:
 
93
  return "filtered_log.txt"
94
  return None
95
 
96
+ with gr.Blocks(theme=gr.themes.Soft(), css="#log_content textarea { font-family: monospace; }") as demo:
97
  filters_state = gr.State([])
98
 
99
  gr.Markdown("## Log File Viewer")
 
119
  with gr.Row():
120
  applied_filters_list = gr.Radio(label="Applied Filters", interactive=True)
121
  remove_filter_button = gr.Button("Remove Selected Filter")
122
+ move_up_button = gr.Button("Move Up")
123
+ move_down_button = gr.Button("Move Down")
124
 
125
  with gr.Row():
126
  save_filtered_log_button = gr.Button("Save Filtered Log")
 
155
  outputs=text_output
156
  )
157
 
158
+ move_up_button.click(
159
+ move_filter_up,
160
+ inputs=[filters_state, applied_filters_list],
161
+ outputs=[filters_state, applied_filters_list]
162
+ ).then(
163
+ update_filter_list,
164
+ inputs=filters_state,
165
+ outputs=applied_filters_list
166
+ ).then(
167
+ apply_filters,
168
+ inputs=[file_input, filters_state],
169
+ outputs=text_output
170
+ )
171
+
172
+ move_down_button.click(
173
+ move_filter_down,
174
+ inputs=[filters_state, applied_filters_list],
175
+ outputs=[filters_state, applied_filters_list]
176
+ ).then(
177
+ update_filter_list,
178
+ inputs=filters_state,
179
+ outputs=applied_filters_list
180
+ ).then(
181
+ apply_filters,
182
+ inputs=[file_input, filters_state],
183
+ outputs=text_output
184
+ )
185
+
186
  save_filters_button.click(
187
  save_filters,
188
  inputs=filters_state,
 
215
  outputs=gr.File(label="Download Filtered Log")
216
  )
217
 
218
+ # Expose functions as API endpoints for external clients
219
+ demo.api_open(fn=load_filters, api_name="load_filters")
220
+ demo.api_open(fn=apply_filters, api_name="apply_filters")
221
+
222
  if __name__ == "__main__":
223
  demo.launch()
224
  demo.launch()
cli_app.py ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from gradio_client import Client
2
+ import os
3
+ import json
4
+ import argparse
5
+
6
+ def run_log_processing(log_file_path: str, filter_file_path: str, output_file_path: str = None):
7
+ """
8
+ Processes a log file using a running Gradio application's API.
9
+
10
+ Args:
11
+ log_file_path (str): Path to the input log file.
12
+ filter_file_path (str): Path to the JSON filter file.
13
+ output_file_path (str, optional): Path to the output file.
14
+ Defaults to a generated name if not provided.
15
+ """
16
+ # Assuming your Gradio app is running on localhost:7860
17
+ # You might need to change this URL if your app is hosted elsewhere.
18
+ client = Client("http://localhost:7860/")
19
+
20
+ print(f"Loading filters from: {filter_file_path}")
21
+ # Call the load_filters function in your Gradio app
22
+ # Gradio client handles file uploads by passing the file path directly.
23
+ loaded_filters = client.predict(
24
+ fn_name="/load_filters",
25
+ inputs=[filter_file_path]
26
+ )
27
+ print(f"Loaded {len(loaded_filters)} filters.")
28
+
29
+ print(f"Applying filters to log file: {log_file_path}")
30
+ # Call the apply_filters function in your Gradio app
31
+ filtered_log_content = client.predict(
32
+ fn_name="/apply_filters",
33
+ inputs=[log_file_path, loaded_filters]
34
+ )
35
+ print("Filters applied.")
36
+
37
+ # Determine output file name if not provided
38
+ if output_file_path is None:
39
+ log_file_name = os.path.basename(log_file_path)
40
+ filter_file_name = os.path.basename(filter_file_path)
41
+
42
+ log_base, log_ext = os.path.splitext(log_file_name)
43
+ filter_base, filter_ext = os.path.splitext(filter_file_name)
44
+
45
+ output_file_path = f"{log_base}_{filter_base}_filtered.txt"
46
+
47
+ # Save the result to the output file
48
+ with open(output_file_path, "w") as f:
49
+ f.write(filtered_log_content)
50
+
51
+ print(f"Filtered log saved to: {output_file_path}")
52
+
53
+ if __name__ == "__main__":
54
+ parser = argparse.ArgumentParser(description="Process log files using a Gradio API.")
55
+ parser.add_argument("log_file", help="Path to the input log file.")
56
+ parser.add_argument("filter_file", help="Path to the JSON filter file.")
57
+ parser.add_argument("-o", "--output_file", help="Optional: Path to the output file. Defaults to a generated name.")
58
+
59
+ args = parser.parse_args()
60
+
61
+ run_log_processing(args.log_file, args.filter_file, args.output_file)
design.md CHANGED
@@ -25,6 +25,8 @@ This is the main application file that creates the user interface and handles al
25
  * A section to display and manage active filters:
26
  * A radio button group (`gr.Radio`) that lists the currently applied filters, showing their type, value, and case sensitivity.
27
  * A "Remove Selected Filter" button (`gr.Button`).
 
 
28
  * A "Save Filtered Log" button (`gr.Button`) to download the currently displayed filtered log content.
29
 
30
  * **State Management:**
@@ -33,6 +35,8 @@ This is the main application file that creates the user interface and handles al
33
  * **Core Logic:**
34
  * **`add_filter()`:** Adds a new filter dictionary to the `filters_state` list.
35
  * **`remove_filter()`:** Removes a filter from the `filters_state` list based on the user's selection.
 
 
36
  * **`apply_filters()`:** This is the main processing function. It reads the uploaded file and applies the sequence of filters stored in `filters_state` by calling the `filter_lines` utility function for each filter.
37
  * **`update_filter_list()`:** Generates a list of strings from the `filters_state` list to display it in the UI.
38
  * **`save_filters()`:** Saves the current `filters_state` to a JSON file.
@@ -43,10 +47,16 @@ This is the main application file that creates the user interface and handles al
43
  * Uploading a file triggers `apply_filters`.
44
  * Clicking "Add Filter" triggers `add_filter`, then `update_filter_list`, and finally `apply_filters`.
45
  * Clicking "Remove Selected Filter" triggers `remove_filter`, then `update_filter_list`, and finally `apply_filters`.
 
 
46
  * Clicking "Save Filters" triggers `save_filters`.
47
  * Uploading a file to "Load Filters" triggers `load_filters`, then `update_filter_list`, and finally `apply_filters`.
48
  * Clicking "Save Filtered Log" triggers `save_filtered_log`.
49
 
 
 
 
 
50
  ### 2.2. `filter_utils.py`
51
 
52
  This module provides the core filtering logic.
@@ -61,6 +71,16 @@ This module provides utilities for parsing and filtering log lines based on time
61
  * **`get_timestamp_range()`:** Extracts the earliest and latest timestamps from a list of log lines.
62
  * **`filter_by_time_range()`:** Filters log lines to include only those within a specified time range, based on parsed timestamps.
63
 
 
 
 
 
 
 
 
 
 
 
64
  ## 3. User Interaction Flow
65
 
66
  1. The user opens the application in their browser.
@@ -71,9 +91,9 @@ This module provides utilities for parsing and filtering log lines based on time
71
  6. To remove a filter, the user selects it from the "Applied Filters" radio group.
72
  7. The user then clicks the "Remove Selected Filter" button.
73
  8. The filter is removed from the list, and the log view is updated to reflect the change.
74
- 9. The user can save the current set of filters by clicking the "Save Filters" button, which will prompt a file download.
75
- 10. The user can load a previously saved set of filters by clicking the "Load Filters" button and selecting a JSON file.
76
-
77
- 11. The user can save the currently displayed filtered log content by clicking the "Save Filtered Log" button, which will prompt a file download.
78
 
79
  This design allows for a flexible and interactive way to analyze log files by building a chain of filters.
 
25
  * A section to display and manage active filters:
26
  * A radio button group (`gr.Radio`) that lists the currently applied filters, showing their type, value, and case sensitivity.
27
  * A "Remove Selected Filter" button (`gr.Button`).
28
+ * "Move Up" button (`gr.Button`) to move the selected filter up in the list.
29
+ * "Move Down" button (`gr.Button`) to move the selected filter down in the list.
30
  * A "Save Filtered Log" button (`gr.Button`) to download the currently displayed filtered log content.
31
 
32
  * **State Management:**
 
35
  * **Core Logic:**
36
  * **`add_filter()`:** Adds a new filter dictionary to the `filters_state` list.
37
  * **`remove_filter()`:** Removes a filter from the `filters_state` list based on the user's selection.
38
+ * **`move_filter_up()`:** Moves the selected filter up in the `filters_state` list.
39
+ * **`move_filter_down()`:** Moves the selected filter down in the `filters_state` list.
40
  * **`apply_filters()`:** This is the main processing function. It reads the uploaded file and applies the sequence of filters stored in `filters_state` by calling the `filter_lines` utility function for each filter.
41
  * **`update_filter_list()`:** Generates a list of strings from the `filters_state` list to display it in the UI.
42
  * **`save_filters()`:** Saves the current `filters_state` to a JSON file.
 
47
  * Uploading a file triggers `apply_filters`.
48
  * Clicking "Add Filter" triggers `add_filter`, then `update_filter_list`, and finally `apply_filters`.
49
  * Clicking "Remove Selected Filter" triggers `remove_filter`, then `update_filter_list`, and finally `apply_filters`.
50
+ * Clicking "Move Up" triggers `move_filter_up`, then `update_filter_list`, and finally `apply_filters`.
51
+ * Clicking "Move Down" triggers `move_filter_down`, then `update_filter_list`, and finally `apply_filters`.
52
  * Clicking "Save Filters" triggers `save_filters`.
53
  * Uploading a file to "Load Filters" triggers `load_filters`, then `update_filter_list`, and finally `apply_filters`.
54
  * Clicking "Save Filtered Log" triggers `save_filtered_log`.
55
 
56
+ * **API Endpoints:**
57
+ * `load_filters`: Exposed via `gr.API` to allow external clients to load filter sets.
58
+ * `apply_filters`: Exposed via `gr.API` to allow external clients to apply filters to log content.
59
+
60
  ### 2.2. `filter_utils.py`
61
 
62
  This module provides the core filtering logic.
 
71
  * **`get_timestamp_range()`:** Extracts the earliest and latest timestamps from a list of log lines.
72
  * **`filter_by_time_range()`:** Filters log lines to include only those within a specified time range, based on parsed timestamps.
73
 
74
+ ### 2.4. `cli_app.py`
75
+
76
+ This is a command-line interface (CLI) application that interacts with the running `app.py` Gradio service via its API.
77
+
78
+ * **Functionality:**
79
+ * Takes a log file path and a filter JSON file path as arguments.
80
+ * Optionally takes an output file path; otherwise, it generates one.
81
+ * Uses `gradio_client` to call the `load_filters` and `apply_filters` API endpoints of the Gradio app.
82
+ * Saves the filtered log content to the specified output file.
83
+
84
  ## 3. User Interaction Flow
85
 
86
  1. The user opens the application in their browser.
 
91
  6. To remove a filter, the user selects it from the "Applied Filters" radio group.
92
  7. The user then clicks the "Remove Selected Filter" button.
93
  8. The filter is removed from the list, and the log view is updated to reflect the change.
94
+ 9. The user can reorder filters by selecting a filter and clicking "Move Up" or "Move Down" buttons.
95
+ 10. The user can save the current set of filters by clicking the "Save Filters" button, which will prompt a file download.
96
+ 11. The user can load a previously saved set of filters by clicking the "Load Filters" button and selecting a JSON file.
97
+ 12. The user can save the currently displayed filtered log content by clicking the "Save Filtered Log" button, which will prompt a file download.
98
 
99
  This design allows for a flexible and interactive way to analyze log files by building a chain of filters.