Indentation error

Where is this bracket that HF is finding?

Added please to the placeholder text.

Syntax error, bracket on line 343

Updated how the gradio looks, and doubled down on the system prompt to ensure that the system refers to the documents as its knowledge base.

Updated the system prompt a bit.

Called results ranked_results

Updated the reranker so that it stops breaking the llm section.

Reducing the batch size to 1 - it'll be slower, but at least this way the model output doesn't collapse.

Updated the reranker function to push the rerank model to the cpu to save on space instead of having it eat up the very limited GPU space that we are working with. Aso reduced the batch type to 8.

Forgot that I called the function reranker_v1.

Added the rerankers package for the reranking package.

Added the reranker to improve the quality of the nodes passed on to the query engine.

Create reranker_v1.py

It was the docstring with the indentation error lol.

Indentation error.

Indent error fix

Forgot to update the package imports to bring in the extractor for the log handler

Changed the temperature to 0.5, and added a function that will extract the condensed question for analysis.

Change the temp to 1

Changed temperature

Update app.py

Added a prompt helper to help manage the tokens, reduced the summary size to 800

Memory issues solved

I now have more GPU, therefore I have now reduced the GPU utililisation to 0.8

Removed the quantization line - that's crashed vLLM.

Increase the max model length, and corrected the quantisation to awq_merlin.

Added a background wait to counter the problem

Changed max model length to 3600 to improve hte KV cache issue..

Reduced the time in the wait for vLLM fnuction.

Reduced max model length, and increased gpu utilisation to 0.9

Specified chat mode, and made sure that the message was streamed for a nice UI action.

Reduced the GPU utilization and specified the quantization method.

Added a memory buffer, and moved the wait llm function to the main bit for gradio.

Checking something

Update vllm_server.py

Changed the GPU utilisation to 0.95

I have added some server specifics because the gradio bit isn't starting up.

Upgraded the max model length to 8092.

Changed the model from FP4 to AWQ

Changed the model from Instruct FP4 to AWQ

Decreased the maximum model lenght to 3408.

Moved the embedding model to the CPU. This will allow me to have more space on the GPU for the LLM.

Changed the time from 5 to 20 seconds.

Switched the model from Llama 3.3-70B to Llama-3.3-70B-Instruct-FP4.

Switching to a pre-quantised version of llama 3.3-70B sourced from Nvidia.

Added missing library socket.

Rectified indentation error on line 47.

Added start server and wait for server

Updated vllm_server to include a wait for vllm portion that ensures that the model is up before the chat section loads.