updated to fix indexing error

Error:
Token indices sequence length is longer than the specified maximum sequence length for this model (5187 > 512).

This suggests that your input sequence length is exceeding the maximum allowable sequence length for the model. TAPAS, like many other transformer models, has a maximum input sequence length. In this case, it's 512 tokens.
IndexError: iloc cannot enlarge its target object

This is the error related to pandas, where it's not allowed to assign a value to an index that doesn't exist in the DataFrame.

Attempted Fix:
I've added checks to ensure that token count doesn't exceed the model's limit.
I've added debugging print statements to help you understand what's going on.

app.py

@@ -10,9 +10,20 @@ model = AutoModelForTableQuestionAnswering.from_pretrained("google/tapas-large-f
 
 def ask_llm_chunk(chunk, questions):
     chunk = chunk.astype(str)
-    inputs = tokenizer(table=chunk, queries=questions, padding="max_length", return_tensors="pt")
-    if inputs["input_ids"].shape[1] > 1024:
+    
+    # Debugging statement to print chunk shape
+    print("Chunk shape:", chunk.shape)
+    print("Sample data:", chunk.head())
+    
+    # Count tokens
+    token_count = len(tokenizer.tokenize(str(chunk) + " ".join(questions)))
+    print("Token count:", token_count)
+    
+    if token_count > 512:
+        print("Warning: Token count exceeds maximum allowable sequence length.")
         return ["Token limit exceeded for chunk"] * len(questions)
+        
+    inputs = tokenizer(table=chunk, queries=questions, padding="max_length", return_tensors="pt")
     outputs = model(**inputs)
     predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
         inputs,
@@ -40,8 +51,7 @@ def summarize_map_reduce(data, questions):
     for chunk in dataframe_chunks:
         chunk_answers = ask_llm_chunk(chunk, questions)
         all_answers.extend(chunk_answers)
-    aggregated_answers = all_answers
-    return aggregated_answers
+    return all_answers  # For now, simply returning the answers for each chunk
 
 st.title("TAPAS Table Question Answering")