Why use Prompt Engineering to trigger AI control operations instead of using function calls?
Prompt engineering places very high demands on a model's instruction following ability , whereas function calling, once trained specifically, achieves more precise invocation and broader generalization capabilities.
https://github.com/gitker/-GELab-Zero I change the Prompt engineering to function call , it seems work well with qwen3 vl 30b a3b instruct
This isn’t prompt engineering. We trained the model using this format, and if we change it, we can’t guarantee there will be no performance degradation.
At the very beginning we did consider using tool calls or JSON as the output format, but we chose this format mainly to save tokens. In the future, we’ll try injecting the ability to follow different output formats into the model.
This isn’t prompt engineering. We trained the model using this format, and if we change it, we can’t guarantee there will be no performance degradation.
At the very beginning we did consider using tool calls or JSON as the output format, but we chose this format mainly to save tokens. In the future, we’ll try injecting the ability to follow different output formats into the model.
I tested this model and found that it often fails to output the format specified in the prompts, leading to errors during Python execution. When I replaced UI operations with function calls and connected them to other multimodal models that support function calls, format errors rarely occurred, and UI operations were more precise. I believe function calls are a more generalizable ability. If the model is trained to operate UIs through function calls, it would be much easier to modify the output format later as needed.
It’s essentially the same thing. In the GitHub code, the parser is responsible for parsing the model’s output into a structured dict. If needed, you can add a wrapper to do some additional conversion.
prompt engineering and funciton calls are the same . if you use the openai format or langchain or whatever you are just not seeing the function( jsons ) ... but its there .
when you extract the calls using openai or langchain etc , when they submit the call they also add a prompt to prompt the model to return in a specific format ...
so when you train a model you generaly consider all these methods ... and it can depend on the data available .. in many training data we do not even have the functions to call , we only have the extracted json ... so we can indeed use any method as the output is the methodology for the give input , so with training if you want to extract these calls , or later create a funciton caller , if you use open ai or langchain it will not matter as they will be injecting thier prompt !
so if you dont find your funciton cal in the openai return ( tool calls or funciton calls ) it should not matter as it only means that with prompt enginerring you will have to specfy the output style you wish the model to return , ie surrounded in xml tags .. so you can parse the raw respones yourself ... so your funciton caller will use both method the potential library your using and your personal method !
in text models you can specifcy any json or xml output format even yaml etc so its down to you ! you ould be using pydantic ( which actually does not truly care about the ouput as they also use templares and extractors so it would seem the model is outputting great objectes but they are not !
the pydantic method is usefull but not correct ! ... but the cobining of the methods above is (even pydantic add to your prompt ! )
so we find that the more librays and steps your prompt goes through the more potential injections which you have no control over hence raw is best !
If you train a model to use tools—and through tools to manipulate the UI—its generalization ability is higher and it is less prone to errors, because using tools becomes, to some extent, a core capability of the model.
If, instead, you train the model to output a specific format to operate the UI, then it can only operate the UI, and this kind of training will, to a certain degree, impair the model’s foundational capabilities.
If you train a model to use tools—and through tools to manipulate the UI—its generalization ability is higher and it is less prone to errors, because using tools becomes, to some extent, a core capability of the model.
If, instead, you train the model to output a specific format to operate the UI, then it can only operate the UI, and this kind of training will, to a certain degree, impair the model’s foundational capabilities.
I think ,
that you need to train all different methods !~
In the begining i was training models with various datasets which produced a varity of output shapes ! even chain of thought , forest of thoughts etc ... some very good datasets out there ! but i did not see the model apating to the outputs that i expected it to produce despite traiing it until it did generalize ? ... or indicated that it was trained ... i never realised that to actually change the behaviour of the model it needed grpo ! ...... now we can design any form of output style and it will make them !
so when you do mass traiing then for sepecfic behaviours you need to do the reward training for the new output style ! .. then it will take 100 percnent ie the deepseek R1 thinker !
so we could make a explainer instead of a thinker ! ie th model always give a big explanation for the answer instead of a massive pre thought ! ...
so we train generally first ... then we lock in the desired beaviour or leave it raw for the next finetuner ! who will grpo the style of response they desire .. and it will converge quickly as it has been pretrained on the n=method !
ie when i attempted to make the deepthinker it took the training very quickly as it had been trained before ! ( but never added the beaviour it was embedded deep !)
so finetuning should be very easy if you train on many taks types !
Now i can train on sparce data and mad bits of meta data ! which make no sense to me but the model accepts it ! .. and later it gains knowledge !
these methods allow for intergrating potential keywords for your responses and task types ! like return the reponse with tool calls in the shpape of ttol call and reasoning in the shape of .. and artifacts in the shape of .... so your UI can extract the various components easily !
just take all content as the response and remove and extract the components leaving the actual response behind
its all methodologies bro !