Evals
All Evals
Get all Evaluatorsfrom orq_ai_sdk import Orq
import os
with Orq(
api_key=os.getenv("ORQ_API_KEY", ""),
) as orq:
res = orq.evals.all(limit=10)
# Handle response
print(res)
Show Parameters
Show Parameters
A limit on the number of objects to be returned. Limit can range between 1 and 50, and the default is 10
A cursor for use in pagination.
starting_after is an object ID that defines your place in the list. For instance, if you make a list request and receive 20 objects, ending with 01JJ1HDHN79XAS7A01WB3HYSDB, your subsequent call can include after=01JJ1HDHN79XAS7A01WB3HYSDB in order to fetch the next page of the list.Create an Eval
Create an Evaluatorfrom orq_ai_sdk import Orq
import os
with Orq(
api_key=os.getenv("ORQ_API_KEY", ""),
) as orq:
res = orq.evals.create(request={
"output_type": "string",
"type": "http_eval",
"url": "https://total-unit.name",
"method": "GET",
"headers": {
"key": "<value>",
"key1": "<value>",
"key2": "<value>",
},
"payload": {
"key": "<value>",
},
"path": "Default",
"description": "",
"key": "<key>",
})
# Handle response
print(res)
Show Parameters
Show Parameters
The request object to use for the request.
Update an Eval
Update an Evaluatorfrom orq_ai_sdk import Orq
import os
with Orq(
api_key=os.getenv("ORQ_API_KEY", ""),
) as orq:
res = orq.evals.update(id="<id>", path="Default")
# Handle response
print(res)
Show Parameters
Show Parameters
Delete an Eval
Delete an Evaluatorfrom orq_ai_sdk import Orq
import os
with Orq(
api_key=os.getenv("ORQ_API_KEY", ""),
) as orq:
orq.evals.delete(id="<id>")
# Use the SDK ...
Show Parameters
Show Parameters
The request object to use for the request.
Invoke an Eval
Invoke a Custom Evaluatorfrom orq_ai_sdk import Orq
import os
with Orq(
api_key=os.getenv("ORQ_API_KEY", ""),
) as orq:
res = orq.evals.invoke(id="<id>", messages=[
{
"role": "tool",
"content": [],
},
])
# Handle response
print(res)
Show Parameters
Show Parameters
Evaluator ID
Evaluators Versions
Returns version history for a specific evaluatorfrom orq_ai_sdk import Orq
import os
with Orq(
api_key=os.getenv("ORQ_API_KEY", ""),
) as orq:
res = orq.evals.get_v2_evaluators_id_versions(id="<id>", limit=10)
# Handle response
print(res)
Show Parameters
Show Parameters
A limit on the number of objects to be returned. Limit can range between 1 and 50, and the default is 10
A cursor for use in pagination.
starting_after is an object ID that defines your place in the list. For instance, if you make a list request and receive 20 objects, ending with 01JJ1HDHN79XAS7A01WB3HYSDB, your subsequent call can include after=01JJ1HDHN79XAS7A01WB3HYSDB in order to fetch the next page of the list.Show Response
Show Response