Skip to main content
In this cookbook, we chain two deployments to process financial data from image files, to create a workflow. First, we perform data extraction on a JPG file, converting unstructured receipt information into structured JSON. We then run an evaluator to verify that the output is valid JSON and check whether the tax amount has been correctly extracted. Next, the validated financial data is passed to a second deployment that summarizes the extracted information, providing clear, actionable insights. Separating these tasks improves accuracy by allowing each step to be optimized independently, and it enables granular checking at each stage. This ensures better validation, making it easier to diagnose errors and refine the process. By leveraging Orq’s capabilities, this workflow delivers a scalable and efficient approach to processing image-based receipts. To make things even easier, we’ve created this Google Colab file that you can copy and run straight away after replacing the API key—the deployment is already live and ready in the deployment section. Below, we’ll run through the code step by step for further explanation. Ready to unlock Orq’s magic? Sign up to get started and keep the process rolling! Step 1: Preparing the Environment Before diving into image processing, the necessary tools must be in place. Installing the Orq SDK is quick and straightforward, setting the stage for seamless integration. 
!pip install orq-ai-sdk

#import
import pandas as pd
from google.colab import auth
With the SDK installed, the focus shifts to setting up the client and preparing the workflow. Step 2: Setting Up the Orq Client The Orq client bridges your environment with Orq’s powerful APIs. By authenticating with an API key, it provides access to deployments that simplify data extraction from images. After you are logged into the platform, you can find your API key here. 
import os

from orq_ai_sdk import Orq

client = Orq(
  api_key=os.environ.get("ORQ_API_KEY", "your_api_key_here"),
)
Once connected, the client is ready to process image files for extraction. Step 3: Converting Images to Base64 To process images with Orq’s deployments, they must first be encoded into Base64 format. This section outlines how to process a folder of .jpg and .png files, preparing them for data extraction. To get you started, we’ve provided a Google Drive folder filled with .jpg files of receipts that you can copy and use to test and explore the workflow. 
import os
import base64

# Specify the folder containing image files
folder_path = '/content/drive/MyDrive/receipts_test'

# Get all .jpg and .png files from the folder
image_files = [file for file in os.listdir(folder_path) if file.endswith(('.jpg', '.png'))]

# List to store Base64-encoded data for each image
base64_images = []

# Iterate through image files and convert them to Base64
for image_file in image_files:
    file_path = os.path.join(folder_path, image_file)
    try:
        with open(file_path, 'rb') as img_file:
            # Encode the image to Base64
            base64_data = base64.b64encode(img_file.read()).decode('utf-8')
            base64_images.append(base64_data)
            print(f"Encoded {image_file} to Base64.")
    except Exception as e:
        print(f"Error processing {image_file}: {e}")

# Output the Base64-encoded data for each image
print("Base64-encoded images ready for processing.")
Step 4: Data Extraction Deployment With images in Base64 format, the final step is to send each encoded image to Orq’s DataExtraction_Receipts deployment.

​
Model Configuration in Orq

In this Orq model setup, we’re using Gemini-2.0-Flash-Experimental as the primary model for fast and efficient data extraction, with Claude-3.7-Sonnet as a heavier and more expensive model as fallback—just in case things don’t go as planned. This way, we ensure smooth processing without hiccups. We’ve set the temperature to 0.2, keeping things precise and predictable. Why? Because when dealing with financial data, we don’t want the model getting too creative—we need structured, reliable results that stick to the schema. A lower temperature helps keep responses on track, ensuring accurate extractions every time.

​
Prompt

To achieve accurate extraction, we use a well-defined prompt that provides clear instructions on identifying key financial details. It specifies exactly what information should be extracted—transaction date, vendor name, amounts (total, pre-tax, and tax), and payment details. By explicitly requesting tax type differentiation and category classification, the prompt ensures a granular and precise extraction. 
Extract the following key financial data from the receipt/invoice images:

1. Date: Transaction date in MM/DD/YYYY format
2. Vendor: Business or individual name providing goods/services
3. Total Amount: Complete transaction value with currency symbol
4. Pre-tax Amount: Subtotal before taxes
5. Tax Amount: All tax charges combined
6. Category: Appropriate expense classification (e.g., Dining, Transportation, Office Supplies, Utilities)
7. Payment Method: How the transaction was completed (e.g., Credit Card, Cash, Bank Transfer, Check)
8. Invoice/Receipt Number: Unique transaction identifier (if present)

Format all extracted information as structured JSON according to the provided schema. Ensure accuracy of monetary values and consistency in categorization.

​
Tools

To keep things structured, we use a JSON schema as our data blueprint. This schema acts as a quality control tool, ensuring that all extracted fields are correctly formatted and validated. Beyond just verification, this structured output makes the data programmatically accessible, allowing seamless integration into other systems, automated workflows, or financial analyses—without the need for manual intervention. 
{
  "type": "object",
  "properties": {
    "Date": {
      "type": "string",
      "description": "The date of the transaction in ISO 8601 format (YYYY-MM-DD)."
    },
    "Vendor_Name": {
      "type": "string",
      "description": "The name of the company or individual from whom the goods or services were purchased."
    },
    "Total_Amount": {
      "type": "number",
      "description": "The total amount spent, including any applicable taxes."
    },
    "Pre_Tax_Amount": {
      "type": "number",
      "description": "The total amount before taxes are applied."
    },
    "Tax_Amount": {
      "type": "number",
      "description": "The total amount after taxes are applied."
    },
    "Category": {
      "type": "string",
      "description": "An appropriate category for the expense (e.g., Travel, Food, Office Supplies)."
    },
    "Payment_Method": {
      "type": "string",
      "description": "The method of payment used (e.g., Credit Card, Cash, Bank Transfer)."
    },
    "Invoice_Number": {
      "type": "string",
      "description": "The unique identifier for the invoice, if available."
    }
  },
  "required": [
    "Date",
    "Vendor_Name",
    "Total_Amount",
    "Pre_Tax_Amount",
    "Tax_Amount",
    "Category",
    "Payment_Method",
    "Invoice_Number"
  ]
}
Step 5: Evaluation Configuration for JSON and Tax Check To ensure the integrity and completeness of the extracted data, we run two key evaluations. These can be configured in Deployment > Settings > Output Guardrails. JSON Check: Verifies that the extracted output follows the expected JSON schema, ensuring proper formatting and the presence of all required fields. Tax Check: Confirms whether the tax amount has been successfully extracted and, if applicable, whether multiple tax types are correctly identified. These evaluations help maintain data quality, flag potential issues early, and improve the reliability of downstream processing. Step 6: Run Deployment This code processes a set of Base64-encoded images by invoking Orq’s data extraction deployment to extract structured receipt information. The extracted data is parsed into JSON format and stored in extraction_results, making it ready for further processing, such as financial summarization or validation. 
import json

# Initialize an empty list to store extraction results
extraction_results = []

# Iterate through each Base64-encoded image and invoke the deployment
for base64_image in base64_images:
    try:
        # Construct the invocation payload
        generation = client.deployments.invoke(
            key="data_extraction_receipt_or_invoice",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"text": "Extract what is on the receipt", "type": "text"},
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": "data:image/png;base64," + base64_image
                            },
                        },
                    ],
                }
            ],
        )

        # Parse the response and append to extraction_results
        extracted_data = json.loads(generation.choices[0].message.content)
        extraction_results.append(extracted_data)

    except Exception as e:
        print(f"Error invoking deployment for an image: {e}")

# At this point, extraction_results is ready for the next deployment

​
Step 7: Financial Summarization Deployment

The second deployment summarizes the extracted financial data, providing a high-level overview of expenses. It highlights total spending, detects unusual charges, and identifies potential cost-saving opportunities, ensuring a clear and concise financial snapshot.

​
Model Configuration in Orq

For summarizing all that extracted receipt data, we’re using Claude 3.5 Haiku as our primary model—it’s fast and efficient. If it ever needs backup, Claude 3.7 Sonnet is ready to jump in as the fallback. We’ve set the temperature to 0.5, which keeps things balanced—structured enough to stay accurate, but still flexible enough to offer insightful takeaways about spending patterns and tax deductions. The Top P (0.7) and Top K (5) settings make sure responses stay focused and relevant, without the model going off on financial tangents.

​
Prompt

This prompt guides the model to act as an experienced financial analyst, focusing on expense management and receipt analysis. It goes beyond simple data extraction by requesting a detailed expense breakdown, identification of unusual charges, and an analysis of spending patterns. Additionally, it includes actionable insights, such as cost-saving recommendations, tax deduction opportunities, and compliance checks. By summarising totals for each category and flagging receipts that may need further review, this prompt ensures a comprehensive financial assessment, making it useful for both accounting and tax reporting. 
You are an experienced financial analyst specializing in expense management and receipt analysis. Please review the following receipts and provide:

- A detailed breakdown of all expenses by category
- Identification of any unusual or potentially erroneous charges
- Analysis of spending patterns and trends
- Recommendations for potential cost-saving opportunities
- Tax deduction possibilities based on the expenses shown
- Any compliance issues or missing information that should be addressed

Please include totals for each category and highlight any receipts that may require additional documentation for accounting or tax purposes.

​
Step 8: Passing Extracted Data to the Next Deployment

Here, we invoke the next deployment, using the JSON output from the previous extraction step as input. This ensures a seamless transition between deployments, allowing for structured financial summarization based on the extracted receipt data. 
import json

try:
    # Convert the combined JSON list to a JSON string
    combined_json_string = json.dumps(extraction_results)

    # Invoke the financial summarization deployment
    summarization = client.deployments.invoke(
        key="financial-analyst-summarizer",
        context={
            "environments": []
        },
        inputs={
            "json": combined_json_string
        },
        metadata={
            "custom-field-name": "custom-metadata-value"
        }
    )

    # Print the summarization result
    print(summarization.choices[0].message.content)

except Exception as e:
    print(f"Error invoking financial summarization deployment: {e}")
What’s Next? With this workflow, you now have a concrete example of how to configure and chain multiple deployments in Orq, transforming unstructured data into structured insights. Beyond receipt processing, this approach can be applied to a wide range of workflows, such as document parsing, customer feedback analysis, or automated compliance checks.
  • Scale and Adapt: Apply the same principles to process different data types, from text documents to audio transcripts or sensor data.
  • Optimize with Evaluations: Run targeted evaluations at key steps to ensure accuracy, detect anomalies, and refine model performance.
  • Automate Decision-Making: Use chained deployments to create intelligent workflows that extract, analyze, and act on data with minimal manual intervention.
By leveraging Orq’s flexible deployment framework, businesses can design custom AI-driven pipelines and integrate them in their software product to fuel their Gen AI based features, unlocking new efficiencies across various domains.