Heddle Logoeddle

Language. Orchestration Engine. Built to eliminate complexity.

Heddle is a strictly-typed Domain-Specific Language (DSL) and high-performance orchestration engine built to eliminate the complexity. It provides a functional, modular language with declarative syntax for orchestrating logic while ensuring consistency through:

  • Strict Typing & Inference: Resolving the types during the planning phase to eliminate runtime errors.
  • Embedded Relational Logic: Utilizing native PRQL support for side-effect-free data manipulation over memory from multiples sources.
  • Modular Architecture: Defining reusable Resources, Steps, and Error Handlers that strictly separate orchestration from imperative execution.
  • Zero-Copy Performance: Compiling workflows into optimized Directed Acyclic Graphs (DAGs) for maximum throughput and minimal latency.
Example
import "fhub/kafka" kafka
import "fhub/postgresql" pg
import "fhub/clickhouse" ch
import "fhub/llm" openai
import "fraud-score/detect" fraud_detection

// 1. Centralized Resources (State/Connections)
// PostgreSQL
resource pg_db = pg.connection {
  host: "pg.internal"
} 

// Clickhouse
resource ch_db = ch.connection {
  host: "ch.internal"
}

// Kafka
resource kf_broker = kafka.connection {
  broker: "kafka.internal:9092"
}

// 2. Bound Imperative Steps with Resource Injection
step fetch_user_data = <connection=pg_db> pg.query {
  query: "SELECT id AS user_id, country FROM users WHERE id = @user_id"
}

step fetch_risk_profile = <connection=ch_db> ch.query {
  query: "SELECT user_id, velocity_score FROM risk_metrics WHERE user_id = @user_id"
}

step generate_audit = openai.prompt {
  system: "Analyze transaction, location, and velocity score. Generate a fraud audit text report."
}

// Global error catcher
handler alert_on_fail {
  *
    | <broker=kf_broker> kafka.produce { topic: "dlq_alerts" }
}

// Step error catcher
handler alert_step_fail {
  *
    | <broker=kf_broker> kafka.produce { topic: "dlq_alerts" }
}


// 3. Strict DAG Workflow
workflow FraudDetection ? alert_on_fail {

  <broker=kf_broker> kafka.consume { topic: "live_transactions" }
  > tx_stream

  // 1. Filter: High-value txns isolated via native PRQL
  // 2. Process: Imperative logic with localized error trap
  // 3. Enrich: Joined with user data & risk metrics
  // 4. Audit: LLM generates natural language report
  tx_stream
    | (
        from input
        filter amount > 10000
        select {user_id, amount}
      ) 
    | fraud_detection.process ? alert_step_fail
    | (
        from input
        join fetch_user_data (==user_id)
        join fetch_risk_profile (==user_id)
      )
    | generate_audit
    | <broker=kf_broker> kafka.produce { topic: "fraud_audits" }
}

Performance-First Execution Unified Integration

Heddle establishes a boundary between orchestration logic (the what) and imperative computation (the how). It simplifies the reuse of disparate infrastructure—including databases, event streams, and LLMs—while providing a seamless local-to-cloud bridge for production-scale clusters.

Example
# local/security.py
from typing import TypedDict
from heddle.core import Table

class HashConfig(TypedDict):
    algorithm: str
    rounds: int

def hash_password(config: HashConfig, input_table: Table) -> Table:
    # input_table is a zero-copy Apache Arrow table!
    arrow_data = input_table.to_pandas()
    
    # ... perform hashing logic ...
    
    return Table.from_pandas(arrow_data)

Relational Native Pushdown

Embedded PRQL support enables side-effect-free data enrichment directly within the orchestration layer. Workers engine execute relational transformations locally over memory, ensuring consistency across all languages.

Example
// Native PRQL support inside Heddle orchestration
workflow Analytics {
  (from fetch_active_users select username, id)
    | (from input filter id > 100)
    | io.print
}

High-Fidelity Observability

Heddle maintains an append-only lineage of all data flows step-by-step. Because data is immutable, developers can trace the exact state of data at any preceding step, enabling precise diagnostics and time-travel debugging.

Example
// Advanced Error Handling
handler detailed_error_handler {
  *
    | io.stderr

  *
    | (from input select error_msg, stack_trace)
    | kafka.retry_queue
}

Polyglot SDKs: Implement once, run anywhere.

Heddle provides high-performance SDKs for major languages. Each SDK follows idiomatic patterns while leveraging zero-copy memory exchange via Apache Arrow for maximum efficiency.

Type-Safe Resource Injection: Automatically inject stateful dependencies into your logic.

Zero-Copy Data Exchange: Move massive datasets between languages without serialization overhead.

Unified Plugin System: Standardized registration for resources, steps, and handlers.

Example
from heddle.sdk.plugin import Plugin
from heddle.core.table import Table
from heddle.core.step import StepConfig
from heddle.core.resource import ResourceConfig, Resource

# 1. Define Specific Table Types
class CredentialsTable(Table):
    """Input: contains 'id' and 'password'."""
    pass

class SecureTable(Table):
    """Output: contains 'id' and 'hash'."""
    pass

# 2. Define Resource & Step Configurations
class VaultConfig(ResourceConfig):
    api_key: str

class HashConfig(StepConfig[VaultConfig]):
    algorithm: str

# 3. Define the Resource Instance
class Vault(Resource):
    def __init__(self, api_key: str):
        self.api_key = api_key

# 4. Initialize and Register
plugin = Plugin(namespace="security")

@plugin.resource(name="vault")
def init_vault(config: VaultConfig) -> Vault:
    return Vault(config.api_key)

@plugin.step(name="hash", resource="vault")
def hash_password(config: HashConfig, input: CredentialsTable) -> SecureTable:
    # The 'vault' resource is automatically injected into the config
    vault = config.resource 
    
    df = input.to_pandas() # Contains 'id' and 'password'
    # ... use vault.api_key for salt or pepper ...
    return SecureTable.from_pandas(df) # Returns 'id' and 'hash'

if __name__ == "__main__":
    plugin.serve()

Developer Experience & Radical Reusability

Heddle prioritizes developer productivity by removing friction between local prototyping and distributed production environments

Fhub (Functions Hub)

An open-source standard library of reusable resources and steps. Heddle promotes modularity by allowing developers to write once and share logic across any workflow.

Workflow Control

Execute and manage complex distributed workflows running in cluster locally, providing total control over the orchestration lifecycle.

Frictionless Data Transfer

Transfer massive volumes of data seamlessly between local and production environments using high-performance, zero-copy protocols.

Integrated Observability

System visibility by integrating logs, metrics, and traces into a unified execution view. Mmonitor distributed workflows with real-time insights into system health.

Distributed Execution Architecture

Heddle employs a decoupled architecture that separates orchestration logic from the data management layer to maximize throughput and minimize latency.

The Control Plane

Manages execution state, DAG optimization, and JIT code loading. It routes logic without handling payload data, facilitating low-latency coordination and cross-workflow batching.

Polyglot Workers

A bifurcated layer consisting of a main orchestrating daemon and language-specific plugins (Python, Go, Rust, Node.js). They execute imperative logic via zero-copy memory exchange.

Data Locality Registry

An optimized memory-mapping subsystem that manages data flow and localization. It enables zero-copy routing and offloads large datasets that exceed local RAM capacity.

Communication Protocols

Utilizes gRPC for control signals and Apache Arrow Flight for data exchange. This eliminates serialization overhead, moving data at the speed of local shared memory.

Cross-Workflow Batching

Converges structurally independent workflows that share the same imperative functions. It consolidates data tables for vector processing efficiency.