Tom Sawyer Data Streams, Web Edition

Release 1.0.0

Overview

Tom Sawyer Data Streams introduces a powerful new approach to Extract Transform Load (ETL) and unification of streaming and legacy data into a single, governed, query-ready knowledge graph in real time. The platform seamlessly subscribes to Apache Kafka (or Confluent) topics—sourced from relational databases, data warehouses, unstructured data, APIs, and more—allowing you to consolidate disparate data without expensive migrations. This release supports Kafka authentication and encryption along with OAuth 2.0 and Keycloak. Deployment is streamlined across cloud, on-premises, and air-gapped environments using a dedicated Docker installer.

The core of the release emphasizes agility and precision in data preparation. An intuitive, web-based designer enables visual data flow design, allowing users to define sources and sinks, and apply powerful transformations without writing custom code. Data flows can run continuously, providing real-time, low-latency processing that keeps the knowledge graph perpetually current for operational decisions and analytics. The resulting graph can be persisted in a graph database and published back to Kafka, supporting downstream tools like AI, visualization, and retrieval-augmented generation (RAG).

Data Streams pairs directly with Tom Sawyer Perspectives and Tom Sawyer Explorations to instantly visualize the resulting knowledge graph. This direct link moves you from data ingestion to interactive insight in minutes, providing automatic graph layouts, powerful filtering, and intuitive data exploration, enabling all stakeholders to validate models and share critical findings. Data Streams has been built with Tom Sawyer Perspectives 13.4.0.

New

Data Integration

• Data integration from Kafka topics
• Data integration from PostgreSQL Debezium sources
• Save models to JanusGraph databases
• Save models to Kafka topics
• Real-time CRUD support

Schema View

• Automatic schema extraction for Kafka element types, attribute names, and data types
• Support for Kafka sampling
• Support for Confluent Schema Registry

Data Flow

• Visual environment to define data sources, transformations, conditions, and sinks
• Automatic graph layout of data flows
• Duplication and renaming of data flows
• Data flow validation
• Editing of data flows
• Synchronized with schema view for immediate reflection of data source changes and schema transformations
• Transformations
  • Convert from node type to edge type
  • Join
  • Rename element
  • Set attribute
  • Set ID
  • Split
  • Support for Spring Expression Language (SpEL)
• Batch processing
• Real-time, continuous processing

Authentication

• OAuth 2.0 authentication
• Keycloak authentication

Installer

• Docker-based installer
• Support for airgapped environments

Documentation

• Setup Guide
• User Guide