As we bring an end-to-end Data Management platform story, this is a combination of several Data Engines, Distributed Processing Systems and Micro-Services. One Engine or combination of Multiple such Data Engines could serve your specific Use Cases around Data.
While this is Single IP story, we have given a view of major architectural blocks (Engines & Processing systems) here
• No-Code’ wrapper on top of Apache Spark, supporting all transformations, aggregations, and Window functions.
• Supports both Batch and Streaming data, with Pluggable Connectors and User-defined Functions.
• Proprietary Streaming Data Engine built with Apache Akka support,
• Supports Column-wise Transformations and Data Quality metrices.
• Flexibility to accommodate Engines / Scripts for future Business Cases,
• Allows re-run of pipeline from point of failure. Scheduled execution along with post-execution actions is possible.
• Searchable storage of metadata as Lucene indexes.
Metadata goes beyond standard labels to include:
• Annotations.
• Transformation definitions,
• Hierarchies
making NLP based data processing possible.
An Infrastructure Management framework to support the parent Data Platform – Deployment, Upgrades, Monitoring & Maintenance.
