Divyendu Singh
Staff Data Framework Engineer at Teradata
Building enterprise data platforms, AI pipelines, and the frameworks that engineering organizations rely on daily — with 16+ years of depth across the full stack.
What I Do Now
I'm a Technical Lead at Teradata, and the through-line of my work is leverage. I find where an organization is solving the same problem fifty different ways, build the system that unifies the approach, and drive adoption until it becomes the standard. Right now that means data engineering frameworks, Airflow orchestration, governance platforms, and AI/ML pipelines — but the pattern has been the same at every stage of my career.
A junior engineer with AI can produce more code. A Staff engineer with AI can produce more leverage. The hard part was never writing the code — it's knowing what's worth building, designing it so it scales, and getting an entire org to actually use it. That's what I do.
The Full Circle
Before enterprise data platforms, I spent years building systems closer to the hardware — and it started with a contrarian bet.
In 2009, for my final year engineering project, I chose to implement David Lowe's SIFT algorithm — a computer vision technique — on a GPU using OpenGL shaders. The premise was contrarian for its time: GPU computing isn't limited to games. My teachers questioned the ambition. We had to learn Objective-C, OpenGL, the shader language, and the underlying mathematics simultaneously. The math-to-shader translation proved harder than expected, and the GPU drivers had bugs. But we reduced scope, got feature detection working on the GPU, and I received the highest grade in my batch.
Sixteen years later, that intuition has been validated beyond anything I could have imagined. GPU computing is now the foundation of the AI revolution — the same parallel processing architecture I explored in a college lab now powers the deep learning models I integrate into production systems. It has come full circle.
Since that project, I've built systems at every layer of the stack — enterprise data platforms and AI-powered governance tools, deep learning analytics for semiconductor manufacturing, IoT-to-cloud data pipelines, factory automation software controlling robot arms, real-time video streaming and image processing, and mobile apps in the early App Store era. At each stage, I've built frameworks and platforms that other engineers build on top of.
Current Work — Teradata, Data & AI Organization
Technical Lead in Teradata's Data & AI organization, leading a cross-functional team of Senior Data Engineers, Data Scientists, and BI Engineers. I turn narrow asks into reusable infrastructure and let the results drive adoption.
Reusable Data Engineering Frameworks
Asked to build a file download feature. I saw the platform opportunity and built it as a config-driven, massively parallel framework from day one — handling OAuth, pagination, state management, and failure recovery. Proved it at scale on an AI pipeline processing 112K documents. The results turned leadership into advocates.
Now adopted org-wide by 50+ engineers.
Enterprise Workflow Orchestration
Championed and leading a migration to Apache Airflow — not just the technical build, but the organizational change: standards, support model, and phased rollout. Deep in Airflow internals, from cluster architecture to custom plugin development.
Governance & Platform Tooling
Designing and building a Data & AI Governance platform from scratch — declarative form engines, reusable UI component libraries, and schema versioning that treats form data as evolvable artifacts.
AI/ML Pipelines
Building end-to-end pipelines: OCR, vector embeddings, RAG-based semantic search, and AI assistants — connecting data that was previously siloed into tools teams can actually use.
Data Pipelines
Building and maintaining marketing analytics data infrastructure, including cloud data warehouse ingestion from external sources like BigQuery and Google Analytics.
PythonApache AirflowREST APIsBigQueryTeradata VantageKubernetesDockerVector DatabasesRAG
Previous — Sapphire Automation, Industrial IoT & Analytics
7 years building data platforms and automation systems across semiconductor, consumer electronics, and industrial IoT. At each stage, the pattern was the same: find the fragmentation, build the platform, drive adoption.
Deep Learning Analytics (2022–2023)
Identified that each analytics team was building one-off detection pipelines. Architected a pluggable platform where any model could be configured and deployed across use cases. Processing 5GB of high-resolution imagery daily with leaderless high-availability architecture.
Stream Processing (2017–2018)
Recognized divergent analytics backends and unified them into a single platform. Led a cross-departmental initiative across 5 teams to standardize AI model commissioning workflows. Automated IIoT device provisioning. Reduced development time by 50%, improved reliability by 8%, increased R&D efficiency by 20%.
IoT Data Pipelines (2019)
Saw disconnected factory-floor and cloud systems. Built multi-stage edge-to-cloud data pipelines on AWS for real-time manufacturing quality monitoring.
Factory Automation (2016–2017)
Built software controlling industrial equipment (robot arms, conveyors, test fixtures) with database-backed monitoring and analytics. Led development teams with technical mentorship.
PythonC#CAWSDockerKafkaRabbitMQMySQLMicroservicesREST APIsModbusTCP/IP
Earlier Career
2013 – 2016
Programmer Analyst — TechJini Solutions (now Datamatics Digital)
Developed macOS industrial automation software — recipe management, equipment control, data collection, and analytics dashboards. Built communication frameworks using Modbus and TCP/IP for production floor systems.
2010 – 2013
Senior Software Engineer — Livestream (now Vimeo)
Developed iOS applications for live video streaming. Built a photo workflow app for the Volvo Ocean Race that processed 200MB images on iPad hardware in real-time. Deep work in ffmpeg, image processing, and streaming protocols.
2009 – 2010
Mobile App Developer — Nuvus Technologies
One of the first employees at a 13-day-old startup during the early App Store era. Built cross-platform apps for iOS, Android, and Windows.
2005 – 2009
B.Tech in Computer Science — UP Technical University
Where it all started. Final project: GPU-accelerated implementation of David Lowe's SIFT algorithm using OpenGL shaders — the contrarian bet that GPU computing would go beyond games. Received highest grade in the batch. Also built a sorting algorithm benchmarking system to empirically verify textbook asymptotic complexity — because theory should meet reality.
Technical Expertise
Languages
PythonC#CObjective-CJavaSQL
Data & Orchestration
Apache AirflowApache KafkaRabbitMQETLStream ProcessingBigQueryTeradata VantageIceberg
Cloud & Infrastructure
AWSDockerKubernetesMicroservicesREST APIsCI/CD
Architecture
Distributed SystemsHigh AvailabilityPlatform DesignData PipelinesVector DatabasesRAG
Domains
Industrial IoTFactory AutomationDeep Learning AnalyticsData EngineeringAI Governance
Platforms & Protocols
LinuxmacOSWindowsiOSAndroidModbusTCP/IPRTSP
Beyond Work
When I'm not building systems, you'll find me outdoors. I love hiking and camping — there's something about disconnecting from screens and being in nature that resets the way I think about problems. I also enjoy planning trips with friends, which honestly exercises a lot of the same project management muscles as engineering (logistics, contingencies, keeping everyone aligned on the plan).
On the athletic side, I'm a long-time badminton player and currently learning tennis — humbling myself with a new skill after years of being decent at another racquet sport. There's a useful lesson in that for engineering too: staying a beginner at something keeps you empathetic to the learning curve.