Writing about platform engineering, agentic AI systems, Python, and cloud infrastructure.
How I protect 15+ projects across Python, JavaScript, and Rust from supply chain attacks using a three-layer defence: registry-level delays, automated PR scheduling, and lockfile discipline.
Convention-driven PRD-to-HLD-to-LLD pipelines and why schema-gen should be your first commit.
Start from a single business SLA — data freshness under 60 seconds — and trace backwards through dependency trees, metadata layers, known-error memory, and automated fixes to build an AI-operated production platform.
Uptime tools tell you a service is running. Gatus tells you the data pipeline is actually working. How I use 73 custom health checks to monitor infrastructure, data freshness, and pipeline completeness.
Building a benchmarking utility for shuffle and network transfer metrics in Databricks clusters.
How we built an end-to-end pipeline using Spark and H2O Sparkling Water to get machine learning models from source control into production with minimal friction.
A practical approach to building an organisational machine learning pipeline that supports multiple tools, PMML-based model deployment, CI/CD practices, and A/B testing.
A practical troubleshooting guide for common Hive and Sqoop job failures on YARN, covering memory tuning, CPU allocation, and where to find the right logs.
Personal views on the Open Data Platform (ODP) announcement and what it means for Hadoop users, BI vendors, and the major distribution companies.
Mesa is a highly scalable analytic data warehousing system that stores critical measurement data related to Google's Internet advertising business.
Key design considerations for storing time series data, including access pattern analysis, windowed storage strategies, and the trade-offs between granularity and performance.
Lessons learned from migrating a large Hadoop cluster over a single weekend -- planning, data migration with distcp, code migration of 300+ Oozie jobs, and HBase migration.
Notes from the book SBT in Action covering Maven concepts, SBT basics, tasks, settings, plugins, and build definitions.
A comparison of the new HDFS centralized cache management feature in Hadoop 2.3 with Spark RDDs, and why Spark still held the edge for in-memory processing.
A concise overview of why Apache Spark was created, how RDDs enable in-memory processing for iterative and interactive workloads, and its key programming abstractions.
A comparison of YARN and Mesos as cluster resource managers, including their architectural differences, recent developments, and insights from the Google Omega paper.
Approaches for handling schema evolution in Hadoop using Avro and ORC file formats, including a practical workflow for managing schema changes with Hive.
How to use the ChainMapper class in Hadoop to call multiple mappers in sequence, with a working example and key points about configuration and type compatibility.
The small files problem in Hadoop and five approaches to solve it: HDFSConcat, IdentityMapper/Reducer, FileUtil.copyMerge, Hadoop File Crush, and Hive concatenate.
Understanding HBase minor compaction -- how files are selected for compaction using the ratio algorithm, with a worked example showing the selection logic.
Understanding HBase major compaction -- how it differs from minor compaction, the configuration properties that control it, and the three methods that trigger it.
To do bench marks of Hadoop cluster is an ongoing process as we use it inside the organization. The main thing which we don't know when we buy new cluste...
Below are some of the learning's which i thought to jot down for quick reference in the future also. These are lessons which i learnt while working in real...
At last its 1 May 2012 Cloudera has opened certifications through vue to worldwide people Details are as follows from Developer Exam Exam...
How to set up a Hadoop development environment in Eclipse with the WordCount MapReduce example.
Error java.io.IOException: Failed to set permissions of path: file:/tmp/hadoopjj/mapred/staging/jj1931875024/.staging to 0700 The details for my Hadoop we...
Quick guide to installing Hadoop on Windows with Cygwin, including Java configuration and verification steps.
CSS3 Speech modulehttp://www.w3.org/TR/css3speech/ is now in last call state Aug 18 , 2011 . Journey for making accessible web standards have come far from...
Graceful degradation and Progressive enhancement are two different web design strategies Progressive enhancement In progressive enhancement you design...
There are two ways to setup environment variables in linux systems Ubuntu , Red Hat , Fedora etc Using the export method of adding environment variables...
Step-by-step guide to manually installing the Java JDK on Linux, including download, permissions, and environment variable setup.
Step-by-step guide to manually installing Apache Tomcat on Linux, including download, permissions, environment variables, and startup.
The explains How to fix the problem when excel cell drag function don’t work or plus sign don’t come when you want to drag the cells in Excel 2010 , 2007 o...
Quick tip on using the uname command to check your Linux kernel version and system information.