I recently attended the Data Engineering Summit, looking to understand where the industry is heading. Across multiple sessions, from startups to tech giants, a clear pattern emerged. We are no longer just struggling with how to store data. The rise in data volume, the shift toward real-time streaming, and the massive demand for AI have create a new bottleneck: context. AI agents and machine learning models are hungry for data, but feeding them raw, messy database tables leads to hallucinations and broken pipelines. To bridge this gap, two keywords kept dominating the summit presentations: Semantic Layer and Ontology. ...

Recently, a friend of mine panicked because I went offline for a few days without replying to messages. They actually thought I was in danger. I realized I needed a passive way to reassure people that “I’m OK” without having to send active “I’m here” texts every single day. So, I built I’m OK, a serverless app to solve this problem, and to finally try out the Google Cloud stack. ...

Recently, I built a CLI tool called BulkPR for my job. It’s a simple app, but it works for what I needed. However, one challenge I faced was distribution. My colleagues use different systems-some on Windows, some on MacOS-and I need a way to ensure the tool worked smoothly across all environments. The Problem: Cross-Platform Distribution with Docker and Go At first, I tried to solve the cross-platform issue by using Go’s built-in GOOS and GOARCH environment variables to build different binaries for each platform. This approach worked in theory, but I didn’t have access to the devices I needed to test on, so just made a workflow for that. ...

Recently, I noticed that the Docker image size for my Next.js project was quite large. So I decided to do some research and found ways to reduce the image size. Baseline Here’s what my initial Dockerfile looked like: FROM node:18 WORKDIR /app COPY . . RUN yarn install RUN yarn build EXPOSE 3000 CMD ["yarn", "start"] It ended up being around 3GB. Step 1: Use a Smaller Base Image Before making any changes, my Docker image was using the standard Node.js base image. To reduce the image size, I switched to the Alpine Linux version. Here’s the updated Dockerfile with a smaller base image: ...