Data Engineering
Explore the essentials of Data Engineering, delving into how data systems are built and maintained. From organizing data flows to automating complex data processes, discover the tools and techniques that make data easily accessible and useful for everyday projects and insights.
Data Engineers are responsible for designing, constructing, installing, testing, and maintaining highly scalable data management systems. They build algorithms to help give easier access to raw data, work to improve data reliability, quality, and efficiency, and use system architectures that handle large data sets. Their work involves setting up and populating analytics databases, using programming languages like SQL, Python, or Scala, and tools such as Hadoop, Spark, and Kafka. This role requires strong software engineering skills and a deep understanding of data warehousing solutions.
Ecto Custom Field Setting
Elixir Snippet: How to set a custom field in ecto changeset pipelines based on previous data fields.
Ecto Custom Field Validation
Minimal snippet to define a custom changeset validation function for a field in Elixir’s Ecto library.
Filter function for arrays/slices in Go
How to write a functional Filter function in Go using generics, including a unit test. A taste of functional code.
Find function for arrays/slices in Go
How to write a functional Find function in golang using generics, including a unit test. A taste of functional programming!
FlatMap function for arrays/slices in Go
How to write a functional FlatMap function in golang using generics, including a unit test. Basically map and flatten in one go!
Get File Encoding
Simple shell command to guess the encoding of a text file.
GroupBy function for arrays/slices in Go
How to write a functional GroupBy function in Go using generics, including a unit test. Returns a map with slices.
Import SQLite records from another database
See how to copy SQLite rows from an already existing (old) database into another database, table by table, using the ATTACH DATABASE built-in feature from within a bash script.
IndexOf function for arrays/slices in Go
How to write a functional IndexOf function in Go using generics. Surprisingly helpful during mundane tasks!
Map function for arrays/slices in Go
How to write a functional Map function in Go using generics. The classical foundation of data pipeline code!
ParallelMap function for arrays/slices in Go
How to write a concurrent functional Map function in golang using generics and goroutines. Leverage all your CPU cores!
Reduce function for arrays/slices in Go
How to write a functional Reduce function in Go using generics, including a unit test. The foundation for many functional programming libraries!
Shell Command To Resize Images
The quick and efficient way to ad-hoc resize an image file right from bash or zsh terminals using image magick.
SQLite On Delete Callbacks
Leverage the built-in callback handlers of sqlite to ensure that no dangling foreign keys or orphan records exist in your database. Pure SQL solution.
Transform All Image Files to webp
Bash script that replaces all image files within a folder into webp files of the same size.
UniqueBy function for arrays/slices in Go
How to write a functional UniqueBy function in golang using generics, including a unit test. Stop writing this by hand every single time!
Unique function for arrays/slices in Go
How to write a functional Unique function in golang using generics, including a unit test. Quite handy when processing data!
Blog Posts →
AI Alignment Problem: a fresh perspective
1 months agoExploring a fresh approach to solving the AI alignment problem by focusing on intrinsic motivation and long-term thinking to ensure harmonious development between AI systems and humanity.
AI Alignment Proposal: v0.1
1 months agoMy personal shot at approaching the AI alignment problem. Allows for iterative refinement as well as immediate experimentation with current LLMs today.
Related Categories →
System Administration
Explore the critical world of System Administration, where efficiency and reliability keep computer systems running smoothly. Learn about the tools and techniques for managing servers, networks, and software to ensure optimal performance and security.
Related Technologies
Elixir
Harness the power of concurrent, fault-tolerant programming for scalable, maintainable applications. Code that flows like water!
Ecto
A toolkit for data mapping and language integrated query. The Elixir answer to Ruby's ActiveRecord.
Shell
Shell scripting: Automate and streamline your tasks with powerful command-line scripts. Unlock efficiency in every command! Contains bash, zsh, sh, ...
Go
Fast, simple, and efficient. Ideal for solopreneurs, Go's straightforward syntax and powerful performance allow for quick development and deployment.
SQLite
The lightweight and surprisingly fast database option for relational data. Ideal for traditional VPS-style deployments with persistent disk, or in-memory for nontrivial aggregations.