If you’re familiar with web frameworks such as Rails and Django, you probably know that these frameworks come with ORM and database migration. This is a great feature to have it enables evolutionary database design in your application. The problem with using built-in migration tools such as this is that you’re locked to your web framework. Moreover, independent migration tools such as Liquibase or Flyway may have some features that are not available off-the-self in your framework.

Assuming you already have some familiarity with Kubernetes, I’m going to lay out the core concepts of Kubernetes that are really important to understand - Pods First is the concept of Pods. Going back to the idea of containerization, containers are only supposed to run one process. In situations where closely related processes are meant to be run together, Kubernetes groups them together into a group called Pod. To summarize, Pod is something where one or more containers run where each container runs a single process and the processes in each container are closely related.

Apache Kafka is an open-source framework that allows you to develop real-time applications. In this article, I will jot down some points that may help you save some time and frustration if you’re just learning about Apache Kafka. First of all, to setup a development Kafka environment, it will save you a lot of hassle if you just use confluent distribution of Kafka as opposed to the native Apache version. Download the Confluent Platform from https://docs.

Most of us are familiar with writing database queries with SQL. But there are also other ways you can query your data from the database or from a file directly. One way is through a Python package called Pandas or through Apache Spark. Both of them are very popular these days in the Data Science field. If you can fit your data in memory in a single computer, I’d suggest to use Pandas.

Previously in this article, I wrote about Kafka Connect. Today, I’m going to get into the details of a type of Kafka Connect called Kafka HDFS Connect that usually comes pre-installed in the confluent distribution of Kafka. If not, it can be easily installed from the Confluent Hub by running the following command from the command line: confluent-hub install confluentinc/kafka-connect-hdfs:latest You can check all the connectors that are installed by:

These days individuals share personal data more than ever before. Technology has also advanced to allow companies to make use of personal data on an unprecedented scale. With so much personal data out there, you might wonder, who has personal data about me and how they will look after it. There are several common concerns among people about their personal data theses days and here are some of them: Since I booked a holiday online, I’m getting loads of emails from companies trying to sell me car hire, airport parking, and hotels.

Step 1 Check the existing kernel of your Ubuntu Linux: uname -a Step 2 Ubuntu maintains a website for all the versions of kernel that have been released. At the time of this writing, the latest stable release of Ubuntu kernel is 4.15. If you go to this link: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15/, you will see several links for download. Step 3 Download the appropriate files based on the type of OS you have.

Git allows you to create versions of your files and therefore it is known as the “Version Control System”. The following table compares others types of systems that allow to create versions and the features they provide: Versions Any Editor Use Offline Manual Save Manual Yes Yes Yes Drop Box Yes No No Google Doc No No No Wikipedia No No Yes Git Yes Yes Yes SVN Yes No Yes Fundamental concept Git system comprises of the local repository and the remote repository.

Linux Services that can be controlled with Linux’s service command, for example sudo service nginx status are controlled by scripts in the folder /etc/init.d/. These scripts define functions such as start(), stop(), reload(), status() etc that tells what to do when a service command is given. For example: when a service command such as sudo service nginx start, it calls the start() function defined in the script file nginx under the folder /etc/init.

From Jenkins home page, click on “New Item” and select a “Freestyle Project” and fill in the general details such as “Project Name and Description” Under the Source Code Management tab, add your code repository URL and set up the ssh credential so that Jenkins will be able to poll after certain intervals. You will need to generate a pair of public and private RSA key and put the public key in your repository and the private key in your Jenkins Source Code Management section by clicking on the “add” button which opens up a new window as shown below where you add in the ssh credential.

Generally, vim installation comes with a .vim diectory and .vimrc file in your home directory. If these are not present in your directory, please feel free to create them manually. Configurations for vim goes in the .vimrc file and the plugins are stored in the .vim directory. Here is what my current .vimrc configurations look like: 1 "Vundle Setup 2 set nocompatible " required 3 filetype off " required 4 5 " set the runtime path to include Vundle and initialize 6 set rtp+=~/.

So you have a new server either on premise or a virtual server from companies like Linode or Rackspace. What do you do next? How do you do the initial set-ups like making new users, and keeping it secured from malicious attacks among others. Here are some of the steps that I follow pretty much every time (Note: this tutorial is based on Ubuntu Server which will apply more or less to any Linux based server)

Discrete Structure and its Applications Discrete Mathematics and its Applications, by Kenneth H. Rosen JAVA Absolute Java, by Walter Savitch from Pearson Publishing Effective Java, by Joshua Bloch Data Structures and Algorithm Data Structures and Algorithms in JAVA, by Goodrich, Tamassia and Goldwasser Introduction to Algorithms, by Cormen, Leiserson, Rivest and Stein Algorithms by Dasgupta, Papadimitriou, and Vazirani Algorithms by Sedgewick and Wayne Mathematics Essential Calculus by James Stewart Differential Equations and Linear Algebra by Farlow, Hall, McDill and West Applied Linear Algebra by Olver and Shakiban Introduction to Graph Theory by Douglas B.

If you are tired of entering password everytime you ssh into your remote machine, you may want to create public-provate key pair and store your public key into your remote machine and the private key into your local machine. You can create such key pair by running the following command from your local machine - ssh-keygen This will create two files in your .ssh directory on your $HOME folder - one with the name id_rsa which is your private key and the other with id_ras.

Cron is a system daemon used to execute desired tasks (in the background) at designated times. A crontab is a simple text file with a list of commands meant to be run at specified times. It is edited with a command-line utility. These commands (and their run times) are then controlled by the cron daemon, which executes them in the system background. crontab -e : opens an editor to edit the crontab file

Users Add a user: sudo adduser [username] Delete a user: sudo userdel [username] All user information is stored in a file /etc/passwd Change a password: sudo passwd [username] Groups Groups are usually used to change permissions for a group of users as changing permissions individually would be a tedious task. Add a group: sudo groupadd [groupname] Delete a group: