eLitmus Blog

What's the deal with secret_key_base in Rails?

2025-09-11 18:52:53 +0530T00:00:00-00:00

If you’ve spent any time with a Rails app, you’ve probably stumbled across this mysterious setting called secret_key_base. Maybe you saw it in config/secrets.yml, or like me, you were upgrading Rails, forgetting that credentials are the new norm and suddenly you saw something like this:


Missing `secret_key_base` for 'production' environment, set this string with `bin/rails credentials:edit`

The short version of what it is:

secret_key_base is Rails’ way of keeping things safe. It’s a big, random string of characters that Rails uses under the hood to:

Sign and verify cookies (so nobody can tamper with them).
Encrypt and decrypt sensitive data.
Make sure session data is legit and not forged.

Think of it as the master lock on your app. If someone gets hold of it, they could impersonate users or mess with your sessions. That’s why Rails treats it like a deal-breaker if the key isn’t set.

Where does it come from?

In older Rails apps, you’ll see it inside config/secrets.yml:


production:
  secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>

In newer Rails versions, it’s usually stored in config/credentials.yml.enc. Either way, it usually gets pulled in from an environment variable. You don’t want this string hard-coded in your code. But guess which genius hard-coded it? ;)

How do you get one?

Rails makes this easy. Just run: rails secret

You’ll get a nice long random string (128 characters). Copy that into your environment variable, and you’re good to go.

In developmentand test environments, even if you don’t run this command, a secret will be generated for you when you either start the server or the console.

You can find this generated file in tmp/local_secret.txt.

In a way this file is proof that Rails is looking for config/credentials.yml.enc and not config/secrets.yml. To override this, and to make Rails fetch this value from secrets.yml, just add the following in your application.rb:


config.secret_key_base = Rails.application.config_for(:secrets)[:secret_key_base]

To sum it up:

secret_key_base isn’t something you’ll deal with every day, but it’s quietly protecting your Rails app all the time. The good news is: once you set it up properly, you mostly forget about it, unless you’re upgrading to Rails 7 or 8 and you still want to continue using secrets.yml.

What's the deal with secret_key_base in Rails? was originally published by eLitmus.com at eLitmus Blog on September 11, 2025.

Kamal App Deployment Tool

2024-11-14 10:48:20 +0530T00:00:00-00:00

Kamal is a simple, dedicated orchestration tool built specifically for deploying containerized applications (mainly Rails). In this blog post, I will take a deep dive into the internal workings of Kamal, exploring its high-level architecture and key deployment phases.

High Level Architecture of Kamal

We can divide Kamal deployment in 3 parts

Build
Push Image to container registry
Deploy

Build Phase

The build process can occur either locally or on a remote server. Its primary purpose is to create images compatible with both amd64 and arm64 architectures. Kamal employs a straightforward Docker-in-Docker strategy for cross-platform builds, making it seamless to create images that work across different architectures.

Container Registry

When using kamal build command, the tool automatically pushes the newly built image to your container registry. If you’re handling the build process separately, you’ll need to manage the registry push manually. Kamal supports most major container registries out of the box (e.g. Docker hub / AWS ECR).

Deploy Phase: The Core Magic

This is the core of Kamal’s deployment process, which goes beyond just pulling the image and starting a container. Here’s a step by step breakdown of what Kamal does under the hood during deployment:

SSH into servers & start by removing any outdated image with the same tag on each server, ensuring the environment is clean.
Then, it pulls the latest version of the Docker image from the registry
Kamal integrates with Kamal-proxy (or similar proxies) to manage network routing. It checks if the kamal-proxy is active, which is essential for rerouting traffic to the new container when it’s ready.
Kamal Now, first replaces the old container name to _replaced than start this new primary container & will wait for healthcheck to be passed to consider it as healthy.
Once the above step completed Kamal starts the secondary containers.
After confirming that the new containers are stable, Kamal prunes old containers and images, freeing up disk space and reducing clutter on servers. This cleanup ensures efficient resource use over time.

Practical Uses

Kamal can be used in multiple ways -

As a cross platform image builder.
For deploying pre-built containerized applications (Build step can be taken care by someone else).
- Deploy step can be done for any containerized application.
- Platform specific deployment to multiple nodes in parallel.
As an end-to-end solution handling both build and deploy phases.

Prerequisites to use Kamal

Ruby Environment.
A containerized application.
Access to container registry.
Server Infrastructure (Bare Metal / EC2 / Google Cloud )
- SSH Keys Setup so that kamal can access the servers during deploy.

Steps

Install latest ruby
Initialize kamal
- Install kamal gem. gem install kamal
- In app directory run kamal init (This can be run outside of project directory if build is not part of the deployment)
Configure deployment settings in config/deploy.yml
For multiple environments, create specific config files(e.g. staging / UAT / production) config/deploy.staging.ymlconfig/deploy.production.yml

Build Configuration

Kamal offers flexible options for managing the build process.

Within same project folder using config/deploy.yml

builder:
  arch: amd64

Specify external project locations, if config/deploy.yml is in different folder:

builder:
    arch: amd64
    context: 'path_to_project_base' 
    dockerfile: 'path_to_dockerfile'
    args:
      COMMIT_SHA: 1.0.0 # default it picks up the latest commit hash.

Image Tagging

By default, Kamal uses Git commit hashes for image tagging. This provides automatic versioning based on your Git history. However, you can customize this behavior if you want to create your own tags:

builder:
  # Other configurations...
  args:
    # Override default git hash-based tag
    COMMIT_SHA: 1.0.0

Once configuration are set use kamal build to finish up the build process.

To deploy a specific version we need to pass VERSION during deploy

VERSION=1.0.0 kamal deploy -P

Deploy configurations

Kamal gives us following commands to deploy application -

kamal setup    # First-time setup and deployment
kamal deploy   # Standard deployment
kamal redeploy # Just Deploy without bootstrapping servers / proxy containers

Pro Tip: Add -P or --skip-push to any command to skip the build and push phases. This is particularly useful when you’re using a separate CI/CD pipeline for building images:

kamal deploy -P  # Deploy without image build & push step

Managing Database Migrations

We can use following approaches to run the migrations so that migrations runs only in primary server.

Using Hooks

Hooks are Kamal’s way of executing commands at specific points in the deployment process. Here’s the complete hooks list.

docker-setup
pre-connect
pre-build
pre-deploy
post-deploy
pre-proxy-reboot
post-proxy-reboot

For migrations, the pre_deploy hook can be used, insert the below code in hooks/pre-delploy file:

VERSION=$KAMAL_VERSION kamal app exec -p "./bin/rails db:prepare"

Server Tags for Migration Control

To run migrations using server tags we can do following steps

Add a tag to the server
Create Env variable for that tag
Use that ENV variable in docker-entrypoint

config.yml
servers:
  - 172.0.0.1: db
  - 172.0.0.2
  - 172.0.0.3
  
env:
  clear:
    MYSQL_USER: app
  secret:
    - MYSQL_PASSWORD
  tags:
    db:
      MIGRATION: 1

if [ "$RUN_MIGRATIONS" = "1" ]; then
  echo "Running database migrations..."
  bundle exec rails db:migrate
fi

For Detailed discussion on Database migration visit Github Discussion

References

Kamal App Deployment Tool was originally published by eLitmus.com at eLitmus Blog on November 14, 2024.

Improving Drag in React JS: A Smoother Approach to Draggable Elements

2024-10-15 19:39:07 +0530T00:00:00-00:00

When building interactive web components, such as a draggable element, you might run into the problem where the native draggable attribute leaves the original element in place while dragging a semi-transparent copy. This behavior can feel clunky and disrupts the user experience. In this blog, I’ll show you how to build a smooth draggable component where the element itself follows the mouse pointer, rather than a ghost image.

By default, when you set an HTML element to draggable="true", the browser shows a semi-transparent copy of the element that moves with your mouse. While this behavior is native and functional, it often doesn’t look great. You want to make sure that the original element moves smoothly with the cursor without showing the browser’s default ghost image.

Solution

The solution is to create an overlay element that follows the mouse pointer while hiding the original element during dragging. This approach improves the user experience and creates a smoother drag-and-drop interaction.

Step-by-Step Guide

Setup Initial State

We are going to need some state to handle the Drag & Drop.

const [isDragging, setIsDragging] = useState(false);
    const [offset, setOffset] = useState(null);
    const [position, setPosition] = useState(null);
    const draggableRef = useRef(null);

isDragging: This will store the current drag state of the element.

offset: When the user starts dragging, they can click anywhere within the draggable element. This stores the coordinate distance between the element’s origin point and the clicked position.

position: This will store the current cursor position.

draggableRef: We’ll use this to retrieve the origin position of the draggable element.

Create the Draggable Element

First, let’s set up a simple HTML element to drag:

return (
      <div className="App">
        {
          isDragging && position && (
            <div
              className="draggable-item draggable-overlay"
              style={ `top: ${position.y}px; left: ${position.x}px`}
            >
              Drag Me!
            </div>
          )
        }
        <div
          ref={draggableRef}
          onMouseDown={handleMouseDown}
          className="draggable-item"
          style={`opacity: ${isDragging ? 0 : 1};`}
        >
          Drag Me!
        </div>
      </div>
    );

Handle Drag Start: Hide the Original and Create an Overlay

We are not going to use native drag events. Instead we will be using onMouseDown event as it will give us more flexibility.

const getInitialPosition = () => {
      const draggableElement = draggableRef?.current?.getBoundingClientRect();
      return {
        x: (draggableElement?.x || 0) + window.scrollX, // It will handle edge case when there is scroll in the page.
        y: (draggableElement?.y || 0) + window.scrollY,
      };
    };

    const handleMouseDown = (event) => {
      setIsDragging(true);
      const initialPosition = getInitialPosition();
      setOffset({
        x: event.clientX - initialPosition.x,
        y: event.clientY - initialPosition.y,
      });
      setPosition(initialPosition);
      document.addEventListener('mouseup', handleMouseUp);
    };

    useEffect(() => {
    if (offset) {
      document.addEventListener('mousemove', handleMouseMove);
    } else {
      document.removeEventListener('mousemove', handleMouseMove);
    }

    return () => {
      document.removeEventListener('mousemove', handleMouseMove);
    };
  }, [offset]);

Move the Overlay with the Mouse

In last step we had added an eventListener on mousemove event. Now, let’s make sure that on mouse move the overlay follows it.

const handleMouseMove = (event) => {
      if (!isDragging) return;

      if (event.clientX > 0 && event.clientY > 0) {
        setPosition({
          x: event.clientX - (offset?.x || 0),
          y: event.clientY - (offset?.y || 0),
        });
      }
    };

Clean Up on Drag End

Once the dragging is finished, we need to remove the overlay and make the original element visible again. Also we need to remove the mousemove event listener.

const handleMouseUp = () => {
      setIsDragging(false);
      setPosition(null);
      setOffset(null);

      document.removeEventListener('mouseup', handleMouseUp);
    };

Styling the Draggable & Overlay

Finally, you can style the overlay so that it looks like the original element. This CSS will ensure that the overlay matches the original element’s appearance.

.draggable-item {
      border: 2px solid #707070;
      padding: 7px 14px;
      border-radius: 8px;
      width: fit-content;
      cursor: grab;
      user-select: none;
      background-color: coral;
    }

    .draggable-overlay {
      position: absolute;
      cursor: grabbing;
      box-shadow: 0px 20px 25px -5px #0000001a;
    }

Conclusion

By using an overlay element to follow the mouse and hiding the original element, you can avoid the default ghost image that the browser shows when dragging. This method provides a smoother and more visually pleasing drag-and-drop experience.

Feel free to experiment with this solution in your projects, and let me know if you find any other creative ways to enhance the draggable experience!

Improving Drag in React JS: A Smoother Approach to Draggable Elements was originally published by eLitmus.com at eLitmus Blog on October 15, 2024.

Book review: Never let me go - Kazuo Ishiguro

2024-09-27 16:03:24 +0530T00:00:00-00:00

I was recently given 2 books as a gift by a close friend, who is a great fan of Murakami’s. The books in question are “Norwegian woods” by Murakami and “Never Let Me Go” by Kazuo Ishiguro.

The task was to read both of these books, and to validate the said friend’s belief that Murakami,as an author, is far superior to Ishiguro. Being a die-hard Murakami fan, she told me beforehand that “Norwegian Woods” is Murakami’s only story without any magical realism, and probably one of his lesser works, while on the other hand, “Never Let Me Go” is Ishiguro’s finest.

So you see, the balance was already distorted. Comparing Ishiguro’s finest to Murakami’s worst? Only to prove that Murakami was still the better author! Being a fan of neither, but a true friend, I believed what I was told, and picked up “Norwegian Woods” first.

While much of what I felt while I read the book is already lost, for it has been around a month since, I remember the writing style quite vividly. “Norwegian Woods” follows a very level-headed and mature protagonist, with the tone of the book being serious mostly.

It is a story of the past, present and future. Without going into much detail (since this blog is mostly about the second book), the protagonist learns to remember and cherish, while also letting go of the past. He fights his demons, and finally accepts and embraces his present, while on the very last page it is hinted that he is finally looking towards his future.

I felt like death was looming around the corner, throughout the book. And towards the end, it became quite predictable what was about to happen. Nevertheless, I was still distraught when what was supposed to happen, happened.

The sadness that I felt throughout this book made me purchase a whole set of Murakami’s best works. If this is one of his worst books, I expect nothing short of heaven from his best.

I next picked up “Never Let Me Go” by Kazuo Ishiguro, the “wannabe japanese” author, as i was told by my friend. I could instantly see the difference in the writing styles. While Murakami’s was a more somber and mature tone, Ishiguro’s was more childlike.

With this bias already in my mind, I put the book down only after a few pages. But as luck would have it, I had to sit in a hospital for hours the next couple of days and I found this book in my bag. So, I gave the book another try, expecting to be let down. Needless to say, I couldn’t put down the book before finishing it. It was that good.

Being as cautious as I can be about not giving spoilers, I’ll go through the essence of what I went through while I read this gem.

“Never Let Me Go” makes you remember your childhood. It makes you remember what being innocent (that’s right, being and not pretending to be) felt like. And it also highlights how you come to lose your innocence. How you finally come to see this world for what it is, instead of what you want it to be.

In this regard, there is this specific moment in the book where the students prank “Madame” - only to realize that while they could predict how she would react, they weren’t ready for the way it made them, and especially our protagonist, feel.

Like I said above, I found the tone to be child-like. But only after I was engrossed in the book, did I realize that that was how it was supposed to be. Because the tone was child-like, I could feel the innocence of our protagonist. I could really believe that it was a child who was narrating the book.

The book makes you go through a variety of emotions - love, jealousy, hatred, apathy, sympathy, and a plethora of others. There were times when I had tears in my eyes reading the book, only to feel hopeful a couple of pages later. And a few more pages down the line, I would go from being hopeful to feeling completely hopeless.

In particular, I loved the character of “Ruth”. A fighter, a leader, a go-getter. Someone who wanted to belong. Someone who wanted to be validated or accepted by others.

I went from respecting the child she was, to hating the teenager she grew up to be, to slightly understanding, but also feeling apathy towards the adult she became, to finally coming to terms with the good-hearted person she really was, especially towards the end.

I feel there is a “Ruth” inside all of us. All we want is to be accepted and validated. We would like to belong. Many times, I find myself doing things that I normally wouldn’t. Things I do in order to be validated by others.

From laughing too hard to a joke that I didn’t really find funny, just to be a part of the group, to being extremely courteous towards elders, because that’s how I am supposed to behave. Not that there’s anything wrong with this, I just realize while doing these things, that I am not being myself.

Towards the end, the book made me feel grateful for the freedom that I enjoy. It made me understand How the most important things in our lives, we take for granted! How stupid and shallow and narrow-minded we are. How evil we really are.

In summary, “Never Let Me Go” is a masterpiece. While fiction, I found it to be a representation of our current dystopian society. You read that right - Not a “dystopian representation of our current society” but a “representation of our current dystopian society”.

Once I was done reading, I immediately called my friend to let her know that Ishiguro was in no way inferior to Murakami.

To Mr. Murakami, I look forward to reading your best. I have been suggested to read “Kafka on the shore”. I apologize for ever doubting you, Mr. Ishiguro. In my defense, I hadn’t read “Never Let Me Go.”

Book review: Never let me go - Kazuo Ishiguro was originally published by eLitmus.com at eLitmus Blog on September 27, 2024.

Mastering Multi Tenant setup with rails - background jobs

2024-05-05 12:39:14 +0530T00:00:00-00:00

Welcome back to the Rails multi-tenant architecture series! If you’re just joining in, be sure to check out Part 1, where you’ll find an introduction to multi-tenancy and a detailed walkthrough on setting up a multi-tenant Rails application.

Part 1

Quick Recap

In the previous blog post, the focus was on delving into the concept of multi-tenancy in software design, with a specific emphasis on managing separate databases for each tenant. After exploring three types of multi-tenant application architectures, a step-by-step guide was provided for setting up a multi-tenant Rails blog application. This included configuring databases for each tenant, implementing automatic connection switching in Rails 6/7, and using Nginx to run multiple databases simultaneously on different ports.

Introduction

In this blog post, the focus is on background job processing within a multi-tenant Rails environment. Specifically, it addresses the challenges of running background jobs across multiple databases and proposes solutions to ensure seamless execution of jobs.

Sidekiq

First we will setup Sidekiq, A popular background job processing library for Ruby. Here’s a quick guide on how to set it up:

Add sidekiq( use > 6 version) in Gemfile. Follow This Guide for setup.
Create a sidekiq job rails generate sidekiq:job multi_db_testing

# app/sidekiq/multi_db_testing_job.rb
class MultiDbTestingJob < ApplicationJob

  def perform
    p "Number of articles is #{Article.count}"
  end
end

Running up application along with sidekiq

To start both the Rails server and Sidekiq, follow these steps:

Install foreman gem to start both rails server and sidekiq.
In Gemfile add foreman gem & run bundle install.
Create a Procfile to define the processes:

# procfile
web: bin/rails server --binding=0.0.0.0 --port=3000 --environment=development
sidekiq: bundle exec sidekiq

Triggering Background jobs

Create a route and controller action to trigger the Sidekiq job:

# config/routes.rb
    resources :articles do
      collection do
        get :run_background_job
      end
    end

    # app/controllers/articles_controller.rb  def run_background_job
      MultiDbTestingJob.perform_later

      redirect_to root_path
    end

    # app/views/articles/index.html.erb
    <%= link_to "Run sidekiq job", run_background_job_articles_path %>

Start the server using foreman start
Navigate to http://localhost:3000, and trigger the job.
You’ll notice that the job is executed, but it retrieves data only from the default database. why? Continue reading to find out the reason.

Problem?

When a Sidekiq server initializes, it establishes a connection pool to manage database queries. During job execution, it retrieves a connection from this pool. If a specific database is not specified for the job, it defaults to the primary database (default - db 1).

Addressing the Database Connection Issue

To ensure that background jobs access the correct database, we need to pass the database name as a parameter to each job and modify the job accordingly:

# /app/controllers/articles_controller.rb
def run_background_job
  MultiDbTestingJob.perform_later(shard_name)

  redirect_to root_path
end

# /app/sidekiq/multi_db_testing.rb
class MultiDbTestingJob < ApplicationJob
  def perform(shard)
    ActiveRecord::Base.connected_to(shard: shard) do
      p "Number of articles in DB is #{Article.count}"
    end
  end
end

Now, you’ll get the desired result for both databases.

However, this approach has its drawbacks:

For each background job, we need to pass an additional parameter.
We need to write additional code to connect to the correct database for each background job.

To address these issues, we can create a Sidekiq adapter that will decide which database to connect to based on the database that initiated the background job. But before creating the adapter, we need a global attribute to remember which database we are connected to. To achieve this, Rails CurrentAttributes and Sidekiq Middleware will be utilized.

Current Attributes

From the definition of Current Attributes, Abstract super class that provides a thread-isolated attributes singleton, which resets automatically before and after each request. This allows you to keep all the per-request attributes easily available to the whole system.

# app/models/current.rb
class Current < ActiveSupport::CurrentAttributes
  attribute :tenant
end

# app/controllers/application_controller.rb
before_action :setup_tenant

def setup_tenant
  tenants = Rails.application.config_for(:settings)[:tenants]
  current_tenant = tenants.keys.find { |key| tenants[key][:hosts].include?(request.env['HTTP_HOST']) } || :app1_shard
  Current.tenant = current_tenant.to_sym
end

Note - Sidekiq also introduced the cattr feature, this will help in persisting the value of current attributes when sidekiq job runs. Read More

Sidekiq Middleware

It is a set of customizable modules that intercept and augment the behavior of Sidekiq job processing in Ruby on Rails applications. Sidekiq Middleware

Create file config/initializers/sidekiq.rb and paste following code.

# config/initializers/sidekiq.rb
    require 'sidekiq'
    require 'sidekiq/web'
    require 'sidekiq/middleware/current_attributes'
    require_relative '../../app/middleware/sidekiq_adapter'

    Sidekiq::CurrentAttributes.persist('Current')

    Sidekiq.configure_server do |config|
      config.server_middleware do |chain|
        chain.add Middleware::SidekiqAdapter
      end
    end

Create file app/middleware/sidekiq_adapter.rb and paste following code.

module Middleware
      class SidekiqAdapter
        include Sidekiq::ServerMiddleware

        def call(job_instance, job_payload, queue)
          shard = current_shard(job_payload)
          ApplicationRecord.connected_to(shard: shard, role: :writing) do
            yield
          end
        rescue StandardError => e
          p "Error occured #{e}"
        end

        def current_shard(job_payload)
          job_payload.try(:[], 'cattr').try(:[], 'tenant')&.to_sym
        end
      end
    end

With the middleware in place, we can simplify our Sidekiq job and remove the shard logic from it. The middleware will handle connecting to the correct shard.

# multi_db_testing_job.rb
  class MultiDbTestingJob < ApplicationJob
    def perform
      p "Number of articles in DB is #{Article.count}"
    end
  end


  # /app/controllers/articles_controller.rb
  def run_background_job
    MultiDbTestingJob.perform_later(shard_name)

    redirect_to root_path
  end

Run the project again and subsqeuently run the sidekiq job to test it out.

You will notice that with the middleware in place, when executing a background job, it connects to the correct database.
Code - Github Link

Summary

In this blog post, we solved database issue with background job processing in a multi-tenant Rails application. We introduced a custom Sidekiq middleware adapter, that fixes the issue of running background jobs across multiple databases. This approach provides a robust & scalable framework for managing background job execution in complex multi-tenant environments.

Mastering Multi Tenant setup with rails - background jobs was originally published by eLitmus.com at eLitmus Blog on May 05, 2024.

Mastering Multi Tenant setup with rails part 1

2023-12-17 14:17:45 +0530T00:00:00-00:00

Multi-tenancy is a software design where a single instance of a software application serves multiple customers or tenants (individual users or organizations). In a multi-tenant architecture, each tenant’s data and configuration are logically isolated from one another, providing a sense of individuality and privacy while sharing the same underlying infrastructure, codebase, and application instance.

Single Tenant application

In a single-tenant application, each hosted instance has its dedicated database. Upon addition of a new organization that requires segregated data, a new application is hosted with a different database.

Multi Tenant Application types

Single Database shared rows
- Each table in database will contain an additional row known as tenant_id.
- Whenever data is stored and retrieved from table this coloumn will be used to get/store the data.
- Only the data that belongs to a specific customer/tenant will be fetched.
Single Database shared schema
- For each tenant a different table will be maintained in same database.
- Data will be segregated table wise.
Dedicated Database for Each Tenant
- For each tenant a new database schema will be maintained, it can be termed as shard.

In this blog post, we’ll take an in-depth look at the third approach, where we opt to manage separate databases for each tenant. To demonstrate this, we’ll walk through the process of creating a basic Rails blog application from the ground up.

Goal

Setting up a multi-tenant application in development mode.
dynamically switching databases according to the requesting host name.

What features rails 6 brings in

Rails 6 introduced the multiple database setup with following features -

Multiple writer databases and a replica for each.
Automatic connection switching for the model you’re working with.
Automatic swapping between the writer and replica depending on the HTTP verb and recent writes.
Rails tasks for creating, dropping, migrating, and interacting with the multiple databases.

Setup

Create new rails app

rails new multi_db_blog
update gemfile to use mysql2 instead of sqlite3

Setup databases

In database.yml file update the database with name.

development:
  app1:
    adapter: mysql2
    encoding: utf8
    reconnect: false
    database: app1_development
    pool: 5
    username:
    password:
    socket: /tmp/mysql.sock
    host: 127.0.0.1
  app2:
    adapter: mysql2
    encoding: utf8
    reconnect: false
    database: app2_development
    pool: 5
    username:
    password:
    socket: /tmp/mysql.sock
    host: 127.0.0.1

bin/rake db:create create databases for both the tenants.
You have the option to execute specific rake commands for each database. For instance, you can create the app1 database using the command: bin/rake db:create:app1

Generate Models and Controller

Model

bin/rails generate model Article title:string body:text
Run migrations

bin/rake db:migrate
Controller

bin/rails generate controller Articles index --skip-routes
update routes.rb file.

root "articles#index"
resources :articles

Complete the Articles Controller, Model and respective views by following This Guide

Start App

Run bin/rails s to start the server.
By default rails will connect to db1 now.
This will act as a default database for the current application.

Running up both databases simaltaneously

Install nginx & paste the following code in nginx.conf file.

http {
  server {
   listen 3000;
   server_name localhost;

   location / {
        proxy_pass http://127.0.0.1:3000; # Rails app running on port 3000
        proxy_set_header Host $host:$server_port;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }

  server {
    listen 4000;
    server_name localhost; # Change this to your actual domain if needed

    location / {
        proxy_pass http://127.0.0.1:3000; # Rails app running on port 3000
        proxy_set_header Host $host:$server_port;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }
}
events { }

Above nginx configurations listens to port 3000 and 4000 and redirect to rails application running in port 3000.

Additional Rails changes

Since We are using Rails 7 we can use automatic shard swap feature provided by rails. if using rails 6.1 or 6, a middleware can be introduced to automatic switch the tenants depending on request. Visit next section for the details.

Mention list of tenants in a .yml file. You can maintain these records in a separate database as well, for now I will create a settings.yml file.

development:
  tenants:
    app1:
      hosts:
        - localhost:3000
    app2:
      hosts:
        - localhost:4000

update application.rb with following configurations.

Rails.application.configure do
  config.active_record.shard_selector = { lock: true }

  tenants = Rails.application.config_for(:settings)[:tenants]  # maintaining list of tenants with host
  config.active_record.shard_resolver = ->(request) {
    tenants.keys.find { |key| tenants[key][:hosts].include?(request.env['HTTP_HOST']) } || :app1
  }
end

update application_record.rb

# connects_to shards: {
   #  app1: { writing: :app1 },
   #  app2: { writing: :app2 }
   # }
   # OR

  TENANTS = Rails.application.config_for(:settings)[:tenants]
  connects_to TENANTS.keys.map { |shard| [shard, { writing: shard }] }.to_h

Creating Middleware for automatic shard switching(ignore if using rails 7 or above)

Create a middleware named middleware/tenant_selector.rb
Add following code

module Middleware
    class TenantSelector
      def initialize(app, tenants)
        @app = app
        @tenants = tenants
      end

      attr_reader :tenants

      def call(env)
        request = ActionDispatch::Request.new(env)
        tenant = selected_tenant(request)

        set_tenant(tenant) do
          @app.call(env)
        end
      end

      private
      def selected_tenant(request)
        tenants.keys.find { |key| tenants[key][:hosts].include?(request.env['HTTP_HOST']) } || :app1
      end

      def set_tenant(tenant, &block)
        ActiveRecord::Base.connected_to(shard: tenant.to_sym, role: :writing) do
          yield
        end
      end
    end
  end

Update application.rb file with following changes.

tenants = Rails.application.config_for(:settings)[:tenants]
    config.app_middleware.use Middleware::TenantSelector, tenants

Final Steps

Follow these final steps to confirm your multi-tenant Rails application is up and running smoothly:

Run bin/rails s
Access localhost:3000 to connect to db1
Access localhost:4000 to connect to db2
If you wish to add more databases, simply update the database.yml and settings.yml files

What Next?

In the upcoming series of blog posts, we will delve into the following topics:

Maintaining Background Jobs.
Running Rake Tasks with Cron Jobs for Multiple Databases.
ActiveStorage Data Management with Different Storage Types for Each Tenant.
Caching.

Summary

In this blog post we covered creating a multi tenant application from scratch and setting it up in development environment. We were able to automatically switch databases according to type of database.

References

Mastering Multi Tenant setup with rails part 1 was originally published by eLitmus.com at eLitmus Blog on December 17, 2023.

An in-depth look at Database Indexing

2023-12-10 15:49:39 +0530T00:00:00-00:00

In this article, we will explore Database Indexing. We will begin by installing the Docker & running a Postgres container on it. Subsequently, to execute queries and comprehend how the database uses various indexing strategies, we will insert millions of rows into a Postgres table.

Following that, we will explore different tools to gaining insights into the SQL query planner and optimizers. After that, we will delve into understanding database indexing, examining how various types of indexing works with examples, and do a comparison between different types of database scan strategies.

Finally, we will then demystify how database indexes operates for the WHERE clause with the AND and OR operators.

Prerequisite

Installing Docker & Running a Postgres Container:
1. Install Docker by following the instructions provided in the getting started guide on the official Docker website.
2. Verify that Docker is installed by running the command docker --version
Running a PostgresSQL Container:
1. Spin up the Docker container by using the official Postgres image.
```
docker run -e POSTGRES_PASSWORD=secret --name pg postgres
```
2. Start the Postgres command shell:
```
docker exec -it pg psql -U postgres
```

Inserting a Million Rows into a Postgres Table:

Create a table named employees:

create table employees(id serial primary key, employeeid integer, name TEXT);

Insert into the employees table using the generate_series function:

create or replace function gen_random_string(length integer)
RETURNS VARCHAR as
$$
DECLARE
result VARCHAR := '';
BEGIN
FOR i IN 1..length LOOP
	result := result || chr((floor(random() * 26) + 65)::integer);
END LOOP;
RETURN result;
END;
$$ language plpgsql;

INSERT INTO EMPLOYEES(employeeid, name)
SELECT *, gen_random_string((random() * 10 + 5)::integer)
FROM generate_series(0, 1000000);

Confirm the result by executing the count query:

select count(*) from employees;
count
--------
	1000001
(1 row)

This sequence of steps creates a table named employees and inserts one million rows into it, generating random values for the employeeid and name columns. The final count query verifies the successful insertion of the specified number of rows.

The SQL Query Planner and Optimizer:
- Explanation: The explain command displays the execution plan generated by the PostgresSQL planner for the provided statement. This plan illustrates how the table(s) referenced in the statement will be scanned, whether through plain sequential scans, index scans, etc.
- Examples:
  1. Select All Query:
```
postgres=# explain select * from employees;
QUERY PLAN
-------------------------------------------------------------
Seq Scan on employees (cost=0.00..16139.01 rows=1000001 width=19)
```
    - Seq Scan: Directly goes to the heap and fetches everything, similar to a Full Table Scan in other databases. In Postgres, with multiple threads, it's called Parallel Seq Scan.
    - Cost=0.00..16139.01: The first number represents work before fetching (e.g., aggregating, ordering), and the second number is the total estimated execution time.
    - rows=1000001: An approximate number of rows to be fetched.
    - width=19: The sum of bytes for all columns.
  2. Select All Query with Order By (Indexed Column):
```
postgres=# create index employees_employeeid_idx ON employees(employeeid);
CREATE INDEX
postgres=# explain select * from employees order by employeeid;
QUERY PLAN
-----------------------------------------------------------------
Index Scan using employees_employeeid_idx on employees (cost=0.42..32122.44 rows=1000001 width=19)
```
    - cost=0.42: Postgres performs work, ordering by employeeid. An index on employeeid leads to an Index Scan.
  3. Select All Query with Order By (Non-Indexed Column):
```
postgres=# explain select * from employees order by name;
QUERY PLAN
--------------------------------------------------------------
Sort (cost=136306.96..138806.96 rows=1000001 width=19)
	Sort Key: name
	-> Seq Scan on employees (cost=0.00..16139.01 rows=1000001 width=19)
```
    - Seq Scan & Sort: Seq Scan on the table, followed by sorting. Sorting cost is critical.
  4. Select Only ID:
```
postgres=# explain select id from employees;
QUERY PLAN
---------------------------------------------------
Seq Scan on employees (cost=0.00..16139.01 rows=1000001 width=4)
```
    - width=4: Fetching only id, resulting in a smaller width of 4 bytes (integer).
  5. Select All Query for a Particular ID:
```
postgres=# explain select * from employees where id = 10;
QUERY PLAN
-------------------------------------------------------------------
Index Scan using employees_pkey on employees (cost=0.42..8.44 rows=1 width=19)
	Index Cond: (id = 10)
```
    - rows=1: Fetching only 1 record using the primary key index.

What is Database indexing?

An index is a data structure that speeds up data retrieval without needing to scan every row present in the table. Index improves lookup performance but decreases write performance because every time a new row is created, indexes need to be updated.

Indexes are typically stored on the disk. An index is typically a small table with two columns: a primary/candidate key and address. Keys are made from one or more columns.

The data structure used for storing the index is B+ Trees. In the simplest form, an index is a stored table of key-value pairs that allows searches to be conducted in O(logn) time using binary search on sorted data.

Types of Indexes:

Clustered Index
- Index and data reside together and are ordered by the key. A Clustered Index is basically a tree-organized table. Instead of storing the records in an unsorted Heap table space, the clustered index is actually B+Tree index having the Leaf Nodes, which are ordered by the clusters key column value, store the actual table records, as illustrated by the following diagram.
Nonclustered Index
- A nonclustered index contains the key values and each key value entry has a pointer to the data row that contains the key value. Since the Clustered Index is usually built using the Primary Key column values, if you want to speed up queries that use some other column, then you'll have to add a Secondary Non-Clustered Index. The Secondary Index is going to store the Primary Key value in its Leaf Nodes, as illustrated by the following diagram

How database indexes works under the hood?

We have already created a database index on the employeeid column in our employees table using the CREATE INDEX statement. Behind the scenes, Postgres creates a new pseudo-table in the database with two columns: a value for employeeid and a pointer to the corresponding record in the employees table. This pseudo-table is organized and stored as a binary tree with ordered values for the employeeid column. Consequently, the query operates with O(logn) efficiency and typically executes in a second or less.

Let’s delve into two scenarios:

SELECT * FROM employees WHERE employeeid = 4;
Here, with an index on the employeeid column, the query initiates an Index Scan. The process begins by accessing the Index table, retrieving the reference for the Page number, and obtaining the row number for the specific record on that page. Subsequently, it navigates to the corresponding page in the heap and fetches the entire row. This method, known as an Index Scan.
SELECT employeeid FROM employees WHERE employeeid = 4;
In this instance, there is no need to access the heap to retrieve the complete record. Since the required value for employeeid is already present in the index table, the operation is streamlined, and it directly performs an Index Only Scan. This approach allows the system to retrieve the specific employeeid directly from the index table without the additional step of fetching the complete row from the heap. This can lead to improved performance, particularly when the index includes all the columns needed for the query, minimizing the amount of data that needs to be processed.

What are different type of database scan strategies?

Index Only Scan
```
postgres=# EXPLAIN ANALYZE select id from employees where id = 100;
							QUERY PLAN
-----------------------------------------------------------------------------
 Index Only Scan using employees_pkey on employees  (cost=0.42..4.44 rows=1 width=4) (actual time=2.529..2.542 rows=1 loops=1)
   Index Cond: (id = 100)
   Heap Fetches: 0
 Planning Time: 0.510 ms
 Execution Time: 2.708 ms
```
If we examine the given query, we retrieve the ID using a filter on the ID column, which serves as the primary key and has an index on it. Let's break down the query output:
1. Index Only Scan: In the case of an Index Only Scan, Postgres scans the index table, resulting in faster performance as the Index table is significantly smaller than the actual table. With Index Only Scan, results are directly fetched from the Index table when querying columns for which indexes have been created.
2. Heap Fetches: 0: This indicates that the queried ID value did not necessitate accessing the heap table to retrieve information. The information was obtained inline, and this is referred to as an Inline query.
3. Planning Time: 0.510 ms: This represents the time taken by Postgres to determine whether to use the index or perform a full table scan.
4. Execution Time: 2.708 ms: This is the time taken by Postgres to actually fetch the records from the table.

Index Scan

postgres=# EXPLAIN ANALYZE select name from employees where id = 1000;
								QUERY PLAN
----------------------------------------------------------------------------
 Index Scan using employees_pkey on employees  (cost=0.42..8.44 rows=1 width=11) (actual time=1.250..1.260 rows=1 loops=1)
   Index Cond: (id = 1000)
 Planning Time: 0.703 ms
 Execution Time: 1.655 ms

If we examine the given query, we are retrieving the name using a filter on the ID column, which serves as the primary key and has an index on it. In this case, the process begins with an index scan on the Index table to retrieve information about the Page number and row number on the Heap. Since the name is not available in the Index table, we must go to the heap to fetch the name. This type of scan is referred to as an Index Scan.

postgres=# EXPLAIN ANALYZE select name from employees where id < 1000;
                      QUERY PLAN
----------------------------------------------------------------------------------------------------------
 Index Scan using employees_pkey on employees  (cost=0.42..40.75 rows=1047 width=32) (actual time=0.062..1.139 rows=999 loops=1)
   Index Cond: (id < 1000)
 Planning Time: 4.948 ms
 Execution Time: 1.215 ms
(4 rows)

In this case, we are filtering the record using the filter on the id with '<' operator and filtering out record which have id less than 1000. So the process begins with an Index scanning on the Index table then fetching the rows from the heap. Same as in case of fetching single id.

postgres=# EXPLAIN ANALYZE select name from employees where id > 1000;
                    QUERY PLAN
---------------------------------------------------------------------------
 Seq Scan on employees  (cost=0.00..18639.01 rows=998953 width=32) (actual time=0.104..168.884 rows=999001 loops=1)
   Filter: (id > 1000)
   Rows Removed by Filter: 1000
 Planning Time: 0.158 ms
 Execution Time: 198.259 ms
(5 rows)

In this case, we are filtering the record using the filter on the id with '>' operator and filtering out records which have id greater than 1000. So, in this case, as Postgres knows it has to fetch 99% of the data anyway, it prefers to use the Seq Scan on the heap table. Rather than going to the Index table to filter out the records and then again going to the heap to filter those Index-scanned rows.

Parallel Seq Scan

postgres=# EXPLAIN ANALYZE select id from employees where name = 'WABOY';
								QUERY PLAN
----------------------------------------------------------------------------
 Gather  (cost=1000.00..12347.44 rows=1 width=4) (actual time=3.970..120.383 rows=1 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   ->  Parallel Seq Scan on employees  (cost=0.00..11347.34 rows=1 width=4) (actual time=64.894..102.448 rows=0 loops=3)
		 Filter: ((name)::text = 'WABOY'::text)
		 Rows Removed by Filter: 333333
 Planning Time: 0.898 ms
 Execution Time: 120.850 ms
(8 rows)

If we examine the given query, we are retrieving the id using a filter on the name column, which doesn't have an index on it. As we don't have an index on the name column, that means we have to actually search for the name WABOY one by one and perform a sequential scan on the employees table. Postgres efficiently addresses this by executing multiple worker threads and conducting a parallel sequential scan.

Bitmap Scan
Let's create a Bitmap Index on the name column to get started.
```
postgres=# CREATE INDEX employees_name_idx ON employees(name);
CREATE INDEX
```
Let's explore how Bitmap Scan works in PostgreSQL.

Heap pages are stored on disk, and loading a page into memory can be expensive. When using an Index Scan, if the query yields a large number of rows, the query's performance may suffer because each row's retrieval involves loading a page into memory.
In contrast, with a Bitmap Scan, instead of loading rows into memory, we set a bit to 1 in an array of bits corresponding to heap page numbers. The operation then works on top of this bitmap.
Here's a simplified breakdown of above image:
- In a bitmap index scan, rows are not loaded into memory. PostgreSQL sets the bit to 1 for page number 1 when the name is 'CD' and 0 for other pages.
- When the name is 'BC', page number 2 is set to 1, and others are set to 0.
- Subsequently, a new bitmap is created by performing an OR operation on both bitmaps.
- Finally, PostgreSQL executes a Bitmap Heap Scan where it fully scans each heap page and rechecks the conditions.
This approach minimizes the need to load entire pages into memory for individual rows, improving the efficiency of the query. If the query results in a lot of rows located in only a limited number of heap pages then this strategy will be very efficient.

Now let's filter out the id, name by the name
```
postgres=# EXPLAIN ANALYZE select id, name from employees where name = 'WABOY';
                  QUERY PLAN
--------------------------------------------------------------------------------
 Bitmap Heap Scan on employees  (cost=111.17..6277.29 rows=5000 width=36) (actual time=0.348..0.369 rows=1 loops=1)
   Recheck Cond: (name = 'WABOY'::text)
   Heap Blocks: exact=1
   ->  Bitmap Index Scan on employees_name_idx  (cost=0.00..109.92 rows=5000 width=0) (actual time=0.274..0.274 rows=1 loops=1)
         Index Cond: (name = 'WABOY'::text)
 Planning Time: 0.905 ms
 Execution Time: 0.734 ms
```
Upon analyzing the provided query, we extract the id and name by applying a filter on the name column, which has an index. Let's clarify the process:
1. Bitmap Index Scan: This step involves scanning the index table for the name column since an index exists on it. It retrieves the page number and row number to obtain references to the corresponding records in the heap.
2. Bitmap Heap Scan: Since we are filtering based on both id and name, this step is necessary to visit the heap and retrieve the values for both attributes for a specific record. The reference to the record is obtained from the preceding Bitmap Index Scan.

Combining Database Indexes

Prerequisite: Let's create a table to learn how to combine indexes.

CREATE TABLE NUMBERS(id serial primary key, a integer, b integer, c integer);
INSERT INTO NUMBERS(a, b, c) select (random() * 100)::integer, (random() * 1000)::integer, (random() * 2000)::integer from generate_series(0, 10000000);

Now let's create index on the columns A and B.

CREATE INDEX numbers_a_idx on numbers(a);
CREATE INDEX numbers_b_idx on numbers(b);

Select column c for a particular value of column a

postgres=# EXPLAIN ANALYZE SELECT c FROM numbers WHERE a = 88;
                  QUERY PLAN

---------------------------------------------------------------------------------
 Bitmap Heap Scan on numbers  (cost=1101.09..57496.05 rows=98665 width=4) (actual time=41.110..683.631 rows=99888 loops=1)
   Recheck Cond: (a = 88)
   Heap Blocks: exact=45619
   ->  Bitmap Index Scan on numbers_a_idx  (cost=0.00..1076.42 rows=98665 width=0) (actual time=29.403..29.403 rows=99888 loops=1)
         Index Cond: (a = 88)
 Planning Time: 1.569 ms
 Execution Time: 687.152 ms

Here, we can analyze that since we have an index only on column a, a bitmap index scan is performed on column a. To retrieve column c, it jumps to the heap and performs a bitmap heap scan.

Select column c but we are going to query on both a and b with AND operation

postgres=# EXPLAIN ANALYZE SELECT c FROM numbers WHERE a = 90 AND b = 500;
                  QUERY PLAN
-----------------------------------------------------------------------------
 Bitmap Heap Scan on numbers  (cost=1320.12..1746.88 rows=110 width=4) (actual time=32.300..38.262 rows=107 loops=1)
   Recheck Cond: ((b = 500) AND (a = 90))
   Heap Blocks: exact=107
   ->  BitmapAnd  (cost=1320.12..1320.12 rows=110 width=0) (actual time=32.079..32.081 rows=0 loops=1)
         ->  Bitmap Index Scan on numbers_b_idx  (cost=0.00..110.88 rows=9926 width=0) (actual time=4.494..4.494 rows=9974 loops=1
)
               Index Cond: (b = 500)
         ->  Bitmap Index Scan on numbers_a_idx  (cost=0.00..1208.93 rows=111000 width=0) (actual time=26.799..26.800 rows=99868 l
oops=1)
               Index Cond: (a = 90)
 Planning Time: 3.362 ms
 Execution Time: 38.604 ms

Here, we can analyze the following:

PostgreSQL executed a bitmap index scan on column 'A'.
Concurrently, a bitmap index scan was performed on column 'B'.
Subsequently, PostgreSQL executed a bitmap AND operation to combine the results of the scans on 'A' and 'B'.
After obtaining the references for the rows to be retrieved, PostgreSQL proceeds to perform a bitmap heap scan.

Select column c but we are going to query on both a and b with OR operation.

postgres=# EXPLAIN ANALYZE SELECT c FROM numbers WHERE A = 50 OR B = 500;
                QUERY PLAN

-------------------------------------------------------------------------------
 Bitmap Heap Scan on numbers  (cost=1164.23..57490.68 rows=101835 width=4) (actual time=37.957..600.439 rows=109466 loops=1)
   Recheck Cond: ((a = 50) OR (b = 500))
   Heap Blocks: exact=46998
   ->  BitmapOr  (cost=1164.23..1164.23 rows=101926 width=0) (actual time=25.625..25.626 rows=0 loops=1)
         ->  Bitmap Index Scan on numbers_a_idx  (cost=0.00..1002.43 rows=92000 width=0) (actual time=24.309..24.309 rows=99602 lo
ops=1)
               Index Cond: (a = 50)
         ->  Bitmap Index Scan on numbers_b_idx  (cost=0.00..110.88 rows=9926 width=0) (actual time=1.313..1.314 rows=9974 loops=1
)
               Index Cond: (b = 500)
 Planning Time: 1.135 ms
 Execution Time: 604.165 ms

Here, we can analyze the following:

PostgreSQL executed a bitmap index scan on column a.
Concurrently, a bitmap index scan was performed on column b.
Subsequently, PostgreSQL executed a BitmapOr operation to combine the results of the scans on columns a and b.
After obtaining the references for the rows to be retrieved, PostgreSQL proceeds to perform a bitmap heap scan.

Composite Index
First, we need to drop the indexes on both columns a and b, and then create a composite index on columns a and b.
```
postgres=# CREATE INDEX numbers_a_b_idx on numbers(a, b);
CREATE INDEX
```

Select column c for a particular value of column a

postgres=# EXPLAIN ANALYZE SELECT c FROM numbers WHERE a = 70;
							QUERY PLAN

-----------------------------------------------------------------------
Bitmap Heap Scan on numbers  (cost=1189.93..56830.03 rows=106000 width=4) (actual time=38.779..610.173 rows=99789 loops=1)
	Recheck Cond: (a = 70)
	Heap Blocks: exact=45549
	->  Bitmap Index Scan on numbers_a_b_idx  (cost=0.00..1163.43 rows=106000 width=0) (actual time=27.796..27.797 rows=99789 loops
=1)
		Index Cond: (a = 70)
Planning Time: 5.188 ms
Execution Time: 613.305 ms
(7 rows)

Here, we can analyze the following:

This time, PostgreSQL decided to use the composite index numbers_ab_idx on both columns a and b.
Subsequently, it performs a Bitmap Heap Scan on the selected rows based on the composite index.

Select column c for a particular value of column b

postgres=# EXPLAIN ANALYZE SELECT c FROM numbers WHERE b = 900;
							QUERY PLAN
-----------------------------------------------------------------------
Gather  (cost=1000.00..108130.94 rows=9926 width=4) (actual time=24.402..395.326 rows=10027 loops=1)
	Workers Planned: 2
	Workers Launched: 2
	->  Parallel Seq Scan on numbers  (cost=0.00..106138.34 rows=4136 width=4) (actual time=9.913..317.809 rows=3342 loops=3)
		Filter: (b = 900)
		Rows Removed by Filter: 3329991
Planning Time: 0.574 ms
JIT:
	Functions: 12
	Options: Inlining false, Optimization false, Expressions true, Deforming true
	Timing: Generation 4.899 ms, Inlining 0.000 ms, Optimization 3.039 ms, Emission 25.030 ms, Total 32.968 ms
Execution Time: 398.820 ms
(12 rows)

Here, we can analyze the following:

This time, Postgres did not use the index numbers_a_b_idx. Even though we have a composite index on both columns a and b. Why? Because we cannot use this composite index when scanning a filter. The filter condition is on column a, and the composite index can be used for conditions involving both columns a and b or just column a. However, it cannot be used for conditions involving only column b. Therefore, if we have a composite index on columns a and b, querying on column b alone will not utilize the index.

Select column c but we are going to query on both A and B with AND operation

postgres=# EXPLAIN ANALYZE SELECT C FROM numbers WHERE A = 60 AND B = 600;
							QUERY PLAN
-------------------------------------------------------------------------
Bitmap Heap Scan on numbers  (cost=5.44..386.39 rows=98 width=4) (actual time=0.732..6.281 rows=102 loops=1)
	Recheck Cond: ((a = 60) AND (b = 600))
	Heap Blocks: exact=101
	->  Bitmap Index Scan on numbers_a_b_idx  (cost=0.00..5.42 rows=98 width=0) (actual time=0.513..0.513 rows=102 loops=1)
		Index Cond: ((a = 60) AND (b = 600))
Planning Time: 0.756 ms
Execution Time: 6.659 ms

Here, the situation remains the same as earlier when we had an index on both columns A and B.

Select column c but we are going to query on both A and B with OR operation

postgres=# EXPLAIN ANALYZE SELECT C FROM numbers WHERE A = 60  or B = 80;
							QUERY PLAN
------------------------------------------------------------------------
Gather  (cost=1000.00..128404.51 rows=108495 width=4) (actual time=20.721..388.512 rows=109443 loops=1)
	Workers Planned: 2
	Workers Launched: 2
	->  Parallel Seq Scan on numbers  (cost=0.00..116555.01 rows=45206 width=4) (actual time=8.325..304.804 rows=36481 loops=3)
		Filter: ((a = 60) OR (b = 80))
		Rows Removed by Filter: 3296853
Planning Time: 1.009 ms
JIT:
	Functions: 12
	Options: Inlining false, Optimization false, Expressions true, Deforming true
	Timing: Generation 5.795 ms, Inlining 0.000 ms, Optimization 2.561 ms, Emission 21.798 ms, Total 30.154 ms
Execution Time: 397.675 ms

Here, we can analyze the situation as follows:

As observed earlier, it's not feasible to use a composite index on column B individually. The option is either to use it on column A alone or on both columns A and B. Consequently, Postgres opts for a Parallel Sequential Scan in this scenario.

An in-depth look at Database Indexing was originally published by eLitmus.com at eLitmus Blog on December 10, 2023.

How to setup AWS Cloudwatch alarm for your SES reputation metrics

2023-11-20 18:58:55 +0530T00:00:00-00:00

Amazon Simple Email Service (SES) is an email platform that offers a straightforward and cost-effective way for you to send and receive emails using your own email addresses and domains.

AWS SES has associated reputation metrics (Bounce & Complaint rate), and if these metrics exceed the threshold limit, AWS may disable your email service, potentially causing a significant impact on your business.

Why not create an alarm that monitors these reputation metrics and notifies you when they approach the threshold value? This way, you can prevent email service downtime.

Fortunately, AWS provides a few services that, when combined, can help you easily set up the SES reputation metrics alarm.

Amazon Simple Email Service

I recommend following deliver the mail with Amazon SES and Rails article to set up AWS SES for your Ruby on Rails application, as I’ll be using Ruby on Rails as my backend language.

Amazon Simple Notification Service

Amazon Simple Notification Service (SNS) is a web service that coordinates and manages message delivery from publishers to subscribers. You can learn more about it here.

We will configure SNS to send notifications to both an email address and an API endpoint in your backend server.

Steps to create SNS topic
- Go to AWS SNS dashboard and click on the Create Topic button.
- Select Standard as the type of topic.
- Type a name for the topic. For example, ses-reputation-notifier.
- Click on Create topic button.
Steps to create subscription for SNS topic
- Go to AWS SNS dashboard and click on the Create Subscription button.
- Select the SNS topic you created from Topic ARN
- Choose the protocol from the list of protocols.
  - Email
    - Select the Email protocol.
    - Enter your email in the endpoint.
    - You’ll receive a subscription URL on your email. Visit this URL to subscribe to the SNS topic.
  - HTTP/HTTPS
    - We will configure the HTTP/HTTPS protocol after creating the public API endpoint in later part of the blog.

Amazon Cloudwatch

Amazon CloudWatch monitors your Amazon Web Services (AWS) resources and the applications you run on AWS in real time. You can use CloudWatch to collect and track metrics, which are variables you can measure for your resources and applications. You can learn more about it here.

For our SES reputation monitoring alarm, we’ll require four alarms. Two will be for monitoring bounce rate, and two will be for monitoring complaint rate. We create two alarms for each reputation metric because the first alarm triggers when the reputation metric exceeds the threshold, and the second alarm triggers when the reputation metric returns to normal.

Steps to create alarm

1. Bounce rate (OK -> ALARM)

This alarm will activate when the Bounce rate surpasses the set limit. This transition will change the alarm state from OK to ALARM.
- Go to AWS Cloudwatch and click on Create alarm.
- Click on Select metrics and select SES > Account Metrics > Reputation.BounceRate and click on Select metrics
- Select 1 hour from period dropdown.
- Fill Define the threshold value with 0.05 (suggested by AWS).
- Click on Additional configuration and from dropdown select Treat Missing data as Good.
- From Alarm state trigger select In alarm.
- From Send a notification to… select the SNS topic you created.
- Type a name for the alarm. For example, bounce-rate-threshold-exceeded and click on Create alarm.
2. Bounce rate (ALARM -> OK)

This alarm will activate when the Bounce rate returns within the specified limit. This transition will change the alarm state from ALARM to OK.
- The procedure will be same for as for Alarm #1.
- Just OK will be selected for Alarm state trigger.
- Type a name for the alarm. for example, bounce-rate-threshold-inlimit
3. Complaint rate (OK -> ALARM)

This alarm will activate when the Complaint rate surpasses the set limit. This transition will change the alarm state from OK to ALARM.
- Go to AWS Cloudwatch and click on Create alarm.
- Click on Select metrics and select SES > Account Metrics > Reputation.ComplaintRate and click on Select metrics
- Select 1 hour from period dropdown.
- Fill Define the threshold value with 0.001 (suggested by AWS).
- Click on Additional configuration and from dropdown select Treat Missing data as Good.
- From Alarm state trigger select In alarm.
- From Send a notification to… select the SNS topic you created.
- Type a name for the alarm. For example, complaint-rate-threshold-exceeded and click on Create alarm.
4. Complaint rate (ALARM -> OK)

This alarm will activate when the Complaint rate returns within the specified limit. This transition will change the alarm state from ALARM to OK.
- The procedure will be same for as for Alarm #3.
- Just OK will be selected for Alarm state trigger.
- Type a name for the alarm. for example, complaint-rate-threshold-inlimit

API endpoint to receive POST request

You’ll need an API endpoint to receive POST requests from AWS SNS.

Create a file in app > controllers > sns_notification_controller.rb

class SnsNotificationController < ApplicationController
      skip_before_action :verify_authenticity_token
      before_action :authenticate_request

      def ses_reputation_notifier
        case message_body['Type']
        when 'SubscriptionConfirmation'
          Rails.logger.error(message_body['SubscribeURL'])
        when 'Notification'
          message = JSON.parse(message_body['Message'])

          alarm_active = message['NewStateValue'] == 'ALARM'
          // Your logic based on alarm status
        end

        head :ok
      end

      private

        def authenticate_request
          head :unauthorized if raw_post.blank? || !message_verifier.authentic?(raw_post)
        end

        def raw_post
          @raw_post ||= request.raw_post
        end

        def message_body
          @message_body ||= JSON.parse(raw_post)
        end

        def message_verifier
          @message_verifier ||= Aws::SNS::MessageVerifier.new
        end
    end

The code above uses the official AWS SNS SDK.

Add a route for the ses_reputation_notifier action in the sns_notification within the config/routes.rb file.

post '/ses_reputation_notifier', to: 'sns_notification#ses_reputation_notifier'

The API endpoint needs to be a public endpoint so that SNS can send notifications without requiring any token. Since it’s a public endpoint, we need to verify the authenticity of the request to ensure it comes from SNS.

There are two types of notifications sent by SNS:

Subscription Confirmation
Notifications

Before being able to receive notifications, we must confirm the subscription by visiting the subscribeUrl sent in the request body. That’s why we log the subscribeURL. Once you visit that URL, you’ll be subscribed to the SNS topic. After subscription, you’ll start receiving notifications.

Steps to create HTTP/HTTPS subscription for SNS topic

Select HTTP or HTTPS protocol.
Enter the public API endpoint URL in the endpoint field. For example: https://your-domain/ses_reputation_notifier
A POST request will be sent to the API endpoint, and the subscription URL will be logged (as specified in the code above). Visit this URL to confirm the subscription.

Now, your SNS topic is configured to publish messages to the specified email and API endpoint.

Conclusion

Now that we’ve set up a CloudWatch alarm to monitor SES reputation metrics, it will notify both via email and API endpoint using SNS. With the notifications received by the server, you can ensure that any potential issues won’t significantly impact the current flow.

References

AWS SES - Here
AWS SNS - Here
AWS Cloudwatch - Here
AWS SNS SDK (Ruby) - Here
Reputation monitoring alarms using CloudWatch - Here

How to setup AWS Cloudwatch alarm for your SES reputation metrics was originally published by eLitmus.com at eLitmus Blog on November 20, 2023.

Building a Collaborative code-editor & Whiteboard: For tech interviews.

2023-11-18 14:53:31 +0530T00:00:00-00:00

A Collaborative code-editor and Whiteboard aims to diversify the current interviewing scenario to Live - Coding, Sketching, Crafting, and Conceptualisation of ideas between the Interviewer and Interviewee for better communication which was until now only limited to Video and audio chats.

The Blog Blueprint:

Need for collaboration: Unveiling the Why
Cracking the Code: Research and Concluded Solutions
- Collaborative conflict management.
- Code-editor research
- Existing code-editors
- Existing whiteboards
Feature Showcase
Architecture
Actors: Use case diagram
Dockerization, Deployment and scale
Conclusion

Need for Collaboration: Unveiling the Why

In the post-pandemic landscape, the majority of tech interviews and initial screening rounds have transitioned to virtual formats through platforms like Google Meet, Zoom, or Microsoft Teams. While these platforms excel in facilitating video and audio interactions, they fall short when it comes to assessing candidates’ coding problem-solving skills in a live, interview-pressure setting. Typically, recruiters resort to screen sharing candidates’ local code editors, but this approach presents limitations.

Recruiters lack the ability to actively edit the code or navigate through different code files seamlessly. Simple tasks such as pointing out errors, saving code for later review, brainstorming designs, and optimizing code quality become cumbersome, relying on manual instructions for candidates to scroll up or down.

As a result, the demand for a collaborative code editor and whiteboard becomes not just beneficial but critical in addressing these challenges efficiently.

Cracking the Code: Research and Concluded Solutions

What if two users press two different letters on the same line at the exact same time?

This conflict is a major problem that all collaborative and distributed softwares have to address. Therefore, when the user makes a change to the document that change needs to be synchronised with other users as well and if the user has to wait for this synchronisation for every letter they type this will make the application very slow. So in order to get a near real-time collaborative editing experience every client maintains a local replica of the document and the main issue now is just to maintain consistency with other clients local replica.

Consider the following example - In an empty line, Alice inserts letter ‘A’ at index 0 and Bob inserts letter ‘B’ at index 0 at the exact same time. Now how should we combine these changes together? Should we prefer Alice’s changes over Bob’s or vice-versa?

Well there is no logical answer and quite frankly it doesn’t matter, what actually matters is that at the end of combining these operations it should “Converge to an identical state” i.e both Alice and Bob should get the same text as soon as possible and if they are not satisfied they will re-edit just like in git you would have merge conflicts and you would have to merge manually but here we can automate this merging using various concurrency control and consistency models.

There are two conceptual protocols when dealing with such problems -

Consensus based protocol - Pick one & reject the rest. In this case it will select any one between Alice and Bob, thereby losing one client’s data completely. This is mainly used in decentralised blockchain applications.
Collaboration based protocol - Merge & Keep them all.

But that’s not the end of the story as we can take advantage of commutative operations like consider a double insertion situation - the text initially is “APP”, Alice inserts “H” at the beginning and Bob inserts “Y” at the end at the exact same time as Alice. In such a situation there is a line conflict but since insertions at different positions are commutative we can merge these operations automatically to get the final text “HAPPY”. Thereby not discarding Alice’s changes or Bob’s changes, providing a real-time conflict-free editing experience and there is no need for someone to manually merge these changes for us.

The concurrency control algorithms are widely studied for collaborative tools and a lot of research is currently going into such algorithms to attain distributed concurrency. The two most widely used ones are OT and CRDT.

Operational Transformation (OT)

Also known as Event passing. Any key event happening on the client-side will be sent to the server. Events can be inserting/deleting a character. The operation received by the server from the client is transformed against its operation which results in a new operation to be performed on the client-side. OT implementations are used in Google docs and many more such collaborative applications pre-2018 ( before CRDT optimisations started to outperform their OT counterparts ).

This can be better understood with the help of an example -

There are many models based on Operational Transformation that are used in prod addressing different issues and there is a plethora of dense academic research whose references can be found in the Documentation. As this blog is written mainly from a developer point of view but academic references are also mentioned wherever needed.

Conflict-free Replicated Data Type (CRDT)

CRDT works on two core principles -

Commutative - Re-ordering different operations will not change the final result.
Idempotency - No matter how many times an operation is performed the final result remains the same.

Here, every character in the document is assigned a unique ID, and when a new character is inserted, the new character would get an ID based on the average of its neighbours (ideally could even take non-integer IDs), which helps to make the algorithm less complex in conflict resolution.

Just like OT had its own various models, CRDTs too have different implementations developed & optimised over time. To get a conceptual understanding consider the example below - using Lagoot’s CRDT Algorithm

CRDTs have a lenient approach to the order of operations. In theory, this reduces the design complexity of distribution mechanisms since there is no need for strict serialisation protocols. By tolerating out-of-order updates, the distribution mechanism has to meet simpler integrity guarantees.

OT Vs CRDT: Final Solution

CRDTs serve our requirements much better because of these two main reasons -

Speed: CRDTs are much much faster than many OT implementations applying operations should be possible with just a O(log(n)) lookup. But in the past the main problem was the large memory overhead. Which is now resolved after significant optimisations in YJS CRDT implementation. ( Exact benchmarks are provided in the references section of Documentation. )
Flexibility and Usability: CRDTs support a wider range of data types. Compared to the OT library, which requires a thorough modelling of business logic as different types of operation data structures, when using the CRDT library, you only need to perform the same operation on common data structures, such as Map and Array.

Code-editor Research

The main code-editing component libraries explored were -

Out of which CodeMirror 6 was selected due to the following reasons -

Highly modular and lightweight: seamless integration of only the necessary components, allowing for a tailored and resource-efficient implementation.
Has a modern, extensible API with excellent documentation.
Is easy to customise, style, and reconfigure.

Existing Browser based code-editors

Here are some more projects and companies whose products were researched thoroughly, but to prevent this blog from becoming super lengthy only the names with references are mentioned and all the other details are mentioned in the Proof Of Concept given alongside the Documentation:

Existing Whiteboards

tldraw

Docs
Github
22k+ stars
130 contributors actively maintaining.
Well funded $2.7M seed
Supports YJS ( Shared-editing framework )
Easy to integrate.

Other options researched:

Feature Showcase

Client Workspace

Whiteboard

All features are satisfied as specified in FRS:

Collaborative Code Editor

Real-time code editing and sharing capabilities using websockets.
Syntax highlighting for multiple programming languages using codemirror 6 language packages.
Support for multiple users editing code simultaneously using YJS shared-editing framework.
Files navigator bar for multi-file editing using Depth First Search Algorithm in react.
Create a new meet in just one click!
Fast and low-latency using websockets + sending only new changes made to the document instead of sending the whole document each time.
Modular code as specified in Architecture below.
Secure groups/ rooms using YJS update serialisation.
Additional features provided -
- Zip downloads the entire codebase.
- Pdf download all whiteboard designs.
- Role management for different actors ( or users ) specified below in the use-case diagram.

Whiteboard

Interactive whiteboard for visual collaboration and problem-solving using tldraw.
Drawing tools, shapes, and text annotations.
Real-time updates for all participants using YJS.
Ability to save whiteboard content by downloading a local pdf copy: Simply by pressing Ctrl + P.
Additional features provided -
- Upload any new shape or image for UML diagrams, flowcharts or any other design pattern.
- Dark mode and multi page editing.
- Set custom opacity, texture patterns and border levels.
- Use laser pointers for better presentations.

Architecture

Frontend

The Frontend architecture is composed of CodeMirror 6, a powerful code-editing tool. To enable multi-file support, a file explorer is seamlessly integrated, allowing users to create and delete files or folders within a nested structure. The status bar dynamically displays all open files until they are either closed or removed.

Additionally, the whiteboard functionality is implemented using Tldraw as a separate component. Users can effortlessly switch between the Code Editor and Whiteboard using the left sidebar.

The system adopts a modular approach, breaking down the Code Editor into smaller, manageable components such as Editor-screen, FileTree, and LeftPanel. This modular design enhances code maintainability and fosters a more scalable and extensible architecture.

Backend

Utilising a Node.js server with the Express framework, the application listens for incoming HTTP requests through RESTful API routes. The server efficiently handles data persistence by saving user input code to the database. Additionally, it provides administrative functionality to retrieve all saved data sets from the database.

The system is Augmented with a WebSocket server for low-latency communication and real-time updates, this system further enhances its capabilities. The WebSocket server efficiently receives updates from diverse clients, manages document state updates, and broadcasts these changes to other clients within the same room. This functionality ensures synchronised, real-time communication across connected clients for a seamless and interactive user experience.

Testing

I have used Jest, in conjunction with React Testing Library, for a comprehensive testing suite. This included rigorous testing scenarios for:

Buttons
Connection management.
Form submissions
Page redirections
Error message handling within the project

Actors: Use case diagram

Dockerization, Deployment and scale

I’ve partitioned the project into distinct frontend and backend components, each independently containerized using Docker. This modularization facilitates the potential for separate hosting on different servers, promoting a decoupled architecture. Although, in the current deployment setup, both containers reside on the same EC2 instance, leveraging Docker Compose for streamlined single-host deployment.

This configuration ensures swift and straightforward deployment with efficient container orchestration. Please note that since frontend & backend are decoupled you need to enter the correct ip address of the backend container so that frontend can connect with it properly. For in-depth instructions and troubleshooting guidance during deployment, refer to the comprehensive documentation provided in the code repository.

docker-compose.yml

version: "3.8"

services:
frontend:
build: ./Frontend
container_name: frontend_c
ports: - "3000:3000"
stdin_open: true
tty: true
depends_on: - backend
networks: - mern-network

backend:
build: ./Backend
container_name: backend_c
restart: always
ports: - "5000:5000"
networks: - mern-network

networks:
mern-network:
driver: bridge

Docker run command

sudo docker compose up --build -d

Scaling

Maximum number of users in one room
In this system, WebSockets are adopted over WebRTC for a client-server architecture. The rationale behind this decision lies in the inherent limitations of WebRTC, particularly in scenarios with a large number of users in a room.

WebRTC relies on a mesh topology, where each user establishes connections with every other user in the room. However, as the user count surpasses 15 to 30, the P2P connections become inefficient, leading to increased latency. In contrast, the client-server model with WebSockets exhibits superior scalability, comfortably supporting over 100 users in a room without encountering performance issues.

Reducing user bandwidth
Optimising user bandwidth is achieved through a granular approach of selectively transmitting updates or modifications made to the document, as opposed to repetitively sending the entire document.

In the collaborative environment, the CRDT algorithm is leveraging this approach, conflict resolution is executed solely based on the transmitted changes, eliminating the need to transmit the entire document each time and ensuring a more efficient and bandwidth-conscious collaborative experience.

More detailed scaling information about YJS are specified in the references documentation.

Conclusion

In this exploration, we dived into the intricacies of a Collaborative Code Editor and Whiteboard, aiming to transform the interview process. This enhances communication beyond the traditional bounds of video and audio chats.

The challenges faced during development served as valuable lessons, fortifying my grasp of web development intricacies and inspiring innovative solutions. This journey was not without its hurdles, researching various repositories, digging one article after another, and experimenting with unknown codebases but with perseverance and effective problem-solving, I navigated these challenges, gaining valuable insights and skills in web development.

Feel free to reach out if you’d like to learn more about this project—I’m here to help and answer any questions you might have.

Building a Collaborative code-editor & Whiteboard: For tech interviews. was originally published by eLitmus.com at eLitmus Blog on November 18, 2023.

Puma: From Daemonization to Process Control with Systemctl and Monit

2023-10-21 21:06:27 +0530T00:00:00-00:00

Puma is a popular Ruby web server that is known for its speed and scalability. It has undergone significant changes in recent versions(starting 5.0.0). One of the most notable alterations is the removal of the daemonization feature. But what does it mean?

Daemonization, in the context of web servers, is a process that allows a program to run in the background as a system service. In older versions, Puma made it simple for users to daemonize their processes with a straightforward configuration snippet:

#config/puma.rb
daemonize

However, in recent versions, attempting to use the daemonize code will result in an error, as this functionality has been removed from the codebase.

Why daemonization should not be part of gem?

Incorporating daemonization directly within a gem can lead to undesirable consequences: as explained by Mike Perham in a Blog Post. Here are some key points that should be considered -

Complexity: Adding daemonization features to a gem can make its code more complex and challenging.
Maintenance: The responsibility of maintaining daemonization, automatic restart, and similar core features becomes an additional burden.
Efficiency: System processes are better equipped to manage tasks like daemonization. Delegating this function to the system ensures more efficient and reliable execution, rather than embedding it within the gem.

As a result of these considerations, Puma decided to remove the daemonization feature from the gem.

This decision led us to make some changes in our setup to ensure the smooth running of our applications.

Using Systemd

We had previously implemented daemonization for Sidekiq, which was a process similar to Puma’s needs. Although there were some minor adjustments required for Puma. Here are steps to achieve daemnization through systemctl:

Remove daemonization from config/puma.rb file

Create a file in /lib/systemd/system/puma.service. Below is sample systemd service configuration example, modify it according to your needs.

[Unit]
      Description=Puma HTTP Server
      After=network.target

      [Service]
      Type=notify
      User=username

      WorkingDirectory=/dir/path
      ExecStart=/bin/pumactl start -F /path/puma_config --environment env
      ExecStartPost=/bin/sh -c '/bin/echo $MAINPID > /usr/myapp/shared/pids/puma.pid'
      ExecStop=/bin/kill -TSTP $MAINPID

      RestartSec=10
      Restart=on-failure

      [Install]
      WantedBy=multi-user.target

Two prominent Puma restart strategies are Phased and Hot restarts. Phased restarts are slower but ensure that all workers finish their existing requests before restarting the server, while Hot restarts are faster but come with increased latency during the restart.
To initiate Puma with a phased restart, you can pass the phased-restart option. This choice offers flexibility to adapt Puma's behavior according to specific needs. More about puma restarts Here.

Monit configurations
Monit is a utility for managing and monitoring processes, programs, files, directories and filesystems on a Unix system Monit Docs.
Updated monitrc file

check process puma with pidfile "/usr/myapp/shared/pids/puma.pid"
      start program = "/bin/bash -l -c 'sudo systemctl start puma'" with timeout 20 seconds
      stop program = "/bin/bash -l -c 'sudo systemctl stop puma'" with timeout 20 seconds
      if totalmem is greater than 800 MB for 3 cycles then restart
      if cpu is greater than 65% for 2 cycles then exec "/etc/monit/slack_notifier.sh" else if succeeded then exec "/etc/monit/slack_notifier.sh"

To check if puma is running correctly follow the commands.
```
ps aux | grep puma
    sudo monit summary
```

Exploring Other Alternatives

As alternative to this we considered using puma-daemon gem, which essentially replicated the removed code and maintained it in a separate gem. However, after careful consideration, we chose not to adopt this alternative for the following reasons:

Violation of system standards.
Additional gem and maintainence burden.

Summary

While the removal of daemonization from Puma may require some adjustments, it aligns with the best practices of modern web server management Managing processes at the system level, using tools like systemd and Monit, is considered a more efficient and maintainable approach. Daemonizing processes within application code is discouraged, as it’s a task that falls under the system level. Ultimately, the shift towards system-level process management ensures the stability and efficiency of web applications.

Puma: From Daemonization to Process Control with Systemctl and Monit was originally published by eLitmus.com at eLitmus Blog on October 21, 2023.

Demystifying Rails 7 System Tests: Configuring CI Pipeline

2023-08-28 17:28:05 +0530T00:00:00-00:00

In Rails 5.1 and later versions, system tests were introduced as a new type of test to simulate a user interacting with a web application. These tests use a headless browser, typically powered by Capybara and a WebDriver, to mimic a user’s actions like clicking buttons, filling forms, and navigating through the application.

Why do we need System Tests?

System tests let you test applications in the browser. Because system tests use a real browser experience, you can test all of your JavaScript easily from your test suite.
Typically used for:
- Acceptance testing: verify that the app has implemented a specific feature
- Smoke testing: verify that the app is functional on a fundamental level and doesn't have code issues.
- Characterization testing: is a type of software testing that involves examining and documenting the behavior of an existing system or application without making any modifications to its code

How we can run System Test?

System Test interacts with your app via an actual browser to run them.
From a technical perspective, system tests aren’t necessarily required to interact with a real browser; they can be set up to utilize the rack test backend, which emulates HTTP requests and processes the HTML responses. While system tests based on rack_test run faster and more dependable than front-end tests involving an actual browser, they have notable limitations in mimicking a genuine user experience as they are incapable of executing JavaScript.

The Anatomy of a System Test?

Minitest
- Minitest is a small and incredibly fast unit testing framework.
- It provides the base classes for test cases. For Rails System Tests, Rails provides an ApplicationSystemTestCase base class which is in turn based on ActionDispatch::SystemTestCase:

require "test_helper"

  class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
    driven_by :selenium, using: :chrome, screen_size: [1400, 1400]
  end

In ActionDispatch::SystemTestCase we require the capybara/minitest library.
It provides basics assertions like assert_equal, assert_nil, assert_same, assert_raises, assert_includes.
A runner to run the tests and report on their success and failure.

Capybara
- Capybara starts your app in a separate process before running the tests. This ensures that the tests are run against the correct version of your app.
- Capybara provides a high-level API that makes it easy to write tests in a natural way. For example, you can write a test that says "click the button" instead of having to write code to find the button and click it.
- Here is an example of a test written with Capybara's DSL (Domain Specific Language):

visit('/login')
  fill_in('email', with: 'user@example.com')
  fill_in('password', with: 'password')
  click_button('Login')

Selenium-Webdriver
- Capybara uses the Selenium Webdriver library to interact with real browsers. Selenium WebDriver is a cross-platform library that provides a way to control web browsers from code. Capybara uses Selenium WebDriver to translate its high-level DSL (Domain Specific Language) into low-level commands that the browser can understand.

require "selenium-webdriver"

  driver = Selenium::WebDriver.for :firefox
  driver.navigate.to "http://google.com"

  element = driver.find_element(name: 'q')
  element.send_keys "Hello WebDriver!"
  element.submit

  puts driver.title

  Driver.quit

You can see how it’s a bit lower-level than the Capybara example further up. The selenium-webdriver library translates these calls into WebDriver Protocol, which it speaks to a webdriver executable.

Webdriver Protocol
- The Selenium WebDriver library translates its calls into the WebDriver Protocol. The WebDriver Protocol is a HTTP-based wire protocol that is used to communicate between the Selenium WebDriver library and the web browser.
- In order to start a chrome browser window and navigate to google.com. We need to startup geckodriver.
- We send it a “new session” command with a HTTP post request

curl -X POST 'http://127.0.0.1:9515/session' -d '{"capabilities":{"firstMatch":[{"browserName":"firefox"}]}}'

This return a session id along with data

{ ... "sessionId":"f1776ba558e28309299dc5f62864e977" ... }

Then we make another post request with a session id. And url in data parameters

curl -X POST 'http://127.0.0.1:9515/session/f1776ba558e28309299dc5f62864e977/url' -d '{"url": "https://google.com"}'

Webdriver
- Webdriver is a tool that speaks “Webdriver protocol” and controls the browser.
- Every major browser there is an associated webdriver tool. Chrome has chromedriver. Firefox has a geckodriver. MS Edge has edgedriver. Safari has safaridriver.
- WebDriver tools act as servers: when you execute them, they start a persistent process that listens for HTTP requests until it is terminated.

Webdrivers gem
- Before selenium-webdriver 4.11, webdrivers gem automatically determines which WebDriver executable needs to be downloaded for your platform and selected browser, downloads it, and arranges for that executable to be used by selenium-webdriver.
- From version 4.11, they have incorporated the functionality in selenium-webdriver gem using selenium-manager.

Running Rails 7 System Tests with Docker and Gitlab Runner on Arm64 and Amd64 linux machines

Step 1: Prepare the Rails 7 application for testing

Run the command below to generate a very basic Ruby on Rails 7 app:

rails new minitest-rails-app

Go ahead and open up the project in your favourite editor and proceed to the Gemfile, specifically to the test block:

group :test do
    # Use system testing [https://guides.rubyonrails.org/testing.html#system-testing]
    gem "capybara"
    gem "selenium-webdriver"
    gem "webdrivers"
  end

Next, let’s do a quick scaffold generation to have something to work with:

rails generate scaffold Blog title:string body:text

Usually, generating a scaffold will automatically generate the application_system_test_case.rb and everything you need for the system tests

application_system_test_case.rb (default) 
  
  require "test_helper"
  
  class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
    driven_by :selenium, using: :chrome, screen_size: [1400, 1400]
  end

Run the database commands

rails db:setup
  rails db:migrate

Running a Basic System For the First Time

rails test:system

Step 2: Exclude the gem webdrivers from the list of dependencies

Before selenium-webdriver 4.11, webdrivers gem automatically download webdriver executable.
From version 4.11, they have incorporated the functionality in selenium-webdriver gem using selenium-manager.
We can comment out the webdrivers line from Gemfile.
After change, Gemfile looks like this

group :test do
  # Use system testing [https://guides.rubyonrails.org/testing.html#system-testing]
  gem "capybara"
  gem "selenium-webdriver", "~> 4.11"
  #gem "webdrivers"
  end

Step 3: Point the Selenium-webdriver to use the firefox browser

As chrome has not released binary compatible with linux/arm64 machine. So the test failed on the arm64 linux machine. I tried multiple approaches to make it work with headless_chrome, but didn’t work and commend the issue in details in this issue tracker
We need to change the browser to the firefox.

#application_system_test_case.rb (change driver to Firefox)
 
  require "test_helper"
  
  class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
    driven_by :selenium, using: :firefox, screen_size: [1400, 1400]
  end

Step 4: Prepare the docker image

Create Dockerfile

FROM ruby:3.1.2-slim-buster

  RUN apt-get update
  RUN apt-get -y install gnupg curl wget xvfb unzip

  ENV NODE_VERSION 19

  RUN curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash -  && \
  apt-get install --yes nodejs && \
  apt-get install --yes libxss1 libappindicator1 libindicator7 python2

  RUN apt-get update && \
  apt-get install --yes software-properties-common build-essential libssl-dev sqlite3 libsqlite3-dev pkg-config ca-certificates firefox-esr

  RUN apt-get install -y git-all
  RUN npm install yarn -g
  ADD . /data

This Dockerfile sets up an image with Ruby 3.1.2 and Node.js 19 installed. It installs system dependencies like Git, Yarn, various libraries for sqlite and Firefox.
Build Docker image

docker buildx build -t dockermanishelitmus/systemtest-rails-app:latest1.0 . --platform linux/amd64,linux/arm64 --push

Command is building a Docker image using the buildx extension, targeting two different platforms (Intel/AMD 64-bit and ARM 64-bit), tagging the image as latest1.0, and pushing the resulting image to a container registry.

Step 5: Prepare the gitlab-runner

In the project root directory create a file .gitlab-ci.yml with content

image: "dockermanishelitmus/systemtest-rails-app:latest1.0"
services:
 - redis:latest
variables:
 RAILS_ENV: "test"

cache:
 paths:
   - vendor/ruby
   - node_modules/

before_script:
 - gem install bundler  --no-document
 - bundle config set force_ruby_platform true
 - bundle install
 - bin/rake db:drop
 - bin/rake db:setup
 - bin/rake db:migrate

stages:
 - tests

SystemTests:
 stage: tests
 script:
   - yarn install
   - bin/rake assets:precompile
   - bin/rails test:system
 artifacts:
   when: on_failure
   name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
   paths:
     - coverage/
   expire_in: 1 day

Finally run your test suite

gitlab-runner exec docker SystemTests

Output

$ bin/rails test:system
  Running 4 tests in a single process (parallelization threshold is 50)
  Run options: --seed 13031

  # Running:

  Capybara starting Puma...
  * Version 5.6.7 , codename: Birdie's Version
  * Min threads: 0, max threads: 4
  * Listening on http://127.0.0.1:33385
  ....

  Finished in 7.865541s, 0.5085 runs/s, 0.5085 assertions/s.
  4 runs, 4 assertions, 0 failures, 0 errors, 0 skips
  Saving cache for successful job
  Creating cache SystemTests/main...
  WARNING: vendor/ruby: no matching files. Ensure that the artifact path is relative to the working directory
  node_modules/: found 2 matching files and directories
  No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
  Created cache
  Job succeeded

Conclusion

Now we have a setup that enables us to run system tests in both arm64 and amd64 linux machines with minimal customizations we may want to add. A few tips and tricks should help to get your first system tests up and running in CI pipeline.

Demystifying Rails 7 System Tests: Configuring CI Pipeline was originally published by eLitmus.com at eLitmus Blog on August 28, 2023.

Interview-Platform:Insights and Learnings

2023-07-27 13:58:13 +0530T00:00:00-00:00

An interview platform is a platform that offers a digital answer to the interview procedure. The conventional hiring procedure is frequently ineffective, time-consuming, and difficult. The platform was developed to address the issue by streamlining the hiring process and offering an effective means to carry it out.

Requirements

It was chosen to incorporate the following features in the application taking into account current interview circumstances, which are as follows:

Multi-User Conferencing
The ability to communicate via video and voice,
Chat feature
Screen-sharing Capabilities

Exploring the Application’s Fundamentals

To build a solution incorporating the above features it is necessary to have a clear understanding of the following concepts.

WebRTC (Web Real-Time Communication): It is a free and open-source project that provides web browsers and mobile applications with real-time communication (RTC) via application programming interfaces (APIs). It allows audio and video communication to work inside web pages by allowing direct peer-to-peer communication.
Socket IO: It is a JavaScript library that enables real-time, bidirectional communication between the server and clients. By leveraging WebSockets, it establishes persistent connections, allowing instantaneous data exchange.
Websocket: It is a protocol that provides a full-duplex communications channel over a network connection. WebRTC is standardised on WebSocket as the way to send information from a web browser to the signalling server and vice versa.
Adapter JS: It takes care of the differences between browsers, so developers don’t have to worry about compatibility issues and can focus on building their applications.
Signaling: The signalling process involves exchanging messages between two peers using an intermediary, the signalling server. WebRTC does not define a signalling protocol.
RTCPeerConnection: RTCPeerConnection is a web API in the JavaScript language used for enabling real-time communication (RTC) between web browsers. It is a fundamental part of the Web Real-Time Communication (WebRTC) technology
Media Servers: A media server is a device or software that stores digital media such as video, audio, or images and makes it available over a network.
SDP (Session Description Protocol): A protocol for describing media communication sessions is used. It is used for peer-to-peer negotiation of different audio and video codecs, network topologies, and other device characteristics but does not deliver the media data.

Now let us understand briefly what signalling taking an example:

User A creates an offer that contains its local SDP.
User A attaches that offer to something known as an RTCPeerConnection object.
User A sends its offer to the signalling server using WebSocket.
User B receives User A’s offer using WebSocket.
User B creates an answer containing her local SDP.
User B attaches its answer, along with User A’s offer, to its RTCPeerConnection object.
User B returns its answer to the signalling server using WebSocket.
User A receives User B’s offer using WebSocket.

You may have a basic understanding of the technology used to create the solution up to this point, so let’s take a quick look at the approach used to develop the solution.

Approach to Build the Solution

The first milestone was accomplished achieved after finishing room functionality, which essentially means that a unique meeting can be made and participants can enter the room with a unique id.
To make the application more purpose-specific, admin features and a chat component were included. With this feature, the interviewer has more control over the meeting, and participants can only participate after being accepted by the admin.
It was important to understand how to use the AWS platform and how to incorporate networking concepts to host the application.

Architecture to be used:

Different Architecture that can be used to enable Multiple User Video Calling Apps:

Mesh Topology
SFU Topology
MCU Topology

Comparing the Mesh & SFU Topology to be used:

In the Mesh architecture a user when joining the room, needs to establish a connection with every other present in the room.
And in turn the guests present needs to establish a connection with the new user.
Mesh architecture can be suitably implemented in a group of 3-4 people.
The advantage is that it is less expensive and less complex.
The disadvantage lies in the extent of scalability.
SFU topology has an edge over the Mesh taking into Scalability as a factor.
The bandwidth decreases taking the number of participants to be constant in both cases.
The one disadvantage is that it is more complex and relatively difficult to implement.

Topology to be followed:

Selective forwarding unit

In this various clients connect to the media server.
Here P2P is taken into use.
Each device is connected to the server.
The server combines them in the stream and puts them in a single stream.

Comparision between Mediasoup and Kurento

Until now we have a fair idea of which topology we need to implement in the application. We need a media server to get implemented for Selective Forwarding Unit. There are many options available in media servers that we can proceed but taking into multiple factors we are left with two options and we need to proceed with one.

Media connections are established 80% quicker than Kurento
Provides rich scalability and performance with its robust selective forwarding unit
Can be used as a NodeJS package or a Rust library
MediaSoup is designed to work in a distributed environment, making it suitable for large-scale deployments. It can handle multiple rooms and numerous concurrent calls, which aligns with your scalability requirement
MediaSoup has good community support and regular updates over time
MediaSoup allows you to write automated test cases for all the features, helping ensure the stability and functionality of your application.

Insights of the application

We currently have a good understanding of the architecture that will be used in the application. According to the demands of the application, SFUs (single forwarding units) will be deployed. And we conclude that we will keep using Mediasoup as the application’s media server.

Talking about the features to be implemented, which include audio and video chat and screen sharing, we need to have a better understanding.

The application requests authorization to use the available camera and microphone to establish communication and transfer media, such as video and audio.

navigator.mediaDevices 
	.getDisplayMedia({ video: true, }) 
	.then(streamSuccess) 
	.catch((err) => { console.log(err); });

The ability to share the screen must be implemented in addition to audio and video. To do this, we can use the code snippet below.

navigator.mediaDevices 
	.getDisplayMedia({ video: true, }) 
	.then(streamSuccess) 
	.catch((err) => { console.log(err); });

As fetching the streams we can proceed with the following steps of implementing the mediasoup architecture. Here is a brief overview of the steps implemented.

Create a media device (mediasoup-client) for capturing media.
Create a transport for sending media to the server.
Connect the send transport and produce audio/video tracks.
Signal a new consumer transport for a remote producer.
Get the list of available producers from the server.
Connect the receiving transport and consume remote media.
Render the local video, screen sharing video, and other controls.
Implement event handlers for muting, camera, and screen sharing toggles.
Implement event handlers for leaving the meeting and accepting new users.

Building the Solution

React Js is used to develop the application’s front end. Redux is utilised for state management, and after establishing the page routes, work on UI design began.
After reading the documentation, we began implementing Mediasoup in the backend and preferred Node Js as the backend framework.

This is an overview of the Mediasoup WebRTC server application. The application is built using Node.js, Express, MongoDB, and Socket.IO to facilitate real-time communication and media streaming.

Dependencies

dotenv: Loads environment variables from a .env file.
express: Web framework for building the server.
httpolyglot: Provides HTTPS server functionality for secure communication.
socket.io: Enables WebSocket communication for real-time events.
mediasoup: A WebRTC media server library for media processing.

Server Setup

Import required modules and set up the Express server.
Create an HTTPS server using httpolyglot for secure communication with SSL certificates.
Connect to MongoDB using Mongoose to store user information and other data.
Define the MongoDB schema for the “users” collection.

Mediasoup Integration

Create a Mediasoup worker to manage media processing.
Define media codecs for audio and video.
Set up Socket IO to handle WebSocket communication for real-time media streams.
Create maps and arrays to manage Mediasoup peers, transports, producers, and consumers.

Socket IO Event Handlers

Implement event handlers for various actions:
- Joining a room and creating WebRTC transports.
- Joining a room and creating WebRTC transports.
- Producing and consuming media.
- Sending and receiving messages between peers.
- Confirming admin status and accepting user requests.

Mediasoup Transports

Implement functions to create Mediasoup WebRTC transports with specific options.
Handle transport events like DTLS state changes and transport closure.

Final Setup

Create the Express app and set it to listen on the specified port (default: 3002).

Deployment

Dockerize the application

FROM node:20 
WORKDIR /app 
COPY package*.json ./ 
RUN npm cache clean --force 
COPY . . 
EXPOSE 8000 
CMD [ "npm", "start" ]

Initialize an EC2 Instance on AWS
Run the docker container with Ngnix WebServer
Expose TCP UDP Ports for EC2 Instance for media transmission.

Challenges Faced

During building the product the major problem I faced was to establish a connection between the clients using the application. Every Client who was joining was producing media but the media was not transporting to other clients. After debugging and revamping the application’s state, the conclusion was drawn that the React state was not behaving as decided and solved the issue after fixing it. During development, the socket instance needs to be properly handled so that it gets mapped to the proper room and doesn’t get broadcasted to every other instance.

During deployment, the major issue was assigning the proper IP address to be used in the application as the application has the requirement of broadcasting the IP address to every applicant with a public IP. After using the Amazon EC2 instance the issue was solved and then I implemented Docker to containerize the application and run it with the nginx server.

Conclusion

The first version of the application is built with the multi-user functionality joining with their desired choice of media.
The admin can accept the desired user and on accepting the user can enter the room.
The participants can communicate over audio, see each other’s video and chat in the room. The entire process of building the application is a great chance of learning and rewarding experience and future improvements can be done to make the application more consistent and reliable.

References

Comparision of Kurento and Mediasoup Mediaserver

Interview-Platform:Insights and Learnings was originally published by eLitmus.com at eLitmus Blog on July 27, 2023.

Building a Frontend Scoring Engine: Automating Frontend Evaluation

2023-10-05 20:17:46 +0530T00:00:00-00:00

The frontend scoring engine is a powerful tool designed to assess the frontend skills of candidates based on code quality, responsiveness, and functionality. It aims to streamline the evaluation process for frontend development by automating the assessment of code quality, best practices, and functionality.

What you’ll learn from this blog

In this blog, we will dive into the technical aspects of building a frontend scoring engine.

The need for frontend scoring engine in today’s technology landscape.
The technical requirements gathering and Research phase involved.
Generation of Test script for Test automation using Puppeteer.
Dockerizing the Application.
Features and Process of building the application.

Need for the Frontend Scoring Engine

In today’s technology-driven world, the demand for skilled frontend developers is at an all-time high. With the rapid evolution of web applications and user interfaces, companies are constantly seeking talented individuals who can create visually appealing, intuitive, and responsive frontend experiences. However, evaluating frontend development skills can be a complex and time-consuming task. This is where a frontend scoring engine comes into play Automating the Evaluation Process, Measurement of Code Quality and Ensuring Mobile Responsiveness. By allowing users to input HTML, CSS and JavaScript code, and generating scores based on predefined test cases, the scoring engine provides a comprehensive evaluation of candidates’ frontend skills.

Research Work

Before starting the implementation of the frontend scoring engine project, extensive research was conducted to understand the need for such a system, evaluate existing systems, explore testing tools, and plan the evaluation process. This research phase played a crucial role in shaping the project and ensuring its successful execution. Let’s take a brief look on highlight and the key areas of research conducted during the project’s inception.

Evaluating Existing Systems : To gain insights into the existing solutions available in the market, a comprehensive evaluation of similar systems was conducted. Various frontend scoring engines, online code editors were explored to understand their features, functionalities, strengths, and weaknesses. This evaluation provided valuable insights that influenced the design decisions and feature set of the new scoring engine.
Some similar existing systems:
Testing Tools and Technologies : During our research, we explored various testing tools and technologies to find the perfect fit for executing test cases, assessing code quality, and evaluating frontend functionalities. The evaluation revolved around factors like capabilities, ease of use, and compatibility with our project requirements. Tools such as Selenium, Cypress, Jest, csslint, eslint were taken into consideration.
Read more about the tools:
- Selenium
- Cypress
- Jest
Puppeteer : Puppeteer was chosen over Selenium primarily due to its compatibility with Docker and its ability to control headless Chrome or Chromium instances. Docker provides an efficient and scalable environment for running tests, and Puppeteer seamlessly integrates with Docker containers. Additionally, Puppeteer offers a more modern and concise API, making it easier to write test scripts and perform browser automation tasks.
- Puppeteer vs Selenium
- Puppeteer Docs
Docker Integration : We explored the benefits of Docker, a widely-used containerization platform, and discovered how it could greatly enhance our project. Docker allows us to create lightweight, portable, and isolated containers, which provide a consistent and reproducible environment. Leveraging Docker, we encapsulated and ran our scoring engine, testing tools, and other dependencies, ensuring seamless integration and efficient execution.
We pulled various Docker images from Docker Hub, enabling us to set up the required tools effortlessly.
- csslint
- eslint
- jest
Real-Time Code Editor : To provide a user-friendly and real-time code editing experience, we started searching for frontend code editors and existing projects available on GitHub. Various code editor projects were evaluated, and their source code were studied to understand the implementation details. This research helped in selecting the most suitable code editor framework and implementing it within our frontend scoring engine.
Problem Statement and Test Case Creation : The goal was to design problem statements that accurately reflect real-world frontend development challenges and create test cases that thoroughly evaluate candidates’ code. Puppeteer test scripts were written to simulate user interactions, perform assertions, and capture screenshots for image comparison using the PixelMatch JavaScript library.
Cloud Deployment and Infrastructure : For our final Deployment and integration Amazon Web Services (AWS) was choosen. The research covered various AWS services, including EC2 instances for hosting the scoring engine, S3 for storage, and other relevant services for infrastructure setup. The deployment process, security considerations, and scaling options were thoroughly explored to ensure a robust and scalable deployment architecture.

Test Script Generation

In the frontend scoring engine, we ensure evaluation of user-submitted HTML, CSS, and JavaScript code by subjecting it to comprehensive testing against predefined test cases. These tests are designed to assess the code quality, functionality, and adherence to best practices, providing a total assessment of candidates’ frontend development skills. By conducting these thorough evaluations, we can accurately determine the proficiency of developers in creating efficient and reliable frontend solutions. Throughout this section, you’ll get an overview of the various types of tests performed, explaining their significance in evaluating code quality and functionality.

Heading/Element Testing This test focuses on ensuring the presence and correctness of specific HTML elements within the user’s code. Test cases are designed to check if required headings, such as h1, h2, p or specific elements identified by ID or class, are present. The purpose of this test is to assess the structure and semantic correctness of the user’s HTML code.
CSS Properties Testing This test aims to verify the correct usage of CSS properties in the user’s code. It includes checking for the presence of essential CSS properties, such as margin, padding, font-size, or specific properties required for a particular problem statement. This test ensures that the user’s code adheres to the defined CSS requirements and best practices.
Form Validation Testing Form validation testing focuses on assessing the user’s code for proper form validation techniques. Test cases can include checking for required fields, validating email formats, enforcing password complexity, or implementing custom validation logic. This test ensures that the user’s code handles form validation correctly and provides appropriate error messages.
Function Testing This test evaluates the functionality and correctness of JavaScript functions implemented by the user. Test cases are designed to cover different scenarios and edge cases to ensure that the functions perform as expected. This test assesses the user’s ability to write functional and efficient JavaScript code.
API Testing API testing involves verifying the integration of API calls in the user’s code. Test cases may include checking if an API request is made, handling the API response correctly, and displaying the data from the API on the page. This test ensures that the user’s code effectively interacts with external APIs.
Button Testing Button testing focuses on evaluating the behavior and interactivity of buttons implemented by the user. Test cases may include checking if a button triggers a specific action, updates the UI, or performs a navigation action. This test ensures the proper functionality of user-defined buttons.
Redirection Testing This test aims to assess the behavior of navigation and redirection implemented by the user’s code. Test cases may include checking if clicking a link or a button redirects the user to the correct page or if the page refreshes as intended. This test ensures that the user’s code correctly handles navigation and redirection scenarios.

Dockerizing the Puppeteer with Chrome Browser Support

Dockerfile:

# Use the node:slim base image

FROM node:slim

# Set an environment variable to skip Puppeteer Chromium download during installation

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
RUN apt-get update && apt-get install gnupg wget -y && \
 wget --quiet --output-document=- https://dl-ssl.google.com/linux/linux_signing_key.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/google-archive.gpg && \
 sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' && \
 apt-get update && \
 apt-get install google-chrome-stable -y --no-install-recommends && \
 rm -rf /var/lib/apt/lists/\*

# Set the working directory inside the container

WORKDIR /usr/src/app

# Copy the package.json file to the working directory

COPY package.json ./

# Install project dependencies using npm

RUN npm install

# Expose port 3000 to allow access to the app outside the container

EXPOSE 3000

# Run the app using the "npm test" command when the container starts

CMD ["npm", "test"]

Build command:

docker build -t bhushan21z/puppchrome .

Publish it to Docker Hub:

docker push bhushan21z/puppchrome:tagname

Pull commnd:

docker pull bhushan21z/puppchrome

Run command:

docker run -it --rm -v $(pwd)/files:usr/src/app/files puppeteerchrome

Features and Architecture

Scoring Engine:

Inputs: The scoring engine takes HTML, CSS, and JavaScript files created by users on the client side, as well as the test cases file generated on the backend.
Code Quality Assessment: The engine assesses code quality using ESLint CSSlint and similar tools.
Scoring: The engine generates a score based on code quality, along with the results of the test cases executed on the client-side code.
Modular Architecture: The scoring engine is a separate entity, independent of the frontend and backend code.
Technology Stack: Python Flask framework is used to implement the scoring engine.
Working: Flask runs various Docker run commands to execute test script.

Backend:

MySql Database: Schema Created with various tables such as users, questions, testcases and submissions.
Node JS: Express framework is used to implement Rest APIs.
User auth: Contains user register and login APIs.
Questions: Questions create/get APIs.
Test Cases: Testcases create/get APIs and joining it with Questions table with question id as foreign key.
Scoring Engine: POST request to get user data and sending it to scoring engine and returning scoring engine response to frontend.
Submissions: User Submissions create/get APIs and joing it with users table and questions table.

Frontend (Admin Side):

Problem Creation: Admins can create problem statements, describing the problem to be solved.
Problem Settings: Problems can include various settings such as score weightage, best practices to check, and mobile responsiveness evaluation.
Test Cases: Admins can add multiple test cases related to each problem statement.
Test Case Visibility: Some test case outputs will be visible to users, while others will be hidden, showing only whether the score passed or failed.
User-Friendly Test Case Creation: Adding test cases are straightforward, even for users with limited programming knowledge.

Frontend (Client Side):

Problem List: Users can view a list of problems on their screen.
Code Editor: Users can write HTML, CSS, and JavaScript code for each problem, similar to the CodePen editor.
Code Compilation: Users can compile their code and generate the output.
Score Display: Users can view the scores generated by the scoring engine based on the performed test cases.

Tools & Technologies

Frontend:

ReactJS is used develop the frontend of the scoring engine.

Backend:

Node.js is employed for building the backend of the scoring engine.
MySQL is used as the database management system.

Scoring Engine

Puppeteer is used for implementing testcases and browser testing.
Docker containers are utilized for testing code quality and running test cases.
Flask is used to make scoring engine server which takes data and interacts with docker.

Conclusion

By implementing a frontend scoring engine, we can automate frontend development evaluation, resulting in a streamlined and efficient assessment process. This blog has explored the goals, research, features, technical requirements, and tools and technologies involved in developing a frontend scoring engine. The automation of code assessment, real-time editing, and integration of testing tools have resulted in an efficient and comprehensive evaluation platform. The challenges we faced during development have strengthened our understanding of frontend development and inspired innovative solutions. As we move forward, we remain committed to enhancing the scoring engine to meet the evolving needs of the tech industry.
If you have any questions, doubts or suggestions feel free to reach out to me on LinkedIn

Building a Frontend Scoring Engine: Automating Frontend Evaluation was originally published by eLitmus.com at eLitmus Blog on July 21, 2023.

Revamping eLitmus.com | Stand-Alone Front-end Module

2023-07-20 04:10:15 +0530T00:00:00-00:00

The current elitmus.com is a web application built with Ruby on Rails Framework, and the views are sent directly from the backend server whenever requested. This was quite good before, but in present scenario of internet and web technologies, these seem to lack some very basic requirements. And Hence, an upgradation is required.

Formally, current elitmus.com has a monolithic structure i.e. the front-end and the back-end are tightly coupled together. As a result of this, it is not possible to divide the project’s logic and team for front-end and back-end. Only Full Stack Developers having knowledge of both the domains are required in order to work in this project. This somehow limits the people who are more expertised in one of the domains.

Also, the present elitmus.com is not using the latest web technologies available. This greatly impacts the user experience.

So, What’s the solution for this ?

Well, we can separate the front-end and back-end. This will solve all the problems faced by the developers who work or tends to work in this project. This solves some of the major issues faced today by developers.

Now, we can have a distributed system, with the views ( front-end ) in one place and the Models and Controllers in the other. The Front-end we plan to build can be built using the latest and efficient web technologies currently available. This helps to improve the User Experience as well.

What Benefits ?

Developer Experience
- Team Separation → We can have Dedicated teams for front-end and back-end, each expertised in their own domains
- Logic Separation → We can separate the Logic of course for the frontend and backend
- Easy to Manage
- Easy to Scale
User Experience
- Latest Web Tech like React can be used to Build Views
- Improved Speed
- Improved Performance
- Consistency in design

How do we do it?

Well, now that we know what we have to do. We are halfway there already ( Just Kidding ). Let’s discuss some of the things we can use to make the front-end efficient and reliable.

React JS
- Its component architecture , helps us building a consistent design across the site.
- It’s fast and performant.
Tailwind CSS
- This is a light-weight CSS framework which is highly reliable and easy to use.
- This has a good community, which can help to borrow UI components rather than making it from scratch.
Redux Toolkit
- Redux Toolkit is a light version of Redux, which extracts away a lot of boilerplate codes and provides us easy to use APIs to manage state.
Jest
- Jest is the most popular library for writing tests in a react application. Infact, Create-React-App provides support for this out of the box when we initiate a new react project.

So, that’s all the core technologies we can use to build an efficient and reliable front-end. But, here is the catch: we can even improve more by following certain practices, which will be fruitful in the long run.

What else can we Improve ?

Following are some of the best practices that we can use to further improve the frontend application.

ES Lint
- Enforcing a code style guide is important to maintain the source code of the application. It helps to maintain consistency across the application.
- More Particularly, we can use the AirBNB Style Guide. This is the most popular style guide for React Application.
- We can add rules as per our need and requirements in the .eslintrc.js
Nested Routes
- This is one of the features of react. We can nest the routes under other routes to maintain a route intuition.

<Route path="/jobs" element={<JobsAndInterviews />}>
    <Route index element={<AllJobs />} />
    <Route path="my_jobs" element={<MyJobs />}>
      <Route index element={<ActiveJobs />} />
      <Route path="active" element={<ActiveJobs />} />
      <Route path="inactive" element={<InActiveJobs />} />
      <Route path="interviews" element={<Interviews />} />
    </Route>
    <Route path="all_jobs" element={<AllJobs />} />
  </Route>

Like in this example snippet, we have a parent route for job and under that my_jobs and inside that we have active, inactive, interviews.

/jobs/my_jobs/active → this route path is really gives a lot of information of the pages.
Dynamic Routes
- This is another feature of React Itself. This allows us to only load the pages that are requested by the user and not all.
- Just imagine, our site has hundreds of pages. When the user wants to visit the homepage, we are trying to send him all the hundred pages. This doesn’t make any sense right ?

// Jobs Page Routes
  export const JobsAndInterviews = lazy(() => import('../pages/Jobs'));
  export const AllJobs = lazy(() => import('../pages/Jobs/AllJobs'));
  export const ApplyJob = lazy(() => import('../pages/Jobs/ApplyJob'));
  export const JobDetails = lazy(() => import('../pages/Jobs/JobDetails'));
  export const MyJobs = lazy(() => import('../pages/Jobs/MyJobs'));
  export const ActiveJobs = lazy(() => import('../pages/Jobs/MyJobs/Active'));
  export const InActiveJobs = lazy(() => import('../pages/Jobs/MyJobs/Inactive'));
  export const Interviews = lazy(() => import('../pages/Jobs/MyJobs/Interviews'));

This above snippet shows how to import the components dynamically. But for this thing to work, we need to wrap the Routes in a Suspense Component which takes fallback. The component given inside the fallback is rendered in between the dynamic loads. So, we can put our page loader here. Below is the snippet showing how to do it.

import jobsRoutes from './routes/jobsRoutes';
  const App = () => (
  <Router>
    <Provider store={store}>
      <Layout>
        <Suspense fallback={() => <Loader />}>
          <Routes>
            {jobsRoutes}
          </Routes>
        </Suspense>
      </Layout>
    </Provider>
  </Router>
  );

Now, this makes the website immensely faster than before.

Intuitive File and Folder Organization -> Organizing the files and folders properly is a very important task because it significantly helps the new developers. It lowers the learning curve for the new fellas.

/src
    /__tests__
      /categoryA
        /page1.test.js
        /page2.test.js
      /categoryB
        /page1.test.js
        /page2.test.js
    /assets
    /components
      /customElements
      /Layout
    /features
      /redux_slices.js
    /pages
      /categoryA
        /page1.jsx
        /page2.jsx
      /categoryB
        /page1.jsx
        /page2.jsx
    /routes
    /store
      /redux_store.js
    /styles

That’s how we can improve our codebase even more.

Then, we have to make sure if our application runs the same on every device, OS, and system specs. For that we can dockerize the react app.

Dockerization and Deployment

Dockerizing the react app gives us the following benefits:

Consistency: Docker ensures the app runs consistently across different environments.
Dependency Management: Docker encapsulates app dependencies, preventing conflicts.
Easy Deployment: Docker simplifies deployment to various environments.
Scalability: Docker facilitates easy scaling to handle increased traffic.
Versioning and Rollbacks: Docker images can be versioned, enabling controlled updates and rollbacks.
Development and Testing: Docker streamlines development and testing in a consistent environment.
Infrastructure Agnostic: Docker allows running the app on various infrastructures.
Resource Efficiency: Docker containers are lightweight and efficient in resource utilization.
Easy Collaboration: Docker promotes seamless collaboration among developers and teams.
Security: Docker provides isolation, adding an extra layer of security to the app.

We can dockerize the react app by adding docker files i.e.

Dockerfile → contains environment and installation instructions for the app.

FROM node:18 as builder
  WORKDIR /app
  COPY package.json .
  RUN npm install
  COPY . .
  RUN npm run build
  FROM nginx
  EXPOSE 80
  COPY --from=builder /app/build /usr/share/nginx/html

docker-compose.yml → contain commands to run our docker container.

version: '3'
  services:
    web:
      build:
        context: .
        dockerfile: Dockerfile
      ports:
        - '80:80'

Now, we have successfully containerized our react application. Finally, we need to deploy it to some cloud services such as AWS.

We can first push our docker image to docker hub

docker push iamsmruti/elitmus-frontend

Then we can login to EC2 instance and then pull the docker image

docker pull iamsmruti/elitmus-frontend

Finally, we can run the docker image

docker run -d -p 5000:5000 iamsmruti/elitmus-frontend

That wraps up our frontend application which can now be live. It is fully capable of consuming the APIs from the backend. Now, the business logic is in the backend and doesn’t put much load on the frontend and hence it is performant and reliable.

If you have any questions, doubts, you can ping me at smrutiranjanbadatya2@gmail.com.

I would definitely get back to you.

I Hope this was a helpful and insightful guide for making a better frontend application with all the necessary good practices to maintain sustainability of the project.

See Ya 👋🏻 … Peace ✌🏻

References

React Docs - Here
Tailwind Docs - Here
Redux Toolkit Docs - Here
Jest Docs - Here
ES Lint Docs - Here
Docker Docs - Here

Revamping eLitmus.com | Stand-Alone Front-end Module was originally published by eLitmus.com at eLitmus Blog on July 20, 2023.

My Experience as a Summer Intern at eLitmus: Building a Telegram Bot

2023-07-14 15:18:53 +0530T00:00:00-00:00

As a summer intern at eLitmus, I had the opportunity to work on an exciting project that involved building a Telegram Bot. In today’s digital era, effective communication channels play a crucial role in connecting businesses with their stakeholders. eLitmus, a talent-tech platform, identified the need for a two-way communication channel between the platform and candidates. To achieve this, Telegram bots were chosen as the ideal starting point. This blog post will delve into the Telegram Bot Integration project.

How it Began:

The project started with the idea of leveraging the Telegram platform as a communication channel between eLitmus and its candidates. The goal was to create a two-way communication channel, enabling candidates to access information, receive updates, and engage in various activities through Telegram bots. This opened up possibilities for automating communication, collecting data, running quizzes, and providing valuable services to candidates.

Design:

Before diving into development phase, thorough planning and design are crucial. I begin by defining the core functionalities of the Telegram bots. I discovered that creating a bot through Bot Father (Telegram’s official bot) was the standard approach. As I was tasked with implementing the project using Ruby on Rails, I focused on two key aspects: developing the Telegram bot and designing the Admin panel. Designing such an application involves three key aspects: architecture design, database design, and UI/UX design. Let’s dive into each of these parts in more detail:

Architecture : The Telegram bots interact with users through messages and commands. Users can access FAQs, participate in quizzes, and receive responses based on their interactions with the bots. The bots handle user inputs, validate quiz answers, and provide feedback and results accordingly. An intuitive admin panel is developed using Ruby on Rails to facilitate easy management of the bot’s functionalities. The admin panel allows administrators to add, update, and delete FAQs, quizzes, and other content. It also provides insights and analytics related to user engagement and bot usage.
Database: The project utilizes a MySQL database to store and manage data related to users, FAQs, quizzes, quiz attempts, analytics, and other relevant information. The database schema is designed to efficiently store and retrieve data, ensuring optimal performance.
UI/UX: To ensure a visually appealing and user-friendly Telegram bot interface, I delved into various UI options and explored the best ways to present information and interact with users. This research helped me identify the most effective strategies for creating an engaging and intuitive bot interface. And for the Admin panel, I took the initiative to design the entire interface using Figma. By visualizing the layout, components, and functionalities, I was able to ensure a cohesive and user-friendly experience for administrators managing the bot’s functionalities. Figma provided a powerful toolset for creating wireframes, mock-ups, and interactive prototypes, allowing me to iterate and refine the design before implementation.

Development:

Before starting this project, I had experience developing mobile applications, and most of them followed the Model-View-Template (MVT) pattern for backend, such as Django. However, for this project, I needed to learn and work with Ruby on Rails, which follows the Model-View-Controller (MVC) architectural pattern. Fortunately, my previous experience with backend development made it easier for me to understand Rails, and within the first two weeks, I was able to develop the basic functionalities of both the FAQ and Quiz bots.

Integrating the telegram bot consists of 3 steps:

Creating a bot using Bot Father ( Official bot of telegram for creating telegram bot) and get the token that was generated by the bot father.
Initalizing the bot in the ruby file and declare a listening function that listens every messsage from the bot.
Writing the message specified functions that is called only when a specified message if recieved from the bot.

I have used Ruby on Rails for both front-end and back-end to develope admin panel. For database I have used mysql and for hosting purpose I have used AWS, EC2 to host admin panel using docker and telegram and RDS for database.

Using docker to host the bot and admin panel was another part of the development that gave me an idea of how to does docker used by most of the companies, it was my personal goal in the year to learn docker so it got done by this project. And to say using docker wasn’t the difficult part. I had to learn how to write a docker file and docker compose file.

Features Developed

User Flow

I focused on refining the functionalities and user flow of the bots, particularly in the context of the Telegram channels. The FAQ bot is connected to the Telegram channel, and when a user posts a question in the channel’s comment section, it gets stored in the database. The admin can then view and answer the question, which is sent back to the user personally through Telegram. Additionally, users can access the FAQ bot to view existing FAQs and request the addition of new ones.

FAQ bot flow

Quiz bot flow

Admin Panel

On the other hand, the admin panel allows the admin to create quizzes and questions. These quizzes are then posted in the Telegram channel, with a button redirecting users to the Quiz bot. Users can access multiple quizzes and attempt them through the bot.

By developing these functionalities, I was able to establish a seamless flow for users, ensuring they can interact with the bots and access relevant information easily. The admin panel provides the necessary tools for managing FAQs, quizzes, and user interactions, allowing for efficient administration and engagement with the users.

In the Admin panel, I implemented the design that I had previously created using Figma. The Admin panel offers various functionalities to enhance the administration and management of the Telegram bots. Here are some key features of the Admin panel:
- User Management: The Admin panel allows the admin to view active users and access individual user data. This includes information about the user’s activities, quiz attempts, and questions asked through the bot.
- FAQ Management: The Admin can view and manage the FAQs. They have the ability to add, edit, or remove FAQs as needed. Additionally, the Admin can track the number of reads by users, providing insights into the popularity and relevance of different FAQs.
- Quiz Management: The Admin can create quizzes and manage them within the Admin panel. They can add questions, set multiple options, and define correct answers. The Admin also has access to the responses of the quizzes, allowing them to analyze individual question analytics and gain insights into user performance. This can also be used to host surveys on telegram channels.
- Analytics: The Admin panel provides analytics on user activities related to both the FAQ and Quiz bots. The Admin can view data such as the number of attempts per day, week, month, or year, as well as the number of FAQ reads per day, week, month, or year. These analytics help the Admin understand user engagement and make data-driven decisions.
- Post Management: The Admin can utilize the post section in the Admin panel to create and publish posts in the Telegram channel directly from Telegram and to make it effective I have created two phases of create and publishing the post so that post get reviewed before publishing the post. This feature streamlines the process of sharing content and updates with users in the channel.
Admin Panel

By incorporating these functionalities into the Admin panel, I ensured that the administrative tasks associated with managing the Telegram bots were streamlined and efficient. The panel provides comprehensive control and insights, empowering the admin to effectively manage user interactions, content, and analytics.

Challenges Faced:

Working with 3rd party API’s is one of the most challenging task and that is the challenging task of the project using telegram bot API. I could able to use telegram api to minimal amount of data of user, for example I couldn’t able to get users contact details, and I have crossed this challenge by finidng a feature of telegram that is by using permissions to access user details and request user to send the mobile number and location, but I couldn’t able to get location from the web or laptop. The biggest challenge I have faced was setting up and displaying analytics using charts and graphs. Initially, I tried using gems like Chartkick and FusionCharts, but faced issues with rendering the graphs correctly. Despite spending considerable time troubleshooting, the graphs weren’t displaying as expected. Eventually, I opted for Chart.js, which proved to be a more suitable solution for my needs. With Chart.js, I could create visually appealing and interactive charts to showcase the data collected through admin panel. The transition to Chart.js was smooth, and it enabled me to present data insights effectively, providing a valuable user experience.

Conclusion:

In summary, working on this project presented its fair share of challenges. However, with perseverance and problem-solving skills, I was able to overcome these obstacles and achieve success. I was able to develop the Telegram bots and the Admin panel effectively. I am thrilled to share that my hard work did not go unnoticed, and my project was selected for use by the company. This recognition is truly gratifying, as it demonstrates the value my work brings to the organization and the impact it can have on the company operations. Overall, this project was a rewarding journey that expanded my knowledge and skills in web development.

My Experience as a Summer Intern at eLitmus: Building a Telegram Bot was originally published by eLitmus.com at eLitmus Blog on July 19, 2023.

Resume Parsing: Insights and Steps to Create Your Own Parser

2023-06-20 13:40:00 +0530T00:00:00-00:00

Resume parsing is the automated process of extracting relevant information from resumes or CVs. It analyzes the unstructured text of a resume and extracts specific details like contact information, work experience, education, skills, and achievements. The extracted data is then converted into a structured format, allowing for easy analysis and integration into recruitment systems.

Benefits of Resume Parsing

It is a time-saving automation
It increases efficiency in candidate screening
Improves accuracy in data extraction
It standardizes the data extraction and formatting

What you’ll learn from this blog:

Resume parsing techniques for different file formats.
Extracting specific details from resumes.
Leveraging NLP techniques for parsing.
Handling multicolumn resumes.
Dockerizing the Application: Simplifying Deployment and Scalability
Hosting it on AWS EC2.

Let’s get Started 🎉

We’ll utilize Python and its Flask framework to create a resume parsing server.

Application Flow Chart:

We will be primarily working on 3 categories of file formats:

PDF
DOCX
Images (.png, .jpg, etc.)

Data that we will be extracting

Embedded links in PDF
Personal data:
2.1. Name: First name and last name
2.2. Email
2.3. Phone Number
2.4. Address: City, Country, and Zip code
2.5. Links: Social and Coding Platform links
Education
3.1. Institute name
3.2. Duration: Start date and End date
3.3. Grade/CGPA
3.4. Degree
Experience
4.1. Company name
4.2. Role
4.3. Durations: Start date and End date
4.4. Skills
Certification:
5.1. Description
5.2. Duration
5.3. Skill
Project:
6.1. Project name
6.2. Skills
6.3. Description
Skills
Achievements
Exam scores
9.1. Exam name
9.2 Score
All other sections present in resume

Date/Duration Extraction

To extract dates from text, we will use datefinder module, and regexp to extract years. Then we will combine these two and sort dates to get start and end date for our duration.

import re
from datetime import date
import datefinder


def get_date(input_string):
    '''Get date from text'''
    matches = list(datefinder.find_dates(input_string))

    res = []
    for i in matches:
        date_str = str(i).split(' ')
        extracted_date = date_str[0]

        res.append(extracted_date)
    return res


def get_years(txt):
    '''Get years from text'''
    pattern = r'[0-9]{4}'
    lst = re.findall(pattern, txt)

    current_date = date.today()
    current_year = current_date.year
    res = []
    for i in lst:
        year = int(i)
        if 1900 <= year <= (current_year + 10):
            res.append(i + "-01-01")
    return res


def get_duration(input_text):
    '''Get duration from text'''

    dates = get_date(input_text)
    years = get_years(input_text)

    for i in years:
        dates.append(i)
    dates.sort()

    duration = {
        "start_date": "",
        "end_date": ""
    }
    if len(dates) > 1:
        duration["start_date"] = dates[0]
        duration["end_date"] = dates[len(dates) - 1]
    return duration

Extracting links from PDF:

To extract links from the PDF, we will use the python module PDFx.

import pdfx

def get_urls_from_pdf(file_path):
    '''extract urls from pdf file'''
    url_list = []

    # for invalid file path
    if os.path.exists(file_path) is False:
        return url_list

    pdf = pdfx.PDFx(file_path)

    # get urls
    pdf_url_dict = pdf.get_references_as_dict()

    if "url" not in pdf_url_dict.keys():
        return url_list

    url_list = pdf_url_dict["url"]

    return url_list

PDF to Text

import pdfx
def get_text_from_pdf(file_path):
    '''extract complete text from pdf'''

    # for invalid file path
    if os.path.exists(file_path) is False:
        return ""

    pdf = pdfx.PDFx(file_path)

    pdf_text = pdf.get_text()

    return pdf_text

Extracting Personal Details:

We will extract text from the PDF and move ahead with further extractions.

Name

Extracting the name from the text is one of the challenging tasks.

For this, we will be using NLP: Named Entity Recognition to extract name from the text.

NLP function:

def get_name_via_nltk(input_text):
    '''extract name from text via nltk functions'''
    names = []
    for sent in nltk.sent_tokenize(input_text):
        for chunk in nltk.ne_chunk(nltk.pos_tag(nltk.word_tokenize(sent))):
            if hasattr(chunk, 'label'):
                name = ' '.join(c[0] for c in chunk.leaves())
                names.append(name)
    return names

The text is tokenized into sentences using nltk.sent_tokenize().
Each sentence is further tokenized into words using nltk.word_tokenize().
The part-of-speech tags are assigned to each word using nltk.pos_tag().
The named entities are identified by applying the named entity recognition (NER) using nltk.ne_chunk().
For each identified named entity chunk, if it has a ‘label’, indicating it is a named entity, the individual words are concatenated to form a name.
The extracted names are appended to the names list.

Phone Number

To extract the Phone number, we use the following module phonenumbers, we extract users country from text and using that we will extract relevant phone numbers.

import geotext
from phonenumbers import PhoneNumberMatcher

def get_phone(input_text):
    '''extract phone number from text'''

    phone_numbers = []

    countries_dict = geotext.GeoText(input_text).country_mentions
    
    country_code = "IN"
    for i in countries_dict.items():
        country_code = i[0]
        break

    search_result = PhoneNumberMatcher(input_text, country_code)

    phone_number_list = []
    for i in search_result:
        i = str(i).split(' ')
        match = i[2:]

        phone_number = ''.join(match)
        phone_number_list.append(phone_number)

    for i in phone_number_list:
        if i not in phone_numbers:
            phone_numbers.append(i)

    return phone_numbers

Email

To extract the Email, we use the following regexp: [^\s]+@[^\s]+[.][^\s]+

def get_email(input_text):
    '''extract email from text'''
    email_pattern = '[^\s]+@[^\s]+[.][^\s]+'

    emails = []
    emails = re.findall(email_pattern, input_text)

    # pick only unique emails
    emails = set(emails)
    emails = list(emails)

    return emails

Address

To Extract address, we use the geotext module; we get City, Country, and Zipcode.

import geotext
def get_address(input_arr):
    '''get address information from input array'''

    input_text = " \n ".join(input_arr)

    res = {}
    # getting all countries
    countries_dict = geotext.GeoText(input_text).country_mentions

    res["country"] = []
    for i in countries_dict:
        res["country"].append(i)

    # getting all cities
    res["city"] = geotext.GeoText(input_text).cities

    # zip code
    pattern = "\b([1-9]{1}[0-9]{5}|[1-9]{1}[0-9]{2}\\s[0-9]{3})\b"
    res["zipcode"] = re.findall(pattern, input_text)

    return res

Links

As we already have a URL list from 1st operation, we will match links from a list of our own, this can be saved in any database or hard-coded, and categorize them into social or coding sections.

Other Sections

There can be many sections in a resume, that we cannot always account for. To extract them, we will create a list of possible section heading and match them against each line from the resume that we have extracted.

The code will be as following:


from utils import dynamo_db

RESUME_SECTIONS = dynamo_db.get_item_db("RESUME_SECTIONS")


def extract_resume_sections(text):
    '''Extract section based on resume heading keywords'''
    text_split = [i.strip() for i in text.split('\n')]

    entities = {}
    entities["extra"] = []
    key = False
    for phrase in text_split:
        if len(phrase.split(' ')) > 10:
            if key is not False:
                entities[key].append(phrase)
            else:
                entities["extra"].append(phrase)
            continue

        if len(phrase) == 1:
            p_key = phrase
        else:
            p_key = set(phrase.lower().split()) & set(RESUME_SECTIONS)

        try:
            p_key = list(p_key)[0]
        except IndexError:
            pass

        if p_key in RESUME_SECTIONS and (p_key not in entities.keys()):
            entities[p_key] = []
            key = p_key
        elif key and phrase.strip():
            entities[key].append(phrase)
        else:
            if len(phrase.strip()) < 1:
                continue
            entities["extra"].append(phrase)

    return entities

Education

To extract education, we need to identify a line from our education section that represent the school/institute name, and a line that represents the degree. After which we can search for CGPA or Percentage using regexp. For name recognition, we will make use of a list of keywords that can be present in the name.

Code to get school name, similarly we can implement to get degree as well.

import re
from utils import helper, dynamo_db

SCHOOL_KEYWORDS = dynamo_db.get_item_db("SCHOOL_KEYWORDS")


def get_school_name(input_text):
    '''Extract list of school names from text'''
    text_split = [i.strip() for i in input_text.split('\n')]

    school_names = []

    for phrase in text_split:
        p_key = set(phrase.lower().split(' ')) & set(SCHOOL_KEYWORDS)

        if (len(p_key) == 0):
            continue

        school_names.append(phrase)
    return school_names

Code to extract CGPA/GPA or Percentage grade

def get_percentage(txt):
    '''Extract percentage from text'''
    pattern = r'((\d+\.)?\d+%)'
    lst = re.findall(pattern, txt)
    lst = [i[0] for i in lst]
    return lst


def get_gpa(txt):
    '''Extract cgpa or gpa from text in format x.x/x'''
    pattern = r'((\d+\.)?\d+\/\d+)'
    lst = re.findall(pattern, txt)
    lst = [i[0] for i in lst]
    return lst


def get_grades(input_text):
    '''Extract grades from text'''
    input_text = input_text.lower()
    # gpa
    gpa = get_gpa(input_text)

    if (len(gpa) != 0):
        return gpa

    # percentage
    percentage = get_percentage(input_text)

    if (len(percentage) != 0):
        return percentage

    return []

Skills

In order to extract skills from the text, a master list of commonly used skills can be created and stored in a database, such as AWS DynamoDB. Each skill from the list can be matched against the text to identify relevant skills. By doing so, a comprehensive master skill list can be generated, which can be utilized for more specific skill extraction in subsequent sections.


from utils import dynamo_db

skills = dynamo_db.get_item_db("ALL_SKILLS")


def get_skill_tags(input_text):
    '''Extract skill tags from text'''
    user_skills = []
    for skill in skills:
        if skill in input_text.lower():
            user_skills.append(skill.upper())

    return user_skills

Experience

To extract company names and roles, a similar strategy can be employed as we used for finding school names and degrees. By applying appropriate techniques, such as named entity recognition or pattern matching, we can identify company names and associated job roles from the text. Additionally, for skill extraction, we can match the text against our previously calculated list of skills to identify and extract relevant skills mentioned in the text

Achievements and Certifications

We can use the section text that we extracted previously and for each line of it, we can search for duration and skills in it.


from utils import helper, skill_tags


def get_certifications(input_array):
    '''Function to extract certificate information'''

    res = {
        "description": input_array,
        "details": []
    }

    try:

        for cert in input_array:
            elem_dict = {
                "institute_name": str(cert),
                "skills": skill_tags.get_skill_tags(cert),
                "duration": helper.get_duration(cert)
            }
            res["details"].append(elem_dict)

    except Exception as function_exception:
        helper.logger.error(function_exception)

    return res

Projects

When it comes to extracting project titles, it can be challenging due to the variations in how individuals choose to title their projects. However, we can make an assumption that project titles are often written in a larger font size compared to the rest of the text. Leveraging this assumption, we can analyze the font sizes of each line in the text and sort them in descending order. By selecting the lines with the largest font sizes from the top, we can identify potential project titles. This approach allows us to further segment the project section and extract additional details such as skills utilized and project durations.

Link: How to find the Font Size of every paragraph of PDF file using python code?

import fitz

def scrape(keyword, filePath):
    results = [] # list of tuples that store the information as (text, font size, font name) 
    pdf = fitz.open(filePath) # filePath is a string that contains the path to the pdf
    for page in pdf:
        dict = page.get_text("dict")
        blocks = dict["blocks"]
        for block in blocks:
            if "lines" in block.keys():
                spans = block['lines']
                for span in spans:
                    data = span['spans']
                    for lines in data:
                            results.append((lines['text'], lines['size'], lines['font']))

    pdf.close()
    return results

Using this we find our project titles:

from utils import helper, skill_tags
from difflib import SequenceMatcher

def similar(string_a, string_b):
    '''Find similarity between two string'''
    return SequenceMatcher(None, string_a, string_b).ratio()

def extract_project_titles(input_array, text_font_size):
    ls = []
    for line_tuple in text_font_size:
        line = line_tuple[0]
        for s in input_array:
            if similar(line,s) > 0.85:
                ls.append([line_tuple[1], s])
    ls.sort(reverse=True)

    title_font_size = ls[0][0] if(len(ls) > 0) else 0
    project_title = []
    for i in ls:
        if i[0] == title_font_size:
          project_title.append(i[1])
    return project_title

def get_projects(input_array, text_font_size):
    '''extract project details from text'''
    res = {
        "description": input_array,
        "details": []
    }
    txt = ' \n '.join(input_array)

    project_titles = helper.extract_titles_via_font_size(
        input_array, text_font_size)

    project_sections = helper.extract_sections(txt, project_titles)

    try:
        for i in project_sections.items():
            key = i[0]
            txt = '\n'.join(project_sections[key])

            elem_dict = {
                "project_name": key,
                "skills": skill_tags.get_skill_tags(txt),
                "duration": helper.get_duration(txt)
            }

            res["details"].append(elem_dict)
    except Exception as function_exception:
        helper.logger.error(function_exception)

    return res

Handling multicolumn resumes

Up until now, we have explored techniques to handle single-column resumes successfully. However, when it comes to two-column or multicolumn resumes, a direct extraction of text may not be sufficient. If we attempt to extract text from a multicolumn PDF using the same method as before, we will encounter challenges such as, the text from different columns will merge together, as our previous approach scans the text from left to right and top to bottom, rather than column-wise.

To overcome this issue, let’s delve into how we can solve this problem and effectively handle multicolumn resumes.

Drawing textboxes

Optical Character Recognition (OCR) comes to the rescue by identifying textboxes and providing their coordinates within the document. By utilizing OCR, we can pinpoint the location of these textboxes, which serve as a starting point for further analysis.

To tackle the challenge of multicolumn resumes, a line sweep algorithm is implemented. This algorithm systematically scans along the X-axis and determines how many textboxes intersect each point. By analyzing this distribution, potential column divide lines can be inferred. These lines act as reference markers, indicating the boundaries between columns.

Once the column lines are established, the text can be extracted from the identified textboxes in a column-wise manner. Following the order of the column lines, the text can be retrieved and processed accordingly.

By leveraging OCR, the line sweep algorithm, and the concept of column lines, we can effectively handle multicolumn resumes and extract the necessary information in an organized and structured manner.

Code:

import cv2
import fitz
from fitz import Document, Page, Rect
import pytesseract
import functools

def textbox_recognition(file_path):
    '''Extract text_boxes from image'''

    img = cv2.imread(file_path, cv2.IMREAD_GRAYSCALE)

    ret, thresh1 = cv2.threshold(
        img, 0, 255, cv2.THRESH_OTSU | cv2.THRESH_BINARY_INV)

    # kernel
    kernel_size = 10
    rect_kernel = cv2.getStructuringElement(
        cv2.MORPH_RECT, (kernel_size, kernel_size))

    # Applying dilation on the threshold image
    dilation = cv2.dilate(thresh1, rect_kernel, iterations=1)

    # Finding contours
    contours, hierarchy = cv2.findContours(
        dilation, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

    segments = []
    text_boxes = []
    # Looping through the identified contours
    for cnt in contours:
        x, y, w, h = cv2.boundingRect(cnt)
        cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 2)
        segments.append([x, x+w])
        text_boxes.append((x, y, w, h))

    return (segments, text_boxes)


def detect_column_lines(segments):
    '''Detect column lines from segments'''

    mx = max(i[1] for i in segments)

    line_sweep_arr = [0 for _ in range(mx+10)]
    for i in segments:
        line_sweep_arr[i[0] + 1] += 1
        line_sweep_arr[i[1]] -= 1

    for i in range(1, mx+10):
        line_sweep_arr[i] += line_sweep_arr[i-1]

    line_mean = sum(line_sweep_arr)/len(line_sweep_arr)

    potential_points = []
    for i in range(1, mx+10):
        if line_sweep_arr[i] < int(line_mean/2.5):
            potential_points.append(i)

    line_points = []
    for i in potential_points:
        if len(line_points) == 0:
            line_points.append(i)
            continue
        prev = line_points[len(line_points) - 1]

        if i == prev + 1:
            line_points[len(line_points) - 1] = i
        else:
            line_points.append(i)

    return line_points


def get_text(img, box_data):
    '''Extract text from given box data'''
    (x, y, w, h) = box_data
    cropped_image = img[y:y+h, x:x+w]

    # to show image
    txt = pytesseract.image_to_string(cropped_image)
    return txt


def box_coverage_percentage(x, w, line):
    '''Extract coverage area in percentage for box'''

    covered_width = line - x
    cover_percentage = covered_width / w
    return cover_percentage


def clean_text(txt):
    '''Clean text'''
    txt = txt.strip()
    txt = txt.replace("•", '')
    return txt


Y_LIMIT = 10


def custom_sort(a, b):
    '''custom sort logic'''
    if a[1] - Y_LIMIT <= b[1] >= a[1] + Y_LIMIT:
        return -1 if (a[0] <= b[0]) else 1
    return -1 if (a[1] <= b[1]) else 1


def get_boxes_for_line(text_boxes, line, ordered_text_box, prev_line):
    '''get boxes with line constraints'''
    temp_boxes = [i for i in text_boxes]
    temp_boxes.sort(key=functools.cmp_to_key(custom_sort))

    res = []

    # check if 90% of box is before line
    for box in temp_boxes:
        if box in ordered_text_box:
            continue

        (x, y, w, h) = box

        if (x >= prev_line - Y_LIMIT and x < line and box_coverage_percentage(x, w, line) >= 0.9):
            res.append(box)
    res.sort(key=lambda x: x[1])
    return res


def map_size(x, org, new):
    '''map box co-ordinates from image to pdf'''
    return (x*new)/org


def get_text_from_pdf(box, img_shape, pdf_shape, page):
    '''extract text from pdf box'''
    (x, y, w, h) = box
    (height, width) = img_shape
    (W, H) = pdf_shape
    x = map_size(x, width, W)
    w = map_size(w, width, W)
    y = map_size(y, height, H)
    h = map_size(h, height, W)
    rect = Rect(x, y, x+w, y+h)
    text = page.get_textbox(rect)
    return text


def image_to_text(file_path, pdf_file_path=""):
    '''extract text from image'''
    segments, text_boxes = textbox_recognition(file_path)
    column_lines = detect_column_lines(segments)

    # if single column
    if len(column_lines) < 3:
        return ""

    # align text boxes by column
    # text boxes within columns
    ordered_text_box = []
    for i in range(len(column_lines)):
        prev_line = column_lines[i-1] if ((i-1) >= 0) else 0
        boxes = get_boxes_for_line(
            text_boxes, column_lines[i], ordered_text_box, prev_line)
        for b in boxes:
            ordered_text_box.append(b)

    # boxes that are not in any column
    # text boxes not in any column
    non_selected_boxes = []
    for i in text_boxes:
        if i not in ordered_text_box:
            non_selected_boxes.append(i)

    for i in non_selected_boxes:
        y = i[1]
        if y <= ordered_text_box[0][1]:
            ordered_text_box.insert(0, i)
        else:
            ordered_text_box.append(i)

    img = cv2.imread(file_path, cv2.IMREAD_GRAYSCALE)
    ret, thresh = cv2.threshold(img, 225, 255, 0)
    img_shape = img.shape

    pdf_shape = (0, 0)
    page = None
    if pdf_file_path != "":
        doc = fitz.open(pdf_file_path)
        page = doc[0]
        pdf_shape = (page.rect.width, page.rect.height)

    resume_text = ""
    for i in ordered_text_box:
        if pdf_file_path != "":
            txt = clean_text(get_text_from_pdf(i, img_shape, pdf_shape, page))
        else:
            txt = clean_text(get_text(thresh, i))
        resume_text += txt + "\n"

    # clean text
    txt = resume_text.split("\n")

    res = []
    for line in txt:
        if len(line) == 0:
            continue
        res.append(line)

    resume_text = ' \n '.join(res)
    return resume_text

Dockerizing the Application

To make deploying the application easy we will be Dockerizing the Application.

Dockerfile

# syntax=docker/dockerfile:1

FROM python:3.9-buster

WORKDIR /resume-parser-docker

RUN mkdir input_files
RUN pip3 install --upgrade pip

COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

# download nltk required
RUN python -m nltk.downloader punkt
RUN python -m nltk.downloader averaged_perceptron_tagger
RUN python -m nltk.downloader maxent_ne_chunker
RUN python -m nltk.downloader words

RUN apt-get update \
  && apt-get -y install tesseract-ocr

RUN apt-get update && apt-get install ffmpeg libsm6 libxext6  -y

COPY . .

EXPOSE 5000/tcp

CMD [ "python3", "-u" , "main.py"]

Then run following commands to create image and run it.

Build Image

docker build --tag jhamadhav/resume-parser-docker .

Run Image at port 5000

docker run -d -p 5000:5000 jhamadhav/resume-parser-docker

Check images
```
docker ps
```

Stop once done

docker stop jhamadhav/resume-parser-docker

Hosting on AWS

Now that we have a docker image of our application.

We can publish it to dockerHub:

docker push jhamadhav/resume-parser-docker

Then login to your EC2 instance and pull the image:

docker pull jhamadhav/resume-parser-docker

Run the image:

docker run -d -p 5000:5000 jhamadhav/resume-parser-docker

🎉🎉🎉 We have a fully functional Resume parser ready.

Future Work

We can make use of Large Language Models (LLM), train on datasets and fine tune LLM model to make extraction of below fields more accurate:

School/Institute name
Degree
Company name
Role in a job

Conclusion

In conclusion, resume parsing using NLP techniques offers a streamlined approach to extract crucial information from resumes, enhancing the efficiency and accuracy of candidate screening.
By leveraging OCR, named entity recognition, and line sweep algorithms, we can handle various resume formats, including multicolumn layouts.
The power of NLP automates the parsing process, empowering recruiters to efficiently process resumes and make informed hiring decisions.
Embracing resume parsing techniques ensures fair and objective evaluation of applicants, leading to successful recruitment outcomes.
With this skillset, you can revolutionize resume processing and contribute to more efficient hiring practices.

If you have any questions, doubts, or just want to say hi, feel free to reach out to me at contact@jhamadhav.com ! I’m always ready to chat about this cool project and help you out. Don’t be shy, drop me a line and let’s geek out together!

Resume Parsing: Insights and Steps to Create Your Own Parser was originally published by eLitmus.com at eLitmus Blog on June 20, 2023.

Debugging & Fixing mysql deadlock issue

2023-06-12 18:31:00 +0530T00:00:00-00:00

Recently, during one of our tests, we encountered a deadlock issue that was reported by Sentry. The deadlock occurred while attempting to insert scores into a table after completing a candidate’s test. We were initially unsure about the cause of this deadlock. Upon investigation, we discovered that it was due to the interplay of various locks in our MySQL database. In this blog post, we will deep dive into the nature of these locks, understand their impact on transactions, and present the solutions we implemented to mitigate deadlock occurrences.

Understanding deadlocks

To understand the deadlock situation, let’s familiarize ourselves with the different types of locks involved, as defined by the official MySQL documentation:

GAP Lock:

A gap lock is a lock on a gap between index records, or a lock on the gap before the first or after the last index record. A gap might span a single index value, multiple index values, or even be empty.

If id is not indexed or has a nonunique index, the statement does lock the preceding gap.

Next Key Lock:

A next-key lock is a combination of a record lock on the index record and a gap lock on the gap before the index record. in simple words If one session has a shared or exclusive lock on record R in an index, another session cannot insert a new index record in the gap immediately before R in the index order.

Insert Intention Lock:

An insert intention lock is a type of gap lock set by INSERT operations prior to row insertion. This lock signals the intent to insert in such a way that multiple transactions inserting into the same index gap need not wait for each other if they are not inserting at the same position within the gap.

Problem Scenario

In our case, we have two tables, table1 and table2, with a has_many relationship. All operations are performed on table2, which has an index on table1 as a foreign key.

Transaction A

BEGIN;
DELETE FROM table2 WHERE table2.table1_id=127;
Query OK, 1 row affected (0.00 sec)

Resulting data locks

mysql> SELECT INDEX_NAME, LOCK_TYPE,LOCK_DATA,LOCK_MODE,LOCK_STATUS, EVENT_ID FROM performance_schema.data_locks;
+-----------------------------------------+-----------+-----------+---------------+-------------+----------+
| INDEX_NAME                | LOCK_TYPE | LOCK_DATA | LOCK_MODE     | LOCK_STATUS | EVENT_ID |
+-----------------------------------------+-----------+-----------+---------------+-------------+----------+
| NULL                      | TABLE     | NULL      | IX            | GRANTED     |      408 |
| index_table2_on_table1_id | RECORD    | 127, 92   | X             | GRANTED     |      408 |
| PRIMARY                   | RECORD    | 92        | X,REC_NOT_GAP | GRANTED     |      408 |
| index_table2_on_table1_id | RECORD    | 128, 93   | X,GAP         | GRANTED     |      408 |
+-----------------------------------------+-----------+-----------+---------------+-------------+----------+
4 rows in set (0.00 sec)

This query acquires a gap lock on table2 and an insert intention lock on table1_id values 126 and 127.

Transaction B

BEGIN;
INSERT INTO table2(table1_id) VALUES(126);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

Resulting data locks

mysql> SELECT INDEX_NAME,LOCK_TYPE,LOCK_DATA,LOCK_MODE,LOCK_STATUS, EVENT_ID FROM performance_schema.data_locks;
+-----------------------------------------+-----------+-----------+------------------------+-------------+----------+
| INDEX_NAME                  | LOCK_TYPE | LOCK_DATA | LOCK_MODE              | LOCK_STATUS | EVENT_ID |
+-----------------------------------------+-----------+-----------+------------------------+-------------+----------+
| NULL                        | TABLE     | NULL      | IX                     | GRANTED     |      351 |
| index_table2_on_table1_id   | RECORD    | 127, 92   | X,GAP,INSERT_INTENTION | WAITING     |      351 |
| NULL                        | TABLE     | NULL      | IX                     | GRANTED     |      408 |
| index_table2_on_table1_id   | RECORD    | 127, 92   | X                      | GRANTED     |      408 |
| PRIMARY                     | RECORD    | 92        | X,REC_NOT_GAP          | GRANTED     |      408 |
| index_table2_on_table1_id   | RECORD    | 128, 93   | X,GAP                  | GRANTED     |      408 |
+-----------------------------------------+-----------+-----------+------------------------+-------------+----------+
6 rows in set (0.01 sec)

As Transaction A holds the lock on table1_id 126 due to the gap lock, Transaction B waits for the lock. However, it eventually times out, resulting in a lock wait timeout error.

To create a deadlock, one must perform a delete query in Transaction B. Then, when attempting to insert a record in Transaction A, a deadlock error is thrown, with Transaction B becoming the victim. This deadlock situation arises due to the conflicts in the next-key lock, preventing Transaction B from inserting the record.

In a nutshell

Lets understood the above queries in nutshell to create a deadlock.

Transaction A -> BEGIN;
Transaction A -> DELETE records on table2 with table1_id=x.
Transaction B -> BEGIN;
Transaction B -> DELETE record on table2 with table1_id=y;
Transaction B -> INSERT a record on table2 and table1_id is x-1.
Transaction A -> INSERT a record on table2 and table1_id is y-1.
A deadlock occurs, with Transaction A being the victim.

Practical example of GAP lock & Next Key Lock.

Gap lock is basically on range of values & will be aquired on a range if we try to delete a record which does not exist.

table1

+----+
| id |
+----+
| 73 |
| 74 |
| 81 |
| 82 |
+----+

table2

+-----+-----------+
| id  | table1_id |
+-----+-----------+
| 1   | 73        | 
| 2   | 82        |
+-----+-----------+

Transaction A

BEGIN;
DELETE from table2 where table1_id=75;
Query OK, 0 rows affected (0.00 sec)

This transaction will aquire a gap lock on range from 74-80. this means if we try to insert new values in table2(in another session) with table1_id ranging from 74-80 it will wait until delete transaction commits.

Other issues

In addition to addressing the deadlock issues caused by gap locks, we also encountered problems related to AASM records. We were using the AASM gem, a library that manages state transitions. In our case, this library was responsible for changing the state of the test to “completed” and executing several callback functions. These operations were performed as part of a single transaction, which sometimes resulted in prolonged transaction durations and increased the likelihood of deadlocks.

Model dummy code

aasm do
  state :active, initial: true
  state :complete
  event :complete, after: [:method1, :method2, :method3] do
      transitions from: :active, to: :complete
  end
end

When the test is marked as complete and the state changes, all the MySQL-related queries are executed as part of a single transaction.

Due to the execution of all these methods within a single transaction, there were instances where the transaction took a considerable amount of time to complete. These prolonged transactions duration increased the risk of deadlocks occurrence and also resulted in issues related to lock wait time.

FIX

To fix this we moved the insertion of records as a separate transaction out of the aasm state change.
Optimized transaction size: We optimized the other badly written queries in the transaction.
Reduced transaction duration: Only limited number of queries were part of the state change transaction (to keep the transaction short).
We further optimized the GAP lock by avoiding unnecessary delete queries when the records were not present in the table with the corresponding ID.

References

Debugging & Fixing mysql deadlock issue was originally published by eLitmus.com at eLitmus Blog on June 12, 2023.

Website Monitor Using Google App Script

2022-12-30 02:04:27 +0530T00:00:00-00:00

Recently, I was looking for a solution to notify me when a website is down and when it is back up. I found a few solutions, but they all had a learning curve. So I thought of an alternative solution using Google App Script, which I had recently learned about.

Requirements

Can run every 5 minutes.
Can send emails when the website is down.
Trustworthy.

I wasn’t sure if the first requirement was possible with Google App Script, but the other two were. After reading the documentation, I found that it was possible to create a time-based trigger for a script.

Steps to follow:

Create a new Google App Script project.
Create a function to track the website. Here is an example:

function myFunction() {
   const initialUrls = [
     { uri: 'https://mock.codes/200', status: ''},
     { uri: 'https://mock.codes/500', status: ''}
   ];
 
 const properties = PropertiesService.getScriptProperties();
 let urls =  JSON.parse(properties.getProperty('URL_LIST')) || initialUrls;
 const errorResponseCodes = [500, 502, 503, 504];
 const alertEmail = 'alertmail@gmail.com';
 
 const options = { muteHttpExceptions: true };
 
 urls.forEach((url) => {
   let responseCode = UrlFetchApp.fetch(url.uri, options).getResponseCode();
 
   const isErrorResponse = errorResponseCodes.includes(responseCode);
   const wasPreviouslyDown = url.status === 'down';

   if (isErrorResponse && !wasPreviouslyDown) {
     // Site is now down for the first time
     const subject = `Alert: Your site ${url.uri} is currently down`;
     const body = `${url.uri} has encountered an error with status code ${responseCode}`;
     MailApp.sendEmail(alertEmail, subject, body);
     url.status = 'down';
   } else if (!isErrorResponse && wasPreviouslyDown) {
     // Site was previously down, but is now back up
     const subject = `Your site ${url.uri} is now back up`;
     const body = `${url.uri} has recovered and is now back up`;
     MailApp.sendEmail(alertEmail, subject, body);
     url.status = '';
   }
 });

 properties.setProperty('URL_LIST', JSON.stringify(urls));
}

Go to the “Triggers” menu in the left sidebar of the Google App Script project.
Click the “Add Trigger” button and select the function to run.
Choose the options to run the trigger every 5 minutes and click “Save”

Explanation

This above code uses the UrlFetchApp service to make HTTP requests to the websites and check their status. it stores the value of each trigger in a variable so that whenver site goes live again it can send email of website backed up.

You can also check the logs for each trigger execution in the “Execution” menu on the left side of the project.

Conclusion

In conclusion, Google App Script is a useful tool for creating a customized website tracker that can notify the user when a website is down. The process of setting up the tracker is straightforward and the logs can be easily accessed to track the execution of the function. this basic functionality can be enhanced more to record the status in a csv file. also interesting graphs and charts can be made using that data.

Additional investigations

Upptime is one of the good open-source alternative which can be used to monitor a website. it uses github actions to make sure the website is up and creates a issue if website is down for some reason. it also logs the information about the website speed.

Website Monitor Using Google App Script was originally published by eLitmus.com at eLitmus Blog on December 30, 2022.

The revamp of a Video Proctoring Solution: A Behind-the-Scenes Look

2022-12-27 12:33:45 +0530T00:00:00-00:00

The story of how we took a good platform and made it even better

Background

For the past few months, the number of test takers and clients at eLitmus has increased significantly. Conducting all of these tests remotely poses a significant challenge in terms of preventing cheating. To address this issue, eLitmus has developed an in-house solution using the open-source Kurento media server. While this solution has been effective in terms of recording videos, it is not horizontally scalable.

In search of a more effective solution, eLitmus turned to Amazon Kinesis and worked with the AWS team to conduct a proof-of-concept. While this approach allowed for live proctoring, it was not possible to record the exams.

How did it get begin?

As I was learning about WebRTC and Amazon Kinesis during this time, I had the opportunity to attend a session by a company called 100ms. This company is focused on solving problems related to live conferencing, and I was eager to learn more about their approach.

After connecting with the co-founder of 100ms, I received a message from their salesperson to schedule a demo call. During the call, we determined that 100ms could be a potential solution for eLitmus’ scalability problem. However, we needed to weigh the costs of maintaining engineering time and effort to maintain the solution against the opportunity cost of using that time to build a new product, as well as overall server and bandwidth costs.

Based on this analysis, we decided to proceed with a proof-of-concept for live remote proctoring. I spent the next week working on the proof-of-concept and was able to complete it successfully. From there, we saw potential synergies between 100ms and eLitmus and decided to make the product(Knights Watch) an open-source platform.

Designing & Developing

I created a document outlining the requirements for the video proctoring solution, including features such as a proctoring dashboard, candidate tests screen, cheating analysis and verification dashboard, admin dashboard, and auto proctoring. For the first version (v0.1), we planned to roll out the proctoring dashboard with multiple streams visible to the proctor, storage of the video stream on an s3 server, retrieval of the video stream in the cheating analysis and verification dashboard, and admin configuration.

After outlining the requirements for the video proctoring solution, I designed the architecture for the solution, diagrammatically representing how all of the components would be connected. The main components of the app were the 100ms server API, the eLitmus server, and the candidate or proctor’s browser.

Next, I created a milestone on Github and listed out the issues that needed to be addressed, including the integration of the proctor dashboard, candidate test screen, algorithm for assigning candidates and proctors to rooms, and storage of videos on the eLitmus prescribed directory structure on an s3 server.

I began working on these issues and was able to roll out the v0.1 of the proctoring solution within a few weeks. During this time, our team encountered various challenges and suggested various features to 100ms.

Challenges Faced

As we worked on storing videos on an AWS s3 server in our prescribed directory structure, we encountered a challenge with the 100ms API. The webhook provided by 100ms was only for the composite recording of the room, not for individual recordings. However, we needed webhooks to notify us of the success of each individual recording. In addition, 100ms had the functionality for only a single webhook per account, but we needed to support multiple environments with multiple applications within a single account. We requested this feature from 100ms.

While working on an algorithm to assign candidates and proctors to rooms, I faced the challenge of storing authentication tokens in the user’s browser and in Redis storage in production. I wrote an algorithm to handle the expiration of tokens from both ends and to handle multiple events.

As we configured 100ms for various environments including staging, production, and edge, we encountered several issues and suggested various features to 100ms. These included the ability to delete apps and templates from the 100ms dashboard from the front-end, team management options in the dashboard, and handling of access keys and secrets for multiple environments.

Testing live remote proctoring solution

After completing the first version (v0.1) of the video proctoring solution, we were ready to test it in production. eLitmus was conducting an internal hiring event at the time, and we used the live video proctoring feature for this event with around 400 candidates. The event went smoothly, with minor issues. The proctor was able to hear the voices of the candidates and all of the videos were recorded throughout the session.

This success gave us confidence in the solution, and we made some minor tweaks. However, our main concern from the start had been scalability, and we wanted to test the solution at a larger scale. We had an in-person test at IITK with over 600 candidates, and decided to conduct the event with live proctoring. The event went smoothly, but the next day we conducted data analysis and discovered that 14 out of 600+ videos had some data loss or were not recorded.

We had a meeting with 100ms to discuss this issue, and after working with their engineering team, we determined that the issue was caused by network connectivity problems. We fixed the issue and the proctoring solution became more stable, with 97% of the videos being recorded.

Open-sourcing video proctoring solution

After this event, we had discussions with 100ms about pricing and suggested various features, including pricing on the 100ms dashboard itself and the option to opt-in or opt-out of composite recording and browser-based recording.

After making the video proctoring solution an open-source project, I focused on documenting the project so that it could be used by others in the community and more developers could contribute to it. I wrote several documents, including a readme file, information on the architecture and prerequisites, installation guidelines, development guidelines, deployment guidelines, a code of conduct, and guidelines for contributing and welcoming new contributors.

Conclusion

In conclusion, the development of the video proctoring solution at eLitmus was a challenging but rewarding process. By identifying a need to solve the problem of vertical scalability, we were able to explore various solutions and ultimately choose 100ms as a partner to help us build a scalable and effective video proctoring platform. Through the development process, we encountered various challenges and were able to work closely with the 100ms team to find solutions and improve the stability of the platform. We are proud to have made the video proctoring solution an open-source project and to have contributed to the community by documenting the project and welcoming new contributors. We hope that others will find this project useful and will be able to build upon it to create even better solutions in the future.

The revamp of a Video Proctoring Solution: A Behind-the-Scenes Look was originally published by eLitmus.com at eLitmus Blog on December 27, 2022.

Fixing Capybara Flaky Tests

2022-12-20 00:21:23 +0530T00:00:00-00:00

When writing system tests for a user interface, it is common to encounter test cases that fail randomly. One of the common failure can occur when the JavaScript on a page takes time to render, causing issues with the test case.

For example, imagine a test case that clicks a button on a page and then checks for the presence of certain content after the click.

Demo Code

visit submit_page
click_on 'Submit'
assert page.has_content 'Some content after clicking on submit'

In most cases, this test will run without any issues. However, occasionally the test may fail on the third line with the error “Expected false to be truthy”. This error can occur when the page is visited and the JavaScript on the page takes a few seconds to load. During this time, the submit button may be clicked, but because there is no JavaScript associated with the button yet, the button click does not do anything. As a result, the test is still on the submit page when it tries to assert that the expected content is present, causing the test to fail.

Solution

One solution to this problem is to increase the wait_time setting in capybara. However, this approach has several limitations. First, the wait_time setting is global and applies to all test cases, so if it is set to a high value, it will increase the overall execution time of the test suite. Additionally, the wait_time setting only waits for a fixed amount of time before moving on with the test, without checking whether the page has finished loading. This means that if the page takes longer to load than the wait_time

The other solution is to use the execute_script method provided by Capybara to click the button instead of the click_on method. The execute_script method allows you to execute JavaScript code within the context of the current page. By using this method to click the button, the click action is added to the end of the browser’s call stack. This means that the click action will be executed after any existing JavaScript code on the page has finished running, ensuring that the button is fully initialized and ready to be interacted with before the test tries to click it.

To use the execute_script method to click the button, you can use the following code:

page.find_button('Submit').execute_script('this.click()')

This way we can ensure that click method will run only after the page javascript is fully loaded.

Browser Call Stack

|               |
          |               |
          |   JavaScript  |  <-- existing code on the page(1)
          |_______________|
          |               |
          |   JavaScript  |  <-- existing code on the page(2)
          |_______________|
          |               |
          |  click action |  <-- added by execute_script method(3)
          |_______________|

Fixing Capybara Flaky Tests was originally published by eLitmus.com at eLitmus Blog on December 20, 2022.

eLitmus Blog

What's the deal with secret_key_base in Rails?

The short version of what it is:

Where does it come from?

How do you get one?

To sum it up:

Kamal App Deployment Tool

High Level Architecture of Kamal

Build Phase

Container Registry

Deploy Phase: The Core Magic

Practical Uses

Prerequisites to use Kamal

Steps

Build Configuration

Image Tagging

Deploy configurations

Managing Database Migrations

Using Hooks

Server Tags for Migration Control

References

Improving Drag in React JS: A Smoother Approach to Draggable Elements

Book review: Never let me go - Kazuo Ishiguro

Mastering Multi Tenant setup with rails - background jobs

Quick Recap

Introduction

Sidekiq

Running up application along with sidekiq

Triggering Background jobs

Problem?

Addressing the Database Connection Issue

Current Attributes

Sidekiq Middleware

Code - Github Link

Summary

Mastering Multi Tenant setup with rails part 1

Single Tenant application

Multi Tenant Application types

Goal

What features rails 6 brings in

Setup

Creating Middleware for automatic shard switching(ignore if using rails 7 or above)

Summary

References

An in-depth look at Database Indexing

Prerequisite

What is Database indexing?

How database indexes works under the hood?

What are different type of database scan strategies?

Combining Database Indexes

How to setup AWS Cloudwatch alarm for your SES reputation metrics

Building a Collaborative code-editor & Whiteboard: For tech interviews.

The Blog Blueprint:

Need for Collaboration: Unveiling the Why

Cracking the Code: Research and Concluded Solutions

What if two users press two different letters on the same line at the exact same time?

Operational Transformation (OT)

Conflict-free Replicated Data Type (CRDT)

OT Vs CRDT: Final Solution

Code-editor Research

Existing Browser based code-editors

Existing Whiteboards

Feature Showcase

Client Workspace

Multifile Navigation & Room info

Whiteboard

Collaborative Code Editor

Whiteboard

Architecture

Frontend

Backend

Testing

Actors: Use case diagram

Dockerization, Deployment and scale

docker-compose.yml

Docker run command

Scaling

Conclusion

Puma: From Daemonization to Process Control with Systemctl and Monit

Why daemonization should not be part of gem?