RATUL'S RANTS

Intro to Web - A Webmaker Event at AIUB

2013-09-27T00:00:00Z

The Firefox @ AIUB community is relatively a new Firefox Club, in fact it isn't actually a registered club yet. Our idea is to be objective than being subjective and we found that if we had the AIUB authority supporting us, then registering as a club is not the most important thing at this moment. Thus we started our journey to spread the philosophy of Open Web and Open Source in our campus :D

On a small adda, almost two months ago, someone proposed we have a Webmaker event. Since most of us have very little idea about how actually websites work, the proposal was greeted with mass appraisal :) We talked to our honorable faculties and after many days of planning arranged Intro to Web - A Webmaker event. And it was a blast! :D :D All thanks goes to Saef sir for guiding us through the whole event!

We had no prior experience in event organizing, so we did some rigorous planning before this one. And it paid us well ^_^ We even had plan B, plan C in case of emergencies which actually occurred! When three of our Mozillian attendees couldn't make it to the event, we triggered our back up plan and improvised with anything else. Personally, I am grateful that my teammates are so awesome that we had almost all the ends tied up :)

Around 2 PM when half of the computer lab was full and we were still copying local version of Thimble, Mr. Mashiour Rahman, Assistant Professor and Head, Computer Science, dropped in to see our progress! We did ask our honorable teacher to visit us in his free time but this was so unexpected! Mashiour sir even gave a short speech, which was truly inspiring for us :)

Anam started with a introduction to Mozilla. He explained why Mozilla is concerned about the Open Web and why we should care. Then he gave a short review of Mozilla Bangladesh community's activities to date. Afterwards Anam transcended in and out after sessions. A few participants had some questions about Mozilla and it's activities, Anam explained those nicely :)

How can I join Mozilla?

After that Sharif came with X-ray Goggles :D We started with X-ray Goggles because with this cool tool the participants will be able to get a clear view of how websites are constructed.

Sharif with X-ray Goggles

After Sharif, Rifaz came forward with Mozilla Thimble. The participants were eagerly waiting for this one because this is when they get to try out their newly learned coding skills :D

Rifaz explaining tags in HTML

Now that the participants had two tools under their belt, we gave them some time to get creative and build something :D

Let's create something!

Hello from Mozilla!

Volunteers and facilitators helping the participants

Here are a few of the works done by our participants :D

And my personal favorite:

https://reznov9185.makes.org/thimble/wolverine-vs-hulk-edited-by-sajid-

Afterwards we selected 3 of the best Remixes and gave them prizes :D

Around 5 PM when we were at the end of the event the speakers also received some gift packs from AIUB. We gifted lanyards to our honorable faculties.

Thus the event officially ended :D

More photos at Flickr: http://www.flickr.com/photos/minhazr/sets/72157635934429984/

Organized by Firefox @ AIUB community (http://www.facebook.com/groups/fxaiub/)

A part of Mozilla Bangladesh (http://www.facebook.com/groups/MozillaBD/)

How to Have a Mozilla Event at Your University

2013-11-22T00:00:00Z

So you have got selected for Firefox Student Ambassadors program? And already opened a Firefox club at your very own campus? :D Yay! Congratulations! Now the real work begins :D

As a beginner, you, the Firefox Club, can organize a number of events that don’t require extensive experience at event management. One of them, and my personal favorite is the Webmaker. Search it on the web and Mozilla’s wiki for more information.

STEP 1

After establishing your Club and discussing the type of event with your fellow Ambassadors, the first thing you need to do is let your local Mozilla community (Mozilla Bangladesh in this case) know your intentions. You can ask for suggestions and advices in our Facebook groups: Mozilla Bangladesh and Firefox Student Ambassadors, Bangladesh. When you have got a clear view of the plan, send an email to the local Mozilla community’s mailing list (community-bangladesh).

STEP 2

By now you have got okay sign from community members, plus one or more ReMo are ready to help your out :D Good job. I take that you explained every part of your plan and adjusted it as other Mozillians suggested. Now you need to find someone at your university who can get you the permission from authority to arrange the event. You can go to-

The Office of Student Affairs
President of your university’s IT related club
A super-enthusiastic faculty

Explain your plan to them too. They too would have some suggestion and advice, revise your plan accordingly.

STEP 3

You have let both the ends know about what you want to do. Now it is time to get official. You need to send a proposal letter through email or as a printed copy (ask your contact which is better) to your university authority. It might be the Head of the Department, an Executive Officer at Office of Student Affairs, or a Faculty in charge of these activities. You can find a demo proposal letter from asking at Mozilla Bangladesh communication channels. Ask on the mailing list thread about any confusion. Add the necessary details to your proposal letter:

Topic of the event
Duration and date
Speakers and sessions
Target audience and estimated participants
Necessary resources- auditorium, projector, speaker etc.
Advertisement- booths, banners and posters.

You are all set to send the letter :)

Note: Try to fix a date for the event after at least 6 weeks from the confirmation of your proposal. In that way we can ensure swags and have plenty of time to plan in details.

STEP 4

You have all sorts of permissions for your activities. It is high time you had planned the details. Get a few experienced Mozillians to help you out in this. They might also be the ones taking sessions in this event, or they can suggest who can. You need to-

Fix segments for the sessions and gap between them.
Decide how to distribute swags
Always have plan B, C, even D in case of emergency
Select some of your finest as volunteers/facilitator
A photographer friend or someone from the Photography Club for awesome photo memories

STEP 5

It's time for the event :D Check if everything is okay a day before the event. Remember to-

Check your task list/planning page
Keep the next speaker and their instruments ready
Put the volunteers on the move so that they can help the participants
Distribute swags carefully
Be careful with food (if you plan to give refreshments)
Take lots and lots of photos
And have tons of fun :D

STEP 6

I hope the event was a success :D Congratulations you and your awesome mates! Not time to sit back yet :P You have to share your amazing experience with us!

Post on Facebook/Google+/Twitter with the FSA page attached with it
Write a blog post :D with the great photos!
Upload the photos to the Mozilla Bangladesh Flickr group
Email all the links to the mailing list thread!

Aaaaaand you are done! Hope to see more and more events :D

Thanks to my buddy another Mozillian Tanha for reminding me to write a step by step guide :) True that not every event will follow these steps, but it will help to get some idea of where to start.

Mozilla Awareness Booth at AIUB, Spring 2014

2014-02-22T00:00:00Z

"My gosh! I had no idea Mozilla has all these things going on besides the browser!" is how a visitor exclaimed their astonishment after getting to know about Mozilla a bit more from Mozilla Awareness Booth at AIUB. The booth was a successful effort of Firefox at AIUB's members to let the students at their university discover Mozilla from the perspective of a worldwide organization constantly working on to keep the beautiful Web open. We know it was successful when we are still getting messages through social sites and email with queries about joining this collective force, to be a part of the future of Web.

The crowd at Mozilla Awareness Booth

Although the Firefox at AIUB community was born almost a year ago, it's members felt that their fellow students of the institution are still in the dark of their voluntary efforts for Mozilla. Even after organizing a Webmaker event and a meet up in November, last year, the students seem to have no clue on what actually Mozilla does, or how to get involved with it. For this very reason we took contact information from interested people who came to our booth and sent them useful links and resources on this matter.

Interested people provided us with their contact info

A potential Mozillian

This was a great opportunity to reach out for potential web developers and tech enthusiasts who are very eager to know the latest ideas and technologies. In the booth we talked about-

Mozilla's mission
Philosophy of Open Web
Firefox OS and it's potential
Firefox Student Ambassador activities
Necessity of WoMoz

Maliha eplaining why WoMoz is important

We gave the visitors demonstration of Firefox OS with Developer Preview Keons. It might seem a bit harsh but we did enjoy the shock and awe in the faces of the visitors when we told them the entire interface they tried out in those phones were in fact web pages. We also hosted a small game competition of Mozzle, the very first game from Firefox at AIUB group, by rocker coder Anis Tuhin. Sadly the winner is still undecided.

We hope to see at least 100 of new Firefox Student Ambassadors from this two day long Mozilla Awareness Booth. We did learn a lot of new and interesting things about arranging booths. We are researching on these points as you read this blog post, so that we can implement the ideas from the next time.

Mentoring Mozilla Contributors of EWU

2014-06-08T00:00:00Z

Last Thursday we went to East West University (EWU) to mentor the new contributors on how to get involved with Mozilla's cool projects. They recently had a MozAwareness Booth at their campus, so all of the participants were very eager to know all the tricks and tips! They were so hyped up after the sessions that some of them started contributing right away! Here is a part of the contributors:

We started out for EWU a bit late but got there right in time for the event. The lab booked for this event was already house full when we stepped into it :D Around 40 participants were present at the time. As a matter of fact, Dr. Md. Nawab Yousuf Ali sir, Chairperson of the CSE department of EWU was there to welcome us with a short but resourceful speech.

Honorable Chairperson of CSE Department, EWU

House full!

We started the Contributor Mentoring day with an introduction to Mozilla. The participants already had a gross idea of what Mozilla is and what philosophy drives us, but this introduction session cleared up lots of confusions like whether all Mozilla's projects are open source or not and stuff like this. This session was taken by me :D

Me, asking question instead of answering :3

After answering all the curious questions from the participants we gave them hands on tour of IRC, the mailing lists, different Facebook groups of Mozilla. Other than a little problem with IRC chat, they got through all these steps of getting connected to Mozilla Bangladesh.

It was Zarin who gave them idea of the Facebook groups and how to get help on these groups. The also talked about WoMoz and Webmaker. An interesting aspect was that many male and female participants were equally interested in WoMoz, which proves their willingness to promote women in the field of technology. Webmaker was another project that got lots of applause because of it's innovativeness.

Zarin talking about Webmaker

After Zarin, Iqram came up with MDN and SuMo. Many of the participants who work as web developers already knew about MDN so Iqram showed them how we localized almost all of the MDN in to Bangla. He also coverd SuMo, describing it as,"One stop helpline for Mozillians."

Iqram explaining the importance of MDN

After we finished guiding the participants through all the communication channels, there was a question and answer session conducted by Iqram. Whatever confusion they new Mozillians had was wiped clean, thanks to him! :D

Finally, we gave the participants stickers and other swags. For all their effort and work we gave lanyards to the volunteers and a Firefox T-shirt to Sayed Mohammad Amir, the club lead of Firefox@EWU club.

Firefox@EWU's Club Lead Amir

Fahmida, an WoMoz from EWU

The event was a total success, and it ended with a few group photos and selfies :D

Here is another blog post by Maruf Islam about the event(Bangla). He is an active Firefox Student Ambassador from EWU who likes to talk about Firefox OS.

Photos:

If .gitignore Does Not Ignore

2015-04-15T00:00:00Z

Sometimes after changing your .gitignore the new (un)ignored files don't get get behind the scenes, as you would have expected. The reason behind this is you have already committed the files that you added to .gitignore and now git is tracking the files. Unless you delete them git will keep an eye on any changes on those files as it would do with any other. To get the new .gitignore rule registered with the repo you will have to manually remove tho file(s) from git's cache. So here is what you need to do:

First of all, YOU MUST COMMIT any staged progress, otherwise they will get lost.

Then you can go on to removing the problematic files from git's tracking. For only one file, you will have to run this commad and that's it. Git will smartly ignore whatever you tell it to ignore:

git rm -r --cached <file>

But in case you have way too many files to mention one by one, you might want to remove all the files at once. In this case git will stop tracking all files, so you will also have to add them once again manually.

git rm -r --cached <file>
git add .

Lastly, simply commit again:

git commit -m "fixed .gitignore"

Tada, problem solved! If you ever want to undo this, just add the file to git:

git add <file>

Scraping Github

2015-10-18T00:00:00Z

The Setup

Data science is a buzzword these days and everyone is asking how to be a data scientist. A data scientist's job is definitely awesome, because if anyone is close to predicting the future, they are the ones who probably do it. But what does a data scientist actually do? The process of "data science" roughly consists of three things: mining, analysis and visualization. Each one of these three segments is interesting in it's own right. In data mining you get to collect data from large data sources, mostly from the Internet using clever techniques like scraping.

Many websites provide API so that other people can collect data from those sites to use in their work. For example you might want to know a Facebook page's growth over a certain period of time. Facebook made an API just so that you can get the data to do the analysis needed to answer your question. This sort of API is provided by Github and Twitter too. You can not imagine how vastly useful this public data is, they even came in handy as much as detecting earthquakes faster than the U.S. Geological Survey!

Enough talk, let's have some action! In this tutorial, I will get you started with scraping data from API using Python. We will use Github's API as data source and keep the scraped data in a database called RethinkDB. Don't worry if you have never heard of it before, it's pretty cool. If you want to try out other databases, you will just have to tweak a few lines of code. Let's set up the necessary softwares.

You will need to install a few softwares, preferably in a Linux workstation. I am going to use Ubuntu. If you are using Windows, then switch to Ubuntu or any other *nix. There is no way around it. You will need-

Python 3
Virtualenv and virtualenvwrapper
RethinkDB 2.0.x
Sublime Text 3 or any good text editor for coding

Virtualenv and virtualenvwrapper

You probably have already used these if you tried to solve version dependency in your code. These are handy tools that create Python virtual environments that are separate and isolated from eachother. Follow this link to read more and install them. Create and activate a virtual environment for this project:

mkvirtualenv scraping
workon scraping

RethingDB

If you are new to Ubuntu or Linux, installing RethinkDB might be a little bit complicated for you. Nothing you can't do though, just a few lines of extra commands to copy and paste. RethinkDB has really nice docs starting with installation and trying it for the first time.

TIP: RethinkDB server won't start automatically everytime you log on to your PC, so you might want to set it up for autostart.

You will also need the Python driver to connect to the database using your code. Start up the virtual environment you created earlier and install the driver:

pip install rethinkdb

Now you are worthy enough to move on to next stage :D

Fetching Data

API, or Application Programming Interface was created to make things easier for developers who want to use anything made by other people. That's the watered down version of what an API is, you can(should) Google and learn more about it. There are tons of APIs out there that you can use to access anything starting from social media posts to map location data. A site called Programmable Web keeps track of over 13,000 APIs like Google Maps, Twitter, Youtube, Flickr etc.

Are you ready to begin scraping? Definitely you are, you have set up all the tools needed. But the question is, what are you going to scrape? Remember the API I talked about last time?

Exploring Github's API

Websites like Github and Facebook have API or Application Programming Interface so that you can interact with their services and data through your programs. They usually have extensive documentation on how to do this. In Github's API documentation you will see how to send requests to Github, what you will get in reply and how you can filter the reply for your specific needs. How about you check it out? Open up a new tab in the browser and go to the address https://api.github.com/users/username with your own username in place of "username". You will see something similar to this:

{
  "login": "gvanrossum",
  "id": 2894642,
  "avatar_url": "https://avatars.githubusercontent.com/u/2894642?v=3",
  "gravatar_id": "",
  "url": "https://api.github.com/users/gvanrossum",
  "html_url": "https://github.com/gvanrossum",
  "type": "User",
  "site_admin": false,
  "name": "Guido van Rossum",
  "company": "Dropbox",
  "blog": "http://python.org/~guido/",
  "location": "San Francisco Bay Area",
  "email": "guido@python.org",
  "hireable": null,
  "bio": null,
  "public_repos": 4,
  "public_gists": 5,
  "followers": 911,
  "following": 0,
  "created_at": "2012-11-26T18:46:40Z",
  "updated_at": "2015-12-23T18:35:37Z"
}

This is what I see when I visit the address with Python's daddy Guido's username: https://api.github.com/users/gvanrossum. I left out a few lines so that it's easier to read. You will notice that we can know many things about Guido's Github account: when it was created, how many public repositories he has, even how many people he is following. This is how it works: Github takes a few parameters from you (eg. Guido's username) and looks up the related information in their database. Depending on your access permission, Github then shows you a webpage with the available info. The page is in JSON format, that is, list of data in key-value pairs. Not very unlike Python's dictionaries.

Let's start scraping!

Who and what to scrape

Now that we know what sort of data we can get, let's start coding. First of all we need to import the modules we are going to use in this project. We will also be defining a few variables:

import requests

def main():
    users = ['google','facebook','apache']
    baseurl = "https://api.github.com/users/"
    properties = ['login', 'id', 'html_url', 'public_repos', 'created_at']

You probably don't have requests module installed, so type pip install requests in your terminal for that. We can put any Github username in the users list, but for now let's keep Google, Facebook and Apache in it. baseurl is the URL to access Github's API. We will add usernames at the end of this URL to access that user's info. Finally, the properties list has a few properties of the JSON file that we are going to use.

Test run

We have three users in our users list, so we will have to loop through it. But at let's test with one user first. Add these lines to your code:

    # to grab and show info from Github
    # make API url with Github username
    current_page = baseurl + "facebook"  # "https://api.github.com/users/facebook"
    # take response from the url
    response = requests.get(current_page)
    # conversion to human readable format
    json = response.json()
    # print properties in json dictionary
    print(json["name"])
    print(json["id"])
    print(json["html_url"])


if __name__ == '__main__':
    main()

Now run the code. Did you see something like this?

Facebook
69631
https://github.com/facebook
147
2009-04-02T03:35:22Z

Now time for line by line explanation. After putting Facebook's API URL in current_page, we grab the page by this:

    response = requests.get(current_page)

response object now holds the exact response that Github's server replied us with when we try to access the address in current_page. It has got the JSON information of Facebook, along with some other HTTP metadata. We take the JSON info from that using json() method and convert it into Python's familiar dictionary:

    page_info = response.json()

We have all the necessary info in json, you can try testing that by printing it. But we only need the name, id and profile address this time, so:

    # print properties in json dictionary
    print(page_info["name"])
    print(page_info["id"])
    print(page_info["html_url"])

Looping through

We already completed our main objective, that is, scrap info from APIs. Congratulations to you! But we are not done yet. This is not an efficient way to get information of multiple users. So we need to loop through the users list as well as the properties list to get all the user's info. Let's rewrite our test code with the following loop like this:

    # to grab and show info from Github
    for username in users:
        # make API url with Github username
        current_page = baseurl + username
        # take response from the url
        response = requests.get(current_page)
        # conversion to human readable format
        page_info = response.json()
        for property in properties:
            print("{} : {}".format(property, page_info[property]))
        print('')

This time when we run the code, it will grab info of all the usernames in users list. Then print the properties of them too. We can move on to next stage: saving the info in to RethinkDB's database. Here is the full code for further use.

Storing in RethinkDB

RethinkDB is a cool database system that is used for realtime apps. It has a flexible query language, monitoring APIs and is really easy to set up. It does not use SQL to run queries, so you can use your knowledge of Python to do all the stuff you would do with SQL in other databases like MySQL or PostgreSQL. Interestingly data is stored in RethinkDB in the format of JSON. And Python + JSON = <3

Sentiment Analysis: An Intertwinement of Disciplines

2015-11-26T00:00:00Z

How often do you get the urge to know what others think about you, that is, your actions and preferences? No matter what the number is, you are not alone. Humans as individuals, have always been interested in knowing the opinion of the people around them. Should I talk to her? Is it a good idea to buy that car? Who should I vote? These are only the simplest examples of opinion related questions that arise in our minds every single day. Getting a nod from your best friend, a positive gesture from your dad or a better campaign slogan helps a lot to answer these questions. Similar questions exist not only in our personal lives but in the academia, business and governments as well. The hunting ground for opinions on these questions used to be friends and family, acquaintances, consumer reports et cetera.

But have you ever wondered how this scenario has turned out to be in the post web era? During the last decade, mining for public opinion has gotten a lot organized, thanks to the abundance of information in the web. This is what we call opinion mining, otherwise known as sentiment analysis.

By definition sentiment analysis is the process of extraction and analysis of public opinion from any sort of source material using natural language processing, text analysis and computational linguistics. In essence, we try to get the subjective information from a document, usually written. This can help determining the attitude and emotional state of the author or speaker on some certain topic. The simplest task of sentiment analysis is finding out whether the text has a negative or positive attitude. It can even be translated to good or bad. A little more complex is to rank the attitude in a scale of 1 to 5. This gets way more complicated when emotional states like "happy", "sad", or "angry" are added as parameters. Based on this analysis of sentiment can be classified in a few ways:

Polarity or valence
Emotional state or feelings
Subjectivity or objectivity identification

Sentiment analysis draws ideas and techniques from statistics, natural language processing, machine learning and a few more. The massive amount of data found in the web is overwhelming if not sorted out. Statistics helps to get a count of words and sentence patterns to give a overview of the document's polarity. Natural language processing techniques give more accuracy by breaking the sentences in to tokens with weights. Machine learning refine the result further more by learning from previously analyzed data and modifying the algorithms.

Natural language processing is breaking down sentences into a language's bare basic components, nouns, verbs, adverbs, pronouns, punctuations etc. By breaking down a sentence we can understand how a sentence is formed, whether it has negated words or not etc.

An example of application of machine learning in sentiment analysis is the use of Support Vector Machine (SVM). This standard machine-learning technique is said to be the most accurate text classifier method. Naive Bayes classifier is another technique, a Bayes theorem based simple probabilistic classifier widely used in spam filtering, sorting emails, language detection and sentiment detection.

There are many examples of sentiment analysis in action in the internet. For example Sentdex goes through thousands of news articles, analyses words and brand names and provides infographics on the polarity of them.

Setting Up Ember CLI Sass

2017-01-27T00:00:00Z

Trying to set up Sass with Ember can be quiet problematic if you are using a version manager for Node. For example I was using a version manager called 'n' and could not integrate Sass in anyway with my Ember app. It all boiled down to my Ember installation not finding libraries needed to compile Sass. These two modules were the reason of the problem: node-gyp and node-sass.

After fiddling around for a while I found out the proper way to setup Sass with Ember. First, ensure these are not installed in the project's node_modules:

npm uninstall node-gyp
npm uninstall node-sass

Then install them again with the right version on Node:

n latest                        # switch to the latest version of Node
npm install -g node-gyp
npm install -g node-sass
ember install ember-cli-sass    # this will install Ember addons for Sass

Don't let your negative thoughts poison your present moment

2020-07-15T00:00:00Z

In these uncertain times, it always feels like things will get only worse. It's hard to not think about what's going to happen, specially if you were in a good chapter in your life just before shit hit the fan. Your mistakes might feel way bigger than anyone else's, you might have a really hard time letting go of the downsides of decisions you made or events occurring around you. I know I am feeling like that a lot lately.

But it's not real, whatever you are feeling. It's a hard concept to grasp but our minds are like lenses, not mirrors. We don't see things happening around us as they are, we take in the information and perceive it in our own way, warped by our emotions at the time and twisted by how our peers also see them happen. You can be more or less sure the intense regret you are feeling or the longing for a comparatively good time in your heart is almost always an amplified version of the real incidents.

Tens of thousands of years ago we used to live in caves, and when darkness fell upon at the day end, our world would become a terrifying hell of death. Every moving creature in the bushes became humongous predators in our minds. I mean even if the slight rustling was because of the smallest bunny in the land, you would have a better chance of living if you assumed it was just another predator. Years of practiced habit still lingers in us, so much so that we actually have a name for this assume the worst behavior: the negativity bias. This is the reason you can't seem to let go of all those bad thoughts about your decisions, constantly self-doubting and feeling sad.

As I often say to myself, there is no bad time, or good time, as a matter of fact. You are a being living in the moment, you are taking your decisions in the moment, with all the resources you have at hand. Trust in yourself. "I shouldn't have done that","I will regret this","What if I made a huge mistake?" these thoughts will come into your mind, your brain wants you to always review again and again if you are gonna survive whatever you are going through. Gently hold those thoughts and blow them into the air. The deed is done, only way now is to move forward.

Do new things, enjoy them. Savor the long forgotten memories of joy. Rejuvenate your soul with new habits. You got this!

The Fall

2021-02-23T00:00:00Z

If you cup your hands on both of your ears, you will hear a swishing sound that almost feels increases every few seconds. You might say that is the sound of your blood pulsating in your body, recycling carbon dioxide and bringing in new oxygen every now and then.

To me is is the sound of wind passing by us, as we fall through the eternal void. It is a reminder of that inevitable crash we all have to face when this brief flight we have been in since our birth ends and we stop moving forever.

When you first realize this you can throw your arms outwards and cry in fear, you can try to clutch on to anything you can grab a hold of. Suddenly there is something in your extended fists! Another hand is clutching on to yours! The vibration of the wind rushing to meet you from below feel ecstatic almost. You were going to brace yourself for the entirety of this journey towards sure obliteration, but now you have someone to occupy your mind instead of those thoughts. Whenever you look right over your shoulder, there is that person smiling back at you like bright red petunia. You feel like it's not so bad, may be this is what was meant to be, may be we were doomed to meet the ground with someone holding our hands.

You look outwards and see the ones without a hand to grab on to. Something breaks inside you. The wind grows stronger, but you feel like you need to hold that person's hands. You stretch out to them and surprisingly see the other hand, the hand that gave you the warmth in the first place loosens up a bit. You cannot ignore your instinct of flying towards the other body falling with all it's weight centered in it's heart. It's a very heavy one indeed, and you feel like you need to lend some help to carry it.

The other body drops as if it's bound to a rock, you can't seem to get a hold of it. Your heart bleeds each time you try to nudge yourself a bit towards it, but just want to give it some warmth, that's all it needs, right? The warmth you got changed the fall for you, to you it's not the fall that worries you anymore, you are more anxious how to fly in the sky rest of the way! You almost grab the other person by their collar. You close your eyes as you embrace them for one warm and fuzzy hug.

The wind grows colder. What is the end of this fall, anyway? Will it be a big splash? Or a thunderous kapoot! Will you remember the sound when it all ends and you are lying on the floor, looking up towards the gold-blue afternoon sky? You can't think of anything with the anticipation of impending conjuncture. But oh? You wave your hands at nothing! You open your eyes and suddenly discover you have flown off far away. The heavy hearted one is still falling, you haven't been able to give them the purpose to fly, you couldn't show them the dream of the avian marvels of the prehistoric era. Instead you are so far away from those set of warm hands that gave you this newfound joy in meeting your crash, that you don't even know how to get back to it. One single snowflake lands on your fingertip, the freeze starting to spread...

The last thing you see before your eyes turn into crystal orbs of ice is those two pair of hands, the ones that held on to you when you were alone in despair. You have lost the amicable ardor that you were so fond of. You don't hear the sound your body makes when it meets the concrete street and breaks into a thousand piece, you are gone way before.

Not All Data Engineers Are Made Equally - 4 Types of Data Engineer Roles

2023-07-20T00:00:00Z

… or how I realized there isn't one single path to becoming a data engineer.

I had been lurking forums like r/dataengineering for a while now and every few months I see these two questions pop up again and again:

What is a data engineer?

and

How much code do I need to know to be a data engineer?

And every single time, it sparks vibrant yet inconclusive discussions. This is not surprising, though. It's a relatively new field that's evolving like crazy, and it's interesting to see the different roles and backgrounds that make up this diverse community.

I switched to the glamorous job of data plumbing a while back, but I used to be just a plain backend developer. So I believe I can give a bit of shape to the different varieties of data engineering roles we see. I couldn't think of different names for each of the roles since they all indeed do the one essential thing: move data from one place to another so that it makes more sense. If data engineering were a role-playing game, there would be 4 classes: The Wizards with their magical SQL queries, The Jedis who wield the force of software engineering principles, The Ninjas of no-code solutions, and last but not least, The Chameleons who can adapt to any data needs.

1. The Wizards: Traditional SQL-Driven Data Engineer

First up, we have the SQL-based traditional data engineers, the Parselmouths of the data world. They are proficient in writing extremely complex SQL queries, designing and optimizing database structures, and transforming data for reporting and analytics purposes. Often, these data engineers come from backgrounds in SQL development, data analysis, or business intelligence (BI).

Strengths

Proficient in data querying and modeling, enabling quick and accurate data extraction and analysis.
Strong understanding of databases, optimizing their structures for efficient data processing.
Swift responders to business data requests in data-driven organizations.

Weaknesses

Limited ability to handle complex ETL processes or automate data workflows without additional coding skills.
May face challenges in adopting modern tech stacks or integrating machine learning algorithms.

Skills Needed

To join the ranks of the wizards, you'll need a solid foundation in SQL and a deep understanding of databases:

Start with the Basics: Familiarize yourself with SQL syntax, data types, and simple SELECT statements.
Master Data Manipulation: Learn how to filter, aggregate, and join data using advanced SQL concepts like GROUP BY, JOIN, and subqueries.
Understand Database Concepts: Grasp the fundamentals of relational databases and their design principles, including normalization and indexing.
Real-world Practice: Gain practical experience in data manipulation using SQL by working on personal projects or analyzing open datasets on sites like Kaggle.

2. The Jedis: When Software Engineers Use The Force

Next, we've got the coders turned data Jedis, who use the Force aka software engineering principles to build data-intensive platforms. With a coding background, they thrive in utilizing technologies like Apache Spark or Apache Airflow to tackle complex tasks.

The fundamental difference between these engineers and the SQL-driven DEs is the ability to use application development tools like docker, logging, microservices, and APIs to implement domain-driven solutions.

Strengths

Proficient in coding and software engineering practices.
Skilled in building robust data pipelines and automating ETL processes.
Comfortable with building complex data architectures and services.
Adaptive in integrating modern tech stacks and AI/ML algorithms into data workflows.

Weaknesses

May lack in-depth knowledge of SQL querying and modeling techniques.
Might be overkill in certain environments where no/low code or SQL based is preferred.

Skills Needed

Coders looking to transition into data engineering can leverage their programming skills to become Data Jedis:

Programming languages: Create data pipelines using programming languages like Python or Java, integrating them with data sources and storage solutions.
Learn Data Engineering Tools: Familiarize yourself with popular data engineering frameworks like Pandas/Polars, Apache Airflow, and Prefect. Online tutorials and documentation can be valuable resources.
Embrace Big Data Technologies: Dive into distributed computing software like Apache Spark, Apache Flink, Dask, and storage technologies like Hadoop and Amazon S3, which are crucial in handling large-scale data.
Explore Cloud Platforms: Familiarize yourself with cloud services like AWS, Google Cloud, or Azure, as they offer robust data engineering tools and infrastructure. Also learn Docker, Kubernetes, etc to deploy your code there.

3. The Ninjas: No-Code or Low-Code Tools Instead of Katanas

These data engineers may not be hardcore coders or SQL junkies, but they can amaze you with their GUI-based tools that make data workflows a breeze. Often coming from non-technical backgrounds like finance or business, they've picked up data engineering skills through BI tools or drag-and-drop ETL platforms. Their main focus is to get the job done as soon as possible, be fast just like a ninja.

Strengths

Excel in using GUI-based tools and low-code platforms for building data workflows quickly and efficiently.
Ideal for business-oriented teams or individuals with non-technical backgrounds seeking to engage in data engineering tasks.
Rapid adoption and ease of use make them valuable assets for fast-paced projects.

Weaknesses

May lack the flexibility and customization options of traditional coding approaches.
Some GUI tools may not support complex data engineering tasks or scale well with growing data volumes.

Skills needed

No-code and low-code platforms empower non-technical individuals to participate in data engineering. Aspiring Wizards can follow these steps:

Discover No-Code Tools: Explore user-friendly platforms like Microsoft Power Automate, Zapier, or Google Cloud Data Fusion, which enable data workflows without coding.
Data Integration: Learn how to connect various data sources and automate data transformations using drag-and-drop interfaces.
Master Data Transformation: Utilize GUI tools to cleanse, enrich, and prepare data for downstream analytics and reporting.
Practice with Business Use Cases: Work on real-world business scenarios, creating data workflows that cater to specific business needs.

4. The Chameleon: Hybrid Data Engineers

Last but not least, we got Hybrid Data Engineers who are essentially the chameleons of the data world. As data engineering practices have evolved, a hybrid role has emerged that combines software engineering expertise with a strong focus on data processes and business needs. These data engineers understand the technical aspects of data engineering while also aligning solutions with the organization's broader goals. Just like how chameleons adapt to any environment depending on the requirement, these engineers can encompass data/business analysis, ETL pipeline development, visualization, etc.

In comparison to software developers turned data engineers, these folks take just the bare minimum required skills from software engineering and focus the rest of their attention on understanding the business.

Strengths

Effective translators between technical teams and stakeholders, navigating both software engineering and data management challenges.
Proficient in crafting data solutions that align with specific business needs and requirements.
Valuable problem solvers, capable of navigating both software engineering and data management challenges.

Weaknesses

Striking the right balance between technical best practices and business agility can be a delicate challenge.
May require a diverse skill set, making their learning curve potentially steeper compared to specialized roles.

Skills needed

Hybrid Data Chameleons combine technical skills with business acumen:

Develop Data and Business Skills: Master SQL querying, data modeling, and programming languages while also enhancing your understanding of business operations and requirements.
Communication and Collaboration: Hone your communication skills to effectively translate technical concepts to non-technical stakeholders and vice versa.
Infrastructure: Learn different IaC tools such as Docker Compose, Terraform, and Pulumi to maintain reproducible data infrastructure to help with setting up

Challenges and Opportunities in Data Engineering

From what I have seen, I believe the data engineering landscape will keep changing in the next 5 years, branching into distinct specializations within the field. Some key points to consider:

1. Legacy vs. Modern Tech Stacks

Many data engineering teams face the challenge of dealing with legacy technologies and processes that might not align with modern software engineering practices. Transitioning from traditional BI-focused platforms to modern data stacks can be a bumpy road, especially if the team lacks a strong software development background.

2. The Human Element: Embracing Change

Change is hard, and this holds true in data engineering too. Introducing new technologies, best practices, and workflows can sometimes meet resistance from team members who prefer sticking to familiar methods. Building a culture of openness to change and continuous learning is crucial for successful data engineering teams.

3. Balancing Flexibility and Best Practices

Striking a perfect balance between business pressure and software engineering principles is essential. While serving the pressing needs of the stakeholders is our foremost priority, it is important that we adopt modern software dev practices like TDD, version control, and log management. This ensures the data infrastructure is something that can be recreated from scratch again and again, making it the resilient data backbone of the organization.

Into The Looking Glass: The Future of Data Engineering

With the rise of machine learning and AI, data engineers will play an even more pivotal role in extracting meaningful insights from the vast ocean of data. With that in mind, more and more teams will adopt cloud-centric products for cost-effective solutions with better ML integration. DataOps practices and DataOps engineering teams will emerge, leading to greater collaboration between data engineers, data scientists, and other stakeholders.

As data privacy and security become paramount concerns, data engineers will embrace practices that prioritize data governance and compliance. These are pure speculations, however, the future is bright for data engineers. The more AI and ML get embedded in our data-driven decision-making process, the need for clean and reliable data pipelines becomes more important.

Data engineering is an exciting and ever-evolving domain with diverse roles and backgrounds shaping its landscape. From SQL Wizards to software Jedis and no-code Ninjas, every type of data engineer has a place in the world of data.

Thank you for reading my analysis of the data engineering role in the year 2023. Feel free to hit me up through my email address or DM me on Twitter @ratulotron.