Python for Data Engineers

Introduction

• Python's role in building the data pipelines.
• Python installation.
• Operators.
• Sequential or Collection or Array data types.

                Lists

                - Discussion on lists

                - Uses of lists

                - Creating a list

                - Checking the datatype

                - Creating an empty list

                - Drop a list using the  del statement

                - Finding the length of the list

                - Creating a list with the elements of different data types

                - Accessing the elements of the list

                - Fetching the last element of the list

                - Remove the last element of the list using pop()

                - Adding an element to the list using append()

                - Adding an element in a given position to the list using insert()

                - Removing an element to the list using remove()

                - Replacing an element of the list

                - Sort the elements based on the value

                - Sort the elements based on the length            

                - Sort the elements in descending order

                - Reverse the elements of the list using reverse()

                - Removing the duplicate elements from the list using dict()

                - Finding the no. of times a given element of a list is repeated using count()

                - Combine the elements of the two lists using the plus operator

                - Find the differences between two lists using the minus operator

                - Add/subtract/multiply/divide the elements of the two lists using zip()

                - Finding the exclusive elements from the two lists using set() and '-' operator

                - Finding the common elements from the two lists using set() and '&' operator

                - What is the rule for converting a string into a list

                - Converting a string into a list using split()

                - Two-step method converting a date into a list using str(), list() and split()

                - Converting a tuple into a list using list()

                - Make a copy of a list using copy()

                - Converting a list into a set using set()

                - Place a list as an element in another list.

                - Converting a list into a dictionary

                  - tuples in the list or nested list using dict()

                  - two lists, one has headers and the other one has the values using zip()

                - Create a list with the values from a dictionary

                - Create a list with the keys from a dictionary

                - Converting a list into a dataframe

                - Create a list from the dataframe first row using tolist()

                - Create a nested list from the dataframe rows

                - Find the size of a list using getsizeof()

                Tuples

                - Discussion on tuples and properties

                - Uses of tuples

                - Making a nested tuple

                Sets

                - Discussion on sets and properties

                - Uses of sets

                - Creating a set from a string using split()

                - Performing minus operation (-)

                - Performing union operation (|)

                - Performing intersection operation (&)

                - Finding uncommon elements (^)

                Dictionary

                - Discussion on dictionary and properties

                - Uses of dictionaries

                - Creating the dictionary

                - Finding the no. of items in the given dictionary using len()

                - Creating a dictionary using dict() function

dict([('language_1', 'Python'), ('language_2','Java'), ('language_3','C#')])

dict(language_1 = 'Python', language_2 = 'Java', language_3 = 'C#')

                - Creating a nested dictionary

z=dict([('language_1', 'Python'), ('language_2','Java'), ('languages',dict([('language_1','Go'),('language_2','JavScript'), ('language_3','C++')]))])

                - Creating a dictionary using lists

>>> questions = ['name', 'quest', 'favorite color']

>>> answers = ['lancelot', 'the holy grail', 'blue']

>>> dict(zip(questions,answers))

{'name': 'lancelot', 'quest': 'the holy grail', 'favorite color': 'blue'}

                - Getting the list of keys and values

list(z.keys()) or list(z) 

list(z.values())

                - Accessing the value of a given key

z['languages']

z['languages']['language_2']

- Deleting an item from the dictionary

del z['language_1']

- Checking an element exists or not

'language' in z

'language' not in z

- Converting the dictionary into tuples or reading key & values into variables using item()

z.items()

>>>for k, v in z.items():

...     print(k)

...     print(f'\t{v}')

- Converting a dictionary into Pandas Dataframe

df=pd.Dataframe(<dictionary name>)

• Comments & Print statements

                - Single-line comments (# Hi, ' Hi ', " Hi ")

                - Multi-line comments (""" Hi """ or ''' Hi ''')

                - Print an empty line

                - Print in the next or new line

                - Print after a few spaces

                - Print a statement with variable values or formatted print statement

• Date and time operations

                - Classes in the package datetime

date, time, datetime, timedelta, timezone

                - Print the current day date

date.today()

                - Print the current date time

datetime.today() or datetime.now()

                - Converting a string into a date

datetime.strptime('2023-feb-10', '%Y-%b-%d')

datetime.strptime('2023-feb-10', '%Y-%b-%d').date()

datetime.strptime('Fri, 2023-feb-10', '%a, %Y-%b-%d')

datetime.strptime('Fri, 2023-feb-10', '%a, %Y-%b-%d').date()

                - Converting a string into date time or specifying the format of date time.

datetime.strptime('2023-feb-10 09:10:23.018371', '%Y-%b-%d %H:%M:%S.%f')

                - Converting a string into a time

datetime.strptime('2023-feb-10 09:10:23.018371', '%Y-%b-%d %H:%M:%S.%f').time()

                - Converting a string (2023-Feb-10) into a date

                - Converting a string (Fri, 2023-Feb-10) into a date

Best resource: https://www.w3schools.com/python/python_datetime.asp

                - Print the weekday of a given date

                - Extract the day, month, and year of the given date

                - Extract the hour, minutes, and seconds of the given date time

                - Add days, months, weeks, years to a given date using timedelta()

a=datetime.strptime('2023-feb-10 09:10:23.073698', '%Y-%b-%d %H:%M:%S.%f')+timedelta(days=3)

b=datetime.strptime('2023-feb-10 09:10:23.073698', '%Y-%b-%d %H:%M:%S.%f')+timedelta(days=4)

a-b

b-a

                - Converting the date format to Mar, 10, 2022 using function strftime()

print(datetime.now().strftime('%b, %Y %d %H:%M:%S.%f'))

                - Working with Time zones

- Getting the timezone of the current time

datetime.today().astimezone()

datetime.today().astimezone().tzname()

- Changing the timezone of the current time to utc

datetime.utcnow()

- Changing a given time to any timezone

- pytz module for timezones

print(datetime.now().astimezone(pytz.timezone('US/Eastern')).strftime('%Y-%m-%d %H:%M:%S.%f %Z%z'))

• String functions

capitalize(),casefold(),count(),

center(),encode(),endswith(),expandtabs(),

find(),format(),format_map(),

index(),isalnum(),isalpha(),isascii(),isdecimal(),isdigit(),isidentifier(),islower(),isnumeric(),isprintable(),isspace(),istitle(),isupper(),

join(),

ljust(),lower(),lstrip(),

maketrans(),

partition(),

removeprefix(),removesuffix(),replace(),rfind(),rindex(),rjust(),rpartition(),rsplit(),rstrip(),

split(),splitlines(),startswith(),strip(),swapcase(),

title(),translate(),

upper(),

zfill()

• If statement
• For loop
• While loop
• Comprehensions

               - List Comprehension

               - Dictionary Comprehension

               - Set Comprehension

               - Generator Comprehension

• Create ranges

               - Print a range of dates between the two given dates.

• Functions
• Classes
• Package
• File operations

- Creating a file  

f=open('C:/Users/farid/sample_py_file.txt','w')

The above command creates a file. If there is already a file with the same name it will be overwritten.

- Write some content into the file created

file.write('India')

Allows to write one message at a time.

- View the content or save the file

file.close(). Open the file and check.

- Add some more data to the file

f=open('C:/Users/farid/sample_py_file.txt','a')

If the file doesn't exist, it will error.

- Write the content in a new line 

file.writelines(['\nSrilanka\nBangladesh'])

or file.writelines('\nSrilanka\nBangladesh')

It allows writing multiple messages at a time. For example each line as a separate element in a list.

file.close()

- Read the full file.

f=open('C:/Users/farid/sample_py_file.txt','r')

file.read(). It is the default mode.

file.close()

- Read the file line by line

f=open('C:/Users/farid/sample_py_file.txt','r')

file.readline() 

file.readline()

file.readline()

file.close()

- Read the first 3 characters of the file

f=open('C:/Users/farid/sample_py_file.txt','r')

file.readline(3) 

file.readline(3)

file.readline(3)

file.close()

- Read the file as a sequence of elements in a list

f=open('C:/Users/farid/sample_py_file.txt','r')

file.readlines()

file.close()

- Reading the data from 3rd character.

f=open('C:/Users/farid/sample_py_file.txt','r')

f.seek(3)

f.readline()

f.close()

- Other file modes available for file operations

w+, a+, r+

If the file doesn't exist in the modes a+, r+ it will error.

• Exception handling

- Need for exception handling

- try..except block 

- Multiple except blocks

- try..except..else

- try..except..finally

- Nesed try..except handling

- Types of exceptions

   - Built-in exceptions

     - list all the exceptions

         - https://docs.python.org/3/library/exceptions.html#exception-hierarchy    

   - Custom exceptions

• Lambda/ Anonymous function

- Use of Lambda functions

- Syntax and basics of Lambda functions

- Using Lambda functions in built-in functions.

• Built-in functions

- abs(),aiter(),all(),any(),anext(),ascii()

- bin(),bool(),breakpoint(),bytearray(),bytes()

- callable(),chr(),classmethod(),compile(),complex()

- delattr(),dict(),dir(),divmod()

- enumerate(),eval(),exec()

- filter(),float(),format(),frozenset()

- getattr(),globals()

- hasattr(),hash(),help(),hex()

- id(),input(),int(),isinstance(),issubclass(),iter()

- len(),list(),locals()

- map(),max(),memoryview(),min()

- next()

- object(),oct(),open(),ord()

- pow(),print(),property()

- range(),repr(),reversed(),round()

- set(),setattr(),slice(),sorted(),staticmethod(),str(),sum(),super()

- tuple(),type()

- vars()

- zip(), __import__()