4 Levels of GitHub Actions: A Guide to Data Workflow Automation

has change into an indispensable factor for making certain operational effectivity and reliability in trendy software program improvement. GitHub Actions, an built-in Steady Integration and Steady Deployment (CI/CD) device inside GitHub, has established its place within the software program improvement business by offering a complete platform for automating improvement and deployment workflows. Nevertheless, its functionalities prolong past this … We are going to delve into the usage of GitHub Actions inside the realm of knowledge area, demonstrating the way it can streamline processes for builders and information professionals by automating information retrieval from exterior sources and information transformation operations.

GitHub Motion Advantages

Github Actions are already well-known for its functionalities within the software program improvement area, whereas in recent times, additionally found as providing compelling advantages in streamlining information workflows:

• Automate the info science environments setup, corresponding to putting in dependencies and required packages (e.g. pandas, PyTorch).
• Streamline the info integration and information transformation steps by connecting to databases to fetch or replace data, and utilizing scripting languages like Python to preprocess or rework the uncooked information.
• Create an iterable information science lifecycle by automating the coaching of machine studying fashions at any time when new information is accessible, and deploying fashions to manufacturing environments routinely after profitable coaching.
• GitHub Actions is free for limitless utilization on GitHub-hosted runners for public repositories. It additionally gives 2,000 free minutes of compute time per thirty days for particular person accounts utilizing non-public repositories. It’s straightforward to arrange for constructing a proof-of-concept merely requiring a GitHub account, with out worrying about opting in for a cloud supplier.
• Quite a few GitHub Actions templates, and group sources can be found on-line. Moreover, group and crowdsourced boards present solutions to widespread questions and troubleshooting assist.

GitHub Motion Constructing Blocks

GitHub Motion is a characteristic of GitHub that permits customers to automate workflows straight inside their repositories. These workflows are outlined utilizing YAML information and will be triggered by varied occasions corresponding to code pushes, pull requests, difficulty creation, or scheduled intervals. With its intensive library of pre-built actions and the flexibility to write down customized scripts, GitHub Actions is a flexible device for automating duties.

• Occasion: If in case you have come throughout utilizing an automation in your units, corresponding to turning on darkish mode when after 8pm, then you’re acquainted with the idea of utilizing a set off level or situation to provoke a workflow of actions. In GitHub Actions, that is known as an Occasion, which will be time-based e.g. scheduled on the first day of the month or routinely run each hour. Alternatively, Occasions will be triggered by sure behaviors, like each time adjustments are pushed from an area repository to a distant repository.
• Workflow: A workflow consists by a sequence of jobs and GitHub permits flexibility of customizing every particular person step in a job to your wants. It’s usually outlined by a YAML file saved within the .github/workflow listing in a GitHub repository.
• Runners: a hosted setting that permits operating the workflow. As a substitute of operating a script in your laptop computer, now you’ll be able to borrow GitHub hosted runners to do the job for you or alternatively specify a self-hosted machine.
• Runs: every iteration of operating the workflow create a run, and we will see the logs of every run within the “Actions” tab. GitHub gives an interface for customers to simply visualize and monitor Motion run logs.

4 Ranges of Github Actions

We are going to display the implementation GitHub actions via 4 ranges of issue, beginning with the “minimal viable product” and progressively introducing extra elements and customization in every stage.

1. “Easy Workflow” with Python Script Execution

Begin by making a GitHub repository the place you need to retailer your workflow and the Python script. In your repository, create a .github/workflows listing (please observe that this listing should be positioned inside the workflows folder for the motion to be executed efficiently). Inside this listing, create a YAML file (e.g., simple-workflow.yaml) that defines your workflow.

The reveals a workflow file that executes the python script hello_world.py primarily based on a handbook set off.

title: simple-workflow

on: 
    workflow_dispatch:
    
jobs:
    run-hello-world:
      runs-on: ubuntu-latest
      steps:
          - title: Checkout repo content material
            makes use of: actions/checkout@v4
          - title: run hey world
            run: python code/hello_world.py

It consists of three sections: First, title: simple-workflow defines the workflow title. Second, on: workflow_dispatch specifies the situation for operating the workflow, which is manually triggering every motion. Final, the workflow jobs jobs: run-hello-world break down into the next steps:

• runs-on: ubuntu-latest: Specify the runner (i.e., a digital machine) to run the workflow — ubuntu-latest is a typical GitHub hosted runner containing an setting of instruments, packages, and settings obtainable for GitHub Actions to make use of.
• makes use of: actions/checkout@v4: Apply a pre-built GitHub Motion checkout@v4 to tug the repository content material into the runner’s setting. This ensures that the workflow has entry to all crucial information and scripts saved within the repository.
• run: python code/hello_world.py: Execute the Python script situated within the code sub-directory by operating shell instructions straight in your YAML workflow file.

2. “Push Workflow” with Setting Setup

The primary workflow demonstrated the minimal viable model of the GitHub Motion, but it surely didn’t take full benefit of the GitHub Actions. On the second stage, we are going to add a bit extra customization and functionalities – routinely arrange the setting with Python model 3.11, set up required packages and execute the script at any time when adjustments are pushed to most important department.

title: push-workflow

on: 
    push:
        branches:
            - most important

jobs:
    run-hello-world:
      runs-on: ubuntu-latest
      steps:
          - title: Checkout repo content material
            makes use of: actions/checkout@v4
          - title: Arrange Python
            makes use of: actions/setup-python@v5
            with:
              python-version: '3.11' 
          - title: Set up dependencies
            run: |
              python -m pip set up --upgrade pip
              pip set up -r necessities.txt
          - title: Run hey world
            run: python code/hello_world.py

• on: push: As a substitute of being activated by handbook workflow dispatch, this enables the motion to run at any time when there’s a push from the native repository to the distant repository. This situation is usually utilized in a software program improvement setting for integration and deployment processes, which can be adopted within the Mlops workflow, making certain that code adjustments are constantly examined and validated earlier than being merged into a special department. Moreover, it facilitates steady deployment by routinely deploying updates to manufacturing or staging environments as quickly as adjustments are pushed. Right here we add an non-obligatory situation branches: -main to solely set off this motion when it’s pushed to the principle department.
• makes use of: actions/setup-python@v5: We added the “Arrange Python” step utilizing GitHub’s built-in motion setup-python@v5. Utilizing the setup-python motion is the advisable means of utilizing Python with GitHub Actions as a result of it ensures constant habits throughout completely different runners and variations of Python.
• pip set up -r necessities.txt: Streamlined the set up of required packages for the setting, that are saved within the necessities.txt file, thus pace up the additional constructing of knowledge pipeline and information science resolution.

If you’re within the fundamentals of establishing a improvement setting to your information science tasks, my earlier weblog put up “7 Tips to Future-Proof Machine Learning Projects” gives a bit extra clarification.

3. “Scheduled Workflow” with Argument Parsing

On the third stage, we add extra dynamics and complexity to make it extra appropriate for real-world purposes. We introduce scheduled jobs as they convey much more advantages to an information science venture, enabling periodic fetching of newer information and lowering the necessity to manually run the script at any time when information refresh is required. Moreover, we make the most of dynamic argument parsing to execute the script primarily based on completely different date vary parameters in accordance with the schedule.

title: scheduled-workflow

on: 
    workflow_dispatch:
    schedule:
        - cron: "0 12 1 * *" # run 1st day of each month

jobs:
    run-data-pipeline:
        runs-on: ubuntu-latest
        steps:
            - title: Checkout repo content material
              makes use of: actions/checkout@v4
            - title: Arrange Python
              makes use of: actions/setup-python@v5
              with:
                python-version: '3.11'  # Specify your Python model right here
            - title: Set up dependencies
              run: |
                python -m pip set up --upgrade pip
                python -m http.consumer
                pip set up -r necessities.txt
            - title: Run information pipeline
              run: |
                  PREV_MONTH_START=$(date -d "`date +%Ypercentm01` -1 month" +%Y-%m-%d)
                  PREV_MONTH_END=$(date -d "`date +%Ypercentm01` -1 day" +%Y-%m-%d)
                  python code/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END
            - title: Commit adjustments
              run: |
                  git config consumer.title ''
                  git config consumer.electronic mail '[email protected]>'
                  git add .
                  git commit -m "replace information"
                  git push

• on: schedule: - cron: "0 12 1 * *": Specify a time primarily based set off utilizing the cron expression “0 12 1 * *” – run at 12:00 pm on the first day of each month. You should use crontab.guru to assist create and validate cron expressions, which observe the format: “minute/hour/ day of month/month/day of week”.
• python code/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END: “Run information pipeline” step runs a sequence of shell instructions. It defines two variables PREV_MONTH_START and PREV_MONTH_END to get the primary day and the final day of the earlier month. These two variables are handed to the python script “fetch_data.py” to dynamically fetch information for the earlier month relative to at any time when the motion is run. To permit the Python script to simply accept customized variables through command-line arguments, we use argparse library to construct the script. This deserves a separate matter, however right here is fast preview of how the python script would seem like utilizing the argparse library to deal with command-line arguments ‘–begin’ and ‘–finish’ parameters.

## fetch_data.py

import argparse
import os
import urllib

def most important(args=None):
	  parser = argparse.ArgumentParser()
	  parser.add_argument('--start', kind=str)
	  parser.add_argument('--end', kind=str)
	  args = parser.parse_args(args=args)
	  print("Begin Date is: ", args.begin)
	  print("Finish Date is: ", args.finish)
	  
	  date_range = pd.date_range(begin=args.begin, finish=args.finish)
	  content_lst = []
	
	  for date in date_range:
	      date = date.strftime('%Y-%m-%d')
	
		  params = urllib.parse.urlencode({
	          'api_token': '',
	          'published_on': date,
	          'search': search_term,
	      })
		  url = '/v1/information/all?{}'.format(params)
		    
		  content_json = parse_news_json(url, date)
		  content_lst.append(content_json)

	  with open('information.jsonl', 'w') as f:
	      for merchandise in content_lst:
	          json.dump(merchandise, f)
	          f.write('n')
	  
      return content_lst

When the command python code/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END executes, it creates a date vary between $PREV_MONTH_START and $PREV_MONTH_END. For every day within the date vary, it generates a URL, fetches the day by day information via the API, parses the JSON response, and collects all of the content material right into a JSON record. We then output this JSON record to the file “information.jsonl”.

- title: Commit adjustments
  run: |
      git config consumer.title ''
      git config consumer.electronic mail '[email protected]>'
      git add .
      git commit -m "replace information"
      git push

As proven above, the final step “Commit adjustments” commits the adjustments, configures the git consumer electronic mail and title, levels the adjustments, commits them, and pushes to the distant GitHub repository. It is a crucial step when operating GitHub Actions that end in adjustments to the working listing (e.g., output file “information.jsonl” is created). In any other case, the output is simply saved within the /temp folder inside the runner setting, and seems as if no adjustments have been made after the motion is accomplished.

4. “Safe Workflow” with Secrets and techniques and Setting Variables Administration

The ultimate stage focuses on bettering the safety and efficiency of the GitHub workflow by addressing non-functional necessities.

title: secure-workflow

on: 
    workflow_dispatch:
    schedule:
        - cron: "34 23 1 * *" # run 1st day of each month

jobs:
    run-data-pipeline:
        runs-on: ubuntu-latest
        steps:
            - title: Checkout repo content material
              makes use of: actions/checkout@v4
            - title: Arrange Python
              makes use of: actions/setup-python@v5
              with:
                python-version: '3.11'  # Specify your Python model right here
            - title: Set up dependencies
              run: |
                python -m pip set up --upgrade pip
                python -m http.consumer
                pip set up -r necessities.txt
            - title: Run information pipeline
              env:
                  NEWS_API_TOKEN: ${{ secrets and techniques.NEWS_API_TOKEN }} 
              run: |
                  PREV_MONTH_START=$(date -d "`date +%Ypercentm01` -1 month" +%Y-%m-%d)
                  PREV_MONTH_END=$(date -d "`date +%Ypercentm01` -1 day" +%Y-%m-%d)
                  python code/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END
            - title: Examine adjustments
              id: git-check
              run: |
                  git config consumer.title 'github-actions'
                  git config consumer.electronic mail '[email protected]'
                  git add .
                  git diff --staged --quiet || echo "adjustments=true" >> $GITHUB_ENV
            - title: Commit and push if adjustments
              if: env.adjustments == 'true'
              run: |
                  git commit -m "replace information"
                  git push
                  

To enhance workflow effectivity and cut back errors, we add a verify earlier than committing adjustments, making certain that commits and pushes solely happen when there are precise adjustments for the reason that final commit. That is achieved via the command git diff --staged --quiet || echo "adjustments=true" >> $GITHUB_ENV.

• git diff --staged checks the distinction between the staging space and the final commit.
• --quiet suppresses the output — it returns 0 when there aren’t any adjustments between the staged setting and dealing listing; whereas it returns exit code 1 (normal error) when there are adjustments between the staged setting and dealing listing
• This command is then linked to echo "adjustments=true" >> $GITHUB_ENV via the OR operator || which tells the shell to run the remainder of the road if the primary command failed. Due to this fact, if adjustments exist, “adjustments=true” is handed to the setting variable $GITHUB_ENV and accessed on the subsequent step to set off git commit and push conditioned on env.adjustments == 'true'.

Lastly, we introduce the setting secret, which boosts safety and avoids exposing delicate info (e.g., API token, private entry token) within the codebase. Moreover, setting secrets and techniques supply the good thing about separating the event setting. This implies you’ll be able to have completely different secrets and techniques for various levels of your improvement and deployment pipeline. For instance, the testing setting (e.g., within the dev department) can solely entry the check token, whereas the manufacturing setting (e.g. in the principle department) will be capable of entry the token linked to the manufacturing occasion.

To arrange setting secrets and techniques in GitHub:

1. Go to your repository settings
2. Navigate to Secrets and techniques and Variables > Actions
3. Click on “New repository secret”
4. Add your secret title and worth

After establishing the GitHub setting secrets and techniques, we might want to add the key to the workflow setting, for instance beneath we added ${{ secrets and techniques.NEWS_API_TOKEN }} to the step “Run information pipeline”.

- title: Run information pipeline
  env:
      NEWS_API_TOKEN: ${{ secrets and techniques.NEWS_API_TOKEN }} 
  run: |
      PREV_MONTH_START=$(date -d "`date +%Ypercentm01` -1 month" +%Y-%m-%d)
      PREV_MONTH_END=$(date -d "`date +%Ypercentm01` -1 day" +%Y-%m-%d)
      python code/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END

We then replace the Python script fetch_data.py to entry the setting secret utilizing os.environ.get().

import os api_token = os.environ.get('NEWS_API_TOKEN')

Take-Residence Message

This information explores the implementation of GitHub Actions for constructing dynamic information pipelines, progressing via 4 completely different ranges of workflow implementations:

• Degree 1: Fundamental workflow setup with handbook triggers and easy Python script execution.
• Degree 2: Push workflow with improvement setting setup.
• Degree 3: Scheduled workflow with dynamic date dealing with and information fetching with command-line arguments
• Degree 4: Safe pipeline workflow with secrets and techniques and setting variables administration

Every stage builds upon the earlier one, demonstrating how GitHub Actions will be successfully utilized within the information area to streamline information options and pace up the event lifecycle.