Monitoring Airflow with Streamlit

J April, 2024 / guda

Having a lot of customers on Airflow and monitoring their data pipelines could become a problem.

There is no way to find which one of 100 pipelines didn’t run today or has been processed more than once.

If you want to know the average times and when all the client pipelines finished you have to dig into the airflow database.

Do you have retries of your tasks? How many?

How your dag executions compared to the one from last week?

For that reason I decided to bring a dashboard with the metrics which I missed.

starting a new project with python poetry is easy and stable. Here are dependencies for postgresql and snowflake

I will use the open source dashboarding tool from https://streamlit.io/

poetry init
poetry add streamlit snowflake-connector-python sqlalchemy

pyproject.toml:

[tool.poetry]
name = "dashboard"
version = "0.1.0"
description = ""
authors = ["gudata <i.bardarov@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
streamlit = "^1.33.0"
psycopg2 = "^2.9.9"
snowflake-connector-python = "^3.9.0"
sqlalchemy = "^2.0.29"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Then we define our connections into .streamlit/secrets.toml

[connections.snowflake-admin]
account = "mysnowflakeaccount.us-east-1"
user = "mysnowflakeuser"
password = "thepassword"
database = "theTenantsDatabase"
schema = "theSchema"

[connections.airflow]
dialect = "psycopg2"
url = "postgresql://root:mypassword@myhost-us-east-1.rds.amazonaws.com/airflow_us_ea
st_1"

And we are almost there… Creating the and dashboard is like:

import streamlit as st

snowflake_connection = st.connection("snowflake-admin", type="snowflake")
postgres_connection = st.connection("airflow", type="sql")

Then querying and displaying a chart/table is very easy:

import pandas as pd

def all_dags():
    sql_query = """SELECT dag.dag_id
FROM dag
WHERE is_paused = false
    AND is_active = true
GROUP BY dag.dag_id
"""
    df = postgres_connection.query(sql_query)
    return df

def runs_per_dag(dag_id):
    sql_query = f"""SELECT dag_id,
        ROUND(
            (EXTRACT(EPOCH FROM (end_date - start_date)) / 60)
        , 0) as duration_in_minutes
    FROM dag_run
    WHERE 1=1
        AND state = 'success'
        AND external_trigger=false
        AND dag_id='{dag_id}'
    ORDER BY duration_in_minutes;
    """
    df = postgres_connection.query(sql_query)
    return df

dag_id = st.selectbox("Select one:", all_dags())

st.header(f"Dag times for {dag_id}", divider="rainbow")
st.dataframe(runs_per_dag(dag_id))

The result will be

Then you can continue adding elements to dashboard and finally deploy it with Docker where you want to be visible.

For the deploy I used shipit and it is not the shopify one. That one is 4 years old and is small & fast. You can start deploying now, not after 3 days learning curve. I have no idea why such a precious gem is not popular.

You need only one file

.shipit

host='airflow-us-east.example.com'
path='/home/ec2-user/dashboard'

[deploy:local]
echo "Deploying to: $path"
poetry export --without-hashes --format=requirements.txt > requirements.txt
rsync -ahzv --delete ./ $host:$path/
echo "Open http://192.168.1.4:8501/"


[deploy]
  docker build -t airflow_dashboard .
  docker stop airflow_dashboard || true
  docker rm airflow_dashboard || true
  docker run --name airflow_dashboard -d --env-file .env -p 8501:8501 airflow_dashboard

[status]
uptime

The full dashboard code is on GitHub, awaiting you with a lot of useful reports and charts.

summary of spotted problems
comparing snowflake(other) table with the dag runs for a specific date
failed tasks and displays the number of the retries
Detailed list of the tasks – are they running on the right pool and workers?
Times for the dags, avg, sum, min, max
Finish time of all dags, tasks per day
table view of single dag history

Unlocking the Mysteries of Chmod: A Guide to Understanding Linux Commands”

J April, 2023 / guda

People would likely experience greater happiness if they began learning about computers and became familiar with the meaning of the “chmod” command.

To aid in this pursuit, I have prepared a table containing a list of such commands. It is unclear why computer experts continue to employ abbreviations, though perhaps there are hardware constraints that necessitate their use.

The use of a terminal interface today is not necessarily the same as it was in the early days of computing. Today’s terminals may have larger screens, more powerful hardware, and more efficient input methods (such as keyboards with better tactile feedback and key rollover). Additionally, modern operating systems have graphical user interfaces (GUIs) that provide alternative ways to interact with the system. These factors make it easier for users to type out full command names and make abbreviations less critical for efficiency.

Nevertheless, in modern times, it seems somewhat peculiar to type “mkdir” instead of “new-folder”.

Abbreviated Command	Full Command
cat	concatenate	show
cd	change directory	go
chmod	change mode	set-mode
chown	change owner	set-owner
cp	copy	copy
df	disk free	disk-free
du	disk usage	disk-usage
grep	globally search for a regular expression and print
ls	list	list
mkdir	make directory	new-folder
mv	move	move
ps	process status	process
pwd	print working directory	show-path
rm	remove	delete
rmdir	remove directory	delete-folder
scp	secure copy	secure-copy
ssh	secure shell	secure-shell
sudo	superuser do	root-do
tar	tape archive	archive
top	table of processes	system
unzip	uncompress zip file	decompress
wget	web get	get
zip	compress zip file	compress
sed	stream editor	steam-edit
awk	Aho, Weinberger, and Kernighan

A modern software developer may choose not to use abbreviations when typing out commands in order to avoid confusion and ensure clarity. Typing out full command names can make it easier to understand what the command does and can help avoid mistakes that might occur when using abbreviations. Additionally, modern development environments and IDEs often provide tools such as auto-complete that make it easy to type out full command names quickly. Using full command names can also make it easier to collaborate with other developers and to share code with a wider audience. Ultimately, whether to use abbreviations or not is a matter of personal preference, but choosing not to use them can help ensure clear communication and minimize errors in software development.

Using oil stone for sharpening

J March, 2023 / guda

how to clean oil sharpening stone

Cleaning an oil sharpening stone is essential to maintain its effectiveness and longevity. Here are the steps you can follow to clean an oil sharpening stone:

First, remove any metal shavings or debris from the surface of the stone with a brush or a dry cloth.
Next, use a mild cleaning solution such as dish soap or a specialized sharpening stone cleaner. Add a few drops of the solution to a bowl of warm water.
Soak the sharpening stone in the water and let it sit for 5-10 minutes.
Scrub the stone gently with a soft-bristled brush or sponge. Be sure to clean both sides of the stone.
Rinse the stone thoroughly with clean water and dry it with a towel.
Let the stone air-dry completely before storing it.
If the stone is particularly dirty or has been used heavily, you can also use a honing oil to clean it. Apply a few drops of oil to the stone, rub it in with a soft cloth, and then wipe off the excess oil with a clean cloth.

By following these steps, you can effectively clean an oil sharpening stone and keep it in good working condition for many years to come.

should I clean it every time I use it?

It is a good practice to clean your oil sharpening stone every time you use it. This will help prevent the accumulation of metal shavings, debris, and oil residues, which can affect the sharpening performance of the stone over time. Additionally, cleaning the stone after each use will make it easier to maintain and prolong its lifespan. It only takes a few minutes to clean the stone, and it’s a simple way to ensure that you’re getting the best possible results from your sharpening tool.

does the water damage the oil sharpening stone?

Oil sharpening stones are designed to be used with oil as a lubricant and cooling agent. While water may be used to clean the stone, prolonged exposure to water can damage an oil sharpening stone. Water can seep into the pores of the stone and displace the oil, which can lead to cracking, warping, or other damage.

That said, if you clean your oil sharpening stone with water as recommended and promptly dry it afterward, you should not encounter any issues with water damage. To be safe, you should avoid soaking the stone in water for an extended period and make sure to dry it thoroughly before storing it. Additionally, if you are using a honing oil with your stone, it is best to clean it with oil rather than water to maintain its lubricating properties.

why people use oil for sharpening historically?

The use of oil for sharpening dates back thousands of years to the time when humans first began to make and use metal tools. In ancient times, natural oils such as olive oil, animal fats, and even whale oil were commonly used as lubricants for sharpening stones.

Oil was used for sharpening for several reasons. First, it helps to reduce friction and heat buildup, which can cause the edge of a blade to overheat and become damaged. The oil also helps to keep the metal particles generated during the sharpening process from clogging the pores of the stone, which can reduce its effectiveness over time. Additionally, oil helps to protect the metal from rust and corrosion by forming a barrier between the metal and the environment.

Today, specialized honing oils are used for sharpening as they provide better lubrication and cooling properties than natural oils. However, the use of oil for sharpening remains a popular and effective method for maintaining sharp edges on metal tools and weapons.

what are the wrong oils for sharpning?

When it comes to sharpening with oil, it is essential to use the right type of oil to achieve the best results. While many oils may seem suitable for use with sharpening stones, some oils can actually damage the stone or the tool you’re sharpening. Here are some examples of oils that should not be used for sharpening:

Motor oil: Motor oil is not suitable for sharpening as it contains additives and contaminants that can clog the pores of the sharpening stone, reducing its effectiveness over time. Additionally, motor oil is harmful to your health and the environment.
Vegetable oil: While vegetable oil may seem like a natural and safe option for sharpening, it can turn rancid over time, leaving a foul smell and taste on your tools. It can also attract bugs and other pests.
Mineral oil: Mineral oil is often used as a lubricant and preservative for cutting boards and wooden utensils, but it is not an ideal choice for sharpening stones. It can cause the stone to become slick and reduce its abrasive properties, making it less effective for sharpening.
Cooking oils: Cooking oils such as olive oil, canola oil, and others are not recommended for sharpening stones as they can become sticky and attract debris and contaminants, which can clog the pores of the stone and reduce its effectiveness.

To ensure the best results when sharpening with oil, it is recommended to use specialized honing oils that are designed specifically for use with sharpening stones. These oils are formulated to provide the right level of lubrication and cooling, and they are less likely to cause damage to the stone or the tools you’re sharpening.

what is the best oils for sharpening, order the list by price

Here are some of the best oils for sharpening stones, listed in order from the most affordable to the most expensive:

Mineral oil: Mineral oil is a popular and affordable option for sharpening stones. It is odorless, colorless, and readily available at most drug stores and hardware stores. It is also non-toxic and safe for use on food preparation surfaces.
Baby oil: Baby oil is another affordable option for sharpening stones. It is made with mineral oil and added fragrance and is widely available in stores.
Vegetable oil: Vegetable oil is a natural and affordable option for sharpening stones. It is readily available in most kitchens and can be a good option if you’re in a pinch.
Honing oil: Honing oil is a specialized oil formulated specifically for use with sharpening stones. It is more expensive than mineral oil or vegetable oil, but it provides better lubrication and cooling, which can result in a sharper edge. Honing oils can range in price depending on the brand and quality.
Camellia oil: Camellia oil is a traditional oil used in Japan for sharpening stones. It is a high-quality oil that provides excellent lubrication and cooling. Camellia oil is more expensive than mineral oil or honing oil, but it is highly recommended for use with high-end sharpening stones.

Ultimately, the best oil for sharpening stones will depend on your personal preferences, budget, and the type of tool you’re sharpening. It is important to choose an oil that provides the right level of lubrication and cooling to achieve the best results.

the best oils for sharpening ordered by quality

Here are some of the best oils for sharpening stones, listed in order from the lowest quality to the highest quality:

Vegetable oil: While vegetable oil is a natural and affordable option for sharpening stones, it is not the best quality oil for this purpose. It can become sticky and attract debris and contaminants, which can clog the pores of the stone and reduce its effectiveness.
Mineral oil: Mineral oil is a popular and affordable option for sharpening stones, but it is not as high quality as other options. It can cause the stone to become slick and reduce its abrasive properties, making it less effective for sharpening.
Baby oil: Baby oil is made with mineral oil and added fragrance, but it is not as high quality as other options. It is a convenient and affordable option, but it may not provide the same level of lubrication and cooling as specialized honing oils.
Honing oil: Honing oil is a specialized oil formulated specifically for use with sharpening stones. It provides better lubrication and cooling than mineral oil or vegetable oil, which can result in a sharper edge. Honing oils can range in quality depending on the brand and formulation.
Camellia oil: Camellia oil is a traditional oil used in Japan for sharpening stones. It is a high-quality oil that provides excellent lubrication and cooling. Camellia oil is more expensive than mineral oil or honing oil, but it is highly recommended for use with high-end sharpening stones.

how to make honing oil?

While it’s possible to make your own honing oil, it can be challenging to get the right formulation for optimal sharpening results. Here is a simple recipe for making honing oil at home:

Ingredients:

1 part mineral oil
1 part kerosene

Instructions:

Combine the mineral oil and kerosene in a clean container with a lid.
Close the lid tightly and shake the container to mix the ingredients thoroughly.
Store the honing oil in a clean, dry place, away from direct sunlight and heat.

Please note that kerosene is flammable and potentially hazardous, so use caution when working with it. Additionally, homemade honing oils may not provide the same level of performance as commercially available honing oils. If you’re unsure about making your own honing oil, it’s recommended to purchase a specialized honing oil from a reputable supplier.

how is made Camellia oil ?

Camellia oil is made from the seeds of the Camellia sinensis plant, which is native to China, Japan, and other parts of Asia. The oil is also known as tea seed oil, as the plant is the same one used to produce tea leaves.

The process of making camellia oil involves several steps:

Harvesting: The Camellia sinensis plant is harvested when the seed pods are ripe and ready to be collected.
Drying: The seed pods are spread out and left to dry in the sun. Once they are completely dry, they are cracked open and the seeds are removed.
Roasting: The seeds are then roasted to bring out their flavor and aroma. This step is important for producing high-quality camellia oil.
Pressing: The roasted seeds are then pressed to extract the oil. The oil is typically cold-pressed, which means that it is extracted without the use of heat or chemicals, preserving its natural properties.
Filtering: The extracted oil is then filtered to remove any impurities or sediment.

The resulting oil is a light yellow color and has a mild, nutty flavor. It is high in monounsaturated and polyunsaturated fats, making it a healthy cooking oil. Camellia oil is also prized for its use in traditional Japanese woodworking and sharpening, as it provides excellent lubrication and protection for tools and sharpening stones.

how much oil to put?

The amount of oil to use when sharpening with an oil stone can vary depending on the size of the stone and the type of oil being used. As a general rule of thumb, you want to use enough oil to create a thin, even film over the surface of the stone.
For smaller stones, such as pocket stones or small bench stones, you may only need a few drops of oil to cover the entire surface. For larger stones, such as bench stones or honing stones, you may need to use several tablespoons of oil to create a sufficient film.
It’s important not to use too much oil, as this can lead to a messy and inefficient sharpening process. If you notice excess oil on the surface of the stone, you can blot it with a paper towel or cloth to remove the excess and create a more even film.

Can I use PTFE ?

PTFE, also known as Teflon, is a synthetic polymer that is commonly used as a non-stick coating on cookware and other products. While it is possible to use PTFE as a lubricant for sharpening tools, it is generally not recommended.
One issue with using PTFE is that it can be difficult to remove from the surface of the tool after sharpening, which can leave a residue that affects the performance of the tool. Additionally, PTFE is not formulated specifically for sharpening applications, and may not provide the same level of lubrication and protection as a specialized honing oil.
If you’re looking for a lubricant for sharpening tools, it’s best to use a product that is specifically designed for that purpose, such as honing oil or mineral oil. These products have been formulated to provide the right level of lubrication and protection for sharpening stones and tools, and are less likely to leave residue on the tool after sharpening.

For what people use wd40 on oil stoneS?

WD-40 is a multi-purpose lubricant and cleaner that can be used for a wide range of applications. However, it is not recommended for use on oil stones for sharpening tools.
While WD-40 can provide some lubrication and protection for sharpening stones and tools, it is not formulated specifically for this purpose. Additionally, WD-40 contains solvents that can dissolve and remove the oil from the oil stone, which can affect its performance and lifespan.
Some people may use WD-40 on oil stones for cleaning purposes, as it can help remove dirt and grime from the surface of the stone. However, it’s generally better to use a specialized stone cleaner for this purpose, as these products are formulated specifically for use with sharpening stones and will not damage the stone’s surface or affect its performance.
If you’re looking for a lubricant or cleaner for your oil stones, it’s best to use a product that is specifically designed for use with sharpening stones and tools, such as honing oil or specialized stone cleaners.

How to check if a stone is water or oil ?

It can sometimes be difficult to tell whether a sharpening stone is intended to be used with water or oil, especially if it has been used before and there are no markings or labels indicating its use. However, there are a few things you can look for that can help you determine whether a stone is intended to be used with water or oil:
Color: Some manufacturers use different colors to indicate whether a stone is intended for use with water or oil. For example, oil stones are often brown, while water stones are often light gray or white. However, this is not always a reliable indicator, as some stones may be the same color regardless of their intended use.
Porosity: Water stones are typically more porous than oil stones, which allows them to absorb water and release it slowly over time. You can check the porosity of a stone by placing a few drops of water or oil on the surface and seeing how quickly it is absorbed. If the stone absorbs the water quickly and appears to be porous, it is likely a water stone. If the oil sits on the surface without being absorbed, it is likely an oil stone.
Manufacturer’s instructions: The manufacturer of the stone may provide instructions or labeling indicating whether the stone is intended for use with water or oil. If you still have the packaging or instructions that came with the stone, this may be a good place to check.
If you’re still unsure whether a stone is intended for use with water or oil, it’s generally better to err on the side of caution and use water until you can confirm its intended use. Using the wrong type of lubricant can affect the performance of the stone and your ability to sharpen your tools effectively.

What is truing stone? Can be used with oil stone?

A truing stone, also known as a flattening stone or lapping stone, is a tool used to flatten and maintain the surface of sharpening stones. Over time, sharpening stones can become uneven or develop grooves or hollows, which can affect their performance and the quality of the sharpened edge. Truing stones are used to restore the flatness of the sharpening stone surface and ensure that it provides consistent and accurate results.
Truing stones can be used with both water stones and oil stones, depending on the type of stone and the manufacturer’s instructions. Some truing stones are designed specifically for use with water stones, while others are designed for use with oil stones. It’s important to follow the manufacturer’s instructions and use the correct type of truing stone for your sharpening stone.
When using a truing stone, it’s important to apply light pressure and work evenly across the entire surface of the sharpening stone. This will help ensure that the surface is flattened evenly and does not develop new grooves or hollows during the truing process. It’s also important to clean the sharpening stone thoroughly after using a truing stone, as any abrasive particles or debris left on the surface can affect the quality of the sharpened edge.

What should I be careful of when sharpening with oil stone?

When sharpening with an oil stone, there are a few things to be careful of in order to ensure that you get the best results and avoid damaging your tools or the stone. Here are some things to keep in mind:
Use the right oil: It’s important to use the right type of oil for your stone, as using the wrong type can affect the stone’s performance and lifespan. Some oils can also be flammable or emit harmful fumes, so it’s important to follow the manufacturer’s instructions and use the recommended type of oil.
Keep the stone clean: As you sharpen your tools, metal particles and debris can build up on the surface of the stone. This can affect the stone’s ability to sharpen and lead to scratches on your tools. It’s important to clean the stone regularly with a stone cleaner or mild soap and water to remove any debris.
Use consistent pressure and angle: When sharpening your tools, it’s important to use a consistent amount of pressure and maintain a consistent angle throughout the sharpening process. This will help ensure that your tools are sharpened evenly and that you don’t accidentally damage the stone.
Use a honing guide: A honing guide can help you maintain a consistent angle when sharpening your tools. This can be especially useful if you’re new to sharpening or if you’re working with a particularly delicate tool.
Avoid overheating: When sharpening with an oil stone, it’s important to avoid overheating the metal of your tools. This can cause the metal to lose its temper and become softer, which can make it less effective and more prone to damage. To avoid overheating, it’s important to use light pressure and take breaks between sharpening sessions to allow the metal to cool down.
By following these tips, you can help ensure that you get the best possible results when sharpening with an oil stone and avoid damaging your tools or the stone.

What is the best liquid which can clean the stone?

The best liquid for cleaning an oil stone depends on the type of oil stone and the type of debris or build-up you need to remove. However, there are a few commonly used options that work well for most oil stones:
Honing oil: Honing oil is specifically designed for use with sharpening stones and can help dissolve and remove metal particles and debris from the surface of the stone. It can also help protect the stone and prevent it from drying out.
Mineral spirits: Mineral spirits can be used to dissolve and remove oil-based contaminants from the surface of the stone. It’s important to use this with caution and in a well-ventilated area, as it can be flammable and emit harmful fumes.
Simple Green: Simple Green is a non-toxic cleaner that can be used to clean oil stones. It’s effective at removing built-up grime and dirt without damaging the stone.
Dish soap and water: A mixture of dish soap and water can be used to clean oil stones, especially if the stone is heavily soiled or has a lot of built-up debris. Simply mix a few drops of dish soap into warm water and use a soft brush or cloth to clean the surface of the stone.
It’s important to read the manufacturer’s instructions for your specific oil stone and use the recommended cleaning method. Avoid using harsh chemicals or abrasives that can damage the stone.

I don’t know what a “Simple Green” is

Simple Green is a brand of non-toxic, biodegradable, and multi-purpose cleaner that is commonly used for cleaning various surfaces, including oil stones. It is a water-based cleaning solution that is designed to be safe and effective for a wide range of cleaning applications. It is available in most hardware stores and online retailers. If Simple Green is not available in your area, you can look for other non-toxic, biodegradable, and multi-purpose cleaners that are suitable for cleaning oil stones. Just be sure to read the label to make sure the cleaner is safe to use on your specific type of oil stone.
What can I use in Europe instead of Simple Green?
If Simple Green is not available in your area, there are other alternatives that you can use to clean your oil stone in Europe. Here are a few options:
Dish soap and water: This is a simple and effective solution for cleaning oil stones. Mix a few drops of dish soap with warm water, apply the solution to the surface of the stone, and scrub with a soft-bristled brush.
Isopropyl alcohol: Isopropyl alcohol, also known as rubbing alcohol, can be used to clean oil stones. Apply the alcohol to a cloth or brush and scrub the surface of the stone.
White vinegar: White vinegar is an effective cleaning solution that can be used to clean oil stones. Mix equal parts white vinegar and water, apply the solution to the surface of the stone, and scrub with a soft-bristled brush.
Honing oil: Honing oil can be used to clean oil stones as well as to lubricate them during sharpening. Apply the oil to the surface of the stone and scrub with a soft-bristled brush.
It’s important to use caution when using any cleaning solution on your oil stone and to follow the manufacturer’s instructions for your specific type of stone. Avoid using harsh chemicals or abrasives that can damage the stone.

Good bye Tomboy

J January, 2023 / guda

Tomboy note taking program was working well for years.

It is time to convert my notes to markdown.

Here is how I did it.

Developer jokes

J October, 2022 / guda

Sketchup Under Virtual Box

J August, 2022 / guda

If you get the error message “graphics card’s OpenGL version is 2.1. SketchUp requires a graphics card that supports OpenGL 3.1 or better”

Then probably you can fix it by downloading one dll and putting it in the folder of the sketchup executable.

download dll from here: https://fdossena.com/?p=mesa/index.frag

Extract opengl32.dll from the downloaded file. Save it in the same folder as the .exe you use to start the program that needs OpenGL 3 support.

Run the program to see if the problem is solved.

The solution worked great for Sketchup.

The original post is here

Writing good error messages

J November, 2021 / guda

I am amazed how big companies like AWS with a lot of resources can deliver such error messages.

Using Apache Airflow’s Docker Operator with Amazon’s Container Repository

J October, 2021 / guda

Here is a python version of getting the ECR tokens for an AWS repository.

There is nothing to install and everyting runs smootly in from the airflow docker containers.

To make the script reusable, you need to create a variable called “aws_region_name” and set it to the correct region, for example “eu-central-1”

"""
You need to create a variable called "aws_region_name" and set it to the correct region, for example "eu-central-1"
"""
from datetime import datetime, timedelta

import json
from datetime import datetime

from airflow.decorators import dag, task

from airflow import settings
from airflow.models import Connection

#Default settings applied to all tasks
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 0,
    'retry_delay': timedelta(minutes=1)
}

@dag(default_args=default_args, schedule_interval='* */10 * * *', max_active_runs=1, start_date=datetime(2021, 1, 1), catchup=False, tags=['airflow'])
def refresh_docker_token_DAG():

    @task(multiple_outputs=True)
    def extract():
        import boto3
        from airflow.models import Variable
        aws_region_name = Variable.get("aws_region_name")
        ecr = boto3.client('ecr', region_name=aws_region_name)
        response = ecr.get_authorization_token()
        token = response['authorizationData'][0]['authorizationToken']
        registry_url = response['authorizationData'][0]['proxyEndpoint']

        return {"token": token, "registry_url": registry_url}



    @task()
    def set_token(token: str, registry_url: str):
        import logging
        import base64
        logger = logging.getLogger(__name__)

        connection_name = "docker_default"
        conn_type = "docker"
        host = registry_url
        port = None
        user = base64.b64decode(token).decode().split(":")[0]
        password = base64.b64decode(token).decode().split(":")[1]
        schema = ""
        extra = ""

        session = settings.Session
        try:
            connection_query = session.query(Connection).filter(Connection.conn_id == connection_name)
            connection_query_result = connection_query.one_or_none()
            if not connection_query_result:
                connection = Connection(conn_id=connection_name, conn_type=conn_type, host=host, port=port,
                                        login=user, password=password, schema=schema, extra=extra)
                session.add(connection)
                session.commit()
            else:
                connection_query_result.host = host
                connection_query_result.login = user
                connection_query_result.schema = schema
                connection_query_result.port = port
                connection_query_result.extra = extra
                connection_query_result.set_password(password)
                session.add(connection_query_result)
                session.commit()

        except Exception as e:
            logger.info("Failed creating connection")
            logger.info(e)

    data = extract()
    set_token(data["token"], data["registry_url"])

refresh_docker_token_dag = refresh_docker_token_DAG()

References

The first article which I read about, includes an idea of how to clean the images.
Using the ECR Airflow plugin

jq

J July, 2021 / guda

Pipe-ing

jq works on set of filters. Each filtering step produce a result which can be den further filtered

json='{"person": {"name": "Ivo", "phone": "123"}}'
echo $json | jq .person.name
"Ivo"

is the same as

echo $json | jq ".person | .name"
"Ivo"

because the first | will produce {name: ‘Ivo’} which will be then filtered.

Extract multiple fields

This is done by enumerating the fields with ,

Just do

echo $json | jq ".person | (.name,.phone)"
"Ivo"
"123"

But if you want to concatente them add echo $json | jq “.person | {name_with_phone:(.name + “-” + .phone)}” { “name_with_phone”: “Ivo-123” }

..and finally extract only the name_with_phone

echo $json | jq ".person | {name_with_phone:(.name + \"-\" + .phone)} | .name_with_phone" 
"Ivo-123"

..and to get only the value add -r echo $json | jq -r “.person | {name_with_phone:(.name + “-” + .phone)} | .name_with_phone” Ivo-123

Here is an example where I grab all the location ids from all parents

cat * | jq '.deviceLocationUpdate.location | .locationId,.parent.locationId,.parent.parent.locationId,.parent.parent.parent.locationId,.parent.parent.parent.parent.locationId' | sort | uniq

Calisthenic in programming