Drafts

There are various ways to configure python tools, config files, code, or environment variables. Let’s look at a few projects that allow users to configure them through the use of config files and how they do it. Motivation # [1] This will not include how they are implemented, I’ve looked at a few and its not simple. This will focus on where config is placed and the order in which duplicates are resolved. The motivation of this article is to serve as a bit of a reference guide for those who may want to create their own package that needs configuration. Flake8 # [2] Global # [3] User settings can exist in the users ~/.config/flake8 file to configure how flake8 runs on their machine. - ~/.config/flake8 Per-Project # [4] Only One project config file will be considered, but allows for several options. These files all use the ini format and must have a [flake8] section header to be consideered. Selection of the config file can also be overridden by the --config cli option. An e...

Docs # [1] https://kedro.readthedocs.io/en/stable/04_kedro_project_setup/01_dependencies.html?highlight=install pip-tools # [2] pip-compile # [3] requirements # [4] - requirements.in - requirements.txt References: [1]: #docs [2]: #pip-tools [3]: #pip-compile [4]: #requirements

Generating an api for a blog is much simpler than one might expect with python. Markdown # [1] Frontmatter # [2] Fill in the blanks # [3] fix missing data Fast # [4] References: [1]: #markdown [2]: #frontmatter [3]: #fill-in-the-blanks [4]: #fast

Overrated underrated command line applications - Ship More - Ship Less - get feedback and apply feedback, build the right things kedro YouTube series # [1] Get started on daily kedro shorts, these are indended to be short clips that people can watch a playlist and learn about kedro concepts at their own pace. This is meant to be low barrier to entry for me to create. Like the tmux series, I hope to make a sub three minute video within one or two takes, no edits, all straight from obs. written # [2] - how I kedro 8/16 - what is kedro 8/17 - kedro new 8/18 - three ways to create a kedro environment 8/19 - git [3] init 8/20 - kedro install 8/21 - kedro pipeline create 8/22 staging # [4] - making your first nodes in kedro 8/23 - kedro run 8/24 - kedro catalog create 8/25 - add new dependencies 8/26 - listing things from the kedro cli 8/27 - Storing catalog entries 8/30 - Comprehensive nodes - modular piplines 8/29 - kedro pipeline package - kedro pipeline pull - ...

- ?docstring - ??sourcecode - %run - %debug - %autoreload - %history - autoformat - %reset - !shell commands ?docstring # [1] Stop going to google everytime your stuck and stay in your workflow. The ipython ? is a superhero for productivity and staying on task. from kedro.pipeline import Pipeline Pipeline? Init signature: Pipeline( nodes: Iterable[Union[kedro.pipeline.node.Node, ForwardRef('Pipeline')]], *, tags: Union[str, Iterable[str]] = None, ) Docstring: A ``Pipeline`` defined as a collection of ``Node`` objects. This class treats nodes as part of a graph representation and provides inputs, outputs and execution order. Init docstring: Initialise ``Pipeline`` with a list of ``Node`` instances. Args: nodes: The iterable of nodes the ``Pipeline`` will be made of. If you provide pipelines among the list of nodes, those pipelines will be expanded and all their nodes will become part of this new pipeline. tags: Optional set of tags to be applied to all the pipeli...

Lint/Format/Doc - black - flake8 - interrogate - mypy Pipeline Assertions - pipeline constructs - pipeline as expected nodes - pipeline has minimum nodes - test minimum tags - test alternate tags Catalog Assertions - test catalog follows naming structure - Node Tests - test function does the correct operations on test data Great Expectations

Dynamically generate kedro pipelines with yaml or script Inspiration - dag-factory [1] References: [1]: https://github.com/ajbosco/dag-factory

- simple landing page - https://swyx.io - joel on software [1] - recent - reading lists - More from waylon just above footer - 4x2 grid - link strategy - latest post - next/prev - similar tags - search in nav - tag stickers - simple cards? - bookmarks? - nav style stinks - single post template - flat routes no need to /blog /notes - post types - 🌳 full - 🌱 budding - 🖊 Note - 💻 hot tip - usage of tags - MDX - stories - slides - ⚠ - ❌ - ✔ - kedro viz - charts - inlink component - https://joshwcomeau.com/ - auto-card oneline links - meta posts - about - uses - how site is built - how to search - stories TODO # [2] - review package.json - update package.json Done # [3] - ahrefs - fix canonical urls - fix broken inlinks - convert to one post template - References: [1]: https://www.joelonsoftware.com/ [2]: #todo [3]: #done

Nesting loops inside of each other in python makes for much harder code to understand, it takes more brain power to understand, and is thus more error prone than if its avoidable. One issue with this complexity is that toy examples may make sense, but most real example will grow and become more deeply nested over time. Avoiding this complexity from the start can help simplify the project in the future. setup # [1] Lets take a pretty simple example where we are using a ficticious library to get some sales data for our transportation company. The api allows us to fetch teh sales data for one class of vehicle and one region at a time. import pandas as pd from datastore import get_sales # ficticious library cars = ['sedan', 'coupe', 'hatchback'] regions = ['US', 'CA', 'MX'] ❌ Nesting Loops # [2] We have setup to fetch our data with two lists that represent the vehicles and regions that we want to analyze. We know that we need to make a call to get_sales for every vehicle and regio...

while read file; do echo $(git log --pretty=format:%ad -n 1 --date=raw -- $file) $file; done < <(git ls-tree -r --name-only HEAD | grep static/stories) | sort -r | head -n 3 | cut -d " " -f 3

Learn Kedro in 5 days Day 0 Setup # [1] - vm - install - python - editor Day 1 # [2] - kedro new - kedro viz Day 2 # [3] - catalog - filter catalog - load data - fsspec Day 3 # [4] - pipeline - nodes Day 4 # [5] - filter pipeline - run partial pipeline Day 5 # [6] - kedro docker - GitHub Actions Advanced Kedro # [7] - hooks - custom datasets - modular pipelines References: [1]: #day-0-setup [2]: #day-1 [3]: #day-2 [4]: #day-3 [5]: #day-4 [6]: #day-5 [7]: #advanced-kedro

Upcoming posts to the blog. Have an idea edit this post [1] , submit a PR and we will talk. 🧠 # [2] - Kedro run changed - How I manage Environments - My Data Workflow. - Daily Schedule - desk - keeb - Material Shell - Why blog - search with fuse.js - Testing a blog with ahrefs - matrix testing in github actions - Think like a Senior Dev - Editor # [3] - tmux - vim - shortcuts - gitui - fzf - devinsideyou [4] Core # [5] - gracefully adopt kedro - catalog - in progress - pipeline - kedro - Silent Logger - Templated config loader - params/env - 10 reasons you shouldnt use kedro - 10 reasons to use - filter viz - Steel-toes env - Why framework - How I write pipelines - when I write pipelines - pipeline node templates - Convert a notebooks to pipelines - Testing Pipelines - professional python - cookiecutter - flake8 - black - mypy - pre-commit - click - pytest - git [6] - parametrize - environment variables - My top pandas methods - Ac...