Posts tagged: python

When learning a new skill it’s important to practice along the way. In order for me to show up to practice I need to make it easy to show up. An easy way to show up to practice with python is to use an online repl. With these you can try out something quick. Sometimes I see snippets from blogs or tweets and I need to try the out for myself to really understand.

When learning a new skill it’s important to practice along the way. In order for me to show up to practice I need to make it easy to show up. An easy way to show up to practice with python is to use an online repl. With these, you can try out something quick. Sometimes I see snippets from blogs or tweets and I need to try them out for myself to really understand.

Here are three different options that I have used in the past to try out something at some various levels. I am sure there are plenty more, but these are three that I have tried. I am not covering all of them, because It’s been a while since I have used one other...

...

I am exploring a kedro catalog meta data hook, these are some notes about what I am thinking.

try pandas method -> try spark -> try dict/list -> none

Is there an easy way to create a nosql database in memory from a a list of dictionaries?

Mypy’s config parser seems to be one of the most complex. This is likely in part to it having the largest backwards compatability of all projects that I looked at.

mypy/config_parser

options/config.py

...

I am looking into a way to replace my google reader experience that I had back in 2013 before google took it from us. I am starting by learning how to parse feeds with python, and without much previous knowledge, it proved to be much easier than anticipated thanks to the feedparser library.

This is how I used python to parse rss and setup my own custom feed.

Install the feedparser library.

...

Python *args and **kwargs are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong.

I generally post these as a carousel on LinkedIn based on a full article. Let mw know what you think of it shown inside of a blog @_waylonwalker.

...

While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont even touch read/write steps at all.

Taking a step into kedro by adopting the catalog first will give you a way to organize all of your data loads in one place, and stop manually writing read/write code, which can be different for each data and storage type. You just don’t need to think about it.

“can be dropped into kedro later” Let’s talk a bit more about that

...

kedro 0.16.2 just dropped last week with a long-awaited feature… catalog search! I went as far as monkey patching this into each of my projects. I work jump between a few really big projects that have tons of datasets. Being able to quickly search for what I need is so useful.

The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections listed in one place make it so easy to pick your project up and move it to a completely new environment. That sweet imperative loading style saves so much read/write overhead. I can load all my data with a single command whether it’s in amazon s3, google cloud platform, or a local file.

Just like with most of these articles, I am going to create a conda environment so that I don’t break any existing projects and scaffold up a toy project to learn from.

...

Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing.

Check out this post for a review of how *args **kwargs work in python.

understanding python *args and **kwargs

...

Python *args and **kwargs are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong.

Python *args and **kwargs are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong.

*args are some magical syntax that will collect function arguments into a list, or unpack a list into individual arguments.

...

I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but really it’s easy. It will automatically run checks for you. In some cases, it will even automatically fix them for you. Out of the box, it will do things like automatically trim extra whitespace, fix file endings, and ensure file sizes are not too large for git.

I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but really it’s easy. It will automatically run checks for you. In some cases, it will even automatically fix them for you. Out of the box, it will do things like automatically trim extra whitespace, fix file...

...