Posts tagged: data Archive

💭 Taming file zoos: Data science with DuckDB database files - Al...

2025-06-09T13:31:30Z

The ability to query s3 buckets so seamless looks like such a pleasure to work with if you have a use case for that. Kedro catalog takes care of this most of the time for me, but I wonder if there are some cross project searching use cases I might find for this.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 hotel_bookings.csv

2025-03-15T22:32:44Z

hotel_bookings.csv

Discover what actually works in AI. Join millions of builders, researchers, and labs evaluating agents, models, and frontier technology through crowdsourced benchmarks, competitions, and hackathons.

kaggle.com

nice dataset to use for example / test projects. I’m using it to play with duckdb currently.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

sqlite vacuum

2024-04-16T21:31:24Z

Today I learned how to VACUUM a sqlite database and cut its size in about half. It’s a database that I have had running for quite awhile and has some decent traffic on it.

Why is it important to do a VACUUM? In short its becuase the file system gets fragmented with as data is updated. On delete the files are removed from the database and marked as available for reuse in the filesystem, but the space is not reclaimed.

To VACUUM a database, run the following sql command. You can do it right form the sqlite shell by running sqlite3.

You will need about double the current size of the database as free space to do the VACUUM, you need space for a full copy, journaling or write ahead logs, and the existing database.

VACUUM;

The docs are fantastic for vacuum.

💭 searching my thoughts locally

2024-03-07T23:42:00Z

![[None]]

First I need to fetch my thoughts from the api, and put it in a local sqlite database using sqlite-utils.

fthoughts () {
    # fetch thoughts
    curl 'https://thoughts.waylonwalker.com/posts/waylonwalker/?page_size=9999999999' | sqlite-utils insert ~/.config/thoughts/database2.db post --pk=id --alter --ignore -
}

Now that I have my posts in a local sqlite database I can use sqlite-utils to enable full text search and populate the full text search on the post table using the title message and tags columns as search.

sthoughts () {
    # search thoughts
    # sqlite-utils enable-fts ~/.config/thoughts/database2.db post title message tags
    # sqlite-utils populate-fts ~/.config/thoughts/database2.db post title message tags
    sqlite-utils search ~/.config/thoughts/database2.db post "$*" | ~/git/thoughts/format_thought.py | bat --style=plain --color=always --language=markdown
}

alias st=sthoughts

Now I am ready to search my thoughts, which is a tiny blog format that I created mostly for leaving my own personal comment on web pages, so most of them have a link to some other online content, and their title is based on the authors title.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Open source, not open contribution with Ben Johnson (Changelog...

2023-10-05T01:11:19Z

Open source, not open contribution with Ben Johnson (Changelog Interviews #433)

This week we're talking with Ben Johnson. Ben is known for his work on BoltDB, his work in open source, and as a freelance Go developer. Late January when Ben open sourced his newest project Litest...

Changelog · changelog.com

Ben Johnson was on the Changelog a few years back covering his work on litestream, and talks about why he chose to go open source, but not open contribution.

You should have a good reason to move off of sqlite.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 DjangoCon Europe 2023 | Use SQLite in production - YouTube

2023-10-03T13:22:19Z

Very inspiring talk, TLDR, you probably don’t need a database server. sqlite will probably be faster, simpler to maintain, and simpler to test your application.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Set up minio bucket entrypoint

2023-10-01T20:11:51Z

I recently se tup minio object storage in my homelab for litestream sqlite backups. The litestream quickstart made it easy to get everything up and running on localhost, but I hit a wall when dns was involved to pull it from a different machine.

Here is what I got to work #

First I had to configure the Key ID and Secret Access Key generated in the minio ui.

❯ aws configure
AWS Access Key ID [****************VZnD]:
AWS Secret Access Key [****************xAm8]:
Default region name [us-east-1]:
Default output format [None]:

Then set the the s3 signature_version to s3v4.

aws configure set default.s3.signature_version s3v4

Now when I have minio running on https://my-minio-endpoint.com I can use the aws cli to access the bucket.

Note that https://my-minio-endpoint.com resolves to the bucket endpoint (default 9000) not the ui (default 9001).

aws --endpoint-url https://my-minio-endpoint.com s3 ls my_bucket

Now Configuring Litestream #

Litestream also accepts the endpoint argument via config. I could not get it to work just with the ui.

Note the aws configure step above is not required for litestream, only the aws cli.

dbs:
  - path: /path/to/database.db
    replicas:
      - url: s3://my_bucket/
        endpoint: https://my-minio-endpoint.com
        region: us-east-1
        access-key-id: ****************VZnD
        secret-access-key: ************************************xAm8

Now run a litestream replication.

litestream replicate -config litestream.yml
# or put the config in /etc/litestream.yml and just run replicate
litestream replicate

💭 benbjohnson/litestream: Streaming replication for SQLite.

2023-10-01T02:19:18Z

GitHub - benbjohnson/litestream: Streaming replication for SQLite.

Streaming replication for SQLite. Contribute to benbjohnson/litestream development by creating an account on GitHub.

GitHub · github.com

    `litestream` is a sick cli tool for steaming replicas of sqlite.  It automatically does daily snapshots, and streams all of the writes to the replica live.

install #

Install is fast using installer, no compilation, just copy the binary and run.

curl https://i.wayl.one/benbjohnson/litestream

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

why-is-postgres-default

2023-09-30T21:26:36Z

Serious question.

No one ever got fired for choosing PostgreSQL #

But, why. It’s the most loved db, right? Right? Maybe it’s time to rethink it.

Don’t get me wrong, if I need a relational db as a service, PostgreSQL is going to be my first choice, but why do I need to run a separate application for it?

Tutorials use sqlite #

Why is that? Because there is nothing else to stand up. Nothing else to maintain. And you probably already have it installed on just about anything that has a battery.

SQLite runs in memory #

Don’t need, or maybe don’t want to persist state. Run it in memory. This is a nice feature for running tests.

Less exposure #

SQLite is a file on your filesystem. It’s not a web service. It’s not a cloud service. Not that postgres is insecure, but it is one more endpoint that you have to think about securing.

this means that is probably also cheaper 🤑

SQLite is easy to replicate #

Want to run your new feature with prod data? Pull a replica or backup. It’s a file, you can cp, scp, rsync it whatever you have available.

SQLite Does everything of what I need in an application db #

SQLite has a rock solid set of features that covers everything that I need, and when it’s not I am probably thinking about pulling data into my application code and running something custom anyways.

SQLite runs on the edge #

Most applications are read heavy and light on writes. Services like turso have recognized this and built a business around it. They give you a master db that all writes go to, and read replicas that can run on the edge with your application code. You have a lot of users in different regions you can run your application close to them without suffering from query latency or replication complexity.

SQLite probably does what you need it to #

For a large number of use cases SQLite is probably the best thing to get you off the ground quick, and likely all that you will need. Consider using it before defaulting to postgres.

You probably already know how to maintain it #

If you know a bit of cli commands and remember that your database is just a file, this will feel very intuitive. If not you can probably poke around a file system gui and still make most of it work.

## create a database
touch database.db
## back it up
cp database.db database.db.bak
## drop the database
rm database.db
## restore the backup
cp database.db.bak database.db
## drop the backup
rm database.db.bak
## create an offsite backup
aws s3 cp database.db s3://my-backup-bucket/
scp database.db username@to_host:/remote/directory/
rsync -a database.db username@to_host:/remote/directory/

Need High Concurrent writes #

You might consider postgres.

Need Remote connection string #

You might consider postgres.

💭 Why I Built Litestream - Litestream

2023-09-30T16:16:47Z

Why I Built Litestream - Litestream

Despite an exponential increase in computing power, our applications require more machines than ever because of architectural decisions made 25 years ago. You can eliminate much of your complexity ...

litestream.io

As applications scale to the edge, to put compute as close to the user as possible, database queries back to the master node get slower and slower. Enter sqlite replication, put the database wtih the application code and replicate from master.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 I'm All-In on Server-Side SQLite · The Fly Blog

2023-09-30T14:54:33Z

I'm All-In on Server-Side SQLite

Ben Johnson has joined Fly.io

Fly · fly.io

SQLite is the next big database trend. with more horizontal scaling, close to user read heavy applications, having your database in the same application stack makes a lot of sense. Tools like litestream are going to enable global distribution in an impressive way.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 LiteFS Cloud: Distributed SQLite with Managed Backups · The Fl...

2023-09-30T14:49:41Z

LiteFS Cloud: Distributed SQLite with Managed Backups

Documentation and guides from the team at Fly.io.

Fly · fly.io

Fly.io’s solution to sqlite managed backups.I definitely want to look into this a bit, but moreso the tech under the hook litestream.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 SQLite FTS5 Extension

2023-08-21T13:33:24Z

sqlite has 3 different tokenizers, porter, ascii, trigram.

These can be used with sqlite-utils.

sqlite-utils enable-fts --tokenize porter database.db post title message tags

And with the python api.

db = Database('database.db')
db["post"].enable_fts(
                ["title", "message", "tags"], create_triggers=True, tokenize="trigram"
            )
posts = list(db["post"].search(search))

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 simonw/datasette-render-markdown: Datasette plugin for renderi...

2023-08-16T00:48:07Z

GitHub - simonw/datasette-render-markdown: Datasette plugin for rendering Markdown

Datasette plugin for rendering Markdown. Contribute to simonw/datasette-render-markdown development by creating an account on GitHub.

GitHub · github.com

datasette really does everything doesn’t it!

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 `ValueError: Constraint must have a name` in alembic 1.10.0 · ...

2023-08-05T01:25:07Z

`ValueError: Constraint must have a name` in alembic 1.10.0 · Issue #1195 · sqlalchemy/alembic

Describe the bug ValueError: Constraint must have a name in alembic 1.10.0. Expected behavior Migration succeeds. To Reproduce Please try to provide a Minimal, Complete, and Verifiable example, wit...

GitHub · github.com

After a nasty time with alembic upgrades, thoughts is about to get a new users table. This may have came from incorrectly setting up alembic for sqlite from the start, but I was able to fix the issue with this GitHub issue.

alembic sqlite ValueError: Constraint must have a name

The change I needed to make to get my migration to run.


        + batch_op.create_foreign_key('fk_post_author_id_user', 'user', ['author_id'], ['id'])

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Use Alembic Check to check for possible upgrades

2023-08-05T01:22:06Z

![[None]]

Since using alembic I have been just running out a new revision checking its content and deleting it if its empty, today I learned there is an alembic check command to check for operations that need to be created.

❯ alembic check
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
No new upgrade operations detected.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 DuckDB vs. MotherDuck — should you switch to the cloud version...

2023-07-31T22:36:12Z

DuckDB vs. MotherDuck — should you switch to the cloud version? | Kestra

Why and when to use MotherDuck over local DuckDB

kestra.io

duckdb is a new in process database that has been making its rounds in analytics for its high performance in those applications.

Mother duck is a centeralized server that brings manages storage, data sharing and an ide to duckdb.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 s3-tree · PyPI

2023-07-30T14:37:06Z

Client Challenge

pypi.org

Super useful way to show a tree view of an s3 bucket’s structure!

pip install s3-tree

s3-tree bucketname

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 kndndrj/nvim-dbee: Interactive database client for neovim

2023-07-29T01:08:20Z

GitHub - kndndrj/nvim-dbee: Interactive database client for neovim

Interactive database client for neovim. Contribute to kndndrj/nvim-dbee development by creating an account on GitHub.

GitHub · github.com

A neovim database client that I need to check out.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite-utils now supports plugins

2023-07-28T14:59:37Z

sqlite-utils now supports plugins

sqlite-utils 3.34 is out with a major new feature: support for plugins. sqlite-utils is my combination Python library and command-line tool for manipulating SQLite databases. It recently celebrated...

Simon Willison’s Weblog · simonwillison.net

As the title states sqlite-utils now supports plugins. I dug in just a bit and Simon implemented this completely with entrypoints, no framework or library at all.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Harlequin SQL IDE - DuckDB

2023-07-28T14:59:37Z

External Link

duckdb.org

Harlequin is a pretty sweet example of what textual can be used to create. Its a terminal based sql ide for DuckDB.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Python API - DuckDB

2023-07-28T14:59:37Z

To persist data in duckdb you need to first make a connection to a duck db database.

con = duckdb.connect('file.db')

Then work off of the connection con rather than duckdb.

con.sql('CREATE TABLE test(i INTEGER)')
con.sql('INSERT INTO test VALUES (42)')
# query the table
con.table('test').show()
# explicitly close the connection
con.close()

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 SQL on Pandas - DuckDB

2023-07-28T14:59:37Z

Redirecting…

duckdb.org

duckdb can just query any pandas dataframe that is in memory.

I tried running it against a list of objects and got this error. Great error message that gives me supported types right in the message.

Make sure that "posts" is either a pandas.DataFrame, duckdb.DuckDBPyRelation, pyarrow Table, Dataset, RecordBatchReader, Scanner, or NumPy ndarrays with supported format

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Full-text search - Datasette documentation

2023-07-28T14:59:37Z

Full-text search - Datasette documentation

docs.datasette.io

Enable full-text search in sqlite using sqlite-utils.

$ sqlite-utils enable-fts mydatabase.db items name description

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite-utils command-line tool - sqlite-utils

2023-07-28T14:59:37Z

sqlite-utils command-line tool - sqlite-utils

sqlite-utils.datasette.io

I want to like jq, but I think Simon is selling me on sqlite, maybe its just me but this looks readable, hackable, editable, memorizable. Everytime I try jq, and its 5 minutes fussing with it just to get the most basic thing to work. I know enough sql out of the gate to make this work off the top of my head

curl  https://thoughts.waylonwalker.com/posts/ | sqlite-utils memory - 'select title, message from stdin where stdin.tags like "%python%"' | jq

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite-utils command-line tool - sqlite-utils

2023-07-28T14:59:37Z

sqlite-utils command-line tool - sqlite-utils

sqlite-utils.datasette.io

insert a json array directly into into sqlite with sqlite-utils.

echo '{"name": "Cleo", "age": 4}' | sqlite-utils insert dogs.db dogs -

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Zev Averbach Interview

2021-04-09T17:40:42Z

Zev Averbach, Frustrated spreadsheet jockey to software developer at 36

Q: Tell me about your journey as a spreadsheet jockey into Data Engineering?

A: First of all, it’s hilarious that I accidentally found your questions for this interview by Googling myself. 😊

I’ve always been a frustrated software user, and that frustration led me to be a “power user” (keyboard shortcuts etc) of my most used applications, as well as a “visual coder” using desktop automation like Alfred and Keyboard Maestro (Mac).

Now that I’ve met data analysts and finance people that use Excel all day, I don’t think I’d claim to have been a true “spreadsheet jockey” in comparison to them. However, hitting up against the limitations of spreadsheets for running my transcription business – specifically for bookkeeping – created a new frustration for me:

As the business grew I was spending more and more time copying entries from Google Sheets to the Freshbooks web app for invoicing purposes. I tried to automate that process, but I was doing it using desktop automation, which was finicky and still more time-consuming than I wanted.

The other big frustration was assigning transcription work: My days were punctuated with the task of assigning and re-assigning transcription projects using dropdown menus in a web app someone had built for me previously.

I tried to get devs to automate those two things – invoicing and assignment of work – but I wasn’t successful, and had been waiting a couple of years for this to come through.

The short version of my journey is that I learned to code and ended up building these automations as my first “real world” projects.

The medium version is that it turned out to be really fun to write software, and that I’ve written quite a bit more since then, both for myself and my company as well as for others (open source and contract work). Then I got a job as a data engineer solely based on my Python skills, and have been writing and maintaining a pile of Airflow DAGs, as well as the code that generates many of them, for the past couple years.

Q: Why should a “spreadsheet jockey” add something like Python to their toolbelt?

A: To simply make their life a lot easier. Sure, heavy spreadsheet users have probably hit up against the size and performance limitations of their tool, and Python- or Scala-based tools are definitely viable solutions to that. But for me the power of imperative programming, of translating steps A to Z that you normally perform in a spreadsheet into human- and machine-readable language, is a magic you can’t ignore once you know about it.

On the ground, this magic translates into massive productivity gains, and frees you up to do higher-level thinking and experimentation which you don’t normally have time for.

Q: What tips do you have for others trying to expand their skillset later in life with limited time and resources for learning?

A: That’s a hard one because I actually had abundant time and sufficient resources for learning and noodling around with Python and, later on, JS. I know someone like Danny Thompson would say – or at least model – “sleep less, work when your kids are sleeping,” etc., but I’m not totally sure that’s doable for many. Personally, I’m pretty sensitive to sleep deprivation.

So here’s a path that could work for a subset of the “later in life coding learners” which resembles mine at least a little: Find ways to make your job more productive using Python or JS. If there’s something mind-numbing and/or repetitive which you’d like to not do anymore, make that the project you build as you’re learning to code. Automate the Boring Stuff is a great jumping-off point.

If you can’t think of anything to automate, you’re probably not looking hard enough. 😆 But if you think harder and still can’t come up with anything, email me!

Q: How do you fight imposter syndrome after a mid-career switch?

A: I have no answer for this, as I still fight the syndrome on a daily basis! Anecdotally this doesn’t really go away, even for CS majors with successful startup acquisitions, etc. My current theory for why this could be is the largely solitudinous nature of writing software. I don’t think coding has to be a solo activity, but it’s just worked out that way for most of us. As you know, Waylon, I really enjoy pair programming, especially as a way to teach and learn, but I also think it’s a great way to fight the syndrome. This is partly because you’ll inevitably discover that everyone’s knowledge is uneven and that there’s always something you as a novice can teach a more experienced dev.

To put it in simpler terms, how can you even know whether you’re an imposter if you work in a one-person silo all day?

Q: Where can we find you online?

A: That’d be here!

Minimal Kedro Pipeline

2021-01-20T00:00:00Z

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It’s only a total of 35 lines of python, 8 in setup.py and 27 in mini_kedro_pipeline.py.

📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner.

Minimal Kedro Pipeline #

I have everything for this post hosted in this gihub repo, you can fork it, clone it, or just follow along.

Installation #

pip install git+https://github.com/WaylonWalker/mini-kedro-pipeline

Caveats #

This repo represents the minimal amount of structure to build a kedro pipeline that can be shared across projects. Its installable, and drops right into your hooks.py or run.py modules. It is not a runnable pipeline. At this point I think the config loader requires to have a logging config file.

This is a sharable pipeline that can be used across many different projects.

Usage #

# hooks.py

import mini_kedro_project as mkp

class ProjectHooks:
    @hook_impl
    def register_pipelines(self) -> Dict[str, Pipeline]:
        """Register the project's pipeline.

        Returns:
            A mapping from a pipeline name to a ``Pipeline`` object.

        """

        return {"__default__": Pipeline([]), "mkp": mkp.pipeline}

Implemantation #

This builds on another post that I made about creating the minimal python package. I am not sure if it should be called a package, it’s a module, but what do you call it after you build it and host it on pypi?

[[ minimal-python-package ]]

Directory structure #

.
├── .gitignore
├── README.md
├── setup.py
└── my_pipeline.py

setup.py #

This is a very minimal setup.py. This is enough to get you started with a package that you can share across your team. In practice, there is a bit more that you might want to include as your project grows.

from setuptools import setup

setup(
    name="MiniKedroPipeline",
    version="0.1.0",
    py_modules=["mini_kedro_pipeline"],
    install_requires=["kedro"],
)

mini_kedro_pipeline.py #

The mini kedro pipeline looks like any set of nodes in your project. Many projects will separate nodes and functions, I prefer to keep them close together. The default recommendation is also to have a create_pipelines function that returns the pipeline.

This pattern creates a singleton, if you were to reference the same pipeline in multiple places within the same running interpreter and modify the one you would run into issues. I don’t foresee myself running into this issue, but maybe as more features become available I will change my mind.

"""
An example of a minimal kedro pipeline project
"""
from kedro.pipeline import Pipeline, node

__version__ = "0.1.0"
__author__ = "Waylon S. Walker"

nodes = []


def create_data():
    "creates a dictionary of sample data"
    return {"beans": range(10)}


nodes.append(node(create_data, None, "raw_data", name="create_raw_data"))


def mult_data(data):
    "multiplies each record of each item by 100"
    return {item: [i * 100 for i in data[item]] for item in data}


nodes.append(node(mult_data, "raw_data", "mult_data", name="create_mult_data"))

pipeline = Pipeline(nodes)

Kedro in scripts #

Using Kedro In Scripts

If you enjoyed this one check out this companion article where I build a fully runnable kedro project in a single script.

Go forth and share your pipelines across projects. Let me know, do you share pipelines or catalogs across projects?

Blog Data With Python

2021-01-18T00:00:00Z

Generating an api for a blog is much simpler than one might expect with python.

Markdown #

Frontmatter #

Fill in the blanks #

fix missing data

Fast #

Kedro - My Data Is Not A Table

2021-01-14T00:00:00Z

In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask.

DataFrames are the heart of most pipelines #

These containers for data contain many convenient methods to manipulate table like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types.

What is Kedro

unfamiliar with kedro, check out this post

Sometimes datasets are not tables #

There are times when our data doesn’t fit nicely into a DataFrame. Lucky for us Kedro has pickle support out of the box. Pickle is a way to store any python object to disk. Beware that pickle files coming from an unknown source can run malicous code and are considered unsafe. For the most part though when you read and write your own pickle files they are a good tool to consider.

See more about pickle from python.org.

Cataloging Pickle #

I may have a dictionary that describes some cars.

{
  'truck-012-abc': {
    'type': 'truck'
    'sales': [12, 2, 3, 4, 8]
    'weight': 9024,
    'accesories': ['leather', 'audio-1']
}

In the catalog we will simply set the type as pickle.PickleDataSet and give it a filepath.

cars:
  filepath: data/cars.pkl
  type: pickle.PickleDataSet

This filepath does not have to be on the local filesystem it can be on the cloud thanks to how kedro utilizes fsspec for each of its datasets.

Loading the dataset #

The benefit of cataloging this dataset compared to leaving it as a MemoryDataSet is that you can easily load this data back into memory for further development or debugging without running any of the pipeline.

catalog.load('cars')

Gracefully adopt kedro, the catalog

2020-06-29T03:00:00Z

Why use kedro catalog? #

While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont even touch read/write steps at all.

Taking a step into kedro by adopting the catalog first will give you a way to organize all of your data loads in one place, and stop manually writing read/write code, which can be different for each data and storage type. You just don’t need to think about it.

iperitive loading style
organizes your data
all file locations can be quickly identified
can be dropped into kedro later

“can be dropped into kedro later” Let’s talk a bit more about that

2 Ways to Gracefully adopt the catalog #

How do I get started with the kedro catalog

add with the code api
load from yaml (recommended)

1. Adding to the catalog with the code api #

how to use the kedro catalog code api

It is possible to keep everything inside of one single file if you want by utilizing the code api, which defines the kedro catalog inside of your script with python. I personally like this method as it is a bit more scriptable to create many layers of datasets with a for loop. It can still be carried right into a kedro project, but is not the normal way that other kedro users will be used to seeing. I’ll leave it up to you which technique to use.

from kedro.io import DataCatalog
from kedro.extras.datasets.pandas import CSVDataSet

io = DataCatalog(
    {
        "bikes": CSVDataSet(filepath="../data/01_raw/bikes.csv"),
    }
)

taken right from the kedro docs

👆 This can be done inside of a single Jupyter cell, and pulled out later.

2. Creating a catalog config file #

recommended

This reccommended method it great since it will simply drop right in to a full kedro project if you were ever ready to adopt the framework as a whole. The downside to gracefully adopting the framework is that you need to have a bit of an understanding of the internals to do it. When using the framework as a whole it seemlessly takes care of everything for you.

# conf/base/catalog.yml
# Example 1: Loads / saves a CSV file from / to a local file system

bikes:
  type: pandas.CSVDataSet
  filepath: data/01_raw/bikes.csv

from kedro.config import ConfigLoader
from kedro.io import DataCatalog

conf_loader = ConfLoader(['conf/base'])
conf_catalog = conf_loader.get('catalog*', 'catalog/**')
catalog = DataCatalog.from_config(conf_catalog)

You can even do this 👆 from a Jupyter notebook

If you do happen to be in a module deeper into an existing library I tend to leverage the use of the __file__ magic. __file__ is a string containing the location of the current file. You can initiate a Path object from this and roll up the necessary number of directories with the parents attribute then into the conf/base directory. I find this more more robust as it does not depend on your current working directory.

from pathlib import Path
conf_loader = ConfLoader([Path(__file__).parents[1] / 'conf/base'])
conf_catalog = conf_loader.get("catalog*", "catalog/**")
new_catalog = DataCatalog.from_config(conf_catalog)

I find leveraging the __file__ magic a bit more robust when possible as it does not depend on your current working directory.

Adhoc adding to an existing kedro catalog #

Let’s say that you are already using a kedro project, but you are wanting to develop new features entirely in the context of a notebook. You can easily create a catalog as shown above. Since the kedro catalog variable is typically catalog, we will call it new_catalog and add it to the existing catalog below. If you are not quite sure what that **catalog syntax is doing check out my article on **kwargs.

catalog.add_feed_dict(new_catalog.datasets.__dict__)

python args and kwargs article by @_waylonwalker

The framework makes appending

catalog = DataCatalog({**catalog.datasets.__dict__, **new_catalog.datasets.__dict__})
``



## One word of caution


If you have any hooks running `after_catalog_created`, they will not be run on the appended catalog entries.

How to find things in your kedro catalog

2020-06-22T03:00:00Z

kedro 0.16.2 just dropped last week with a long-awaited feature… catalog search! I went as far as monkey patching this into each of my projects. I work jump between a few really big projects that have tons of datasets. Being able to quickly search for what I need is so useful.

The Catalog #

The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections listed in one place make it so easy to pick your project up and move it to a completely new environment. That sweet imperative loading style saves so much read/write overhead. I can load all my data with a single command whether it’s in amazon s3, google cloud platform, or a local file.

Kick start a toy project #

Just like with most of these articles, I am going to create a conda environment so that I don’t break any existing projects and scaffold up a toy project to learn from.

conda create -n kedro0162 python=3.8 -y
activate kedro0162
pip install kedro
kedro new # call it Kedro 0162 and click-through
cd kedro-0162
kedro install

Expect this set of commands to take a few minutes depending on your system, connection speed, and amount of packages already in your local cache.

Create some catalog #

Now the power of the catalog search really starts to shine when your projects grow legs. You have groups of many datasets containing patterns of data including layer, or source among other things.

vim conf/base/catalog.yml

In the catalog, you will see a few lines of instructions followed by

example_iris_data:
  type: pandas.CSVDataSet
  filepath: data/01_raw/iris.csv

This gives us one stored catalog entry called example_iris_data, it is a CSV file stored in data/01_raw/iris.csv.

Let’s make up a transportation company that is siloed into three different divisions and it is our job to bring their sales and product metadata into a single report. This company makes lifted-trucks, primium-scoots, and luxy-yahts. and we know that we will want raw, int, pri and modin layers to start our project so let’s scaffold up that catalog real quick.

# ――――――――― lifted-truck ―――――――――

raw_lifted_truck_sales:
  type: pandas.CSVDataSet
  filepath: data/01_raw/sales/lifted-truck.csv

int_lifted_truck_sales:
  type: pandas.CSVDataSet
  filepath: data/01_int/sales/lifted-truck.csv

pri_lifted_truck_sales:
  type: pandas.CSVDataSet
  filepath: data/01_pri/sales/lifted-truck.csv

raw_lifted_truck_info:
  type: pandas.CSVDataSet
  filepath: data/01_raw/info/lifted-truck.csv

int_lifted_truck_info:
  type: pandas.CSVDataSet
  filepath: data/01_int/info/lifted-truck.csv

pri_lifted_truck_info:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/lifted-truck.csv

# ――――――――― primium-scoot ―――――――――

raw_primium_scoot_sales:
  type: pandas.CSVDataSet
  filepath: data/01_raw/sales/primium-scoot.csv

int_primium_scoot_sales:
  type: pandas.CSVDataSet
  filepath: data/01_int/sales/primium-scoot.csv

pri_primium_scoot_sales:
  type: pandas.CSVDataSet
  filepath: data/01_pri/sales/primium-scoot.csv

raw_primium_scoot_info:
  type: pandas.CSVDataSet
  filepath: data/01_raw/info/primium-scoot.csv

int_primium_scoot_info:
  type: pandas.CSVDataSet
  filepath: data/01_int/info/primium-scoot.csv

pri_primium_scoot_info:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/primium-scoot.csv

# ――――――――― luxy-yaht ―――――――――

raw_luxy_yaht_sales:
  type: pandas.CSVDataSet
  filepath: data/01_raw/sales/luxy-yaht.csv

int_luxy_yaht_sales:
  type: pandas.CSVDataSet
  filepath: data/01_int/sales/luxy-yaht.csv

pri_luxy_yaht_sales:
  type: pandas.CSVDataSet
  filepath: data/01_pri/sales/luxy-yaht.csv

raw_luxy_yaht_info:
  type: pandas.CSVDataSet
  filepath: data/01_raw/info/luxy-yaht.csv

int_luxy_yaht_info:
  type: pandas.CSVDataSet
  filepath: data/01_int/info/luxy-yaht.csv

pri_luxy_yaht_info:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/luxy-yaht.csv


# ――――――――― combined ―――――――――
pri_combined_sales:
  type: pandas.CSVDataSet
  filepath: data/01_pri/sales/combined.csv

pri_combined_info:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/combined.csv

# ――――――――― modin ―――――――――

modin_main:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/combined.csv

Some examples of common regex uses #

regex gets really complicated fast, but these basic examples are very common use cases and will get you a long way without being very complicated.

term - all catalog entries that include term in the catalog entry
^term - all catalog entries that include term at the beginning of the catalog entry
term$ - all catalog entries that include term at the end of the catalog entry
term1.*term2 - include anything in between term1 and term2.
term1|term2 - all catalog entries that include term1 or term2

Let’s Search this thing #

kedro has long included the catalog.list() feature that returns a list of all datasets. Now the list command takes in a regex_search keyword argument. By default, it is empty and returns the entire catalog.

kedro ipython

list out all of the luxy-yahts #

>>> catalog.list('luxy_yaht`)
['raw_luxy_yaht_sales',
 'int_luxy_yaht_sales',
 'pri_luxy_yaht_sales',
 'raw_luxy_yaht_info',
 'int_luxy_yaht_info',
 'pri_luxy_yaht_info']

List out data by layer #

Easy just search for the layer name.

raw #

>>> catalog.list('raw')
['raw_lifted_truck_sales',
 'raw_lifted_truck_info',
 'raw_primium_scoot_sales',
 'raw_primium_scoot_info',
 'raw_luxy_yaht_sales',
 'raw_luxy_yaht_info']

pri #

 >>> catalog.list('pri')
['pri_lifted_truck_sales',
 'pri_lifted_truck_info',
 'raw_primium_scoot_sales',
 'int_primium_scoot_sales',
 'pri_primium_scoot_sales',
 'raw_primium_scoot_info',
 'int_primium_scoot_info',
 'pri_primium_scoot_info',
 'pri_luxy_yaht_sales',
 'pri_luxy_yaht_info',
 'pri_combined_sales',
 'pri_combined_info']

😲 We just included every primium-scoot dataset!

Here we just encountered our first need for regex. I’ll be the first to admit that I am really bad at regex, it’s incredibly confusing, becomes read-only with much complexity, but is super powerful and used in a lot of places.

`^term` #

beginning of catalog entry

The ^ regex operator searches for catalog entries that include the search term at the very beginning.

 >>> catalog.list('^pri')
['pri_lifted_truck_sales',
 'pri_lifted_truck_info',
 'pri_primium_scoot_sales',
 'pri_primium_scoot_info',
 'pri_luxy_yaht_sales',
 'pri_luxy_yaht_info',
 'pri_combined_sales',
 'pri_combined_info']

`term$` #

end of catalog entry

The $ operator is the opposite of the ^ operator. It means give me all that matches that occur at the end of the catalog entry.

>>> catalog.list('info$')
['raw_lifted_truck_info',
 'int_lifted_truck_info',
 'pri_lifted_truck_info',
 'raw_primium_scoot_info',
 'int_primium_scoot_info',
 'pri_primium_scoot_info',
 'raw_luxy_yaht_info',
 'int_luxy_yaht_info',
 'pri_luxy_yaht_info',
 'pri_combined_info']

`term1.*term2` #

The .* operator in regex means give me all the datasets that include the two terms no matter what is between them. There is also a .? to only allow one character between them. More often than not I really just want the two patterns to exist in the dataset entry.

>>> catalog.list('raw.*info$')
['raw_lifted_truck_info',
 'raw_primium_scoot_info',
  'raw_luxy_yaht_info']

Some real things that we can do with search #

Let’s look at a few examples beyond the obvious of just searching for the dataset that we want to load.

Check Raw Data #

While migrating pipelines between environments it’s important to know if your raw datasets are available. I will argue that you should also consider looking at pipeline.inputs as it cannot lie and gives you a true reading of the pipeline inputs. But another easy check might be to check all the datasets that the Data Engineers have labeled raw.

>>> {dataset: catalog.exists(dataset) for dataset in catalog.list('^raw')}
{'raw_lifted_truck_sales': False,
 'raw_lifted_truck_info': False,
 'raw_primium_scoot_sales': False,
 'raw_primium_scoot_info': False,
 'raw_luxy_yaht_sales': False,
 'raw_luxy_yaht_info': False}

Since we just created a dummy catalog the data does not exist in this example.

Create a new catalog #

Let’s say that we have someone on the team who is from the land division of our company and they want a simplified catalog readily available that does not include any marine data.

To do this we will need to reach a bit into the kedro internals for the DataCatalog class and utilize a new regex operator |.

>>> from kedro.io import DataCatalog
>>> land_catalog = DataCatalog(
    {
        dataset: getattr(catalog.datasets, dataset)
        for dataset in catalog.list('truck|scoot')
        }
    )
>>> land_catalog.list()
['raw_lifted_truck_sales',
 'int_lifted_truck_sales',
 'pri_lifted_truck_sales',
 'raw_lifted_truck_info',
 'int_lifted_truck_info',
 'pri_lifted_truck_info',
 'raw_primium_scoot_sales',
 'int_primium_scoot_sales',
 'pri_primium_scoot_sales',
 'raw_primium_scoot_info',
 'int_primium_scoot_info',
 'pri_primium_scoot_info']

regex recap #

^term - beginning
term$ - end
term1.*term2 - anything in between
term1|term2 - or

How Kedro handles your inputs

2020-06-19T03:00:00Z

Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing.

*args/**args review #

Check out this post for a review of how *args **kwargs work in python.

understanding python *args and **kwargs

python args and kwargs article by @_waylonwalker

All Kedro inputs are catalog Entries #

When kedro runs your pipeline it uses the catalog to imperatively load your data, meaning that you don’t tell kedro how to load your data, you tell it where your data is and what type it is. These catalog entries are like a key-value store. You just need to give the key when setting up a node.

Single Inputs #

These are fairly straightforward to understand. In the example below when kedro runs the pipeline it will load the input from the catalog, then pass that input to the func, then save the returned value to the output catalog entry.

from kedro.pipeline import node

def create_int_sales(sales):
    "cleans up raw sales data"
    ...
    return cleaned_sales

my_node = node(
    func=create_int_sales,
    inputs='raw_sales',
    output='int_sales',
    )

List of inputs #

Let’s look at an example node that combines more than one dataset. When kedro has sees a list of catalog entries it will load up each catalog entry sequentially then pass them in order to the create_sales_report function.

from kedro.pipeline import node

def create_sales_report(sales, products):
    "adds product metadata to the sales data"
    ...


my_node = node(
    func=create_sales_report,
    inputs=['pri_sales', 'pri_products'],
    output='sales_report',
    )

simulating pipeline run using 2 inputs #

Here We can simulate what kedro does during the pipeline run by using *args.

# inputs you gave kedro
inputs=['pri_sales', 'pri_products']
# load data
input_data = [catalog.load(entry) for entry in  inputs]
# run the node
sales_report = create_sales_report(*input_data)
# save the data to the output
catalog.datasets.sales_report.save(sales_report)

More generalizable functions #

We can also use *args to make our functions a little bit more generalizable. The first that comes to my mind is a unioner. The second

def unioner(*dfs: pd.DataFrame): -> pd.DataFrame
    pd.concat(dfs)

Now we can pass any number of DataFrames into our kedro node to get unioned together, but do we really need a function for a one-liner… No we can use an inline function for this case.

my_node = node(
    func=lambda *dfs: pd.concat(dfs),
    input=['sales_2017', 'sales_2018'],
    output='sales',
)

`*args` scares the crap out of me #

It’s great for the unioner example where its a collection of similar things where order does not matter. But for the create_sales_report function. Those are distinctly different inputs. If someone does some refactoring and changes the order in one place or another it’s going to turn into a bad day real fast.

**kwargs are a bit better #

Let’s refactor the create_sales_report before someone tries to ruin our day. We can easily do this by passing a dictionary (keys are the argument name, values are the catalog key) of arguments to kedro instead of a list.

from kedro.pipeline import node

def create_sales_report(sales, products):
    "adds product metadata to the sales data"
    ...

my_node = node(
    func=create_sales_report,
    inputs={'sales': 'pri_sales', 'products': 'pri_products'},
    output='sales_report',
    )

Now if someone tries to refactor the order of arguments we are safe!

Simulating the pipeline run with `**kwargs` #

Pretty much the same as before, except with **kwargs and dictionaries keeping us a bit safer.

# inputs you gave kedro
inputs={'sales': 'pri_sales', 'products': 'pri_products'},
# load data
input_data = {arg: catalog.load(entry) for arg, entry in inputs.items()}
# run the node
sales_report = create_sales_report(**input_data)
# save the data to the output
catalog.datasets.sales_report.save(sales_report)

Stay Safe #

Kedro inputs are quite easy to understand if you already have a grasp of *args and **kwargs and if you don’t it is still fairly intuitive to pick up. Stay on the safe side, if your collection of inputs are clearly different things, use a dictionary for safety.

011

2020-06-07T02:09:58Z

Load data from database into pandas

import pandas as pd
from sqlalchemy import create engine

engine = create_engine('postgresql://scott:tiger@localhost:5432/mydatabase')

sql = 'select * from inventory'

with engine.connect() as connection:
    inventory = pd.read_sql(sql, con)
engine.dispose()

010

2020-06-07T02:07:13Z

load remote data with pandas

import pandas as pd
cars_url = 'waylonwalker.com/cars.csv'
cars = pd.read_csv(cars_url)

009

2020-06-07T02:02:58Z

Combine a directory of csv’s with pandas

import pandas as pd
from pathlib import Path

csvs = Path.glob('raw/*.csv')
csvs_combined = pd.concat(csvs)
csvs_combined.to_csv('processed/combined.csv')

Create Custom Kedro Dataset

2020-05-15T05:00:00Z

Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When running Kedro takes care of all of the read/write, you just reference the catalog key.

But what is happening behind the scenes #

Under the hood there is an AbstractDataSet that each connector inherits from. It sets up a lot of the behind the scenes structure for us so that we dont have to. For the most part kedro has connectors for about anything that you want to load, csv, parquet, sql, json, from about anywhere, http, s3, localfile system are just some of the examples.

Here is a DataSet implementation from their docs. Here you can see the barebones example straight from the docs. Parameters from the yaml catalog will get passed in

from pathlib import Path

import pandas as pd

from kedro.io import AbstractDataSet


class MyOwnDataSet(AbstractDataSet):
    def __init__(self, param1, param2, filepath, version):
        super().__init__(Path(filepath), version)
        self._param1 = param1
        self._param2 = param2

    def _load(self) -> pd.DataFrame:
        load_path = self._get_load_path()
        return pd.read_csv(load_path)

    def _save(self, df: pd.DataFrame) -> None:
        save_path = self._get_save_path()
        df.to_csv(save_path)

 	def _exists(self) -> bool:
        path = self._get_load_path()
        return path.is_file()

    def _describe(self):
        return dict(version=self._version, param1=self._param1, param2=self._param2)

What is YOUR Advice for New Data Scientists

2020-02-26T12:55:00Z

Learn the business
Learn Git
Your code does not need to be amazing
Keep Learning

Learn Git #

You dont have to start out as a git wizard with the cleanest possible commit history. At first dont let yourself get too wrapped up in it, the most important part is that you make commits. You will find needs for more advanced stuff later.

git add .
git commit -m "FEAT added new function to calculate revenue by product family"
git push

Get comfortable with this, then learn how to branch, rebase, stash, etc…

Your code does not need to be amazing #

Get the job done. Keep it in small bite size pieces. Make readable function definitions and variable names. You will thank yourself for naming things well later. Readability counts more than performance in most cases of data science. If it gets the job done try not to over worry about things like performance. A few extra seconds to clean a dataset or build a model is not worth hours of your time. As you go you will have cases that performance is more critical and you will learn what to do from the start to avoid them.

Filtering Pandas

2019-09-24T05:00:00Z

query #

Good for method chaining, i.e. adding more methods or filters without assigning a new variable.

# is
skus.query('AVAILABILITY == " AVAILABLE"')
# is not
skus.query('AVAILABILITY != " AVAILABLE"')

masking #

general purpose, this is probably the most common method you see in training/examples

# is
skus[skus['AVAILABILITY'] == 'AVAILABLE']
# is not
skus[~skus['AVAILABILITY'] == 'AVAILABLE']

isin #

capable of including multiple strings to include

# is in
df[df.AVAILABILITY.isin(['AVAILABLE', 'AVL'])]
# is not in
df[~df.AVAILABILITY.isin(['AVAILABLE', 'AVL'])]

contains #

Good For partial matches

# contains
df[df.AVAILABILITY.str.contains('AVA')]
# not contains
df[~df.AVAILABILITY.str.contains('AVA')]

MASKS #

anything that we put inside of square brackets can be set as a variable then passed in.

service_mask = skus['AVAILABILITY'] == 'AVAILABLE'
name_mask = skus['NAME'] == 'Dell chromebook 11'

Operators #

& - and ~ - not | - or

AVAILABLE and NAME #

df[service_mask & name_mask]

AVAILABLE or NAME #

df[service_mask | name_mask]

AVAILABLE and not NAME #

df[service_mask & ~name_mask]

Clean up Your Data Science with Named Tuples

2019-09-11T00:00:00Z

If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow.

Example #

As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing sales, inventory, and pricing data. You may need to calculate total revenue, inventory on hand. You may need to pass these data sets into various models to drive production or pricing based on predicted volumes.

Load data #

Here we setup functions that can load data from the sales database. Assume that we also have similar functions to get_inventory and get_pricing.

def get_engine():
    engine = create_engine('postgresql://scott:tiger@localhost:5432/mydatabase')

def get_sales():
    '''
    gets sales history from the sales database
    '''
    engine = get_engine()
    with engine.connect() as con:
        sql = '''select * from sales.history'''
        df = pd.read_sql(sql, con)
    engine.dispose()
    return df

def get_inventory():
    ...

def get_pricing():
    ...

Create Metrics #

Here we create our first function to calculate some metrics. There are likely many of these functions that repeat a similar pattern. They use similar data and have their own custom logic for calculations and joins.

def calculate_total_revenue(sales, pricing):
    """calculates the total sales revenue for all of company XYZ"""
    sales = sales.join(pricing.set_index('sku'), on='sku')
    sales['revenue'] = sales['qty'] * sales['price']
    return sales['revenue'].sum()

def calculate_inventory_sale_ratio(sales, inventory, pricing):
    ...

def calculate_inventory_sale_ratio(inventory, sales):
    ...

Furthermore these functions will need to be called somewhere, this might be in a make_report() function that puts this data into an html template to be sent out to stakeholders, or to be served on a website. Notice how we have the same data showing up time and time again. And sometimes we even ask for it in a different order 😲.It is important to recognize this early in the project before this gets our of hand.

def make_report():
    """Makes stakeholder report for company XYZ"""
    sales = get_sales()
    inventory = get_inventory()
    pricing = get_pricing()

    revenue = calculate_total_revenue(sales, pricing)
    sales_ratio = inventory_sale_ratio(sales, inventory, pricing)
    inventory_sale_ratio(inventory, sales)

getting out of hand #

Along the way our features, models, and out supervisors have all had their own needs and we have added new datasets, and several flags. This is the point at which anxiety starts creeping in. We start spending a lot of time double checking the order of each call to make sure that we dont make a mistake. And when someone else touches this model, we know what it looks like and cant help but think, “Oh God I hope they didn’t screw up that horrid module!”

def calculate_total_revenue(sales, pricing, inventory, stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000'):
    sales = (sales
                .join(pricing.set_index('sku'), on='sku')
                .join(inventory.set_index('sku'), on='sku')
            )
    sales['revenue'] = sales['qty'] * sales['price']
    sales  = sales.query(f'price > {min_price}')
    sales  = sales.query(f'sale_date > {min_date}')
    sales  = sales.query(f'sale_date < {end_date}')
    return sales['revenue'].sum()

def calculate_inventory_sale_ratio(sales, pricing, inventory, stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000'):
    ...

def calculate_inventory_sale_ratio(sales, pricing, inventory, stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000'):
    ...

def model_new_prices(sales, pricing, inventory, stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000'):
    ...

def model_production(sales, pricing, inventory, stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000'):
    ...

def completely_custom_metric_for_steve(sales, pricing, inventory, stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000'):
    """Steve has been here 30 years and doesnt trust our metrics unless he can validate against the old metrics"""
    ...

It gets even crazier when you start calling all of these functions! Note that we have a common theme of the same data getting passed into

def make_report(stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000', is_for_steve=False):
    """Makes stakeholder report for company XYZ"""
    sales = get_sales()
    inventory = get_inventory()
    pricing = get_pricing()

    revenue = calculate_total_revenue(stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000')
    sales_ratio = sales_ratio = inventory_sale_ratio(stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000')
    inventory_sale_ratio = inventory_sale_ratio(stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000')
    predicted_prices = model_new_prices(stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000')
    predicted_production_rates = model_production_rate(stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000')
    completely_custom_metric_for_steve(stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000')

    # render report
    ...

This code Stinks #

⌚ Time for refactoring

While our code started out good it has grown stinky over time. It will works fine, but it could be better and more enjoyable to work with. What we have here is a code smell called data clump. This is a group of data that always appears together. It would be appropriate to formally group this data together. One way to do this in python is to use nametuples. It is a very simple technique that allows us to define a data structure that contains a specific set of attributes that we can access using .attribute_name. Replacing this data clump with a formal group of data will allow us to reduce the number of arguments in all of our functions. Add new attributes easily. And not rely on positional arguments. This code will be easier to maintain, read, and write.

What we have here is a code smell called data clump

create a namedtuple

from collections import namedtuple

ModelData = namedtuple('modin_data', 'sales pricing inventory stored_at min_price start_date end_date')

use the namedtuple

data = ModelData(
    sales=get_sales(),
    pricing=get_pricing(),
    inventory=get_inventory(),
    stored_at='LAX'
    min_price=100
    start_date = datetime.today() - datetime.timedelta(days=30)
    end_date = datetime.today()
    )

refactor functions Now that we have a clean data object how do we use it. Simple, we pass in one data object, then access each attribute with the dot operator. These functions are now much cleaner to call and read. Here I have chosen a poor name for our data, but in a real scenario you may have multple namedtuples.

def calculate_total_revenue(data):
    sales = (data.sales
                .join(data.pricing.set_index('sku'), on='sku')
                .join(data.inventory.set_index('sku'), on='sku')
            )
    sales['revenue'] = sales['qty'] * sales['price']
    sales  = sales.query(f'price > {data.min_price}')
    sales  = sales.query(f'sale_date > {data.min_date}')
    sales  = sales.query(f'sale_date < {data.end_date}')
    return sales['revenue'].sum()

...

call the function

Now that all of the data is store in a single object it is really easy to call each of our functions using one data instance.

def make_report(stored_at='LAX', min_price=100, start_date='01-01-1999', end_date='01-01-3000', is_for_steve=False):
    """Makes stakeholder report for company XYZ"""
    data = ModelData(stored_at=stored_at, min_price=min_price, start_date=start_date, end_date=end_date)

    revenue = calculate_total_revenue(data)
    sales_ratio = sales_ratio = inventory_sale_ratio(data)
    inventory_sale_ratio = inventory_sale_ratio(data)
    predicted_prices = model_new_prices(data)
    predicted_production_rates = model_production_rate(data)
    completely_custom_metric_for_steve(data)

    # render report
    ...

🧹 Clean up your data science #

Named Tuples are a great way to clean up your data science code and reduce Data Clumps. Anytime you have multiple data sets that almost always get used together namedtuples are a great way to clean up your code, make it more readable and more maintainable. If you find that you also have functions that are tightly coupled to this data you might want to consider using a class instead of a namedtuple, but that is for another article.

Background Tasks in Python for Data Science

2019-09-10T05:00:00Z

This post is intended as an extension/update from background tasks in python. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started.

I use it in more places than I probably should

Before we get into it, I want to make a shout out to Kenneth Reitz for making this so easy. Kenneth is a python God for all that he has given to the community in so many ways, especially with his ideas in building stupid simple api’s for very complicated things.

Installation #

install via pip #

pip install background

install via github #

I believe one of the later pr’s to the project fixes the way arguments are passed in. I generally clone the repo or copy the module directly into my project.

clone it

git clone https://github.com/ParthS007/background.git
cd background
python setup.py install

copy the module

curl https://raw.githubusercontent.com/ParthS007/background/master/background.py > background.py

🐌 The Slow Function #

Imagine that this function is a big one! This function is fairly realistic as it takes in some input and returns a DataFrame. This is what a good half of my fuctions do in data science. The internals of this function generally will include a sql query, load from s3 or a data catalog, an aggregation from another DataFrame. In general it should do one simple thing.

Feel Free to copy this “boilerplate”

import background
from time import sleep
import pandas as pd

@background.task
def long_func(i):
    """
    Simulates fetching data from a service
    and returning a pandas DataFrame.

    """
    sleep(10)
    return pd.DataFrame({'number_squared': [i**2]})

Calling the Slow Function #

it’s the future calling 🤙

If we were to call this function 10 times it would take 100s. Not bad for a dumb example, but detrimental when this gets scaled up💥. We want to utilize all of our available resources to reduce our development time and get moving on our project.

Calling long_func will return a future object. This object has a number of methods that you can read about in the cpython docs. The main one we are interested in is result. I typically call these functions many times and put them into a list object so that I can track their progress and get their results. If you needed to map inputs back to the result use a dictionary.

%time futures = [long_func(i) for i in range(10)]

CPU times: user 319 µs, sys: 197 µs, total: 516 µs
Wall time: 212 µs

Do something with those `results()` #

Simply running the function completes in no time! This is because the future objects that are returned are non blocking and will run in a background task using the ProcessPoolExecutor. To get the result back out we need to call the result method on the future object.result is a blocking function that will not realease until the function has completed.

%%time
futures = [long_func(i) for i in range(10)]
pd.concat([future.result() for future in futures])

CPU times: user 5.38 ms, sys: 3.53 ms, total: 8.9 ms
Wall time: 10 s

Note that this example completed in 10s, the time it took for only one run, not all 10! 😎

n #

😫 crank it up

By default the number of parallel processes wil be equal to the number of cpu threads on your machine. To increase the number of parallel processes (max_workers) set increase background.n.

background.n = 100

Is it possible to overruse @background.task? #

I use this essentially anywhere that I cannot vectorize a python operation and push the compute down into those fast 💨 c extended libraries like numpy, and the operation takes more than a few minutes. Nearly every big network request I make gets broken down into chunks and multithreaded. Let me know… is is possible to overruse @background.task? Let me know your thoughts @_WaylonWalker.

Repl.It #

Play with the code here! Try different values of background.n and n_runs.

Generating Readme Tables From Pandas

2018-05-16T00:00:00Z

Generating Readme Tables From Pandas #

I commonly have a need to paste the first few lines of a dataset into a markdown file. I use two handy packages to do this, tabulate and pyperclip. Lets say I have a Pandas DataFrame in memory as df already. All I would need to do to convert the first 5 rows to markdown and copy it to the clipboard is the following.

from tabulate import tabulate
import pyperclip
md = tabulate.tabulate(df.head(), df.columns, tablefmt='pipe')
pyperclip.copy(md)

This is a super handy snippet that I use a lot. Folks really appreciate it when they can see a sample of the data without opening the entire file.

Stepping Up My SQL Game

2018-03-25T00:00:00Z

In 2018 I transitioned from a Product Engineering (Mechanical) role to a Data Scientist Role. I entered this space with strong subject matter expertise with our products, our data, munging through data in pyhon, and data visualization in python. My sql skills were lacking to say the least. I had learned what I needed to know to get data from our relational databases, then use pandas to do any further analysis. Just run something like the following and you have data.

SELECT
    *
FROM
    Table
Where
    col_1 = 'col_1_filter'

This technique works great for small data sets that you only need to run once. There is no shame to pull in a big dataset and start munging with it in pandas to get some results, and make decisions. The problem becomes when your dataset becomes too big or you need to run the query on a frequent basis. Doing the aggregations on the server run much quicker, as it reduces the time spent in io. My longest running steps are currently io related. Reducing these steps have improved my workflow. At the point that I was getting server timeout errors, or using the same long running query in many places I would be searching for examples online, because I just did not have the experience with many more techniques. I decided it was time to put away the cheat sheets, step away from Stack Overflow, and improve my speed.

Why Learn SQL in 2018?? #

SQL is far from the hot topic in 2018, AI, Deep Learning, BIG data, Machine Learning, Natural Language Processing take the win here. SQL is so simple why would anyone want to spend time learning SQL? The reason… all of those hot topics in 2018 require data. My data mostly comes from relational databases which require sql to get data from them. Without the ability to efficiently get the data I need to do an aanlysis I cannot even start. Sure I could use an ORM, but I found that to be a bit unwieldy with the thousands of tables we have in formats that were determined many years ago. Plus raw SQL is more transportable. I commonly collaborate with other folks who do not use python. I am proud that I am able to point them to the SQL I use rather than telling them to suck it up an learn python. I truly believe that people are the most effective when they are able to choose their own stack of tools. Taking some time to focus on the basics of Data Science will help be build a strong foundation for my career.

Joining Data in Posgres #

Below are my notes from the Joining Data in Posgres course on DataCamp. I will use these notes as a refresher when I need a quick reference.

Using() #

When joining two tables on the same column the USING clause can be used as a shorthand.

without using

SELECT *

FROM
    Table1 as t1

LEFT JOIN
    Table2 as t2
    ON t1.id = t2.id

with using

SELECT
    *

FROM
    Table1 as t1

LEFT JOIN
    Table2 as t2
    USING (id)

Join Types #

for joining columns of data together into a single table

INNER: Includes only records contained in both tables.

RIGHT: Inlcudes all records from the right, droping values from the left if non-existent in the right, or leaving nulls if non-existant in the left.

LEFT: Inlcudes all records from the left, droping values from the right if non-existent in the left, or leaving nulls if non-existant in the right.

FULL: Combination of Left and Right Join, leaving nulls where data is missing in one table, and not droping any data.

CROSS: returns all pairs from two tables, does not have an on or using clause.

Union #

for concatenating rows of data with the same columns

Union: returns only unique records, does not include duplicates.

Union All: returns all records(including duplicates)

Intersect #

Intersect: returns only records appearing in both tables

Execpt #

Except: returns only records not in the second table

Self-Joins #

Semi-Join: Filters based on results of a subquery. Does not have direct sql syntax. This type of join is achieved through a subquery in the where statement.

Anti-Join: Similar to the Semi-join, but using a not modifier. This is particularly useful for debugging situations.

Subqueries #

This is where I have really stepped up my sql game. I was able to get practice writing more complex queries. I also learned about different methods of joining tables together.

WHERE #

Subqueries are commonly found in the where clause to filter data. Below is an example given in the course to select only the Asian countries with below average fertility rate from the states table.

SELECT
   name,
   fert_rate
FROM
    states
WHERE
    continent = 'Asia'
AND fert_rate <
        (SELECT AVG(fert_rate)
         FROM states;)

SELECT #

Subqueries can be found in the SELECT clause to create new columns of data. This is a different technique than I have used in the past. Previously I have only used GROUPBY statements to get this effect. I can see where this can be really useful because it is not constrained by aggregations any data point can be pulled in with this tecnhique.

SELECT DISTINCT
    continent,
    (SELECT
        COUNT(*)
     FROM
        states
     WHERE
        prime_ministers.continent = states.continent
    ) AS countries_num

From Prime Ministers

FROM #

subqueries found in the FROM clause can be very helpful to create a new dataset from an existing table. I find these the easiest to read as it is not much different than creating a new table. Again this can be very powerful in creating new columns that were not easily available otherwise.

SELECT DISTINCT
    monarchs.continent,
    subquery.max_perc

FROM
    monarchs,
    (SELECT
        continent,
        MAX(women_parli_perc) AS max_perc

    FROM
        states

    GROUP BY
        continent
    ) as subquery

WHERE
    monarchs.continent = subquery.continent

ORDER BY
    continent;

ON #

Challenge Problem 1 This problem was the one that had me more stumped than any other problem in the course. I found the subquery inside the on statement very confusing to understand. In this question we are joining the countries table to a subquery what yields country codes of countries with offial languages from the languages table.

SELECT DISTINCT
    c.name,
    e.total_investment,
    e.imports

FROM
    countries as c
LEFT JOIN
    economies as e
    ON c.code = e.code

    AND c.code in (
    SELECT
        l.code
    FROM
        languages as l
    WHERE
        official = true
    )

WHERE
    c.region = 'Central America'
AND e.year = 2015

ORDER BY
    c.name asc;

background tasks in python

2017-09-16T00:00:00Z

I have tried most of the different methods in the past and found that copying and pasting the threadpoolexecutor example or the processpoolexecutor example from the standard library documentation to be the most reliable. Since this is often something that I stuff in the back of a utility module of a library it is not something that I write often enough to be familiar with, which makes it both hard to write and hard to read and debug. If you are looking for a good overview of the difference concurrency Raymond Hettinger has a great talk about the difference between the various different methods, when to use them and why.

Recently a new python library was released to make running tasks in the background very simple. The background project by Kenneth Reitz is a high level implementation of python 3’s ThreadPoolExecutor. I have been playing around with this project over the last week and I will say that this is definitely the simplest way to run background tasks in python by far. It really simplifes the syntax and lets me focus on my job rather than implementing custom concurrent code that is more difficult to debug.

Background #

I have pulled the latest version of the project in Sept 2017. I found that it had some updates that were important to pass *args and **kwargs compared to the pypi version.

import time
import background as bg

%load_ext watermark
%watermark -d -v -p background

2017-09-16

CPython 3.6.2
IPython 6.1.0

background n�

Define Worker Functions #

Each of these worker functions takes 1s to run, simulating a moderately long calculation that we need to do many times over.

def work():
    time.sleep(1)
    return 1

@bg.task
def bg_work():
    time.sleep(1)
    return 1

Run the Worker Functions #

Blocking function #

This function is blocking each time the function runs, thus taking 1 second to run for each calculation. The example below took exactly 100 s to run 100 calculations. Depending on your use case this may not be fast enough. If the calculations do not rely on the global state

%%time
for _ in range(100):
    work()

Wall time: 1min 40s

Reaction #

I know what half of you are saying to yourselfs..

!!What!! that took 100 s, by now my users have already sent a dozen messages and filed an issue that my feature is down

and the other half

Seriously that wasnt even enough time to grab a coffee. Any real time consuming analysis takes at least 3 dats 14 hours 159 seconds before I start to care about concurrency

To you I say… I am impatient and I got other things to do rather than wait on this maching to finish its work. Let’s get into this concurrency stuff.

Background Function #

This function spins off worker processes and runs much faster. By default background sets the number of processes to the number of cpu cores available, Therefore this function should run in n/4 + (inefficiency). Here we see that the result is just over 13 s.

Note:Since there is a bit of inefficiency added by needing to handle all of the threads it is not exactly divided by the number of workers.

%%time
f_list = [bg_work() for _ in range(100)];

while not all([f.done() for f in f_list]):
    pass

Wall time: 13.1 s

Reaction #

I know what your saying this time.

really a 7.6x improvement… Is that really even woth the extra work.

Fine then lets crank it up to 11!

Lots of Background #

lets set the number of background processes to a value just higher to than the number of workers we need to run in order to start them all simultaneously. With this simple example that is not very CPU intensive we see the result is just over the amount of time that it takes to run 1 worker.

bg.n = 110

%%time
f_list = [bg_work() for _ in range(100)];

while not all([f.done() for f in f_list]):
    pass

Wall time: 1.09 s

Reaction #

91x improvement by putting my calculations into a function, adding a decorator, and some checks, im in.

On Tap #

This week while taking it up to 11 I was enjoying a super thick and rich cup of El Salvador Finca Rosa from Onyx Coffee Labs. Check out their love for letting the bean speak for it self and producing a great cup.

Posts tagged: data Archive

💭 Taming file zoos: Data science with DuckDB database files - Al...

💭 hotel_bookings.csv

sqlite vacuum

💭 searching my thoughts locally

💭 Open source, not open contribution with Ben Johnson (Changelog...

💭 DjangoCon Europe 2023 | Use SQLite in production - YouTube

Set up minio bucket entrypoint

Here is what I got to work #

Now Configuring Litestream #

💭 benbjohnson/litestream: Streaming replication for SQLite.

install #

why-is-postgres-default

No one ever got fired for choosing PostgreSQL #

Tutorials use sqlite #

SQLite runs in memory #

Less exposure #

SQLite is easy to replicate #

SQLite Does everything of what I need in an application db #

SQLite runs on the edge #

SQLite probably does what you need it to #

You probably already know how to maintain it #

Need High Concurrent writes #

Need Remote connection string #

💭 Why I Built Litestream - Litestream

💭 I'm All-In on Server-Side SQLite · The Fly Blog

💭 LiteFS Cloud: Distributed SQLite with Managed Backups · The Fl...

💭 SQLite FTS5 Extension

💭 simonw/datasette-render-markdown: Datasette plugin for renderi...

💭 `ValueError: Constraint must have a name` in alembic 1.10.0 · ...

💭 Use Alembic Check to check for possible upgrades

💭 DuckDB vs. MotherDuck — should you switch to the cloud version...

💭 s3-tree · PyPI

💭 kndndrj/nvim-dbee: Interactive database client for neovim

💭 sqlite-utils now supports plugins

💭 Harlequin SQL IDE - DuckDB

💭 Python API - DuckDB

💭 SQL on Pandas - DuckDB

💭 Full-text search - Datasette documentation

💭 sqlite-utils command-line tool - sqlite-utils

💭 sqlite-utils command-line tool - sqlite-utils

Zev Averbach Interview

Minimal Kedro Pipeline

Minimal Kedro Pipeline #

Installation #

Caveats #

Usage #

Implemantation #

Directory structure #

setup.py #

mini_kedro_pipeline.py #

Kedro in scripts #

Share your pipelines #

Blog Data With Python

Markdown #

Frontmatter #

Fill in the blanks #

Fast #

Kedro - My Data Is Not A Table

DataFrames are the heart of most pipelines #

Sometimes datasets are not tables #

Cataloging Pickle #

Loading the dataset #

Gracefully adopt kedro, the catalog

Why use kedro catalog? #

2 Ways to Gracefully adopt the catalog #

1. Adding to the catalog with the code api #

2. Creating a catalog config file #

Adhoc adding to an existing kedro catalog #

How to find things in your kedro catalog

The Catalog #

Kick start a toy project #

Create some catalog #

Some examples of common regex uses #

Let’s Search this thing #

list out all of the luxy-yahts #

List out data by layer #

raw #

pri #

^term #

`^term` #

`term$` #

`term1.*term2` #

`*args` scares the crap out of me #

Simulating the pipeline run with `**kwargs` #

Do something with those `results()` #