As the title states sqlite-utils now supports plugins. I dug in just a bit and Simon implemented this completely with entrypoints, no framework or library at all.
Posts tagged: data
All posts with the tag "data"
70 posts
latest post 2025-06-09
Publishing rhythm
Harlequin is a pretty sweet example of what textual can be used to create. Its a terminal based sql ide for DuckDB.
To persist data in duckdb you need to first make a connection to a duck db database.
con = duckdb.connect('file.db')
Then work off of the connection con rather than duckdb.
duckdb can just query any pandas dataframe that is in memory.
I tried running it against a list of objects and got this error. Great error message that gives me supported types right in the message.
Enable full-text search in sqlite using sqlite-utils.
I want to like jq, but I think Simon is selling me on sqlite, maybe its just me but this looks readable, hackable, editable, memorizable. Everytime I try jq, and its 5 minutes fussing with it just to get the most basic thing to work. I know enough sql out of the gate to make this work off the top of my head
insert a json array directly into into sqlite with sqlite-utils.
Kedro rich is a very new and unstable (it’s good, just not ready) plugin for kedro to make the command line prettier.
There is no pypi package yet, but it’s on github. You can pip install it with the git url.
pip install git+https://github.com/datajoely/kedro-rich
Kedro run #
You can run your pipeline just as you normally would, except you get progress bars and pretty prints.
kedro run 
Kedro catalog #
Listing out catalog entries from the command line now print out a nice pretty table.
...
I keep my nodes short and sweet. They do one thing and do it well. I turn almost every DataFrame transformation into its own node. It makes it must easier to pull catalog entries, than firing up the pipeline, running it, and starting a debugger. For this reason many of my nodes can be built from inline lambdas.
Here are two examples, the first one lambda x: x is sometimes referred to as an identity function. This is super common to use in the early phases of a project. It lets you follow standard layering conventions, without skipping a layer, overthinking if you should have the layer or not, and leaves a good placholder to fill in later when you need it.
Many times I just want to get the data in as fast as possible, learn about it, then go back and tidy it up.
from kedro.pipeline import node my_first_node = node( func=lambda x: x, inputs='raw_cars', output='int_cars', tags=['int',] ) my_first_node = node( func=lambda cars: cars[['mpg', 'cyl', 'disp',]].query('disp>200'), inputs='raw_cars', output='int_cars', tags=['pri',] )
Note: try not to take the idea...
As you work on your kedro projects you are bound to need to add more dependencies to the project eventually. Kedro uses a fantastic command pip-compile under the hood to ensure that everyone is on the same version of packages at all times, and able to easily upgrade them. It might be a bit different workflow than what you have seen, let’s take a look at it.
Before you start mucking around with any changes to dependencies make sure that your git status is clean. I’d even reccomend starting a new branch for this, and if you are working on a team potentially submit this as its own PR for clarity.
git status git checkout main git checkout -b add-rich-dependency
requirements.in #
New requirements get added to a requirements.in file. If you need to specify an exact version, or a minimum version you can do that, but if all versions generally work you can leave it open.
# requirements.in rich
Here I added the popular rich package to my requirements.in file. Since I am ok with the latest version I am not going to pin anything,...
...
I am a huge believer in practicing your craft. Professional athletes spend most of their time honing their skills and making themsleves better. In Engineering many spend nearly 0 time practicing. I am not saying that you need to spend all your free time practicing, but a few minutes trying new things can go a long way in how you understand what you are doing and make a hue impact on your long term productivity.
practice building pipelines with #kedro today
Go to your playground directory, and if you don’t have one, make one.
...
I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a kedro project to play with so I did what any good kedroid would do, I went to my command line and ran
pipx run kedro new --starter spaceflights
But what I got back was not what I expected!
Fatal error from pip prevented installation. Full pip output in file: /home/walkers/.local/pipx/logs/cmd_2022-01-01_20.42.16_pip_errors.log Some possibly relevant errors from pip install: ERROR: Could not find a version that satisfies the requirement kedro (from versions: none) ERROR: No matching distribution found for kedro Error installing kedro.
This is weird, why cant I run kedro new with pipx? Lets try pip.
pip install kedro
Same issue.
...
kedro catalog create
I use kedro catalog create to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key.
nvim conf 2021 | IDE's are slow | Waylon Walker
Use a graphical IDE if it works for you.
vim is so well integrated into the terminal, take advantage
...
Kedro-Broken-Urls
Broken Urls # https://github.com/josephhaaga) [ ] https://example.com/file.h5 https://raw.githubusercontent.com/kedro-org/kedro/develop/static/img/pipeline_visualisation.png https://example.com/file.txt https://github.com/jmespath/jmespath.py. https://github.com/tsanikgr) https://example.com/file.csv https://kedro.readthedocs.io/en/latest/04_user_guide/15_hooks.html https://kedro.readthedocs.io/en/stable/07_extend_kedro/04_hooks.html https://github.com/EbookFoundation/free-programming-books/blob/master/books/free-programming-books.md#python https://github.com/quantumblacklabs/private-kedro/blob/develop/docs/source/04_user_guide/04_data_catalog.md http://example.com/api/test https://example.com/file.parquet https://kedro.readthedocs.io/en/stable/11_faq/01_faq.html#how-do-i-upgrade-kedro https://example.com/file.xlsx https://www.datacamp.com/community/tutorials/docstrings-python https://github.com/mmchougule) https://example.com/file.tf...
Setting Parameters in kedro
Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files.
...
Writing your first kedro Nodes
Before we jump in with anything crazy, let’s make some nodes with some vanilla data structures.
You will need to import node from kedro.pipeline to start creating nodes.
...
Running your Kedro Pipeline from the command line
Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli.
kedro Virtual Environment
Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment.
I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don’t have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool.
...
Kedro Pipeline Create
Kedro pipeline create is a command that makes creating new pipelines much easier. There is much less boilerplate that you need to write yourself.
The kedro cli comes with the following command to scaffold out new pipelines. Note that it will not add it to your pipeline_registry, to be covered later, you will need to add it yourself.
...