Posts tagged: data

sqlite-utils now supports plugins sqlite-utils 3.34 is out with a major new feature: support for plugins. sqlite-utils is my combination Python library and command-line tool for manipulating SQLite databases. It recently celebrated... Simon Willison’s Weblog · simonwillison.net [1] As the title states sqlite-utils now supports plugins. I dug in just a bit and Simon implemented this completely with entrypoints, no framework or library at all. Note This post is a thought [2]. It’s a short note that I make about someone else’s content online #thoughts References: [1]: https://simonwillison.net/2023/Jul/24/sqlite-utils-plugins/ [2]: /thoughts/

External Link duckdb.org [1] Harlequin is a pretty sweet example of what textual can be used to create. Its a terminal based sql ide for DuckDB. Note This post is a thought [2]. It’s a short note that I make about someone else’s content online #thoughts References: [1]: https://duckdb.org/docs/guides/sql_editors/harlequin [2]: /thoughts/

[1] To persist data in duckdb you need to first make a connection to a duck db database. con = duckdb.connect('file.db') Then work off of the connection con rather than duckdb. con.sql('CREATE TABLE test(i INTEGER)') con.sql('INSERT INTO test VALUES (42)') # query the table con.table('test').show() # explicitly close the connection con.close() Note This post is a thought [2]. It’s a short note that I make about someone else’s content online #thoughts References: [1]: /static/https://duckdb.org/docs/api/python/overview.html [2]: /thoughts/

Redirecting… duckdb.org [1] duckdb can just query any pandas dataframe that is in memory. I tried running it against a list of objects and got this error. Great error message that gives me supported types right in the message. Make sure that "posts" is either a pandas.DataFrame, duckdb.DuckDBPyRelation, pyarrow Table, Dataset, RecordBatchReader, Scanner, or NumPy ndarrays with supported format Note This post is a thought [2]. It’s a short note that I make about someone else’s content online #thoughts References: [1]: https://duckdb.org/docs/guides/python/sql_on_pandas [2]: /thoughts/

Full-text search - Datasette documentation docs.datasette.io [1] Enable full-text search in sqlite using sqlite-utils. $ sqlite-utils enable-fts mydatabase.db items name description Note This post is a thought [2]. It’s a short note that I make about someone else’s content online #thoughts References: [1]: https://docs.datasette.io/en/latest/full_text_search.html#enabling-full-text-search-for-a-sqlite-table [2]: /thoughts/

sqlite-utils command-line tool - sqlite-utils sqlite-utils.datasette.io [1] I want to like jq, but I think Simon is selling me on sqlite, maybe its just me but this looks readable, hackable, editable, memorizable. Everytime I try jq, and its 5 minutes fussing with it just to get the most basic thing to work. I know enough sql out of the gate to make this work off the top of my head curl https://thoughts.waylonwalker.com/posts/ | sqlite-utils memory - 'select title, message from stdin where stdin.tags like "%python%"' | jq Note This post is a thought [2]. It’s a short note that I make about someone else’s content online #thoughts References: [1]: https://sqlite-utils.datasette.io/en/stable/cli.html#querying-data-directly-using-an-in-memory-database [2]: /thoughts/

sqlite-utils command-line tool - sqlite-utils sqlite-utils.datasette.io [1] insert a json array directly into into sqlite with sqlite-utils. echo '{"name": "Cleo", "age": 4}' | sqlite-utils insert dogs.db dogs - Note This post is a thought [2]. It’s a short note that I make about someone else’s content online #thoughts References: [1]: https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data [2]: /thoughts/

Kedro rich is a very new and unstable (it’s good, just not ready) plugin for kedro to make the command line prettier. Install kedro rich # [1] There is no pypi package yet, but it’s on github. You can pip install it with the git [2] url. pip install git+https://github.com/datajoely/kedro-rich Kedro run # [3] You can run your pipeline just as you normally would, except you get progress bars and pretty prints. kedro run [4] Kedro catalog # [5] Listing out catalog entries from the command line now print out a nice pretty table. kedro catalog list [6] Give it a star # [7] Go to the GitHub repo [8] and give it a star, Joel deserves it. References: [1]: #install-kedro-rich [2]: /glossary/git/ [3]: #kedro-run [4]: https://images.waylonwalker.com/kedro-rich-run.png [5]: #kedro-catalog [6]: https://images.waylonwalker.com/kedro-rich-catalog-list.png [7]: #give-it-a-star [8]: https://github.com/datajoely/kedro-rich

I keep my nodes short and sweet. They do one thing and do it well. I turn almost every DataFrame transformation into its own node. It makes it must easier to pull catalog entries, than firing up the pipeline, running it, and starting a debugger. For this reason many of my nodes can be built from inline lambdas. Examples # [1] Here are two examples, the first one lambda x: x is sometimes referred to as an identity function. This is super common to use in the early phases of a project. It lets you follow standard layering conventions, without skipping a layer, overthinking if you should have the layer or not, and leaves a good placholder to fill in later when you need it. Many times I just want to get the data in as fast as possible, learn about it, then go back and tidy it up. from kedro.pipeline import node my_first_node = node( func=lambda x: x, inputs='raw_cars', output='int_cars', tags=['int',] ) my_first_node = node( func=lambda cars: cars[['mpg', 'cyl', 'disp',]].query('disp>200'), inputs='raw_cars', output='int_cars', tags=['pri',] ) Note: try not to take the idea of a one liner too far. If your one line function wraps several lines down it probably deserv...

As you work on your kedro projects you are bound to need to add more dependencies to the project eventually. Kedro uses a fantastic command pip-compile under the hood to ensure that everyone is on the same version of packages at all times, and able to easily upgrade them. It might be a bit different workflow than what you have seen, let’s take a look at it. git status # [2] Before you start mucking around with any changes to dependencies make sure that your git status is clean. I’d even reccomend starting a new branch for this, and if you are working on a team potentially submit this as its own PR for clarity. git status git checkout main git checkout -b add-rich-dependency requirements.in # [3] New requirements get added to a requirements.in file. If you need to specify an exact version, or a minimum version you can do that, but if all versions generally work you can leave it open. # requirements.in rich Here I added the popular rich package to my requirements.in file. Since I am ok with the latest version I am not going to pin anything, I am going to let the pip resolver pick the latest version that does not conflict with any of my dependencies for me. build-reqs # [4] ...

I am a huge believer in practicing your craft. Professional athletes spend most of their time honing their skills and making themsleves better. In Engineering many spend nearly 0 time practicing. I am not saying that you need to spend all your free time practicing, but a few minutes trying new things can go a long way in how you understand what you are doing and make a hue impact on your long term productivity. What is Kedro [1] Start practicing # [2] practice building pipelines with #kedro today Go to your playground directory, and if you don’t have one, make one. cd ~/playground get pipx # [3] Install pipx in your system python. This is one of the very few, and possibly the only python library that deserves to be installed in your system directory, primarily because its used to sanbox clis in their own virtual environment [4] automatically for you. pip install pipx make a new project # [5] From inside your playground directory, start your new kedro project. This is quite simple and painless. So much so that if you mess this one up doing something wild, it might be easier to make a new one that fixing the wild one. pipx run kedro new # answer the questions it asks I u...

I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a kedro project to play with so I did what any good kedroid would do, I went to my command line and ran pipx run kedro new --starter spaceflights But what I got back was not what I expected! Fatal error from pip prevented installation. Full pip output in file: /home/walkers/.local/pipx/logs/cmd_2022-01-01_20.42.16_pip_errors.log Some possibly relevant errors from pip install: ERROR: Could not find a version that satisfies the requirement kedro (from versions: none) ERROR: No matching distribution found for kedro Error installing kedro. This is weird, why cant I run kedro new with pipx? Lets try pip. pip install kedro Same issue. ERROR: Could not find a version that satisfies the requirement kedro (from versions: none) ERROR: No matching distribution found for kedro What is Kedro [1] Curious what kedro is? Check out this article. What’s up # [2] wrong python version The issue is that kedro only runs on up to python 3.8, and on Ubuntu 21.10 when you apt install python3 you get python 3.9 and the standard repos don’t have an old enough version to run kedro. How to fix this? # [3] Theres a couple of wa...