Posts tagged: data

💭 Taming file zoos: Data science with DuckDB database files - Al...

2025-06-09T13:31:30Z

The ability to query s3 buckets so seamless looks like such a pleasure to work with if you have a use case for that. Kedro catalog takes care of this most of the time for me, but I wonder if there are some cross project searching use cases I might find for this.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 hotel_bookings.csv

2025-03-15T22:32:44Z

hotel_bookings.csv

Discover what actually works in AI. Join millions of builders, researchers, and labs evaluating agents, models, and frontier technology through crowdsourced benchmarks, competitions, and hackathons.

kaggle.com

nice dataset to use for example / test projects. I’m using it to play with duckdb currently.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

sqlite vacuum

2024-04-16T21:31:24Z

Today I learned how to VACUUM a sqlite database and cut its size in about half. It’s a database that I have had running for quite awhile and has some decent traffic on it.

Why is it important to do a VACUUM? In short its becuase the file system gets fragmented with as data is updated. On delete the files are removed from the database and marked as available for reuse in the filesystem, but the space is not reclaimed.

To VACUUM a database, run the following sql command. You can do it right form the sqlite shell by running sqlite3.

You will need about double the current size of the database as free space to do the VACUUM, you need space for a full copy, journaling or write ahead logs, and the existing database.

VACUUM;

The docs are fantastic for vacuum.

💭 searching my thoughts locally

2024-03-07T23:42:00Z

![[None]]

First I need to fetch my thoughts from the api, and put it in a local sqlite database using sqlite-utils.

fthoughts () {
    # fetch thoughts
    curl 'https://thoughts.waylonwalker.com/posts/waylonwalker/?page_size=9999999999' | sqlite-utils insert ~/.config/thoughts/database2.db post --pk=id --alter --ignore -
}

Now that I have my posts in a local sqlite database I can use sqlite-utils to enable full text search and populate the full text search on the post table using the title message and tags columns as search.

sthoughts () {
    # search thoughts
    # sqlite-utils enable-fts ~/.config/thoughts/database2.db post title message tags
    # sqlite-utils populate-fts ~/.config/thoughts/database2.db post title message tags
    sqlite-utils search ~/.config/thoughts/database2.db post "$*" | ~/git/thoughts/format_thought.py | bat --style=plain --color=always --language=markdown
}

alias st=sthoughts

Now I am ready to search my thoughts, which is a tiny blog format that I created mostly for leaving my own personal comment on web pages, so most of them have a link to some other online content, and their title is based on the authors title.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Open source, not open contribution with Ben Johnson (Changelog...

2023-10-05T01:11:19Z

Open source, not open contribution with Ben Johnson (Changelog Interviews #433)

This week we're talking with Ben Johnson. Ben is known for his work on BoltDB, his work in open source, and as a freelance Go developer. Late January when Ben open sourced his newest project Litest...

Changelog · changelog.com

Ben Johnson was on the Changelog a few years back covering his work on litestream, and talks about why he chose to go open source, but not open contribution.

You should have a good reason to move off of sqlite.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 DjangoCon Europe 2023 | Use SQLite in production - YouTube

2023-10-03T13:22:19Z

Very inspiring talk, TLDR, you probably don’t need a database server. sqlite will probably be faster, simpler to maintain, and simpler to test your application.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Set up minio bucket entrypoint

2023-10-01T20:11:51Z

I recently se tup minio object storage in my homelab for litestream sqlite backups. The litestream quickstart made it easy to get everything up and running on localhost, but I hit a wall when dns was involved to pull it from a different machine.

Here is what I got to work #

First I had to configure the Key ID and Secret Access Key generated in the minio ui.

❯ aws configure
AWS Access Key ID [****************VZnD]:
AWS Secret Access Key [****************xAm8]:
Default region name [us-east-1]:
Default output format [None]:

Then set the the s3 signature_version to s3v4.

aws configure set default.s3.signature_version s3v4

Now when I have minio running on https://my-minio-endpoint.com I can use the aws cli to access the bucket.

Note that https://my-minio-endpoint.com resolves to the bucket endpoint (default 9000) not the ui (default 9001).

aws --endpoint-url https://my-minio-endpoint.com s3 ls my_bucket

Now Configuring Litestream #

Litestream also accepts the endpoint argument via config. I could not get it to work just with the ui.

Note the aws configure step above is not required for litestream, only the aws cli.

dbs:
  - path: /path/to/database.db
    replicas:
      - url: s3://my_bucket/
        endpoint: https://my-minio-endpoint.com
        region: us-east-1
        access-key-id: ****************VZnD
        secret-access-key: ************************************xAm8

Now run a litestream replication.

litestream replicate -config litestream.yml
# or put the config in /etc/litestream.yml and just run replicate
litestream replicate

💭 benbjohnson/litestream: Streaming replication for SQLite.

2023-10-01T02:19:18Z

GitHub - benbjohnson/litestream: Streaming replication for SQLite.

Streaming replication for SQLite. Contribute to benbjohnson/litestream development by creating an account on GitHub.

GitHub · github.com

    `litestream` is a sick cli tool for steaming replicas of sqlite.  It automatically does daily snapshots, and streams all of the writes to the replica live.

install #

Install is fast using installer, no compilation, just copy the binary and run.

curl https://i.wayl.one/benbjohnson/litestream

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

why-is-postgres-default

2023-09-30T21:26:36Z

Serious question.

No one ever got fired for choosing PostgreSQL #

But, why. It’s the most loved db, right? Right? Maybe it’s time to rethink it.

Don’t get me wrong, if I need a relational db as a service, PostgreSQL is going to be my first choice, but why do I need to run a separate application for it?

Tutorials use sqlite #

Why is that? Because there is nothing else to stand up. Nothing else to maintain. And you probably already have it installed on just about anything that has a battery.

SQLite runs in memory #

Don’t need, or maybe don’t want to persist state. Run it in memory. This is a nice feature for running tests.

Less exposure #

SQLite is a file on your filesystem. It’s not a web service. It’s not a cloud service. Not that postgres is insecure, but it is one more endpoint that you have to think about securing.

this means that is probably also cheaper 🤑

SQLite is easy to replicate #

Want to run your new feature with prod data? Pull a replica or backup. It’s a file, you can cp, scp, rsync it whatever you have available.

SQLite Does everything of what I need in an application db #

SQLite has a rock solid set of features that covers everything that I need, and when it’s not I am probably thinking about pulling data into my application code and running something custom anyways.

SQLite runs on the edge #

Most applications are read heavy and light on writes. Services like turso have recognized this and built a business around it. They give you a master db that all writes go to, and read replicas that can run on the edge with your application code. You have a lot of users in different regions you can run your application close to them without suffering from query latency or replication complexity.

SQLite probably does what you need it to #

For a large number of use cases SQLite is probably the best thing to get you off the ground quick, and likely all that you will need. Consider using it before defaulting to postgres.

You probably already know how to maintain it #

If you know a bit of cli commands and remember that your database is just a file, this will feel very intuitive. If not you can probably poke around a file system gui and still make most of it work.

## create a database
touch database.db
## back it up
cp database.db database.db.bak
## drop the database
rm database.db
## restore the backup
cp database.db.bak database.db
## drop the backup
rm database.db.bak
## create an offsite backup
aws s3 cp database.db s3://my-backup-bucket/
scp database.db username@to_host:/remote/directory/
rsync -a database.db username@to_host:/remote/directory/

Need High Concurrent writes #

You might consider postgres.

Need Remote connection string #

You might consider postgres.

💭 Why I Built Litestream - Litestream

2023-09-30T16:16:47Z

Why I Built Litestream - Litestream

Despite an exponential increase in computing power, our applications require more machines than ever because of architectural decisions made 25 years ago. You can eliminate much of your complexity ...

litestream.io

As applications scale to the edge, to put compute as close to the user as possible, database queries back to the master node get slower and slower. Enter sqlite replication, put the database wtih the application code and replicate from master.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 I'm All-In on Server-Side SQLite · The Fly Blog

2023-09-30T14:54:33Z

I'm All-In on Server-Side SQLite

Ben Johnson has joined Fly.io

Fly · fly.io

SQLite is the next big database trend. with more horizontal scaling, close to user read heavy applications, having your database in the same application stack makes a lot of sense. Tools like litestream are going to enable global distribution in an impressive way.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 LiteFS Cloud: Distributed SQLite with Managed Backups · The Fl...

2023-09-30T14:49:41Z

LiteFS Cloud: Distributed SQLite with Managed Backups

Documentation and guides from the team at Fly.io.

Fly · fly.io

Fly.io’s solution to sqlite managed backups.I definitely want to look into this a bit, but moreso the tech under the hook litestream.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 SQLite FTS5 Extension

2023-08-21T13:33:24Z

sqlite has 3 different tokenizers, porter, ascii, trigram.

These can be used with sqlite-utils.

sqlite-utils enable-fts --tokenize porter database.db post title message tags

And with the python api.

db = Database('database.db')
db["post"].enable_fts(
                ["title", "message", "tags"], create_triggers=True, tokenize="trigram"
            )
posts = list(db["post"].search(search))

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 simonw/datasette-render-markdown: Datasette plugin for renderi...

2023-08-16T00:48:07Z

GitHub - simonw/datasette-render-markdown: Datasette plugin for rendering Markdown

Datasette plugin for rendering Markdown. Contribute to simonw/datasette-render-markdown development by creating an account on GitHub.

GitHub · github.com

datasette really does everything doesn’t it!

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 `ValueError: Constraint must have a name` in alembic 1.10.0 · ...

2023-08-05T01:25:07Z

`ValueError: Constraint must have a name` in alembic 1.10.0 · Issue #1195 · sqlalchemy/alembic

Describe the bug ValueError: Constraint must have a name in alembic 1.10.0. Expected behavior Migration succeeds. To Reproduce Please try to provide a Minimal, Complete, and Verifiable example, wit...

GitHub · github.com

After a nasty time with alembic upgrades, thoughts is about to get a new users table. This may have came from incorrectly setting up alembic for sqlite from the start, but I was able to fix the issue with this GitHub issue.

alembic sqlite ValueError: Constraint must have a name

The change I needed to make to get my migration to run.


        + batch_op.create_foreign_key('fk_post_author_id_user', 'user', ['author_id'], ['id'])

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Use Alembic Check to check for possible upgrades

2023-08-05T01:22:06Z

![[None]]

Since using alembic I have been just running out a new revision checking its content and deleting it if its empty, today I learned there is an alembic check command to check for operations that need to be created.

❯ alembic check
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
No new upgrade operations detected.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 DuckDB vs. MotherDuck — should you switch to the cloud version...

2023-07-31T22:36:12Z

DuckDB vs. MotherDuck — should you switch to the cloud version? | Kestra

Why and when to use MotherDuck over local DuckDB

kestra.io

duckdb is a new in process database that has been making its rounds in analytics for its high performance in those applications.

Mother duck is a centeralized server that brings manages storage, data sharing and an ide to duckdb.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 s3-tree · PyPI

2023-07-30T14:37:06Z

s3-tree

list s3 objects in tree-like format.

PyPI · pypi.org

Super useful way to show a tree view of an s3 bucket’s structure!

pip install s3-tree

s3-tree bucketname

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 kndndrj/nvim-dbee: Interactive database client for neovim

2023-07-29T01:08:20Z

GitHub - kndndrj/nvim-dbee: Interactive database client for neovim

Interactive database client for neovim. Contribute to kndndrj/nvim-dbee development by creating an account on GitHub.

GitHub · github.com

A neovim database client that I need to check out.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite-utils now supports plugins

2023-07-28T14:59:37Z

sqlite-utils now supports plugins

sqlite-utils 3.34 is out with a major new feature: support for plugins. sqlite-utils is my combination Python library and command-line tool for manipulating SQLite databases. It recently celebrated...

Simon Willison’s Weblog · simonwillison.net

As the title states sqlite-utils now supports plugins. I dug in just a bit and Simon implemented this completely with entrypoints, no framework or library at all.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts