Posts tagged: sqlite

💭 Using Litestream to Restore My Database for Easy Development

2025-09-29T18:49:12Z

Using Litestream to Restore My Database for Easy Development | Nic Payne

Litestream see [[using-litestream-to-backup-quadtasks-sqlite-db]] for how I setup litestream replication for [[quadtask]] I have the entrypoint to my app contai

pype.dev

I really like how well the local dev is setup to run off of production data here. I’ll use this as a reminder that I need to set up lite stream on a few of my projects that it’s missing from and include a nice sync prod data Posts tagged: justfile recipe.

Litestreams interface always throws me for a loop. It works fantastic, but the global config stored in /etc and some of the commands break my brain. It’s not you it’s me.

Using real data when you can is goated. Fake data is so often a perfect example of what someone thinks the backend should look like and does not include things that users actually do, running pipelines for days, or setting titles to paragraphs worth of text. Obviously this is not possible everywhere and the more sensitive your data the harder that process becomes.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Meet Gor on X: "SQLite added generated columns in 3.31 (around...

2025-09-16T13:54:52Z

External Link

X (formerly Twitter) · x.com

I’ve never tried generated columns, but it is something that I’m interested in. It is nice to have things computed and in the database if you ever need to query on them. My brain is now churning with possibilities and quesions, does this work with Alembic? How does this compare to views?

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Database Remote-Copy Tool For SQLite (draft)

2024-10-05T20:56:49Z

Database Remote-Copy Tool For SQLite (draft)

Neat new SQLite utilities often show up in branches of the SQLite repository. Here's a new one from last month: sqlite3-rsync, providing tools for efficiently creating and updating copies of …

Simon Willison’s Weblog · simonwillison.net

Simon shared a really cool new utility tool for sqlite ispired by rsync. It checks hashes of each sqlite page and syncs pages. So if nothing in the database has changed it will only require 0.5% the bandwidth as a copy would.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Sqlite-jiff

2024-07-24T03:12:14Z

sqlite-jiff

I linked to the brand new Jiff datetime library yesterday. Alex Garcia has already used it for an experimental SQLite extension providing a timezone-aware jiff_duration() function - a useful new …

Simon Willison’s Weblog · simonwillison.net

Sqlite is getting rust extensions now, and datetimes make it totally worth if if they work well and and fast, two things that don’t always go together in datetime libraries

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sql - How can I list the tables in a SQLite database file that...

2024-04-18T01:13:59Z

External Link

stackoverflow.com

I learned about the sqlite_master table from this stack overflow answer. This helps make a lot of sense to how sqlite works. The master table contains all the sqlite objects and the sql to create them.

The .tables, and .schema “helper” functions don’t look into ATTACHed databases: they just query the SQLITE_MASTER table for the “main” database. Consequently, if you used

sqlite3 database.db "SELECT * from sqlite_master;"

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sql - SQLite: COUNT slow on big tables - Stack Overflow

2024-04-01T20:59:40Z

External Link

stackoverflow.com

Another interesting option for slow count queries in sqlite.

If you haven’t DELETEd any records, doing:

SELECT MAX(ROWID) FROM "table" LIMIT 1;

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Optimizing SQLite for servers

2024-04-01T20:55:23Z

Optimizing SQLite for servers

SQLite is often misconceived as a "toy database", only good for mobile applications and embedded systems because it's default configuration is optimized for embedded use cases, so most ...

Sylvain Kerkour · kerkour.com

Very interesting article by Sylvain, suggested by Simon Willison.

Definitely some things that I want to come back and try later on.

Here is the TLDR of the whole post

PRAGMA journal_mode = WAL;
PRAGMA busy_timeout = 5000;
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 1000000000;
PRAGMA foreign_keys = true;
PRAGMA temp_store = memory;

This is interesting, and something I need to consider. I definitely have an application with slow count queries. I am not sure how to make it better as its not a full count(*) so a count table doesn’t work, nor does counting by index.

I might need to have a table of cached results, and if a write matches the counter increase it, or update all counters on write.

COUNT queries are slow SQLite doesn’t keep statistics about its indexes, unlike PostgreSQL, so COUNT queries are slow, even when using a WHERE clause on an indexed field: SQLite has to scan for all the matching records.

One solution is to use a trigger on INSERT and DELETE that updates a running count in a separate table then query that separate table to find the latest count.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 searching my thoughts locally

2024-03-07T23:42:00Z

![[None]]

First I need to fetch my thoughts from the api, and put it in a local sqlite database using sqlite-utils.

fthoughts () {
    # fetch thoughts
    curl 'https://thoughts.waylonwalker.com/posts/waylonwalker/?page_size=9999999999' | sqlite-utils insert ~/.config/thoughts/database2.db post --pk=id --alter --ignore -
}

Now that I have my posts in a local sqlite database I can use sqlite-utils to enable full text search and populate the full text search on the post table using the title message and tags columns as search.

sthoughts () {
    # search thoughts
    # sqlite-utils enable-fts ~/.config/thoughts/database2.db post title message tags
    # sqlite-utils populate-fts ~/.config/thoughts/database2.db post title message tags
    sqlite-utils search ~/.config/thoughts/database2.db post "$*" | ~/git/thoughts/format_thought.py | bat --style=plain --color=always --language=markdown
}

alias st=sthoughts

Now I am ready to search my thoughts, which is a tiny blog format that I created mostly for leaving my own personal comment on web pages, so most of them have a link to some other online content, and their title is based on the authors title.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Open source, not open contribution with Ben Johnson (Changelog...

2023-10-05T01:11:19Z

Open source, not open contribution with Ben Johnson (Changelog Interviews #433)

This week we're talking with Ben Johnson. Ben is known for his work on BoltDB, his work in open source, and as a freelance Go developer. Late January when Ben open sourced his newest project Litest...

Changelog · changelog.com

Ben Johnson was on the Changelog a few years back covering his work on litestream, and talks about why he chose to go open source, but not open contribution.

You should have a good reason to move off of sqlite.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 DjangoCon Europe 2023 | Use SQLite in production - YouTube

2023-10-03T13:22:19Z

Very inspiring talk, TLDR, you probably don’t need a database server. sqlite will probably be faster, simpler to maintain, and simpler to test your application.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 benbjohnson/litestream: Streaming replication for SQLite.

2023-10-01T02:19:18Z

GitHub - benbjohnson/litestream: Streaming replication for SQLite.

Streaming replication for SQLite. Contribute to benbjohnson/litestream development by creating an account on GitHub.

GitHub · github.com

    `litestream` is a sick cli tool for steaming replicas of sqlite.  It automatically does daily snapshots, and streams all of the writes to the replica live.

install #

Install is fast using installer, no compilation, just copy the binary and run.

curl https://i.wayl.one/benbjohnson/litestream

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Why I Built Litestream - Litestream

2023-09-30T16:16:47Z

Why I Built Litestream - Litestream

Despite an exponential increase in computing power, our applications require more machines than ever because of architectural decisions made 25 years ago. You can eliminate much of your complexity ...

litestream.io

As applications scale to the edge, to put compute as close to the user as possible, database queries back to the master node get slower and slower. Enter sqlite replication, put the database wtih the application code and replicate from master.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 I'm All-In on Server-Side SQLite · The Fly Blog

2023-09-30T14:54:33Z

I'm All-In on Server-Side SQLite

Ben Johnson has joined Fly.io

Fly · fly.io

SQLite is the next big database trend. with more horizontal scaling, close to user read heavy applications, having your database in the same application stack makes a lot of sense. Tools like litestream are going to enable global distribution in an impressive way.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 LiteFS Cloud: Distributed SQLite with Managed Backups · The Fl...

2023-09-30T14:49:41Z

LiteFS Cloud: Distributed SQLite with Managed Backups

Documentation and guides from the team at Fly.io.

Fly · fly.io

Fly.io’s solution to sqlite managed backups.I definitely want to look into this a bit, but moreso the tech under the hook litestream.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 SQLite FTS5 Extension

2023-08-21T13:33:24Z

sqlite has 3 different tokenizers, porter, ascii, trigram.

These can be used with sqlite-utils.

sqlite-utils enable-fts --tokenize porter database.db post title message tags

And with the python api.

db = Database('database.db')
db["post"].enable_fts(
                ["title", "message", "tags"], create_triggers=True, tokenize="trigram"
            )
posts = list(db["post"].search(search))

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite_utils Python library - sqlite-utils

2023-08-20T14:32:39Z

sqlite_utils Python library - sqlite-utils

sqlite-utils.datasette.io

sqlite-utils is primarily a cli tool for sqlite operations such as enabling full text search, and executing searches, but it also has a nice python api that is exposed and pretty straightforward to use.

from sqlite_utils import Database
db = Database("database.db")
db["post"].enable_fts(["title", "message", "tags])
db["post"].search("water")

This returns a generator object that you can iterate over the row objects with.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 simonw/datasette-render-markdown: Datasette plugin for renderi...

2023-08-16T00:48:07Z

GitHub - simonw/datasette-render-markdown: Datasette plugin for rendering Markdown

Datasette plugin for rendering Markdown. Contribute to simonw/datasette-render-markdown development by creating an account on GitHub.

GitHub · github.com

datasette really does everything doesn’t it!

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Full-text search - Datasette documentation

2023-07-28T14:59:37Z

Full-text search - Datasette documentation

docs.datasette.io

Enable full-text search in sqlite using sqlite-utils.

$ sqlite-utils enable-fts mydatabase.db items name description

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite-utils command-line tool - sqlite-utils

2023-07-28T14:59:37Z

sqlite-utils command-line tool - sqlite-utils

sqlite-utils.datasette.io

I want to like jq, but I think Simon is selling me on sqlite, maybe its just me but this looks readable, hackable, editable, memorizable. Everytime I try jq, and its 5 minutes fussing with it just to get the most basic thing to work. I know enough sql out of the gate to make this work off the top of my head

curl  https://thoughts.waylonwalker.com/posts/ | sqlite-utils memory - 'select title, message from stdin where stdin.tags like "%python%"' | jq

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite-utils command-line tool - sqlite-utils

2023-07-28T14:59:37Z

sqlite-utils command-line tool - sqlite-utils

sqlite-utils.datasette.io

insert a json array directly into into sqlite with sqlite-utils.

echo '{"name": "Cleo", "age": 4}' | sqlite-utils insert dogs.db dogs -

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts