Posts tagged: sql

💭 Sqlite-jiff

2024-07-24T03:12:14Z

I linked to the brand new Jiff datetime library yesterday. Alex Garcia has already used it for an experimental SQLite extension providing a timezone-aware jiff_duration() function - a useful new …

Simon Willison’s Weblog · simonwillison.net

Sqlite is getting rust extensions now, and datetimes make it totally worth if if they work well and and fast, two things that don’t always go together in datetime libraries

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sql - SQLite: COUNT slow on big tables - Stack Overflow

2024-04-01T20:59:40Z

External Link

stackoverflow.com

Another interesting option for slow count queries in sqlite.

If you haven’t DELETEd any records, doing:

SELECT MAX(ROWID) FROM "table" LIMIT 1;

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Optimizing SQLite for servers

2024-04-01T20:55:23Z

Optimizing SQLite for servers

SQLite is often misconceived as a "toy database", only good for mobile applications and embedded systems because it's default configuration is optimized for embedded use cases, so most ...

Sylvain Kerkour · kerkour.com

Very interesting article by Sylvain, suggested by Simon Willison.

Definitely some things that I want to come back and try later on.

Here is the TLDR of the whole post

PRAGMA journal_mode = WAL;
PRAGMA busy_timeout = 5000;
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 1000000000;
PRAGMA foreign_keys = true;
PRAGMA temp_store = memory;

This is interesting, and something I need to consider. I definitely have an application with slow count queries. I am not sure how to make it better as its not a full count(*) so a count table doesn’t work, nor does counting by index.

I might need to have a table of cached results, and if a write matches the counter increase it, or update all counters on write.

COUNT queries are slow SQLite doesn’t keep statistics about its indexes, unlike PostgreSQL, so COUNT queries are slow, even when using a WHERE clause on an indexed field: SQLite has to scan for all the matching records.

One solution is to use a trigger on INSERT and DELETE that updates a running count in a separate table then query that separate table to find the latest count.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite_utils Python library - sqlite-utils

2023-08-20T14:32:39Z

sqlite_utils Python library - sqlite-utils

sqlite-utils.datasette.io

sqlite-utils is primarily a cli tool for sqlite operations such as enabling full text search, and executing searches, but it also has a nice python api that is exposed and pretty straightforward to use.

from sqlite_utils import Database
db = Database("database.db")
db["post"].enable_fts(["title", "message", "tags])
db["post"].search("water")

This returns a generator object that you can iterate over the row objects with.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 python - SQLAlchemy ORDER BY DESCENDING? - Stack Overflow

2023-07-29T22:30:25Z

External Link

stackoverflow.com

How to sort results from a sqlalchemy based orm.

.order_by(model.Entry.amount.desc())

I needed this to enable paging on my thoughts api.

@post_router.get("/posts/")
async def get_posts(
    *,
    request: Request,
    session: Session = Depends(get_session),
    hx_request: Annotated[str | None, Header()] = None,
    accept: Annotated[str | None, Header()] = None,
    current_user: Annotated[User, Depends(try_get_current_active_user)],
    page_size: int = 10,
    page: int = 1,
) -> Posts:
    "get all posts"
    statement = (
        select(Post)
        .where(Post.published)
        .order_by(Post.id.desc())
        .limit(page_size)
        .offset((page - 1) * page_size)
    )
    posts = session.exec(statement).all()
    posts = Posts(__root__=posts)

    if isinstance(current_user, RedirectResponse):
        is_logged_in = False
    else:
        is_logged_in = True

    if hx_request and page == 1 and len(posts.__root__) == 0:
        return HTMLResponse('<ul id="posts"><li>No posts</li></ul>')
    if hx_request and len(posts.__root__) == 0:
        return HTMLResponse("")
    if not hx_request and len(posts.__root__) == 0:
        return ["no posts"]
    if hx_request:
        return templates.TemplateResponse(
            "posts.html",
            {
                "request": request,
                "config": config,
                "posts": posts,
                "md": md,
                "is_logged_in": is_logged_in,
                "page": page,
            },
        )

    if accept.startswith("text/html"):
        return templates.TemplateResponse(
            "base.html",
            {
                "request": request,
                "config": config,
                "posts": posts,
                "md": md,
                "is_logged_in": is_logged_in,
                "page": page,
            },
        )

    return posts

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite-utils now supports plugins

2023-07-28T14:59:37Z

sqlite-utils now supports plugins

sqlite-utils 3.34 is out with a major new feature: support for plugins. sqlite-utils is my combination Python library and command-line tool for manipulating SQLite databases. It recently celebrated...

Simon Willison’s Weblog · simonwillison.net

As the title states sqlite-utils now supports plugins. I dug in just a bit and Simon implemented this completely with entrypoints, no framework or library at all.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Column INSERT/UPDATE Defaults — SQLAlchemy 1.4 Documentation

2023-07-28T14:59:37Z

Column INSERT/UPDATE Defaults — SQLAlchemy 1.4 Documentation

docs.sqlalchemy.org

sqlalchemy server_defaults end up as defaults in the database when new values are inserted.

t = Table(
    "test",
    metadata_obj,
    Column("abc", String(20), server_default="abc"),
    Column("created_at", DateTime, server_default=func.sysdate()),
    Column("index_value", Integer, server_default=text("0")),
)

CREATE TABLE test (
    abc varchar(20) default 'abc',
    created_at datetime default sysdate,
    index_value integer default 0
)

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite-utils command-line tool - sqlite-utils

2023-07-28T14:59:37Z

sqlite-utils command-line tool - sqlite-utils

sqlite-utils.datasette.io

I want to like jq, but I think Simon is selling me on sqlite, maybe its just me but this looks readable, hackable, editable, memorizable. Everytime I try jq, and its 5 minutes fussing with it just to get the most basic thing to work. I know enough sql out of the gate to make this work off the top of my head

curl  https://thoughts.waylonwalker.com/posts/ | sqlite-utils memory - 'select title, message from stdin where stdin.tags like "%python%"' | jq

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Stepping Up My SQL Game

2018-03-25T00:00:00Z

In 2018 I transitioned from a Product Engineering (Mechanical) role to a Data Scientist Role. I entered this space with strong subject matter expertise with our products, our data, munging through data in pyhon, and data visualization in python. My sql skills were lacking to say the least. I had learned what I needed to know to get data from our relational databases, then use pandas to do any further analysis. Just run something like the following and you have data.

SELECT
    *
FROM
    Table
Where
    col_1 = 'col_1_filter'

This technique works great for small data sets that you only need to run once. There is no shame to pull in a big dataset and start munging with it in pandas to get some results, and make decisions. The problem becomes when your dataset becomes too big or you need to run the query on a frequent basis. Doing the aggregations on the server run much quicker, as it reduces the time spent in io. My longest running steps are currently io related. Reducing these steps have improved my workflow. At the point that I was getting server timeout errors, or using the same long running query in many places I would be searching for examples online, because I just did not have the experience with many more techniques. I decided it was time to put away the cheat sheets, step away from Stack Overflow, and improve my speed.

Why Learn SQL in 2018?? #

SQL is far from the hot topic in 2018, AI, Deep Learning, BIG data, Machine Learning, Natural Language Processing take the win here. SQL is so simple why would anyone want to spend time learning SQL? The reason… all of those hot topics in 2018 require data. My data mostly comes from relational databases which require sql to get data from them. Without the ability to efficiently get the data I need to do an aanlysis I cannot even start. Sure I could use an ORM, but I found that to be a bit unwieldy with the thousands of tables we have in formats that were determined many years ago. Plus raw SQL is more transportable. I commonly collaborate with other folks who do not use python. I am proud that I am able to point them to the SQL I use rather than telling them to suck it up an learn python. I truly believe that people are the most effective when they are able to choose their own stack of tools. Taking some time to focus on the basics of Data Science will help be build a strong foundation for my career.

Joining Data in Posgres #

Below are my notes from the Joining Data in Posgres course on DataCamp. I will use these notes as a refresher when I need a quick reference.

Using() #

When joining two tables on the same column the USING clause can be used as a shorthand.

without using

SELECT *

FROM
    Table1 as t1

LEFT JOIN
    Table2 as t2
    ON t1.id = t2.id

with using

SELECT
    *

FROM
    Table1 as t1

LEFT JOIN
    Table2 as t2
    USING (id)

Join Types #

for joining columns of data together into a single table

INNER: Includes only records contained in both tables.

RIGHT: Inlcudes all records from the right, droping values from the left if non-existent in the right, or leaving nulls if non-existant in the left.

LEFT: Inlcudes all records from the left, droping values from the right if non-existent in the left, or leaving nulls if non-existant in the right.

FULL: Combination of Left and Right Join, leaving nulls where data is missing in one table, and not droping any data.

CROSS: returns all pairs from two tables, does not have an on or using clause.

Union #

for concatenating rows of data with the same columns

Union: returns only unique records, does not include duplicates.

Union All: returns all records(including duplicates)

Intersect #

Intersect: returns only records appearing in both tables

Execpt #

Except: returns only records not in the second table

Self-Joins #

Semi-Join: Filters based on results of a subquery. Does not have direct sql syntax. This type of join is achieved through a subquery in the where statement.

Anti-Join: Similar to the Semi-join, but using a not modifier. This is particularly useful for debugging situations.

Subqueries #

This is where I have really stepped up my sql game. I was able to get practice writing more complex queries. I also learned about different methods of joining tables together.

WHERE #

Subqueries are commonly found in the where clause to filter data. Below is an example given in the course to select only the Asian countries with below average fertility rate from the states table.

SELECT
   name,
   fert_rate
FROM
    states
WHERE
    continent = 'Asia'
AND fert_rate <
        (SELECT AVG(fert_rate)
         FROM states;)

SELECT #

Subqueries can be found in the SELECT clause to create new columns of data. This is a different technique than I have used in the past. Previously I have only used GROUPBY statements to get this effect. I can see where this can be really useful because it is not constrained by aggregations any data point can be pulled in with this tecnhique.

SELECT DISTINCT
    continent,
    (SELECT
        COUNT(*)
     FROM
        states
     WHERE
        prime_ministers.continent = states.continent
    ) AS countries_num

From Prime Ministers

FROM #

subqueries found in the FROM clause can be very helpful to create a new dataset from an existing table. I find these the easiest to read as it is not much different than creating a new table. Again this can be very powerful in creating new columns that were not easily available otherwise.

SELECT DISTINCT
    monarchs.continent,
    subquery.max_perc

FROM
    monarchs,
    (SELECT
        continent,
        MAX(women_parli_perc) AS max_perc

    FROM
        states

    GROUP BY
        continent
    ) as subquery

WHERE
    monarchs.continent = subquery.continent

ORDER BY
    continent;

ON #

Challenge Problem 1 This problem was the one that had me more stumped than any other problem in the course. I found the subquery inside the on statement very confusing to understand. In this question we are joining the countries table to a subquery what yields country codes of countries with offial languages from the languages table.

SELECT DISTINCT
    c.name,
    e.total_investment,
    e.imports

FROM
    countries as c
LEFT JOIN
    economies as e
    ON c.code = e.code

    AND c.code in (
    SELECT
        l.code
    FROM
        languages as l
    WHERE
        official = true
    )

WHERE
    c.region = 'Central America'
AND e.year = 2015

ORDER BY
    c.name asc;