Posts tagged: python

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log # [1] First, how does one silence a python log? Python loggers can be retrieved by the logging module’s getLogger function. Then their log level can be changed. Much of kedro’s chattiness comes from INFO level logs. I don’t want to hear about anything for my current use case unless it’s essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways. python logging levels # [2] Level Numeric value CRITICAL 50 ERROR 40 WARNING 30 INFO 20 DEBUG 10 NOTSET 0 Get or Create a logger # [3] Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in. logger = logging.getLog...

Setting up python with the native nvim>0.5 lsp was mr lsp-config # [1] https://github.com/neovim/nvim-lspconfig lua << EOF require'lspconfig'.pyright.setup{} EOF pyls#190 # [2] https://github.com/palantir/python-language-server/issues/190 lspconfig.pyls.setup { cmd = {"pyls"}, filetypes = {"python"}, settings = { pyls = { configurationSources = {"flake8"}, plugins = { jedi_completion = {enabled = true}, jedi_hover = {enabled = true}, jedi_references = {enabled = true}, jedi_signature_help = {enabled = true}, jedi_symbols = {enabled = true, all_scopes = true}, pycodestyle = {enabled = false}, flake8 = { enabled = true, ignore = {}, maxLineLength = 160 }, mypy = {enabled = false}, isort = {enabled = false}, yapf = {enabled = false}, pylint = {enabled = false}, pydocstyle = {enabled = false}, mccabe = {enabled = false}, preload = {enabled = false}, rope_completion = {enabled = false} } } }, on_attach = on_attach } mypy # [3] Getting mypy working with ...

change_speed = (speed) => [...document.querySelectorAll('video')].map(v => v.playbackRate=v.playbackRate+speed) Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale. Minimum reproducible error # [1] Minimum reporducible error is one of my superpowers in development. I do this very often to sus out what is really happening. My day to day work is processing data with python, I keep a number of very small data sets handy to break and fix. This helps separate complexities of the project and the problem. Let’s break it # [2] Markata has a lot going on. It’s a plugins all the way down static site generator built in python. Trying to find the root cause through the layers ...

:G :G status :G commit :G add % :Gdiff :G push :Glog Add current file and commit with diff in a split # [1] function! s:GitAdd() exe "G add %" exe "G diff --staged" exe "only" exe "G commit" endfunction :command! GitAdd :call s:GitAdd() nnoremap gic :GitAdd<CR> :on[ly] # [2] C-W o :on[ly] will make the current buffer the only one on the screen. This is super helpful as many of fugitive commands will open in a split by default. C-I C-O # [3] cycle through the jumplist This one has nothing to do with fugitive, but is a native vim feature that makes fugitive glorious. Before I realized how to utilize C-i and C-o, I would get completely lost when using fugitive. Digging deep into the log, opening a file from a specific commit, then no way to get back where I was in the log. C-i jump :jump[s] # [4] show the jumplist The jumplist is sorted Oldest to newest :Telescope jumplist # [5] When navigating the jumplist with :Telescope jumplist, it will add a new entry to the jumpli...

change_speed = (speed) => [...document.querySelectorAll('video')].map(v => v.playbackRate=v.playbackRate+speed) When a python module is called it is assigned the __name__ of __main__ otherwise if it’s imported it will be assigned the __name__ of the module. Concrete example # [1] Let’s create a module to play with __name__ a bit. We will call this module nodes.py. It is a module that we may want to run by it’self or import and use in other modules. #!python # nodes.py if __name__ == "nodes": import sys import __main__ print(f"you have imported me {__name__} from {sys.modules['__main__'].__file__}") if __name__ == "__main__": print("you are running me as main") I have set this module up to execute one of two if statements based on whether the module it’self is being ran or if the module is being imported. Note it is not common to have a if __name__ == "nodes": block, this is just for demnonstration purposes. running python nodes.py # [2] Running a python script with the...

DRAFT - formatters: mine: format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s - %(me)s" handlers: mine_handler: class: logging.StreamHandler level: INFO formatter: mine stream: ext://sys.stdout loggers: me: level: DEBUG handlers: [mine_handler] root: level: INFO handlers: [console, info_file_handler, error_file_handler]

I really like using conda (miniconda) as my python virtual environment [1] manager of choice. It’s simple and it includes its own python interpreter using the version that I specify at creation. Mamba # [2] from their readme [3] --- Mamba is a reimplementation of the conda package manager in C++. - parallel downloading of repository data and package files using multi-threading - libsolv for much faster dependency solving, a state of the art library used in the RPM package manager of Red Hat, Fedora and OpenSUSE - core parts of mamba are implemented in C++ for maximum efficiency At the same time, mamba utilize the same command line parser, package installation and deinstallation code and transaction verification routines as conda to stay as compatible as possible. --- Installing Micromamba # [4] Similar to miniconda micromamba can be installed with a few lines of bash wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba ./bin/...

I am trying to see what an embeded replit

Zev Averbach, Frustrated spreadsheet jockey to software developer at 36 Q: Tell me about your journey as a spreadsheet jockey into Data Engineering? A: First of all, it’s hilarious that I accidentally found your questions for this interview by Googling myself. 😊 I’ve always been a frustrated software user, and that frustration led me to be a “power user” (keyboard shortcuts etc) of my most used applications, as well as a “visual coder” using desktop automation like Alfred and Keyboard Maestro (Mac). Now that I’ve met data analysts and finance people that use Excel all day, I don’t think I’d claim to have been a true “spreadsheet jockey” in comparison to them. However, hitting up against the limitations of spreadsheets for running my transcription business [1] – specifically for bookkeeping – created a new frustration for me: As the business grew I was spending more and more time copying entries from Google Sheets to the Freshbooks web app for invoicing purposes. I tried to auto...

Testing print/log statements in pytest can be a bit tricky, capsys makes it super easy, but I often struggle to find it. capsys # [1] capsys is a builtin pytest fixture that can be passed into any test to capture stdin/stdout. For a more comprehensive description check out the docs on capsys [2] using capsys # [3] Simply create a test function that accepts capsys as an argument and pytest will give you a capsys opject. def test_print(capsys): print('hello') captured = capsys.readouterr() assert 'hello' in captured.out print('world') captured = capsys.readouterr() assert 'world' in captured.out References: [1]: #capsys [2]: https://docs.pytest.org/en/stable/capture.html#accessing-captured-output-from-a-test-function [3]: #using-capsys

Draft Post I’ve really been digging [1]@willmcgugan [2]’s rich [3] library for creating TUI like interfaces in python. I’ve only recently started to take full advantage of it. Dev Server # [4] I am working on a project in which I want to have a dev server running continuously in the background. I really like dev servers theat automatically chooose an unused port and list out the running pid so that I can kill it if I need to. - automatic port number - auto-restart - display ( port, pid, uptime ) finding the port # [5] I am very novice at best when it comes to sockets, the following function came from searching StackOverflow for how to tell if a port is in use. I recursively check if a port is being used, if it is I increment by one until I find an unused port to return. def find_port(port=8000): """Find a port not in ues starting at given port""" import socket with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: if s.connect_ex(("localhost", port)) == 0: retur...

Final Result # [1] git checkout main git reset --hard git rm -rf --cached . echo "* text=auto" > .gitattributes git add . References: [1]: #final-result

I have automated my process to create virtual environments in my python projects, here is how I did it. I’ve really been digging my new tmux session management setup. Now I have leveled it up by adding direnv to my workflow. It will execute a shell script whenever I cd into a directory. One thing I wanted to add to this was, automatic activation of python environments whenever I cd into a directory, or create a new environment if one does not exist. https://waylonwalker.com/tmux-nav-2021/ Direnv # [1] You can learn more about direnv [2] on their website. There are several libraries that seem to do a similar concept, run a bash script when I cd into a directory direnv is the one that I chose to use, but you can likely do the same concept in other ones. Installation # [3] direnv is in many package repos, or can be installed from binary builds with a one liner. See their installation [4] instructions for more information. curl -sfL https://direnv.net/install.sh | bash .envrc # [...

I have started doing more regular PR’s on my teams Kedro [1] pipelines. I generally take a two phase approach to the review in order to give the reviewee both quick and detailed feedback. What is Kedro [2] initial scan (Phase1) # [3] - passing ci - Variable Names - Antipatterns - No commented out code - Docsttrings generally make sense Phase1 is typically a quick scan over the PR right within the PR window in my browser. Passing CI # [4] - flake8 - black - isort - interrogate - pytest - build The very first thing that needs to happen is automated CI. We use things like flake8, black, isort, interrogate to ensure that everyone follows generic style guides like pep8. The project does a build within the PR, but no deploy. Variable Names # [5] I strugle really hard to not impose my own opinion into the PR at this point, and sometimes really want to change a lot of variable names. Typically I make sure they don’t grow longer than necessary, too short, misspelled, or inc...

With the realease of kedro==0.17.2 came a new module in the project template pipeline_registry.py. Here are some notes that I learned while playing with this new module. migrating to pipeline_registry.py # [1] - create a src/<package-name>/pipeline_registry.py file create a - register_pipelines function in pipeline_registry.py that mirrors the - register_pipelines method from your hooks.py module do not bring the - hook_impl decorator remove register_pipelines method on your ProjectHooks - class You should now have something that looks like this in your src/<package-name>/pipeline_registry.py. """Project pipelines.""" from typing import Dict from kedro.pipeline import Pipeline def register_pipelines() -> Dict[str, Pipeline]: """Register the project's pipelines. Returns: A mapping from a pipeline name to a ``Pipeline`` object. """ return {"__default__": Pipeline([])} pipeline_registry only works in kedro>=0.17.2 Conflict Resolution # [2] What happens If I register p...

pytest has open sourced their amazing plugin framework pluggy, it allows library authors to give their users a way to modify the libaries behavior without needing to submit a change that may not make sense to the entire library. Previous Experience # [1] My experience so far as a plugin user, and plugin author has been great. Building and using plugins are incredibly intuitive. I wanted to dive a bit deeper and see how they are implemented inside of a library and its a bit of a mind bend the first time you try to do it. Plugins vs. Hooks # [2] A hook is a single function that has a specific place that it is ran by the PluginManager. A Plugin is a collection of one or more hooks. Layers # [3] - library author - plugin author - end user Using a plugin # [4] For a plugin to be registered is must be registered by the PluginManager which is implemented by the library author. It is the job of the library author to determine what plugins are actively registered or disabled. Ther...

There are so many tutorials out there for creating OG [1] images with nodejs or puppeteer, but I have yet to see many using python. PIL # [2] Template # [3] Text # [4] Overlays # [5] References: [1]: /og/ [2]: #pil [3]: #template [4]: #text [5]: #overlays

There are various ways to configure python tools, config files, code, or environment variables. Let’s look at a few projects that allow users to configure them through the use of config files and how they do it. Motivation # [1] This will not include how they are implemented, I’ve looked at a few and its not simple. This will focus on where config is placed and the order in which duplicates are resolved. The motivation of this article is to serve as a bit of a reference guide for those who may want to create their own package that needs configuration. Flake8 # [2] Global # [3] User settings can exist in the users ~/.config/flake8 file to configure how flake8 runs on their machine. - ~/.config/flake8 Per-Project # [4] Only One project config file will be considered, but allows for several options. These files all use the ini format and must have a [flake8] section header to be consideered. Selection of the config file can also be overridden by the --config cli option. An e...

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It’s only a total of 35 lines of python, 8 in setup.py and 27 in mini_kedro_pipeline.py. 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. Minimal Kedro Pipeline # [1] I have everything for this post hosted in this gihub repo [2], you can fork it, clone it, or just follow along. Installation # [3] pip install git+https://github.com/WaylonWalker/mini-kedro-pipeline Caveats # [4] This repo represents the minimal amount of structure to build a kedro pipeline that can be shared across projects. Its installable, and drops right into your hooks.py or run.py modules. It is not a runnable pipeline. At this point I think the config loader requires to have a logging config file. This is a sharable pipeline that can be used across many different projects. Usage # [5] # hooks.py import mini_kedro_project as mkp class Pro...

This is a post that may be a work in progress for awhile, Its a collections of thoughts on managing my blog, but could be translated into anythiung that is just a collection of markdown. Listing things # [1] - posts - tags - draft posts data # [2] - frontmatter - filepath - content - template - html [3] render content # [4] - Markdown.Markdown - support extentsions frontmatter cleaning. # [5] - provide ways to hook in or clean up the frontmatter Markata.Markata methods # [6] - load - render - save Markata.Post methods # [7] - load - render - save Markata plugins # [8] - before_load - before_post_load - after_load - after_post_load - before_save - before_post_save - after_save - after_post_save Markata plugins # [9] - cleanse_frontmatter - html_feed - json_feed - rss_feed - save_posts CLI # [10] $ markata list tags python data $ markata [ { "title": "post title", "description": "this is a post", "filepath": "path_to.md", "content": "the ...