Check out terminal [1] by microsoft [2]. It’s a well-crafted project with great potential.
The new Windows Terminal and the original Windows console host, all in the same place!
References:
[1]: https://github.com/microsoft/terminal
[2]: https://github.com/microsoft
Publishing rhythm
Clean up Your Data Science with Named Tuples
If you are a regular listener of TalkPython [1] or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow.
Example # [2]
As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing sales, inventory, and pricing data. You may need to calculate total revenue, inventory on hand. You may need to pass these data sets into various models to drive production or pricing based on predicted volumes.
Load data # [3]
Here we setup functions that can load data from the sales database. Assume that we also have similar functions to get_inventory and get_pricing.
def get_engine():
engine = create_engine('postgresql://scott:tiger@localhost:5432/mydatabase')
def get_sales():
'''
gets sales history from the sales database
'''
engine = ge...
Background Tasks in Python for Data Science
This post is intended as an extension/update from background tasks in
python [1]. I started using background
the week that Kenneth Reitz released it. It takes away so much boilerplate
from running background tasks that I use it in more places than I probably
should. After taking a look at that post today, I wanted to put a better data
science example in here to help folks get started.
This post is intended as an extension/update from background tasks in python [1]. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started.
I use it in more places than I probably should
Before we get into it, I want to make a shout out to Kenneth Reitz for making this so easy. Kenneth is a python God for all that he has given to the community in so many w...
If you’re into interesting projects, don’t miss out on starship [1], created by starship [2].
☄🌌️ The minimal, blazing-fast, and infinitely customizable prompt for any shell!
References:
[1]: https://github.com/starship/starship
[2]: https://github.com/starship
alttch [1] has done a fantastic job with rapidtables [2]. Highly recommend taking a look.
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
References:
[1]: https://github.com/alttch
[2]: https://github.com/alttch/rapidtables
Autoreload in Ipython
I have used %autoreload for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated.
What I like about autoreload # [1]
🔥 Blazing Fast
💥 Keeps me in the comfort of my text editor
👏 Allows me to use Jupyter when I need
👟 Extremely Reliable
One of the biggest benefits that I find is that it shortens the distance between my module/library code and test code inside of a terminal/notebook. Now I primarily use jupyter notebooks for the presentation aspect. I develop code from the comfort of my editor with all of the tools I have setup, and run the functions in a notebook to get the output. From there I might do some aggregations or plots, but the 🥩 meat of development is done outside of jupyter.
Now I primarily use jupyter notebooks for the presentation aspect.
Enabling Autoreload # [2]
📐 config
This is a sh...
If you’re into interesting projects, don’t miss out on psutil [1], created by giampaolo [2].
Cross-platform lib for process and system monitoring in Python
References:
[1]: https://github.com/giampaolo/psutil
[2]: https://github.com/giampaolo
If you’re into interesting projects, don’t miss out on promote-open-source-project [1], created by zenika-open-source [2].
📄 How to promote my open source project?
References:
[1]: https://github.com/zenika-open-source/promote-open-source-project
[2]: https://github.com/zenika-open-source
Check out watchtower [1] by kislyuk [2]. It’s a well-crafted project with great potential.
Python CloudWatch Logging: Log Analytics and Application Intelligence
References:
[1]: https://github.com/kislyuk/watchtower
[2]: https://github.com/kislyuk
I recently discovered arrow [1] by apache [2], and it’s truly impressive.
Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
References:
[1]: https://github.com/apache/arrow
[2]: https://github.com/apache
Just starred shell-functools [1] by sharkdp [2]. It’s an exciting project with a lot to offer.
Functional programming tools for the shell
References:
[1]: https://github.com/sharkdp/shell-functools
[2]: https://github.com/sharkdp
Keyboard Driven VSCode
Throw that mouse Away its time to setup some keyboard shortcuts.
These sortcuts were the baseline for switching from tmux/vim to vscode. Most folks posts I was able to find gave great tips on replacing vim, but very few have focused on the hackability of tmux. tmux allows me to rapidly fire up a workspace, create new windows and splits. Then When I switch tasks I can leave that workspace open and and jump right back in later exactly where I left off. There is nothing quite like it. The shortcuts listed here make the transition a bit better. The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse. This first set of keybindings solve that issue.
The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse.
!!! see-also
I have an updated article in my tmux workflow How I navigate tmux in 2021 [1]
Alt+[hjkl] # [2]
navigation
⬅ jump to left split alt+h
⬇ j...
Looking for inspiration? Jupyter-Atom-Dark-Theme [1] by burglarbenson [2].
A dark theme for Jupyter Lab
References:
[1]: https://github.com/burglarbenson/Jupyter-Atom-Dark-Theme
[2]: https://github.com/burglarbenson
tarpas [1] has done a fantastic job with pytest-testmon [2]. Highly recommend taking a look.
Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch.
References:
[1]: https://github.com/tarpas
[2]: https://github.com/tarpas/pytest-testmon
If you’re into interesting projects, don’t miss out on vim-flog [1], created by rbong [2].
A blazingly fast, stunningly beautiful, exceptionally powerful git [3] branch viewer for Vim/Neovim.
References:
[1]: https://github.com/rbong/vim-flog
[2]: https://github.com/rbong
[3]: /glossary/git/
I like mcfunley’s [1] project pugsql [2].
A HugSQL-inspired database library for Python
References:
[1]: https://github.com/mcfunley
[2]: https://github.com/mcfunley/pugsql
I like ggreer’s [1] project the_silver_searcher [2].
A code-searching tool similar to ack, but faster.
References:
[1]: https://github.com/ggreer
[2]: https://github.com/ggreer/the_silver_searcher
Realistic Git Workflow
My git [1] workflow based on real life. Its not always clean and simple.
sometimes things get messy
The Clean Path # [2]
[3]
pull 👉 branch 👉 format 👉 work👉 add 👉 commit 👉 pull 👉 rebase 👉 push
Pull # [4]
As complicated as that seems it is pretty straight forward. When you sit down to work the first thing you do is to pull down the teams latest working “develop” branch from git.
git checkout develop
git pull
Branch # [5]
Next create a new branch with a name that will remind you of what you are working on. For your own sanity choose something descriptive. It is easy to get too many similar branches going and forget which branch is which.
git checkout -b ingest_product_id_table
Format # [6]
If you know which files in existance that you will be editing before you start work it is a good idea to format them in a commit early on to keep your working commits separate from formatting. This will make it easier for reviewers to distinguish from your changes and formatting fixes.
...
Just starred kedro [1] by kedro-org [2]. It’s an exciting project with a lot to offer.
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
References:
[1]: https://github.com/kedro-org/kedro
[2]: https://github.com/kedro-org
Check out forestryio [1] and their project forestry.io [2].
Forestry.io website
References:
[1]: https://github.com/forestryio
[2]: https://github.com/forestryio/forestry.io
Check out maildown [1] by chris104957 [2]. It’s a well-crafted project with great potential.
A super simple CLI for sending emails
References:
[1]: https://github.com/chris104957/maildown
[2]: https://github.com/chris104957
Forestry.io
Testing out forestry.io
Sorry Netlify CMS # [1]
I still ♥️ your product dont be
forestry is simple
I have been playing with the netlify cms for a while now, and it has been a decent experience, but I really struggle configuring it. Forestry is so simple to setup. My favorite part is that I can code up my gatsby.js site, storing all editable text in markdown, and come back later and add the CMS based on existing documents.
Configuration is Simple # [2]
Forestry.io has this amazing feature to create create based on existing document 🤯. This is great because it sets up the .yml config for me without error. And If I really want to come back later to customize it more I have that option, too.
[3]
By far my favorite feature is create based on existing document
Multi-File Gallery # [4]
I have a use case for a photography site where the owner wants to be able to show off sample photos of each type of work she does. I got it working in the netlify cms, although it was not very user...
Just starred eslint-config-wesbos [1] by wesbos [2]. It’s an exciting project with a lot to offer.
No-Sweat™ Eslint and Prettier Setup - with or without VS Code
References:
[1]: https://github.com/wesbos/eslint-config-wesbos
[2]: https://github.com/wesbos
Check out ydataai [1] and their project ydata-profiling [2].
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
References:
[1]: https://github.com/ydataai
[2]: https://github.com/ydataai/ydata-profiling
I came across ydata-profiling [1] from Data-Centric-AI-Community [2], and it’s packed with great features and ideas.
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
References:
[1]: https://github.com/Data-Centric-AI-Community/ydata-profiling
[2]: https://github.com/Data-Centric-AI-Community
mdbartos [1] has done a fantastic job with tabview [2]. Highly recommend taking a look.
Python curses command line CSV viewer
References:
[1]: https://github.com/mdbartos
[2]: https://github.com/mdbartos/tabview
TabViewer [1] has done a fantastic job with tabview [2]. Highly recommend taking a look.
Python curses command line CSV and tabular data viewer
References:
[1]: https://github.com/TabViewer
[2]: https://github.com/TabViewer/tabview
I came across voidrice [1] from LukeSmithxyz [2], and it’s packed with great features and ideas.
My dotfiles (deployed by LARBS)
References:
[1]: https://github.com/LukeSmithxyz/voidrice
[2]: https://github.com/LukeSmithxyz
I like FormidableLabs’s [1] project webpack-dashboard [2].
A CLI dashboard for webpack dev server
References:
[1]: https://github.com/FormidableLabs
[2]: https://github.com/FormidableLabs/webpack-dashboard
Rewrite History with Git
- rebase
- git [1] commit –amend
Unstage # [2]
git reset -- <file>
rage unstage to wipte out history of staged commit
git reset --hard <file>
Undo file # [3]
- rage quit
- git reset HEAD~n
- removes modifications
- keeps hitsory of changes and undoes them
- git checkout HEAD~n –
-
keeps modifications
-
removes history
-
–SOFT
-
–HARD
-
–Mixed
undo n commits back # [4]
locally before push
git reset HEAD~n
after push
git revert HEAD~n
update .gitignore # [5]
after push
git rm -r --cached .
git commit -am "Updated .gitignore"
References:
[1]: /glossary/git/
[2]: #unstage
[3]: #undo-file
[4]: #undo-n-commits-back
[5]: #update-gitignore
It's not all about winning
This is my story into data science.
The Journey Begins # [1]
I am addicted to the process of learning and improving my skills nearly to a fault. The reason I say nearly is because my addiction is fueled with results. I crave the output of my work enhance the work of others. I jump with joy as I see users gain insights they could have never imagined before. My mouth starts watering as I see their boring repetitive data mining activites be completed in a matter of seconds, opening up their mind to focus on their expertise. The day I stop learning will be the day that I start looking for another career path.
It’s not all about winning.
~Mom
This happened to me in 2014. I have a mechanical engineering degree and had a really good position at the time. I owned full engineering control of a small subset of engine components. The problem was that Everything is so proprietary and hardly documented the process of learning did not click with me. I felt like I had learned a lot about the ...
Looking for inspiration? fzf-preview.vim [1] by yuki-yano [2].
The plugin that powerfully integrates fzf and (Neo)vim. It is also possible to integrate with coc.nvim.
References:
[1]: https://github.com/yuki-yano/fzf-preview.vim
[2]: https://github.com/yuki-yano
If you’re into interesting projects, don’t miss out on background [1], created by ParthS007 [2].
Runs things in the background.
References:
[1]: https://github.com/ParthS007/background
[2]: https://github.com/ParthS007
The work on react-select [1] by JedWatson [2].
The Select Component for React.js
References:
[1]: https://github.com/JedWatson/react-select
[2]: https://github.com/JedWatson
I recently discovered vimade [1] by TaDaa [2], and it’s truly impressive.
Vimade let’s you dim, fade, tint, animate, and customize colors in your windows and buffers for (Neo)vim
References:
[1]: https://github.com/TaDaa/vimade
[2]: https://github.com/TaDaa
Update Git User
This morning I log into my VCS and check activity on my projects to find that someone else has been very active on my projects fo the last few weeks. I quicklyhover over the missing avatar to find that It’s Me. What’s wrong here, why do I look like two different people throughout the day! upon further investigation I see the issue. while setting up a new terminal environment I mistyped my email address by one character. After much searching and a few failed attempts I was able to fix it by following an article no longer available (2021) from https://help.github.com/articles.
Bare Clone # [1]
Clone the repo, note it must be a --bare clone.
git clone --bare https://github.com/user/repo.git
cd repo.git
git-author-rewrite # [3]
Curl down the git-author-rewrite script and edit the following variables OLD_EMAIL CORECT_NAME CORRECT_EMAIL
curl https://gist.githubusercontent.com/octocat/0831f3fbd83ac4d46451/raw/c197afe3e9ea2e4218f9fccbc0f36d2b8fd3c1e3/git-author-rewrite.sh > git-author-...
2019 goals
strong {
/* color: goldenrod; */
/* text-shadow: 0rem 1px 1px goldenrod; */
}
The year of intenionality
This is a follow up to my previous post 2018-retrospective [1]
professional # [2]
This year I will become more productive, by intensionally working on a well thought out plan, learning the right technologies, and leave behind a positive legacy.
Productivity # [3]
Last year I was able to make some great strides in my productivity and focus. This year I want to be able to bring it up a notch. Intentionality plays a big part in this. Taking some time to sit down and think about the tasks you realistically want to complete for the day, then focusing on those tasks one at a time.
I started off the year last year using the pomodoro system to focus on one task and one task only for 25 minutes at a time. This worked really well for me but I quickly fell off of the train and fell into my old trap of reacting to the loudest customer over the items I put intensionally on my todo lis...
Check out engineer-man [1] and their project youtube [2].
Code from the Engineer Man YouTube channel. Please do not submit pull requests, they will be ignored/closed. The code in the repo needs to remain as it was in the video.
References:
[1]: https://github.com/engineer-man
[2]: https://github.com/engineer-man/youtube
2018 Retrospective
2018 was a year of many ups and downs, and learning to deal with a whole new
set of problems professionally and at home. In 2018 I logged in to my first
Linux system, setup my own webserver, data pipelines, database. I learned to
use react and d3. Stepped up my python, javascript, and sql.
At home the doctor appointments keep piling in. While I am learning to deal
with it all there were several times throughout the year that I was very
overwhelmed with everything and broke down.
Here are the goals I had listed out for 2018 and how I faired at completing
them.
Positivity # [1]
The Good # [2]
---
- Continue “Favorite Things” with family at dinner time
- Take 2 10min breaks per day clear the mind
---
I can say that the kids are fully on board with favorite things and love to
tell everyone about how their day went. This is a time that they are very
positive and generally give praise to another family member for helping them
through their day.
To Be Improved # [3]
---
- Grati...
Do More of What Brings You Joy
Today I want to take some time to talk about the things that make me happy in my work environment. This is completely free-flow off the cuff, but are things that I do that make me happy, not having them would definitely be a deal breaker for me .
Attitude # [1]
A positive work atmosphere goes a long ways. We all have enough negativity going on in our lives that is out of our control accepting any negativity in the workplace is a no go for me. There was a point in which I was suffering a lot of negativity at work. This began to trickle into every part of life, and it became hard to see positivity anywhere. My wife not only noticed this, but put a polite request in for change. It was definitely a low point and she could tell tell that my work life was not helping anything.
[2]
Throw out the negativity - courtesy giphy [3]
Bring the positivity to your workplace. It is really difficult for folks to tear you down if you are the one that is always up beat and happy. People will notic...
feather [1] by wesm [2] is a game-changer in its space. Excited to see how it evolves.
Feather: fast, interoperable binary data frame storage for Python, R, and more powered by Apache Arrow
References:
[1]: https://github.com/wesm/feather
[2]: https://github.com/wesm
Just starred NES.css [1] by nostalgic-css [2]. It’s an exciting project with a lot to offer.
NES-style CSS Framework | ファミコン風CSSフレームワーク
References:
[1]: https://github.com/nostalgic-css/NES.css
[2]: https://github.com/nostalgic-css
kennethreitz [1] has done a fantastic job with responder [2]. Highly recommend taking a look.
A familiar HTTP Service Framework for Python.
References:
[1]: https://github.com/kennethreitz
[2]: https://github.com/kennethreitz/responder
cheatsheets [1] by rstacruz [2] is a game-changer in its space. Excited to see how it evolves.
Cheatsheets for web development - devhints.io
References:
[1]: https://github.com/rstacruz/cheatsheets
[2]: https://github.com/rstacruz
If you’re into interesting projects, don’t miss out on awesome-vscode [1], created by viatsko [2].
🎨 A curated list of delightful VS Code packages and resources.
References:
[1]: https://github.com/viatsko/awesome-vscode
[2]: https://github.com/viatsko
psp_hourly_rates [1] by rwhitt2049 [2] is a game-changer in its space. Excited to see how it evolves.
My local power company offers market based electricity rates that vary by hour. This is an analysis to determine if it could save me money. https://www.powersmartpricing.org/
References:
[1]: https://github.com/rwhitt2049/psp_hourly_rates
[2]: https://github.com/rwhitt2049
Looking for inspiration? mdx [1] by mdx-js [2].
Markdown for the component era
References:
[1]: https://github.com/mdx-js/mdx
[2]: https://github.com/mdx-js
Just starred pyperclip [1] by asweigart [2]. It’s an exciting project with a lot to offer.
Python module for cross-platform clipboard functions.
References:
[1]: https://github.com/asweigart/pyperclip
[2]: https://github.com/asweigart
Check out tomerfiliba [1] and their project plumbum [2].
Plumbum: Shell Combinators
References:
[1]: https://github.com/tomerfiliba
[2]: https://github.com/tomerfiliba/plumbum
decaporg [1] has done a fantastic job with decap-cms [2]. Highly recommend taking a look.
A Git [3]-based CMS for Static Site Generators
References:
[1]: https://github.com/decaporg
[2]: https://github.com/decaporg/decap-cms
[3]: /glossary/git/