Archive

Custom Python Exceptions

Custom Exceptions # [1] class ProjectNameError(NameError): pass class UserNameError(NameError): pass class CondaEnvironmentError(RuntimeError): pass class BucketNotDefinedError(NameError): pass References: [1]: #custom-exceptions

Filtering Pandas

query # [1] Good for method chaining, i.e. adding more methods or filters without assigning a new variable. # is skus.query('AVAILABILITY == " AVAILABLE"') # is not skus.query('AVAILABILITY != " AVAILABLE"') masking # [2] general purpose, this is probably the most common method you see in training/examples # is skus[skus['AVAILABILITY'] == 'AVAILABLE'] # is not skus[~skus['AVAILABILITY'] == 'AVAILABLE'] isin # [3] capable of including multiple strings to include # is in df[df.AVAILABILITY.isin(['AVAILABLE', 'AVL'])] # is not in df[~df.AVAILABILITY.isin(['AVAILABLE', 'AVL'])] contains # [4] Good For partial matches # contains df[df.AVAILABILITY.str.contains('AVA')] # not contains df[~df.AVAILABILITY.str.contains('AVA')] MASKS # [5] anything that we put inside of square brackets can be set as a variable then passed in. service_mask = skus['AVAILABILITY'] == 'AVAILABLE' name_mask = skus['NAME'] == 'Dell chromebook 11' Operators # [6] & - and ~ - not | - or AVAILABLE and ...

Digital Ocean

I love digital ocean for it’s simplicity and its commitment to open source.

⭐ Kashu7100 Recreation-of-Nature

If you’re into interesting projects, don’t miss out on Recreation-of-Nature [1], created by Kashu7100 [2]. ALife simulation with Python: patterns, behavior, and cognition. References: [1]: https://github.com/Kashu7100/Recreation-of-Nature [2]: https://github.com/Kashu7100

Quick Progress Bars in python using TQDM

tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time. [1] for more gifs like these follow me on twitter @waylonwalker [2] Add a simple Progress bar! from tqdm import tqdm from time import sleep for i in tqdm(range(10)): sleep(1) convenience TQDM also has a convenience function called trange that wraps the range function with a tqdm progress bar automatically. from tqdm import trange from time import sleep for i in trange(range(10)): sleep(1) notebook support There is also notebook support. If you are bouncing between ipython and jupyter I recomend importing from the auto m...

⭐ kennethreitz bake

I’m impressed by bake [1] from kennethreitz [2]. Bake — the strangely familiar workflow utility. References: [1]: https://github.com/kennethreitz/bake [2]: https://github.com/kennethreitz

⭐ microsoft terminal

Check out terminal [1] by microsoft [2]. It’s a well-crafted project with great potential. The new Windows Terminal and the original Windows console host, all in the same place! References: [1]: https://github.com/microsoft/terminal [2]: https://github.com/microsoft

Clean up Your Data Science with Named Tuples

If you are a regular listener of TalkPython [1] or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow. Example # [2] As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing sales, inventory, and pricing data. You may need to calculate total revenue, inventory on hand. You may need to pass these data sets into various models to drive production or pricing based on predicted volumes. Load data # [3] Here we setup functions that can load data from the sales database. Assume that we also have similar functions to get_inventory and get_pricing. def get_engine(): engine = create_engine('postgresql://scott:tiger@localhost:5432/mydatabase') def get_sales(): ''' gets sales history from the sales database ''' engine = ge...

Background Tasks in Python for Data Science

This post is intended as an extension/update from background tasks in python [1]. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. This post is intended as an extension/update from background tasks in python [1]. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. I use it in more places than I probably should Before we get into it, I want to make a shout out to Kenneth Reitz for making this so easy. Kenneth is a python God for all that he has given to the community in so many w...

⭐ starship starship

If you’re into interesting projects, don’t miss out on starship [1], created by starship [2]. ☄🌌️ The minimal, blazing-fast, and infinitely customizable prompt for any shell! References: [1]: https://github.com/starship/starship [2]: https://github.com/starship

⭐ alttch rapidtables

alttch [1] has done a fantastic job with rapidtables [2]. Highly recommend taking a look. Super fast list of dicts to pre-formatted tables conversion library for Python 2/3 References: [1]: https://github.com/alttch [2]: https://github.com/alttch/rapidtables

📝 Bash Notes

Bash is super powerful. File System Full # [1] Show Remaining Space on Drives df -h show largest files in current directory du . -h --max-depth=1 Move files then symlink them mkdir /mnt/mounted_drive mv ~/bigdir /mnt/mounted_drive ln -s /mnt/mounted_drive/bigdir ~/bigdir Fuzzy One Liners # [2] a() {source activate "$(conda info --envs | fzf | awk '{print $ edit in vim vf() { fzf | xargs -r -I % $EDITOR % ;} cat a file vf() { fzf | xargs -r -I % $EDITOR % ;} bash execute bf() { bash "$(fzf)" } git [3] add gadd() { git status -s | fzf -m | awk '{print $2}' | xargs git add && git status -s} git reset greset() { git status -s | fzf -m | awk '{print $2}' |xargs git reset && git status -s} Kill a process fkill() {kill $(ps aux | fzf | awk '{print($2)}')} Finding things # [4] Files # [5] fd-find [6] is amazing for finding files, it even respects your .gitignore file 😲. Install with apt install fd-find. fd md ag -g python find . -n "*.md" ++Vanilla Bonus Content # [7] ** sh...

Autoreload in Ipython

I have used %autoreload for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated. What I like about autoreload # [1] 🔥 Blazing Fast 💥 Keeps me in the comfort of my text editor 👏 Allows me to use Jupyter when I need 👟 Extremely Reliable One of the biggest benefits that I find is that it shortens the distance between my module/library code and test code inside of a terminal/notebook. Now I primarily use jupyter notebooks for the presentation aspect. I develop code from the comfort of my editor with all of the tools I have setup, and run the functions in a notebook to get the output. From there I might do some aggregations or plots, but the 🥩 meat of development is done outside of jupyter. Now I primarily use jupyter notebooks for the presentation aspect. Enabling Autoreload # [2] 📐 config This is a sh...

⭐ giampaolo psutil

If you’re into interesting projects, don’t miss out on psutil [1], created by giampaolo [2]. Cross-platform lib for process and system monitoring in Python References: [1]: https://github.com/giampaolo/psutil [2]: https://github.com/giampaolo

⭐ zenika-open-source promote-open-source-project

If you’re into interesting projects, don’t miss out on promote-open-source-project [1], created by zenika-open-source [2]. 📄 How to promote my open source project? References: [1]: https://github.com/zenika-open-source/promote-open-source-project [2]: https://github.com/zenika-open-source

⭐ kislyuk watchtower

Check out watchtower [1] by kislyuk [2]. It’s a well-crafted project with great potential. Python CloudWatch Logging: Log Analytics and Application Intelligence References: [1]: https://github.com/kislyuk/watchtower [2]: https://github.com/kislyuk

⭐ apache arrow

I recently discovered arrow [1] by apache [2], and it’s truly impressive. Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics References: [1]: https://github.com/apache/arrow [2]: https://github.com/apache

⭐ sharkdp shell-functools

Just starred shell-functools [1] by sharkdp [2]. It’s an exciting project with a lot to offer. Functional programming tools for the shell References: [1]: https://github.com/sharkdp/shell-functools [2]: https://github.com/sharkdp

Keyboard Driven VSCode

Throw that mouse Away its time to setup some keyboard shortcuts. These sortcuts were the baseline for switching from tmux/vim to vscode. Most folks posts I was able to find gave great tips on replacing vim, but very few have focused on the hackability of tmux. tmux allows me to rapidly fire up a workspace, create new windows and splits. Then When I switch tasks I can leave that workspace open and and jump right back in later exactly where I left off. There is nothing quite like it. The shortcuts listed here make the transition a bit better. The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse. This first set of keybindings solve that issue. The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse. !!! see-also I have an updated article in my tmux workflow How I navigate tmux in 2021 [1] Alt+[hjkl] # [2] navigation ⬅ jump to left split alt+h ⬇ j...

⭐ burglarbenson Jupyter-Atom-Dark-Theme

Looking for inspiration? Jupyter-Atom-Dark-Theme [1] by burglarbenson [2]. A dark theme for Jupyter Lab References: [1]: https://github.com/burglarbenson/Jupyter-Atom-Dark-Theme [2]: https://github.com/burglarbenson

⭐ tarpas pytest-testmon

tarpas [1] has done a fantastic job with pytest-testmon [2]. Highly recommend taking a look. Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch. References: [1]: https://github.com/tarpas [2]: https://github.com/tarpas/pytest-testmon

⭐ rbong vim-flog

If you’re into interesting projects, don’t miss out on vim-flog [1], created by rbong [2]. A blazingly fast, stunningly beautiful, exceptionally powerful git [3] branch viewer for Vim/Neovim. References: [1]: https://github.com/rbong/vim-flog [2]: https://github.com/rbong [3]: /glossary/git/

⭐ mcfunley pugsql

I like mcfunley’s [1] project pugsql [2]. A HugSQL-inspired database library for Python References: [1]: https://github.com/mcfunley [2]: https://github.com/mcfunley/pugsql

⭐ ggreer the_silver_searcher

I like ggreer’s [1] project the_silver_searcher [2]. A code-searching tool similar to ack, but faster. References: [1]: https://github.com/ggreer [2]: https://github.com/ggreer/the_silver_searcher

Realistic Git Workflow

My git [1] workflow based on real life. Its not always clean and simple. sometimes things get messy The Clean Path # [2] [3] pull 👉 branch 👉 format 👉 work👉 add 👉 commit 👉 pull 👉 rebase 👉 push Pull # [4] As complicated as that seems it is pretty straight forward. When you sit down to work the first thing you do is to pull down the teams latest working “develop” branch from git. git checkout develop git pull Branch # [5] Next create a new branch with a name that will remind you of what you are working on. For your own sanity choose something descriptive. It is easy to get too many similar branches going and forget which branch is which. git checkout -b ingest_product_id_table Format # [6] If you know which files in existance that you will be editing before you start work it is a good idea to format them in a commit early on to keep your working commits separate from formatting. This will make it easier for reviewers to distinguish from your changes and formatting fixes. ...

⭐ kedro-org kedro

Just starred kedro [1] by kedro-org [2]. It’s an exciting project with a lot to offer. Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular. References: [1]: https://github.com/kedro-org/kedro [2]: https://github.com/kedro-org

⭐ forestryio forestry.io

Check out forestryio [1] and their project forestry.io [2]. Forestry.io website References: [1]: https://github.com/forestryio [2]: https://github.com/forestryio/forestry.io

⭐ chris104957 maildown

Check out maildown [1] by chris104957 [2]. It’s a well-crafted project with great potential. A super simple CLI for sending emails References: [1]: https://github.com/chris104957/maildown [2]: https://github.com/chris104957

Forestry.io

Testing out forestry.io Sorry Netlify CMS # [1] I still ♥️ your product dont be forestry is simple I have been playing with the netlify cms for a while now, and it has been a decent experience, but I really struggle configuring it. Forestry is so simple to setup. My favorite part is that I can code up my gatsby.js site, storing all editable text in markdown, and come back later and add the CMS based on existing documents. Configuration is Simple # [2] Forestry.io has this amazing feature to create create based on existing document 🤯. This is great because it sets up the .yml config for me without error. And If I really want to come back later to customize it more I have that option, too. [3] By far my favorite feature is create based on existing document Multi-File Gallery # [4] I have a use case for a photography site where the owner wants to be able to show off sample photos of each type of work she does. I got it working in the netlify cms, although it was not very user...

⭐ wesbos eslint-config-wesbos

Just starred eslint-config-wesbos [1] by wesbos [2]. It’s an exciting project with a lot to offer. No-Sweat™ Eslint and Prettier Setup - with or without VS Code References: [1]: https://github.com/wesbos/eslint-config-wesbos [2]: https://github.com/wesbos

⭐ ydataai ydata-profiling

Check out ydataai [1] and their project ydata-profiling [2]. 1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames. References: [1]: https://github.com/ydataai [2]: https://github.com/ydataai/ydata-profiling

⭐ Data-Centric-AI-Community ydata-profiling

I came across ydata-profiling [1] from Data-Centric-AI-Community [2], and it’s packed with great features and ideas. 1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames. References: [1]: https://github.com/Data-Centric-AI-Community/ydata-profiling [2]: https://github.com/Data-Centric-AI-Community

⭐ mdbartos tabview

mdbartos [1] has done a fantastic job with tabview [2]. Highly recommend taking a look. Python curses command line CSV viewer References: [1]: https://github.com/mdbartos [2]: https://github.com/mdbartos/tabview

⭐ TabViewer tabview

TabViewer [1] has done a fantastic job with tabview [2]. Highly recommend taking a look. Python curses command line CSV and tabular data viewer References: [1]: https://github.com/TabViewer [2]: https://github.com/TabViewer/tabview

⭐ LukeSmithxyz voidrice

I came across voidrice [1] from LukeSmithxyz [2], and it’s packed with great features and ideas. My dotfiles (deployed by LARBS) References: [1]: https://github.com/LukeSmithxyz/voidrice [2]: https://github.com/LukeSmithxyz

⭐ FormidableLabs webpack-dashboard

I like FormidableLabs’s [1] project webpack-dashboard [2]. A CLI dashboard for webpack dev server References: [1]: https://github.com/FormidableLabs [2]: https://github.com/FormidableLabs/webpack-dashboard

Rewrite History with Git

- rebase - git [1] commit –amend Unstage # [2] git reset -- <file> rage unstage to wipte out history of staged commit git reset --hard <file> Undo file # [3] - rage quit - git reset HEAD~n - removes modifications - keeps hitsory of changes and undoes them - git checkout HEAD~n – - keeps modifications - removes history - –SOFT - –HARD - –Mixed undo n commits back # [4] locally before push git reset HEAD~n after push git revert HEAD~n update .gitignore # [5] after push git rm -r --cached . git commit -am "Updated .gitignore" References: [1]: /glossary/git/ [2]: #unstage [3]: #undo-file [4]: #undo-n-commits-back [5]: #update-gitignore

It's not all about winning

This is my story into data science. The Journey Begins # [1] I am addicted to the process of learning and improving my skills nearly to a fault. The reason I say nearly is because my addiction is fueled with results. I crave the output of my work enhance the work of others. I jump with joy as I see users gain insights they could have never imagined before. My mouth starts watering as I see their boring repetitive data mining activites be completed in a matter of seconds, opening up their mind to focus on their expertise. The day I stop learning will be the day that I start looking for another career path. It’s not all about winning. ~Mom This happened to me in 2014. I have a mechanical engineering degree and had a really good position at the time. I owned full engineering control of a small subset of engine components. The problem was that Everything is so proprietary and hardly documented the process of learning did not click with me. I felt like I had learned a lot about the ...

⭐ yuki-yano fzf-preview.vim

Looking for inspiration? fzf-preview.vim [1] by yuki-yano [2]. The plugin that powerfully integrates fzf and (Neo)vim. It is also possible to integrate with coc.nvim. References: [1]: https://github.com/yuki-yano/fzf-preview.vim [2]: https://github.com/yuki-yano

⭐ ParthS007 background

If you’re into interesting projects, don’t miss out on background [1], created by ParthS007 [2]. Runs things in the background. References: [1]: https://github.com/ParthS007/background [2]: https://github.com/ParthS007

⭐ JedWatson react-select

The work on react-select [1] by JedWatson [2]. The Select Component for React.js References: [1]: https://github.com/JedWatson/react-select [2]: https://github.com/JedWatson

⭐ TaDaa vimade

I recently discovered vimade [1] by TaDaa [2], and it’s truly impressive. Vimade let’s you dim, fade, tint, animate, and customize colors in your windows and buffers for (Neo)vim References: [1]: https://github.com/TaDaa/vimade [2]: https://github.com/TaDaa

Update Git User

This morning I log into my VCS and check activity on my projects to find that someone else has been very active on my projects fo the last few weeks. I quicklyhover over the missing avatar to find that It’s Me. What’s wrong here, why do I look like two different people throughout the day! upon further investigation I see the issue. while setting up a new terminal environment I mistyped my email address by one character. After much searching and a few failed attempts I was able to fix it by following an article no longer available (2021) from https://help.github.com/articles. Bare Clone # [1] Clone the repo, note it must be a --bare clone. git clone --bare https://github.com/user/repo.git cd repo.git git-author-rewrite # [3] Curl down the git-author-rewrite script and edit the following variables OLD_EMAIL CORECT_NAME CORRECT_EMAIL curl https://gist.githubusercontent.com/octocat/0831f3fbd83ac4d46451/raw/c197afe3e9ea2e4218f9fccbc0f36d2b8fd3c1e3/git-author-rewrite.sh > git-author-...

2019 goals

strong { /* color: goldenrod; */ /* text-shadow: 0rem 1px 1px goldenrod; */ } The year of intenionality This is a follow up to my previous post 2018-retrospective [1] professional # [2] This year I will become more productive, by intensionally working on a well thought out plan, learning the right technologies, and leave behind a positive legacy. Productivity # [3] Last year I was able to make some great strides in my productivity and focus. This year I want to be able to bring it up a notch. Intentionality plays a big part in this. Taking some time to sit down and think about the tasks you realistically want to complete for the day, then focusing on those tasks one at a time. I started off the year last year using the pomodoro system to focus on one task and one task only for 25 minutes at a time. This worked really well for me but I quickly fell off of the train and fell into my old trap of reacting to the loudest customer over the items I put intensionally on my todo lis...

⭐ engineer-man youtube

Check out engineer-man [1] and their project youtube [2]. Code from the Engineer Man YouTube channel. Please do not submit pull requests, they will be ignored/closed. The code in the repo needs to remain as it was in the video. References: [1]: https://github.com/engineer-man [2]: https://github.com/engineer-man/youtube

2018 Retrospective

2018 was a year of many ups and downs, and learning to deal with a whole new set of problems professionally and at home. In 2018 I logged in to my first Linux system, setup my own webserver, data pipelines, database. I learned to use react and d3. Stepped up my python, javascript, and sql. At home the doctor appointments keep piling in. While I am learning to deal with it all there were several times throughout the year that I was very overwhelmed with everything and broke down. Here are the goals I had listed out for 2018 and how I faired at completing them. Positivity # [1] The Good # [2] --- - Continue “Favorite Things” with family at dinner time - Take 2 10min breaks per day clear the mind --- I can say that the kids are fully on board with favorite things and love to tell everyone about how their day went. This is a time that they are very positive and generally give praise to another family member for helping them through their day. To Be Improved # [3] --- - Grati...

Do More of What Brings You Joy

Today I want to take some time to talk about the things that make me happy in my work environment. This is completely free-flow off the cuff, but are things that I do that make me happy, not having them would definitely be a deal breaker for me . Attitude # [1] A positive work atmosphere goes a long ways. We all have enough negativity going on in our lives that is out of our control accepting any negativity in the workplace is a no go for me. There was a point in which I was suffering a lot of negativity at work. This began to trickle into every part of life, and it became hard to see positivity anywhere. My wife not only noticed this, but put a polite request in for change. It was definitely a low point and she could tell tell that my work life was not helping anything. [2] Throw out the negativity - courtesy giphy [3] Bring the positivity to your workplace. It is really difficult for folks to tear you down if you are the one that is always up beat and happy. People will notic...

⭐ wesm feather

feather [1] by wesm [2] is a game-changer in its space. Excited to see how it evolves. Feather: fast, interoperable binary data frame storage for Python, R, and more powered by Apache Arrow References: [1]: https://github.com/wesm/feather [2]: https://github.com/wesm

⭐ nostalgic-css NES.css

Just starred NES.css [1] by nostalgic-css [2]. It’s an exciting project with a lot to offer. NES-style CSS Framework | ファミコン風CSSフレームワーク References: [1]: https://github.com/nostalgic-css/NES.css [2]: https://github.com/nostalgic-css

⭐ kennethreitz responder

kennethreitz [1] has done a fantastic job with responder [2]. Highly recommend taking a look. A familiar HTTP Service Framework for Python. References: [1]: https://github.com/kennethreitz [2]: https://github.com/kennethreitz/responder