Amazon Web Services
I started using aws in March 2019. Here are some of my notes.
aws-cli # [1]
ec-2 # [2]
SageMaker # [3]
S3 # [4]
References:
[1]: #aws-cli
[2]: #ec-2
[3]: #sagemaker
[4]: #s3
Minimal Project Structure
TLDR # [1]
Use .gitignore.io [2] and consider adding an alias to your terminal to quickly add a .gitignore to any project missing one.
alias gitignore='curl https://www.gitignore.io/api/vim,emacs,python,pycharm,sublimetext,visualstudio,visualstudiocode,data > .gitignore'
Add a minimal setup.py to the root of your project, and use the following command to install it.
pip install -e .
consider using **cookiecutter [3]
References:
[1]: #tldr
[2]: https://www.gitignore.io
[3]: https://github.com/audreyr/cookiecutter
Python Tips
Dictionaries # [1]
Unpacking # [2]
- **kwargs
- func(**input)
- locals().update(d)
# [3]
References:
[1]: #dictionaries
[2]: #unpacking
[3]: #heading
remove git cruft
inspiration # [1]
My original inspiration for this post came from steven ostermiller’s blog post
that no longer exists from my last check in May, 2024.
https://blog.ostermiller.org/removing-and-purging-files-from-git-history/
I was able to find it on the way back machine though.
https://web.archive.org/web/20240222195617/https://blog.ostermiller.org/removing-and-purging-files-from-git-history/
git log --all --pretty=format: --name-only --diff-filter=D | sed -r 's|[^/]+$||g' | sort -u
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch FILE_LIST' --prune-empty -f -- --all
rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --aggressive --prune=now
git push origin --force --all
git push origin --force --tags
cd MY_LOCAL_GIT_REPO
git fetch origin
git rebase
git reflog expire --expire=now --all
git gc --aggressive --prune=now
References:
[1]: #inspiration
Follow Your Passion
Example # [1]
References:
[1]: #example
Follow Your Passion
Follow Your Passion # [1]
my journey into data science
In January 2018 I started work as a full time data scientist turning my passion into a career. It is something that I didn’t see myself doing 5 years ago, but is something that I love to do. It combines my love of data, visualization, story telling, software development, and writing code. Most of all it allows me to work in a space that promotes learning and creativity. As a mechanical engineer for a company that has been building equipment for nearly a century the mechanical engineering is very well established I felt that there was not a lot of room for creativity.
Find Your Role # [2]
When I first started as a full time mechanical engineer
References:
[1]: #follow-your-passion
[2]: #find-your-role
Generating Readme Tables From Pandas
Generating Readme Tables From Pandas # [1]
I commonly have a need to paste the first few lines of a dataset into a markdown file. I use two handy packages to do this, tabulate and pyperclip. Lets say I have a Pandas DataFrame in memory as df already. All I would need to do to convert the first 5 rows to markdown and copy it to the clipboard is the following.
from tabulate import tabulate
import pyperclip
md = tabulate.tabulate(df.head(), df.columns, tablefmt='pipe')
pyperclip.copy(md)
This is a super handy snippet that I use a lot. Folks really appreciate it when they can see a sample of the data without opening the entire file.
References:
[1]: #generating-readme-tables-from-pandas
Pycon 2018 Roundup
These are my notes from pycon 2018 videos. I love the python community and especially the conference talks. This year I am going to take some notes from my favorite talks and post them here.
This is an Incomplete working post.
Jake VanderPlas - Performance Python: Seven Strategies for Optimizing Your Numerical Code [1] # [2]
- Always profile before making any optimizations.
- Vectorize with Numpy
- Looping in python can be slow
- Use specialized data structures.
- scipy.spacial
- pandas
- xarray
- scipy.sparse
- sparse package
- scipy.sparce.csgraph
- Cython
- Add types
- Numba
- jit
- Fortran Like Speed
- heavy dependencies
- Dask
- distributed tasks
- Can be executed locally or on a cluster
- Look for an existing package
- resist the urge to reinvent the wheel
https://www.youtube.com/watch?v=zQeYx87mfyw
Justin Crown - “WHAT IS THIS MESS?” - Writing tests for pre-existing code bases - PyCon 2018 [3] # [4]
This was a great talk about not only test driven de...
Stepping Up My SQL Game
In 2018 I transitioned from a Product Engineering (Mechanical) role to a Data Scientist Role. I entered this space with strong subject matter expertise with our products, our data, munging through data in pyhon, and data visualization in python. My sql skills were lacking to say the least. I had learned what I needed to know to get data from our relational databases, then use pandas to do any further analysis. Just run something like the following and you have data.
SELECT
*
FROM
Table
Where
col_1 = 'col_1_filter'
This technique works great for small data sets that you only need to run once. There is no shame to pull in a big dataset and start munging with it in pandas to get some results, and make decisions. The problem becomes when your dataset becomes too big or you need to run the query on a frequent basis. Doing the aggregations on the server run much quicker, as it reduces the time spent in io. My longest running steps are currently io related. Reducing these steps have im...
My favorite pandas pattern
My favorite pandas pattern
I work with a lot of transactional timeseries data that includes categories. I often want to create timeseries plots with each category as its own line. This is the method that I use almost data to achieve this result. Typically the data that am working with changes very slowly and trends happen over years not days or weeks. Plotting daily/weekly data tends to be noisy and hides the trend. I use this pattern because it works well with my data and is easy to explain to my stakeholders.
import pandas as pd
import numpy as np
% matplotlib inline
Lets Fake some data # [1]
Here I am trying to simulate a subset of a large transactional data set. This could be something like sales data, production data, hourly billing, anything that has a date, category, and value. Since we generated this data we know that it is clean. I am still going to assume that it contains some nulls, and an irregular date range.
n = 365*5
cols = {'level_0': 'date',
'level_1': 'item',
...
Pug Reveal
Pug Reveal Slides
I recently gave a presentation at the Big Brothers and Big Sisters Data Challenge. I wanted to use reveal to create my slides. I have used it before and it is a really nice package. Compared to PowerPoint it is much easier to incorporate interactive visualizations right into the presentation,easy to re factor and maintain slides. Since you are just working with text you can easily convert from a list of items on one slide to a set of slides.
Avoiding Death by PowerPoint # [1]
If you have not seen David JP Phillips Death By PowerPoint [2] TEDx, stop now and watch it. You will never look at slides the same again. Watching this video ruined me for watching presentations with these issues. Reveal is a tool that makes it very easy to follow these principles
- You are the center of focus
- Reduce clutter
- Focus your users with
- size
- contrast
Reduce clutter # [3]
In the video David talks about reducing the number of points we have down to no more than 6. You...
No More Death By PowerPoint
No More Death By PowerPoint
I Waylon S. Walker vow that from this point forward I will no longer create powerpoints to be considerec DEATH BY POWERPOINT
If you have not seen David JP Phillips Death By PowerPoint [1] TEDx, stop now and watch it. You will never look at slides the same again. Watching this video ruined me for watching presentations with these issues. Reveal is a tool that makes it very easy to follow these principles
I currently work in a company that employs over 100K employees, and to this day I cannot recall a single presentation given where the slides did not violate the rules stated in David’s Talk. This year I am putting a stop to this starting with myself. I am starting a new job role in 2018 and there is no better time to make some drastic changes to my workflow than now. I expect there to be a few followers and many naysayers, but I dont care. I will employ the directives listed below.
1+1=0 # [2]
Very few people (if any) in your will be able to multi-tas...
background tasks in python
I have tried most of the different methods in the past and found that copying and pasting the threadpoolexecutor example [1] or the processpoolexecutor example [2] from the standard library documentation to be the most reliable. Since this is often something that I stuff in the back of a utility module of a library it is not something that I write often enough to be familiar with, which makes it both hard to write and hard to read and debug. If you are looking for a good overview of the difference concurrency Raymond Hettinger [3] has a great talk about the difference between the various different methods, when to use them and why.
Recently a new python library was released to make running tasks in the background very simple. The background [4] project by Kenneth Reitz is a high level implementation of python 3’s ThreadPoolExecutor. I have been playing around with this project over the last week and I will say that this is definitely the simplest way to run background tasks in pyth...
Pycon 2017 Roundup
Pycon 2017 Roundup
Good afternoon fellow Data Geeks. Last week Pycon [1] released 141 videos of greatness. Here are my top picks from the event.
#3 Kelsey Hightower - Keynote - Pycon 2017 # [2]
https://www.youtube.com/watch?v=u_iAXzy3xBA&t=1795s [3]
#2 Al Sweigart Yes, It’s Time to Learn Regular Expressions PyCon 2017 # [4]
https://www.youtube.com/watch?v=abrcJ9MpF60
#1 Trey Hunner Readability Counts PyCon 2017 # [5]
https://www.youtube.com/watch?v=knMg6G9_XCg
What’s on Tap # [6]
This afternoon we have a cup of from one of my favorite roasters Thirty Thiry Coffee. This
[7]
References:
[1]: https://www.youtube.com/channel/UCrJhliKNQ8g0qoE_zvL8eVg
[2]: #3-kelsey-hightower---keynote---pycon-2017
[3]: https://www.youtube.com/watch?v=u_iAXzy3xBA&t=1795s
[4]: #2-al-sweigart-yes-its-time-to-learn-regular-expressions-pycon-2017
[5]: #1-trey-hunner-readability-counts-pycon-2017
[6]: #whats-on-tap
[7]: https://www.thirty-thirtycoffee.com/
Gh Rm Workflow Runs
date: 2022-09-01 19:12:34 templateKey: til title: GitHub Actions Delete all Workflow Runs published: true tags: - bash
Top Banner
Announcement banner shown above the site header -- included via the top_banner slot.
Sidebar
Homepage sidebar content -- included by home.html via include_post.
Now Intro
Homepage now card -- included by home.html via include_post.
Nav Content
Markdown-driven navigation -- replaces the config-driven nav via the nav_content slot.
Left Sidebar
Left sidebar with navigation links -- included via the left_sidebar slot.