Productive one on one
[1]Inspired by https://m.soundcloud.com/front-end-happy-hour/episode-095-shot-for-shot-one-on-ones
References:
[1]: https://images.waylonwalker.com/Screenshot_20200221-114202.png
Drafts
Draft and unpublished posts
0 posts
Long variable names are good
🏷️ Long variable names are a good thing. Self documenting code is more important than poorly documented code. Simply adding a few characters to your variable names can go a long ways.
Containers are plural # [1]
Aliases are welcome # [2]
Scope is important
References:
[1]: #containers-are-plural
[2]: #aliases-are-welcome
What is Refactoring
Make the change easy
Make the easy change
Martin Fowler
When to refactor
RSS feed for your Gatsby Site
Adding an rss feed to your gatsby js site is super simple.
https://www.gatsbyjs.org/packages/gatsby-plugin-feed/
Install # [1]
npm install --save gatsby-plugin-feed
How to use # [2]
// In your gatsby-config.js
module.exports = {
plugins: [
{
resolve: `gatsby-plugin-feed`,
options: {
query: `
{
site {
siteMetadata {
title
description
siteUrl
site_url: siteUrl
}
}
}
`,
feeds: [
{
serialize: ({ query: { site, allMarkdownRemark } }) => {
return allMarkdownRemark.edges.map(edge => {
return Object.assign({}, edge.node.frontmatter, {
description: edge.node.excerpt,
date: edge.node.frontmatter.date,
url: site.siteMetadata.siteUrl + edge.node.fields.slug,
guid: site.siteMetadata.siteUrl + edge.node.fields.slug,
custom_elements: [{ "content:encoded": edge.node.html }],
})
})
},
query: `
{
allMarkdownRemark(
sort: { order: DESC, fields: [frontmatter___date] },
) {
edges {
node {
excerpt
html
fields { slug }
frontmatter {
title
date
}
}
}
}
}
...
Conda Build
Conda Build
Install Conda Build # [1]
conda update conda
conda install conda-build
References:
[1]: #install-conda-build
What DataScientists Should Know About S3
What DataScientists Should Know About S3
Pyspark
I have been using pyspark since March 2019, here are my thoughts.
Making good documentation in python
Tools
Sphinx # [1]
Portray # [2]
I just started using portray and it is amazingly simple to use!
Methodology
References:
[1]: #sphinx
[2]: #portray
Amazon Web Services
I started using aws in March 2019. Here are some of my notes.
aws-cli # [1]
ec-2 # [2]
SageMaker # [3]
S3 # [4]
References:
[1]: #aws-cli
[2]: #ec-2
[3]: #sagemaker
[4]: #s3
Minimal Project Structure
TLDR # [1]
Use .gitignore.io [2] and consider adding an alias to your terminal to quickly add a .gitignore to any project missing one.
alias gitignore='curl https://www.gitignore.io/api/vim,emacs,python,pycharm,sublimetext,visualstudio,visualstudiocode,data > .gitignore'
Add a minimal setup.py to the root of your project, and use the following command to install it.
pip install -e .
consider using **cookiecutter [3]
References:
[1]: #tldr
[2]: https://www.gitignore.io
[3]: https://github.com/audreyr/cookiecutter
Python Tips
Dictionaries # [1]
Unpacking # [2]
- **kwargs
- func(**input)
- locals().update(d)
# [3]
References:
[1]: #dictionaries
[2]: #unpacking
[3]: #heading
remove git cruft
inspiration # [1]
My original inspiration for this post came from steven ostermiller’s blog post
that no longer exists from my last check in May, 2024.
https://blog.ostermiller.org/removing-and-purging-files-from-git-history/
I was able to find it on the way back machine though.
https://web.archive.org/web/20240222195617/https://blog.ostermiller.org/removing-and-purging-files-from-git-history/
git log --all --pretty=format: --name-only --diff-filter=D | sed -r 's|[^/]+$||g' | sort -u
git filter-branch --tag-name-filter cat --index-filter 'git rm -r --cached --ignore-unmatch FILE_LIST' --prune-empty -f -- --all
rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --aggressive --prune=now
git push origin --force --all
git push origin --force --tags
cd MY_LOCAL_GIT_REPO
git fetch origin
git rebase
git reflog expire --expire=now --all
git gc --aggressive --prune=now
References:
[1]: #inspiration
Follow Your Passion
Example # [1]
References:
[1]: #example
Follow Your Passion
Follow Your Passion # [1]
my journey into data science
In January 2018 I started work as a full time data scientist turning my passion into a career. It is something that I didn’t see myself doing 5 years ago, but is something that I love to do. It combines my love of data, visualization, story telling, software development, and writing code. Most of all it allows me to work in a space that promotes learning and creativity. As a mechanical engineer for a company that has been building equipment for nearly a century the mechanical engineering is very well established I felt that there was not a lot of room for creativity.
Find Your Role # [2]
When I first started as a full time mechanical engineer
References:
[1]: #follow-your-passion
[2]: #find-your-role
Generating Readme Tables From Pandas
Generating Readme Tables From Pandas # [1]
I commonly have a need to paste the first few lines of a dataset into a markdown file. I use two handy packages to do this, tabulate and pyperclip. Lets say I have a Pandas DataFrame in memory as df already. All I would need to do to convert the first 5 rows to markdown and copy it to the clipboard is the following.
from tabulate import tabulate
import pyperclip
md = tabulate.tabulate(df.head(), df.columns, tablefmt='pipe')
pyperclip.copy(md)
This is a super handy snippet that I use a lot. Folks really appreciate it when they can see a sample of the data without opening the entire file.
References:
[1]: #generating-readme-tables-from-pandas
Pycon 2018 Roundup
These are my notes from pycon 2018 videos. I love the python community and especially the conference talks. This year I am going to take some notes from my favorite talks and post them here.
This is an Incomplete working post.
Jake VanderPlas - Performance Python: Seven Strategies for Optimizing Your Numerical Code [1] # [2]
- Always profile before making any optimizations.
- Vectorize with Numpy
- Looping in python can be slow
- Use specialized data structures.
- scipy.spacial
- pandas
- xarray
- scipy.sparse
- sparse package
- scipy.sparce.csgraph
- Cython
- Add types
- Numba
- jit
- Fortran Like Speed
- heavy dependencies
- Dask
- distributed tasks
- Can be executed locally or on a cluster
- Look for an existing package
- resist the urge to reinvent the wheel
https://www.youtube.com/watch?v=zQeYx87mfyw
Justin Crown - “WHAT IS THIS MESS?” - Writing tests for pre-existing code bases - PyCon 2018 [3] # [4]
This was a great talk about not only test driven de...
Stepping Up My SQL Game
In 2018 I transitioned from a Product Engineering (Mechanical) role to a Data Scientist Role. I entered this space with strong subject matter expertise with our products, our data, munging through data in pyhon, and data visualization in python. My sql skills were lacking to say the least. I had learned what I needed to know to get data from our relational databases, then use pandas to do any further analysis. Just run something like the following and you have data.
SELECT
*
FROM
Table
Where
col_1 = 'col_1_filter'
This technique works great for small data sets that you only need to run once. There is no shame to pull in a big dataset and start munging with it in pandas to get some results, and make decisions. The problem becomes when your dataset becomes too big or you need to run the query on a frequent basis. Doing the aggregations on the server run much quicker, as it reduces the time spent in io. My longest running steps are currently io related. Reducing these steps have im...
My favorite pandas pattern
My favorite pandas pattern
I work with a lot of transactional timeseries data that includes categories. I often want to create timeseries plots with each category as its own line. This is the method that I use almost data to achieve this result. Typically the data that am working with changes very slowly and trends happen over years not days or weeks. Plotting daily/weekly data tends to be noisy and hides the trend. I use this pattern because it works well with my data and is easy to explain to my stakeholders.
import pandas as pd
import numpy as np
% matplotlib inline
Lets Fake some data # [1]
Here I am trying to simulate a subset of a large transactional data set. This could be something like sales data, production data, hourly billing, anything that has a date, category, and value. Since we generated this data we know that it is clean. I am still going to assume that it contains some nulls, and an irregular date range.
n = 365*5
cols = {'level_0': 'date',
'level_1': 'item',
...