Published

Packages # [1] Click [2] # [3] Inputs # [4] Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. Option # [5] - typically aliased with a shorthand (’-v’, ‘–verbose’) --- **From the Docs [6] To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores. @click.command() @click.option('-s', '--string-to-echo') def echo(string_to_echo): click.echo(string_to_echo) @click.command() @click.option('-s', '--string-to-echo', 'string') def echo(string): click.echo(string) --- Argument # [7] - positional - required - no help text supplied by click Yaspin [8] # [9] 88e1bcff-6a9c-4bd9-955c-fd130f2fa369.mp4 [10] Click Help Colors [11] # [12] [13] # [14] Colorama [15] # [16] Colorama Example [17] Click DidYouMean [18] # [19] References: [1]: #packages [2]: https://click.pal...

See all of my kedro related posts in [[ tag/kedro ]]. #kedrotips [1] # [2] I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips [3]. 🗣 Heads up # [4] Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful more usefule for exploration. 📚 Catalog # [5] [6] Photo by jesse orrico on Unsplash CSVLocalDataSet # [7] python import pandas as pd iris = pd.read_csv('https://raw.githubusercontent.com/kedro-org/kedro/d3218bd89ce8d1148b1f79dfe589065f47037be6/kedro/template/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/iris.csv') data_set = CSVLocalDataSet(filepath="test.csv", load_args=None, save_args={"index": False}) iris_data_set.save(iris) reloaded_iris = iris_data_se...

- jmespath - Tabnine Bulwark # [1] |-|-| |github: |https://github.com/zaxr/bulwark| I definitely want to try this out with kedro. Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+. Example # [2] import bulwark.decorators as dc @dc.IsShape((-1, 10)) @dc.IsMonotonic(strict=True) @dc.HasNoNans() def compute(df): # complex operations to determine result ... return result_df References: [1]: #bulwark [2]: #example

Using pdb # [1] References: [1]: #using-pdb

Pathlib is an amazing cross-platform path tool. Import # [1] from pathlib import Path Create path object # [2] Current Directory cwd = Path('.').absolute() Users Home Directory home = Path.home() module directory module_path = Path(__file__) Others Let’s create a path relative to our current module. data_path = Path(__file__) / 'data' Check if files exist # [3] Make Directories # [4] data_path.mkdir(parents=True, exists_ok=True) rename files # [5] Path(data_path /'example.csv').rename('real.csv') List files # [6] Glob Files # [7] data_path.glob('*.csv') recursively data_path.rglob('*.csv') Write # [8] Path(data_path / 'meta.txt').write_text(f'created on {datetime.datetime.today()}) References: [1]: #import [2]: #create-path-object [3]: #check-if-files-exist [4]: #make-directories [5]: #rename-files [6]: #list-files [7]: #glob-files [8]: #write

Custom Exceptions # [1] class ProjectNameError(NameError): pass class UserNameError(NameError): pass class CondaEnvironmentError(RuntimeError): pass class BucketNotDefinedError(NameError): pass References: [1]: #custom-exceptions

query # [1] Good for method chaining, i.e. adding more methods or filters without assigning a new variable. # is skus.query('AVAILABILITY == " AVAILABLE"') # is not skus.query('AVAILABILITY != " AVAILABLE"') masking # [2] general purpose, this is probably the most common method you see in training/examples # is skus[skus['AVAILABILITY'] == 'AVAILABLE'] # is not skus[~skus['AVAILABILITY'] == 'AVAILABLE'] isin # [3] capable of including multiple strings to include # is in df[df.AVAILABILITY.isin(['AVAILABLE', 'AVL'])] # is not in df[~df.AVAILABILITY.isin(['AVAILABLE', 'AVL'])] contains # [4] Good For partial matches # contains df[df.AVAILABILITY.str.contains('AVA')] # not contains df[~df.AVAILABILITY.str.contains('AVA')] MASKS # [5] anything that we put inside of square brackets can be set as a variable then passed in. service_mask = skus['AVAILABILITY'] == 'AVAILABLE' name_mask = skus['NAME'] == 'Dell chromebook 11' Operators # [6] & - and ~ - not | - or AVAILABLE and ...

I love digital ocean for it’s simplicity and its commitment to open source.