Inputs

The input will be a yaml file containing a list of Items you want to stay up to date with. Inside each item will be a url, and weight.

email:
    max-entries: 10
    recipients:
      - waylon@waylonwalker.com
markdown:
    max-entries: 100
    output:
        - README.md
json:
    max-entries: 1000
    output:
        - feeds/feed.json
rss:
    max-entries: 1000
    output:
        - feeds/feed.xml
html:
    max-entries: 100
    output:
        index.html

items:
    Waylon Walker:
    weight: 5
    url: https://waylonwalker.com/rss.xml
    @_WaylonWalker:
    weight: 3
    twitter: https://twitter.com/_waylonwalker
    DEV Waylon Walker:
    weight: 8
    url: https://dev.to/waylonwalker
    Stack Overflow Kedro:
    weight: 5
    url: https://stackoverflow.com/questions/tagged/kedro
    Kedro GitHub:
    weight: 4
    url: https://github.com/quantumblacklabs/kedro
    Kedro Pypi
        weight: 10
        url: https://pypi.org/project/kedro/

Types

• rss feed (primary source)
• youtube feed
• Stack Overflow tags
• GitHub repo activity
• pypi release
• dev.to post
• Twitter Search # user will need an api key

Methodology

Each url will be pulled in and parsed into a standard data scructure. Some items may yield special feaures, a schemaless/nosql datastructure may be best. Pipeline will decide to how to weight posts based on users weight, recent position on feed, .

a_raw (raw plain text / json items) -> b_int (cleaned items) -> c_pri (single feed of items) -> d_fea (weighted feed of items) -> e_out (requested output formats)

output

Pipeline outputs will be email, json, markdown, xml, html. Each will be able to be configured by the config file (max-entries, output location).

Running

Users will be able to create their own reader. Here is a list of possiblilites. Users will not have the pipeline inside their repo. It will pull the pipeline from a package repo, pypi, dockerhub, GitHub.

• fork a template repo (might be cumbersome to update)
• use a GitHub action from the Marketplace (easier to update)
• GH actions will run the pipeline on a schedule