pythonistas

Pythonistas

Source code for Pythonistas Meetups

#python #automation #docker #gui #terraform #k8s #svelte #flask #django #linux
Demo Download

pythonistas

Summary

This repo contains the source code for Pythonistas Meetups. For the repos that predate this one, they'll be included as submodules.

Caveat Emptor

Very little of this gets tested on Windows hosts. Windows Subsystem for Linux (WSL) is used where necessary with the default Ubuntu LTS install.

Be the change et al if Windows is your main and you wanna raise a PR with broad instructions on getting tooling working under Windows (e.g., docker, poetry, playwright.)

Table of Contents

Setup

Usage

Mac and Linux users

Development environments and tooling are first-class citizens on macOS and *nix. For Windows faithfuls, please setup WSL below.

Windows Subsytem for Linux (wsl)

WSL allows Windows users to run Linux (Unix) locally at a system-level. All of the standard tooling is used and community guides can be followed without standard Windows caveats (e.g., escaping file paths, GNU utilities missing, etc.)

  • Install from the Setup section
  • Enable
    • Start Menu > search for "Turn Windows Features On" > open > toggle "Windows Subsystem for Linux"
    • Restart
  • M1 Macs only (Intel Macs and native Windows boxes need not apply)
    • Revert WSL 2 to WSL 1 due to nested virtualization not being available at a hardware level
      wsl --set-default-version 1
    • Docker won't run without paravirtualization enabled, but the rest of the development environment will work as expected
  • Install Ubuntu
    # enable default distribution (Ubuntu)
wsl --install ubuntu
  • Start Linux and prep for environment setup
    # launch Ubuntu
ubuntu

# upgrade packages (as root: `sudo -s`)
apt update && apt upgrade -y

# create standard user
adduser <username>
visudo

# search for 'Allow root to run any commands anywhere', then append identical line with new user
root            ALL=(ALL)       ALL
<username>      ALL=(ALL)       ALL

# Allow members of group sudo to execute any command
%sudo  ALL=(ALL) NOPASSWD: ALL
  • Additional configuration options
    • Configuration locations
      • WSL 1: /etc/wsl.conf
      • WSL 2: ~/.wslconfig
          # set default user
  [user]
  default=<username>

  # mounts host drive at /mnt/c/
  [automount]
  enabled = true
  options = "uid=1000,gid=1000"

  # WSL2-specific options
  [wsl2]
  memory = 8GB   # Limits VM memory in WSL 2
  processors = 6 # Makes the WSL 2 VM use two virtual processors
    • After making changes to the configuration file, WSL needs to be shutdown for 8 seconds
      • wsl --shutdown
    • OPTIONAL: Change home directory to host Windows' home
      # copy dotfiles to host home directory
cp $HOME/.* /mnt/c/Users/<username>

# edit /etc/passwd
<username>:x:1000:1000:,,,:/mnt/c/Users/<username>:/bin/bash

asdf

  • Install from the Setup section
  • WSL/Ubuntu Linux dependencies
    sudo apt update && sudo apt install \
make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget \
curl llvm libncursesw5-dev xz-utils tk-dev \
libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
  • Fedora dependencies
    sudo dnf install -y bzip2-devel libsqlite3x-devel
  • All operating systems
    # add python plugin
asdf plugin-add python

# install stable python
asdf install python latest

# uninstall version
asdf uninstall python 3.9.6

# refresh symlinks for installed python runtimes
asdf reshim python

# set stable to system python
asdf global python latest

# optional: local python (e.g., python 3.9.10)
cd $work_dir
asdf list-all python 3.9
asdf install python 3.9.10
asdf local python 3.9.10

# verify python shim in use
asdf current

# check installed python
asdf list python

pip

If a basic virtual environment (venv) and requirements.txt are all that's needed, can use built-in tools.

# create a virtual environment via python
## built-in
python3 -m venv .venv

## faster
python3 -m pip install virtualenv # _or_ pipx install virtualenv
virtualenv .venv

# activate virtual environment
source .venv/bin/activate

# install dependencies
python3 -m pip install requests inquirer

# generate requirements.txt
python3 -m pip freeze > requirements.txt

# exit virtual environment
deactivate

Poetry

  • NOTE: it's possible to use the built-in .venv virtual environment (e.g., troubleshooting SolverProblemError dependency hell)
    poetry env use .venv/bin/python
  • Install from the Setup section
  • Normal usage
    # Install (modifies $PATH)
curl -sSL https://install.python-poetry.org | $(which python3) - # append `--no-modify-path` to EOL if you know what you're doing 

# Change config
poetry config virtualenvs.in-project true           # .venv in `pwd`
poetry config experimental.new-installer false      # fixes JSONDecodeError on Python3.10

# Activate virtual environment (venv)
poetry shell

# Deactivate venv
exit  # ctrl-d

# Install multiple libraries
poetry add google-auth google-api-python-client

# Initialize existing project
poetry init

# Run script and exit environment
poetry run python your_script.py

# Install from requirements.txt
poetry add `cat requirements.txt`

# Update dependencies
poetry update

# Remove library
poetry remove google-auth

# Generate requirements.txt
poetry export -f requirements.txt --output requirements.txt --without-hashes

# Uninstall Poetry (e.g., troubleshooting)
POETRY_UNINSTALL=1 bash -c 'curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py' | $(which python3) -

Docker

  • Install from the Setup section
  • Usage
    # clean build (remove `--no-cache` for speed)
docker-compose build --no-cache --parallel

# start container
docker-compose up --remove-orphans -d

# exec into container
docker attach hello

# run command inside container
python hello.py

# stop container
docker-compose stop

# destroy container and network
docker-compose down

Playwright

  • Install from the Setup section
  • Usage
    # install
pip install --upgrade pip
pip install playwright
playwright install

# download new browsers (chromedriver, gecko)
npx playwright install

# generate code via macro
playwright codegen wikipedia.org

Django

  • Follow the official Django Docker Compose article
    • cd django
    • Generate the server boilerplate code
      docker-compose run web django-admin startproject composeexample .
    • Fix upstream import bug and whitelist all hosts/localhost
      $ vim composeexample/settings.py
import os
...
ALLOWED_HOSTS = ["*"]
    • Profit
      docker-compose up
    • Optional: Comment out Django exclusions for future commits
      • Assumed if extracting Django boilerplate from template and creating a new repo
        # .gitignore
# ETC
# django/composeexample/
# django/data/
# django/manage.py

Flask

FastAPI

Kubernetes (k8s)

  • Easiest way to set up a local single node cluster is via one of the following:
  • Add aliases to ~/.bashrc or ~/.zshrc
    # k8s
alias k="kubectl"
alias kc="kubectl config use-context"
alias kns='kubectl config set-context --current --namespace'
alias kgns="kubectl config view --minify --output 'jsonpath={..namespace}' | xargs printf '%s\n'"
KUBECONFIG="$HOME/.kube/config:$HOME/.kube/kubeconfig:$HOME/.kube/k3s.yaml"
  • CLI/TUI (terminal user interface) management of k8s
    • k9s
      • Built-in help: type ?
      • Main Screen: :pod

      • Describe a pod: d

      • Delete a pod:
        • ctrl-d
        • Replicas rebuild

      • Remove resource (permanent)
        • :deploy
        • ctrl-d

  • POC
    git clone https://github.com/jwsy/simplest-k8s.git
k config get-contexts     # should have `rancher-desktop` selected
kc rancher-desktop        # switch to rancher context if not
k apply -f simplest-k8s
k delete -f simplest-k8s

Terraform

  • NOTES:
    • This section depends on Kubernetes and a ~/.kubeconfig from above
    • NodePort was used instead of LoadBalancer for service.type
  • Install terraform via `asdf
    # terraform
asdf plugin-add terraform
asdf install terraform latest
  • Add aliases to ~/.bashrc or ~/.zshrc
    # ~/.bashrc
alias tf='terraform'
alias tfi='terraform init -backend-config=./state.conf'
alias tfa='terraform apply'
alias tfp='terraform plan'
alias tfpn='terraform plan -refresh=false'
  • Navigate to ./terraform/ and initialize the terraform working directory
    cd terraform/
tfi
  • Create an execution plan
    tfp
  • Apply/execute the actions from Terraform plan
    tfa
  • Navigate to http://localhost:<port>
    • Port can be found via kubectl
      k get svc   # 80:31942/TCP
  • Tear down deployment
    tf destroy
    • Real-time view of pod removal

GitHub Actions

Update submodules recursively

  • Add the submodule to the downstream repo

    git submodule add https://github.com/pythoninthegrass/automate_boring_stuff.git
git commit -m "automate_boring_stuff submodule"
git push

  • Create a personal access token called PRIVATE_TOKEN_GITHUB with repo permissions on the downstream repo

    • repo:status
    • repo_deployment
    • public_repo

  • Add that key to the original repo

    • Settings > Security > Secrets > Actions
    • New repository secret

  • Setup a new Action workflow

    on: push: branches: - main - master

    jobs: update: runs-on: ubuntu-latest

      steps:
  - uses: actions/checkout@v2
      with: 
      repository: username/repo_name
      token: ${{ secrets.PRIVATE_TOKEN_GITHUB }}

  - name: Pull & update submodules recursively
      run: |
      git submodule update --init --recursive
      git submodule update --recursive --remote
  - name: Commit
      run: |
      git config user.email "actions@github.com"
      git config user.name "GitHub Actions - update submodules"
      git add --all
      git commit -m "Update submodules" || echo "No changes to commit"
      git push

    ```

Debugging

asdf

No version set for command python

  • Make sure python or python3 isn't aliased in ~/.bashrc or ~/.zshrc

bash - Is it possible to check where an alias was defined? - Unix & Linux Stack Exchange

Docker

  • Watch logs in real-time: docker-compose logs -tf --tail="50" hello
  • Check exit code
    $ docker-compose ps
Name                          Command               State    Ports
------------------------------------------------------------------------------
docker_python      python manage.py runserver ...   Exit 0

PATH

  • asdf, poetry, and python all need to be sourced in your shell $PATH in a specific order
    • asdf stores its Python shims in ~/.asdf/shims
    • poetry lives in ~/.local/bin
      export ASDF_DIR="$HOME/.asdf"
export PATH="$ASDF_DIR/shims:$HOME/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
  • Furthermore, any aliases or alias files need to be sourced as well
    . "$ASDF_DIR/asdf.sh"
. "$ASDF_DIR/completions/asdf.bash"
. /usr/local/etc/profile.d/poetry.bash-completion

    Terraform

  • Verbose logging and redirection to a file
    export TF_LOG="trace"                       # unset via "off"
export TF_LOG_PATH="$HOME/Downloads/tf.log" # `~` doesn't expand
  • Error: cannot re-use a name that is still in use

    I think I resolved the issue. This is what I did: 1) mv the terraform.tfstate to another name, 2) mv the terraform.tfstate.backup to terraform.tfstate, and 3) run 'terraform refresh' command to confirm the state is synchronized, and 4) run 'terraform apply' to delete/create the resource. I will mark your reply as the answer, as it gives me the clue for solving the issue. Thanks! – ozmhsh Dec 9, 2021 at 4:57

    nginx - Stuck in the partial helm release on Terraform to Kubernetes - Stack Overflow

TODO

  • pipx ```bash

    Install

    python3 -m pip install --user pipx python3 -m pipx ensurepath

Usage

...

* Django
   * Merge with [docker_python](https://github.com/pythoninthegrass/docker_python) and put the latter on an ice float
* Flask
   * Bonus points for [Svelte](https://svelte.dev/blog/the-easiest-way-to-get-started) front-end ❤️
* FastAPI
* k8s
 * `~/.kubeconfig`
* ansible
* wsl
 * VSCode
     * Remote WSL install and usage
       * Or at least further reading nods
* Debugging
  * Dependencies
  * script itself via [icecream](https://github.com/gruns/icecream)

## Further Reading
[Basic writing and formatting syntax - GitHub Docs](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax)

[venv — Creation of virtual environments — Python 3.7.2 documentation](https://docs.python.org/3/library/venv.html)

[pip freeze - pip documentation v22.0.3](https://pip.pypa.io/en/stable/cli/pip_freeze/)

[Introduction | Documentation | Poetry - Python dependency management and packaging made easy](https://python-poetry.org/docs/)

[Commands | Documentation | Poetry - Python dependency management and packaging made easy](https://python-poetry.org/docs/cli#export)

[Overview of Docker Compose | Docker Documentation](https://docs.docker.com/compose/)

[Compose file version 3 reference | Docker Documentation](https://docs.docker.com/compose/compose-file/compose-file-v3/)

[Commands](https://k9scli.io/topics/commands/)

[Speed up administration of Kubernetes clusters with k9s | Opensource.com](https://opensource.com/article/20/5/kubernetes-administration)

[Getting started | Playwright Python | codegen macro](https://playwright.dev/python/docs/intro)

[Install WSL | Microsoft Docs](https://docs.microsoft.com/en-us/windows/wsl/install)

[Set up a WSL development environment | Microsoft Docs](https://docs.microsoft.com/en-us/windows/wsl/setup/environment)

[Advanced settings configuration in WSL | Microsoft Docs](https://docs.microsoft.com/en-us/windows/wsl/wsl-config)

[Understanding The Python Path Environment Variable in Python](https://www.simplilearn.com/tutorials/python-tutorial/python-path)

Top categories

tailwind daisyui admin template popup mdsvex portfolio blog form ecommerce ui carousel auth dark seo image routing
Loading Svelte Themes