Unix snippets for reference

command line (zsh)

• find files
  • by name

    find <location> -iname 'filename.ext' # accepts wildcards
    

    or zsh option

    ls -ld -- **/*name_fragment*
    

  • by type

    -find  <location> -type f -name ".conf"
    

• search directory recursively for search expression

    grep --include\*.{extensions} -rnw 'home/folder/' -e 'pattern'
  

• (zsh specific) Replace all instances of foo with bar in all files recursively

    sed -i -- 's/foo/bar/g' **/*(D.)
  

• recursively delete all .git folders (before backing up to an external drive without backing up all thousands of annoying .pack files)

    ( find . -type d -name ".git" && find . -name ".gitignore" && find . -name ".gitmodules" ) | xargs rm -rf
  

• zsh globbing loops

    for f in **/*.ext; do <command> $f; done
  

• symlinks (both files and directories) - particularly useful for overleaf projects

    ln -s <target> <link_name>
  

• hard drive space

    df -H
  

• chmod

    chmod +x script.sh
  

• chown

    chown -R user:domain folder
  

• scp (copy log files from ssh server to local directory)

    apoorva@age4.nber.org:apoorva/Do/\*.log ./
  

• populate sensible gitignores

    'echo "*.csv\n*.dta\n*.ipynb_checkpoints" >> .gitignore'
  

• Update linux distro

    sudo apt-get update
    sudo apt-get upgrade -y
    sudo apt-get dist-upgrade
    sudo do-release-upgrade
  

convert

• pngs into gif (for R/Python plots)

    convert -delay 45 -loop 0 *.png <gif-title>.gif
  

• pdfs to jpgs

    for i in `ls *.pdf`; do convert -density 300 "$i" "${i%%.*}.png"; done
  

• tsv to csv (hahvahd does tsv for some inexplicable reason)

  tr '\t' ',' < input.tab > output.csv
  

pdf operations

• combine

  pdfunite in1.pdf in2.pdf out.pdf
  

• slice large pdfs [either pdftk or qpdf]

  qpdf --empty --pages bigdoc.pdf 1-10 -- smalldoc.pdf
  

wget

• match extension/pattern

    wget <site> -nd -np -l1 -r -v  -A "<pattern>*.<extension>"

• mirror entire site

    wget -mkEpnp <site>

git

• init new .gitignore with sensible defaults

  echo '*.csv\n*.dta\n*.ipynb_checkpoints' >> .gitignore
  

• refresh repo after update to .gitignore

    git rm -r --cached . && git add .
  

• remove current remote and add new remote

    git rm remote origin && git remote add origin <repo link>
  

conda / jupyter / ipython - conda cheatsheet

• convert notebooks into scripts

    jupyter nbconvert --to script *.ipynb
  

• activate environment

    source activate <env>
  

• link pythonstartup file to default ipython profile

  ln -s ~/dotfiles/pythonstartup ~/.ipython/profile_default/startup/start.py
  

• add environment to kernelspec after install so it shows up in Jupyter Notebook/Lab and Atom

  source activate myenv
  conda install ipykernel
  python -m ipykernel install --user --name myenv --display-name "New Environment"
  

• export list of packages from current environment

    conda env export > requirements.txt
  

• delete environment

    conda env remove --name <env-name>
  

pandoc - ref

• generic

    pandoc -s <input file> -o <output file>
  

• markdown to latex

    pandoc -s <md> -o <tex>
  

• markdown to pdf with citations

    pandoc -s --bibliography biblio.bib --filter pandoc-citeproc CITATIONS -o example24a.html