Problem definition

The A4 scrapbook we were meant to use was too small to have full-sized 4x6 prints glued in, so I needed to print several images on one 4x6 card and then cut them apart. 4 images per page seemed to be the “sweet spot” in our case.

Initially, I tried some freely available collage tools for Mac, but found them annoying, cumbersome and very out-of-date - and not really supporting bulk processing 52 images at once - so I ended up with a command-line based solution.

Setup

• My photo setup is pretty basic: my iPhone and its default camera app ¹. I think the native dimensions of images on the iPhone are 4032 px x 3024 px…

• I have an Epson EcoTank 2750 printer, which I absolutely love because it’s an easily refillable colour inkjet. This makes printing all of the squizillion worksheets and handouts in my life inexpensive and reasonably fast. I recently discovered that this printer can print photos in decent quality (at least as well as you used to be able to get at the corner drugstore back when I was a child + we shot on film…), provided you buy good quality photo paper.

Process

To solve the “Koala’s adventures” dilemma we took the following steps:

-rw-r--r--@ 1 darya  macmini  8382495 31 Oct 14:48 koalaexport - 1.jpeg
-rw-r--r--@ 1 darya  macmini  9401795 31 Oct 14:48 koalaexport - 2.jpeg
-rw-r--r--@ 1 darya  macmini  8078024 31 Oct 14:48 koalaexport - 3.jpeg
-rw-r--r--@ 1 darya  macmini  8789539 31 Oct 14:48 koalaexport - 4.jpeg

for file in *; do mv "$file" `echo $file | tr ' ' '_'` ; done

-rw-r--r--@ 1 darya  macmini  8382495 31 Oct 14:48 koalaexport_-_1.jpeg
-rw-r--r--@ 1 darya  macmini  9401795 31 Oct 14:48 koalaexport_-_2.jpeg
-rw-r--r--@ 1 darya  macmini  8078024 31 Oct 14:48 koalaexport_-_3.jpeg
-rw-r--r--@ 1 darya  macmini  8789539 31 Oct 14:48 koalaexport_-_4.jpeg

for ((i=0;i<=12;i++)); do montage -geometry 1800x1200+0+0 -tile 2x2 koalaexport_-_$(( 4*i + 1 )).jpeg koalaexport_-_$(( 4*i + 2 )).jpeg koalaexport_-_$(( 4*i + 3 )).jpeg koalaexport_-_$(( 4*i + 4 )).jpeg ${i}.tiff;done

for ((i=0;i<=13;i++)); do echo $i;done

The “why”

Just like working on our local machines, when working on an HPC cluster we really want to be able to have a consistent R environment for each of our projects that we snapshot at the time of working on said project. This means code that was written months or even years ago will still work as expected, even if the packages on CRAN have since been updated with breaking changes. The renv package is brilliant at managing this - however, it is not intuitive to figure out how to use it when working on an HPC cluster - which is what I work through in the below post.

Assumptions

This post makes a few assumptions:

1. You are planning to use R for analysis, and know how to install packages on your local machine.
2. You are working with a PBS-Pro HPC cluster, and know all of the core commands (this is not an introduction to ssh, qsub, module load etc - see here for an intro). I think some of the below should generalise to SLURM or other cluster schedulers, but don’t have access to them so can’t test.
3. You want to use the renv library to manage the packages in your project (You do, you really do - I cannot recommend this highly enough, especially for sanity of future you). You know the basics of using renv on your local machine, as per the renv vignette - that’s probably the best intro to renv.

Oof, now that that’s out of the way, let’s dive in :dolphin: …

Step 1: ssh into your cluster and fire up an interactive session

After you login to your HPC server, I recommend starting up an interactive session so you’re not working on and hence clogging up the login nodes. (You can do this on a login node too, but depending on how much compute prototyping your analysis requires this may crash :fire: - or get you a cranky email from an admin :rage: ).

qsub -I -l walltime=4:0:0 -l select=1:ncpus=1:mem=10gb

Step 2: Launch R and install renv

After you’re logged in to the worker node, run the following commands to:

• navigate to the project directory
• get rid of everything that may have inadvertently been preloaded
• launch R
• install renv

cd myprojectdirectory
module purge
module load R/4.0.4 # <- or whatver version or R is available on your cluster
R

This will open the R command line interface.

install.packages("renv")

This will return the details that the installation of renv() will occur into a default location:

Installing package into ‘/home/myusername/R/x86_64-pc-linux-gnu-library/4.0’

And ask me to select a CRAN mirror for installation; after choosing the appropriate mirror and pressing return the package will be installed.

Step 3: Install the packages you need, (ideally) prototype your analysis and snapshot the installs using renv

For this demo, I am going to assume that my project depends on the tidyverse family of libraries, and that I want to install them in one go. (In real life, I actually avoid taking this approach, and instead install each of the tidyverse libraries one by one as I use and need them.)

library(renv)
# initialise renv
renv::init()

This will ask me to restart R, after doing which I can start installing packages.

install.packages("tidyverse")
# wait... and wait... for packages to successfully install
library(tidyverse)

I would next prototype my analysis code, installing as many packages as I need using the normal install.packages() syntax - and eventually wrap the analysis up into a script.

After you have installed all of the libraries you need, in the interactive R session, exectute the following command:

# this will tell you whether renv has captured versions 
# of all of the packages you need to install
renv::status()
# if the result of renv::status() was not 
# "The project is already synchronized with the lockfile."
# run the below command to snapshot all currently installed libraries
renv::snapshot()

For this demo, I’m going to assume I want to run the following command that uses the built-in gss_cat dataset from the forcats package, the pipe and the dplyr::filter() command - so nicely tests a few of the tidyverse libraries in one line:

gss_cat %>% head() %>% filter(tvhours < 10)

The output should be:

# A tibble: 3 × 9
   year marital      age race  rincome    partyid     relig     denom    tvhours
  <int> <fct>      <int> <fct> <fct>      <fct>       <fct>     <fct>      <int>
1  2000 Widowed       67 White Not appli… Independent Protesta… No deno…       2
2  2000 Never mar…    39 White Not appli… Ind,near r… Orthodox… Not app…       4
3  2000 Divorced      25 White Not appli… Not str de… None      Not app…       1

(Optional) Step 3a: Look at the renv.lock file to see all of the packages you have installed into the project

The renv.lock file stores details about the R version and all of the packages that are installed in your environment in .json format. So installing the tidyverse packages in one fell swoop like I did above results in quite a long list of libraries (last 5 shown here):

 "vroom": {
      "Package": "vroom",
      "Version": "1.5.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "aac6012f34348b3ca6bf373fe7172b06"
    },
    "withr": {
      "Package": "withr",
      "Version": "2.4.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "ad03909b44677f930fa156d47d7a3aeb"
    },
    "xfun": {
      "Package": "xfun",
      "Version": "0.24",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "88cdb9779a657ad80ad942245fffba31"
    },
    "xml2": {
      "Package": "xml2",
      "Version": "1.3.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "d4d71a75dd3ea9eb5fa28cc21f9585e2"
    },
    "yaml": {
      "Package": "yaml",
      "Version": "2.2.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "2826c5d9efb0a88f657c7a679c7106db"
    }

I sometimes find that renv::status() and renv::snapshot() in step 3 don’t immediately see all of the libraries I have installed in the environment, and - weirdly - I need to restart R and run renv::status() again before it will detect that a bunch of libraries have been installed. So if you’re missing libraries in the renv.lock file you know need to be there - restarting R and re-running renv::status() and renv::snapshot() may be the way to go.

Step 4: Save your script and submit it via PBS

Wrap the above code into myscript.R , with the following contents:

library(tidyverse)
gss_cat %>% head() %>% filter(tvhours < 10)

Then create a shell script (myshellscript.sh) that can be submitted to the PBS queue:

#! /bin/bash
module purge
module load R/4.0.4
cd /to/my/projectpath
Rscript myscript.R

Finally, submit to the queue:

qsub -l walltime=0:10:0 -l select=1:ncpus=1:mem=1gb -N mytestscript myshellscript.sh

Note that I’m only asking for 1 Gb of RAM and 10 minutes of walltime. This is probably a lot less than you’d need for any “real” analysis - but it’s perfectly fine for my script above.

After that executes, I get the following output in mytestscript.o123456, where 123456 is the job number that my PBS scheduler automatically assigned to my script run:

# A tibble: 3 × 9
   year marital      age race  rincome    partyid     relig     denom    tvhours
  <int> <fct>      <int> <fct> <fct>      <fct>       <fct>     <fct>      <int>
1  2000 Widowed       67 White Not appli… Independent Protesta… No deno…       2
2  2000 Never mar…    39 White Not appli… Ind,near r… Orthodox… Not app…       4
3  2000 Divorced      25 White Not appli… Not str de… None      Not app…       1

And the package loading messages in mytestscript.e123456:

Warning message:
renv took longer than expected (66 seconds) to activate the sandbox.

The sandbox can be disabled by setting:

    RENV_CONFIG_SANDBOX_ENABLED = FALSE

within an appropriate start-up .Renviron file. See `?renv::config` for more details.
Warning: program compiled against libxml 209 using older 207
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──
✔ ggplot2 3.3.5     ✔ purrr   0.3.4
✔ tibble  3.1.3     ✔ dplyr   1.0.7
✔ tidyr   1.1.3     ✔ stringr 1.4.0
✔ readr   2.0.0     ✔ forcats 0.5.1
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

This includes a warning that the renv sandbox took a while to load, but this doesn’t seem to affect functionality, so I haven’t disabled it.

That’s it! Hopefully the above helps you use renv() with a PBS-Pro based HPC cluster :blush:, and please do leave a comment if something isn’t clear or doesn’t work as expected.

(Even more) notes

1. This workflow means that your package install information (i.e. the binaries) are stored in the project folder, in the renv folder. So they are not stored on the worker nodes where your job is being executed, but instead in your home or project folder. This is a good thing, as it means you’re not hitting up CRAN to install packages every time you run a job.
2. This workflow also means that while you can rsync your project folder from your local machine to the remove server (or prototype on your local machine), you will need to run renv::restrore() once on the “other” machine prior to being able to execute or deploy the script.

Step 0 (CRITICAL): Close R & RStudio

Make sure you have closed R and RStudio prior to embarking on the below. Updating R versions mid-analysis can have … unintended consequences.

Step 1: Download RSwitch and install it

1. Download RSwitch and install it. Go through all of the hoops of getting it approved by MacOS and able to be run by accepting the risks of running software from an unidentified developer. (Eventually) End up with a nice switch icon in your menubar.

Step 2: Get new R

[As usual] There is more than one way to get a new version of R onto your machine.

Step 3: Apply funky magic

sudo pkgutil --forget org.r-project.R.el-capitan.fw.pkg \
             --forget org.r-project.x86_64.tcltk.x11 \
             --forget org.r-project.x86_64.texinfo \
             --forget org.r-project.R.el-capitan.GUI.pkg

Step 4: Painfully battle the security stuff

Try to open Rstudio. This is probably something that happens only with Option 2 above, but I got a bunch of security warnings from MacOS and had to “Allow Anyway” a ton (twice) before RStudio was able to load R 4.0.

Accept all the things:

Step 5: Switch

To go back to having your existing, “production” version of R active on your machine use RSwitch to select that older version:

I hope this post has been helpful, and let me know in a comment if something isn’t working as described or something’s not clear. (Probably) next post: updating libraries…

My personal teaching setup

• I’m incredibly lucky to have access to several machines, which I use to work through materials on all of the platforms and also while teaching.

• During live teaching, we use two machines: (1) a shared “training” laptop on which we live code or show slides, connected to the projector all day, and (2) our own individual machines, which we use for looking at the notes.

• For teaching online, I use:

  1. A primary machine, with a good camera, as my teaching machine. This has a vanilla setup of RStudio and/or Jupyter, notifications are turned off and all I can see are the things I need to teach. This laptop has only the one, built-in screen.
  2. A secondary machine, a.k.a. my command centre. This machine has my back-channel open, Zoom gallery view (I log in twice to the training) so I can see my learners, Zoom chats and Participants all visible. My notes are printed out on paper, as I have enough screens and windows to try to juggle.
  3. A third machine, which has one (small-ish) screen, and shows me what a learner with only one laptop/desktop screen is seeing. This is not essential, but helps me adjust font sizes and window widths to make sure people can actually live code along with me, without needing to rely on my co-instructors (I still ask students if it’s OK, of course, but this helps me self-adjust faster).
  4. I use an iPad if I need access to whiteboard (see below).

Back-channel communication: whispering in someone’s ear

• You need a quick and easy way to communicate between you and your co-instructors. I recommend a different, secondary chat application for this, for example Microsoft Teams or Slack (or Telegram/Whatsapp - whatever), if you’re using Zoom as your primary teaching tool.

• Ideally, you set up two channels in this: one for urgent messages to the instructor who’s teaching (“YOUR SCREEN IS TINY!”), and another for non-teaching instructors to communicate with each other. This allows your helpers to communicate about challenging software installs (“Do you have any experience updating R libraries on Ubuntu?”), delegate responsibilities (“Can you take over host please? I really need a break for 3 minutes!”) and otherwise manage the class - but all of this is not relevant for the instructor who’s teaching and can actually distract them if they keep getting pinged about it.

Common problems: learner with setup challenges

• First and foremost, send out detailed setup and installation instructions, complete with screenshots, to your learners several days before the course. Remind them to email you with screenshots of any error messages, or - if you’re teaching a large class - get them to post the screenshots into a shared document, and encourage peers to help.

• Second, provide the opportunity for learners who have had issues to join the meeting an hour early and try to get help (yes, this means I’m online at 8 am for a 9 am class).

• My preferred way of helping debug issues is to have learners post a screenshot of the error message into a specific place in the shared document we’re using, and for me to guide them in trying to fix it (via the Zoom chat if at all possible - but note it doesn’t work well with line breaks, so you may need to use the shared doc for pasting code as well).

• Tools like Zoom do provide the functionality to share screens and take over someone’s desktop, BUT I’ve found taking screenshots is faster, doesn’t require you to jump out into a breakout room (so the learner doesn’t miss out on core content), and doesn’t cause bandwidth problems (I’ve tried taking over learners’ desktops with suboptimal bandwidth, and it kicked both of us out of the session).

• Finally, it can be helpful to have a backup cloud platform for teaching as well, as the digital equivalent of spare laptops. We’ve been successful using mybinder.org for python, and have tried but not productionised rstudio.cloud for R. I’d love a suggestion for an online terminal for teaching basic Unix!

Common problems: poor internet connectivity

• If you are having issues with your connection, consider (1) switching to slides/screenshare-only mode, (2) ensuring your device has priority on your home WiFi network, (3) trying to teach via a personal hotspot or (4) swapping to another instructor teaching¹. This is something you can communicate about with your team via the back-channel - and a key reason the back-channel needs to use very little bandwidth!

• For learners, you can try asking them to turn off their video, and possibly also closing other tabs/devices connected to the internet.

Common problems: Tool stack meltdown

• EXPECT your primary tool stack to melt down. If you are using Zoom/Teams/GoToMeeting, expect your platform to go down at least once during the training (it’s great if it doesn’t, but you’re prepared if it does!). Have a plan, with your co-instructors, what you’re going to do: switch to another tool? take an off-schedule break? Something else?

• Explicitly tell your students how you’ll communicate with them to let them know where to go in the event of a breakdown. For us, I prefer using the shared doc, but an email would also work (albeit might be slower, as they’re hopefully not checking email while we’re teaching).

Shared documents

• A shared, student-editable document is essential for successful delivery of training.

• My primary criteria for a shared document include:

  1. It is editable by all students (ideally without logging in),
  2. It doesn’t take up too much screen real estate,
  3. It allows you to post images by copy-pasting them in,
  4. It doesn’t require any cognitive overhead for learners to figure out how to use it.

• The official solution recommended by the University of Sydney is a Microsoft Office Online Word document.

• I have had success using Google Docs. I provide a link to a public doc, so learners don’t have to log in with their Google Account, and ensure that no student data is collected in the doc (i.e. learners can use first names or pseudonyms only on the doc). Most people have used Docs, so there is no overhead in figuring out how to use the tool - we can just dive in and move on.

• There are tools like the Etherpad, hack.md and an open-source, self-hosted version called CodiMD. The latter two support document creation in markdown, which is great because it’s plain text, but also not so great, because it uses more screen real estate than Google docs (especially with the preview pane open side by side with the markdown itself).

• What goes in the Doc? My (non-exhaustive) check-list includes:

  • The title of the course
  • The names of all of the instructors who are part of the teaching team
  • Links to the course materials
  • Links to any data downloads
  • Links to the registration page for the course
  • Links to pre and post workshop surveys
  • Details about the zoom meeting, and every possible way learners can log onto it
  • Links to the setup instructions and tests that setup completed successfully
  • [Links to the mybinder or Rstudio cloud instance, if using as a backup]

Code transfer

• Sometimes you need to share a chunk of code with your learners. While you can paste it into the shared document or Zoom chat, often these tools will do strange things with spaces and quotes.

• Instead, I recommend pasting them into a public GitHub gist, sharing the link with your learners via the chat, and screen sharing how you would access the gist and move it into your R/python session.

Whiteboard

• In an in-person live coding class, I often use the whiteboard when answering student questions or working through problems.

• Digitally, I’m a lot more hesitant to use one, if only because it takes away screen real estate from the other things I’m doing, which are often more important for the learners to see at the time.

• However, I have had success using my iPad and both GoodNotes and Notability when I needed a digital whiteboard, following the instructions here.

I hope these two posts are helpful for you as you prepare to jump into teaching online! Please leave a comment below if you found something useful, unclear or would like to add something else I missed!

Preliminaries

A learners arrives in my training room. They:

1. Sign in on a sign-in sheet
   • The “Host” can take a screenshot of all of the participants, in gallery view, and/or a list of the names of Participants, approximately 30 minutes after the session begins¹.
2. Make a name tag for themselves
   • Rename themselves in Zoom to their preferred name, faculty or school (depending on teaching cohort) and preferred pronoun, so I’d become, for example “Darya (SIH, she/her)”.
   • This can sometimes make it tricky to match names to emails, which is what we use as our unique key for registration. That’s why I try to check people in during the day, and clarify with a private chat message asking what someone’s email is if I’m unable to unambiguously match them to their email.
3. Find a seat
   • There are several ways to allocate learners to a breakout room. First, you can use the faculty details in the naming convention above to group people by discipline, school or faculty.
     • This can present challenges downstream if a particular aspect of your course is better aligned with one domain’s background knowledge than another’s. For example, I use a problem with cancer patients and controls, which for me (and most people with a biomedical background) obviously indicates that the latter group are healthy, possibly/ideally age/gender matched people without cancer. When I grouped people by Faculty, the life and health sciences team powered through the challenge, whereas the engineers ended up spending a lot of time wrapping their heads around what a control was in this context.
     • So in the future I think I’ll stick to what happens in real life, and mix people by first letter of name or order of popping into the course, similar to how learners tend to randomly sit together in a 3D classroom.

Workshop start

1. Chat to their neighbours
   • In a 3D workshop this is part of the preliminaries, but you as the instructor have to actively normalise this online.
   • It’s one of the hardest things to replicate: the productive interactions and relationship-building/networking among learners that happens when you work together on a bunch of tasks for 2 days straight.
   • A somewhat useful replacement that gets learners talking is an icebreaker task (ideas here), which can either be posted into the shared document OR carried out orally in small groups in breakout rooms. I find it helpful, in addition to asking about something relatable (“What was the most interesting thing you learned working from home last week?”) to ask about learners’ setups and screens (“What does your learning setup look like today?”). That way, I can suggest tweaks, such as logging in from a mobile device as a modification of a two screen setup. Another option is to ask the latter as a poll.
2. Code of Conduct
   • All workshops need a Code of Conduct, which establishes the norms of behaviour you expect to support everyone’s learning. Online, I add information about expectations for private messaging and screen sharing, as well as recording (nope!).
   • I also ask people to let me know (via a private chat message, at any point in the workshop) if they’re uncomfortable with me explicitly calling on them, as one of the biggest challenges of teaching online (and off) is extroverts dominating the conversation. To prevent this, I keep a tally of who I’ve called on, but I also want to make sure I don’t make someone who’s wrangling a child or responding to 50 emails about a grant deadline to feel uncomfortable or pressured.
3. Schedule
   • Just like a “normal” class, I use a slide to walk learners through the schedule, explain how the course content is linked and when breaks will happen.
4. Sticky notes
   • In a Carpentries workshop, we have a very special way of using sticky notes to gauge learner state and assess who needs help - without them having to hold their hand up for hours. Zoom is great because it’s a digital platform that allows us to replicate the same - BUT it’s important to recognise that using this feature in Zoom costs screen real estate (and cognitive load!), so may need to be relied on sparingly during some portions of the class, especially the live coding ones!
   • To replicate stickies in Zoom you can use the Non-verbal Feedback functionality:

• While there are a LOT of options, we tended to use only the “Hand up”, “Yes” and “No” options, as (1) you can only use one of these statuses at a time, and (2) we wanted to know whether students were good (“Yes”), needed a helper to reach out (“No”), or wanted to ask a question (“Hand”). The Hand functionality is also quite helpful for the instructor, as it automatically places the person with the hand up at the top left in Gallery view mode.

• I’ll also mention Reactions here, which unlike Non-verbal feedback are a non-persistent way learners can tell you that all is good: they’re displayed on screen for 5 seconds, and are overlaid over a learners picture in Gallery view. You can only show “thumbs up” or “clap”, so they’re not very helpful for getting negative feedback. Our learners used them intuitively, without us providing explicit instructions, to let us know that they didn’t have any questions during the frequent pauses we made to ask “Does anyone have any questions?/Are there any questions?/What’s unclear about what we just did?”.

Slide-centric sessions

In some of our intermediate sessions, we then dive into slides, with a small slide deck (20 minutes max) that provides some theory and a few conceptual challenge tasks. With these sessions, I

1. Start by showing learners how best to set up their screens for learning, including if they want to take notes or ask questions via the chat.

2. Make sure I explicitly provide a link to where they can download the slides, so they can annotate a copy as we go, on their own machine.

3. Set up the norms of asking questions: use the chat (or the shared doc, but ideally not both), and have my hosting co-instructor interrupt me during short pauses if I miss a question that a learner asked that would be best answered during that particular slide/mini-session vs at the end of the presentation.

Live coding sessions

These form the core of our workshops, with sessions of coding along interspersed with short, formative assessment tasks, including multiple choice questions, faded examples and more complex, unscaffolded challenge tasks. Live coding is the most challenging aspect to “port” to digital. The key things that help make these work (somewhat):

1. Make sure you use/share/project only 1/2 of your screen, and use an appropriate coding “tool” that encompasses everything in that 1/2 screen. This means jupyter notebooks and the terminal are in, but RStudio/a .R script, in it’s native implementation, is out - they just take up too much screen real estate! The best work-around I’ve found so far is to use an Rmarkdown document, with inline output for figures. This is what you get the settings to look like:

And this is what the learners’ screens looked like:

2. During the training, start by showing learners how best to set up their screens for learning.

3. If you do decide (or your learners just go) to move to more of a watch me narrate and code, I highly recommend having the challenge tasks commented out in a single code file you distribute throughout the day - so a .py or a .R or (possibly) even a .sh script, although I’m not sure about the latter, as most intro Unix courses don’t necessarily work a lot with shell scripts. Learners can then uncomment the challenges as they go, and work through them at the appropriate times.

Breakout rooms and challenge tasks

Peer learning has been shown in numerous studies to be one of the most effective ways of getting students to learn. In our in-person training sessions, we encourage learners to sit in a group and, for every challenge task, to start working on it themselves, then share their solution or any encountered problems with their neighbour, then their table and - finally - when the group has solves it - with the class. To replicate this in an online environment we used Zoom breakout rooms, with a few caveats:

1. The host - who is NOT the person teaching - needs to set up the maximum number of breakout rooms at the beginning of the training session. Note that ONLY the host, and NOT the co-hosts, can set up breakout rooms and allocate people to them!

2. We send groups of 3-5 learners, plus one co-instructor, into each breakout room. Unlike an in-person event, in a digital skills training we found that it took learners a few minutes to adjust to the breakout context and figure out what they were meant to do and where - so the instructor in each room helped guide this process.

3. Right before a challenge task we’d paste the text of the task into the chat AND add it into our shared document. This meant that all learners had easy access to the activity. Note that the chat in a breakout room can only happen between people IN that room, so if learners are all in a room and want to message the host, who is NOT in their room, they can’t do this! So if you’ve got more rooms than co-instructors it can be helpful to have them post requests for help into the shared document, after which you can jump into the room; they can also ask for help via the app:

• The red/green “No”/“Yes” participant statuses are not visible to the host from outside a breakout room, so using them to replace stickies doesn’t work in this context.
• Note that users joined via the web client, Chromebooks/Chrome OS or Zoom Rooms are unable to join Breakout Rooms! Zoom suggests the main room as an alternative session for these users, but we’d recommend explicitly requesting learners to use an installed version of the app/an individual machine instead of the web client or room.

To mimic the green sticky system of in-person teaching, the instructors can use the back-channel to communicate about where their group is in the task OR - if you’ve got rooms without instructors - I’d recommend learners use the shared document to update you when they’ve completed each of the components of the challenge task.

• When everyone’s back together ask for a volunteer early on, then, later in the day, call on people to prevent extroverts from dominating the reporting.

Casual chats with instructors and other learners

This is impossible to fully replicate online, BUT as a workaround: plan for at least one of your instructors to be in the meeting during the morning and afternoon “tea” breaks (Labelled “HERE” below). My “standard” workshop schedule looks like this:

• 09:00 am - 10:30 am - Training
• 10:30 am - 11:00 am - “Morning tea” <- HERE
• 11:00 am - 12:30 pm - Training
• 12:30 pm - 1:30 pm - Lunch
• 1:30 pm - 3:00 pm - Training
• 3:00 pm - 3:30 pm - “Afternoon tea” <- HERE
• 3:30 pm - 5:00 pm - Training

What does that instructor do? Small talk at the least, and - usually - after getting to know the learners a bit, they WILL ask you questions about their research, your work or other “stuff” related to the course. It also allows you to reassure them you believe they can learn the content AND that even if their setup is “weird” (according to them) they can still succeed in the course and use the tools and techniques later on.

Whew! That ended up being a long post! In the next post of this series, I’ll explore some common problems and how to deal with them, as well as discuss how to replicate instructor-instructor communication. If you’re a learner, this post is for you!

1. Follow your instructor’s instructions

2. Setup your setup

3. Test your setup!

1. Most online meeting tools will allow you to have a meeting with yourself, or to join a test meeting. It’s great to try this the night before the workshop, at the latest, as you set up your workspace. Here is a link for how to do this in zoom.

2. For the programming tools, try opening them and seeing what start-up messages are printed. If there’s a warning or something doesn’t look right or hangs it might be helpful to reach out to your instructor.

4. Set up for notes

Think about how you plan to take notes. In a face to face class, it’s usually easy to have a note-taking app open on your computer, but with the extra windows that you’ll have open with digital teaching, it might be a bit much to try to switch windows all the time. Consider whether you’d prefer to take paper notes, or use a tablet or other secondary device for note-taking instead.

5. Make sure to note the norms

At the beginning of class your instructor will most likely introduce the norms of behaviour:

• How do you ask questions?
• Raise your hand to volunteer to answer?
• Ask for and get help?
• Communicate with other learners?

Make sure to pay attention to this bit, since - just like an airplane - different instructors will use the same tools differently sometimes, and you don’t want to use the “Hand up” feature for 2 hours waiting for someone to help you if you were supposed to paste into the chat instead.

I hope the above check-list is helpful for you as you prepare to jump into learning online! Please leave a comment below if I’ve missed something or something is not clear.

Personal, highly non-exhaustive notes on section I: Explore

library(tidyverse)
theme_set(theme_minimal())

Steps of the data pipeline:

• Import: take data stored in a file, database, or web API, and load it into a data frame in R.

Small data vs big data

• Small/medium data: hundreds of megabytes of data, and with a little care up to 1-2 Gb of data.
• If you’re routinely working with larger data (10-100 Gb, say), you should learn more about data.table.

Is big data really big? Two ways of thinking small about big data

Key definitions for tidy data (reference)

• A variable is a quantity, quality, or property that you can measure.
• A value is the state of a variable when you measure it. The value of a variable may change from measurement to measurement.
• An observation is a set of measurements made under similar conditions (you usually make all of the measurements in an observation at the same time and on the same object). An observation will contain several values, each associated with a different variable. I’ll sometimes refer to an observation as a data point.
• Tabular data is a set of values, each associated with a variable and an observation. Tabular data is tidy if each value is placed in its own “cell”, each variable in its own column, and each observation in its own row.

Variable types

• A variable is continuous if it can take any of an infinite set of ordered values.
• A variable is categorical if it can only take one of a small set of values. In R, categorical variables are usually saved as factors or character vectors.

EDA key definitions

• Variation is the tendency of the values of a variable to change from measurement to measurement.
• Covariation is the tendency for the values of two or more variables to vary together in a related way. T
• he residuals give us a view of the price of the diamond, once the effect of carat has been removed.

ggplot(data = diamonds) + 
  geom_bar(aes(x = cut, y = ..count.. / sum(..count..), fill = color))

# not really new, but I'm sure I'll forget position = "fill"
ggplot(data = diamonds) + 
  geom_bar(mapping = aes(x = cut, fill = clarity), position = "fill")

# pie chart from bar
ggplot(data = diamonds) + 
  geom_bar(mapping = aes(x = 1, fill = clarity)) + coord_polar(theta = "y")

select(flights, time_hour, air_time, everything())

# A tibble: 336,776 x 19
   time_hour           air_time  year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay carrier
   <dttm>                 <dbl> <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>     <dbl> <chr>  
 1 2013-01-01 05:00:00      227  2013     1     1      517            515         2      830            819        11 UA     
 2 2013-01-01 05:00:00      227  2013     1     1      533            529         4      850            830        20 UA     
 3 2013-01-01 05:00:00      160  2013     1     1      542            540         2      923            850        33 AA     
 4 2013-01-01 05:00:00      183  2013     1     1      544            545        -1     1004           1022       -18 B6     
 5 2013-01-01 06:00:00      116  2013     1     1      554            600        -6      812            837       -25 DL     
 6 2013-01-01 05:00:00      150  2013     1     1      554            558        -4      740            728        12 UA     
 7 2013-01-01 06:00:00      158  2013     1     1      555            600        -5      913            854        19 B6     
 8 2013-01-01 06:00:00       53  2013     1     1      557            600        -3      709            723       -14 EV     
 9 2013-01-01 06:00:00      140  2013     1     1      557            600        -3      838            846        -8 B6     
10 2013-01-01 06:00:00      138  2013     1     1      558            600        -2      753            745         8 AA     
# … with 336,766 more rows, and 7 more variables: flight <int>, tailnum <chr>, origin <chr>, dest <chr>, distance <dbl>,
#   hour <dbl>, minute <dbl>

161024_AWS workshop

• SciTeam ~10 ppl

Moving data to AWS

Research on AWS

Lab: Alces Flight & CfnCluster

Apart from Alces flight, what other launch a cluster configurations are there?

In order to relaunch a snapshot you have you can use CfnCluster.

spark R machine learning

jupiter gnu (ask him via email)

Video

Slides

Abstract

In 2001 Bill Clinton unveiled “the most important, most wondrous map ever produced by humankind” - the human genome. This monumental endeavour cost $3 billion, and took hundreds of scientists from all over the world 13 years. Today, a single person can generate such a map in ~2 days for $1000. This dramatic drop in cost means that we now have data for hundreds of thousands of people - and other species - from all corners of the globe, and cohorts are available for every major disease under the sun. Petabytes of new data are also being generated every week.

Most of this data is publicly available, so anyone with an internet connection can try in silico biology from the comfort of their own home. In my talk, I’ll walk through what this data looks like, and how it’s analysed - with a special focus on where python fits into the workflow (;tldr the most interesting parts!). I will also highlight some common pitfalls software engineers and developers face when getting into this space.

Finally, I’ll showcase several other facets of bioinformatics that sorely need contributions from good coders. Genomics is rapidly entering the world of health care in both the public and private hospital sectors, and in direct-to-consumer genetic testing. Understanding this data, the challenges and limitations of its analytics will help us all make better-informed health and medical decisions, affecting our quality of life and those we love.

Note:

If you are trying to follow up on my talk and carry out an analysis of some data (or are looking for some data, or would like to know where to find the data from paper X), please leave a comment below, and I’ll do my best to answer your question and help! I don’t monitor the comments on youtube or slideshare, but I do get a ping if you leave a comment here.