About Me Blog
A tutorial on tidy cross-validation with R Analyzing NetHack data, part 1: What kills the players Analyzing NetHack data, part 2: What players kill the most Building a shiny app to explore historical newspapers: a step-by-step guide Classification of historical newspapers content: a tutorial combining R, bash and Vowpal Wabbit, part 1 Classification of historical newspapers content: a tutorial combining R, bash and Vowpal Wabbit, part 2 Curly-Curly, the successor of Bang-Bang Dealing with heteroskedasticity; regression with robust standard errors using R Easy time-series prediction with R: a tutorial with air traffic data from Lux Airport Exporting editable plots from R to Powerpoint: making ggplot2 purrr with officer Fast food, causality and R packages, part 1 Fast food, causality and R packages, part 2 For posterity: install {xml2} on GNU/Linux distros Forecasting my weight with R From webscraping data to releasing it as an R package to share with the world: a full tutorial with data from NetHack Get text from pdfs or images using OCR: a tutorial with {tesseract} and {magick} Getting data from pdfs using the pdftools package Getting the data from the Luxembourguish elections out of Excel Going from a human readable Excel file to a machine-readable csv with {tidyxl} Historical newspaper scraping with {tesseract} and R How Luxembourguish residents spend their time: a small {flexdashboard} demo using the Time use survey data Imputing missing values in parallel using {furrr} Intermittent demand, Croston and Die Hard Looking into 19th century ads from a Luxembourguish newspaper with R Making sense of the METS and ALTO XML standards Manipulate dates easily with {lubridate} Manipulating strings with the {stringr} package Maps with pie charts on top of each administrative division: an example with Luxembourg's elections data Missing data imputation and instrumental variables regression: the tidy approach Modern R with the tidyverse is available on Leanpub Objects types and some useful R functions for beginners Pivoting data frames just got easier thanks to `pivot_wide()` and `pivot_long()` R or Python? Why not both? Using Anaconda Python within R with {reticulate} Searching for the optimal hyper-parameters of an ARIMA model in parallel: the tidy gridsearch approach Some fun with {gganimate} Split-apply-combine for Maximum Likelihood Estimation of a linear model Statistical matching, or when one single data source is not enough The best way to visit Luxembourguish castles is doing data science + combinatorial optimization The never-ending editor war (?) The year of the GNU+Linux desktop is upon us: using user ratings of Steam Play compatibility to play around with regex and the tidyverse Using Data Science to read 10 years of Luxembourguish newspapers from the 19th century Using a genetic algorithm for the hyperparameter optimization of a SARIMA model Using cosine similarity to find matching documents: a tutorial using Seneca's letters to his friend Lucilius Using linear models with binary dependent variables, a simulation study Using the tidyverse for more than data manipulation: estimating pi with Monte Carlo methods What hyper-parameters are, and what to do with them; an illustration with ridge regression {disk.frame} is epic {pmice}, an experimental package for missing data imputation in parallel using {mice} and {furrr} Building formulae Functional peace of mind Get basic summary statistics for all the variables in a data frame Getting {sparklyr}, {h2o}, {rsparkling} to work together and some fun with bash Importing 30GB of data into R with sparklyr Introducing brotools It's lists all the way down It's lists all the way down, part 2: We need to go deeper Keep trying that api call with purrr::possibly() Lesser known dplyr 0.7* tricks Lesser known dplyr tricks Lesser known purrr tricks Make ggplot2 purrr Mapping a list of functions to a list of datasets with a list of columns as arguments Predicting job search by training a random forest on an unbalanced dataset Teaching the tidyverse to beginners Why I find tidyeval useful tidyr::spread() and dplyr::rename_at() in action Easy peasy STATA-like marginal effects with R Functional programming and unit testing for data munging with R available on Leanpub How to use jailbreakr My free book has a cover! Work on lists of datasets instead of individual datasets by using functional programming Method of Simulated Moments with R New website! Nonlinear Gmm with R - Example with a logistic regression Simulated Maximum Likelihood with R Bootstrapping standard errors for difference-in-differences estimation with R Careful with tryCatch Data frame columns as arguments to dplyr functions Export R output to a file I've started writing a 'book': Functional programming and unit testing for data munging with R Introduction to programming econometrics with R Merge a list of datasets together Object Oriented Programming with R: An example with a Cournot duopoly R, R with Atlas, R with OpenBLAS and Revolution R Open: which is fastest? Read a lot of datasets at once with R Unit testing with R Update to Introduction to programming econometrics with R Using R as a Computer Algebra System with Ryacas

The Raspberry Pi 4B as a shiny server

This blog post will not have any code, but will document how I went from hosting apps on shinyapps.io to hosting shiny apps on my own server, which is a Raspberry Pi 4B with 8 gigs of ram. First of all, why hosting apps on a Raspberry Pi? And why not continue on shinyapps.io? Or why not get one of hose nifty droplets on DigitalOcean? Well for two reasons; one is that I wanted to have full control of the server, and learn some basic web dev/web engineering skills that I lacked. These services simplify the process of deploying and hosting a lot, which of course is a good thing if your only goal is to deploy apps. But I wanted to learn how to do it myself from scratch for some time. True, with a DigitalOcean droplet, I could have learned quite a lot about the whole process as well, but there’s a second problem; the minimum amount of processing power that the droplet needed to run shiny came at 10€ a month. Not a fortune, but already quite expensive for me, since I just wanted to learn some stuff on my free time. Which is why I got a Raspberry Pi 4B with 8 gigs of ram. It’s less than 100€, and now that I have it, I can do whatever I want whenever I want to. If I don’t touch it for several months, no harm done. And if I get tired of it, I’ll make a retro console out of it and play some old schools games. It’s a win-win situation if you ask me.

So first, you should get a Raspberry Pi. Those are quite easy to find online, and there’s many tutorials available on how to install Ubuntu (or any other Linux distro) on it, so I won’t bother with that. I also won’t explain to you how to ssh into your Raspberry Pi, again, there’s many tutorials online. More importantly, is how to get Shiny on it? There’s two solutions; you either install it from source, or you use Docker. I chose to use Docker, but maybe not in the way you’d expect; there’s a lot of talk online about dockerizing apps, complete with all their dependencies and environment. The advantage is that you’re guaranteed that deployment with be very smooth. But the big disadvantage is that these dockerized apps are huge, around 1GB, or sometimes more. It is true that disk space is quite cheap nowadays, but still… so I prefer to run a Shiny server from Docker, and then run the apps out of this server. My apps are thus very small, and it’s only the Shiny server that is huge. I found a Github repository from user havlev that explains how to do it here. I have followed this guide, and created my own docker container, which is based on havlev’s one. I added some dependencies (to the base Debian distro included, as well as some more R packages).

If you’re in a hurry, and want to use my Docker image, you can simply type the following on your Raspberry pi:

mkdir shiny-server
cd shiny-server
mkdir apps
mkdir conf
mkdir logs
docker run -d -p 3838:3838 -v shiny-apps:/srv/shiny-server/ -v shiny-logs:/var/log/ -v shiny-conf:/etc/shiny-server/ --name rpi-shiny-server brodriguesco/shiny_1_5:firstcommit

The first 5 commands will create some folders that we’ll need later on, while the last one will pull my Docker container, which is based on havlev’s one, launch the server and it’ll start listening to port 3838.

I made an app (another blog post, focusing on this app, will follow soon), hosted on my Raspberry Pi that you can find here. I’ll also give you some pointers on how you can achieve that.

But let’s start from the beginning.

Adding dependencies to a Docker container

So let’s suppose that you’re me a few weeks ago, and that you find and follow havlev’s guide here. Getting the docker running is quite easy, you just need to set up Docker, and then find the line in the tutorial that starts with docker run…. You’ll get Shiny running with its hello world app. Now, how can you add more packages, either to the base Debian image, or R packages? For this part, I followed this guide. The idea is to “log in” to the console of the base Debian distro that is running from the container. First, find the ID of the container by typing the following command in the terminal:

docker ps

You should see something like this:

ubuntu@ubuntu:~$ docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED              STATUS              PORTS                    NAMES
69420blazeit        brodriguesco/shiny_1_5:firstcommit   "/etc/shiny-server/i…"   About a minute ago   Up About a minute   0.0.0.0:3838->3838/tcp   rpi-shiny-server

now with the ID in hand, you can start any command line program from your Docker container, for instance bash:

docker exec -it 69420blazeit bash

You’ll be “logged in” as root:

root@69420blazeit:/# 

and from there, you can install Debian packages. The following two packages are necessary to install many R packages from source, so I recommend you install them:

root@69420blazeit:/# apt-get install libssl-dev libxml2-dev

Once these Debian packages are installed, you can start R by simply typing R in the same console, and install whatever packages your Shiny apps will need. In my case, I installed {golem} and several others, but this will be the subject of another blog post. We’re almost done with that; we now need to save the changes because if you restart the container, you’ll lose all these changes. To save these changes, let’s run the following command, but in a new terminal on your Raspberry Pi (on the “local” Ubuntu, not the Debian running in the container):

ubuntu@ubuntu:~$ docker commit -m "added some dependencies" 69420blazeit shiny_with_deps

So now you could run this container with the command from above, by replacing the adequate parts:

docker run -d -p 3838:3838 -v shiny-apps:/srv/shiny-server/ -v shiny-logs:/var/log/ -v shiny-conf:/etc/shiny-server/ --name rpi-shiny-server shiny_with_depsshiny_with_deps

Using your Shiny server

Ok so now that the server is running, you can you deploy apps on it? Remember the folders that we created at the beginning of the blog post (or that you created if you followed havlev’s guide)? This is where you’ll drop your apps, the usual way. You create a folder there, and simply put the ui.R and server.R files in here, and that it. These folders can be found in your $HOME directory, and they are accessible to your docker container as well. Once you dropped one or two apps, you’ll be able to access them on a link similar as this one:

http://192.168.178.55:3838/hello/

where 192.168.178.55 is the local IP address of the Raspberry Pi, 3838 is the port the server is listening to, and /hello/ is the name of the subfolder contained in the ~/shiny-server/apps folder that you created before. What is left doing is making your Raspberry Pi a proper server that can be accessed from the internet. For this, you’ll need to ask your ISP for a dynamic IP address. Generally, you’ll have to pay some money for it; in my case, I’m paying 2€ a month. This address can then be used to access your Raspberry Pi from the internet. The problem, is that being dynamic, the address changes every time you restart your server. To solve this issue, you can use a free dynamic DNS. I use duckdns. This will allow you to have domain that you can share with the world. What’s nice is that if you follow their guide the redirection to the dynamic IP address will happen seamlessly every time it changes, so no need to think about it and do it manually.

Finally, you’ll also have to open up port 3838 on your router. The procedure changes from router to router, but you should be able to find the instructions for your router quite easily. If not, you should also be able to get help from your ISP.

The end result is that you’ll have your own Shiny server running off a Raspberry Pi, and accessible over the internet! You’ll be able to deploy as many apps as you want, but of course, don’t forget that you’re running all this on a Raspberry Pi. While these machines have become quite powerful over the years, they won’t be powerful enough if you’re running some heavy duty apps with hundreds of concurrent users.

In my next blog post, I’ll walk you through the development of a Shiny app using the {golem} package, which you can find here.

Hope you enjoyed! If you found this blog post useful, you might want to follow me on twitter for blog post updates and buy me an espresso or paypal.me, or buy my ebook on Leanpub.

Buy me an EspressoBuy me an Espresso