Open Source Summit + Embedded Linux Conference + OSPOCon 2021

NorthSec started as a yearly on-site Capture The Flag security event in Montreal. Over the years, it has grown to close to a thousand attendees representing as many as 80 teams, while also developing a large conference and a selection of professional trainings on the side. This 3 day CTF is somewhat unique for providing a completely distinct infrastructure for each participating team. In theory making it impossible for one team to affect any of the others. All while providing sometimes hundreds of different virtual servers and services for a team to attack. Providing all of that, to every team, in a reliable, fair and safe way has at times been a bit of a struggle. This talk will be going over the past few editions, looking both at the infrastructure used to provide the CTF and how it evolved as well as the various bugs and configuration issues that were encountered. This covers all kind of interesting problems, Linux kernel bugs, container namespacing issues, resource limits working in odd ways, information leakage and network and storage tuning. Those lessons are in no way specific to operating a CTF and should be of interest for anyone running a very large set of containers in production, especially when untrusted and/or malicious users are involved!