Separation between code and data (specifically configuration data) is a critical concept in any development environment. Having worked a lot with the nodejs stack I found the way configuration is handled in embedded Arduino/PlatformIO projects surprisingly messy. I decided to come up with a convention to simplify this and I will describe it below.
tl;dr, add an unversioned, extra_configs file called private_config.ini to your platformio.ini. Put any user specific parameters in it as macros. Put any user-specific settings there using PlatformIO syntax. Add a config.h(pp) file to your project to handle documentation, defaults for optional parameters and errors for mandatory parameters. Optionally include a versioned private_config.template.ini in your project to help users start.
Introduction
Let’s see how my two favorite (as of now 🙂 ) stacks compare in terms of configuration. My first favorite is nodejs + Visual Studio Code. My second favorite is Arduino + PlatformIO + Visual Studio Code. Note that the hardware layer doesn’t matter for this comparison, though obviously node runs on computers with an operating system (Linux, Windows, etc.) where PlatformIO is meant for embedded devices where you flash the entire firmware at once and it runs on bare metal.
Issues resulting from separation between code and data tend to only manifest when a project has several developers and several deployments. A project that only has a single user will work fine even if you hard code all the configuration in the source files. So for the purpose of this article we will consider projects that are going to be hosted publicly, which others will clone and run on their own, and which others will make pull requests for (i.e. github).
The term “configuration” means several things. There is data that describes the project such as dependencies, name, architecture, build process, etc.; those are not code but they don’t typically change at different deployments of the project, then there is data that changes per “user” of the project such as ports, pins, urls, keys, passwords, etc.
In a nodejs environment, code is compiled/interpreted at runtime, so instead of compile time configuration we have run time configuration. The convention for user data is to set the parameters as environment variables either in memory or in a text file (a dotenv .env file). If a file is used, then it is typically not version-controlled and is “git ignored”. The parameters are then used as “process.env.PARAM” in the code. The convention for project data is to use npm and a well-defined package.json file. This works great because all you need to do is clone a repository, npm install it, define any custom parameters you wish to set in your .env file or set environment variables and run the project. Your local additions are not changes to version controlled files. If you ever pull a new version or want to push a change then your user specific files are separate.
For comparison, an Arduino framework in an Arduino IDE (pre PlatformIO) is total chaos. There is no structured way to save project data in a repository of the project. Most often such Arduino projects are just a single ino file. When downloading it you have to manually configure the IDE for the right platform, device, speed, com port and other parameters. When you come back to the project at a later time you are likely to reconfigure it all again manually, unless you haven’t done anything else in the time-being. You also have to understand the dependencies and manually download and install them yourself. User-specific data is usually hard-coded or specified as define directives at the top of the main file. When you set your own pins and other parameters it becomes a nightmare to keep your code in sync with the original repository.
PlatformIO brought a needed break when it standardized a file format to define project data. Now use of parameters such as architecture, platform, build configuration and dependencies is all automated. You just clone and let PlatformIO build the program for you. Want to close a project, work on another one and then get back to the first? No problem, the settings are all saved per project (like in other up-to-date IDEs). The majority of large, embedded, open-source projects that I have dealt with recently use this convention. What is still a mess though is user-specific configuration.
Most embedded project have compile time parameters. Even if a project has runtime configuration mechanism, like a web or serial interface, it is likely to still have compile time settings to set the physical aspects of the device running the firmware such as pin ids. There is no standard file system that all various embedded devices support and no easy way to write data to an EEPROM or to a data segment in flash that is cross platform. As a result, the easiest method to add parameters to a project is compiling them into the firmware using macros. The macros can be added with define directives (in the source code) or using build flags of the compiler (in build or platformio.ini files).
Let’s take a look at a few examples of PlatformIO projects and how they implement user-specific settings. There is no specific reason for showing these particular projects, it is just what I was dealing with when I was working with LoRa recently. First example is Paxcounter. If you want to deploy pax to a device, you have to clone the repository and then edit platformio.ini, two other files and then make your own copies of two additional files from “sample” files. In the end, you end up with quite a few changes in versioned file, configuration is not central, nor is there one single way to specify the configuration. The second example is ESP-1ch-Gateway. If you want to deploy the gateway to a device, you have to clone the repository and then edit two versioned header files. Once again, you end up with a few changes in versioned file.
Private config
When I wrote the firmware for my last project, adding a Wi-Fi interface to a non-connected appliance, I decided to find an elegant solution for storing user-specific configuration. Nothing that I will describe below is revolutionary, it is made out of existing features of PlatformIO, git and the C compiler. It is the intention of using this as a standard convention that is the novelty here. If more projects use this or similar processes, then the effort to grab and set up a random project will become smaller. I would love to get feedback and suggestions, so feel welcome to comment. I will use the project rinnai-wifi as an example.
We start by creating a file called private_config.ini and adding it to .gitignore. This file will have the same format as a platformio.ini file. Next, we include the new file in our platformio.ini file using a statement called extra_configs.
[platformio]
extra_configs = private_config.iniThis includes everything we put in the private file in the main file. The included file takes precedence if commands appear in both files, so the included file can override values from the main file. You can also include more than one external config file to split your configuration between several files if you prefer, but for the sake of simplicity I will use just one “container” in this example.
Next, for user-specific project parameters such as upload_port, monitor_port, etc. we just add matching entries in the private config file.
[env:usb]
upload_port = COM7
monitor_port = COM7For user-specific parameters that we want to pass to the source code, we will declare macros, I decided to use src_build_flags in the private file and build_flags in the main file to separate them and prevent one overriding the other. It is also possible to manually merge individual settings in the main file if that is required. PlatformIO allows for any statement to be used in the ini files and will allow you to pass and include custom statements with ${env.param} syntax. This allows for more complex constructs but PlantformIO will complain about unknown entries. Perhaps there is a way to disable that warning. LMK if you know how. Ok, back to macros. Define any parameters you want using this syntax:
src_build_flags =
'-D STRING_PARAM="pass"'
'-D STRING_INC_PARAM="${env.param}"'
-D NUMERIC_PARAM1=-1
-D NUMERIC_PARAM2=4Here we will only specify the overrides. Many parameters are likely to have useful defaults. It is preferred to create a versioned file, called private_config.template.ini that will hold an example of what the user might need to create. This will help users get on board faster. If you find yourself needing to troubleshoot the hierarchy of ini files, you can use a useful platformio project config command to print the unified config.
The entire list of macros will be presented and documented in a file called config.h(pp). Add such a file and include it in your main.c(pp). You might want to include it in other modules, too. Personally, I prefer to include it only in main.c(pp) and pass the relevant parameters around to sub-modules through constructors, not macros. The config file should look something like this:
#pragma once
// check (required) parameters passed from the ini
// create your own private_config.ini with the data. See private_config.template.ini
#ifndef SERIAL_BAUD
#error Need to pass SERIAL_BAUD
#endif
// Name of the device
#ifndef DEVICE_NAME
#define DEVICE_NAME "Default device name"
#endifThere is a general description after which we have two types of entries – mandatory parameters or optional parameters. Mandatory parameters will throw a pre-compiler error if they are not defined. Note that they can be defined in the private config file or another relevant config file for the check to succeed. Optional parameters have their default value set if they are not defined. Also note that each entry is also documented, this way all the parameters, their documentation and their defaults are all defined in a single place using a single convention.
Summary
I find the process useful and not too complex. Hopefully you find the proposed method useful, too. Please post a comment if you have improvements or questions.