Folders and files

*Notes

• the specific libpq5 version shown here is required for libpq-dev at time of writing
• poetry install (next step) requires python 3.9, but there are multiple options for resolving this, see here

Pro-tip: we have had at least one large code refactor. You can ignore specific large commits when blaming by setting the Git config's ignoreRevsFile to .git-blame-ignore-revs:

1. If using VSCode, configure workspace folders

   • Add /experimenter/ and /experimenter/experimenter folders to your workspace (File -> Add Folder to Workspace -> path/to/experimenter/experimenter)

   • From the /experimenter/experimenter folder, run yarn install

     • Make sure you are using the correct version of node

       node -v

     • Troubleshooting:

       • Changing node version
       • Clear npm cache: npm cache clean --force

On certain pages an API endpoint is called to receive experiment analysis data from Jetstream to display visualization tables. To see experiment visualization data, you must provide GCP credentials.

0. Prequisites

   • Install GCP CLI
     • Follow the instructions here
     • Project: moz-fx-data-experiments
   • Verify/request project permissions
     • Check if you already have access to the storage bucket here
     • If needed, ask in #nimbus-dev for a project admin to grant storage.objects.list permissions on the moz-fx-data-experiments project

1. Authorize CLI with your account

   • make auth_gcloud
     • this will save your credentials locally to a well-known location for use by any library that requests ADC
     • Note: if this returns Error saving Application Default Credentials: Unable to write file [...]: [Errno 21] Is a directory: ..., delete the directory and try again (rm -rf ~/.config/gcloud)

2. The next time you rebuild the docker-compose environment, your credentials will be loaded as a volume

   • Note that this will require the existing volume to be removed (hint: run make refresh)

3. (optional) Verify access

   • make refresh
   • make bash
   • ./manage.py shell
     • from django.core.files.storage import default_storage
default_storage.listdir('/')

     • Confirm this second command prints a list instead of an error

We support user uploads of media (e.g. screenshots) for some features.

In local development, the default is to store these files in /experimenter/media using Django's FileSystemStorage class and the MEDIA_ROOT and MEDIA_URL settings.

In production, a GCP bucket and credentials are required.

The bucket name is configured with the UPLOADS_GS_BUCKET_NAME setting. For example:

For local testing of a production-like environment, The credentials should be configured as described in the previous section on Google Credentials for Jetstream.

In the real production deployment, credentials are configured via workload identity in Google Kubernetes Engine.

Experimenter uses docker for all development, testing, and deployment.

Build the application container by executing the build script

Build the supporting services (nginx, postgresql) defined in the compose file

Create dummy SSL certs to use the dev server over a locally secure connection. This helps test client behaviour with a secure connection. This task is run automatically when needed.

Stop and delete all docker containers. WARNING: this will remove your database and all data. Use this to reset your dev environment.

Apply all django migrations to the database. This must be run after removing database volumes before starting a dev instance.

Populates the database with dummy experiments of all types/statuses using the test factories

Run kill followed by refresh_db. Useful for resetting your dev environment when switching branches or after package updates.

Run migrate, load_locales_countries, and load_dummy_experiments.

Cirrus is required to run and test web application experiments locally. It is disabled by default. To enable Cirrus run:

This will be done automatically for any Cirrus related make commands.

Start a dev server listening on port 80 using the Django runserver. It is useful to run make refresh first to ensure your database is up to date with the latest migrations and test data.

Start postgresql, redis, autograph, kinto on their respective ports to allow running the Django runserver and yarn watchers locally (non containerized)

Start Django runserver, Celery worker, postgresql, redis, autograph, kinto on their respective ports to allow running the yarn watchers locally (non containerized)

Start all containers in the background (not attached to shell). They can be stopped using make kill.

Pull in the latest Kinto Docker image. Kinto is not automatically updated when new versions are available, so this command can be used occasionally to stay in sync.

Run all test and lint suites, this is run in CI on all PRs and deploys.

If you have a test failing to find an element (or finding too many, etc.) and the DOM is being cut off in the console output, you can increase how much is printed by locally editing the DEBUG_PRINT_LIMIT=7000 in the Makefile (line starts with JS_TEST_NIMBUS_UI).

Run only the python test suite.

Start a bash shell inside the container. This lets you interact with the containerized filesystem and run Django management commands.

You can run the entire python test suite without coverage using the Django test runner:

For faster performance you can run all tests in parallel:

You can run only the tests in a certain module by specifying its Python import path:

For more details on running Django tests refer to the Django test documentation

To debug a test, you can use ipdb by placing this snippet anywhere in your code, such as within a test method or inside some application logic:

Then invoke the test using its full path:

And you will enter an interactive iPython shell at the point where you placed the ipdb snippet, allowing you to introspect variables and call methods

For coverage you can use pytest, which will run all the python tests and track their coverage, but it is slower than using the Django test runner:

You can also enter a Python shell to import and interact with code directly, for example:

And then you can import and execute arbitrary code:

You can also interact with the yarn commands, such as checking TypeScript for Nimbus UI:

Or the test suite for Nimbus UI:

For a full reference of all the common commands that can be run inside the container, refer to this section of the Makefile

The integration tests run Selenium against a full Experimenter stack (app server, nginx, kinto, Firefox) inside Docker containers. They are defined in experimenter/tests/integration/nimbus/ and configured in .circleci/config.yml.

The easiest way to run and debug integration tests is via VNC, which lets you watch Firefox execute the test in real time.

1. Run a single make command with the test you want to run:

This automatically handles environment setup (copies .env.integration-tests, sets SKIP_DUMMY=1, configures PYTEST_ARGS with --reruns 0 and --base-url), rebuilds containers, and drops you into a shell inside the Firefox/Selenium container.

2. Connect a VNC viewer to watch the test:

   • VNC client (e.g. Safari on macOS, VNC Viewer): vnc://localhost:5900, password secret
   • noVNC (browser-based): http://localhost:7902, password secret

3. From the shell inside the container, run the test script:

This installs Firefox, Poetry dependencies, and runs pytest with your PYTEST_ARGS. You can watch the test execute in the VNC window.

Tests are organized with pytest markers. Use -m in PYTEST_ARGS to run a specific group:

If you prefer to run tests against a local Firefox install instead of the Docker container:

1. Install geckodriver (brew install geckodriver on macOS) and confirm it works:

   geckodriver --version

2. Make sure firefox is in your PATH. On macOS, add to your ~/.zshrc:

   alias firefox="/Applications/Firefox.app/Contents/MacOS/firefox"

3. Start Experimenter:

   cp .env.integration-tests .env
   make refresh build_integration_test SKIP_DUMMY=1 up_prod_detached

   Confirm it's running at https://localhost/nimbus.

4. Run the tests:

   PYTEST_ARGS="-k test_archive_experiment[FIREFOX_DESKTOP] --reruns 0" ./experimenter/tests/nimbus_integration_tests.sh

Firefox will launch and run the test. To use a different Firefox binary, set it in the firefox_options fixture in experimenter/tests/integration/nimbus/conftest.py:

• 502 from nginx: The experimenter container IP may have changed after a restart. Run docker restart to re-resolve it.
• Port conflicts: Only run one Experimenter stack at a time -- they share host ports (3001, 5432, 7001, 8000, 8888). Kill existing containers first with docker compose kill in the other directory.
• Feature dropdown selects wrong feature: The feature search is substring-based and selects the first alphabetical match. Use a feature name that won't substring-match other features.
• Flaky button clicks: Some tests may intermittently fail due to Selenium timing. Use --reruns 1 in CI or re-run locally. When debugging, use --reruns 0 to see the actual failure.

Navigate to experimenter/tests/tools

To test a targeting expression, first add an app context named app_context.json to the experimenter/tests/tools directory.

You can then invoke the script with the --targeting-string flag:

The script should return the results, either True, False, or an error.

Note that you can change the app_context live, and run the script again after.

In development you may wish to approve or reject changes to experiments as if they were on Remote Settings. You can do so here: http://localhost:8888/v1/admin/

There are three accounts you can log into Kinto with depending on what you want to do:

• admin / admin - This account has permission to view and edit all of the collections.
• experimenter / experimenter - This account is used by Experimenter to push its changes to Remote Settings and mark them for review.
• review / review - This account should generally be used by developers testing the workflow, it can be used to approve/reject changes pushed from Experimenter.

The admin and review credentials are hard-coded here, and the experimenter credentials can be found or updated in your .env file under KINTO_USER and KINTO_PASS.

Any change in remote settings requires two accounts:

• One to make changes and request a review
• One to review and approve/reject those changes

Any of the accounts above can be used for any of those two roles, but your local Experimenter will be configured to make its changes through the experimenter account, so that account can't also be used to approve/reject those changes, hence the existence of the review account.

For more detailed information on the Remote Settings integration please see the Kinto module documentation.

Experimenter has two front-end UIs:

• core is the legacy UI used for Experimenter intake which will remain until nimbus-ui supersedes it
• nimbus-ui is the Nimbus Console UI for Experimenter that is actively being developed

Learn more about the organization of these UIs here.

Also see the nimbus-ui README for relevent Nimbus documentation.

API documentation can be found here

Please see our Contributing Guidelines

Experimenter uses the Mozilla Public License