End-to-end tests for the Mattermost web app in general use Cypress and Playwright. If you’re not familiar with Cypress, check out the Cypress Developer Guide and API Reference. Feel free to also join us on the Mattermost Community server if you’d like to ask questions and collaborate with us!
Mark As Unread feature.Guest Accounts.E2E tests are located at the root of the repository in the `e2e-tests` folder. The file structure is mostly based on the Cypress scaffold. Here is an overview of some important folders and files:
|-- e2e-tests
|-- cypress
|-- tests
|-- fixtures
|-- integration
|-- plugins
|-- support
|-- utils
|-- cypress.config.ts
|-- package.json
/e2e-tests/cypress/tests/fixtures or Fixture Files:
cy.fixture() command and most often when stubbing network requests./e2e-tests/cypress/tests/integration or Test Files:
/e2e/cypress/tests/integration/messaging/ for “Messaging”)./e2e-tests/cypress/tests/plugins or Plugin Files:
spec file./e2e-tests/cypress/tests/support or Support Files:
spec files./e2e-tests/cypress/tests/utils: this folder contains common utility functions./e2e-tests/cypress/cypress.config.ts: this file is for Cypress configuration./e2e-tests/cypress/package.json: this file is for all the dependencies related to Cypress end-to-end testing.You will need to either add the new test to an existing spec file, or create a new file. Sometimes, you will be informed (for example through issue descriptions) of the specific folder the test file should go in, or the actual test file being amended. As aforementioned, the e2e-tests/cypress/tests/integration folder is where all of the tests live, with subdirectories that roughly divide the tests by functional areas. Cypress is configured to look for and run tests that match the pattern of *_spec.ts, so a good new test file name for an issue like Write Web App E2E with Cypress: "MM-T642 Attachment does not collapse" #18184 would be attachment_does_not_collapse_spec.ts, to ensure that it gets picked up.
Note: There may be some JavaScript
specfiles, but new tests should be written in TypeScript. If you are adding a test to an existingspecfile, convert that file to TypeScript if necessary.
If you don’t know where a test should go, first check the names of the subdirectories, and select a folder that describes the functional area of the test best. From there, look to see if there is already a spec file that may be similar to what you are testing; if there is one, it would be possible to add the test to the pre-existing file.
Test metadata is used to identify each spec file before it is forwarded for a Cypress run, and the metadata is located at the start of a spec file. Currently, supported test metadata fields include the following:
Stage - Indicates the environment for testing; valid values for this include @prod, @smoke, @pull_request. “Stage” metadata in spec files are owned and controlled by the Quality Assurance (QA) team who carefully analyze the stability of tests and promote/demote them into certain stages. This is not required when submitting a spec file and it should be removed when modifying an existing spec file.
Group - Indicates test group or category, which is primarily based on functional areas and existing release testing groups. Valid values for this include: @settings for Settings, @playbooks for Playbooks, etc. This is required when submitting a spec file.
Skip - This is a way to skip running a spec file depending on the capabilities of the test environment. This is required when submitting a spec file if there is a test that has certain limitations or requirements. Forms of capabilities include:
@darwin for Mac, @linux for Linux flavors like Ubuntu, @win32 for Windows, etc.@electron, @chrome, @firefox, @edge, etc.@headless or @headed.A spec file can have zero or more metadata values separated by spaces (for example, // Stage: @prod @smoke). A more full example of what metadata would look like at the start of a spec file (for example, attachment_does_not_collapse_spec.ts) would be:
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
// See LICENSE.txt for license information.
// ***************************************************************
// - [#] indicates a test step (e.g. # Go to a page)
// - [*] indicates an assertion (e.g. * Check the title)
// - Use element ID when selecting an element. Create one if none.
// ***************************************************************
// Stage: @prod
// Group: @incoming_webhook
The metadata is part of a comment block that also includes information on copyright and license, and a section to explain how to tag comments in your code appropriately.
Underneath the comment header, we can add the starter code as defined from the “Test code arrangement” part of the issue. Each test (no matter the situation you’re writing a test for) should have a corresponding test case in Zephyr. Therefore, the describe block encompassing the test code should correspond to folder name in Zephyr (e.g. “Incoming webhook”), and the it block should contain Zephyr test case number as Test Key, and then the test title. For Write Web App E2E with Cypress: "MM-T642 Attachment does not collapse" #18184, in the spec file made for it (attachment_does_not_collapse_spec.ts), the starter code would be:
describe('Integrations/Incoming Webhook', () => {
it('MM-T642 Attachment does not collapse', () => {
// Put test steps and assertions here
});
});
For those writing E2E from Help Wanted tickets with Area/E2E Tests label, the Test Key is available in the Github issue itself. The Test Key is used for mapping test cases per Release Testing specification. It will be used to measure coverage between manual and automated tests. In case the Test Key is not available, feel free to prompt the QA team who will either search for an existing Zephyr entry or if it’s a new one, it will be created for you.
Before writing the main body of the test in the it block, it can help to write some setup code for test isolation using hooks. In a before() hook, you can run tests in isolation using the custom command cy.apiInitSetup(). This command creates a new team, channel, and user which can only be used by the spec file itself. Make use of the cy.apiInitSetup() function as much as possible, as it is recommended to log in as a new user and visit the generated team and/or channel. Avoid the use of sysadmin user or default ad-1 team if possible.
For attachment_does_not_collapse_spec.ts for example:
let incomingWebhook;
let testChannel;
before(() => {
// # Create and visit new channel and create incoming webhook
cy.apiInitSetup().then(({team, channel}) => {
testChannel = channel;
const newIncomingHook = {
channel_id: channel.id,
channel_locked: true,
description: 'Incoming webhook - attachment does not collapse',
display_name: 'attachment-does-not-collapse',
};
cy.apiCreateWebhook(newIncomingHook).then((hook) => {
incomingWebhook = hook;
});
cy.visit(`/${team.name}/channels/${channel.name}`);
});
});
The before() hook is also a good place to add checks if a test requires a certain kind of server license. If test(s) require a certain licensed feature, use the function cy.apiRequireLicenseForFeature('<feature name>'). To check if the server has a license in general, use cy.apiRequireLicense(). You can also add hard requirements in the before() hook, such as: cy.shouldNotRunOnCloudEdition(), cy.shouldRunOnTeamEdition(), cy.shouldHavePluginUploadEnabled(), cy.shouldHaveElasticsearchDisabled(), and cy.requireWebhookServer(). For more information on custom commands and how to select elements, check out the End-to-End (E2E) cheatsheets.
Putting what you’ve gone through so far all together, you should have code that looks similar to this template:
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
// See LICENSE.txt for license information.
// **********************************************************************
// - Use [#] in comment to indicate a test step (e.g. # Go to a page)
// - Use [*] in comment to indicate an assertion (e.g. * Check the title)
// - Query an element with @testing-library/cypress as much as possible
// **********************************************************************
// Group: @change_group
describe('Change to Functional Group', () => {
before(() => {
// Add hard requirement(s) to immediately fail and throw a descriptive error if not met
// cy.shouldNotRunOnCloudEdition();
// Add license requirement(s)
// cy.apiRequireLicense();
// Init basic setup for test isolation
cy.apiInitSetup({loginAfter: true}).then(({team, channel, user}) => {
// Assign return values to variable/s
// # Visit a channel
// Do other setup per test data preconditions
});
});
// Add a title of "[Zephyr_id] - [Zephyr title]" for test case with single step,
// or "[Zephyr_id]_[step_number] - [Zephyr title]" for test case with multiple steps
it('[Zephyr_id] - [Zephyr title]', () => {
// Put test steps and assertions here
});
});
Now, inside the body of the it block , we will write in code the “Steps” part of the E2E issue. The following steps and code are from Write Web App E2E with Cypress: "MM-T642 Attachment does not collapse" #18184. Check out the complete file at: `attachment_does_not_collapse_spec.ts`.
Create an incoming webhook and send it through POST with attachment:
// # Post the incoming webhook with a text attachment
const content = '[very long lorem ipsum test text]';
const payload = {
channel: testChannel.name,
attachments: [{fallback: 'testing attachment does not collapse', pretext: 'testing attachment does not collapse', text: content}],
};
cy.postIncomingWebhook({url: incomingWebhook.url, data: payload, waitFor: 'attachment-pretext'});
View the webhook post that has the attachment: you are already in the channel that has the attachment post, as specified by the line cy.visit('/${team.name}/channels/${channel.name}'); from the setup section of the code.
Type /collapse and press Enter:
// * Check "show more" button is visible and click
cy.getLastPostId().then((postId) => {
const postMessageId = `#${postId}_message`;
cy.get(postMessageId).within(() => {
cy.get('#showMoreButton').scrollIntoView().should('be.visible').and('have.text', 'Show more').click();
});
});
// # Type /collapse and press Enter
const collapseCommand = 'collapse';
cy.uiGetPostTextBox().type(`/${collapseCommand} {enter}`);
Observe the integration post with the Message Attachment: where you ascertain what is expected of the test.
cy.getNthPostId(-2).then((postId) => {
const postMessageId = `#${postId}_message`;
cy.get(postMessageId).within(() => {
// * Verify "show more" button says "Show less"
cy.get('#showMoreButton').scrollIntoView().should('be.visible').and('have.text', 'Show less');
// * Verify gradient
cy.get('#collapseGradient').should('not.be.visible');
});
If the server is not running, launch it by running make run in the server directory. Then, confirm that the Mattermost instance has started successfully. You can also run make test-data in the server directory to preload your server instance with initial seed data (you may need to restart the server again).
server directory and running make config-reset.Change the directory to e2e-tests/cypress, and install dependencies by running npm i.
You can then run tests in a variety of different ways by using the following commands in the e2e-tests/cypress directory:
npm run cypress:run. This does not include the spec files in the /e2e-tests/cypress/tests/integration/enterprise folder because they need an Enterprise license to run successfully.spec metadata: For example, if you want to run all the tests in a specific group, such as those in “accessibility”, the command would be: node run_tests.js --group='@accessibility'.npm run cypress:open. This will start up the Cypress desktop app, where you will be able to do partial testing depending on the spec selected in the app. If you are using Gitpod, the Cypress app will open up in the VNC desktop, which is accessible at port 6080.Don’t forget to check your coding styles! See the Web app workflow page for helpful commands to run.
All tests are run by Mattermost in a CI pipeline, and they are grouped according to test stability.
node run_tests.js --stage='@prod'. These production tests are selected and also labeled with @prod in the test metadata. See link for an example test run posted in our community channel.node run_tests.js --stage='@prod' --invert to run all tests except production tests. These are called “unstable tests” as they either consistently or intermittently fail due to automation bugs, and not because of product bugs.Several environment variables (env variables) are used when testing with Cypress in order to easily change things when running tests in CI and to cater to different values across developer machines.
Environment variables are defined in cypress.config.ts under the env key. In most cases you don’t need to change the values, because it makes use of the default local developer setup. If you do need to make changes, the easiest method is to override by exporting CYPRESS_*, where * is the key of the variable, for example: CYPRESS_adminUsername. See the Cypress documentation on environment variables for details.
| Variable | Description |
|---|---|
| CYPRESS_adminUsername | Admin’s username for the test server. Default: sysadmin when server is seeded by make test-data. |
| CYPRESS_adminPassword | Admin’s password for the test server. Default: Sys@dmin-sample1 when server is seeded by make test-data. |
| CYPRESS_dbClient | The database of the test server. It should match the server config SqlSettings.DriverName.Default: postgres Valid values: postgres or mysql |
| CYPRESS_dbConnection | The database connection string of the test server. It should match the server config SqlSettings.DataSource.Default: "postgres://mmuser:mostest@localhost/mattermost_test?sslmode=disable\u0026connect_timeout=10" |
| CYPRESS_enableVisualTest | Use for visual regression testing. Default: falseValid values: true or false |
| CYPRESS_ldapServer | Host of the Lightweight Directory Access Protocol (LDAP) server. Default: localhost |
| CYPRESS_ldapPort | Port of the LDAP server. Default: 389 |
| CYPRESS_runLDAPSync | Option to run LDAP sync. Default: trueValid values: true or false |
| CYPRESS_resetBeforeTest | When set to true, it deletes all teams and their channels where sysadmin is a member except eligendi team and its channels.Default: falseValid values: true or false |
| CYPRESS_webhookBaseUrl | A server used for testing webhook integrations. Default: http://localhost:3000 when initiated with the command npm run start:webhook in the e2e-tests/cypress directory. |
Review the Test Guidelines for details on how to submit your PR
If test(s) are failing due to another known issue, follow these steps to amend your test:
-- KNOWN ISSUE: [Jira_key]. For example:
describe('Upload Files', () => {
it('MM-T2261 Upload SVG and post -- KNOWN ISSUE: MM-38982', () => {
// Test steps and assertion here
});
});
spec file following the format of <existing_spec_file_name_[1-9].js>. For example:
accessibility_account_settings_spec_1.js and demote the spec file (i.e. remove // Stage: @prod from the spec file)npm run cypress:run In this problem, either the command line exits immediately without running any test or it logs out like the following with the error message:
✖ Verifying Cypress can run /Users/user/Library/Caches/Cypress/3.1.3/Cypress.app
→ Cypress Version: 3.1.3
Cypress failed to start.
This is usually caused by a missing library or dependency.
The solution to this problem is to clear node options by initiating unset NODE_OPTIONS in the command line. Running npm run cypress:run should then proceed with Cypress testing.
ENOSPC This error may occur in Ubuntu when running any Cypress spec:
code: 'ENOSPC',
errno: 'ENOSPC',
The solution to this problem is to run the following command: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p.