UE Autotest Framework
Introduction
The objectives of this autotest framework is to perform automated e2e tests between OAI-UE and OAI-enodeB as a getting process in the continuous integration process of OAI (ref: Wiki page on integration process) . Autotests are configured to run a given test-suite set of UL/DL transmission conditions (Phy tests with various MCS, iperf throuput tests, endurance tests) that are passed when performance criteria are met (duration/link quality/throughput/...).
Autotest framework is based on Python Scripts capable of controlling the execution of Test Components running on a distributed configuration of Test Execution Machines (OAI EUs / OAI eNBs). Machines communicate by means of SSH connections.
Generic Configurations of Test Execution Machines
Generic Configurations of Test Execution Machines include:
- A Test Controller Machine (MTC) executing the Autotest Framework
- Two or more secondary Test Execution Machines (TEM) running oai
Testing Configuration used to support “noS1” Tests
Default configuration using USRP B210
The following configuration is the generic configuration to duplicate OAI 'noS1' autotest set-up.
- Testbed setup
- 1 test controller - see OAI Hardware requirements
- 1 eNodeB USRP B200/B210
- 1 UE USRP B200/B210
- 2x RF cable link with 60dB Attenuator : 60dB attenuators allows to run with default power settings (conf files)
Note: for hardware compatibility, please look at the OAI System requirements page.
The UE autotest framework support only the B200/B210 & ExpressMIMO2 hardware but can be easily extended to other OAI supported RF.
- Testbed capabilities in noS1 configuration
- Phy-test
- Iperf / video
- Ping
List of deployed testbeds (detailed description)
- SYRTEM UED testbed (link to be added)
- Eurecom OAI UE continuous integration testbed (link to be added)
Setting-up Run-Time Environment
Add a sudo user on all machines
Shall be done once on all machines (MTC, eNBs, UEs).
-
$USER : example oai_autotest
-
$PASSWD : example autotest Those user id and password will be set on the launch command line.
-
Add user with sudo privilege on each machine
#> sudo adduser '$USER' #> adduser '$USER' sudo #> Password '$PASSWD' -
Config $USER user sudo without psswd
#> export EDITOR="nano" #> sudo visudo Add following lines '$USER' ALL=(ALL:ALL) ALL '$USER' ALL=(ALL:ALL) NOPASSWD: ALL
Configure ssh access on all machines
Shall be done once on all machines (MTC, eNBs, UEs).
#> sudo apt install openssh-server
Machines executing OAI SW (UE / eNB)
Follow instructions for OpenAirInterface package installation : https://gitlab.eurecom.fr/oai/openairinterface5g/wikis/OpenAirSoftwareSupport
Note : Source code installation and OAI build are not mandatory as they will be done by the test script. But one can check that at least compilation phase is successful before launching tests script.
Master Tester Controller (MTC)
Install OAI software develop branch
#> git clone https://gitlab.eurecom.fr/oai/openairinterface5g.git #> cd openairinterface5g/ #> git checkout develop #> source oaienv
Create local folder for Autotest reports saving with rights access for your Autotest user $USER
#> sudo mkdir /mnt/sradio/TEST_RESULTS #> sudo chown $USER /mnt/sradio/TEST_RESULTS #> sudo chmod +w /mnt/sradio/TEST_RESULTS
Check MTC access to all machine
- Check MTC can reach machine on the network :
#> ping 'MACHINE' - Check MTC can access machine via ssh :
#> ssh 'USER'@'MACHINE'
NoS1 UE test suite
test-suite description
''File : $OPENAIR_DIR/cmake_targets/autotest/testsuite_ue_noS1.xm''
This testsuite is part of the OAI test PLAN under Objective '03' => 'Autotests specific to OAI UE' (see $OPENAIR_DIR/cmake_targets/autotest/Readme.txt for OAI test plan description). This testsuite use 'lte-softmodem-noS1' test class.
All testcases are targeting noS1 OAI configuration. Testcases can be identified by:
- id : unique testcase description identifier – 6 digits
- tag : give a more self explained test configuration (can be parsed easily by reports generation tools)
Testcase identification
- id format : Obj.# Case# Test#
- Obj#
- 03 → Autotests specific to OAI UE
- Case#
- 00..28 : 5MHz MCS00..MCS28
- 33..61 : 10MHz MCS00..MCS28
- 66..94 : 20MHz MCS00..MCS28
- Test# : 2digits
- 1rst digit → UE platform
- 0 = UE/USRPB210
- 1 = UEDv1
- 2..9 : reserved
- 2nd digit → test mode
- 0 = –phy-test performance
- 1 = –phy-test stability
- 2 = ping
- 3 = data iperf UDP performance downlink
- 4 = data iperf UDP stability downlink
- 5 = data iperf UDP performance uplink
- 6 = data iperf UDP stability uplink
- 7 = data iperf TCP performance downlink
- 8 = data iperf TCP performance uplink
- 9 = data iperf TCP performance downlink-uplink
- 1rst digit → UE platform
test-suite configuration
Before running a test campaign, one shall configure the test-suite to meet the specific setup requirements (RF gains, timings, machine names ).
One can find the generic test suite in the autotest directory $OPENAIR_DIR/cmake_targets/autotest/testsuite_ue_noS1.xml
- copy generic test-suite and create one for your setup (example : testsuite_ue_noS1_syrtem.xml)
- Update TEM (Test Execution Machines) description to map the testsuite to your testsetup. In the UE noS1 test-suite you will find the following TEM:
- ue_noS1_TEM_eNB_usrpb210 : replace all occurrences with your eNB machine name
- ue_noS1_TEM_ue_usrpb210 : replace all occurrences with your UE b210 machine name
- ue_noS1_TEM_ue_EXMIMO : replace all occurrences with your UE EXMIMO machine name
- Adapt for each test case OAI runtime parameters ( power, timing advance, ..) according your test setup
- see (https://gitlab.eurecom.fr/oai/openairinterface5g/wikis/HowToCalibrateeNBandUE) for timing calibration
- adjust UE power level in each test case ( --ue-txgain 85 and --ue-rxgain 125 )
NoS1 UE test suite execution
All execution commands are done on the MTC:
`#> cd 'Install dir' #> source oaienv #> $OPENAIR_DIR/cmake_targets/autotests/run_exec_lte-softmodem_tests.py -s -r -g "03+" --skip-oai-install -u $USER -p $PASSWD --skip-git-head-check --test-suite $OPENAIR_DIR/cmake_targets/autotests/testsuite_ue_noS1.xml --generate-html-report ``
Examples:
-
run full test suite
#> $OPENAIR_DIR/cmake_targets/autotests/run_exec_lte-softmodem_tests.py -s -r -g "03+" --skip-oai-install -u $USER -p $PASSWD --skip-git-head-check --test-suite $OPENAIR_DIR/cmake_targets/autotests/testsuite_ue_noS1.xml --generate-html-report -
run UE/USRP testcases 5MHz
$OPENAIR_DIR/cmake_targets/autotests/run_exec_lte-softmodem_tests.py -s -r -g "030000 030030 030900 030930 031000 031030 031600 031630 031700 031730 032800 032810 032830 032840" --skip-oai-install -u oai_autotest -p autotest --skip-git-head-check --test-suite $OPENAIR_DIR/cmake_targets/autotests/testsuite_ue_noS1.xml --generate-html-report
For more detailed command line parameters:
(https://gitlab.eurecom.fr/oai/openairinterface5g/wikis/OAITestSetup) or
$OPENAIR_DIR/cmake_targets/autotests/run_exec_lte-softmodem_tests.py -h :
- -s: This flag MUST be set to start the test cases
- -r: Remove the log directory in autotests
- -g: Run test cases in a group
- -c: Run cleanup scripts on remote machines and exit
- -5GRepo: Repository for OAI 5G to use to run tests (overrides GitOAI5GRepo in test_case_list.xml)
- -5GRepoBranch: Branch for OAI 5G Repository to run tests (overrides the branch in test_case_list.xml)
- -5GRepoHeadVersion: Head commit on which to run tests (overrides the branch in test_case_list.xml)
- -u: use the user name passed as argument
- -p: use the password passed as an argument
- -n: Set the NFS share passed as an argument
- --nrun_lte_softmodem: Set the number of runs for lte-softmodem test case class
- -MachineList : overrides the MachineList parameter in test_case_list.xml
- -MachineListGeneric : overrides the MachineListGeneric parameter in test_case_list.xml
- --skip-git-head-check: skip checking of GitHead remote/local branch (only for debugging)
- --timeout_cmd: Override the default parameter (timeout_cmd) in test_case_list.xml. This parameter is in seconds and should be > 120
- --skip-oai-install: Skips the openairinterface5g installer
- --skip-machine-preparation: skipp the whole system preparation -> direct to test cases
- --skip-sanity-check: skipp the machine sanity checks
- --test-suite: Select a XML test-suite file
- --generate-html-report: Generate an HTML report of the test campaign (supported only for noS1 testsuite !!)