CI: Add UE timing phytest (#188)
Add a new XML file to also measure UE timings, similarly how we do with
gNB.
Reviewed-by: Robert Schmidt <robert.schmidt@openairinterface.org>
Allow XML test cases to specify one or more files with timing stats that
define which container log paths to collect and analyze. AnalyzeRTStatsObject
loops over the provided list, falling back to the default gNB files
(nrL1_stats.log, nrMAC_stats.log) when none are specified.
Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: Jaroslava Fiedlerova <jaroslava.fiedlerova@openairinterface.org>
- Fix vrtsim uneven antenna configurations
- Fix shm_td_iq_channel client-server SHM race
- Set shm size to 2G in unit test docker run to prevent out-of-memory errors
Closes: #130
Signed-off-by: Bartosz Podrygajlo <bartosz.podrygajlo@openairinterface.org>
Use taskset with a dynamic range based on the number of detected processors
for OAI and DPDK build. This ensures both DPDK and OAI build steps use all
available CPU cores on the machine.
Build with XRAN K release
Signed-off-by: Jaroslava Fiedlerova <jaroslava.fiedlerova@openairinterface.org>
Remove CreateTag() and the merge/targetBranch parameters from
CreateWorkspace, passing the final branch name and repository directly
from Jenkins instead. Jenkinsfile computes testBranch and testRepository
(with INTERNAL_REPO as default). create_workspace.sh now does a simple
shallow clone by branch name.
Introduce two optional Jenkins parameters that allow overriding the
default branch reference and git repository URL used across the CI
pipeline.
- customBranch: when set, replaces the entire computed
{sourceBranch}-{commitID}
- customRepository: when set, replaces INTERNAL_REPO as the git remote
used by Create_Workspace to clone the source code onto the test node.
Signed-off-by: Jaroslava Fiedlerova <jaroslava.fiedlerova@openairinterface.org>
Remove the `BuildProxy` method, `Build_Proxy` action, `proxyCommit`
attribute, and all associated XML configuration and unit tests.
The oai-lte-5g-multi-ue-proxy repository has not received any updates
in over two years, and the proxy commit currently used for the build
is outdated as well. Although the proxy image is still required by
RAN-L2-Sim-Test-4G and must remain available on the test machine,
building it as part of the CI pipeline is dropped in this commit.
Instead, the CI will now assume that the corresponding proxy image is
already present as an external dependency, similarly to how CN images
are handled.
Signed-off-by: Jaroslava Fiedlerova <jaroslava.fiedlerova@openairinterface.org>
- enable configurable dockerfile, runtime-opt, and ctest-opt parameters in
the XML file for the Build_Run_Tests step
- pass runtime options to `docker run` to support ctest execution with CUDA
- allow additional ctest options for selective test execution and labeling
- all RAN code, CI code, configuration files, dockerfiles, in CSSL v1.0
- all deployment code (openshift, charts, ancillary files like shell
scripts), in MIT
- documentation in CC-BY-4.0
- exceptions might apply and are listed in NOTICE
- there is a new LICENSES folder with all licenses
- CONTRIBUTIONS.md has been updated accordingly
For automated changes based on OAI PL v1.1:
perl -i~ -0pe 's/\/\*.*Licensed to the OpenAirInterface.*openairinterface.org\n#?/\/*\n * SPDX-License-Identifier: LicenseRef-CSSL-1.0\n/s' **/*.{c,h,cpp}
perl -i~ -0pe 's/\/\*.*Licensed to the OpenAirInterface.*openairinterface.org\n#?/\/*\n * SPDX-License-Identifier: LicenseRef-CSSL-1.0\n/s' **/*.ts
perl -i~ -0pe 's/<!--.*Licensed to the OpenAirInterface.*openairinterface.org\n.*-->/<!-- SPDX-License-Identifier: LicenseRef-CSSL-1.0 -->/s' **/*.xml
The rest (cmake, files with missing license, cmake) manually.
When running phytest with FHI7.2 in Docker, two services are deployed:
dpdk-init and oai-gnb. Previously, AnalyzeRTStatsObject selected the
first deployed service for RT stats analysis.
Allow to explicitly select the service used for RT stats analysis instead
of relying on the first deployed service.
CI: rework log analysis to simplify and allow new use cases
The CI has some log analysis, but it is unmaintainable (huge functions
that mix eNB/gNB or lteUE/nrUE, are not tested, most of it is not used
or useful...) or limited to specific use cases (the timing-phytest time
analysis is hardcoded to source builds instead of also handling analysis
when running from docker images). This MR reimplements both functions.
For the log analysis, the CI team decided to throw away the old
analysis, and reimplement a mechanism that checks the couple of things
we care for.
- Allow to analyze phy-test timing analysis from docker files, and add a
unit test
- Rework existing log analysis to remove useless checks, and make it
more composable
* The undeployment steps (Undeploy_Object(), Terminate_eNB()) take an
<analysis> XML object which then either has a <services> subelement
with check that are whitespace delimited (similar to how other xml
task steps have <services>). For checks that contain whitespace,
they can use <service> (singular) that encodes a single check, such
that e.g., arguments could include spaces.
* Each check is of form service=check[=arguments], where service is
the docker services, check is a checking function (implemented as a
class with run(file, options) function inside cls_loganalysis.py,
and options arguments after the second = that is to be read by run()
(basically, options will contain arguments that can be freeform)
* these definitions can occur multiple times, e.g., it could be
serviceA=Check1 and serviceA=Check2=Args and it will execute both
Check1 and Check2
* if a service is declared, it will execute alse the Default check
which currently checks for an assertion
* Implemented (and inspired from the existing CI analysis) are:
- Default check to check for assertions (will be extended, to e.g.,
address/undefined behavior sanitizer analysis)
- RetxCheck: retransmission check as used in the CI
- LastLineContains to check the last line containes a specific
string
- ContainsString to check a specific string appears in logs
- EndsWithBye: check that towards the end (but not necessarily the
last line), there is a Bye. string (a workaround since we don't
always terminate with 0)
- There is a plan to use return codes after stopping in docker, but most
executables do not end with 0 most of the time, such that we needed to
abandon this plan.
- Additional fixes (see commit messages) and delete unused code
- we agreed to basically throw away the old log checker (as it is
unmaintainable)
- now: provide list of analysis to do
* all checkers in cls_log_analysis, can be unit tested (see also next
commit)
* looked up dynamically to simplify adding of new tests + config
* configured with
<analysis>
<services>DESCRIPTION [DESCRIPTION...]</services>
<service>DESCRIPTION</service>
...
</analysis>
where DESCRIPTION follows service=func[=options]:
- service is name of service to check
- func is function to call, which receives filename
- option are arbitary options to pass to func
<services> is whitespace delimited (so can take multiple service
analysis definitions)
<service> exists to allow service definitions with whitespace
changes that would not work in <services>
- I initially planned to check return code, but most softmodems actually
exit with non-zero return code, so this is still TODO
- Default analyzer (checking for assertions, ...) is always run if a
service is listed
- checks for file size
- add unit tests
We use Aerial exclusively on ARM/gracehopper, so no need to build it on
x86. On this occasion, rename the "kind" (ID which images to build) to
x86 for clarity and align with the other "kinds" (build_cross_arm,
native_armv9, ...).
The generated HTML from CI builds so far only shows the number of
errors, but not the errors themselves. This is inconvenient, as this
means that developers have to go to the logs.
This commit modifies AnalyzeBuildLogs(), which already iterates the
entire file, to store the lines containing "error:" (as it would be
emitted by a compiler). The callers of AnalyzeBuildLogs() now also print
the analysis outcome in a more concise manner.
Also, change the return parameter of AnalyzeBuildLogs() to a tuple to make
the variable handling easier, and remove the unused warnings list.
1. Rename oai-flexric to nearRT-RIC. FlexRIC is a project combining nearRT-RIC, xApps, and E2 agent emulators.
2. Instead of undeploying all the containers at the same time, and not checking the logs for xApps and nearRT-RIC, do the following:
- Stop containers (docker compose stop) - one or subgroup of running containers in the following order:
a) xapp-kpm-moni
b) xapp-kpm-rc
c) xapp-gtp-mac-rlc-pdcp-moni
d) oai-nr-ue oai-nr-ue2
e) oai-gnb
f) nearRT-RIC
- Undeploy containers (docker compose down), collect and analyze all the logs.
For some test cases, we might want to stop a service earlier. This
functionality is done by StopObject().
Test log collection is still done by UndeployObject(), which will
collect the logs of all services (at that point, stopped or not).
GetRunningServices() was returning all services, running or stopped.
Hence, rename to reflect what this function does, and rewrite slightly
to better capture what is actually happening. Also, don't return the
container ID, as we don't use it.
Call stop on all services, as we will down the deployment right after.
The list of services is still used to retrieve logs.
Remove a stray "return False", which would sometimes prevent us from
seeing all error messages.
Following the change to use:
image: ${REGISTRY-oaisoftwarealliance/}${GNB_IMG:-oai-gnb}:${TAG:-develop}
the REGISTRY variable must now include the trailing slash.
For clarity, rename to native_armv9 for clarity. The check inside Python
checked for native_arm, and if we introduce native_armv8 (for jetson),
it would also match that.
Instead of making the function a class member, make it an instance
member because (1) it is more aligned with other functions, notably,
run(), and (2) especially for RemoteCmd() it can be faster, as
successive exec_script() calls do not require a new SSH connection.
CI: speed up and simplify build, remove unnecessary Python code, remove --eNB* parameter list
This cleans up the CI framework for running end-to-end tests, and
improves the integration with OAI code. Specifically
- cleanup build_oai and helpers to output all text, which is captured
better by the CI
- build some images in the Cluster-Build in parallel to increase speed
- remove all --eNB{,1,2}{IPAddress,UserName,Password} variables. Those
were indexed through "instance" and "server ID" and are replaced with
the actual node name. I consider further cleanup to introduce a
"global" XML file node name so we don't need this in each step and
instead define this once.
- individual cleanup
It is superseded by the CreateWorkspace() function and task step (which
removes and recreates a workspace), and this option was not actually
used in any function except for the build of unit tests, but no XML
step actually specified this option.
After having transitioned to direct node names in the XML, these
parameters are not relevant anymore.
The best way to view this patch is with
git show <SHA> --ignore-all-space
Pass node directly from XML to these functions
- BuildClusterImage()
- BuildImage()
- BuildProxy()
- BuildRunTests()
instead of reading the "server ID" and looking up from
eNB_IPAddress/eNB1_IPAddress/eNB2_IPAddress.
Improve the proxy build unit test by properly passing ctx information.
