Table of Contents
UE Configuration for high real time quality
Aim of this page is to describe how to use your CPU at maximum (for the moment with 4 CPU)
As the UE is running with 3 main threads we can put one thread on one CPU (one CPU is used for non real time work).
In order to get good results when doing this experiment you need 4 CPU with a frequency > 3 GHz.
installation from scratch
Install Ubuntu, preferably 16.04 LTS, but any version after 14.04 should be fine
Preferably update it to the latest patch level
sudo apt update
sudo apt upgrade
and cleanup the obsolete stuff
sudo apt autoclean
sudo apt autoremove
install the lowlatency kernel and git
sudo apt install linux-lowlatency git
No need to configure the BIOS and the kernel
A reboot is required to load the updated linux kernel
OpenAirInterface Fresh installation
Download OpenAirInterface (git clone), choose a version (git checkout)
Install additional packages and compile as per regular OpenAirInterface Wiki pages (build_oai -I ...) without dealine scheduler option
Check that OpenAirInterface can run and perform your use case.
Define your machine configuration
We compiled OAI to use FIFO scheduler (the deadline scheduler creates errors in USRP/UHD driver).
We have to make a machine map, depending on the number of cores the machine has.
OAI has been defined for 4 actual cores, not counting the virtual cores created by Intel hyper-threading
for cpu in /sys/devices/system/cpu/cpu[0-9]* ; do
echo -n "Siblings for virtual core: ${cpu##*/} "
cat $cpu/topology/thread_siblings_list
done
If the cpus have more than one sibling with same core id, your machine don't have hyperthreading.
We will manage these virtual cores belonging to same physical core all together to prevent other users to grab the physical core from another virtual instance.
Core 0 is the default Linux core, we will always let this core for non real-time operations
A typical mapping, on a 4 cores machine
| core | usage |
|---|---|
| 0 | for non real-time |
| 3 | for I/Q samples acquisition and OAI internal scheduling |
| 1 and 2 | for the two processing threads that have been defined in OAI source code |
| In this standard configuration, one core processes even sub-frames while the odd sub-frames and processed by a second core. | |
| Therefore, a sub-frame processing can use 2 miliseconds of one core. |
If your machine has only 2 actual cores and if you run non real-time other programs such as XWindow server, you can dedicate only one core to OAI.
The performance will be limited nevertheless it can run as long as the total CPU time to process one sub-frame is always under 1 milisecond.
Configure OpenAirInterface
Depending on OAI branch, version you may have to set the cores for each thread.
This is defined at thread start-up in: targets/RT/USER/lte-ue.c (or lte-enb.c for eNB)
The cores usage have to be made, for each thread is the macro CPU_SET();
Then, the thread should call
pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset);
to send the configuration to the kernel
start the system
Configure your trace level, understanding that each trace line costs a couple of µseconds of CPU and a larger max jitter (more than 10µs in millions of calls)
NEVER use options -V (enable VCD, that is equivalent to a huge number of log traces) and -d (enable scope) when you perform performance tests.
There is two ways to choose the core for the thread :
1. With shield (not recommended) :
sudo bash
cset shield --force --kthread on -c <the list of cores for OAI>
(example : cset shield --force --kthread on -c 1-3)
for f in /sys/devices/system/cpu/cpu[0-9]* ; do
echo "performance" > $f/cpufreq/scaling_governor
done
The second part fix the cores frequency to their maximum frequency, the system could do this automatically, nevertheless UHD/USRP driver works better if we fix it.
Run the system on the chosen cores
sudo cset shield ./lte_noS1_build_oai/build/lte-softmodem-nos1 --U -C266... (then use your normal command line for the execution)
2. With command line :
The mapping of threads to cores can be controlled with the following options on command line:
--threadIQ xx xx is the core number, all threads related to IQ acquisition runs on this core
--threadOddSubframe xx core dedicated to run odd figure (1/3/5/7/9) sub frames: whole Rx and Tx processing from I/Q samples
--threadEvenSubframe xx core dedicated to run even figure (0/2/4/6/8) sub frames: whole Rx and Tx processing from I/Q samples
Run the system on the chosen cores
sudo cmake_targets/lte_<option_noS1_or_not>/build/lte-softmodem <noS1> -U --threadIQ 3 --threadOddSubframe 1 --threadEvenSubframe 2 -C266... (then use your normal command line for the execution)
Example :
sudo -E ./lte_noS1_build_oai/build/lte-softmodem-nos1 -U --threadIQ 3 --threadOddSubframe 1 --threadEvenSubframe 2 -C2680000000 -r100 --ue-scan-carrier --ue-txgain 80 --ue-rxgain 100 -d |tee UE.log
Measure the performance
A set of macros are defined in openair2/UTIL/LOG/log.h to track times
To enable them, set DEBUG_REALTIME in log.h, else the following C macros doesn't generate any assembly instructions
The macros are
- initRefTimes to create a memory for statistical measurements
- pickTime() picks the current time in a local variable, initStaticTime() and pickStatic time() do the same, in a static variable
- updateTimes(start, Meas *M, period, txt) this macro picks a time, computes the delay from 'sart' (taken with pickTime() or pickStaticTime()) stores in M structure the statistical data, and prints out the statistics each 'period' number of calls, associated with the text "txt"
- ckeck(max_time) is a macro that tests max delay between two calls of this macro, local to a thread. If the max_time is crossed, it prints out a trace with the source code line reference and the actual delay
Improve the HW interruptions latency
Move out all movable IRQ signals to the core 0
for f in /proc/irq/[0-9]* ; do echo '0' > $f/smp_affinity_list; done
Move the USRP/USB IRQ to the core 3: checkout the IRQ number by command 'top': this IRQ should come out under the lte-softmodem process
Move the IRQ to the same core as the I/Q manager (in above example: core 3)
echo '3' > /procirq/30/smp_affinity_list
Results and trace example
Doing the above configuration, including the interruption vector configuration Then, lauching a simple test, 5MHz, SISO on a i5-4570 CPU with USRP B200 can stay in real time like in this output example.
[PHY][W][printMeas] Delay to wake up UE_Thread_Rx (case 2) avg=1, iterations=650000, max=28/28/28/28/29/31/37/37/43/44
[PHY][W][printMeas] Delay to process sub-frame (case 3) avg=76, iterations=650000, max=286/286/288/289/291/298/315/320/322/371
[PHY][W][printMeas] Delay to wake up UE_Thread_Rx (case 2) avg=1, iterations=650000, max=22/22/27/27/28/28/29/30/32/32
[PHY][W][printMeas] Delay to process sub-frame (case 3) avg=66, iterations=650000, max=254/255/258/265/271/272/272/276/290/403
[PHY][W][printMeas] Delay between two IQ acquisitions (case 1) avg=999, iterations=1300000, max=1302/1306/1312/1314/1315/1372/1380/1380/1392/1474
Decoding of the trace:
- We processed 1.3 Million of sub frames, so about 20 minutes real time
- half (650000) have been processed in each of the twin threads that process sub-frames
- The acquisition time average is 999, as it should be 1000 (rounded) µseconds (OAI acquire I/Q samples per sub-frame)
- The top maximum delay encountered for this 1.3M acquisitions are: 1302/1306/1312/1314/1315/1372/1380/1380/1392/1474. This is due to the UHD driver and the packetization over the USB
- thread to thead signaling is 1 µsecond on average and absolute worse case is 44µs
- time to process a sub frame average is near 70 µsec, but we can observe peak times around 300µs
Meanwhile the test, I was browsing the web, performed some tests of internet traffic, launching other programs such as "libreoffice"