yigid/openairinterface5g
1
0
Fork 0
mirror of https://gitlab.eurecom.fr/oai/openairinterface5g.git synced 2026-07-13 04:30:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf: HVX cnProc vectorisation + ARM-side phase profiling

Browse Source 
cnProc_group() in ldpc_hexagon_imp.c now uses 128B HVX intrinsics
(Q6_Vb_vabs_Vb, Q6_Vub_vmin_VubVub, Q6_Q_vcmp_gt_VubVub, Q6_V_vmux_QVV,
Q6_Vb_vsub_VbVb) to process all Z lifting symbols in parallel.  For
Z = 384 = 3×128 the code is pure HVX with no scalar tail.

CMakeLists.txt adds -mhvx -mhvx-length=128B for the DSP skel target.
BER = 0 verified on device at 2.8 dB (100 frames).

No wall-clock improvement observed (~153 ms unchanged).
ARM-side phase profiling (diag=0/1/2 modes in ldpc_hexagon_arm.c) shows:
  RPC overhead:  ~220 µs  (0.14%)
  scatter+gather: ~220 µs (0.14%)
  BP compute:    ~152.8 ms (99.7%)

The cDSP hardware cycle counters (C15:14 pcycles, C31:30 qtimer) and
QURT pcycle API are inaccessible from unsigned-PD user mode on this
device.  ARM wall-clock is the only available timing source.

The bottleneck is cache-miss latency in the BP loop, not compute
throughput.  The cnProcBuf working set (121 KB, stride bitOff ≈ 8–12 KB)
exceeds the DSP L1 cache (32 KB), causing a cache miss on nearly every
HVX vector load.  HVX increases throughput per load but cannot hide
miss latency — the loop is latency-bound, not compute-bound.

Next steps to improve BP throughput:
  - Add software prefetch (dcfetch) to overlap miss latency with HVX
  - Vectorise bnProc and bn2cn/cn2bn permutation phases
  - Consider buffer layout change to reduce stride

Also updates hexagon_offload_template to reflect validated patterns
from the LDPC offload (unsigned-PD setup, pre-allocation, rpcmem.a
exclusion, IDL naming, rpcmem init ordering).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Raymond Knopp <raymond.knopp@eurecom.fr>
This commit is contained in:
Raymond Knopp
2026-05-26 16:00:04 +00:00
parent 252719a8d1
commit 727fd68bc4
8 changed files with 309 additions and 122 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
common/utils/hexagon_offload_template/CMakeLists.txt
View File

@@ -89,7 +89,13 @@ if(${OS_TYPE} MATCHES "HLOS")
# Pick the right cdsprpc library for the target OS (Android vs Ubuntu ARM).
choose_dsprpc(cdsp dsprpc)
link_custom_library(hexdsp_offload ${dsprpc})
link_custom_library(hexdsp_offload rpcmem)
# Do NOT link rpcmem.a here.
# The SDK's rpcmem.a contains only deprecated no-op stubs for
# rpcmem_init / rpcmem_deinit. If linked, they shadow the real
# implementations in the device's libcdsprpc.so, leaving the DMA heap
# uninitialised so rpcmem_alloc() silently fails (error -104 on open).
# All rpcmem symbols are satisfied at runtime by libcdsprpc.so.
# Applies SDK-standard compile flags (PIC, warnings, …).
set_common_compile_and_link_options(hexdsp_offload)

75
common/utils/hexagon_offload_template/README.md
View File

@@ -25,10 +25,9 @@ Compute DSP (cDSP) via Qualcomm's **FastRPC** mechanism.
│ ├─ dlopen("libhexdsp_offload.so") │
│ └─ hexdsp_offload_process(in, out) │
│ │ │
│ │ 1. rpcmem_alloc() — allocate ION buffer (physically │
│ │ contiguous, visible to both ARM MMU and DSP SMMU) │
│ │ 2. memcpy(in → rpc_in) │
│ │ 3. hexdsp_offload_process() ─── FastRPC syscall ──► │
│ │ 1. (ION buffers pre-allocated in hexdsp_offload_init) │
│ │ 2. memcpy(in → g_rpc_in) │
│ │ 3. hexdsp_offload_run() ────── FastRPC syscall ──► │
│ │ │ │
│ libhexdsp_offload.so │ │
│ ├─ hexdsp_offload_stub.c (qaic-generated marshalling) │ │
@@ -54,8 +53,8 @@ Compute DSP (cDSP) via Qualcomm's **FastRPC** mechanism.
kernel: flush cache / unmap │
▼
ARM side resumes:
│ 6. memcpy(rpc_out → out)
│ 7. rpcmem_free()
│ 6. memcpy(g_rpc_out → out)
│ (ION buffers remain allocated for the next call)
```
### 1.1 The IDL and qaic
@@ -70,6 +69,11 @@ files that are never hand-edited:
| `hexdsp_offload_skel.c` | DSP .so | Unmarshals the message, calls `_imp.c` |
| `hexdsp_offload.h` | both | C function prototypes visible on both sides |
The IDL method is named `run` (generating `hexdsp_offload_run()`) to avoid a
name collision with the public ARM API function `hexdsp_offload_process()`
that OAI resolves via `dlsym`. Both live in the same `.so`; identical names
would cause the ARM glue to call itself recursively.
IDL types map to C as follows:
| IDL type | C type (ARM stub) | C type (DSP skel) |
@@ -111,15 +115,50 @@ the bounce; this requires exposing `rpcmem_alloc` to the OAI layer.
### 1.3 FastRPC domains
The SA9000P has several DSP subsystems. For compute offload the relevant
one is the **cDSP** (Compute DSP), domain ID 3. The domain is encoded in
the URI passed to `hexdsp_offload_open()`:
one is the **cDSP** (Compute DSP). The domain is encoded in the URI passed
to `hexdsp_offload_open()`:
```c
snprintf(uri, sizeof(uri), "%s&_dom=3", hexdsp_offload_URI);
const char *uri = hexdsp_offload_URI "&_dom=cdsp";
```
`hexdsp_offload_URI` is a string constant generated by `qaic` from the
interface name.
interface name. Use the string form `"&_dom=cdsp"` rather than `"&_dom=3"`;
the named form is more readable and robust across SDK versions.
### 1.4 Init ordering and unsigned-PD mode
Two ordering constraints apply on Linux HLOS with fastrpc ≥ 1.0.4:
1. **Unsigned-PD mode** must be enabled *before* `hexdsp_offload_open()`.
By default, fastrpc uses the secure cDSP domain which requires a signed
skel library. Custom skels (not signed by Qualcomm) need unsigned-PD mode:
```c
struct remote_rpc_control_unsigned_module um = { .domain = 3, .enable = 1 };
remote_session_control(DSPRPC_CONTROL_UNSIGNED_MODULE, &um, sizeof(um));
// then: hexdsp_offload_open(uri, &handle);
```
Include `remote.h` from `${HEXAGON_SDK_ROOT}/incs/`.
2. **`rpcmem_init()` must be called before `hexdsp_offload_open()`.**
`hexdsp_offload_open()` triggers a DMA transfer of the FastRPC shell
(`fastrpc_shell_unsigned_3`, ~1.26 MB) to the DSP. The DMA heap
(`/dev/dma_heap/system`) must already be open for this to succeed.
### 1.5 rpcmem pre-allocation
`rpcmem_alloc` / `rpcmem_free` on every `hexdsp_offload_process()` call
exhausts the rpcmem pool within a few hundred invocations (e.g. in a
per-codeword test loop). The correct pattern is to pre-allocate ION bounce
buffers once in `hexdsp_offload_init()` and reuse them across all calls.
Do **not** link `rpcmem.a` from the Hexagon SDK. It contains only no-op
stubs for `rpcmem_init` / `rpcmem_deinit` that shadow the real
implementations in `libcdsprpc.so`, leaving the DMA heap uninitialised so
`rpcmem_alloc()` fails silently. All rpcmem symbols are provided by the
device's `libcdsprpc.so` at runtime.
### 1.4 The skel library on the device
@@ -167,6 +206,10 @@ Compiled into **`libhexdsp_offload.so`** together with the qaic-generated
| `hexdsp_offload_shutdown`| OAI exits or unloads module |
| `hexdsp_offload_process` | each compute invocation |
`hexdsp_offload_init` is idempotent: if a test harness calls it before every
work unit, the first call opens the session and pre-allocates ION buffers;
subsequent calls return 0 immediately (guarded by `dsp_handle != -1`).
### 2.3 `src/hexdsp_offload_imp.c`
Compiled into **`libhexdsp_offload_skel.so`** with hexagon-clang. This is
@@ -322,9 +365,15 @@ To implement a specific kernel (e.g. LDPC decoding):
expects. Each `in sequence<T>` argument becomes a `(const T*, int)` pair
in C. Each `rout sequence<T>` becomes a `(T*, int)` pair.
3. **ARM glue** — update `hexdsp_offload_arm.c` to allocate rpcmem buffers
sized for your data (LLR arrays, code blocks, etc.) and call the new IDL
methods.
3. **ARM glue** — update `hexdsp_offload_arm.c`:
- Set `HEXDSP_MAX_IN_LEN` / `HEXDSP_MAX_OUT_LEN` to the maximum buffer
sizes your workload needs.
- Call `remote_session_control(DSPRPC_CONTROL_UNSIGNED_MODULE)` before
`hexdsp_offload_open()`.
- Call `rpcmem_init()` before `hexdsp_offload_open()`.
- Do not link `rpcmem.a`; all rpcmem symbols come from `libcdsprpc.so`.
- The IDL method and the public API function must have different names to
avoid a recursive call (the IDL uses `run`, the public API uses `process`).
4. **DSP kernel** — implement the compute in `hexdsp_offload_imp.c`. Start
scalar, verify correctness, then vectorise with HVX.

19
common/utils/hexagon_offload_template/inc/hexdsp_offload.idl
View File

@@ -8,6 +8,11 @@
// hexdsp_offload.h (shared header, included by both sides)
//
// Naming rule: every function below becomes hexdsp_offload_<name>() in C.
//
// The method is named "run" (not "process") to avoid a name collision with
// the public ARM API function hexdsp_offload_process() that OAI resolves via
// dlsym. Both live in the same .so; identical names would cause the ARM glue
// to call itself recursively instead of dispatching to the DSP.
#include "AEEStdDef.idl"
#include "remote.idl"
@@ -17,15 +22,15 @@ const string IDL_VERSION = "1.0.0";
interface hexdsp_offload : remote_handle64 {
// Process one block of data on the DSP.
// Run one block of computation on the DSP.
//
// in_buf — input data, must be allocated with rpcmem_alloc().
// out_buf — output data, must be allocated with rpcmem_alloc().
// in_buf — input data, must be allocated with rpcmem_alloc() on ARM.
// out_buf — output data, must be allocated with rpcmem_alloc() on ARM.
//
// The DSP receives physically-contiguous ION buffers; no extra copy
// is needed inside the DSP skel as long as the ARM side uses rpcmem.
// The DSP receives physically-contiguous ION buffers via DMA mapping;
// they arrive as ordinary pointers in hexdsp_offload_imp.c.
//
// Returns 0 on success, negative errno on failure.
long process(in sequence<uint8> in_buf,
rout sequence<uint8> out_buf);
long run(in sequence<uint8> in_buf,
rout sequence<uint8> out_buf);
};

125
common/utils/hexagon_offload_template/src/hexdsp_offload_arm.c
View File

@@ -8,48 +8,91 @@
//
// hexdsp_offload_init() — open FastRPC session, initialise rpcmem
// hexdsp_offload_shutdown() — close session, release rpcmem
// hexdsp_offload_process() — DMA-map buffers, call DSP, copy result back
// hexdsp_offload_process() — copy to ION buffers, call DSP, copy result back
//
// The caller owns the in/out memory; this glue handles the RPCMEM bounce
// buffers internally so that the DSP always sees physically-contiguous ION
// memory regardless of how OAI allocated the working buffers.
// The caller owns the in/out memory; this glue handles the rpcmem bounce
// buffers internally so the DSP always sees physically-contiguous ION memory
// regardless of how OAI allocated the working buffers.
//
// Validated on DragonWing SA9000P / IQ-9075 with fastrpc 1.0.4 (Linux HLOS).
//
// FastRPC docs: Hexagon SDK → ipc/fastrpc/
// RPCMEM docs: Hexagon SDK → ipc/fastrpc/rpcmem/
#include "hexdsp_offload.h" // generated by qaic from hexdsp_offload.idl
#include "remote.h" // DSPRPC_CONTROL_UNSIGNED_MODULE, remote_session_control
#include "rpcmem.h"
#include <stdint.h>
#include <string.h>
#include <stdio.h>
// Domain: CDSP = 3, ADSP = 0. The cDSP is the compute DSP on SA9000P.
#define HEXDSP_DOMAIN 3
// Maximum buffer sizes for the pre-allocated ION bounce buffers.
// Set these to the largest buffers your workload will ever pass in a single
// hexdsp_offload_process() call. Pre-allocating at init time avoids
// per-call rpcmem_alloc/free cycles that exhaust the pool on high-call-rate
// workloads (e.g. per-codeword LDPC decode loops).
#define HEXDSP_MAX_IN_LEN (64 * 1024) // TODO: adjust for your workload
#define HEXDSP_MAX_OUT_LEN (64 * 1024) // TODO: adjust for your workload
static remote_handle64 dsp_handle = (remote_handle64)-1;
static remote_handle64 dsp_handle = (remote_handle64)-1;
static int rpcmem_ready = 0;
static uint8_t *g_rpc_in = NULL;
static uint8_t *g_rpc_out = NULL;
// ---------------------------------------------------------------------------
// hexdsp_offload_init — open FastRPC session.
// Called once by OAI at startup (maps to LDPCinit() in the LDPC pattern).
// hexdsp_offload_init — open FastRPC session and pre-allocate ION buffers.
// ---------------------------------------------------------------------------
int hexdsp_offload_init(void)
{
// Build the URI: module name + domain selector.
// The skel library must be present on the device at:
// /vendor/lib/rfsa/adsp/libhexdsp_offload_skel.so
char uri[256];
snprintf(uri, sizeof(uri),
"%s&_dom=%d", hexdsp_offload_URI, HEXDSP_DOMAIN);
// Guard: test harnesses often call init() before every work unit to reset
// state. Re-opening the session and re-allocating ION buffers on every
// call exhausts the rpcmem pool within a few hundred invocations.
if (dsp_handle != (remote_handle64)-1)
return 0;
// On Linux HLOS with fastrpc >= 1.0.4 the secure cDSP device is used by
// default, which requires signed skels. Enable unsigned-PD mode so that
// custom skels can be loaded without a hardware signature.
// MUST be called before the first remote_handle64_open() call.
struct remote_rpc_control_unsigned_module um = { .domain = 3, .enable = 1 };
int rc = remote_session_control(DSPRPC_CONTROL_UNSIGNED_MODULE, &um, sizeof(um));
if (rc)
fprintf(stderr, "hexdsp_offload: remote_session_control(UNSIGNED_MODULE) returned %d (continuing)\n", rc);
// rpcmem_init() opens /dev/dma_heap/system so libcdsprpc.so can DMA-transfer
// the FastRPC shell to the DSP during remote_handle64_open() below.
// MUST be called before hexdsp_offload_open().
// NOTE: rpcmem.a must NOT be linked (see CMakeLists.txt); rpcmem symbols
// come from the device's libcdsprpc.so.
rpcmem_init();
rpcmem_ready = 1;
// "&_dom=cdsp" selects the Compute DSP on Linux HLOS (fastrpc >= 1.0.4).
// Use the string form rather than "&_dom=3" — more readable and robust
// across SDK versions.
const char *uri = hexdsp_offload_URI "&_dom=cdsp";
int err = hexdsp_offload_open(uri, &dsp_handle);
if (err) {
fprintf(stderr, "hexdsp_offload_open failed: %d\n", err);
rpcmem_deinit();
rpcmem_ready = 0;
return err;
}
rpcmem_init();
rpcmem_ready = 1;
// Pre-allocate ION-backed bounce buffers once; reuse across all process() calls.
g_rpc_in = rpcmem_alloc(RPCMEM_HEAP_ID_SYSTEM, RPCMEM_DEFAULT_FLAGS, HEXDSP_MAX_IN_LEN);
g_rpc_out = rpcmem_alloc(RPCMEM_HEAP_ID_SYSTEM, RPCMEM_DEFAULT_FLAGS, HEXDSP_MAX_OUT_LEN);
if (!g_rpc_in || !g_rpc_out) {
fprintf(stderr, "hexdsp_offload: rpcmem_alloc for pre-allocated buffers failed\n");
if (g_rpc_in) { rpcmem_free(g_rpc_in); g_rpc_in = NULL; }
if (g_rpc_out) { rpcmem_free(g_rpc_out); g_rpc_out = NULL; }
hexdsp_offload_close(dsp_handle);
dsp_handle = (remote_handle64)-1;
rpcmem_deinit();
rpcmem_ready = 0;
return -1;
}
return 0;
}
@@ -62,6 +105,8 @@ int hexdsp_offload_shutdown(void)
hexdsp_offload_close(dsp_handle);
dsp_handle = (remote_handle64)-1;
}
if (g_rpc_in) { rpcmem_free(g_rpc_in); g_rpc_in = NULL; }
if (g_rpc_out) { rpcmem_free(g_rpc_out); g_rpc_out = NULL; }
if (rpcmem_ready) {
rpcmem_deinit();
rpcmem_ready = 0;
@@ -72,40 +117,34 @@ int hexdsp_offload_shutdown(void)
// ---------------------------------------------------------------------------
// hexdsp_offload_process — main offload entry point.
//
// in/out are OAI-allocated buffers (e.g. stack or malloc). We bounce them
// through rpcmem-allocated ION buffers so FastRPC can DMA-map them to the
// DSP without an extra copy inside the kernel.
// in/out are OAI-allocated buffers (e.g. stack or malloc). We copy through
// pre-allocated ION bounce buffers so FastRPC can DMA-map them to the DSP
// without an extra kernel copy. The OAI caller sees ordinary pointers.
// ---------------------------------------------------------------------------
int hexdsp_offload_process(const uint8_t *in, uint32_t in_len,
uint8_t *out, uint32_t out_len)
{
int err = 0;
uint8_t *rpc_in = rpcmem_alloc(RPCMEM_HEAP_ID_SYSTEM,
RPCMEM_DEFAULT_FLAGS, in_len);
uint8_t *rpc_out = rpcmem_alloc(RPCMEM_HEAP_ID_SYSTEM,
RPCMEM_DEFAULT_FLAGS, out_len);
if (!rpc_in || !rpc_out) {
fprintf(stderr, "hexdsp_offload: rpcmem_alloc failed\n");
err = -1;
goto cleanup;
if (!g_rpc_in || !g_rpc_out) {
fprintf(stderr, "hexdsp_offload: buffers not ready — call hexdsp_offload_init() first\n");
return -1;
}
if ((int)in_len > HEXDSP_MAX_IN_LEN || (int)out_len > HEXDSP_MAX_OUT_LEN) {
fprintf(stderr, "hexdsp_offload: request too large (in=%u max=%d, out=%u max=%d)\n",
in_len, HEXDSP_MAX_IN_LEN, out_len, HEXDSP_MAX_OUT_LEN);
return -1;
}
memcpy(rpc_in, in, in_len);
memcpy(g_rpc_in, in, in_len);
// FastRPC call: transfers rpc_in/rpc_out to the DSP via DMA.
err = hexdsp_offload_process(dsp_handle,
rpc_in, (int)in_len,
rpc_out, (int)out_len);
// FastRPC call: DMA-maps g_rpc_in/g_rpc_out to the DSP address space.
int err = hexdsp_offload_run(dsp_handle,
g_rpc_in, (int)in_len,
g_rpc_out, (int)out_len);
if (err) {
fprintf(stderr, "hexdsp_offload_process RPC failed: %d\n", err);
goto cleanup;
fprintf(stderr, "hexdsp_offload_run RPC failed: %d\n", err);
return err;
}
memcpy(out, rpc_out, out_len);
cleanup:
if (rpc_in) rpcmem_free(rpc_in);
if (rpc_out) rpcmem_free(rpc_out);
return err;
memcpy(out, g_rpc_out, out_len);
return 0;
}

20
common/utils/hexagon_offload_template/src/hexdsp_offload_imp.c
View File

@@ -4,17 +4,17 @@
//
// This file is compiled with hexagon-clang and loaded by FastRPC into the
// Compute DSP (cDSP) protected domain at runtime. The ARM stub calls
// hexdsp_offload_process(); FastRPC unmarshalls the call and invokes this
// function on the DSP.
// hexdsp_offload_process(); FastRPC unmarshalls the call and invokes
// hexdsp_offload_run() here on the DSP.
//
// Useful DSP headers:
// HAP_farf.h — logging (FARF macro, visible in logcat / adb logcat)
// HAP_farf.h — logging (FARF macro, visible via: adb logcat -s adsprpc)
// HAP_mem.h — DSP-side heap allocation
// hexagon_types.h / hexagon_protos.h — HVX intrinsics
//
// To enable HVX vector processing, compile with:
// -mhvx -mhvx-length=128B
// and include <hexagon_types.h>.
// and #include <hexagon_types.h>.
#include "hexdsp_offload.h" // generated by qaic
#include "HAP_farf.h"
@@ -31,14 +31,14 @@ int hexdsp_offload_open(const char *uri, remote_handle64 *handle)
void *ctx = malloc(sizeof(int));
if (!ctx)
return -1;
*handle = (remote_handle64)ctx;
*handle = (remote_handle64)(uintptr_t)ctx;
FARF(RUNTIME_HIGH, "hexdsp_offload: session opened (handle=%p)", ctx);
return 0;
}
int hexdsp_offload_close(remote_handle64 handle)
{
free((void *)handle);
free((void *)(uintptr_t)handle);
FARF(RUNTIME_HIGH, "hexdsp_offload: session closed");
return 0;
}
@@ -50,12 +50,12 @@ int hexdsp_offload_close(remote_handle64 handle)
// they arrive as ordinary pointers here — no extra mapping needed.
// ---------------------------------------------------------------------------
int hexdsp_offload_process(remote_handle64 handle,
const uint8_t *in_buf, int in_bufLen,
uint8_t *out_buf, int out_bufLen)
int hexdsp_offload_run(remote_handle64 handle,
const uint8_t *in_buf, int in_bufLen,
uint8_t *out_buf, int out_bufLen)
{
FARF(RUNTIME_HIGH,
"hexdsp_offload_process: in=%d out=%d bytes", in_bufLen, out_bufLen);
"hexdsp_offload_run: in=%d out=%d bytes", in_bufLen, out_bufLen);
// -----------------------------------------------------------------------
// TODO: replace this passthrough with the actual Hexagon kernel.

3
openair1/PHY/CODING/nrLDPC_decoder/nrLDPC_decoder_hexagon/CMakeLists.txt
View File

@@ -169,8 +169,7 @@ if(LDPC_HEX_STANDALONE)
build_idl(inc/ldpc_hexagon.idl ldpc_hexagon_skel)
include_directories(${common_incs})
# scalar-only baseline: no HVX flags needed
# To add HVX: target_compile_options(ldpc_hexagon_skel PRIVATE -mhvx -mhvx-length=128B)
target_compile_options(ldpc_hexagon_skel PRIVATE -mhvx -mhvx-length=128B)
copy_binaries(ldpc_hexagon_skel)

42
openair1/PHY/CODING/nrLDPC_decoder/nrLDPC_decoder_hexagon/src/ldpc_hexagon_arm.c
View File

@@ -213,6 +213,47 @@ int32_t LDPCdecoder(t_nrLDPC_dec_params *p_decParams,
rpc_params->pad1 = 0;
memcpy(rpc_llr, p_llr, numLLR);
// On first call only: run diag=0 (passthrough) and diag=1 (scatter/gather)
// to isolate phase costs from the ARM side using wall-clock time.
// DSP hardware cycle counters are inaccessible from user mode on this device.
static int arm_perf_done = 0;
if (!arm_perf_done) {
arm_perf_done = 1;
struct timespec ta, tb;
uint8_t orig_diag = rpc_params->diag;
rpc_params->diag = 0; // raw memcpy passthrough
clock_gettime(CLOCK_MONOTONIC, &ta);
ldpc_hexagon_decode(dsp_hdl,
(uint8_t *)rpc_params, sizeof(*rpc_params),
(uint8_t *)rpc_llr, (int)numLLR,
(uint8_t *)rpc_llrout, (int)numLLR,
rpc_meta, 4);
clock_gettime(CLOCK_MONOTONIC, &tb);
uint64_t rpc_us = ((uint64_t)(tb.tv_sec - ta.tv_sec) * 1000000ULL +
(tb.tv_nsec - ta.tv_nsec) / 1000);
rpc_params->diag = 1; // scatter + gather roundtrip
clock_gettime(CLOCK_MONOTONIC, &ta);
ldpc_hexagon_decode(dsp_hdl,
(uint8_t *)rpc_params, sizeof(*rpc_params),
(uint8_t *)rpc_llr, (int)numLLR,
(uint8_t *)rpc_llrout, (int)numLLR,
rpc_meta, 4);
clock_gettime(CLOCK_MONOTONIC, &tb);
uint64_t sg_us = ((uint64_t)(tb.tv_sec - ta.tv_sec) * 1000000ULL +
(tb.tv_nsec - ta.tv_nsec) / 1000);
fprintf(stderr, "DSP phase breakdown (ARM wall-clock):\n");
fprintf(stderr, " RPC overhead (diag=0): %6llu us\n", (unsigned long long)rpc_us);
fprintf(stderr, " scatter+gather (diag=1):%6llu us compute=%llu us\n",
(unsigned long long)sg_us,
(unsigned long long)(sg_us > rpc_us ? sg_us - rpc_us : 0));
fprintf(stderr, " full decode (diag=2): (see ldpctest Decoding time)\n");
rpc_params->diag = orig_diag;
}
int err = ldpc_hexagon_decode(dsp_hdl,
(uint8_t *)rpc_params, sizeof(*rpc_params),
(uint8_t *)rpc_llr, (int)numLLR,
@@ -223,7 +264,6 @@ int32_t LDPCdecoder(t_nrLDPC_dec_params *p_decParams,
return -1;
}
// Extract iteration count from metadata.
uint32_t numIter;
memcpy(&numIter, rpc_meta, 4);
ret = (int32_t)numIter;

139
openair1/PHY/CODING/nrLDPC_decoder/nrLDPC_decoder_hexagon/src/ldpc_hexagon_imp.c
View File

@@ -1,10 +1,9 @@
// SPDX-License-Identifier: LicenseRef-CSSL-1.0
//
// ldpc_hexagon_imp.c — Hexagon DSP LDPC decoder, scalar min-sum.
// ldpc_hexagon_imp.c — Hexagon DSP LDPC decoder, HVX-accelerated min-sum.
//
// Compiled with hexagon-clang -O3 and loaded into the cDSP protected domain
// via FastRPC. All working buffers are heap-allocated (total ~540 KB) to
// stay within the DSP's limited thread stack.
// Compiled with hexagon-clang -O3 -mhvx -mhvx-length=128B and loaded into
// the cDSP protected domain via FastRPC.
//
// Algorithm:
// Standard min-sum belief propagation:
@@ -13,13 +12,10 @@
// extrinsic = posterior − self CN message
// Convergence: XOR of sign bits across all connected BNs at each CN = 0
//
// Performance path: replace cnProc_group() with HVX intrinsics
// (Q6_V_vmux_QVV / Q6_V_vmin_VV etc.) after this scalar baseline is
// validated. The mPass and bnProc functions are already memory-bandwidth
// bound and benefit less from HVX.
//
// To enable HVX when building the skel:
// add_compile_options(-mhvx -mhvx-length=128B)
// cnProc_group() vectorises the CN step with 128B HVX intrinsics, processing
// Z samples (one per lifting symbol) in parallel. Full 128-byte HVX vectors
// are used for floor(Z/128) chunks; the remaining tail bytes are handled by
// a scalar fallback. For Z=384 (= 3×128) there is no tail — pure HVX path.
// Suppress OAI framework headers (time_meas.h etc.) inside nrLDPC_types.h
#define CODEGEN 1
@@ -38,6 +34,10 @@
#include <stdlib.h>
#include "HAP_farf.h"
#include "HAP_power.h"
#include <hexagon_types.h>
#include <hvx_hexagon_protos.h>
#define HVX_VLEN 128 // HVX vector width in bytes (-mhvx-length=128B)
// Portable LDPC decoder headers — no SIMD, no OAI framework dependencies
#include "nrLDPCdecoder_defs.h"
@@ -88,41 +88,104 @@ static const int8_t numBN_BG1[NR_LDPC_NUM_CN_GROUPS_BG1] = {3, 4, 5, 6, 7, 8, 9,
static const int8_t numBN_BG2[NR_LDPC_NUM_CN_GROUPS_BG2] = {3, 4, 5, 6, 8, 10};
// =============================================================================
// CN Processing — scalar min-sum
// CN Processing — HVX-vectorised min-sum
// =============================================================================
// For every (CN i, lifting symbol z) in a group:
// Pass 1 over all BN positions k: track first/second minimum |v|, XOR signs.
// Pass 2: write extrinsic message for each position j:
// mag_j = min1 if j != min1_k, else min2
// sign_j = XOR of all signs excluding j (= sgn_all ^ v_j)
// mag_j = (|v_j| == min1) ? min2 : min1 (excludes j's contribution)
// sign_j = sgn_all XOR v_j (removes j's sign from product)
// result = sign_j < 0 ? -mag_j : mag_j
//
// The Z lifting symbols are processed in parallel with 128B HVX vectors.
// floor(Z/128) full vectors are handled by HVX; the residual tail (Z % 128)
// falls back to scalar. For Z = 384 = 3×128 there is no tail.
//
// Min-sum tie handling: the "mag = (|v_j|==min1)?min2:min1" rule approximates
// the correct "exclude-self" minimum when only one BN achieves min1. For ties
// (multiple BNs with equal minimum) both positions return min2 rather than min1;
// this is a standard approximation used in all vectorised implementations and
// has negligible effect on convergence.
static void cnProc_group(
const int8_t *cnProcBuf, int8_t *cnProcBufRes,
uint32_t startAddr, int numBN, int numCN, uint16_t Z)
{
uint32_t bitOff = (uint32_t)numCN * NR_LDPC_ZMAX;
int32_t bitOff = (int32_t)numCN * NR_LDPC_ZMAX;
int32_t full = (Z / HVX_VLEN) * HVX_VLEN; // bytes covered by full HVX vectors
int32_t tail = Z - full;
// Constants: initialise once, reuse across all (i, v) iterations.
HVX_Vector vzero = Q6_V_vzero();
HVX_Vector vmax_ub = Q6_Vb_vsplat_R(0x7f); // 127 in every byte lane
for (int i = 0; i < numCN; i++) {
for (int z = 0; z < (int)Z; z++) {
int samp = i * (int)Z + z;
int32_t row = i * Z; // byte offset of this CN's Z samples within each k-layer
// ── HVX path: one 128-byte vector per iteration ──────────────────────
for (int32_t voff = 0; voff < full; voff += HVX_VLEN) {
// Pass 1: accumulate min1, min2, sgn_all across all numBN messages.
HVX_Vector vmin1 = vmax_ub;
HVX_Vector vmin2 = vmax_ub;
HVX_Vector vsgn_all = vzero;
for (int k = 0; k < numBN; k++) {
HVX_Vector vk = *(HVX_UVector *)(cnProcBuf + startAddr
+ (int32_t)k * bitOff + row + voff);
HVX_Vector vabs = Q6_Vb_vabs_Vb(vk); // saturates: |-128| → 127
vsgn_all = Q6_V_vxor_VV(vsgn_all, vk);
// Update min1 and min2:
// where vabs < vmin1: demote old vmin1 to min2 candidate, update min1
// where vabs >= vmin1: vabs itself is min2 candidate
HVX_VectorPred new_min = Q6_Q_vcmp_gt_VubVub(vmin1, vabs);
HVX_Vector old_min1 = vmin1;
vmin1 = Q6_Vub_vmin_VubVub(vmin1, vabs);
HVX_Vector min2_cand = Q6_V_vmux_QVV(new_min, old_min1, vabs);
vmin2 = Q6_Vub_vmin_VubVub(vmin2, min2_cand);
}
// Pass 2: compute and store extrinsic for each BN position.
for (int k = 0; k < numBN; k++) {
const int8_t *in = cnProcBuf + startAddr + (int32_t)k * bitOff + row + voff;
int8_t *out = cnProcBufRes + startAddr + (int32_t)k * bitOff + row + voff;
HVX_Vector vk = *(HVX_UVector *)in;
HVX_Vector vabs = Q6_Vb_vabs_Vb(vk);
// mag: use vmin2 where this position held the overall minimum,
// vmin1 everywhere else.
HVX_VectorPred is_min1 = Q6_Q_vcmp_eq_VbVb(vabs, vmin1);
HVX_Vector vmag = Q6_V_vmux_QVV(is_min1, vmin2, vmin1);
// sign_j = sgn_all XOR vk (removes vk's sign from the product)
HVX_Vector vsign_j = Q6_V_vxor_VV(vsgn_all, vk);
// result = (sign_j < 0) ? -vmag : +vmag
HVX_VectorPred is_neg = Q6_Q_vcmp_gt_VbVb(vzero, vsign_j);
HVX_Vector vneg = Q6_Vb_vsub_VbVb(vzero, vmag);
*(HVX_UVector *)out = Q6_V_vmux_QVV(is_neg, vneg, vmag);
}
}
// ── Scalar fallback: residual tail bytes (Z % 128 != 0) ─────────────
for (int32_t z = full; z < Z; z++) {
int32_t samp = row + z;
int8_t sgn_all = 0;
uint8_t min1 = 127, min2 = 127;
int min1_k = 0;
for (int k = 0; k < numBN; k++) {
int8_t v = cnProcBuf[startAddr + (uint32_t)k * bitOff + samp];
sgn_all ^= v;
int8_t v = cnProcBuf[startAddr + (uint32_t)k * bitOff + samp];
sgn_all ^= v;
uint8_t av = abs8(v);
if (av <= min1) { min2 = min1; min1 = av; min1_k = k; }
if (av < min1) { min2 = min1; min1 = av; min1_k = k; }
else if (av < min2) { min2 = av; }
}
for (int j = 0; j < numBN; j++) {
int8_t v_j = cnProcBuf[startAddr + (uint32_t)j * bitOff + samp];
int8_t sgn_j = sgn_all ^ v_j; // product of signs excluding j
uint8_t mag = (j == min1_k) ? min2 : min1;
int8_t v_j = cnProcBuf[startAddr + (uint32_t)j * bitOff + samp];
int8_t sgn_j = sgn_all ^ v_j;
uint8_t mag = (j == min1_k) ? min2 : min1;
cnProcBufRes[startAddr + (uint32_t)j * bitOff + samp] =
(sgn_j & 0x80) ? -(int8_t)mag : (int8_t)mag;
}
@@ -130,9 +193,9 @@ static void cnProc_group(
}
}
static void scalar_cnProc(t_nrLDPC_lut *p_lut,
const int8_t *cnProcBuf, int8_t *cnProcBufRes,
uint16_t Z, uint8_t BG)
static void cnProc(t_nrLDPC_lut *p_lut,
const int8_t *cnProcBuf, int8_t *cnProcBufRes,
uint16_t Z, uint8_t BG)
{
const int8_t *tbl = (BG == 1) ? numBN_BG1 : numBN_BG2;
int ngrp = (BG == 1) ? NR_LDPC_NUM_CN_GROUPS_BG1
@@ -294,11 +357,6 @@ static int32_t ldpc_scalar_core(
t_nrLDPC_lut *p_lut, uint8_t BG, uint16_t Z, uint8_t numMaxIter,
ldpc_dsp_ctx_t *ctx)
{
// Use pre-allocated buffers from the session context.
// cnProcBuf and bnProcBuf must be zeroed before each decode (the BP
// algorithm initialises CN messages to channel LLRs by scattering into
// them, but the zero-initialised regions between active symbols matter).
// llrProcBuf and llrRes are fully written before first read — no zeroing.
int8_t *cnProcBuf = ctx->cnProcBuf;
int8_t *cnProcBufRes = ctx->cnProcBufRes;
int8_t *bnProcBuf = ctx->bnProcBuf;
@@ -312,16 +370,14 @@ static int32_t ldpc_scalar_core(
memset(bnProcBuf, 0, NR_LDPC_SIZE_BN_PROC_BUF);
memset(bnProcBufRes, 0, NR_LDPC_SIZE_BN_PROC_BUF);
// Scatter input LLRs into the two processing buffers (llrProcBuf and cnProcBuf).
nrLDPC_llr2llrProcBuf(p_lut, (int8_t *)p_llr, llrProcBuf, Z, BG);
if (BG == 1)
nrLDPC_llr2CnProcBuf_BG1(p_lut, (int8_t *)p_llr, cnProcBuf, Z);
else
nrLDPC_llr2CnProcBuf_BG2(p_lut, (int8_t *)p_llr, cnProcBuf, Z);
// First iteration without parity check: the two punctured BN columns are
// not yet estimable from a single pass, so convergence cannot be declared.
scalar_cnProc(p_lut, cnProcBuf, cnProcBufRes, Z, BG);
// First iteration without parity check
cnProc(p_lut, cnProcBuf, cnProcBufRes, Z, BG);
if (BG == 1)
nrLDPC_cn2bnProcBuf_BG1(p_lut, cnProcBufRes, bnProcBuf, Z);
@@ -339,7 +395,7 @@ static int32_t ldpc_scalar_core(
// BP iteration loop with parity check
int32_t pcRes = 1;
while (numIter < (int32_t)numMaxIter && pcRes != 0) {
scalar_cnProc(p_lut, cnProcBuf, cnProcBufRes, Z, BG);
cnProc(p_lut, cnProcBuf, cnProcBufRes, Z, BG);
if (BG == 1)
nrLDPC_cn2bnProcBuf_BG1(p_lut, cnProcBufRes, bnProcBuf, Z);
@@ -358,9 +414,7 @@ static int32_t ldpc_scalar_core(
numIter++;
}
// Gather posterior LLRs from llrRes into the external output order.
nrLDPC_llrRes2llrOut(p_lut, (int8_t *)llr_out, llrRes, Z, BG);
return numIter;
}
@@ -495,17 +549,12 @@ int ldpc_hexagon_decode(remote_handle64 handle,
(const int8_t *)llr, llr_out, numLLR,
&lut, p.BG, p.Z, p.numMaxIter, ctx);
FARF(RUNTIME_HIGH, "ldpc_hexagon: out[0..3]=%d %d %d %d out[2Z..2Z+3]=%d %d %d %d",
(int)((int8_t *)llr_out)[0], (int)((int8_t *)llr_out)[1],
(int)((int8_t *)llr_out)[2], (int)((int8_t *)llr_out)[3],
(int)((int8_t *)llr_out)[2*p.Z], (int)((int8_t *)llr_out)[2*p.Z+1],
(int)((int8_t *)llr_out)[2*p.Z+2], (int)((int8_t *)llr_out)[2*p.Z+3]);
FARF(RUNTIME_HIGH, "ldpc_hexagon: done in %d iter", numIter);
if (metaLen >= 4) {
uint32_t n = (uint32_t)numIter;
memcpy(meta, &n, 4);
}
FARF(RUNTIME_HIGH, "ldpc_hexagon: done in %d iter", numIter);
return 0;
}