feat: Enable QASYMM8_SIGNED to floating-point assembly dequantization

[morgolock]

Enable QASYMM8_SIGNED input and weights to use the F32 DequantizeFloat assembly output stage, including the direct convolution selection path.

Propagate input and weight zero-points plus the unrounded mathematical K depth so asymmetric offset correction uses the real GEMM/convolution depth rather than arm_gemm's padded internal K.

Add NEON validation coverage for the direct I8S8F32 convolution path.

Performance was checked on a A76, pinned to CPU 4 with one thread. The change is neutral on large workloads and improves the small NHWC signed int8 to F32 convolution path: QASYMM8_SIGNED/RunSmallDequantizeF32 NHWC/no-activation cases show a 1.36x geomean speedup over github/main, with the tiniest single-batch cases improving by roughly 1.45x to 3.13x.

Change-Id: Ie723d3da629d48de6de737c425bf7ad48e0f7feb

[Passavee-Losripat]

Hi, I've been testing this PR for OpenVINO's ARM CPU plugin int8 enablement and found a possible validation gap in the new QASYMM8_SIGNED->F32 dequantization path.

QSYMM8_PER_CHANNEL weights are correctly rejected by validate() on this path. However, QASYMM8_SIGNED weights whose QuantizationInfo carries a per-channel scale vector are accepted and the kernel then dequantizes every output channel with scale[0] (the scalar uniform().scale product in create_arm_gemm_dequant).

I create minimal repo to test this: all-ones i8 input, all-ones 3×3×3 i8 weights, bias 0, distinct per-channel scales s[c] -> each interior output should be 27·s[c]:

channel    got      expected(27*s[c])    ratio
c00      0.0270        0.0270            1.0000
c01      0.0270        0.0540            0.5000
c02      0.0270        0.1080            0.2500
...
c11      0.0270       55.2960            0.0005

validate() returns OK, but every channel is computed with s[0]. Since quantization frameworks (NNCF, PyTorch, TF) produce per-channel weight scales by default, this input shape is what integrations like OpenVINO naturally pass, so it fails silently on real models rather than erroring out.

I am thinking of rejecting scale().size() > 1 on this path the same way QSYMM8_PER_CHANNEL is rejected, unless per-channel support is planned. I can share the full standalone test program if useful.

[morgolock]

Hi, I've been testing this PR for OpenVINO's ARM CPU plugin int8 enablement and found a possible validation gap in the new QASYMM8_SIGNED->F32 dequantization path.

QSYMM8_PER_CHANNEL weights are correctly rejected by validate() on this path. However, QASYMM8_SIGNED weights whose QuantizationInfo carries a per-channel scale vector are accepted and the kernel then dequantizes every output channel with scale[0] (the scalar uniform().scale product in create_arm_gemm_dequant).

I create minimal repo to test this: all-ones i8 input, all-ones 3×3×3 i8 weights, bias 0, distinct per-channel scales s[c] -> each interior output should be 27·s[c]:

channel    got      expected(27*s[c])    ratio
c00      0.0270        0.0270            1.0000
c01      0.0270        0.0540            0.5000
c02      0.0270        0.1080            0.2500
...
c11      0.0270       55.2960            0.0005

validate() returns OK, but every channel is computed with s[0]. Since quantization frameworks (NNCF, PyTorch, TF) produce per-channel weight scales by default, this input shape is what integrations like OpenVINO naturally pass, so it fails silently on real models rather than erroring out.

I am thinking of rejecting scale().size() > 1 on this path the same way QSYMM8_PER_CHANNEL is rejected, unless per-channel support is planned. I can share the full standalone test program if useful.

Hi @Passavee-Losripat

Good catch. I added validation to reject QASYMM8_SIGNED weights with multiple scales on this dequantized path, since the current output stage only supports a single scalar scale.

Please try the latest patchset and let us know if this works for you.

Hope this helps

[gunes-arm]

Quite confused here. When we have per channel quantized weights, the data type should be provided as QSYMM8_PER_CHANNEL. This should be the correct usage assumption throughout the library and how we validate against. We may need to think about trying to prevent this misuse in another way.

@Passavee-Losripat would you mind sharing your reproducible? And, do we have any workloads/integration points that OpenVino relies on ACL accepting a per-channel quantized weights tensor with QASYMM8_SIGNED/QASYMM8? Because, to me, we might prevent some TensorInfo being constructed this way, as it's not the correct usage of the types, but that's going to cause crashes instead of silently validating inside a function/operator.

[Passavee-Losripat]

Hi, @gunes-arm and @morgolock. I attached acl_per_channel_dequant_repro.cpp here as a self-contained repro: all-ones i8 input, all-ones 3*3*3 i8 weights, zero bias, distinct per-channel scales, so each interior output must equal 27*s[c] with no reference implementation needed. I re-ran it against the latest patchset and the per-channel vector case now fails validate() (raised from CpuConv2d::validate), and the per-tensor scalar case still validates and computes bit-exactly, so the added check resolves the issue I reported without over-rejecting.

On your question about integration points, OpenVINO relies on ACL accepting QASYMM8_SIGNED weights with a per-channel scale vector today, in two places:

1. The int8 FullyConnected path that is already merged in OpenVINO master (src/plugins/intel_cpu/src/nodes/executors/acl/acl_lowp_fullyconnected.cpp) constructs exactly this shape: QASYMM8_SIGNED weights with QuantizationInfo(dequantizationScales) where the scale vector is per-output-channel.
2. The int8 Convolution enablement in progress (which this PR would serve) follows the same pattern.

On the dtype question, OpenVINO's quantization pipeline produces symmetric per-channel i8 weights (zero-point 0), which semantically maps to QSYMM8_PER_CHANNEL but we got two reasons why we currently used QASYMM8_SIGNED + vector instead:

• The scale vector OpenVINO attaches is not strictly the weights' own quantization scale but it's the combined dequantization scale (src_scale * weight_scale, per output channel, folded together by our graph-level transformations), with the src tensor's QuantizationInfo set to trivial (scale=1) to compensate. It's effective scale attached to the weights tensor rather than a literal property of the weights data.
• On the standard requantize path, this already works correctly: quantization::compute_quantized_multipliers_and_shifts (src/core/utils/quantization/AsymmHelpers.cpp) sizes its multiplier array from weights->quantization_info().scale().size() and iterates every entry, with no dtype check at all. So the library's own output-stage math already reads and applies every entry in the scale vector, regardless of whether the tag is QSYMM8_PER_CHANNEL or QASYMM8_SIGNED. The FC integration above already relies on exactly this behavior, merged in OpenVINO master today.

Because of that, rejecting this combination when constructing TensorInfo could break an already-working OpenVINO integration on the standard requantization path, where the per-channel vector is handled correctly. I think new rejection at validate() looks quite right to me. It would reject only the output stage that cannot actually consume the vector, while allowing the existing supported path to continue working. It also turns the current silent incorrect result into a clear validation failure at the point where the unsupported combination is used.

Happy to help test further if useful.

[gunes-arm]

Hi @Passavee-Losripat, I agree that breaking what's already working is undesirable and we can keep the validation checks introduced in this PR because they are not harmful to current functionality in any sense.

But, for the per-channel quantized tensors, we expect the data type to be QSYMM8_PER_CHANNEL. Although this coupling is not ideal, for us it's too late to change this :) So, using QASYMM8_SIGNED or any other type with a quantization scale vector of length > 1 is an API misuse we couldn't prevent.

Many different places in the library rely on this type. We check is_data_type_quantized_per_channel in both Cpu and Gpu backends, including matrix multiplicaton, convolution and depthwise routines. So, I expect we can find more cases where validation doesn't fail but functional correctness does if the types aren't used properly. Also, having these validations in only some places is another validation gap for us as it seems like we are accepting length > 1 quantization info with QASYMM8_SIGNED in other operators.

As I said, I think the current set of additional checks can be added, but I recommend using QSYMM8_PER_CHANNEL where applicable. Could you check if this is achievable for a future release so that we can add guards in TensorInfo as well?

[gunes-arm]

Hi @Passavee-Losripat, any thoughts on the timeline for fixing the data type usage for per-channel quantized models in OpenVino?

[Passavee-Losripat]

Hi @gunes-arm, sorry for the slow reply. I wanted to check with the team first.

We’re planning to move the per-channel weight tensors to QSYMM8_PER_CHANNEL in both places where we currently use QASYMM8_SIGNED with a multi-value scale vector. I’ll be leading this work and plan to start in the first week of September. I also tested this PR's latest patchset against our downstream OpenVINO usage (our ARM CPU plugin's Conv+FQ test suite with int8-quantized model inference) and confirmed it doesn't cause any new problems for us today.

Thanks for keeping the validation guards in this PR as a temporary safeguard. We’re also fine with the TensorInfo-level ticket whenever you’re ready to open it. Once the changes are merged, I’d be happy to help test it against both paths.

[gunes-arm]

There is a bit of a weird situation here.

I'd remove this check because that's what the switch/case is for and remove the S8 case from the switch. Would we breaking anything? I'm not too sure. Only if there is some misuse of data types. Same situation goes with comments and U8 type.

So, I'd suggest

• Ignore S8 was in the case statement above and just remove a->data_type() == DataType::QASYMM8_SIGNED from the ifs here as it's already checked.
• Create a ticket to resolve the S8/U8 confusion with tests involved.

[morgolock]

Thanks, agreed. I’ll keep this PR scoped to the QASYMM8_SIGNED -> F32 path, remove the redundant a->data_type() check inside the QASYMM8_SIGNED case, and avoid changing the S8/U8 handling here. I’ll create a follow-up MLCE/Jira ticket for the broader S8/U8 vs QASYMM8/QASYMM8_SIGNED validation cleanup with dedicated tests.

[gunes-arm]

I think we inadvertently added S8 handling as well. Could you remove that?

[gunes-arm]

Can you remove the added logic for case DataType::S8 in both validate & configure? I think it sends the wrong message as if we support S8, but we shouldn't. Since they were there before, I'd leave the case statement as is, but I wouldn't add any logic below that case statement and create a ticket to remove S8 and U8 from this switch case. Especially, if we're not breaking anything, we could just do that in a single PR. If we're breaking anything, we can deprecate and say this is not the right type to use.

[morgolock]

Done. I kept the existing S8 case labels untouched, but removed the new DequantizeFloat/F32/F16 logic that had been added under the S8 path. This PR now stays scoped to QASYMM8_SIGNED input/weights with F32/F16 output. The broader S8/U8 vs QASYMM8/QASYMM8_SIGNED cleanup should be handled separately with dedicated tests.

[gunes-arm]

But, you've added a->data_type() == DataType::QASYMM8_SIGNED checks back 🤔 See my initial message in the thread.

[gunes-arm]

Can you make the commit & PR title start with uppercase letter? Thanks :)

[gunes-arm]

I think we inadvertently added S8 handling as well. Could you remove that?

[gunes-arm]

Hi @Passavee-Losripat, I agree that breaking what's already working is undesirable and we can keep the validation checks introduced in this PR because they are not harmful to current functionality in any sense.

But, for the per-channel quantized tensors, we expect the data type to be QSYMM8_PER_CHANNEL. Although this coupling is not ideal, for us it's too late to change this :) So, using QASYMM8_SIGNED or any other type with a quantization scale vector of length > 1 is an API misuse we couldn't prevent.

Many different places in the library rely on this type. We check is_data_type_quantized_per_channel in both Cpu and Gpu backends, including matrix multiplicaton, convolution and depthwise routines. So, I expect we can find more cases where validation doesn't fail but functional correctness does if the types aren't used properly. Also, having these validations in only some places is another validation gap for us as it seems like we are accepting length > 1 quantization info with QASYMM8_SIGNED in other operators.

As I said, I think the current set of additional checks can be added, but I recommend using QSYMM8_PER_CHANNEL where applicable. Could you check if this is achievable for a future release so that we can add guards in TensorInfo as well?

[morgolock]

Can you make the commit & PR title start with uppercase letter? Thanks :)

Updated the commit title to start with an uppercase letter. I will update the PR title to match.

[Passavee-Losripat]

I tested this PR against our downstream OpenVINO usage (ARM CPU plugin's Conv+FQ subgraph test suite and an int8-quantized YOLO model). There're no errors so far.