Testing M2M and Consumer RSP beyond the Specifications: The Device [PART 3]
Written by Iain Maxwell.
Welcome back again to this blog on testing M2M and Consumer RSP entities beyond the specifications. Last week we looked testing the eUICC for the M2M and Consumer RSP domains. Therefore, this week I will concentrate on testing the device mainly in the Consumer RSP domain.
Let’s now concentrate on the device looking at what is tested and any issues and problems associated with testing it, to determine if we need to test beyond the specifications, and also how to do this in practice. From a M2M and Consumer perspective there is a lot less testing required as compared to the eUICC, and this table summarizes the scope of the M2M and Consumer RSP device testing where in fact nothing required for M2M from an embedded SIM technology perspective and only functional testing is required for Consumer RSP.
Table 1: Consumer RSP Device Testing Overview
Functional M2M device testing is not required as the device has no additional specific eUICC functionality therefore it is only tested in accordance to the 3GPP and ETSI test specifications.
Consumer RSP devices on the other hand does have specific eUICC support, therefore functional device testing is delivered by GCF and PTCRB, following the SGP.23 test specification. Both GCF and PTCRB have activated the device test cases and since then the GCF 110 day rule has expired but PTCRB devices are still under the 9 month rule. This means that the OEMs have approximately 3 months left to get their SGP.22 v2.X devices validated for PTCRB as after the 9 months are up, SGP.22 v2.1 and v2.2 devices will be mandated to require conformance to SGP.23 v1.2. For GCF all SGP.22 v2.1 and v2.2 devices are now mandated to require conformance to SGP.23 v1.2. It should be noted that similar to M2M, all Consumer Devices in addition to SGP.23 are also tested in accordance to 3GPP and ETSI test specifications.
Device Certification is required by GSMA for devices to enter the Consumer RSP ecosystem based only on the functional testing so it is much simpler than the eUICC certification process.
Figure 1: Consumer RSP Device Certification Overview
This is applicable to only Consumer RSP devices. So only functional testing is performed, then the OEM submits all the results of the testing including the certifications acquired for the GCF or the PTCRB, and the GSMA then performs an evidence check to analyse the validity of the results. If the evidence check is successful, the OEM can declare that their product is fully compliant with the GSMA certification process for Consumer RSP devices.
As mentioned previously M2M and Consumer RSP devices are also tested in accordance to 3GPP and ETSI test specifications.
Device Testing: Issues and problems
Issues and problems related to device testing are more generic by nature, often related to decisions made by the RSP-TEST group when writing the SGP.23 test specification. Consumer RSP devices are tested via Wi-Fi in accordance to SGP.23, but most devices spend an equal amount of time connected OTA to MNO networks, therefore many MNOs are already testing the device and LPAd beyond SGP.23 to ensure that profiles can be downloaded OTA. The approach used is very similar to the GCF and PTCRB field trial testing.
The end user experience is also an area that needs to be considered. Although the LPAd is tested in SGP.23, different LPAd implementations have to interact with the end user in Consumer via the ESeu interface and UL are finding that some of the OEMs deliver a wide range of end user experiences.
Devices with embedded eUICCs result in the ES10X interfaces not being fully tested with an eUICC simulation, so these interfaces are only implicitly tested – but this may not be the limitation that it initially seems as it means that in advance the OEM has to do their ‘homework’ and ensure that the eUICC that they are using is fully compliant with SGP.23 and also their device. Therefore UL feels that there is no explicit need to test the ES10X interfaces and you can effectively treat the device and eUICC combination as a ‘black-box’ in that regard.
But a major problem here is for devices that have removable eUICCs, as they may cause future interoperability issues. If the device is qualified for SGP.22 v2.1 but in the future a customer inserts an old eUICC compliant to v2.0 or a new eUICC compliant to, say for example v3.0, then potential issues might occur. This same issue is also applicable to M2M.
Device Test Tools – solving the problems
To solve these problems and in general perform the required testing, UL provides ‘UL SGP.23 Device Test Suite’ offering GCF and PTCRB functional compliance testing against SGP.23.
Figure 2: UL Consumer Device Test Suite
The pink area in the tool is the part of the architecture that the tool is simulating in order to perform the testing. So for Consumer RSP the ES8+, ES9 and ES11 interfaces are explicitly tested and the ES10X interfaces are implicitly tested. Then ES End User interface (ESeu) is not actually required for testing as it is just the End User interacting with the LPAd.
Using this official GCF and PTCRB compliance test suites UL also offers official device qualification services in our GCF and PTCRB qualified labs. Note that all testing is currently performed via Wi-Fi but in the future UL hope to offer testing OTA as a service, albeit not for GCF and PTCRB validation.
UL are also building up our range of RF testing services and via our business partners hope to soon offer M2M RF testing for NB-IoT and LTE-M devices.
GCF and PTCRB have always maintained a fairly strong position in the world of the MNOs and MVNOs, therefore the OEMs have had to wholeheartedly embrace the certification processes of these two validation bodies to ensure that they can offer their handsets to their MNO and MVNO customers. So it would initially seem then that device testing is a pretty healthy state.
But since Device validation only requires the use of Wi-Fi there is a real need to test all devices OTA, albeit not repeating the full range of tests performed in SGP.23, but at least performing a subset of the tests from a confidence perspective. In addition, performing extra interoperability testing on devices with removable eUICCs is necessity to future proof against possible interoperability issues.
For Consumer devices it is also suggested that MNOs and MVNOs perform user acceptance testing via the ESeu interface to ensure that the desired level of user experience is achieved if not consistently across all devices. This will deliver a consistent user experience and at the same time protect the MNO brand.
For M2M devices an optional addition might be to test the BIP performance of the device modem during the SGP.11 functional testing to determine if there are any scenarios that might leave the Service Provider using the device without connectivity for any unacceptable periods of time. It might also be possible to perform a similar type of testing on the Consumer devices on the ES10X interfaces during the SGP.23 functional testing to determine if there are any scenarios that might leave the End User using the device without connectivity for any unacceptable periods of time.
Next week we will focus on all the testing issues related to the Subscription Manager, also known as the server. So please join me then for another deep dive analysis.