Testing M2M and Consumer RSP beyond the specifications: General debug and Conclusion [PART 5]
Welcome to the final instalment of this series of blogs on testing M2M and Consumer RSP entities beyond the specifications. In the past 4 weeks we have detailed how we defined the scope of entities that are ‘in-scope’ and those that are ‘out of scope’ of the testing in the M2M and Consumer RSP domains. We also specifically looked at testing the entities defined as being ‘in-scope’; the eUICC, device and Subscription Manager. Now in this final blog in the series I will cover general debugging issues related to the eUICC, device and Subscription Manager before we reflect and analyse the findings in the final conclusion.
Let’s concentrate on the task of debugging any of the potential issues or problems that might arise during testing any of the entities that we previously defined as being ‘in-scope’.
The debug of most issues and problems frequently relies on the ability to spy on the active M2M or Consumer RSP interfaces. But the biggest issue with eSIM technologies is how to physically spy on M2M and Consumer RSP interface traffic?
Subscription Manager or server side spying is problematic as Subscription Manager vendors will be reluctant to allow third party tools into their data centres. Then on the device side, M2M and Consumer devices may have embedded eUICCs which makes spying on interface traffic difficult as physically attaching a spy tool probe is problematic.
Plus even if you get past all the physical impracticalities, even though most of the authentication process traffic can by spied on, all profile content interface traffic is encrypted, therefore you need the PSKs for M2M eUICCs or the associated Consumer RSP secret key and certificates.
UL Mobile Spy – solving the problems
To solve the problem of how to debug any issues that are uncovered during testing of the Device, eUICC and also potentially the Subscription Managers, UL provide the UL Mobile Spy tool. It translates and visualizes communication between devices or handsets and UICCs or eUICCs on the ISO 7816 interface and also any contactless or NFC traffic on the ISO14443 and SWP interface between the UICC, CLF and contactless readers.
Figure 1: UL's Spying Solution
The tool also supports the powerful feature of dynamically decrypting Secure Channel Protocols; SCP02, SCP03, SCP03t, SCP80 & SCP81 as long as it has the PSKs for M2M eUICCs or the Consumer RSP secret key and certificates.
The spy tool can then be used in conjunction with any of the UL test suites or test tools to assist in the debugging of any issues that the test tool might find with the entity under test.
Over the past month we have learned that to guarantee interoperability and scalability all components and interfaces in the M2M and Consumer RSP domains need to be fully tested & certified. Let’s now consider each entity under test in turn:
The eUICC has a fairly robust M2M and soon to be Consumer RSP certification process for functional, profile, logical security and physical security. Currently there isn’t any pressing need to functionally test beyond the specifications if the EUM can at least demonstrate that they have performed this testing, but an interesting addition might be to test the BIP performance of the eUICC as part of the M2M and Consumer functional testing. Plus the GSMA and their MNO members could really improve the certification process if they mandated the eUICC functional and profile testing.
When it comes to Logical Security the EUMs are rejecting the Common Criteria audits that the GSMA have and are trying to put in place, so it’s up to GSMA to resolve this as soon as possible as any security breaches will be quite harmful. Therefore UL recommends some additional care be taken during the logical security testing that many of the EUMs are delivering as a self-assessment, especially since they may try to leverage and demonstrate equivalent security via other industry audits that are not explicitly aimed at the M2M or Consumer RSP architecture and ecosystem. But at least the physical security offered by GSMA SAS-UP has been adopted by the EUMs worldwide and offers more than adequate safeguards.
Device testing is only applicable for Consumer RSP devices. GCF and PTCRB have always maintained a fairly strong position in the world of the MNOs and MVNOs, therefore the OEMs have had to wholeheartedly embrace the certification processes of these two validation bodies to ensure that they can offer their handsets to their MNO and MVNO customers. So it would initially seem then that device testing is a pretty healthy state, but as it only uses Wi-Fi there is a real need to test all devices OTA, albeit not repeating the full range of tests performed in SGP.23, but at least performing a subset of them from a confidence perspective. Plus an addition approach might be to test the BIP performance of the M2M device modem during the SGP.11 functional testing to determine if there are any scenarios that might leave the Service Provider using the device without connectivity for any unacceptable periods of time. It may also be possible to perform a similar type of testing on the Consumer devices on the ES10X interfaces during the SGP.23 functional testing.
Then for devices with removable eUICCs it is viewed as necessity to perform additional interoperability testing to future proof against possible interoperability issues. But this testing can never be fully comprehensive resulting in unknown or unexpected behaviours.
It is also recommended that MNOs and MVNOs perform Consumer RSP device user acceptance testing via the ESeu interface to ensure that the desired level of user experience is achieved if not consistently across all devices and at the same time protect the MNO brand.
Subscription Manager Testing
As there is no designated certification body for the Subscription Manager certification process for M2M and Consumer RSP it relies on self-certification by the Subscription Managers themselves, which inherently is not a reliable approach. Subscription Managers vendors should be able to demonstrate that they have performed functional testing defined in SGP.11 for M2M and SGP.23 for Consumer against their products. However, UL believes that GSMA could mandate in SGP.24 for Consumer RSP or the equivalent SGP.24 process document from M2M, the use of a third party test tool which would surely help to improve the process. Subscription Managers also have many optional features that they may or may not support and it is expected that many Subscription Manager vendors will choose to test only the features that they need for their given customers. Obviously it isn’t reasonable to ask all Subscription Manager to support all the optional features so it is advised that any customers of the Subscription Manager vendors pay special attention to which optional features have been tested by the Subscription Manager to ensure that they get the feature set that they as a customer are expecting. In addition it is recommended to test the Subscription Manager beyond the SGP.11 and SGP.23 test specifications to help identify and resolve any possible technical interoperability issues such as issues recently resolved in the SM-SR Change Process for M2M or potential ES2+ issues caused by that interface being out of scope in SGP.23 for Consumer respectively.
Debugging of any issues uncovered during testing may need to take place with the use of a spy tool, but physical access might prove difficult, plus you need to get the eUICCs secret keys to do so – which may prove to be difficult for test engineers to acquire. But if you do manage to get physical access and have access to the necessary keys and certificates using a powerful spy tool is a very powerful tool to help you debug all of the eUICC-device interfaces. The ability to compliment this with a network spy tool on the Subscription Manager side would have delivered excellent E2E debugging capabilities but access to the data centre where the Subscription manager is located and also agreeing on a point of access in their system is expected to be blocked.
It becomes clear then that the world of M2M and Consumer RSP is still quite immature and a bit of additional testing here and there will go a long way to protecting your investment in these new technologies, for any player in the ecosystem. Plus if you get the right tools and even a spy tool to help you do the job in hand then it won’t be as an expensive an overhead to do the extra testing as you’d initially feared.
Written by Iain Maxwell.