LinuxDig.com Request For Comments

RFC Number : 2967 Title : TISDAG - Technical Infrastructure for Swedish Directory Access Gateways. Network Working Group L. Daigle Request for Comments: 2967 Thinking Cat Enterprises Category: Informational R. Hedberg Catalogix October 2000 TISDAG - Technical Infrastructure for Swedish Directory Access Gateways Status of this Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Copyright Notice Copyright (C) The Internet Society (2000). All Rights Reserved. Abstract The strength of the TISDAG (Technical Infrastructure for Swedish Directory Access Gateways) project's DAG proposal is that it defines the necessary technical infrastructure to provide a single-access- point service for information on Swedish Internet users. The resulting service will provide uniform access for all information -- the same level of access to information (7x24 service), and the same information made available, irrespective of the service provider responsible for maintaining that information, their directory service protocols, or the end-user's client access protocol. Table of Contents 1.0 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1 Project Goal. . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2 Executive Summary of Technical Study Result . . . . . . . . . 5 1.3 Document Overview . . . . . . . . . . . . . . . . . . . . . . 6 1.4 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.0 Requirements. . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1 End-User Requirements . . . . . . . . . . . . . . . . . . . . 8 2.2 WDSPs Requirements. . . . . . . . . . . . . . . . . . . . . . 8 2.3 DAG-System Requirements . . . . . . . . . . . . . . . . . . . 9 3.0 Functional Specification. . . . . . . . . . . . . . . . . . . 9 3.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.2 The DAG Core. . . . . . . . . . . . . . . . . . . . . . . . . 10 3.3 Client Interface. . . . . . . . . . . . . . . . . . . . . . . 11 3.3.1 Acceptable User Input . . . . . . . . . . . . . . . . . . . 12 Daigle & Hedberg Informational [Page 1]  RFC 2967 TISDAG October 2000 Supported Query Types. . . . . . . . . . . . . . . . . . . . . 12 Matching Semantics . . . . . . . . . . . . . . . . . . . . . . 12 Character Sets . . . . . . . . . . . . . . . . . . . . . . . . 13 3.3.2 Data Output Spec. . . . . . . . . . . . . . . . . . . . . . 14 Schema Definition. . . . . . . . . . . . . . . . . . . . . . . 14 Referral Definition. . . . . . . . . . . . . . . . . . . . . . 14 Error conditions . . . . . . . . . . . . . . . . . . . . . . . 14 3.4 Directory Server Interface. . . . . . . . . . . . . . . . . . 14 4.0 Architecture. . . . . . . . . . . . . . . . . . . . . . . . . 15 4.1 Software Components . . . . . . . . . . . . . . . . . . . . . 15 4.1.1 Internal Communications . . . . . . . . . . . . . . . . . . 15 4.1.2 Referral Index. . . . . . . . . . . . . . . . . . . . . . . 15 4.1.3 DAG-CAPs. . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.1.4 DAG-SAPs. . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.2 Important Architectural Notes . . . . . . . . . . . . . . . . 17 4.2.1 2 Distinct Functions: Referrals and Chaining . . . . . . . 17 4.2.2 Limited Query and Response Semantics. . . . . . . . . . . . 17 4.2.3 Visibility. . . . . . . . . . . . . . . . . . . . . . . . . 17 4.2.4 Richness of Query semantics . . . . . . . . . . . . . . . . 18 4.2.5 N+M Protocol Mappings . . . . . . . . . . . . . . . . . . . 18 4.2.6 DAG-CAPs and DAG-SAPs are completely independent of each other. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.2.7 The Role of the DAG-CAP . . . . . . . . . . . . . . . . . . 18 4.2.8 The Role of the DAG-SAP . . . . . . . . . . . . . . . . . . 19 4.2.9 DAG/IP is internal. . . . . . . . . . . . . . . . . . . . . 19 4.2.10 Expectations . . . . . . . . . . . . . . . . . . . . . . . 19 4.2.11 Future Extensions. . . . . . . . . . . . . . . . . . . . . 19 5.0 Software Specifications . . . . . . . . . . . . . . . . . . . 19 5.1 Notational Convention . . . . . . . . . . . . . . . . . . . . 19 5.2 DAG-CAP Basics. . . . . . . . . . . . . . . . . . . . . . . . 20 5.2.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . 20 5.2.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . 21 5.2.3 Error handling. . . . . . . . . . . . . . . . . . . . . . . 21 5.2.4 Pruning of results. . . . . . . . . . . . . . . . . . . . . 22 5.3 DAG-SAP Basics. . . . . . . . . . . . . . . . . . . . . . . . 22 5.3.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . 22 5.3.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . 23 5.3.3 Error handling. . . . . . . . . . . . . . . . . . . . . . . 23 5.3.4 Pruning of results. . . . . . . . . . . . . . . . . . . . . 23 5.3.5 Constraint precedence . . . . . . . . . . . . . . . . . . . 23 5.4 The Referral Index. . . . . . . . . . . . . . . . . . . . . . 24 5.4.1 Architecture. . . . . . . . . . . . . . . . . . . . . . . . 24 5.4.2 Interactions with WDSPs (CIP) . . . . . . . . . . . . . . . 24 5.4.3 Index Object Format . . . . . . . . . . . . . . . . . . . . 24 5.4.4 DAG-Internal I/O. . . . . . . . . . . . . . . . . . . . . . 24 5.4.5 The Index Server. . . . . . . . . . . . . . . . . . . . . . 24 5.4.6 Configuration . . . . . . . . . . . . . . . . . . . . . . . 25 5.4.7 Security. . . . . . . . . . . . . . . . . . . . . . . . . . 25 Daigle & Hedberg Informational [Page 2]  RFC 2967 TISDAG October 2000 5.5 Mail (SMTP) DAG-CAP . . . . . . . . . . . . . . . . . . . . . 25 5.5.1 Mail DAG-CAP Input. . . . . . . . . . . . . . . . . . . . . 26 5.5.2 Translation from Mail query to DAG/IP . . . . . . . . . . . 28 Querying the Referral Index. . . . . . . . . . . . . . . . . . 28 Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 29 5.5.3 Chaining queries in Mail DAG-CAP. . . . . . . . . . . . . . 31 5.5.4 Expression of results in Mail DAG-CAP . . . . . . . . . . . 31 5.5.5 Expression of Errors in Mail DAG-CAP. . . . . . . . . . . . 31 5.6 Web (HTTP) DAG-CAP. . . . . . . . . . . . . . . . . . . . . . 32 5.6.1 Web DAG-CAP Input . . . . . . . . . . . . . . . . . . . . . 32 5.6.2 Translation from Web query to DAG/IP. . . . . . . . . . . . 33 Querying a DAG-SAP Directly. . . . . . . . . . . . . . . . . . 33 Querying the Referral Index. . . . . . . . . . . . . . . . . . 33 Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 35 5.6.3 Chaining queries in Web DAG-CAP . . . . . . . . . . . . . . 36 5.6.4 Expression of results in Web DAG-CAP. . . . . . . . . . . . 36 text/html results. . . . . . . . . . . . . . . . . . . . . . . 36 application/whoispp-response Results . . . . . . . . . . . . . 37 5.6.5 Expression of Errors in Web DAG-CAP . . . . . . . . . . . . 37 Standard Errors. . . . . . . . . . . . . . . . . . . . . . . . 38 5.7 Whois++ DAG-CAP . . . . . . . . . . . . . . . . . . . . . . . 38 5.7.1 Whois++ DAG-CAP Input . . . . . . . . . . . . . . . . . . . 38 5.7.2 Translation from Whois++ query to DAG/IP. . . . . . . . . . 39 Querying the Referral Index. . . . . . . . . . . . . . . . . . 39 Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 39 5.7.3 Chaining in Whois++ DAG-CAP . . . . . . . . . . . . . . . . 40 5.7.4 Expression of results in Whois++. . . . . . . . . . . . . . 41 5.7.5 Expression of Errors in Whois++ DAG-CAP . . . . . . . . . . 41 5.8 LDAPv2 DAG-CAP. . . . . . . . . . . . . . . . . . . . . . . . 42 5.8.1 LDAPv2 DAG-CAP Input. . . . . . . . . . . . . . . . . . . . 42 5.8.2 Translation from LDAPv2 query to DAG/IP . . . . . . . . . . 44 Querying the Referral Index. . . . . . . . . . . . . . . . . . 44 Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 46 5.8.3 Chaining queries in LDAPv2 DAG-CAP. . . . . . . . . . . . . 48 5.8.4 Expression of results in LDAPv2 . . . . . . . . . . . . . . 48 5.8.5 Expression of Errors in LDAPv2 DAG-CAP. . . . . . . . . . . 48 5.9 LDAPv3 DAG-CAP. . . . . . . . . . . . . . . . . . . . . . . . 50 5.9.1 LDAPv3 DAG-CAP Input. . . . . . . . . . . . . . . . . . . . 50 5.9.2 Translation from LDAPv3 query to DAG/IP . . . . . . . . . . 51 Querying the Referral Index. . . . . . . . . . . . . . . . . . 51 Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 54 5.9.3 Chaining queries in LDAPv3 DAG-CAP. . . . . . . . . . . . . 55 5.9.4 Expression of results in LDAPv3 . . . . . . . . . . . . . . 55 5.9.5 Expression of Errors in LDAPv3 DAG-CAP. . . . . . . . . . . 56 5.10 Whois++ DAG-SAP. . . . . . . . . . . . . . . . . . . . . . . 57 5.10.1 Input. . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.10.2 Translation from DAG/IP to Whois++ query . . . . . . . . . 58 5.10.3 Translation of Whois++ results to DAG/IP . . . . . . . . . 58 Daigle & Hedberg Informational [Page 3]  RFC 2967 TISDAG October 2000 5.11 LDAPv2 DAG-SAP . . . . . . . . . . . . . . . . . . . . . . . 59 5.11.1 Input. . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.11.2 Translation from DAG/IP to LDAPv2 query. . . . . . . . . . 59 5.11.3 Translation of LDAPv2 results to DAG/IP. . . . . . . . . . 61 5.12 LDAPv3 DAG-SAP . . . . . . . . . . . . . . . . . . . . . . . 62 5.12.1 Input. . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.12.2 Translation from DAG/IP to LDAPv3 query. . . . . . . . . . 62 5.12.3 Translation of LDAPv3 results to DAG/IP. . . . . . . . . . 64 5.13 Example Queries. . . . . . . . . . . . . . . . . . . . . . . 64 5.13.1 A Whois++ Query. . . . . . . . . . . . . . . . . . . . . . 65 What the Whois++ DAG-CAP Receives. . . . . . . . . . . . . . . 65 What the Whois++ DAG-CAP sends to the Referral Index . . . . . 65 What the Whois++ DAG-CAP Sends to an LDAP DAG-SAP. . . . . . . 65 5.13.2 An LDAP Query. . . . . . . . . . . . . . . . . . . . . . . 66 What the LDAP DAG-CAP Receives . . . . . . . . . . . . . . . . 66 5.13.3 What the LDAP DAG-CAP sends to the Referral Index. . . . . 67 What the LDAP DAG-CAP Sends to a Whois++ DAG-SAP . . . . . . . 67 What the LDAP DAG-CAP Sends to an LDAP DAG-SAP . . . . . . . . 68 6.0 Service Specifications. . . . . . . . . . . . . . . . . . . . 68 6.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . 68 6.2 WDSP Participation. . . . . . . . . . . . . . . . . . . . . . 69 6.3 Load Distribution . . . . . . . . . . . . . . . . . . . . . . 69 6.4 Extensibility . . . . . . . . . . . . . . . . . . . . . . . . 72 7.0 Security. . . . . . . . . . . . . . . . . . . . . . . . . . . 73 7.1 Information credibility . . . . . . . . . . . . . . . . . . . 73 7.2 Unauthorized access . . . . . . . . . . . . . . . . . . . . . 73 8.0 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 74 Appendix A - DAG Schema Definitions . . . . . . . . . . . . . . . 75 A.1 DAG Personal Information Schema (DAGPERSON Schema). . . . . . 76 A.2 DAG Organizational Role Information Schema (DAGORGROLE Schema). . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Appendix B - Schema Mappings for Whois++ and LDAP . . . . . . . . 77 B.1 LDAP and the DAG Schemas. . . . . . . . . . . . . . . . . . . 78 B.2 Whois++ and the DAG Schemas . . . . . . . . . . . . . . . . . 81 Appendix C - DAG-Internal Protocol (DAG/IP) . . . . . . . . . . . 82 C.1 A word on the choice of DAG/IP. . . . . . . . . . . . . . . . 83 C.2 DAG/IP Input and Output -- Overview . . . . . . . . . . . . . 83 C.3 BNF for DAG/IP input and output . . . . . . . . . . . . . . . 83 C.3.1 The DAG/IP Input Grammar. . . . . . . . . . . . . . . . . . 84 C.3.2 The DAG/IP Response Grammar . . . . . . . . . . . . . . . . 87 C.4 DAG/IP Response Messages. . . . . . . . . . . . . . . . . . . 89 Appendix D - DAG/IP Response Messages Mapping . . . . . . . . . . 93 Appendix E - DAG CIP Usage. . . . . . . . . . . . . . . . . . . . 95 E.1 CIP Index Object. . . . . . . . . . . . . . . . . . . . . . . 95 E.2 CIP Index Object Creation . . . . . . . . . . . . . . . . . . 97 E.3 CIP Index Object Sharing. . . . . . . . . . . . . . . . . . . 98 E.3.1 Registration of Servers . . . . . . . . . . . . . . . . . . 98 E.3.2 Transmission of Objects . . . . . . . . . . . . . . . . . .100 Daigle & Hedberg Informational [Page 4]  RFC 2967 TISDAG October 2000 Appendix F - Summary of Technical Survey Results. . . . . . . . .100 Appendix G - Useful References. . . . . . . . . . . . . . . . . .102 Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . .102 Authors' Addresses. . . . . . . . . . . . . . . . . . . . . . . .104 Full Copyright Statement. . . . . . . . . . . . . . . . . . . . .105 List of Tables Table 3.1 DAG-supported queries . . . . . . . . . . . . . . . . .12 Table 5.1 Allowable Whois++ Queries . . . . . . . . . . . . . . .38 Table A.1 DAGPERSON schema attributes . . . . . . . . . . . . . .76 Table A.2 DAGORGROLE schema attributes. . . . . . . . . . . . . .77 Table B.1 Canonical DAGPERSON schema & LDAP inetorgPerson attributes . . . . . . . . . . . . . . . . . . . . . . . . . .79 Table B.2 Reasonable Approximations for LDAP organizationalRole attributes . . . . . . . . . . . . . . . . . . . . . . . . . .79 Table B.3 Canonical mappings for LDAP organizationalRole attributes . . . . . . . . . . . . . . . . . . . . . . . . . .81 Table B.4 Canonical DAGPERSON schema & Whois++ USER attributes. .81 Table B.5 Canonical mappings for Whois++ ORGROLE attributes . . .82 Table C.1 List of system response codes . . . . . . . . . . . . .90 Table D.1 LDAPv2/v3 resultcodes to DAG/IP response codes mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . .93 Table D.2 Mapping from DAG/IP response codes to LDAPv2/v3 resultcodes. . . . . . . . . . . . . . . . . . . . . . . . . .94 Table D.3 Mapping between DAG/IP and Whois++ response codes . . .94 Table F.1 Summary of TISDAG Survey Results: Queries . . . . . . 101 Table F.2 Summary of TISDAG Survey Results: Operational Information. . . . . . . . . . . . . . . . . . . . . . . . . 101 1.0 Introduction 1.1 Project Goal The overarching goal of this project is to develop the necessary technical infrastructure to provide a single-access-point service for searching for whitepages information on Swedish Internet users. The service must be uniform for all information -- the same level of access to information (7x24 service), and the same whitepages information made available, irrespective of the service provider responsible for maintaining that information. 1.2 Executive Summary of Technical Study Result The strength of the TISDAG project's DAG proposal is that it defines the necessary technical infrastructure to provide a single-access- point service for information on Swedish Internet users. The resulting service will provide uniform access for all information -- Daigle & Hedberg Informational [Page 5]  RFC 2967 TISDAG October 2000 the same level of access to information (7x24 service), and the same information made available, irrespective of the service provider responsible for maintaining that information, their directory service protocols, or the end-user's client access protocol. Instead of requiring centralized mirroring of complete information records from Swedish directory service providers, the DAG system uses a well-defined index object summary of that data, updated at the directory service provider's convenience. When an end-user queries the DAG, the referral information is used (by the end-user's software, or by a module within the DAG, as appropriate) to complete the final query directly at the directory service provider's system. This ensures that the end-user gets the most up-to-date complete information, and promotes the directory service provider's main interest: its service. The architecture of the DAG itself is very modular; support for future protocols can be added in the operational system. 1.3 Document Overview This document is broken into 5 major sections: Requirements: As a service, the DAG system will have several different types of users. In order to be successful, those users' needs (requirements) must be met. This in turn defines certain constraints, or system requirements, that must be met. This section aims to capture the baseline requirement assumptions to be addressed by the system, and thus lays the groundwork on which the rest of the proposed system is built. Functional Specification Overview: Working from the users' requirements, specific technologies and functionality details are outlined to architect a system that will meet the stated requirements. This includes a conceptual architecture for the system. While the Requirements section outlines the needs the different users have for the eventual DAG system, implementing and providing the eventual service will entail constraints or conditions that need to be met in order to be able to participate in the overall system. Architecture: Once the system has been defined conceptually, a proposed software architecture is specified to produce the desired functionality and meet the stated requirements. Software Specifications: This section provides the specifications for software components to meet the architecture described above. Daigle & Hedberg Informational [Page 6]  RFC 2967 TISDAG October 2000 Service Specifications: Once the software has been designed, the success of the DAG system will rest on its operational characteristics. Details of service requirements are given in this section. 1.4 Terminology DAG-CAP: Client Access Point -- point of communication between client-access software and the DAG system. DAG-System: The Directory Access Gateway system resulting from the TISDAG project. A collection of infrastructural software and services for the purpose of providing unified access to Swedish whitepages information. DAG/IP: DAG-Internal Protocol -- communication protocol used between software components of the DAG. End-User: People performing White Pages searches and look-ups (via various forms of client software). DAG-SAP: Service Access Point -- point of communication between the DAG and WDSP software. WDSP: Whitepages Directory Service Provider -- ISPs, companies, or other interested entities. Whitepages Information: Collected information coordinates for individual people. This typically includes (but is not limited to) a person's name, and e-mail address. 2.0 Requirements There are 2 primary classes of users for the proposed Whitepages directory access gateway: - End-users - WDSPs As outlined below, needs of each of these user classes imposes a set of constraints on the design of the DAG system itself. Some of the requirements shown below are assumed starting criteria for the DAG service; others have been derived from data collected in the Technical Survey or other expertise input. Daigle & Hedberg Informational [Page 7]  RFC 2967 TISDAG October 2000 2.1 End-User Requirements The End-User is to be provided with a specific set of search types: Name Name + Organization Role + Organization Name + Locality Name + Organization + Locality Role + Organization + Locality The search results will, if available, include the following information for each 'hit': - Full name - E-mail address - Role - Organization - Locality - Full address - Telephone numbers Access to the service must be available through reasonable and current protocols -- such that directory-service-aware software can make use of it seamlessly, and there are no reasonable technological impediments to making this service useful to all Swedish Internet users. Following on that, its responses are expected to be timely; a standard search should not take more time than the average access to a web-server. 2.2 WDSPs Requirements Given that the WDSPs that participate in this service are already in the business of providing a service of whitepages information, they have certain requirements that must be respected in order to make this a successful and useful service to all concerned. The DAG system must provide reasonable assurances of data integrity for WDSPs; the information the End-User sees should correspond directly to that provided by the WDSPs. The DAG system should be non-preferential in providing whitepages information -- the service is to the End-User, and the source of whitepages information should not influence the search and information presentation processes. Daigle & Hedberg Informational [Page 8]  RFC 2967 TISDAG October 2000 The DAG system must be able to reflect information updates within a reasonable time after receipt from WDSPs; on the flip side, while the DAG system will function best with regular updates from WDSPs, the update and participation overhead for WDSPs should be held within reasonable bounds of what the WDSP should do to support regular access to its information. Furthermore, given that WDSPs provide directory service information with an eye to value-added service, wherever possible End-Users should be redirected to the WDSP responsible for individual directory service entries for final and further information. 2.3 DAG-System Requirements In order to address the requirements of End-Users and WDSPs, the DAG system itself has certain design constraints that must be taken into account. The system must be implementable/operational by Dec 31/98 -- which implies that it must be designed and constructed with already extant technologies. The System will have certain requirements for participation -- e.g., 7x24 WDSP availability. In terms of scaling, the system should be able to handle 8M records at the outset, with a view to handling larger information systems in the future. The system must also be capable of extension to other, related applications (e.g., serving security certificate information). 3.0 Functional Specification In the TISDAG pilotservice we have decided to apply some limitations as to what is specified for the DAG/IP. These limitations are presented in this text in the following manner: TISDAG: This is a TISDAG comment 3.1 Overview The conceptual environment of the DAG system can be described in three major components: - client access software for end-users - the DAG system core - WDSP directory service software Daigle & Hedberg Informational [Page 9]  RFC 2967 TISDAG October 2000 This is illustrated in Figure 3.1 The DAG (Directory Access Gateway) is the infrastructural core of the service; it maintains the necessary data and transformation facilities to permit the smooth connection of diverse directory service Client Software to the existing WDSPs' directory servers. The key challenges in designing this portion of the system are: Quantity of data -- the quantity of whitepages information that will be made available, and diversity of its sources (different WDSPs) introduce challenges in terms of finding a structure that will allow efficient searching, and facilitate the timeliness of updating the necessary information. Multiplicity of access protocols -- in order to support the use of existing whitepages-aware software with a minimum of perturbation, the DAG system will have to present a uniform face in several different access protocols, each with its own information search and representation paradigm. This specification will outline the following areas: - the functioning of the DAG core itself - the interface between the DAG core and End-Users' Directory Service Access software - the interface between the DAG core and Directory Services Servers 3.2 The DAG Core In order to reduce the quantity of data the DAG itself must maintain, and to keep the maintenance of the whitepages information as close as possible to the source of information (the WDSPs themselves), the DAG will only maintain index information and will use 'query routing' to efficiently refer End-User queries to WDSPs for search refinement and retrieval of information. Although originally developed for the Whois++ protocol, query routing is being pursued in a protocol- independent fashion in the IETF's FIND WG, so the choice of this approach does not limit the selection and support of whitepages access protocols. The DAG will look after pursuing queries for access protocols that do not support referral mechanisms. In order to achieve the support of multiple access protocols and differing data paradigms, the DAG will be geared to specifically support a limited set of whitepages queries. Daigle & Hedberg Informational [Page 10]  RFC 2967 TISDAG October 2000 +---------+ @ + ->| | -+- /|Protocol| | | / | / +---------+ / / | 'B' + | / | |<- +-------+ | | O | | | | -+- | |<--------->| | | | | Protocol | | / | | 'A' | |<- +-------+ | |Protocol | | + | 'A' +---------+ @ | | | -+- | ->| | | | +---------+ / + The End Client DAG Directory Directory Users Software System Server Service Core Software Providers Figure 3.1 The role of the DAG system 3.3 Client Interface The DAG will respond to End-User queries in - e-mail (SMTP) - WWW (HTTP) - LDAPv2 - Whois++ - LDAPv3 The DAG will provide responses including the agreed-upon data. For access protocols that can handle referrals, responses will be data and/or referrals in that query protocol. These are Whois++ and LDAPv3. N.B.: the LDAPv3 proposal defines a referral as a URL; no limitation is placed on the access protocol. However it cannot be assumed that all clients will be able to handle all access protocols, so only referrals to LDAPv3 servers will be returned. Daigle & Hedberg Informational [Page 11]  RFC 2967 TISDAG October 2000 3.3.1 Acceptable User Input User Input is defined in terms of - Searchable Attributes - Matching semantics - Character sets These, in conjunction with the DAG schema, defined in Appendix A, form the basis of the required query expression. Individual queries are discussed in more detail in the Client Access Point (DAG-CAP) component descriptions for supported protocols. Supported Query Types The DAG system is designed to support fragment-matching queries on a limited set of data attributes -- 'Name', 'Organizational Role', 'Organization', and 'Locality'. The selected permissible query combinations of attributes are listed in Table 3.1. From the table it can be seen that not all combinations of the three attributes are supported -- only those that are needed for the desired functionality. Symbol Description ------- ----------- N Name NL Name + Locality NO Name + Organization NOL Name + Organization + Locality RO Role + Organization ROL Role + Organization + Locality Table 3.1 DAG-supported queries The RO and ROL queries are separated from the rest as they are searches for 'virtual' persons -- roles within an organization (e.g., president, or customer service desk) for which one might want to find contact information. Matching Semantics As befits the individual client query protocols, more string matching expressions may be provided. The basic semantics of the DAG expect the following to be available in all client access software (as relevant): Daigle & Hedberg Informational [Page 12]  RFC 2967 TISDAG October 2000 - Full word, exact match - Word substring match (E.g., 'cat' would match 'scatter') - Case-sensitive and case-insensitive matching TISDAG: LDAP/X.500, supports case-sensitivity as such but some of the most used attributes, such as the commonName attribute, are defined in the standard to be of the case-insensitive attributetypes. The impact on the DAG system is that even if the index collected from a LDAP/X.500 server might have upper and lower case letters in the tokens, they can not be handled as such since that would be inferring meaning in something which is natively regarded as meaningless. The conclusion of the above is that The Referral Index should be case-insensitive and case- sensitivity should be supported by the SAPs if the native access protocol supports it. Character Sets Wherever possible, the DAG System supports and promotes the use of Unicode Version 2.0 for character sets (see [21]) specifically the UTF-8 encoding (see Appendix A.2 of [21] or [20]) Accommodation is made, where necessary, to support the deployed base of existing software. Specifically: DAG/IP: All internal communications using the DAG/IP are carried out in UTF-8. TISDAG: not just UTF-8, but UTF-8 based on composed UNICODE version 2 character encodings. DAG-CAP input: Where specific access protocols permit selection of character sets, DAG-CAPs must support UTF-8. They may additionally support other anticipated character set encodings. DAG-SAP communications with WDSPs: Where specific access protocols permit selection of character sets, DAG-SAPs must support UTF-8 and use UTF-8 whenever the remote WDSP supports it. They may additionally support other character set encodings. CIP Index Objects: The Index Objects supplied by the WDSPs to the DAG system shall contain data encoded in UTF-8. TISDAG: The same limitation as for DAG/IP, that is the basic data should be UTF-8 encoded composed UNICODE version 2 character encodings. Daigle & Hedberg Informational [Page 13]  RFC 2967 TISDAG October 2000 3.3.2 Data Output Spec Schema Definition The schema used for the DAG service is defined in Appendix A. This is a very basic information schema, intended to carry the necessary information for the DAG service, and not more. Although generic 'whitepages' schema definitions do exist the more sophisticated and detailed the information presentation, the more difficult it is to map the schema seamlessly across protocols of different paradigms. Thus, the 'KISS' ('Keep it simple, sir') principle seems appropriate here. Individual DAG-CAPs define how they express this schema. Referral Definition For client access protocols that make use of the concept of referrals, DAG-CAP definitions will define the expression of referrals in those protocols. The DAG/IP defines the expression of referrals (see Appendix C). Error conditions Each DAG-CAP may provide more detailed error messages, but will define minimally the support for the following error conditions: - unrecognized query - too many hits Apart from these errors, the DAG-CAP may choose to refuse a query by redirecting the end-user to a different DAG-CAP of the same protocol. 3.4 Directory Server Interface The DAG will use the Common Indexing Protocol (CIP) server-server protocol to obtain updated index objects from WDSPs. For query- routing purposes, WDSPs are expected to provide Whois++, LDAPv2 or LDAPv3 interface to their data (although their preferred access may be something completely different). N.B.: In the responses from the technical survey, all respondents currently provide access to their service in one of these protocols. In order to provide a useful and uniform service, WDSPs are expected to provide 7x24 access to their whitepages information. WDSPs are also expected to implement operations, administration, maintenance, and provisioning processes designed to minimize service down time for both planned and unplanned administration and maintenance activities. Daigle & Hedberg Informational [Page 14]  RFC 2967 TISDAG October 2000 4.0 Architecture 4.1 Software Components The conceptual architecture of the DAG is represented in Figure 4.1. General architectural specifications are described below, followed by individual component specifications Sections 5.5 through 5.12. 4.1.1 Internal Communications Communications between components of the DAG will be by TCP/IP connections, using the DAG-Internal Protocol (DAG/IP). DAG/IP is used by DAG-CAPs to communicate with the Referral Index and DAG-SAPs. Thus, the DAG/IP defines - the DAG-CAPs' range of query ability in the Referral Index (to gather referrals in response to the end-user's requests) - the responses (and their formats) of the Referral Index to the DAG-CAP requests - the DAG-CAPs' range of query ability to the DAG-SAPs for pursuing referrals when the DAG-CAP needs to do chaining for the client access software - the responses (and their formats) of the DAG-SAPs to the DAG-CAPs. The detail of the planned DAG/IP is given in Appendix C. The detail of the DAG-CAP--Referral Index and DAG-CAP--DAG-SAP interactions is given in the definitions of individual DAG-CAPs and DAG-SAPs, below (Sections 5.5 through 5.12). 4.1.2 Referral Index The Referral Index is responsible for maintaining the index of WDSP information, and providing a list of reasonable referrals in response to DAG-CAP search requests. These 'referrals' provide pointers to identify WDSPs that may have information that matches the end-user's query. 4.1.3 DAG-CAPs Individual DAG-CAPs are responsible for providing a particular client access protocol interface to the DAG service. DAG-CAPs receive end- user queries in a particular query access protocol, convert the request into a query for the Referral Index ( i.e., expressed in DAG/IP), and then convert the Referral Index's response into a form that is appropriate for the client access protocol. This may mean passing back the referrals directly, calling on DAG-SAPs to do the work of translating the referral into results ('chaining'), or a combination of both. Daigle & Hedberg Informational [Page 15]  RFC 2967 TISDAG October 2000 +-------------------------------------+ |+====+ | HTTP <-->+| |<------+ (Full chaining) | || | | | |+====+ | | | | +----+| | | Referral-->| || | | Result <--| |+<--> Whois++ | | +----+| |+====+ | | SMTP <-->+| |<------+ (Full chaining) | || | | | |+====+ | | | | +----+| | | Referral-->| || | | Result <--| |+<--> LDAPv2 | | +----+| |+====+ | | Whois++<-->+| |<------+ (Chain LDAPv2/3) | || | | | |+====+ | | | | +----+| | | Referral-->| || | | Result <--| |+<--> LDAPv3 | | +----+| |+====+ | | LDAPv2 <-->+| |<------+ (Full chaining) | || | | | |+====+ | | | | | |+====+ | | LDAPv3 <-->+| |<------+ (Chain Whois++) | || | | | |+====+ | | | | | | v | | +-----------------------+ | | | Referral Index |<---------------> Common | | | | Indexing Protocol | +-----------------------+ | (CIP) +-------------------------------------+ All internal communications are in DAG/IP. Figure 4.1 Conceptual Architecture of the DAG Daigle & Hedberg Informational [Page 16]  RFC 2967 TISDAG October 2000 4.1.4 DAG-SAPs Individual DAG-SAPs are called upon (by DAG-CAPs) to take DAG- generated referrals and pursue them -- issuing the indicated query at the specified WDSP service. Results from individual WDSPs are converted back into DAG/IP-specific format for the DAG-CAP that made the request. Each DAG-SAP is responsible for handling referrals to WDSPs of a particular protocol (e.g., LDAPv2, Whois++, etc). 4.2 Important Architectural Notes This section notes some of the thinking that has driven the architectural and software design specification for the DAG system. This helps to provide the context in which to understand the software specifications that follow, and should give clues for the eventual extension of the DAG system. This section also acts, in some ways, as an FAQ (Frequently Asked Questions) section, as the content is shaped by questions received during the tech spec development phase. It attempts to illuminate context that may not otherwise be apparent on a first reading of the software specifications. 4.2.1 2 Distinct Functions: Referrals and Chaining At all times, it must be kept in mind that the primary function of the DAG system is to provide users with referrals to WDSP services that may have the information they seek. Since it is the case that not all supported client protocols can handle referrals, the DAG system also provides a chaining service to pursue referrals that the user's client software cannot handle itself. This chaining service does attempt to match the user's query against data from WDSPs, but this is to be seen as a secondary, or support function of the DAG system. In the perfect future, all access protocols will be able to handle all referrals! 4.2.2 Limited Query and Response Semantics The DAG system does not attempt to be a chameleon, or the ultimate whitepages query service. It focuses on providing referrals for information on the limited number of query types outlined in the functional specifications of the DAG service. This makes the DAG system a good place to start a search, but refinements and detailed inquiries are beyond its scope. 4.2.3 Visibility Given the limited query syntax of the DAG system it will not always be possible to exactly match a query posed to a CAP into a query posed to a SAP. This will have the effect that for instance a LDAPv2 Daigle & Hedberg Informational [Page 17]  RFC 2967 TISDAG October 2000 client that issues a query to the DAG system which by the DAG system is chained to a LDAP server might not get the same results as if the client where directly connected to the server in question. 4.2.4 Richness of Query semantics Even the limited query syntax of the DAG system is capable of expressing queries that might NOT be possible to represent in the access protocols to the WDSPs. In these cases the DAG-SAP either can refuse the query or try to emulate it. 4.2.5 N+M Protocol Mappings As part of the chaining service offered by the DAG system, a certain amount of mapping between protocols is required -- in theoretical terms, there are 'N' allowable end-user query access protocols, and 'M' supported WDSP server protocols. The architecture of the software is constructed to use a single internal protocol (the DAG/IP) and data schema, providing a common language between all components. Without this, each input protocol module (DAG-CAP) would have to be constructed to be able to handle every WDSP protocol -- NxM protocol mappings. This would make the system complex, and difficult to expand to include new protocols in future. 4.2.6 DAG-CAPs and DAG-SAPs are completely independent of each other For the above reasons, the DAG-CAP and DAG-SAP modules are intended to be completely independent of each other. A DAG-SAP responds to a query that is posed to it in the DAG/IP, without regard to the protocol of the DAG-CAP that passed the query. 4.2.7 The Role of the DAG-CAP Thus, the DAG-CAP is responsible for using the DAG/IP to obtain referral information and, where necessary, chained responses. Where necessary, it performs adjustments to accommodate the differences in semantics between the DAG/IP and its native protocol. This might involved doing post-filtering of the results returned by the DAG-SAPs since the query issued in DAG/IP to the DAG-SAP might be 'broader' then the original query. Thus, the DAG-CAP 'knows' only 2 protocols: its native protocol, and the DAG/IP. Daigle & Hedberg Informational [Page 18]  RFC 2967 TISDAG October 2000 4.2.8 The Role of the DAG-SAP Similarly, the DAG-SAP is responsible for responding to DAG/IP queries by contacting the designated WDSP server. Where necessary, it performs adjustments to accommodate the differences in semantics between the DAG/IP and its native protocol. These adjustments might mean that, as a consequence, the DAG-SAP will receive results that do not match the original query. In such cases the DAG-SAP should attempt to do post-pruning in order to reduce the mismatch between the original query and the results returned. Thus, the DAG-SAP 'knows' only 2 protocols: its native protocol, and the DAG/IP. 4.2.9 DAG/IP is internal No module outside of the DAG system should be aware of the DAG/IP's construction. End-users use the query protocols supported by DAG- CAPs; WDSPs are contacted using the query protocols supported in the DAG-SAPs. 4.2.10 Expectations The expectation is that the DAG system, although defined as a single construct, will operate by running modules on several different, perhaps widely distributed (in terms of geography and ownership), computers. For this reason, the DAG/IP specified in such a way that it will operate on inter-machine communications. 4.2.11 Future Extensions The DAG system architecture was constructed with a specific view to extensibility. At any time, an individual component may be improved (e.g., the Mail DAG-CAP may be given a different query interface) without disrupting the system. Additionally, future versions of the DAG system may support other access protocols -- for end-users, and for WDSPs. 5.0 Software Specifications 5.1 Notational Convention It is always a challenge to accurately represent text protocol in a printed document; when is a new line a 'newline', and when is it an effect of the text formatter? Daigle & Hedberg Informational [Page 19]  RFC 2967 TISDAG October 2000 In order to be adequately illustrated, this document includes many segments of protocol grammars, sample data, and sample input/output in a text protocol. In order to distinguish newlines that are significant in a protocol, the symbol is used. For example, This is an example of a very long line of input. There is only one newline in it (at the end), in spite of the fact that this document shows it spanning several lines of text. 5.2 DAG-CAP Basics 5.2.1 Functionality Every DAG-CAP must support the full range of DAG queries, as defined in 3.3.1. Each DAG-CAP accepts queries in its native protocol. Individual DAG-CAP definitions define the expected expression of the DAG queries in the native protocol. The DAG-CAP is then responsible for: - converting that expression into a query in the DAG/IP to obtain relevant referrals from the Referral Index. This might mean that parts of the original query are disregarded (e.g., if the query included attributes not supported by the DAG application, or if the query algebra was not supported by the DAG application); - returning referrals in the client's native protocol, where possible; - expressing the client query to the necessary DAG-SAPs, given the limitations mentioned above, to chain those referrals not usefully expressible in the client's native protocol; - possibly doing post-filtering on the DAG-SAP results; and - converting the collected DAG-SAP results for expression in the client's native protocol (and schema, where applicable). Each DAG-CAP defines the nature of the interaction with the end-user (e.g., synchronous or asynchronous, etc). Additionally, each DAG-CAP must be able to carry out the following, in order to permit load- limiting and load-balancing in the DAG system: - direct the client to a different DAG-CAP of the same type (for load-balancing) Daigle & Hedberg Informational [Page 20]  RFC 2967 TISDAG October 2000 - decline to return results because too many referrals were generated (to discourage data-mining). Ideally, this should include the generation of a message to refine the query in order to produce a more manageable number of referrals/replies. DAG-CAPs must be capable of accepting and respecting DAG-SAP service referrals (for DAG-SAP load-sharing). In protocols that permit it, the DAG-CAP should indicate to the end- user which services were unavailable for chaining referrals (i.e., to indicate there were parts of the search that could not be completed, and information might be missing). TISDAG: Any CAP that receives commands other than queries, like help, answers those on its own. A CAP should not pass any system command on to the RI. 5.2.2 Configuration It must be possible to change the expected address of the DAG-CAP by configuration of the software (i.e., host and port, e-mail address, etc). For DAG-CAPs that need to access DAG-SAPs for query chaining, for each type (protocol) of DAG-SAP that is needed, the DAG-CAP must be configurable in terms of: - at least one known DAG-SAP of every necessary protocol to contact - for each DAG-SAP, the host and port of the DAG-SAP software The DAG-CAPs must also be configurable in terms of a maximum number of referrals to handle for a user transaction (i.e., to prevent data mining, the DAG-CAP will refuse to reply if the query is too general and too many hits are generated at the Referral Index). The DAG-CAP must be configurable in terms of alternate DAG-CAPs of the same type to which the end-user software may be directed if this one is too busy. 5.2.3 Error handling Apart from error conditions arising from the operation of the DAG-CAP itself, DAG-CAPs are responsible for communicating error conditions occurring elsewhere in the system that affect the outcome of the user's query (e.g., in the DAG-RI, or in one or more DAG-SAPs). Daigle & Hedberg Informational [Page 21]  RFC 2967 TISDAG October 2000 If the DAG-CAP sends a query to the DAG-RI and receives an error message, it should attempt to match the the received DAG errorcode into its native access protocol's error codes. The same action is appropriate when the DAG-CAP is 'chaining' the query to one DAG-SAP. There are also occasions when the DAG-CAP may have to combine multiple errorcodes into a single expression to the user. When the DAG-CAP is 'chaining' the query through DAG-SAPs to one or more WDSPs, situations can arise when there is a mix of responsecodes from the DAG-SAPs. If this happens, the DAG-CAP should try to forward information to the end-user software that is as specific as possible, for instance which of the WDSPs has not been able to fulfill the query and why. See Appendix D for more information concerning error condition message mappings. 5.2.4 Pruning of results Since there is no perfect match between the query syntaxes of the DAG system on one hand and the different access protocols that the DAG- CAPs and DAG-SAPs supports on the other, there will be situations where the results a DAG-CAP has to collect is 'broader' then what would have been the case if there had been a perfect match. This might have adverse effects on the system to the extent that administrative limits will 'unnecessary' be exceeded on WDSPs or that the collected results exceeds the sizelimit of the DAG-CAP. Since the DAG-CAP is the only part of the DAG system that actually knows what the original query was, the DAG-CAP can prune the results received from the DAG-SAPs in such a way that the results presented to the client better matches the original question. 5.3 DAG-SAP Basics 5.3.1 Functionality Every DAG-SAP must support the full range of DAG queries, as defined in 3.3.1. Results must be complete DAG schemas expressed in well- formed DAG/IP result formats (see Appendix C). Each DAG-SAP accepts queries in DAG/IP and converts them to the native schema and protocol for which it is designed to proxy. The DAG-SAP is then responsible for - converting the query into the native schema and protocol of the WDSP to which the referral points. (If the query is not representable in the native protocol, it must return an error Daigle & Hedberg Informational [Page 22]  RFC 2967 TISDAG October 2000 message. If it is emulatable, the DAG-SAP can attempt emulate it by posing a related query to the WDSP and post-pruning the results received); - contacting that WDSP, using the host, port, and protocol information provided in the referral; - negotiating the query with the remote WDSP; - accepting results from the WDSP, possibly doing post-filtering on the result set; and - conveying the results back to the calling DAG-CAP using the DAG/IP and its schema. Note that this implicitly means that the DAG-SAP is responsible for chaining and pursuing any referrals it receives from WDSP services. The DAG-SAP returns only search results to the DAG-CAP that called it. 5.3.2 Configuration DAG-SAPs must be configurable to accept connections only from recognized DAG components. DAG-SAPs that have service limits must be configurable to redirect DAG-CAPs to alternate DAG-SAPs of the same type when necessary. 5.3.3 Error handling A DAG-SAP must translate error codes received from a WDSP server to DAG error codes according to Appendix D. 5.3.4 Pruning of results Since it might not be possible to exactly map a DAG query into a query in the access protocol supported by the a DAG-SAP, the DAG-SAP should try to translate it into a more general query (or if necessary into a set of queries). If so, the DAG-SAP must then prune the result set received before furthering it to the DAG-CAP. 5.3.5 Constraint precedence Some constraints, search and case, can appear both as local and global constraints. If this happens in a query then the local constraint specification overrides the global. For a query like the following: fn=leslie;search=exact and org=think:search=substring the resulting search constraint for 'fn=leslie' will be 'exact' while it for 'org=think' will be 'substring'. Daigle & Hedberg Informational [Page 23]  RFC 2967 TISDAG October 2000 5.4 The Referral Index 5.4.1 Architecture The Referral Index contains (only) information necessary to deliver referrals to DAG-CAPs based on the query types supported by the DAG itself. The Referral Index creates an index over these objects so that it can respond to DAG-CAP queries using the DAG/IP. The information is drawn directly from interactions with participating WDSPs' software, using the Common Indexing Protocol (CIP). 5.4.2 Interactions with WDSPs (CIP) WDSPs that wish to participate in the DAG system must register themselves (see Section 5.4.6). Once registered, the Referral Index will interact with the WDSPs using the Common Indexing Protocol as defined in [1], using the Index Object defined in Section 5.4.3. 5.4.3 Index Object Format The CIP index object type is based on the Tagged Index Object as defined in [12]. Appendix E details the expected content of the index objects as they are to be provided by the WDSPs. TISDAG: The tokens in the Tagged Index Object should be UTF-8 encoded composed UNICODE version 2 character encoding. 5.4.4 DAG-Internal I/O The Referral Index interacts with the rest of the DAG internal modules (DAG-CAPs) by listening for queries and responding in the DAG/IP (defined in Appendix C). 5.4.5 The Index Server The Referral Index must index the necessary attributes of the CIP index object in order to respond to queries of the form described in Table 3.1. The semantics of the chosen CIP object (defined in Appendix E) are such that a referral to a WDSP server is sent back if (and only if) - the index object of the WDSP contains all the tokens of the query, in the attributes specified, according to the logic of the DAG/IP query, and - all of those tokens are found with a common tag. Daigle & Hedberg Informational [Page 24]  RFC 2967 TISDAG October 2000 This means that a query for the name 'Fred Flintstone' (2 tokens) will yield a referral to a server that has a record for 'Fred Amadeus Flintstone', but not to a WDSP with 2 differently tagged records, for 'Fred Amadeus' and 'Julie Flintstone'. Depending on the access protocol being used and the original end-user query, the referral to the WDSP with 'Fred Amadeus Flintstone' may yield a successful result, or it may not. But, it is known that the other WDSP would not have yielded successful searches. That is, the referral approach may yield false-positive results, but will not miss appropriate WDSPs. 5.4.6 Configuration The Referral Index must provide the ability to register interested WDSPs, as outlined in Appendix E. The Referral Index must be able to configure the port for DAG/IP communications. Also, it must be configurable to recognize only registered DAG-CAPs. 5.4.7 Security The Referral Index will accept queries only from recognized (registered) DAG-CAPs. This will reduce 'denial of service' attack types, but is also a reflection on the fact that the Referral Index uses the DAG/IP, (i.e., internal) protocol, which should not be exposed to non-DAG software. The Referral Index must be able to use authenticated communication to receive data from WDSPs (see Appendix E). 5.5 Mail (SMTP) DAG-CAP This is the default Mail DAG-CAP. More sophisticated ones could certainly be written -- e.g., for pretty-printed output, or for handling different philosophies of case-matching. This DAG-CAP has been designed on the assumption that mail queries will be human-generated (i.e., using a mail program/text editor), as opposed to being queries formulated by software agents. The input grammar should therefore be simple and liberal in acceptance of variations of whitespace formatting. Daigle & Hedberg Informational [Page 25]  RFC 2967 TISDAG October 2000 5.5.1 Mail DAG-CAP Input Mail DAG-CAP input is expected to be a regular or MIME-encoded (see [9] and [10]) SMTP mail message, sent to an advertised mail address. The mail DAG-CAP parses the message and replies to it with a MIME- encoded message containing the results of the DAG search. One query is accepted per e-mail message -- text after a single valid query has been read is simply ignored. The body of the query message must follow the syntax defined below. Note that all input control terms ('type=', 'name=' etc) are shown in lower case for convenience, but could be upper case or mixed case on input. mailquery = [mnl] [controls] mnl terms mnl controls = [msp] 'searchtype' [msp] '=' [msp] ( matchtype / casetype / matchtype msp casetype / casetype msp matchtype / ) matchtype = 'substring' / 'exact' ; default: substring casetype = 'ignore' / 'sensitive' ; default: ignore terms = n / n-l / n-o / n-o-l / r-o / r-o-l n = n-term n-l = ( n-term l-term / l-term n-term) n-o = ( n-term o-term / o-term n-term ) n-o-l = ( n-term o-term l-term / n-term l-term o-term / l-term n-term o-term / l-term o-term n-term / o-term l-term n-term / o-term n-term l-term ) r-o = ( r-term o-term / o-term r-term ) r-o-l = ( r-term o-term l-term / r-term l-term o-term / l-term o-term r-term / l-term r-term o-term / o-term l-term r-term / o-term r-term l-term ) n-term = [msp] 'name' [msp] '=' [msp] string mnl o-term = [msp] 'org' [msp] '=' [msp] string mnl Daigle & Hedberg Informational [Page 26]  RFC 2967 TISDAG October 2000 l-term = [msp] 'loc' [msp] '=' [msp] string mnl r-term = [msp] 'role' [msp] '=' [msp] string mnl string = ISO-8859-1 or UTF-8 except nl and sp> msp = 1*(sp) sp = ' ' mnl = 1*(nl) nl = The following are valid mail queries: Example 1: searchtype = name = thinking cat Example 2: searchtype = exact ignore name=thinking cat Example 3: role=thinking cat org =space colonization Example 4: name=thinking cat My signature line follows here in the most annoying fashion Note that the following are not acceptable queries: Example 5: searchtype= exact substring name = thinking cat Example 6: name=thinking cat org= freedom fighters anonymous Daigle & Hedberg Informational [Page 27]  RFC 2967 TISDAG October 2000 In Example 5, two conflicting searchtypes are given. In Example 6, no linebreak follows the n-term. 5.5.2 Translation from Mail query to DAG/IP Querying the Referral Index A key element of translating from the Mail DAG-CAP input into the DAG/IP query format is to 'tokenize' the input terms into single token elements for the DAG/IP query. For example, the n-term name= thinking cat is tokenized into 2 n-tokens: thinking cat which are then mapped into the following in the DAG/IP query (dag-n- terms): FN=thinking and FN=cat The same is true for all r-terms, l-terms and o-terms. The primary steps in translating the mail input into a DAG/IP query are: translate quoted-printable encoding, if necessary translate base64 encoding, if necessary tokenize the strings for each term construct the DAG/IP query from the resulting components, as described in more detail below DAG/IP constraints are constructed from the searchtype information in the query. dag-matchtype = 'search=' / 'search=substring' ; if matchtype not ; specified dag-casetype = 'case=ignore' / ; if casetype not ; specified or ; casetype=ignore 'case=consider' ; if casetype=sensitive constraints = ':' dag-matchtype ';' dag-casetype The terms for the DAG/IP query are constructed from the tokenized strings from the mail input. Daigle & Hedberg Informational [Page 28]  RFC 2967 TISDAG October 2000 dag-n-terms = 'FN=' n-token 0*( ' and FN=' n-token) dag-o-terms = 'ORG=' o-token 0*( ' and ORG=' o-token) dag-l-terms = 'LOC=' l-token 0*( ' and LOC=' l-token) dag-r-terms = 'ROLE=' r-token 0*( ' and ROLE=' r-token) This means that the relevant DAG/IP queries are formulated as one of two types: dagip-query = ( ( ( n-query / nl-query / no-query / nol-query ) [' and template=DAGPERSON']':' dag-matchtype ';' dag-casetype) / ( ( ro-query / rol-query ) [' and template=DAGORGROLE']':' dag-matchtype ';' dag-casetype) ) n-query = dag-n-terms nl-query = dag-n-terms ' and ' dag-l-terms no-query = dag-n-terms ' and ' dag-o-terms nol-query = dag-n-terms ' and ' dag-o-terms ' and ' dag-l-terms ro-query = dag-r-terms ' and ' dag-o-terms rol-query = dag-r-terms ' and ' dag-o-terms ' and ' dag-l-terms The examples given earlier are then translated as follows. Example 1: FN=thinking and FN=cat:search=substring;case=ignore Example 2: FN=thinking and FN=cat:search=exact;case=ignore Example 3: ROLE=thinking and ROLE=cat and ORG=space and ORG=colonization:search=substring;case=ignore Querying a DAG-SAP In querying a DAG-SAP (irrespective of the protocol of that DAG-SAP), the DAG/IP query must include information about the target WDSP server. This information is drawn from the Referral Index SERVER- TO-ASK referral information, and is appended to the query as specified in Appendix C): Daigle & Hedberg Informational [Page 29]  RFC 2967 TISDAG October 2000 ':host=' quoted-hostname ';port=' number ';server-info=' quoted-serverinfo ';charset=' charset where the response from the Referral Index included: '# SERVER-TO-ASK ' serverhandle nl ' Server-info: ' serverinfo nl ' Host-Name: ' hostname nl ' Host-Port: ' number nl ' Protocol: ' prot nl ' Source-URI: ' source nl ' Charset: ' charset nl '# END' nl and the 'quoted-hostname' and 'quoted-serverinfo' are obtained from 'hostname' and 'serverinfo' respectively, by quoting the DAG/IP special characters. For example, the referral # SERVER-TO-ASK dagsystem01 Server-info: o=thinkingcat, c=se Host-Name: thinkingcat.com Host-Port: 2839 Protocol: ldapv2 Source-URI: http://www.thinkcat.com Charset: T.61 # END would yield the addition :host=thinkingcat.com;port=2839;server-info=o=thinkingcat, c=se;charset=T.61 in its query to an LDAPv2 DAG-SAP. (N.B.: See Appendix C for further definitions of the terms used in the SERVER-TO-ASK response). Note that it is the DAG-SAP's responsibility to extract these terms from the query and use them to identify the WDSP server to be contacted. See the individual DAG-SAP definitions, below. Daigle & Hedberg Informational [Page 30]  RFC 2967 TISDAG October 2000 5.5.3 Chaining queries in Mail DAG-CAP The Mail DAG-CAP has to chain all referrals -- to the Whois++ DAG- SAP, LDAPv2 DAG-SAP, or LDAPv3 DAG-SAP as appropriate for the referral. 5.5.4 Expression of results in Mail DAG-CAP The results message is sent to the 'Reply-To:' address of the originating mail, if available (see [4] for appropriate interpretation of mail originator headers). The original query is repeated, along with the message-id. The remainder of the body of the mail message is the concatenation of responses from the DAG-SAP calls, each result having the WDSP's SOURCE URI (from the referral) appended to it, and the system messages also having been removed. At the end of the message, the WDSP servers that failed to respond (i.e., the DAG-SAP handling the referral returned the '% 403 Information Unavailable' message) are listed with their server-info. 5.5.5 Expression of Errors in Mail DAG-CAP If the mail DAG-CAP receives a message that is not parsable using the query grammar described above, it returns an explanatory message to the query mail's reply address saying that the query could not be interpreted, and giving a description of valid queries. If the number of referrals sent by the Referral Index is greater than the pre-determined maximum (for detecting data-mining efforts, or otherwise refusing over-general queries, such as 'FN=svensson'), the mail DAG-CAP will send an explanatory message to the query mail's reply address describing the 'over-generalized query' problem, suggesting the user resubmit a more precise query, and describing the list of valid query types. If the mail DAG-CAP receives several different result codes from the DAG-SAPs it should represent those in an appropriate manner in the response message. A mail DAG-CAP may redirect a connection to another mail DAG-CAP for reasons of load-balancing. This is done simply by forwarding the mail query to the address of the alternate mail DAG-CAP. Daigle & Hedberg Informational [Page 31]  RFC 2967 TISDAG October 2000 5.6 Web (HTTP) DAG-CAP 5.6.1 Web DAG-CAP Input The web DAG-CAP provides its interface via standard HTTP protocol. The general expectation is that the web DAG-CAP will provide a form page with radio buttons to select 'substring or exact match' and 'consider case or ignore case'. Other information (about name, role, organization, locality) is solicited as free-form text. The DAG-CAP receives queries via an HTTP 'post' method (the outcome of the form action for the page described above, or generated elsewhere). The rest of this section describes the variables that are to be expressed in that post. The actual layout of the page and most user interface issues are left to the discretion of the builder. Note that the Web DAG-CAP may be called upon to provide responses in different content encoding, and must therefore address the 'Accept- Encoding:' request header in the HTTP connection. Although the Web protocol, HTTP, is not itself capable of handling referrals, through the use of two extra variables this client is given the option of requesting referral information and then pursuing individual referrals through the Web DAG-CAP itself, as a proxy for those referrals. This is handled through the extra 'control variables' to request referrals only, and to indicate when the transaction is a continuation of a previous query to pursue a referral. There has been call to have a 'machine-readable' version of the search output. As HTML is geared towards visual layout, user agents that intend to do something with the results other than present them in an HTML browser have few cues to use to extract the relevant information from the HTML page. Also, 'minor' visual changes, accomplished with extensive HTML updates, can disrupt user agents that were built to blindly parse the original HTML. Therefore, provision has been made to return 'raw' format results. These are requested by specifying 'Accept-Content: application/whoispp- response' in the request header of the HTTP message to the HTTP DAG-CAP. Daigle & Hedberg Informational [Page 32]  RFC 2967 TISDAG October 2000 The variables that are expected are: transaction = 'new' / 'chain' ; default is 'new'. This ; should not be user-settable. It is used ; in constructed URLs resulttype = 'all' / 'referrals' ; default is 'all' matchtype = 'substring' / 'exact' casetype = 'case ignore' / 'case sensitive' n-term = string o-term = string l-term = string r-term = string host-term = string port-term = string servinfo-term = string prot-term = string ; the protocol of the referral string = / / 5.6.2 Translation from Web query to DAG/IP Querying a DAG-SAP Directly If the transaction variable is 'chain', the information in the POST is used to pursue a particular referral, not do a search of the Referral Index. The appropriate DAG-SAP (deduced from the prot-term) is contacted and issued the query directly. Results from this type of query are always full results (i.e., not referrals). Querying the Referral Index A key element of translating from the Web DAG-CAP input into the DAG/IP query format is to 'tokenize' the input terms into single token elements for the DAG/IP query. For example, the n-term name= thinking cat is tokenized into 2 n-tokens: thinking cat which are then mapped into the following in the DAG/IP query (dag-n- terms): FN=thinking and FN=cat Daigle & Hedberg Informational [Page 33]  RFC 2967 TISDAG October 2000 The same is true for the r-term, l-term and o-term. The primary steps in translating the HTTP input into a DAG/IP query are: translate encodings, if necessary tokenize the strings for each term construct the DAG/IP query from the resulting components, as described in more detail below DAG/IP constraints are constructed from the searchtype information in the query. dag-matchtype = 'search=' / 'search=substring' ; if matchtype not ; specified dag-casetype = 'case=ignore' / ; if casetype not ; specified or ; casetype='case ignore' 'case=consider' ; if casetype= ; 'case sensitive' constraints = ':' dag-matchtype ';' dag-casetype The terms for the DAG/IP query are constructed from the tokenized strings from the HTTP post input. dag-n-terms = 'FN=' n-token 0*( ' and FN=' n-token) dag-o-terms = 'ORG=' o-token 0*( ' and ORG=' o-token) dag-l-terms = 'LOC=' l-token 0*( ' and LOC=' l-token) dag-r-terms = 'ROLE=' r-token 0*( ' and ROLE=' r-token) This means that the relevant DAG/IP queries are formulated as one of two types: dagip-query = ( ( ( n-query / nl-query / no-query / nol-query ) [' and template=DAGPERSON']':' dag-matchtype ';' dag-casetype) / ( ( ro-query / rol-query ) [' and template=DAGORGROLE']':' dag-matchtype ';' dag-casetype) ) n-query = dag-n-terms Daigle & Hedberg Informational [Page 34]  RFC 2967 TISDAG October 2000 nl-query = dag-n-terms ' and ' dag-l-terms no-query = dag-n-terms ' and ' dag-o-terms nol-query = dag-n-terms ' and ' dag-o-terms ' and ' dag-l-terms ro-query = dag-r-terms ' and ' dag-o-terms rol-query = dag-r-terms ' and ' dag-o-terms ' and ' dag-l-terms Querying a DAG-SAP In querying a DAG-SAP (irrespective of the protocol of that DAG-SAP), the DAG/IP query must include information about the target WDSP server. This information is drawn from the Referral Index SERVER- TO-ASK referral information, and is appended to the query as specified in Appendix C: ':host=' quoted-hostname ';port=' number ';server-info=' quoted-serverinfo ';charset=' charset where the response from the Referral Index included: '# SERVER-TO-ASK ' serverhandle ' Server-info: ' serverinfo ' Host-Name: ' hostname ' Host-Port: ' number ' Protocol: ' prot ' Source-URI: ' source ' Charset: ' charset '# END' and the 'quoted-hostname' and 'quoted-serverinfo' are obtained from 'hostname' and 'serverinfo' respectively, by quoting the DAG/IP special characters. For example, the referral # SERVER-TO-ASK dagsystem01 Server-info: o=thinkingcat, c=se Host-Name: thinkingcat.com Host-Port: 2839 Protocol: ldapv2 Source-URI: http://www.thinkingcat.com Charset: T.61 # END would yield the addition Daigle & Hedberg Informational [Page 35]  RFC 2967 TISDAG October 2000 :host=thinkingcat.com;port=2839;server-info=o=thinkingcat, c=se;charset=T.61 in its query to an LDAPv2 DAG-SAP (N.B.: See Appendix C for further definitions of the terms used in the SERVER-TO-ASK response). Note that it is the DAG-SAP's responsibility to extract these terms from the query and use them to identify the WDSP server to be contacted. See the individual DAG-SAP definitions, below. 5.6.3 Chaining queries in Web DAG-CAP If the resulttype was 'all', all of the referrals received from the Referral Index are chained using the appropriate DAG-SAPs. If only referrals were requested, the Referral Index results are returned. 5.6.4 Expression of results in Web DAG-CAP text/html results The default response encoding is text/html. If the resulttype was 'all', the content of the chaining responses from the DAG-SAPs, without the system messages, is collated into a single page response, one result entry per demarcated line ( e.g., bullet item). The FN or ROLE value should be presented first and clearly. The SOURCE URI for each WDSP referral should be presented as an HREF for each of the WDSPs results. At the end of the message, the WDSP servers that failed to respond (i.e., the DAG-SAP handling the referral returned the '% 403 Information Unavailable' message) are listed with their server-info. If, however, the resulttype was 'referrals', the results from the Referral Index are returned as HREF URLs to the Web DAG-CAP itself, with the necessary information to carry out the query (including the 'HOST=', etc, for the referral). For example, if the original query: n-term='thinking cat' resulttype='referrals' drew the following referral from the Referral Index: # SERVER-TO-ASK DAG-Serverhandle Server-Info: c=se, o=tce Daigle & Hedberg Informational [Page 36]  RFC 2967 TISDAG October 2000 Host-Name: answers.tce.com Host-Port: 1111 Protocol: ldapv3 Source-URI: http://some.service.se/ Charset: UTF-8 # END the response would be an HTML page with an HREF HTTP 'POST' URL to the Web DAG-CAP with the following variables set: n-term='thinking cat' transaction='chain' servinfo-term='c=se, o=tce' host-term='answers.tce.com' port-term='1111' prot-term='ldapv3' The Source-URI should be established in the response as its own HREF URI. application/whoispp-response Results If Accept-Encoding: ' HTTP request header had the value 'application/whoispp-response', the content of the HTTP response will be constructed in the same syntax and attribute mapping as for the Whois++ DAG-CAP. If the resulttype was 'all', all the referrals will have been chained by the Web DAG-CAP, and the response will include only full data records. If the resulttype was 'referrals', then all referrals are passed directly back in a single response, in correct Whois++ referral format (conveniently, this is how they are formulated in the DAG/IP). Note that this will include referrals to LDAP-based services as well as Whois++ servers. 5.6.5 Expression of Errors in Web DAG-CAP A Web DAG-CAP may redirect a connection to another web DAG-CAP for reasons of load-balancing. This is done simply by using an HTTP redirect. Daigle & Hedberg Informational [Page 37]  RFC 2967 TISDAG October 2000 Standard Errors If the web DAG-CAP receives a message that is not parsable using the query grammar described above, it sends an explanatory HTML page saying that the query could not be interpreted, and giving a description of valid queries. If the number of referrals sent by the Referral Index is greater than the pre-determined maximum (for detecting data-mining efforts, or otherwise refusing over-general queries, such as 'FN=svensson'), the web DAG-CAP will send a page with an explanatory message describing the 'over-generalized query' problem, suggesting the user resubmit a more precise query, and describing the list of valid query types. If the web DAG-CAP receives more than one result code from the DAG- SAPs, it must represent them all in a appropriate manner in the response. application/whoispp-response Errors An invalid query is responded to with a simple text response with the error: '% 500 Syntax Error'. If too many referrals are generated from the Referral Index, the simple text response will have the message '% 503 Query too general'. 5.7 Whois++ DAG-CAP TISDAG: The system commands polled-for/-by should elicit the empty set as a return value until we better understand the implications of doing otherwise. 5.7.1 Whois++ DAG-CAP Input Input to the Whois++ DAG-CAP follows the Whois++ standard ([6]). Minimally, the Whois++ DAG-CAP must support the following queries: Query Type Expression in Whois++ ----------- ------------------------------------ N One or more 'name=' and template=USER NL One or more 'name=' and One or more 'address-locality=' and template=USER NO One or more 'name=' and one or more 'organization-name=' and template=USER Daigle & Hedberg Informational [Page 38]  RFC 2967 TISDAG October 2000 NOL One or more 'name=' and one or more 'organization-name=' and one or more 'address-locality=' and template=USER RO One or more 'org-role=' and one or more 'organization-name=' and template=ORGROLE ROL One or more 'org-role=' and one or more 'organization-name=' and one or more 'address-locality=' and template=ORGROLE Table 5.1 Allowable Whois++ Queries The following constraints must be supported for queries: 'search=' (substring / exact) 'case=' (ignore / consider) If no constraints are defined in a query the default is exact and ignore. For example, FN=foo and loc=kista and fn=bar is a perfectly valid Whois++ NL query for 'Foo Bar' in 'Kista'. 5.7.2 Translation from Whois++ query to DAG/IP Querying the Referral Index The Whois++ DAG-CAP formulates a DAG/IP query by forwarding the search terms received (as defined in Table 5.1). For example, the above query would be expressed as: FN=foo and LOC=kista and FN=bar and template=DAGPERSON Querying a DAG-SAP In querying a DAG-SAP (irrespective of the protocol of that DAG-SAP), the DAG/IP query must include information about the target WDSP server. This information is drawn from the Referral Index SERVER- TO-ASK referral information, and is appended to the query as specified in appendix C: ':host=' quoted-hostname ';port=' number ';server-info=' quoted-serverinfo ';charset=' charset Daigle & Hedberg Informational [Page 39]  RFC 2967 TISDAG October 2000 where the response from the Referral Index included: '# SERVER-TO-ASK ' serverhandle ' Server-info: ' serverinfo ' Host-Name: ' hostname ' Host-Port: ' number ' Protocol: ' prot ' Source-URI: ' source ' Charset: ' charset '# END' and the 'quoted-hostname' and 'quoted-serverinfo' are obtained from 'hostname' and 'serverinfo' respectively, by quoting the DAG/IP special characters. For example, the referral # SERVER-TO-ASK dagsystem01 Server-info: o=thinkingcat, c=se Host-Name: thinkingcat.com Host-Port: 2839 Protocol: ldapv2 Source-URI: http://www.thinkingcat.com/ Charset: T.61 # END would yield the addition :host=thinkingcat.com;port=2839;server-info=o=thinkingcat, c=se;charset=T.61 in its query to an LDAPv2 DAG-SAP. (N.B.: See Appendix C for further definitions of the terms used in the SERVER-TO-ASK response). Note that it is the DAG-SAP's responsibility to extract these terms from the query and use them to identify the WDSP server to be contacted. See the individual DAG-SAP definitions, below. 5.7.3 Chaining in Whois++ DAG-CAP The Whois++ DAG-CAP relies on DAG-SAPs to chain any non-Whois++ referrals (currently, the LDAPv2 and LDAPv3 DAG-SAPs). Daigle & Hedberg Informational [Page 40]  RFC 2967 TISDAG October 2000 5.7.4 Expression of results in Whois++ Results are expressed in Whois++ by collating the DAG/IP results received from DAG-SAPs (using the FULL response), and using the template and attribute mappings defined in Appendix B. For each result from a given referral, the SOURCE attribute is added, with the value of the SOURCE-URI from the referral. Any referrals to other Whois++ servers provided by the Referral Index are sent directly to the Whois++ client as follows: server-to-ask = '# SERVER-TO-ASK ' DAG-Serverhandle ' Server-Handle: ' SERVER-INFO ' Host-Name: ' HOST ' Host-Port: ' PORT ' Protocol: ' PROTOCOL '# END' where SERVER-INFO, HOST, PORT, PROTOCOL are drawn from the referral provided in the DAG/IP, and the SOURCE-URI information is lost. 5.7.5 Expression of Errors in Whois++ DAG-CAP As appropriate, the Whois++ DAG-CAP will express operational errors following the Whois++ standard. There are 4 particular error conditions of the DAG system that the DAG-CAP will handle as described below. When the Whois++ DAG-CAP receives a query that it cannot reply to within the (data) constraints of the DAG, it sends an error message and closes the connection. The error message includes % 502 Search expression too complicated If the number of referrals sent by the Referral Index is greater than the pre-determined maximum (for detecting data-mining efforts, or otherwise refusing over-general queries, such as 'FN=svensson'), the Whois++ DAG-CAP will send an error message and close the connection. The error message includes % 503 Query too general (N.B.: this is different from the 'Too many hits' reply, which does send partial results.) Daigle & Hedberg Informational [Page 41]  RFC 2967 TISDAG October 2000 A Whois++ DAG-CAP may redirect a connection to another Whois++ DAG- CAP for reasons of load-balancing. This is expressed to the end-user client software using the SERVER-TO-ASK response with appropriate information to reach the designated alternate DAG-CAP. If a Whois++ DAG-CAP receives several different response codes from DAG-SAPs it should try to represent them all in the response to the end-user client. The proposed mapping between DAG/IP response codes and Whois++ response codes are given in Appendix D. 5.8 LDAPv2 DAG-CAP 5.8.1 LDAPv2 DAG-CAP Input Input to the LDAPv2 DAG-CAP follows the LDAPv2 standard ([19]). Minimally, the LDAPv2 DAG-CAP must support the following queries (adapted from the ASN.1 grammar of the standard): BindRequest ::= [APPLICATION 0] SEQUENCE { version INTEGER (1 .. 127), name LDAPDN, authentication CHOICE { simple [0] OCTET STRING, krbv42LDAP [1] OCTET STRING, krbv42DSA [2] OCTET STRING } } BindResponse ::= [APPLICATION 1] LDAPResult SearchRequest ::= [APPLICATION 3] SEQUENCE { baseObject 'dc=se', scope wholeSubtree (2), derefAliases ENUMERATED { neverDerefAliases (0), derefInSearching (1), derefFindingBaseObj (2), derefAlways (3) }, sizeLimit INTEGER (0 .. maxInt), timeLimit INTEGER (0 .. maxInt), attrsOnly BOOLEAN, filter Filter, Daigle & Hedberg Informational [Page 42]  RFC 2967 TISDAG October 2000 attributes SEQUENCE OF AttributeType } Filter ::= CHOICE { and [0] SET OF Filter, or [1] SET OF Filter, not [2] Filter, equalityMatch [3] AttributeValueAssertion, substrings [4] SubstringFilter } SubstringFilter ::= SEQUENCE { type AttributeType, SEQUENCE OF CHOICE { initial [0] LDAPString, any [1] LDAPString, final [2] LDAPString } } Queries against attributes in the prescribed LDAP standard schema (see Appendix B) are accepted. N.B., this is a minimal set of supported queries, to achieve the basic DAG-defined queries. An LDAP DAG-CAP may choose to support more complex queries than this, if it undertakes to do the translation from the DAG/IP to the LDAPv2 client in a way that responds to the semantics of those queries. TISDAG: Since LDAPv2 didn't specify any characterset but relied on X.500 to do so, in practice several different charactersets are in use in Sweden today. That the LDAPv2 CAP has no way of knowing which characterset that are in use by a connecting client is a problem that the TISDAG project can not solve. Users of the DAG system will have to configure their specific client according to information on the TISDAG web page. That page provides very specific information (including port number) that can be given to LDAPv2 users. The LDAP DAG-CAP listening on the default port (389) will be the LDAPv3 one. Daigle & Hedberg Informational [Page 43]  RFC 2967 TISDAG October 2000 5.8.2 Translation from LDAPv2 query to DAG/IP Querying the Referral Index The essential stratagem for mapping LDAP queries into DAG/IP Referral Index queries is to tokenize the string-oriented LDAP AttributeValueAssertions or SubstringFilters and construct an appropriate DAG/IP token-oriented query in the DAG/IP. This will generalize the LDAP query and yield false-positive referrals, but should not miss any appropriate referrals. There are 3 particular cases to be considered: equalityMatch queries substring queries combination equalityMatch and substring queries TISDAG: If the LDAP filter contains a cn-term and no objectclass specification it is unclear if the search is for a person or a role. When this happens the DAG query should cover all bases and map the query into a query for both people and roles. EqualityMatch queries can be handled by simply tokenizing the AttributeValueAssertions, making one DAG/IP query term per token (using the appropriate DAGSchema attribute) and carrying out an exact match in the DAG/IP. Consider the following example, represented in the ASCII expression of LDAP Filters as described in [13]): (& (cn=Foo Bar)(objectclass=inetOrgPerson)) This query can be represented in the DAG/IP as FN='Foo' and FN='Bar':search=exact N.B. The search is set up to be 'case=ignore' (the DAG/IP's default) because the relevant LDAP schema attributes are all derivatives of the 'name' attribute element, which is defined to have a case insensitive match. If no objectclass were defined the query in DAG/IP would have been (FN='Foo' and FN='bar') or (ROLE='Foo' and ROLE='bar'):search=exact Daigle & Hedberg Informational [Page 44]  RFC 2967 TISDAG October 2000 inetOrgPerson is used as the objectclass in this and the following examples, although person or organizationalPerson could also have been used. This query will yield false-positive referrals; the original LDAP query should only match against records for which the 'cn' attribute is exactly the phrase 'Foo Bar', whereas the DAG/IP query will yield referrals any WDSP containing records that include the two tokens 'foo' and 'bar' in any order. For example, this DAG/IP query will yield referrals to WDSPs with records including: cn: Bar Foo cn: Le Bar Foo cn: Foo Bar AB LDAP substring queries must also be tokenized in order to construct a DAG/IP query. The additional point to bear in mind is that LDAP substring expressions are directed at phrases, which obscure potential token boundaries. Consequently, all points between substring components must be considered as potential token boundaries. Thus, the LDAP query (& (cn=black) (o=c*t) (objectclass=inetOrgPerson)) could be expressed as a DAG/IP query with 3 tokens, in a substring search: FN=black and ORG=c and ORG=t:search=substring This query will yield false-positive results as the tokenized query does not preserve the order of appearance in the LDAP substring, and it doesn't preserve phrase-boundaries. That is, ORG=c and ORG=t:search=substring will match tabacco which is not a match by the LDAP query semantics. Combined EqualityMatch and Substring queries need special attention. When an LDAP query includes both EqualityMatch components and substring filter components, the DAG/IP query to the Referral Index Daigle & Hedberg Informational [Page 45]  RFC 2967 TISDAG October 2000 can be constructed by following the same mechanisms of tokenization, but the whole search will become a substring search, as the DAG/IP defines only search types across the entire query for Referral Index queries. Thus, (& (cn=Foo Bar) (o=c*t) (objectclass=inetOrgPerson)) can be expressed as FN=Foo and FN=Bar and ORG=c and ORG=t:search=substring Alternatively, the LDAP DAG-CAP could conduct two separate queries and take the intersection (the logical 'AND') of the two sets of referrals returned by the Referral Index. Note that DAG/IP can accept phrases for searches -- the query FN=Foo bar (note the escaped space) is perfectly valid. However, it would match only those things which have been tokenized in a way that preserves the space, which is the empty set in the case of the data stored here. Querying a DAG-SAP It is never invalid to use the same substantive query to a DAG-SAP as was used to obtain referral information from the Referral Index. However, the over-generalization of these queries may yield excessive numbers of results, and will necessitate some pruning of results in order to match the returned results against the semantics of the original LDAP query. It is the LDAP DAG-CAP that is responsible for this pruning, as it is the recipient of the original query, and responsible for responding to its semantics. In concrete terms, when making the DAG/IP query which is to be sent to a DAG-SAP the above mentioned queries are still valid queries, but an alternative finer-grained query is also possible, namely: FN=foo and FN=bar and ORG=c;search=lstring and ORG=t;search=tstring Particularly in the case of the LDAPv2 DAG-CAP, however, there will be cause to use LDAP(v2/v3) DAG-SAPs. Since these DAG-SAPs also deal in phrase-oriented data, a less-over-generalized query can be passed to them: FN=Foo Bar:search=exact Daigle & Hedberg Informational [Page 46]  RFC 2967 TISDAG October 2000 In querying a DAG-SAP (irrespective of the protocol of that DAG-SAP), the DAG/IP query must include information about the target WDSP server. This information is drawn from the Referral Index SERVER- TO-ASK referral information, and is appended to the query as specified in Appendix C: ':host=' quoted-hostname ';port=' number ';server-info=' quoted-serverinfo ';charset=' charset where the response from the Referral Index included: '# SERVER-TO-ASK ' serverhandle ' Server-info: ' serverinfo ' Host-Name: ' hostname ' Host-Port: ' number ' Protocol: ' prot ' Source-URI: ' source ' Charset: ' charset '# END and the 'quoted-hostname' and 'quoted-serverinfo' are obtained from 'hostname' and 'serverinfo' respectively, by quoting the DAG/IP special characters. For example, the referral # SERVER-TO-ASK dagsystem01 Server-info: o=thinkingcat, c=se Host-Name: thinkingcat.com Host-Port: 2839 Protocol: ldapv2 Source-URI: http://www.thinkingcat.com Charset: T.61 # END would yield the addition :host=thinkingcat.com;port=2839;server-info=o=thinkingcat, c=se;charset=T.61 in its query to an LDAPv2 DAG-SAP. (N.B.: See Appendix C for further definitions of the terms used in the SERVER-TO-ASK response). Note that it is the DAG-SAP's responsibility to extract these terms from the query and use them to identify the WDSP server to be contacted. See the individual DAG-SAP definitions, below. Daigle & Hedberg Informational [Page 47]  RFC 2967 TISDAG October 2000 5.8.3 Chaining queries in LDAPv2 DAG-CAP The LDAPv2 DAG-CAP relies on DAG-SAPs to resolve every referral. 5.8.4 Expression of results in LDAPv2 As described above, results from DAG-SAPs will have to be post- processed in cases where the original query was generalized for expression in DAG/IP. Acceptable results are expressed in the LDAP search response: SearchResponse ::= CHOICE { entry [APPLICATION 4] SEQUENCE { objectName LDAPDN, attributes SEQUENCE OF SEQUENCE { AttributeType, SET OF AttributeValue } }, resultCode [APPLICATION 5] LDAPResult } where LDAPDN = DN / 'cn=' (FN/ROLE) [',o='ORG] ',dc=se' attributes = 'objectClass = inetOrgPerson', 'objectClass = top', 'objectClass = person' or 'objectClass = organizationalRole', as appropriate, and 'labeledURI = ' for each result from a given referral> (Where DN,FN,ORG and ROLE are the values from the DAG schema). I.e., where available, the entry's true DN is used; otherwise (e.g., for data coming from Whois++ servers), a reasonable facsimile is constructed. 5.8.5 Expression of Errors in LDAPv2 DAG-CAP As appropriate, the LDAPv2 DAG-CAP will express system responses following the LDAPv2 standard. Daigle & Hedberg Informational [Page 48]  RFC 2967 TISDAG October 2000 Appendix D gives the proposed mapping between DAG/IP response codes and LDAPv2 resultcodes. There are 4 particular error conditions of the DAG system that the DAG-CAP will handle as described below. When the LDAPv2 DAG-CAP receives a query that it cannot reply to within the (data) constraints of the DAG queries, it sends an error message and closes the connection. The error message includes the LDAPv2 resultCode: noSuchAttribute (for incorrect schema attributes) inappropriateMatching (when a match type other than those supported is used, e.g. approxMatch) unwillingToPerform (when the query is not one of the defined types) If the number of referrals sent by the Referral Index is greater than the pre-determined maximum (for detecting data-mining efforts, or otherwise refusing over-general queries, such as 'FN=svensson'), the LDAPv2 DAG-CAP will send an error message. The error message includes one of the following resultCodes: sizeLimitExceeded timeLimitExceeded An LDAPv2 DAG-CAP may redirect a connection to another LDAPv2 DAG-CAP for reasons of load-balancing. This is expressed to the end-user client software using the 'umich referral' convention to direct the client software to an alternate DAG-CAP by passing the URL in an error message. Since a LDAPv2 DAG-CAP only can send one resultcode back to a client; If a LDAPv2 DAG-CAP receives several different result codes from the DAG-SAPs it will have to construct a resultmessage that to some extent represents the combination of those. It is proposed that in these cases the following actions are taken: - All the response codes are collected - Each response code are translated into the corresponding LDAPv2 resultcode. - A resultcode is chosen to represent the collected response on the following grounds: If 'success' is the only resultcode represented after these steps the return that result code. If apart from 'success' there is one other resultcode represented return that other resultcode. Daigle & Hedberg Informational [Page 49]  RFC 2967 TISDAG October 2000 If apart from 'success' there are two or more resultcodes represented return the resultcode 'other'. 5.9 LDAPv3 DAG-CAP 5.9.1 LDAPv3 DAG-CAP Input Input to the LDAPv3 DAG-CAP follows the LDAPv3 definition (currently defined in [17]). Minimally, the LDAPv3 DAG-CAP must support the following queries (adapted from the ASN.1 grammar of the standard): BindRequest ::= [APPLICATION 0] SEQUENCE { version INTEGER (1 .. 127), name LDAPDN, authentication AuthenticationChoice } AuthenticationChoice ::= CHOICE { simple [0] OCTET STRING, -- 1 and 2 reserved sasl [3] SaslCredentials } SaslCredentials ::= SEQUENCE { mechanism LDAPString, credentials OCTET STRING OPTIONAL } BindResponse ::= [APPLICATION 1] SEQUENCE { COMPONENTS OF LDAPResult, serverSaslCreds [7] OCTET STRING OPTIONAL } SearchRequest ::= [APPLICATION 3] SEQUENCE { baseObject c=se, scope wholeSubtree (2) }, derefAliases ENUMERATED { neverDerefAliases (0), derefInSearching (1), derefFindingBaseObj (2), derefAlways (3) }, sizeLimit INTEGER (0 .. maxInt), timeLimit INTEGER (0 .. maxInt), typesOnly BOOLEAN, filter Filter, attributes AttributeDescriptionList } Daigle & Hedberg Informational [Page 50]  RFC 2967 TISDAG October 2000 Filter ::= CHOICE { and [0] SET OF Filter, or [1] SET OF Filter, not [2] Filter, equalityMatch [3] AttributeValueAssertion, substrings [4] SubstringFilter } SubstringFilter ::= SEQUENCE { type AttributeDescription, -- at least one must be present substrings initial [0] LDAPString, substrings any [1] LDAPString, substrings final [2] LDAPString} Queries against attributes in the proscribed LDAP standard schema (see Appendix B) are accepted. N.B., this is a minimal set of supported queries, to achieve the basic DAG-defined queries. An LDAP DAG-CAP may choose to support more complex queries than this, if it undertakes to do the translation from the DAG/IP to the LDAPv3 client in a way that responds to the semantics of those queries. 5.9.2 Translation from LDAPv3 query to DAG/IP Querying the Referral Index The essential stratagem for mapping LDAP queries into DAG/IP Referral Index queries is to tokenize the string-oriented LDAP AttributeValueAssertions or SubstringFilters and construct an appropriate DAG/IP token-oriented query in the DAGschema. This will generalize the LDAP query and yield false-positive referrals, but should not miss any appropriate referrals. There are 3 particular cases to be considered: equalityMatch queries substring queries combination equalityMatch and substring queries TISDAG: If the LDAP filter contains a cn-term and no objectclass specification it is unclear if the search is for a person or a role. When this happens the DAG query should cover all bases and map the query into a query for both people and roles. Daigle & Hedberg Informational [Page 51]  RFC 2967 TISDAG October 2000 EqualityMatch queries can be handled by simply tokenizing the AttributeValueAssertions, making one DAG/IP query term per token (using the appropriate DAGSchema attribute) and carrying out an exact match in the DAG/IP. Consider the following example, represented in the ASCII expression of LDAP Filters as described in [13]): (& (cn=Foo Bar)(objectclass=person)) This query can be represented in the DAG/IP as FN='Foo' and FN='Bar':search=exact N.B. The search is set up to be 'case=ignore' (the DAG/IP's default) because the relevant LDAP schema attributes are all derivatives of the 'name' attribute element, which is defined to have a case insensitive match. If no objectclass where defined the query in DAG/IP would have been (FN='Foo' and FN='bar') or ( ROLE='Foo' and ROLE='bar'):search=exact Although person is used as objectclass in this and the following examples, inetOrgPerson or organizationalPerson could also have been used. This query will yield false-positive referrals; the original LDAP query should only match against records for which the 'cn' attribute is exactly the phrase 'Foo Bar', whereas the DAG/IP query will yield referrals any WDSP containing records that include the two tokens 'foo' and 'bar' in any order. For example, this DAG/IP query will yield referrals to WDSPs with records including: cn: Bar Foo cn: Le Bar Foo cn: Foo Bar AB LDAP substring queries must also be tokenized in order to construct a DAG/IP query. The additional point to bear in mind is that LDAP substring expressions are directed at phrases, which obscure potential token boundaries. Consequently, all points between substring components must be considered as potential token boundaries. Daigle & Hedberg Informational [Page 52]  RFC 2967 TISDAG October 2000 Thus, the LDAP query (& (cn=black) o=c*t) (objectclass=person)) should be expressed as a DAG/IP query with 3 tokens, in a substring search: FN=black and ORG=c and ORG=t:search=substring This query will yield false-positive results as the tokenized query does not preserve the order of appearance in the LDAP substring, and it doesn't preserve phrase-boundaries. That is, ORG=c and ORG=t:search=substring will match tabacco which is not a match by the LDAP query semantics. Combined EqualityMatch and Substring queries need special attention. When an LDAP query includes both EqualityMatch components and substring filter components, the DAG/IP query to the Referral Index can be constructed by following the same mechanisms of tokenization, but the whole search will become a substring search, as the DAG/IP defines search types across the entire query. Thus, (& (cn=Foo Bar) (o=c*t) (objectclass=person)) can be ex