Knowledge graph (aka direct displays, cards, etc.) has become an increasingly common and important component in search engine result pages (SERPs). The objective of this new pilot task accepted to run at NTCIR-13 is to foster research on generating knowledge graphs that are optimised for facilitating users' actions (buying, booking, downloading, comparing, creating, etc.). To achieve this objective, we set two subtasks: Action Mining (AM) subtask and Actionable Knowledge Graph Generation (AKGG) subtask.
You can participate either of subtasks or both!
In this pilot task, we set two subtasks to achieve and advance the technologies related to actionable knowledge graph presentations that can be used for search engines.
For a given entity type (e.g., Place) and instance entity (e.g., "poland"), participants will be asked to find potential actions that can be taken (e.g., "visit poland", "buy a house in poland", "find weather in poland"). Participants are allowed to use any external resources (e.g., Action section in schema.org) to return a ranked list of potential actions. Up to 100 actions should be submitted for each query (pair of entity and its type). The format of each submitted action should contain verb and object (called also the modifier of a verb), where the object's length is limited to 50 characters. For example, in the above-mentioned case of "poland" as the entity instance of the Place type, "buy a house in poland" would be an action composed of a verb "buy" and object "a house in poland". Participants are allowed to submit up to three actions that share the same verb. Note that actions can sometimes lack objects (actions containing only a verb), in which case, the object is actually considered NULL. For example, for the entity "outlook express" of the type Product, "download" is an example of correct action that does not require any explicit object specified. This subtask can be seen as an open information extraction task, and allows us to accumulate a comprehensive set of actions that are related to a given entity type and entity instance. The returned actions will be assessed by crowdsourcing to be scored from 1 to 5. Actions will be evaluated not only based on their relevance but also based on diversity in order to prevent submissions of many similar actions (e.g., actions where verbs are synonyms or the objects have very similar meaning). For sample data see: http://ntcirakg.github.io/tasks.html
For a given search query, entity included in that query, the type of the entity, and action (e.g., "kyoto budget travel", "kyoto", location, "visit a temple"), participants will be asked to rank entity properties based on their relevance to the query. The query (input) can be ambiguous as in realistic search queries, and participants need to return the ranked list of relevant entity properties to create an actionable knowledge graph. Actions in the test queries will be taken from the outcomes of the Action Mining (AM) Subtask. Properties to be returned will be those defined as attributes of the entity type in schema.org vocabulary.
Effectiveness of the returned properties will be judged by pair-wise comparison performed by crowdsourcing for all pairs of submitted runs. We are currently considering to propose a new metric that can take preference based judgements into account, by applying Bradley-Terry model.
Participants may use any external sources for the subtasks. Potential document collection and knowledge base are here and here.
Please note:
Task participants must submit a paper to the NTCIR-13 Conference (unrefereed) and at least one member of each participating group must attend the conference (December 5-8, 2017, NII, Tokyo, Japan) to present their work.
Please visit task website for more information: http://ntcirakg.github.io/index.html
| Aug 24, 2016 | NTCIR-13 Kick-Off Event |
| Oct 1, 2016
Oct 15, 2016 |
AM: Dry run topics + training data release |
| Nov 1, 2016
Nov 15, 2016 |
AM: Dry run submissions due |
| Dec 1, 2016 | AM: Dry run results release |
| Dec 15, 2016
Dec 31, 2016 | Task registration due for AM subtask |
| AM: Formal run topics release | |
| AM: Formal run submissions due | |
| AM: Formal run results release | |
| AKGG: Dry run topics + sample data release | |
| AKGG: Dry run submissions due | |
| AKGG: Dry run results release | |
| Jun 1, 2017 | Task registration due for AKGG subtask |
| Jun 15, 2017 | AKGG: Formal run topics release |
| Jul 15, 2017 | AKGG: Formal run submissions due |
| Sep 1, 2017 | AKGG: Formal run results release Partial Overview paper release |
| Oct 1, 2017 | Partcipants paper due |
| Nov 1, 2017 | Camera-ready due |
| Dec 5, 2017 | NTCIR-13 Conference |