The discussion mining system generates knowledge discovery from discussion contents during face-to-face meetings. This previously developed system Nagao et al. (2005), shown in Figure 5, generates structured minutes for meetings semi-automatically and links them with audiovisual data. This system summarizes discussions using a personal device, which captures information, called the discussion commander. The created content is then viewed using the discussion browser mentioned later, which provides a search function that lets users browse the discussion details.
Recording and structuring discussions
Discussions in our meetings are automatically recorded and these meeting records are composed of structured multimedia contents including texts and videos. In the contents, meeting scenes are segmented based on discussion chunks. The segments of contents are connected with visual and auditory data corresponding to the segmented meeting scenes.
Previous studies on structuring discussions and supporting discussions by referring past structured discussion contents include IBIS and gIBIS Conklin and Begeman (1988) that consider semantic discussion structures. However, most studies that provide technology for discussions and minutes generations have focused on automatic recognition techniques for auditory and visual data. For example, Lee et al. (2002) proposed a method that records the participants’ actions using cameras and microphones and then produces indexed minutes using automatic recognition techniques. Chiu et al. (2001) integrated audio-visual information and information for presentation materials.
We analyze meetings not only with natural language processing to support the comprehension of arguments in a discussion but also form diversified perspectives using auditory and visual information in slides as well as other presentation content. We also use metadata to deal with discussion content. Overall, our discussion mining system supports the creation of minutes for face-to-face meetings, records the meeting environment with cameras and microphones, and generates meta-information that relates elements in the contents.
In addition, the system can graphically display the structure of a discussion to facilitate understanding of the minutes and encourage effective statements. Our discussion commander has some functionality for discussion facilitation such as pointing/highlighting some areas and underlining some texts in the presentation slides displayed on the main screen. We also developed a method to define visual referents in the presentation slides that are pointed and referred by meeting participants.
Our method can handle sharing and re-referring the visual referents. This method then contributes to finding central topics of the discussion chunks. A discussion chunk has a tree structure and it consists of participants’ utterances and relationships between two utterances. An utterance has one of two types: start-up and follow-up. The start-up type is assigned to the utterance when it introduces a new topic while the follow-up type is assigned when the utterance inherits the predecessor’s topic. The discussion content of a meeting has several discussion chunks that have tree structures of utterances as shown in Figure 6.
The summarization of discussion content is performed as follows:
Based on common visual referents in utterances included in discussion chunks, a graph structure is generated. Spreading activation is applied to the graph structure where external inputs are assigned based on marking agreeable/disagreeable utterances which are decided by using discussion commander. Highly activated utterances are selected as more significant elements of the content. The discussion browser allows the users to adjust some parameters such as the ratio of summary and the weight of marking. The whole system provides functions of generating and publishing multimedia meeting records and their in-depth search and summarization.
On-time visualization of discussion structures and histories of visual referents contributes to the facilitation of current discussion and modification of discussion structures by changing parent nodes of follow-up utterances and by re-referring previous visual referents. Such modification is performed using each participant’s discussion commander. The discussion commander also works for annotating agree or disagree attributes to the current utterance by pressing + or - buttons. The time of pressing the buttons, the user who pressed the buttons, and the target utterance are recorded and used for summarization. The target utterance of the agree annotation has a high-valued external input when spreading activation is performed.
Since our main mission is to train students’ discussion skills, the previous system was extended and new functions were added in order to obtain user-specific data such as the quality of statements and level of understanding the discussions, which led to the creation of the Leaders’ Saloon (Section ‘Leaders’ Saloon: a new physical-digital learning environment’).
Discussion browser
The information accumulated by the discussion mining system is presented synchronously in the discussion browser shown in Figure 7. This system consists of a video view, a slide view, a discussion view, a search menu, and a layered seek bar.
The discussion browser provides the function of searching and browsing discussion details in correspondence to the users’ requests. For example, when the participant of the meeting wants to refer to certain important previous discussions, the participant will search for the statements using keywords or the speakers’ names, and then browse the details of the statements in the search results. Users who did not participate in the meeting can search and browse the important meeting elements displayed in the layered seek bar by inquiring into discussions containing statements that form agreements by using the discussion commanders, or by surveying the frequency distributions of keywords.
Video view
The video view provides recorded videos of the meeting, including the participants, presenter, and screen. The participant video shows the scene of the speaker if the speaker is not a presenter or the whole span of the meeting room if the speaker is the presenter.
Discussion view
The discussion view consists of text forms, in which the contents of the discussion primarily constitute of information inputted by a secretary and relation links, which visualize the structure of the discussion. This view supports the understanding of the contents of the discussion, because the users can survey the structure of the discussion. The user can also tag the meeting contents for searching by selecting accurate tags from a tag cloud containing tags extracted from the text of statements and presentation materials.
Search menu
In the search menu, three types of search queries are available: speaker name, the target of the search (either the contents of the slide or the statement, or both), and keywords. The users will search for the necessary information using combined queries. The search results are shown in the layered seek bar (matched elements in the timeline are highlighted) and in the discussion view (discussions where the matched elements appear are highlighted).
Layered seek bar
The elements that compose a meeting content are displayed in the layered seek bar. Various bars are generated according to each type of element and it also presents the details. The left edge of each bar corresponds to the start time of the meeting, and the right edge corresponds to the end time. The discussion browser enables effective reuse of meeting contents. Additionally, summarization is possible by acquiring relevant discussion from links between statements. The entire operation history of the discussion media browser is saved in the database. This history is used for the personalization of meeting contents.
Importing discussion mining system into Leaders’ Saloon
We developed an extended version of the discussion mining system working at the Leaders’ Saloon. The discussion tables are used to operate and visualize discussion structures. The users also use discussion commanders and the previously described discussion mining system.
In this section, we explain two systems implemented on the discussion tables to visualize real information recorded by the discussion mining system: (1) discussion visualizer, a system to visualize the structure of an ongoing discussion, and (2) discussion reminder, a system to retrieve and visualize past discussions.
Discussion visualizer
The discussion visualizer shown in Figure 8 is a system to visualize the structure of meeting discussions shown in the discussion table (Section ‘Discussion table’). This visualizer consists of a meeting view, a slide list, a discussion segment view, and a discussion segment list.
The meeting view provides a preview of camera records showing the participants, a list of all attendances, and elapsed time of presentation. A list of slide thumbnails displayed in the presentation is also shown and the thumbnail of the currently displayed slide is emphasized in the slide list. Speakers can operate the slide show by selecting the thumbnail in this view using the touch panel.
The discussion segment view shows the information about the discussion segment, which contains the current statement. The texts of the start-up statement, which was the trigger of the discussion, and the parent statement of the current statement (if it is a follow-up statement) are shown at the upper side of this view. The structure of the discussion segment is shown at the bottom side of this view. Users can also make corrections of parent statements. Participants confirm the stream of discussion at the meeting through the discussion segment list. In this list, the nodes representing main topics are shown as rectangle nodes while the subtopics are shown as circle nodes. These discussion segment topics are displayed as a chain structure in the middle, the keywords of multiple discussion segments are displayed on the left, and the keywords of the main topics or subtopics are displayed on the right. Moreover, the nodes that involve questions and answers are represented by the specific character Q. The amount of agreements on the statements inputted by the discussion commanders is represented as a density of the color of the nodes. The icons are displayed next to the node containing the statements marked by discussion commanders. Therefore, it enables participants to confirm when important discussions occur.
There are various kinds of discussion segments created by the discussion mining system. For example, short segments with only comments on the presentation and long segments that contain a lot of statements as a result of a hot debate. There is also a possibility that the long discussion segments have follow-up statements whose content derives from the topic of the start statement. Thus, we think that the start statement is the root node of the discussion segment and some subtopics derive from this root node.
Discussion reminder
A review and sharing of previous discussion contents lead to a uniformed knowledge level among all participants, wherein low-level participants can make remarks actively. This will also prevent redundant discussion. From here, we can then think about topics from a new point of view and figure out solutions to problems that have not been solved due to lack of technology. Therefore, we develop a system to retrieve and browse past discussions on time, called discussion reminder.
There are two concerning issues in the development of the discussion reminder. One is an accurate understanding of discussion contents, and another issue is the quick retrieval of discussion contents preventing any disruption in the ongoing discussion. Unclear and inadequate sharing of discussion contents will inhibit the achievement of a uniformed knowledge level and will lead to misunderstandings and confusion. Thus, the discussion reminder provides a function to browse videos of past discussions for accurate understanding.
However, all of the participants need to interrupt the ongoing discussion for a review of discussion contents, thus it is desirable to finish the review in no time using the above method to find the things required in the audiovisual information. For an efficient review, the discussion reminder provides an interface to narrow down the browsing information, such as discussion content matched with queries, slides matched discussion content, and statements associated with matched slide, and to retrieve cooperatively by participants. A participant who notices the existence of the discussion, which he/she wants to review, inputs queries to the discussion reminder. Various types of information, such as names of presenters, dates of meetings, and keywords, are available as queries. The contents of retrieved results are displayed on the discussion table as shown in Figure 9.
Participants conduct various operations using the touch panel in this interface. This interface consists of a discussion content list, a slide list, and a discussion segment view. The discussion content list displays titles of the discussion contents, which contains the discussion matched queries. When a participant selects a title using the touch panel, slide thumbnails comprised in the selected discussion content are shown at the bottom of the slide list. Participants can preview the larger slide thumbnail at the top of the slide list.
The discussion segment view shows information about the discussion segments associated with the slide selected in the slide list. Examples of discussion segment information include structures of discussion segments, speaker’s ID, keywords of statement, and so on. In the discussion segment view, full text of the statement can also be previewed. Participants can browse videos in the video view displayed on the table from the start time of the selected statement in the discussion segment view.
Employing machine learning techniques
In this study, machine learning techniques are employed to obtain deep structures of presentation and discussion contents. Techniques like deep neural networks Bengio (2009) integrate several context information such as operation histories of users. By integrating the results of subject experiments on presentations and discussions, different methods to evaluate the quality of students’ intellectual activities and to increase their skills are discovered. The system tries to perform some consensus-building processes to make evaluation results appropriate for each student.